►
Description
Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Debuggers and eBPF: Bringing Debugging to Production - Derek Parker, Red Hat
Your program isn't working... now what?! When it comes to diagnosing misbehaving software a debugger is an essential tool in your developer toolbox. A good debugger is a valuable resource in your development environment, but what about in production? Often times inspecting a program running in your production environment with a debugger is a nonstarter. The overhead incurred when spying on your program can be too much to make this a viable path towards fixing your software. The big question is, does it have to be this way? Can we do better? I think the answer is yes, and that is the focus of this talk. I will explain how to get the best of both worlds: a solid source-level debugger and low-overhead debugging and tracing. This hybrid approach opens up new possibilities for resolving problems in production environments and increasing telemetry and insight in your software while incurring negligible performance overhead. Attendees will walk away with not only a deeper understanding of debuggers and eBPF but also practical information that they can bring back to their development workflow. This talk will be approachable to newcomers while also providing deep technical insights geared towards more advanced engineers.
A
All
right,
thank
you
very
much,
as
mentioned.
My
name
is
derek
parker
and
first
I
would
just
like
to
start
off
by
thanking
the
organizers
for
allowing
me
to
present
this
talk
here
and
for
everybody
in
the
audience
in
person,
in
virtual,
for
checking
this
talk
out,
it's
something
that
I'm
very
excited
about.
A
A
You
expect
some
sort
of
responses,
and
hopefully
you
figure
out
what's
going
on,
but
what
I
want
to
talk
about
today,
specifically,
is
more
spying
on
your
program,
just
kind
of
seeing
what
it's
doing
under
the
hood
from
a
user
space
perspective,
the
actual
functions
that
you've
written
and
doing
that
in
a
way.
That's
that's
performance
so
that
maybe
you
can
actually
spy
on
your
your
software.
That's
running
in
in
production.
A
So
we've
already
done
a
little
bit
of
introductions,
but
again
my
name
is
derek
parker.
I'm
a
senior
software
engineer
at
red
hat
where
I
work
on
upstream
go,
and
I
also
work
on
delve
as
well.
So
this
being
kind
of
my
experience
using
this
with
the
delve
debugger,
some
of
this
stuff
might
be
a
little
bit
ghost-centric,
but
the
bulk
of
it
is
about
how
the
tool
uses
ebpf.
A
A
Essentially
it's
process
trace
it's
a
feature
of
a
lot
of
unixy
systems,
there's
parallels
and
other
non-unix
systems
as
well,
but
it
essentially
provides
a
means
for
a
user
space
program
to
take
over
inspect
control,
another
user
space
program.
It's
basically
how
debuggers
work
underneath
the
hood.
A
However,
there's
a
problem
with
it
p
trace
is
pretty
slow,
and
I
mentioned
petraeus
specifically
in
this
talk,
but
sis
calls
are
slow,
is
really
what
it
comes
down
to
at
the
end
of
the
day
and
I'll
I'll
kind
of
dig
into.
Why
that
that
is
so
problematic
for
us,
but
p-trace
and
syscalls
kind
of
in
general
are
slow,
they're
very
slow.
A
So
I,
when
I
was
initially
starting
to
work
on
this
implementation,
I
did
a
little
bit
of
testing
and
some
measuring
and
I
wrote
just
kind
of
like
a
small
example
toy
program,
measured
how
long
it
took
to
execute
and
then
measured
how
long
it
took
to
execute
with
different
kinds
of
tracing
implementations
on
top
of
it.
So,
as
you
can
see,
the
program
execution
just
by
itself
is
around
23
microseconds
we're
talking
microseconds,
not
milliseconds,
we're
talking
like
extremely
extremely
quick.
A
So
with
the
ebpf
base
tracer,
you
can
see
it
balloons
up
a
decent
amount,
so
we
go
from
23
to
about
683,
but
but
still
remember
we're
talking
about
microseconds
like
this
is
this
is
not
this
is
you
know
a
decent
amount
of
overhead,
but
nothing
too
crazy.
Then
we
look
at
the
traditional
based
the
the
traditional
approach
of
p-trace
space
tracing
and
we
go
all
the
way
up
to
2.3
seconds
now
for
a
program
that
executes
in
that
small
amount
of
time
ballooning
up.
A
Essentially
what
it
comes
down
to
is
the
syscall
overhead,
the
user,
space
and
kernel
context.
Switching
gets
very,
very
expensive,
especially
when
you
have
to
do
those
kinds
of
operations,
multiple
times
per
trace,
so
say,
for
example,
you're
tracing
the
the
the
entry
point
and
the
exit
point
of
a
function.
A
So
our
solution
to
this
is:
let's
use
eppf.
Why
do
we
have
to
do
any
of
this
context?
Switching
can
we
we
can
do
better.
The
the
technology
is
there.
So
ebpf
turns
out
to
be
very
fast
again
when
we
look
at
the
benchmarking,
there
is
still
some
overhead
there,
but
but
in
terms
of
of
the
other
solutions
that
we
have
right
now,
it's
it's
pretty
negligible.
A
A
So
the
execution
of
those
programs
happens
very
quickly,
so
we're
not
kind
of
running
in
huge
loops
and
unconstrained
kind
of
behavior
and
that
single
program
that
single
stop
can
can
by
itself
gather
all
of
the
data
that
we
need
and
send
it
back
to
user
space
without
doing
multiple
points
of
contact.
Switching.
A
So
let's
talk
about
the
requirements
that
we
had
for
our
tracing
back
end,
because
I
think
this
is
where
it
gets
the
most
interesting.
In
my
opinion,
so
one
of
the
things
that
we
wanted
to
do
one
of
the
requirements
was
essentially
it
has.
It
has
to
maintain
parity
with
the
existing
tracing
implementation
right,
so
we
need
to
be
able
to
trace
arbitrary
functions.
That's
very
interesting
to
me,
because
a
lot
of
the
use
cases
for
ebpf
programs
are
usually
very
small
and
targeted.
A
You
kind
of
already
know
what
syscall
you're
going
to
be
attaching
to
or
what
thing
you're
going
to
be
inspecting.
So
it's
it
there's
not
a
lot
of
guesswork
there.
You
can
kind
of
build
in
a
lot
of
the
logic.
Just
within
your
program
we
for
for
go
programs
and
in
the
context
of
delve
we
need
to
be
able
to
retrieve
the
go
routine
id.
So
we
need
to
know
where
to
find
it
and
all
that
stuff.
A
So
let's
talk
a
little
bit
about
tracing
arbitrary
functions,
so
in
general,
in
order
to
make
this
stuff
work,
we
use
lib
bpf
and
lib
bpf
go
so
we've
heard
about
libby
pf
and
for
anybody
trying
to
experiment
with
ebpf
stuff
and
go
there's,
there's
a
bunch
of
different
frameworks,
but
we've
decided
to
go
the
route
of
using
libby
pf
go.
It's
worked
out
very
well
for
us.
A
A
So
initially
we
we
embed
the
ebpf
object
in
the
dell
binary.
This
is
something
that
I
think
is
really
cool.
It's
a
feature
of
the
go
programming
language
in
general,
but
it
allows
us
to
basically
continue
to
ship
delve
as
a
single
binary
without
any
kind
of
dependencies
on
disk,
or
anything
like
that
in
terms
of
trying
to
find
where
this
ebpf
program
is
or
anything
like
that.
A
A
A
So
I
want
to
get
into
some
kind
of
low-level
implementation
details,
because
that's
what
I
think
is
is
most
exciting
for
these
kinds
of
talks.
So
in
order
to
communicate
back
and
forth
between
the
debugger
and
the
ebpf
program,
we
use
some
pretty
standard.
Eppf
tough
stuff
ring
buffers
and
maps,
and
we
use
them
in
what
I
think
is
some
somewhat
creative
ways.
A
So
one
of
the
things
that
we
communicate
from
user
space
to
the
ebpf
program
is
all
of
the
information
that
it
needs
to
know
to
find.
How
many
arguments
does
this
function
have?
Where
do
they
live?
We're
talking
specifically
in
the
context
of
go,
you
may
or
may
not
be
aware
that
go
has
recently
changed
its
abi
from
a
stack
based
calling
convention
to
a
registered
base
calling
convention
within
delve.
We
have
to
support
both
of
these
versions,
so
we
have
to
know
where
to
find
arguments
on
the
stack.
A
We
also
have
to
know
where
to
find
them
in
in
registers,
and
if
we
have
pointers,
we
need
to
know
how
to
follow
them
and
get
that
data,
and
we
need
to
know
that
it
is
a
pointer
in
the
first
place.
So
we
we
try
to
convey
as
much
of
this
information
as
possible
from
the
debugger
and
we
store
this
information
in
the
map.
That's
keyed
by
the
the
instruction
address,
so
we
can
say
when
the
ebpf
program
is
hit.
A
It
can
look
at
what
the
current
instruction
pointer
value
is
look
up
in
the
map,
the
information
that
it
needs
to
be
able
to
decode
all
of
the
information
and
then
the
the
program
can
kind
of
run
and
do
its
thing.
So,
as
you
can
see
here,
we
kind
of
put
a
lot
of
information
in
like
the
go
id
offset
so
where
the
go
id
struct
is
from
the
offset
of
the
go
id
from
the
the
actual
like
where
the
go
routine
struct
lives.
A
We
described
the
g
adder
offset,
so
the
the
the
offset
of
the
go
routine
from
thread:
local
storage,
it's
a
lot
of
kind
of
low
level
information,
but
it's
this
kind
of
information
that
the
debugger
already
has
so
instead
of
like
re-implementing
a
dwarf
parser
in
ebpf,
or
something
like
that.
We
try
to
provide
as
much
context
as
we
can
from
the
user
space
side
of
things,
and
we
do
that
ahead
of
time
so
that
by
the
time
the
ebpf
program
is
actually
triggered.
A
It
has
all
the
information
that
it
needs
to
be
able
to
just
quickly
as
quickly
as
possible.
Read
all
this
information
and
send
it
back
to
user
space
so
again,
more
information,
so
for
each
function,
parameter
input
or
output.
We
have
a
ton
of
information
about
it.
What
kind
of
variable
is
this?
What's
the
size
of
it?
A
What's
the
offset
from
the
stack
pointer
if
this
is
stack
based
api,
if
it's
in
a
register,
all
this
all
these
kind
of
information
and
then
also
the
information
that
we
want
to
convey
back
to
user
space
like
the
values,
the
actual
raw
bytes
of
of
these
variables,
we
want
to
pass
that
back
to
user
space.
So
we
do
it
through
these
various
kind
of
structs
that
we
pass
around
using
ring
buffers
and
maps.
A
A
So
from
the
delve
side
of
things,
we
attach
u
probes
into
rep
probes.
So
if,
if
you're
not
familiar
with
this
through
ebpf,
you
can
you
can
attach
user
space
probes
and
typically,
you
probe
would
be
at
the
function,
entry
point
and
then
there's
also
uret
probes,
which
can
trigger
whenever
a
function
returns.
A
Now,
specifically
with
you
reprobes
when
you're
talking
about
using
them
with
go
programs,
they
get
a
little
bit
tricky.
So
the
way
that
you
rep
probes
actually
work
is
they
modify
some
information
on
the
stack,
so
they
modify
actually
where
the
function
returns
to
it
actually
ends
up
returning
into
kind
of
like
a
trampoline
that
ends
up
executing
the
ebpf
program.
A
This
has
a
tendency
to
make
go
very,
very
upset
because
go
likes
to
look
in
the
mirror,
a
lot
it
likes
to
inspect
itself.
So
if
you're
not
super
familiar
with
go,
go
has
this
concept
of
go
routines,
which
are
these
very,
very
small
kind
of
units
of
execution,
and
they
start
with
really
small
stacks
and
that
stack
grows
and
gets
copied
over
time.
A
A
real
break
point
like
a
p
tracy
break
point
on
on
the
runtime
function
that
handles
this
copy
stack.
So
we
kind
of
do
this
like
real,
quick
when
go's
about
to
copy
the
the
stack
we
remove
the
urep
probes
and
then
when
it's
done,
we
kind
of
put
everything
back.
So
it's
kind
of
like
a
weird
little
hack,
but
it's
mostly
working
for
us
right
now.
A
A
That's
long
running
starts
communicating
and
and
just
kind
of
getting
this
information
back
from
the
ebpf
program,
and
it
can
parse
it
and
essentially
the
the
nice
thing
about
this
approach
is
it
can
impart
it
can
start
parsing
it
at
its
at
its
leisure.
For
the
most
part,
I
mean
the
hard
work
is
done
like
what
we
really
want
to
do
is
we
want
to.
We
want
to
prevent
the
program
from
stopping
for
too
long
right.
We
want
to
prevent
the
overhead,
so
we
just
need
to
get
the
data
in
the
ebpf
land.
A
A
You
know
not
not
slowly,
but
there's,
there's
less
time
constraints
there,
because
that
doesn't
actually
affect
the
the
program
that's
being
run.
So
this
interface
is
really
nice
and
it's
been
working
out
really
great
for
us
now.
There's
there's
a
lot
of
like
upsides
about
this.
This
approach
and
this
rewrite
and
it's
been
a
lot
of
fun.
It's
been
very
exciting,
but
as
with
everything,
there
are
a
little
bit
of
downsides.
A
I
think
I
on
you
know
I
understand
why
it
needs
it
needs
to
be
that
way.
But
that
is,
in
my
opinion,
just
a
little
bit
of
a
downside.
There's
there's
a
little
bit
more
that
you
have
to
think
about,
and
you
have
to
be
really
creative
with
certain
things.
So,
for
example,
like
lack
of
of
being
able
to
write
loops
in
an
ebpf
program
from
for
this
particular
implementation,
we
need
to
be
able
to
say
well,
I
want
to
parse
three
input
arguments
or
four
or
five.
How
do
you
do
that?
A
If
you
can't
loop
and
kind
of
you
know,
do
this
kind
of
arbitrary
thing?
So
in
our
case,
we
kind
of
took
advantage
of
c
style
switch
statements
and
automatic
follow-through
to
make
like
weird
pseudo,
loops
and
stuff
like
that,
so
there
is
workarounds,
but
it's
just
something
that
I
think
people
should
be
aware
of.
A
So
also,
if
you're
not
familiar
when
you,
when
you
load
an
ebpf
program
into
the
kernel,
it
goes
through
like
a
verifier,
where
it
just
makes
sure
that
this
function
or
that
this
particular
program
is
going
to
execute
the
way
that
it
it
expects.
It's
not
going
to
do
anything
dangerous,
it's
not
looping
or
and
verify
that
it
actually
will
exit
deterministically.
A
I
mentioned
this
a
little
bit,
but
the
small
stack
requirement
can
be
a
little
bit
of
a
hindrance
and
something
to
that
you
might
have
to
work
around
a
lot
again.
No
loops
limited
control
flow.
It
forces
you
to
be
very
creative
and,
as
I
mentioned,
you
rep
probes
do
not
play
with
play
well
with
with
go
programs
by
default.
If
you're
not
really
really
careful
with
how
you
use
your
rep
probes,
you're
guaranteed
pretty
much
guaranteed
to
crash
any
go
program
that
you
use
them
with.
A
A
So
I'll
repeat,
the
question,
as
I
understand
it
is:
are:
are
these
probes
handled
on
a
go
on
a
per
go
routine
basis,
or
are
they
global
so
in
in
the
strictest
sense?
They
are
global,
but
that's
one
of
the
things
that
we
do
from
the
debugger
side
of
things
is
try
to
figure
out
how
to
collate
all
of
this
information
and
present
it
linearly.
So
that's
one
of
the
things.
A
That's
why
it's
important
when,
when
I
showed
some
of
the
code,
examples
where
we're
parsing
the
go
routine
structure
and
we're
getting
the
go
id
so
we're
we're
using
that
information
to
take
that
back
to
user
space
and
kind
of
present,
like
a
cohesive
story
of
when
a
function
is
hit
and
when
it's
returned
and
making
sure
that
that
that
the
input
and
the
return
values
are
all
being
associated
with
the
same
the
context
of
execution.
So
the
same
go
routine.
A
A
A
So
yes,
so
right
now,
you
do
have
to
be
a
privileged
user
to
be
able
to
load
the
ebpf
program
into
the
kernel
or
have
syscap
admin.
I
think,
if
you
have
syscap
admin,
you
can
do
whatever
you
want,
but
that
that's
another
kind
of
workaround
and
but
and
you
right
now,
it's
it's
still
like
this.
This
implementation
is
ongoing
and
something
that
I'm
still
kind
of
currently
working
on
so
right
now
it's
kind
of
you
have
to
build
like
a
like.
A
Great
question
so
just
to
repeat
the
question
since
we're
talking
about
this
in
the
context
of
production,
a
lot
of
people
will
strip
debug
information
out
of
their
production
binaries
to
make
them
even
smaller
when
they,
when
they
run
them
on.
You
know
in
any
kind
of
production
context.
So
yes,
this
that
that
is
a
huge
for
this
aspect,
that
that
would
be
a
huge
hindrance
and
limitation.
A
workaround
for
that
is
delve,
does
allow
you
to
supply
external
debug
information.