►
From YouTube: Building Your Own Kubectl Trace on Steroids Using BPF Libraries in Go, Rust, Python an... Kyle Quest
Description
Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Building Your Own Kubectl Trace on Steroids Using BPF Libraries in Go, Rust, Python and C - Kyle Quest, DockerSlim
You know the BPF fundamentals, you've tried "bpftrace" and you've used "kubectl trace" with your Kubernetes cluster. Now you want to build something more useful that goes beyond making "printf" calls and the basic data processing capabilities in "bpftrace". You are ready to build your tracing engine, but you are stuck. There are a lot of different libraries to choose from and you don't know what to use. This is a journey exploring different BPF libraries in Go, Rust, Python and C and exploring their capabilities.
You'll learn about the library ecosystem, how the libraries are different and which ones can be used to build system tracing applications using BFP tracepoints and other BPF program types useful for tracing and profiling. You'll also learn about the different options available in BFP and why you'd want to choose tracepoints instead of kprobes and vice versa.
We'll explore these languages and libraries:
Go - iovisor/gobpf, cilium/ebpf, dropbox/goebpf, libbpfgo
Python - bcc, pyebpf
Rust - libbpf-rs, redbpf, aya
C - libbpf, bcc
We'll also briefly explore what's available in other languages (lua, node.js and ruby).
The code samples will be available in Github, so you'll be able to experiment with the tracer app code in different languages and there'll be a "kubectl-tracex" that provides an alternative to "kubectl-trace" powered by a custom tracer engine from this talk.
A
This
is
going
to
be
a
little
different,
I'm
not
going
to
be
presenting
a
brand
new
tool
with
uber
capabilities
and
all
of
that
I'll
focus
more
on
building
a
custom
tool
and
today
we'll
be
building
a
kubernetes
tracer,
an
abpf
kubernetes
tracer
that
will
watch
file
access
basic.
But
again
the
focus
is
on
what
it
takes
to
build.
Something.
A
A
Bpf
trace
is
probably
the
number
one
tool
you'll
start
with.
It
provides
a
high
level
interface
to
eppf
capabilities
and
it's
awesome.
It
allows
you
to
hook
into
a
lot
of
different
parts
of
the
kernel
and
the
system
in
general,
but
its
functionality
is
limited.
There's
only
so
many
functions,
they
provide
you
and
if
you
want
something
more,
you
need
to
look
elsewhere.
A
You
have
cube
control
trace
that
pretty
much
provides
an
ability
to
run
bpf
trays
on
kubernetes,
but
actually
one
of
the
newer
capabilities
in
cube
control
trays
is
an
ability
to
run
other
things.
So,
in
a
way,
one
potential
design
option
is
to
take
cube,
control,
trace
and
create
your
own
tracer
and
hook
it
up
together
together,
so
that
that
can
be
a
shortcut
if
you're.
A
A
Then
you
also
need
to
figure
out
the
I
o.
How
are
you
going
to
talk
to
the
kernel
part
of
your
application,
how
you're
going
to
provide
the
config
parameters?
How
you're
going
to
get
the
data
out
and
all
of
that?
So
usually,
when
you
build
an
ebpf
based
application,
you
have
a
kernel
piece
and
a
user
piece
and
then
in
the
kernel
you
have
a
number
of
hooks
and
you
know
for
us:
it's
going
to
be
tracing
hooks,
then
you're
going
to
have
a
map
or
a
set
of
maps.
A
Where
that
you
would
use
for
input
and
output,
mostly
configuration
and
basic
stats
on
the
way
out
and
for
any
kind
of
dynamic
data.
You
probably
want
to
use
perv
buffers
or
ring
buffers
and
ring
buffers
are
kind
of
perf
buffers
plus
plus
much
nicer
easier
to
use
single
queue
and
all
of
that,
and
then
you
might
also,
if
you're,
using
newer
versions.
You
might
also
use
global
variables
for
basic
config
parameters.
A
A
There
are
quite
a
few
program
types,
a
lot
of
different
places
where
you
can
hook
in
and
most
of
the
program
types
they
fall
into
these.
You
know
high
level
categories
and
the
most
the
biggest
categories
are
tracing
and
networking,
and
today
we'll
focus
on
tracing,
but
there's
a
lot.
You
can
do
with
networking
and
lots
of
great
talks
about
that.
A
So
with
tracing
you,
you
have
a
choice
to
make.
There
are
different
program
types
to
choose
from
and
they
allow
you
to
do
static,
tracing
or
dynamic
tracing
so
with
static
tracing
the
nice
thing
about
it.
You
use
stable
hook
points
in
the
system
that
will
not
change
from
one
version
of
the
system
to
another.
So
that's
great
with
dynamic
tracing.
A
You
have
more
flexibility,
but
it's
more
brittle
in
the
dynamic
tracing
bucket.
You
have
your
k,
probes
and
k
red
probes
or
u
probes
and
your
red
probes,
and
there
you
can
hook
any
function.
You
want
the
problem
there
is
that,
if
all
of
a
sudden
you
know
the
the
function
goes
away,
you're
out
of
luck,
but
with
the
static
trace
points,
there's
a
well-defined
interface,
that's
stable!
So
if
you
want
a
tracing,
app,
that's
stable
and
it
will
run
from
one
version
to
another.
A
You
definitely
want
to
pick
trace
points
and
do
static
tracing
and
that's
gonna
simplify
a
lot
in
in
your
application
and
then,
with
you
know,
with
the
trace
points,
there
are
a
few
different
options.
For
example
in
the
kernel
you
have
trace,
trace
points
and
row
trace
points
and,
depending
on
you
know
how
advanced
your
application
is.
You
might
want
to
pick
raw
trace
points.
It's
pretty
much
trace
points,
but
without.
A
A
With
cnc,
plus
plus
the
most
known
library
is
pcc,
so
if
you're
not
brand
new
to
ebpf,
you
know
that
it's,
it's
probably
the
the
biggest
library
out
there,
the
biggest
ecosystem,
with
lots
of
examples,
it's
most
known
for
its
python
wrapper.
So
a
lot
of
talks
this
year
previous
years,
any
talk
about
ebp.
Almost
any
talk
about
ebpf,
most
likely
included
python
snippets,
the
main
gotcha
with
bcc
and
the
wrappers
around
it
is
that
you
need
to
compile
the
bpf
program
on
the
target
machine
and
grant
talked
about
that.
A
A
Ebpf
pub
is,
is
a
new
library.
It
was
created
for
voice
query,
it's
a
simple
wrapper
for
libby
pf
designed
for
tracing,
haven't
used
it
much,
but
it
looks
interesting.
The
code
is
pretty
clean
lots
of
go
libraries.
The
first
one
in
the
original
library
is
go
bpf
from
io
visor,
it's
it's
a
wrapper
for
bcc
and
you
get
the
the
same
benefits
most
benefits
of
bcc
and
the
you
know
the
downsides
from
it.
A
A
I
don't
know
it's
it's
confusing!
There's
enough
confusing
stuff
there,
but
ebpf
any
more
confusing
stuff
is
going
to
make
things
more
complicated,
but
one
of
the
nice
features
there
is
the
bpf2go
module.
So
it
allows
you
to
embed
the
ebpf
program,
compiled
ebpf
program
code
into
your
go
code
and
it
provides
a
nice
wrapper
to
load
it
and
to
work
with
that
and
they're.
Also
working
on
the
compile
once
run
everywhere
support
they.
They
support
a
lot
of
it
already
and
I've
seen
samples
but
they're.
A
Almost
there
there's
a
library
from
dropbox,
it's
pretty
nice
much
nicer
than
the
serium
library
in
terms
of
its
structure
and
how
easy
it
is
to
understand.
Unfortunately,
it
doesn't
support
enough
program
types.
So
if
you
want
to
build
a
tracing
application
using
trace
points,
you're
out
of
luck,
there's
a
library
from
aqua,
libby,
pfgo
and
tracy
is
using
it,
and
this
is
actually
one
of
the
libraries
we'll
be
using
today.
A
A
A
A
A
A
A
So
it
means
that
you
don't
need
to
write
any
c
code
and,
if
you're
a
rust
person,
it's
it's
super
cool.
It's
still
a
little
early,
the
tooling
ecosystem
looks
promising
yeah
a
little
tough.
I
have
a
few
samples
to
show
there.
There
are
a
few
other
libraries
but
they're,
mostly
bcc
rappers.
The
lua
rapper
is
the
official
wrapper.
A
We
can
take
a
look
at
the
kubernetes
side
of
things,
and
the
first
thing
you
probably
want
to
figure
out
is
the
cubecontrol
plug-in
so
for
the
kubernetes
side
of
things,
you'll
need
to
have
two
parts,
the
the
plug-in.
Technically,
you
can
work
around
that
with
and
just
use,
cubecontrol
and
and
manually
set
up
resources,
but
it's
nice,
that's
what
cubecontrol
trace
does
and
then
you
also
need
to
figure
out
the
the
the
part
that
you
run
in
the
cluster
so
with
the
but
the
cubecontrol
plugins.
A
A
And
you
can
use
libraries
if
you
want
to
there's
a
there's,
a
nice
helper
library
for
go
cli
runtime,
that's
what
cubecontrol
uses
so
you
know
internally.
I
think
it
was
extracted
from
it.
I
didn't
find
it
super
useful,
because
I
I
don't
use
cobra
for
the
cli
options
and
a
lot
of
the
you
know
helpers
are
around
that,
but
the
client
configuration
helpers,
they're,
actually
nice
and
then,
with
the
tube
control
plug-in.
A
You
also
need
to
figure
out
how
you'll
interact
with
the
what
kubernetes,
what
client
library
you
you'll
want
to
use
and
technically
the
the
language
you
use
for
the
cubecontrol
plugin
and
the
language
you
use
for
the
tracing
engine.
They
don't
need
to
be
the
same.
So
they
can.
You
know
the
the
cube,
control
plugin
could
be
go,
the
the
engine
could
be
c,
but
there
are
a
few
options:
there's
obviously
the
official
client
go
library
and
there's
a
an
official
python
library.
It
has
a
lot
of
examples.
A
Pretty
nice
there
is
an
an
unofficial,
rust
library,
cube
rs.
Also,
nice
probably
has
the
most
examples.
I've
seen
today,
I'm
not
going
to
show
examples
of
in
each
language
possible
language,
but
it's
it's
nice
to
know
that
there
are
options.
There
are
a
number
of
options
for
other
languages,
but
yeah
I
feel
free
to
explore.
I
I'd
stick
with
those
three.
A
A
Control
plugin
will
will
schedule
a
job
on
demand
and
it
will
figure
out
where
to
run
it
and
and
then
the
the
bpf
trace
or
the
other
tracer
will
run
on
the
selected
nodes
where
the
job
is
running
so
when,
but
when
you're
using
jobs,
you
need
to
figure
out
how
to
where
to
run
the
jobs,
because
if
you
need
to
create
an
application
in
a
certain
pod,
you
need
to
make
sure
that
the
job
runs
there.
There
are
a
few
options
available
there.
A
The
most
basic
one
is
to
set
the
node
name
in
the
pod
spec,
but
it's
brittle
and
it's
not
recommended
the
next
best
option
is
to
use
node
selectors
and
that's
pretty
much
the
most
basic
option.
That's
that
you
can
use
if
you
don't
need
any
kind
of
fancy
way
to
select
the
nodes
where
you
want
to
run
the
job.
That's
that
would
be
the
way
to
do
it
and
the
last
one.
The
last
option
is
to
use
node
affinity,
but
node
affinity
provides
more
advanced,
node
selection
capabilities.
A
And
this
is,
I
think,
that
that's
what
cube
control
trace
uses
internally,
node
affinity,
the
other
option
to
use
is
demon
sets,
and
it's
actually
the
option
that
a
number
of
other
tools
use
the
the
trade-off
there
is
that,
with
with
jobs
you
get
tracing
on
demand.
So
if
something
happens,
and
then
you
decide
to
run
cube
control
trace
and
it's
nice
on
one
hand.
On
the
other
hand,
it's
you
know
it.
It
might
have
gaps
because
by
the
time
something
happens
and
you
run
cube
control
trace
with
a
job.
A
What
you
need
is
already
gone,
for
example,
if
you
need
to
trace
how
an
application
binds
to
a
port
using
a
job
isn't
really
a
good
design
there,
because
by
the
time
you
decide
that
hey,
I
want
to
run
cubecontrol
trace.
The
application
already
called
the
buying
call,
and
all
of
that,
so
that's
when
demon
sets
are
nice.
Kubernetes
takes
care
of
running
your
pod
on
every
machine.
A
This
is
the
cubecontrol
plugin
for
the
sample
application
and
it's
using
go
and
like
I
said
it
doesn't
need
to
be
go.
It
doesn't
even
need
to
be
a
in
a
real
programming
language,
but
here
it's
it
has
this
install
command
and
the
install
command
connects
to
kubernetes,
and
then
it
launches
a
daemon
set.
A
A
A
A
A
A
A
Then
it
loads
the
bpf
module
and
then
attaches
the
trace
points.
So
this
is
what
I
was
referring
to
you
have
to
for
each
trace
point
you
have,
you
need
to
call
attach
train
support,
a
trace
point
explicitly
with
the
lib
bpf
library.
There
is
a
single
call
that
you
can
make
to
attach
everything,
so
it
would
be
nice
to
do
that.
So,
once
you
attach
everything.
A
It
consumes
events
from
the
roy
van
channel,
and
this
is
the
channel
that
was
passed
to
the
perv
buffer
setup
call
and
that's
one
of
the
nice
things
in
in
the
lib
bpf
go
library.
It
provides
a
go
abstraction
to
interact
with,
but
the
low
level
primitives
and
the
data
that
you
get.
You
need
to
decode
it
so
the
program
itself,
so
the
other
side.
A
A
Open
trace,
point
and
says,
enter
openeth
tracepoint
and
says
exit
open
at
tracepoint
pretty
much.
We
hook
the
the
entry
and
the
exit
to
the
file
operations,
so
we
can
get
the
status
codes
on
the
way
out
and
get
more
information,
and
this
is
what
the
map
is
used
for.
It's
used
for
temporary
storage.
So
when
we
see
the
entry
trace
point,
we
save
the
data
into
a
structure.
A
A
The
c
version
is
pretty
similar.
So
there's
also
there's
also
a
temporary
map.
There
is
a
ring
buffer
and
that
that's
the
nicer
version
of
the
perf
buffer
and
the
the
coded
side
looks
almost
identical.
The
user
side
of
it
looks
different
and
here's
here's
the
call
that
I
like
the
most.
So
you
open
the
bpf
program,
the
ppf
module.