►
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon North America 2021 in Los Angeles, CA from October 12-15. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
eBPF on the Rise - Getting Started - Quentin Monnet, Isovalent
eBPF is rising as an essential component for observability and networking programmability on Linux and in containers. Due to its performance, flexibility, and scalability, Liz Rice listed it as one of the five CNCF technologies to watch for 2021! In an opportunity to fully comprehend its potential, this introduction will help you get started with eBPF. First, the core architecture. What is an eBPF program? How to attach and run it safely in the kernel, and most importantly, what for? And what are the other components of the eBPF subsystem? Come and learn the basics. Then, augment your toolbox with utilities to manipulate eBPF objects. Inspect, debug or trace with bpftool, bcc tools, bpftrace, or libbpf and its Golang equivalent. Equipped with knowledge and tools, here you come, now prepared to instrument clusters with eBPF! Before you go load-balancing traffic and chasing metrics, the last part focuses on eBPF's benefits for cloud-native environments. Ready to surf the eBPF wave?
A
A
My
objective
today
is
to
help
you
getting
started
with
ebpf
and
we'll
do
that
in
three
parts.
The
first
section
will
be
about
understanding
how
ebpf
works
and
then,
in
a
second
time,
we'll
see
what
tools
are
available
for
working
with
ebpf
and
the
last
section
will
be
about
what
benefits
we
can
have
for
cloud
native
environments
from
ebpf.
A
A
To
really
understand
that?
Let's
observe
that
the
linux
system
is
being
used
at
the
basis
for
everything
in
cloud
environments
nowadays,
and
we
have
some
kind
of
paradox
with
linux,
because
everything
that
allows
you
to
get
some
observability
to
understand
what
resources
are
used
by
a
given
process
of
pod
is
happening
on
the
kernel
side,
but
you
have
very
little
flexibility
at
the
center
in
the
kernel.
You
are
free
to
program.
Whatever
you
want
in
user
space,
but
in
user
space
you
won't
get
a
direct
access
to
those
kernel,
data
structures.
A
So
do
we
have
a
way
to
introduce
more
programmability
in
the
kernel
we
have
kindle
modules,
but
kernel
modules
can
be
tricky
to
implement
or
they
can
have
issues
in
terms
of
safety.
You
are
likely
to
crash
your
kernel
if
you
made
some
insects,
some
mistakes
in
your
modules,
you
do
not
have
any
guarantees
in
terms
of
api
stability
from
the
canal
from
one
version
to
the
other
one.
So
that
can
break
your
modules
too.
A
A
And
if
so,
can
you
leverage
that
to
get
some
benefits
in
your
cloud
native
environments?
The
answer,
of
course,
is
ebpf.
So
ebpf
is
some
generic
purpose
execution
engine
to
already
implement
your
programs
defined
in
user
space
inside
of
the
kernel.
So
historically,
it
was
built
on
top
of
what
is
now
called
the
classic
bpf,
which
was
used
with
this
pinum
or
second
to
filter,
packets
to
computer
user
space
or
cisco
arguments
respectively,
and
nowadays,
ebpf
is
using.
A
You
also
have
a
benefits
in
terms
of
safety,
because
your
ebpf
program
will
be
checked
and
verified
when
you
load
it
into
the
kernel
to
make
sure
that
it
terminates
and
won't
be
hanging
your
kernel
and
that
it
is
safe.
So
that
means
you
won't
be
able
to
have
infinite
loops
in
your
programs,
but
that
makes
it
short
to
terminate
and
you
won't
be
able
to
leak
some
sensitive
memory
to
the
user
space
or
to
perform
out-of-bound
accesses
in
your
ebps
program
and
to
risk
crashing
your
camera.
A
A
A
You
have
a
number
of
maps
too
ebpf
maps
are
some
kernel
memory
areas
available
to
programs,
usually
key
value,
storage
areas
like
arrays
or
hash
maps.
A
few
other
ones
for
dedicated
use
cases
too,
and
those
maps
can
be
shared
between
different
instances
of
a
program
to
store
some
state,
for
example,
some
counter
some
metrics
whatever
you
can
also
share
them
between
different
programs
to
correlate
some
data.
A
You
can
also
share
them
with
user
space,
to
share
information
with
user
space,
to
collect
metrics
or
to
pass
down
some
configuration
options.
In
addition
to
those
features,
we
get
a
number
of
things
that
the
execution
engine
itself
is
supporting.
As
the
bpf
system
subsystem
is
getting
improved,
so
we
have
now
up
to
one
million
instructions
in
the
program
which
makes
it
flexible
for
a
number
for
a
wide
number
of
use
case.
Even
advanced.
We
have
tech
codes,
we
have
ebpf
to
bpa
function,
calls
we
have
banded
loops
support.
A
Now
we
have
a
number
of
things
like
this.
That
makes
it
really
closer
and
closer
to
just
a
regular
program
that
you
would
compile
from
c
or
from
any
other
language.
Already
it's
getting
close
to
something
very
generic
and
that
you
can
use
for
all
type
of
usage
so
about
the
use
cases.
They
mostly
fall
down
into
two
big
categories,
which
are
network
packet,
processing
on
one
side
and
tracing
and
monitoring.
On
the
other
side.
A
As
for
networking,
we
have
a
number
of
hooks
in
the
kernel,
so
we
have,
for
example,
a
hook
on
the
tc
traffic
control,
both
ingress
and
egress
path.
We
have
the
xdp
hook,
which
is
about
going
very
low
in
front
of
the
kennel
stack
for
processing
packets,
just
at
the
exit
of
the
driver,
and
we
can
retrieve
our
bike
packets
from
there
and
process
them.
Even
before
the
circuit
buffer
is
allocated
so
even
before
we
spend
time
and
resources
processing
the
packets
in
the
linux
stack.
A
Those
hooks
make
ebpf
very
suited
very
well
suited
for
applications
like
protection
against
denial
of
service
attacks
or
for
load
balancing
to,
because
it's
located
just
in
front
of
the
stack
you
have
some
other
applications
to
like
routing
overlay,
nat
many
others.
You
can
parent
some
options
for
tcp
sessions
and
even
re-implement
the
congestion
algorithms
used
by
tcp.
A
A
You
have
also
static
probes
with
trace
points
or
the
user
space
equivalent
and
a
few
other
probes
too,
and
you
can
use
them
to
inspect
and
trace
and
profile
your
kernel
or
your
user
space
applications,
making
it
very
suitable
for
understanding.
What's
going
on
in
terms
of
resources,
usage
and
optimizing,
your
programs
too
one
big
advantage
of
ebpf.
Is
that
because
it's
a
program
running
in
your
canon,
you
can
aggregate
and
correlate
matrix
and
all
the
data
inside
of
your
program
and
just
send
what
you
need
the
meaningful
information
to
your
user
application.
A
You
don't
need
to
sample
out
everything
to
use
a
space,
and
that
means
a
lot
of
gains
in
terms
of
overhead.
I
mean
you're
saving
a
lot
of
overhead.
You
have
a
few
other
use
cases
too.
So,
for
example,
you
have
a
linux
security
module
that
is
built
on
top
of
ebpf.
Now
in
the
kernel,
you
have
proposals
that
were
made
about
using
ebpa
for
file
systems
or
storage.
A
So
the
list
is
growing
that
more
and
more
people
getting
interested
in
bpf
and
proposing
use
cases,
and
that's
something
that
is
very
interesting
to
see.
I'm
looking
forward
to
seeing
new
applications
using
ebpf,
but
we
have
all
those
use
cases.
How
can
we
use
ebpf
concretely
to
to
start
tracing
systems,
for
example?
Nowadays,
the
first
tool
I
would
like
to
present
is
first
the
llvm
backend
to
generate
ebpf
bytecode.
A
This
bytecode
is
very
close
to
assembly,
but
nobody
really
likes
spending
the
time
writing
assembly.
I
think
so
we
have
the
the
clang
and
llc
tools
that
have
been
adapted
to
generate
ebpa
bytecode
from
c
in
particular,
and
so
you
can
just
write
c
programs
and
turn
them
into
ebpf
bytecode
that
will
be
stored
in
l5,
object
files
and
load
it
from
there.
So
here
is
a
very
simple
ebpf
program
for
networking.
A
It
would
be
attached
to
an
interface
and
drop
everything
which
is
not
ipv4
packets.
So
this
is
a
standalone
program.
I
can
just
compile
this
file
with
clang
and
attach
it
with
the
command
ip
link
set
xdp
to
drop
everything
which
is
not
ipv4
and
the
way
it
works
is
we
have
two
checks
in
this
program.
A
A
A
A
We
just
need
two
commands
to
generate
this
graph,
the
first
one
to
run
the
bcc
tool
to
extract
the
data
from
the
cpu
to
to
sample
the
cpu
and
extract
the
data,
and
the
second
command
is
just
running
a
script,
to
process
its
data
and
generate
the
graph
itself.
So
you
can
use
that
to
profile
cpu
for
your
kernel,
but
also
for
applications
written
in
a
variety
of
languages.
So
there
is
python,
ruby,
php,
see
like
languages
and
so
on
and
so
forth.
A
A
So
the
first
example
I
have
here
is
about
reproducing
the
tracing
of
the
open
system
call,
but
it's
just
on
a
one
line
comment.
So
that's
very
short:
you
have
two
more
examples
below
about
printing
the
size
distribution
by
process
of
the
length
written
read
by
the
read
system
code.
So
you
you
have
programs
using
read
and
read
returns
length
of
the
memory
chunk,
that's
being
read
and
you
can
just
print
a
histogram
in
the
console
with
that.
A
The
first
one
is
bad
counting.
The
llc
cache
misses
for
your
processes,
so
really
really
short
comments,
but
potentially
really
helpful
information
to
help.
You
troubleshoot
your
programs.
If
you
want
to
go
beyond
bcc
and
bpf
trace,
you
can
build
your
own
applications.
You
have
libraries
to
help
you
with
that.
You
are
not
starting
from
scratch.
You
have,
for
example,
libpf,
which
is
the
reference
library
for
everything
related
to
the
through
bpf.
A
You
have
a
number
of
libraries
available
in
go
also,
I'm
probably
a
little
bit
biased
here,
but
I
would
definitely
recommend
going
with
the
ebpf
library
maintained
by
cloudflare
and
cdm,
which
is
a
pure
go
library
very
useful
to
to
manage
ebpf
objects.
There
are
some
options
in
rust,
too,
and
possibly
in
other
languages.
A
A
I
want
to
be
able
to
understand,
what's
going
on
in
my
system,
like
what
programs
are
loaded
and
to
inspect
them,
so
there
is
bpf
tool
that
you
can
use
to
load
programs
with
ppf
prog
load
from
the
command
line
or
epf
to
proc
show
to
just
list
the
programs
that
are
loaded
on
your
system.
So
here
I
have
two
programs
attached
to
circuits
and
c
groups
and
one
xdp
program.
A
I
can
just
dump
the
bytecode
of
those
programs
to
to
see.
What's
in
the
program
that
that's
loading
on
the
kernel,
I
can
download
the
cheat
compile
instructions
in
case
it
helps.
I
can
manage
maps
too.
I
can
list
the
maps
I
can
look
up
for
given
entry.
I
can
update
the
map
and
I
have
a
few
more
options
available
as
well.
I
can
test
run
programs.
A
I
can
list
the
ebpf
related
features
supported
by
my
kernel,
if
you're
interested
in
all
those,
you
should
probably
have
a
look
at
the
manual
pages
for
bpf2
and
that's
it
for
the
tools
now.
Can
we
use
some
of
those
tools
and
some
of
the
advantages
we've
seen
about
the
bpf
into
cloud
native
environments?
So,
as
a
reminder,
we
have
a
number
of
advantages
brought
forward
by
ebpa.
If
we
have
safety,
we
have
performance.
A
We
have
a
big
deal
in
terms
of
observability,
it's
something
very
versatile,
it's
in
the
kernel,
but
it
remains
flexible
and
having
it
in
the
corner.
It's
also
a
huge
advantage
for
the
reasons
in
particular,
because
it's
available
by
default,
you
don't
need
to
install
everything
to
anything
to
use
ebpf,
it's
already
here
on
your
system.
It
has
a
stable
user
api.
So
it's
not
subject
to
breakages
from
one
conservation
to
the
next
one.
It's
also
really
easy
to
update
an
ebpf
program.
A
You
don't
have
to
hack
into
your
kernel
and
then
send
the
contributions
upstream
and
wait
for
them
to
be
merged
and
then
to
be
available
in
your
version
that
you're
using
you
just
change
your
program
and
recompile
it
and
reload
it.
You
no
need
to
reboot
your
system
to
load
it
and
if
you're
processing,
packets,
for
example,
you
don't
even
have
any
loss
of
packets
between
an
update
from
the
previous
to
the
next
program.
A
A
If
I
don't
need
ipv4
just
ipv6,
I
just
don't
compile
any
ipv4
related
features,
and
that
gives
me
something
really
clean
and
fast
and
scalable
too,
that
I
can
use
to
really
implement
my
solution
to
solve
my
real
world
production
use
cases,
and
that's
something
really
important
to
have
this
kind
of
flexibility,
especially
if
we
consider
that
the
linux
systems
are
really
everywhere
in
the
cloud
and
used
for
for
building
everything
in
data
centers.
Having
ebpf
with
all
this
flexibility
and
all
those
advantages
brings
us
huge
benefits.
A
A
So
what
it
does
is
basically
launching
a
worker
pod
to
just
run
the
command
on
the
node
that
you
want
to
trace
or
profile,
and
you
can
have
epf
trace,
one-liners
and
scripts.
Sending
you
information
about
your
your
system
on
the
same
model.
We
have
inspector
gadget,
which
is
doing
like
the
same
thing
really,
but
for
bcc
tools
and
that's
already
available
upon
source.
So
you
can
use
it
already.
A
If
you
were
to
focus
more
on
networking,
cdm
is
probably
the
reference
here,
so
we
do
networking
and
observability
and
security
and
all
of
that
with
ebpf.
So,
for
example,
we
have
a
cube
proxy
replacement.
We
can
get
rid
of
cube
proxy,
which
is
a
huge
advantage,
because
cube
proxy
heavily
relies
on
ip
tables
rules
and
those
rules
may
come
by
the
servants.
A
So
that
leads
to
huge
gains
in
terms
of
performance
and
makes
things
a
lot
cleaner
too.
Another
example
of
senior
optimized
data
path
is
when
we
use
an
android
proxy
to
implement
a
later
seven
policies.
So
if
I
want
to
tell
this
pod
the
system
that
this
spot
can
use,
this
http
rest
api
command
in
particular.
But
not
this
part.
This
one
doesn't
have
the
permission.
A
We
also
have
a
number
of
other
use
cases
for
ebpf
for
networking
load,
balancing
network
security,
observability
and
service
mesh,
but
I
will
have
time
to
dive
into
all
of
them.
So
just
have
a
look
at
synonyms
documentation
or
join
our
community
slack
if
you're
interested
to
discussing
them
further.
A
Let's
move
on
with
the
big
ebpf
players
that
are
actually
contributing
to
or
using
bpf
there
is
facebook
using
it
a
lot
for
tracing
and
monitoring
and
for
network
processing.
They
have
an
open
source
load
balancer
in
particular,
called
catran
that
you
can
find
on
github
netflix
is
using
ebpf2
for
tracing
monitoring,
mostly
google
for
number
of
use.
Cases
too
cloudflare,
mostly
for
protection
against
the
idea
of
service
attacks
and,
of
course,
everywhere.
A
Cdm
is
deployed.
Ebpf
is
used
a
lot
to
implement
the
data
path
and
the
network
policies.
There
are
also
a
number
of
other
projects
using
ebpf,
so,
for
example,
we
have
farco
or
tracy
using
ebpf
for
security
purposes.
In
the
cloud
we
have
hubble,
which
already
implements
unprecedented
visibility
for
network
flows
on
your
clusters,
we
have
weave
works.
Shuriketa
2
has
some
xdp
mode
for
capturing
packets
for
security
purposes,
so
ebpf
is
a
thriving
ecosystem.
Really
there
are
an
increasing
number
of
projects
in
addition
to
those
represented
using
the
technology.
A
This
leads
also
to
some
new
startups.
Productizing
ebpa
for
continuous
profiling
network
analytics
for
security
in
the
cloud
too,
and
some
of
those
startups
have
been
acquired
already.
So
I
think
that
was
late.
Late
last
year,
we've
had
a
pixie
acquired
by
new
relic,
also
flo
milo
acquired
by
splunk,
and
that
shows
that
there
is
some
interest
into
the
products
into
the
technology
itself
and
that
there
is
a
lot
of
momentum
here.
On
the
kennel
side.
A
On
the
community
side,
we've
had
a
dedicated
mailing
list
for
ebpf
contributions
which
received
about
which
has
been
receiving
about
50
emails
per
day
on
average,
and
we
also
have
free
maintenance,
five
senior
code
reviewers
to
keep
up
with
the
load
and
they
come
from
facebook,
isoven
and
google,
and
all
that
makes
ebpf
one
of
the
fastest
growing
subsystem
in
linux.
At
the
moment,
there's
a
lot
going
on.
We've
had
our
first
ebpf
summit
last
year,
organized
by
the
cdm
community,
and
it
was
a
huge
success.
A
If
you
want
to
see
the
videos
have
a
look
on
youtube,
they're
all
available
two
tweets
that
I
would
like
to
present
to
show
so
the
momentum
that
is
going
on
with
bpf
the
first
one
from
mark
rusinovic
is
about
microsoft,
looking
at
implementing
some
system-like
utility
for
tracing
on
linux,
and
that
sounds
very
interesting
to
you
to
see
microsoft,
focusing
on
ebpf
to
implement
things.
The
second
one
is
from
steven
rusted
about
bpf
that
may
replace
linux
in
the
future.
A
So
to
wrap
up
this
presentation.
Ebpf
brings
a
lot
of
programmability
of
programmability
to
the
kernel.
It's
safe,
efficient,
it's
versatile!
It's
scalable!
It's
ideally
located
to
gather
data
about.
What's
going
on
on
resources,
usage
on
the
different
codes
that
the
pods
are
executing
on
the
system
and
also
for
processing
packets.
All
of
these
for
individual
systems
or
in
cloud
native
environments-
and
we
have
a
number
of
tools
already
that
are
being
improved
by
the
day
to
work
with
ebpf,
so
we
have
bcc
bpf
trace.
A
A
All
of
these
open
source
and
already
available
ebpf
is
on
the
rise,
so
it
solves
real-world
production
problems
and
that's
something
really
important.
A
lot
of
big
companies
are
using
it
for
that,
because
it
brings
them
some
flexibility.
They
need
to
to
change
some
behavior
in
the
kernel
right
now,
without
having
to
wait
for
upstream
changes.
A
So
selium's,
optimized
data
path
and
network
policies
is
a
very
good
example
on
how
you
can
leverage
ebpf
to
to
implement
advanced
advanced
features
in
cloud
native
environments,
and
there
is
a
buzzing
community
behind
bpf
to
to
a
new
tools.
New
features
all
the
time,
so
I
hope
you'll
be
able
to
to
join
the
community
and
to
ride
the
bpf
wave.
A
So
that's
it
for
this
presentation.
Thank
you
for
attending.
If
you
need
more
information,
there's
one
link
in
particular
that
you
should
check
which
is
ebpf.io
and
which
contains,
in
particular,
all
the
pointers
that
you
would
need
if
you
wanted
to
get
more
information
and
documentation
about
the
bpf
use
cases
and
internal
sentences.