►
Description
Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
A Beginner's Guide to eBPF Programming for Networking - Liz Rice, Isovalent
eBPF has been described as “Superpowers for Linux,” and recently we’ve seen an explosion of tools that use it to power networking, observability and security in the Cloud Native world. It's an exciting technology that enables running bespoke programs directly in the kernel. In this talk Liz uses live-coding examples to explore how eBPF programs are loaded and run in the kernel, and attached to a variety of networking-related events. You might have seen Liz give a similar talk before, with examples hooking into system calls. This updated version focuses networking examples, giving insight into how eBPF programs can inspect and manipulate packets to form the basis of sophisticated and high-performance networking tools.
A
A
A
You'll
probably
have
seen
examples
involving
system
calls,
for
example,
the
k
probe,
which
is
the
function
entry
point
to
well
the
entry
point
to
a
function
that
runs
when
a
particular
system
call
is
called
by
a
user
space
application,
but
system
calls
are
really
just
the
beginning.
There
are
many
many
other
places
in
a
linux
machine
where
we
can
attach
evpf
programs.
A
So
the
typical
hello
world
program
that
I
would
normally
write
for
an
illustration
would
attach
to
a
system
call
like
exec
ve
and
whenever
a
user
space
application
calls
that
system
call
exec
ve
is
used
to
run
a
new
executable,
so
every
time
user
space,
for
example,
you
type
a
new
program
name
into
your
shell.
You
hit
return
that
causes
xactv
to
run
that
program
and
we
might
write
a
hello
world
example
that
attaches
to
the
entry
point.
The
k
probe
of
that
exec
ve
function
and
that
would
generate
some
tracing
every
time.
A
Someone
ran
a
new
program
and
that
tracing
would
include
some
contextual
information
about
the
process
that
was
running
that
triggered
that
call
that
particular
function
call
that
system
call
in
this
case,
but
k
probes
are
really
just
one
of
many
different
types
of
evpf
program.
There
are
a
long
list
of
different
ebpf
program
types
that
we
can
attach
to
different
types
of
events,
so
ebpf
is
not
just
about
tracing
system
calls
system
calls
are
the
interface
between
user
space
and
the
kernel
there's
a
huge
wealth
of
other
places
where
we
can
attach
ebpf
programs.
A
A
A
A
So
the
first
place
to
start
would
be
with
k
probes.
There
are
lots
and
lots
of
functions
that
relate
to
networking
every
time
we
call
into
the
entry
or
exit
from
any
function
in
the
kernel
we
can
trigger
an
ebpf
program.
We
can
attach
it
using
a
k
probe
or
a
k,
rep
probe
for
the
exit
and
as
an
example,
we
can
use
the
kernel
function
that
gets
called
when
a
tcp
connection
is
set.
A
A
So
my
user
space
code
starts
by
reading
and
compiling
the
evpf
functions
that
I've
written
into
this
c
file.
Network.C.
We'll
look
at
that
in
a
moment,
and
it's
going
to
attach
a
function.
That's
called
tcp
connect
to
the
event,
that's
associated
with
the
entry
to
tcp
v4
connect
and
I've
got
a
little
bit
of
debugging.
That's
going
to
tell
us
when
my
ebpf
programs
are
loaded
and
we're
ready
to
go,
and
then
my
user
space
code
is
going
to
print
out,
whatever
tracing
has
been
generated
by
my
ebpf
code
and
let's
take
a
look.
A
A
Okay,
so
that's
k
probes.
Now,
let's
move
on
to
some
more
networking
specific
program
types
and
the
first
one
we'll
look
at
is
socket
filter,
so
I'm
gonna
attach
to
a
raw
socket
using
this
socket
filter
program
type
and
of
raw
socket
sees
traffic
coming
in
and
out
very
close
to
the
network
connection.
So
it's
kind
of
closer
to
the
network
interface
than
the
rest
of
the
networking
stack,
socket
filtering
was
designed
for
observability
and
it's
worth
noticing
this.
A
We
can
look
at
packets
and
we
can
make
decisions
about
whether
we
want
to
pass
a
copy
up
to
user
space
for
observability
purpose.
So
the
idea
is
that
if
you
were
running
some
kind
of
tracing
utility
in
user
space
you'd
probably
want
to
trace
a
subset
of
packets.
Maybe
you
would
want
to
trace
tcp
packets,
but
not
other
sorts
of
networking
activity.
A
A
Then
it
looks
at
the
ip
header
to
see
what
type
of
request
or
what
type
of
packet
this
is,
and
here
I'm
going
to
trace
out
whether
it's
icmp
icmp
commonly
known
as
ping
or
tcp
traffic,
and
if
it's
tcp
traffic,
I'm
going
to
return
-1
and
that
tells
the
kernel.
I
want
to
send
a
copy
of
this
packet
to
user
space.
A
I'm
going
to
need
to
write
some
code
in
my
user
space
side
to
listen
for
this
traffic.
So,
first
of
all,
let's
stop
listening
to
tcp
connect
events
and
instead
we're
going
to
load
a
function
load
that
socket
filter,
ebpf
program,
attach
it
to
a
raw
socket
and
then
we're
also
going
to
create
a
socket
that
can
where
we
can
receive
these
copies
of
the
network
packets
and
then,
instead
of
printing
out
the
tracing.
A
A
A
A
Now,
let's
look
at
xdp,
which
stands
for
express
data
path.
Now
daniel
balkman,
who
is
a
kernel
maintainer
who
works
at?
I
surveillance
told
me
a
great
story
about
how
the
idea
for
xdp
came
about,
and
it
was
essentially
saying:
wouldn't
it
be
great
if
we
could
run
ebpf
programs
rather
than
running
them
on
the
cpu,
what
if
we
could
run
them
on
the
hardware
itself
for
networking?
A
So
if
you
have
a
network
interface
card
or
a
network
driver
that
supports
xdp,
you
can
load
vpf
programs
into
that
hardware
itself.
Hardware
offload
and
when
a
packet
arrives
on
that
network
interface,
it
can
trigger
the
ebpf
program
that
can
make
decisions
about
what
to
do
with
that
packet,
whether
or
not
to
pass
it
up
to
the
networking
stack
or
not.
A
A
There
is
some
performance
gain
because
it
hasn't
had
to
traverse
the
networking
stack,
but
it's
it's
not
quite
what
was
intended
by
doing
that
hardware.
Offload
importantly,
if
we're
using
containers,
we
can
still
run
xdp
because,
with
the
kernel
implementation
of
xdp,
it
can
be
attached
to
virtual
network
connections
as
well
as
physical
network
connections,
but
this
is
only
about
inbound
packets.
If
we
think
about
it
in
the
context
of
a
network
card,
it
really
makes
sense
to
process
inbound
packets
as
quickly
as
possible
before
we
pass
them
to
the
cpu.
A
But
if
the
cpu
has
already
handled
a
packet
and
it's
on
its
way
out,
then
the
cpu
has
already
done.
Whatever's
necessary,
so
let's
take
a
look
at
an
example
of
an
xdp
program,
so
in
our
user
space
code
this
is
pretty
straightforward.
We
can
load
a
function
that
I've
called
it
xdp
and
we
attach
it
to
xdp
on
the
interface
f0,
which
is
the
same
as
what
we
used
for
the
socket
filter
before
it's
defined
up
here
also,
I
want
to
go
back
to
printing
out
tracing
and
we're
not
dumping
the
data
anymore.
A
There
we
go
so
we
see
the
ping
request
coming
in
through
xdp,
so
think
of
that
as
the
packet
arriving
at
the
network
interface
and
then
it
gets
to
the
raw
socket
where
we
can
see.
First
of
all,
the
request
which
is
destined
for
an
address
ending
in
zero
two
and
then
we're
actually
seeing
the
response
going
back
through
the
socket
filter,
because
the
socket
filter
sees
traffic
in
both
directions.
A
So
we
can
attach
these
filters
to
what's
called
queuing
disciplines.
We
can
do
this
in
both
ingress
and
egress.
My
example
is
going
to
use
ingress,
so
a
packet's
going
to
arrive
the
idea
of
this
traffic
control
is,
you
might
use
it
to
prioritize
certain
types
of
traffic
and
do
traffic
shaping
you
could
also
use
it
for
filtering
certain
types
of
traffic.
A
A
Okay,
so
my
program
again
doing
something
very
similar:
we're
going
to
chase
out
the
fact
that
we've
got
a
packet
and
then
we're
going
to
look
through
the
packet
using
the
same
function
as
before,
to
check
whether
it's
a
ping
request,
and
in
this
case
I'm
going
to
drop
it.
This
tc
act
shot
says
this
packet
is
shot
just
don't
we
don't
want
to
pass
it
on
up
the
stack.
A
So
tc
is
gonna
when
that
ping
request
reaches
this
tc
ingress
part
of
the
stack
ping
requests
are
going
to
get
dropped,
so
I
need
to
load
this.
It's
a
little
bit
more
involved
here,
because
we
need
to
create
the
queuing,
discipline
and
add
the
filter,
and
before
I
forget,
I
need
to
make
sure
it
gets
unloaded
at
the
end.
A
A
This
looks
like
some
other
packet.
That's
arrived
on
that
same
interface.
My
guess
is
that
that's
probably
some
kind
of
art
traffic
going
on.
We
could
add
some
code
to
to
check
whether
that's
true,
let's
run
another
ping,
just
to
demonstrate
yeah
we're
not
seeing
those
pings
being
responded
to
100
packet
loss,
we're
successfully
dropping
traffic
using
tc,
but
we
don't
just
have
to
drop
packets.
We
can
also
modify
them
and
we
can
even
make
decisions
about
sending
them
through
different
interfaces.
A
A
A
We
need
to
modify
the
type
of
the
message,
so
this
is
saying
this
is
a
response
rather
than
a
request
hidden
within
this
function.
It's
also
updating
the
icmp
checksum
in
the
header,
and
then
we
have
to
do
something,
maybe
a
little
bit
special
here.
What
we're
going
to
do
is
say:
throw
away
the
original
packet.
A
And
that's
loaded
so
we're
hoping
to
see
successful
ping
responses
this
time
and
we
did.
That
was
pretty
quick.
We
saw
the
packet
responded,
zero
percent
packet
loss
and
what
we
could
see
there
packet.
The
request
comes
in
xdp,
it
goes
through
socket,
filter
gets
to
tc,
tc,
looks
at
the
the
message,
but
gives
us
a
little
bit
more
information
about
that
message,
and
then
we
can
see
the
response
which
is
destined
for
address,
ending
in
one
being
sent
back
out
through
the
socket
filter.
A
A
One
way
we
can
check
is
to
use
that
perf
utility
to
trace
out
networking
related
events
that
happen
as
those
ping
messages
traverse.
The
networking
stack
so
instead
of
just
sending
ping
directly,
I'm
going
to
send
it
with
perf
trace,
trace
and
I'm
interested
in
all
the
network
events
and
I'm
going
to
send
it
to
an
output
file
called
let's
say,
with
tc,
and
then
I
want
to
run
ping
I'll,
send
it
once
and
we're
sending
it
to
the
same
dress
address
as
before.
A
A
I
ran
that
earlier
without
tc,
and
this
is
the
result
and
we
should
see
that
the
result
without
cc
is
a
little
bit
longer
than
with
yeah,
and
we
should
see,
starts
with
the
same
calls
and
we're
missing
out
on
some
network
interface.
Rx
calls
here.
So
that's
telling
us
that
something
further
up
the
stack
that
was
receiving
packets.
A
Those
pods
have
their
own,
typically
have
their
own
networking
namespace
separate
from
the
host's
networking
namespace
and
in
each
of
them
there's
a
separate
tcpip
stack.
So
imagine
a
packet
destined
for
our
application
running
in
a
pod.
It
arrives
at
the
physical
interface
to
the
host
it
traverses
the
whole
ip
stack
on
the
host
then
gets
passed
up
through
the
virtual
ethernet
connection
to
the
pod,
where
it
goes
through.
The
pods
networking
stack
with
ebpf.
A
We
can
bypass
a
lot
of
the
host
networking
stack
because
we
can
see
the
address
that
that
packet
is
destined,
for.
We
know
that
that
is
one
of
the
pods,
because
we're
a
kubernetes,
cni
and
we've
been
in
we've
been
responsible
for
setting
up
those
endpoints.
So
we
know
the
packet
is
destined
for
the
pod
and
we
can
send
it
straight
to
the
appropriate
virtual
ethernet
interface
that
makes
networking
significantly
faster
and
there's
a
really
great
blog
post
about
cni
benchmarking
on
the
civilian
website
that
you
might
want
to
check
out
for
more
details.
A
A
A
If
we
think
about
networking
in
particular,
that
creates
a
lot
of
really
cool
capabilities.
So
the
fact
that
we
can
inspect
packets
gives
us
some
really
powerful
observability
tools
that
are
really
performant.
That
can
be
mapped
to
other
aspects
like
kubernetes
identity.
So
we
can
have
really
detailed
mapping
between
what
network
packets
are
flowing
and
which
kubernetes,
pods
and
services
they
relate
to.
We
can
get
some
really
in-depth
security
forensics
from
this
information.
A
We
can
use
the
ability
to
drop
packets
or
to
modify
them
to
create
network
policies.
We
can
do
encryption
in
ebpf.
These
are
all
really
great
benefits
for
security
and
the
fact
that
we
can
send
packets
to
we
can
change
the
destination
for
a
packet
allows
us
to
create
all
sorts
of
interesting
network
functionality.
A
Things
like
load,
balancing
routing
services,
and
particularly
interesting,
I
think,
is
the
idea
of
an
evpf
enabled
service
mesh
and,
if
that's
something,
that
you're
interested
in
do
come
and
speak
to
us
about.
Psyllium
ebpf
is
enabling
the
next
generation
of
service
mesh,
because
we
don't
have
to
instrument
the
pods
that
make
up
those
services
in
order
to
have
these
service
mesh
capabilities.