►
Description
The Future of eBPF in Cloud Native - Thomas Graf, Isovalent
eBPF is taking the cloud native world by storm. Where will it lead us? This talk introduces eBPF by looking at the ecosystem of eBPF-enabled CNCF projects focusing on what they provide for end-users and how they will evolve in the coming years. The projects covered span a wide set of use cases including networking, security, service mesh, observability, and performance monitoring. We will answer questions such as "Why is eBPF so powerful?" "What is different to prior similar technologies?", and "What values does eBPF provide for end users?". To sum it all up, we will together look at how eBPF itself will evolve in the coming years and what impact that will have to the cloud native world.
A
I
think
many
of
us
of
you
have
may
have
seen
a
slight
similar
to
this
kind
of
ebpf
being
able
to
execute
code
in
the
kernel
when
some
event
happens
and
one
very
common
one
is
when
a
system
call
is
happening
and
then,
instead
of
the
system
call
being
performed
automatically
by
the
kernel,
we
can
execute
some
code
such
as
shown
here.
This
is
a
common
example
on
how
some
of
the
system
call
tracers
would
generate
an
event
in
the
kernel
to
extract
visibility.
A
A
A
What
I
mean
by
that
is,
when
you
think
back
20
years
ago,
many
of
us
probably
remember
we
had
to
upgrade
our
browsers,
our
netscape
navigator,
our
internet
explorers,
whatever
browser
we're
using
opera,
whatever
to
view
certain
websites,
and
if
you
often
see
this
message.
Oh
you
cannot
view
this
website,
because
your
browser
is
trolled
that
has
disappeared
because
of
javascript,
because
javascript
and
other
programming
languages
that
are
able
to
run
sandboxed
in
the
browser
made
the
browser
programmable,
so
web
developers
could
innovate
and
write
applications
without
having
to
build
something
into
the
browser.
A
First
and
all
of
a
sudden,
we
no
longer
need
to
upgrade
the
browsers
constantly.
This
led
to
massive
innovation.
Most
of
us
would
have
probably
never
guessed
that
something
as
complex
as
an
entire
word
processing
tool.
Microsoft
word
would
get
potentially
replaced
by
something
that's
running
in
a
browser,
but
it
has
happened,
and
it
has
happened
because
of
this
programmability.
A
It
has
happened
because
web
application
engineers
can
safely
innovate
as
part
of
the
browser.
Ebpf
is
exactly
the
same
for
the
linux
kernel
and
hence
it
will
lead
to
a
very
similar
wave
of
innovation
and
that's
what
we're
exactly
seeing
on
going
on
right
now,
we're
seeing
a
massive
innovation
of
networking
security,
observability
service
smash
performance
troubleshooting.
A
All
of
these
fields
are
going
through
a
massive
leap
of
innovation
because
of
this
programmability
I
was
a
kernel
developer
for
about
20
years,
and
this
was
a
massive
problem
and
it
shows
the
the
reason
why
innovation
was
so
challenging
in
in
the
in
the
corner
was
often
in
this
situation.
That
somebody
would
ask
me
a
question.
Can
I
get
this
into
the
kernel?
We
need
this
feature.
I
need
to
observe
my
app
better.
A
I
need
this
and
yes,
that's
cool
and
we
did
it
and
we
added
it
to
the
kernel,
but
it
took
years
and
years
and
years
for
those
kernel
versions
to
make
it
into
the
hands
of
users.
In
particular,
if
users
were
using
enterprise
linux
distributions,
they
would
lag
behind
many
years.
This
made
it
essentially
impossible
to
innovate
in
the
kernel
because
you
were
lagging
behind
so
much
because
of
the
dynamic
programmability
of
ebpf.
A
This
is
solved
and
all
of
a
sudden
we
can
innovate
and
give
new
code
give
new
functionality
into
the
hands
of
end
users,
essentially
on
a
daily
basis,
very
similar
to
how
javascript
has
done
this
to
the
browser,
and
this
is
leading
to
this
massive
innovation
in
the
kernel
space
that
was
not
possible.
Before
some
of
you
may
have
seen
the
ebpf
landscape
that
is
available
on
ebpf.io.
A
For
today,
I've
created
a
new
landscape.
It's
not
an
official
one.
That's
just
mine!
I've
just
put
it
together
for
educational
purposes
here
with
a
a
very
strict
cloud
native
focus.
So
I
think
not
all
of
the
projects
that
you
will
find
on
ebpf.org
are
in
here,
but
the
ones
that
are
found
are
very
compelling
for
a
for
a
cloud
native
world.
A
On
the
lower
level
we
have
like
the
platform
layer
layer.
We
can
see
that
ebpf
can
run
on
windows
and
on
linux,
then
we
have
the
sdk
and
packaging
layer.
This
is
the
layer
that
makes
consuming
ebpf
easier
because
ebpf
itself
is
very
low
level.
Byte
code
kernel
level,
so
this
this
middle
layer
makes
it
easier
to
actually
use
ebpf.
There's
a
variety
of
programming
or
language
bindings,
go
rust,
c,
plus,
plus
bumblebee
making
packets
easier,
and
then
application,
observability,
networking
and
service
mesh
and
security,
and
you
can
see
a
variety
of
projects.
A
Some
of
them
are
in
the
cloud
id
foundation,
spxe
psyllium,
chachugon
falco,
we'll
talk
about
chatagon
in
a
little
bit
for
many
of
you
probably
hear
about
it
for
the
first
time
requirement,
not
all
ebpf
projects
need
to
be
in
the
cloud
native
foundation
in
order
to
be
useful.
Of
course,
if
you
look
into
that
close
into
that
lower
level
a
bit
more
because
that's
what
most
people
regard
as
ebpf,
it's
this
runtime.
So
how
does
that
run
time
actually
look
like.
I
only
have
one
slide
here
kind
of
on
the
technical
side.
A
In
this
case,
I'm
using
clang
as
a
way
to
compile
pseudo
c
code
into
ebpf
bytecode,
but
you
could
use
another
tool
chain
as
well.
Essentially
what
you
need
is
bytecode
that
you
can
then
inject
into
the
kernel.
You
then
use
typically
a
library
whether
it's
lib
ppf
or
go
ppf
or
the
rust
or
io
or
whatever
it
is
to
actually
load
the
ebpf
code
into
the
kernel
with
a
system
call
so
there's
a
specific
ebpf
system
call.
A
You
want
to
to
run
your
your
ebpf
code
on,
for
example,
execute
this
program
when
a
certain
system
call
is
being
made
or
when
this
network
packet
is
being
processed
or
when
new
data
is
available
in
the
socket
or
when
a
tcp
retransmission
happens
or
when
a
disk
is
being
accessed
and
so
on,
then
we
have
ebpf
maps.
These
are
data
structures
and
a
great
way
to
extract
visibility
from
kernel
space
into
user
space,
so
these
are
hash
tables,
maps
and
so
on.
A
So
to
summarize
kind
of
the
ebp
for
the
zbpf
ebff
gives
us
amazing
programmability
and
is
it's
leading
to
this
innovation
very
similar
to
what
javascript
has
done
to
the
browser?
I
think
most
of
us
see
the
excellent
visibility
and
control
as
a
as
a
cornerstone
or
everything.
We
can
see
a
lot
more
things
with
ebpf.
We
have
deeper
visibility,
but
we
also
have
a
lot
more
control
and
we'll
see
a
couple
of
examples
of
this
control.
A
A
A
year
later,
the
llvm
compiler
received
its
ebpf
backend
back
in
these
days,
ebpf
was
really
really
really
low
level,
so
it
was
only
meant
for
kernel
engineers,
kernel
kernel
developers.
So
having
an
llbm
background,
oh
we
can.
We
can
write
sudo
c
code.
That's
not
that's
great!
That's
a
high
level
language
like
for
kernel
developers.
That's
a
high
level
language!
Also,
this
cls
bpf
was
merged.
A
It
was
a
set
of
python
based
scripts
that
would
allow
you
to
introspect
system
applications
and
and
so
on,
and
today,
even
today,
bcc
is
massively
used.
It
has
a
wide
variety
of
use
cases
that
it
can
cover
a
year
later,
xdp
was
merged.
Xdp
is
the
high
performance.
Networking
layer
of
ppf
allows
ppf
to
compete
with
technologies
like
dpdk
user
space,
networking,
for
example,
and
psyllium.
Was
it
was
announced?
So
in
2016
we
announced
psyllium
a
year
later,
brandon
coins,
brandon
greg.
Some
of
you
may
have
heard
him.
Talk
on.
A
Ebpf
is
a
massive
figure
involved
in
the
evolution
of
ebpf
coin
to
term
linux.
Bpf
superpowers,
and
it
was
very
very
fitting,
like
bpf,
is
a
super
power
in
the
operating
system
that
allows
us
to
do
things
that
were
not
possible.
Before
also
same
year,
facebook
came
out
and
basically
said,
hey
world.
This
is
how
we
do
load
balancing
now
and
it's
10
times
faster
as
ipvf
ipvs
and
by
the
way
we're
using
vpf
and
the
world
was
like
wow.
A
How
did
that
happen?
Like
10x
10x
improvements
are
very
rare,
like
for
mature
production
in
production
in
infrastructure,
this
code
is
known
as
catron.
Today,
it's
available
open
source
also
cloudflare
started
migrating
all
of
their
ddos
use
cases
that
these
ddos
codes
over
to
bpf-
and
I
think
in
this
year
it
became
clear.
Okay,
ebpf
is
going
to
be
a
thing
like
there's
massive
players.
Betting
on
ebpf
ebpf
is
going
to
be
a
technology
that
changes
the
world
bpf
trace
was
announced
2019
as
a
replacement
or
as
an
evolution
of
bcc.
A
You
may
have
seen
to
you,
may
have
seen
data
plane.
We
too
in
gk,
that's
zillion-based.
It's
ebpf-based,
I
think
major
milestone
for
ebpf,
for,
like
a
major
cloud
provider
is,
is
making
their
products
based
on
ebpf.
So
in
one
nine
with
maglev
low
balancing
support
and
the
first
acquisitions
happened,
so
2020
new,
relic
acquired
pixie
in
the
observability
space
and
pixi,
is
today
a
sandbox
project.
A
I
think
this
was
the
second
ebpf
summit
that
we
had
completely
overrun
with
over
3000
attendees,
virtually
a
massive
event
still
in
111,
with
topology
routing,
bgp,
open,
telemetry
support
and
so
on,
like
lots
of
innovation
going
on
which
leads
us
to
this
year
and
the
year
is
not
even
that
old
yet,
but
we
already
have
a
couple
of
very
new
thoroughly
items
still
in
service
mesh.
We
announced
this
year
and
tetragon.
I
will
have
a
slide
later
on
on
this
we're
announcing
this
week.
An
ebpf
summit
2022
will
happen
in
the
in
september.
A
That
was
a
brief
history
and
if
you
look
back
kind
of
it
started
in
the
same
year
as
kubernetes
and
ebpf
has
gone,
come
a
very,
very,
very
long
way
and
we're
not
only
seeing
the
big
hyperscalers,
google,
facebook,
netflix
and
so
on,
leverage
ebpf
internally
today,
ebpf
is
used
in
enterprises.
End
users
use
ebpf
directly
and
there
is
a
wide
set
of
end
user
projects
that
are
available.
If
you
look
back
into
the
landscape,
let's
look
a
little
bit
into
the
future
as
well
like.
A
A
Service
mesh
is
for
those
who
have
never
heard
about
service
mesh
service.
Mesh
is
a
concept
where
essentially
code,
that
used
to
live
in
applications
themselves,
to
secure
communication,
to
perform
client-side
load,
balancing
to
gain
introspection
to
do
retries
to
move
that
code
out
of
the
applications
themselves
and
move
it
to
somewhat
transparent.
So
that
code
doesn't
have
to
be
written
for
each
application
separately
and
the
most
appealing
first
way
of
doing
this
was
to
run
a
so-called
sidecar
and
run
this
code
in
a
proxy.
A
So
the
same
proxy
could
be
placed
in
front
of
every
application
and
run
this
code,
for
example,
perform
mtls
or
retries
or
layer,
7,
load,
balancing,
retrieve,
telemetry
and
so
on.
But
if
you
look
at
service
mesh
and
the
desire
for
it,
it's
actually
close
to
tcp
or
ipsec.
It's
like.
We
want
visibility.
We
want
security.
We
want
connectivity,
it's
just
a
much
better
version
of
this.
So
if
we're
talking
about,
for
example,
http
retries,
it's
conceptually
very
similar
to
a
tcp
retry
right,
we
want
reliable,
consistent
communication.
A
A
So
I
think
one
way
to
look
at
this
is
not
necessarily
to
think
about
that.
All
the
functionality
that
is
currently
done
by
a
proxy
has
to
move
into
ebpf.
I
think
the
concept
should
be
move
some
of
it
into
the
kernel
actually
integrates
the
proxy
with
ebpf
and
collaborate.
I
have
one
example
here
where
it
shows
the
impact
on
the
lower
left.
You
can
see
performance
for
http,
based
visibility,
and
the
numbers
shown
here
are
latency
numbers,
so
lower
is
better
and
you
see
a
bar
for
no
visibility.
A
A
I
think
we've
already
seen
a
lot,
but
we've
only
just
begun.
So
there
have
been
acquisitions,
there
have
been
new
tools
announced,
but
in
terms
of
potential
and
what
can
be
done
with
ebpf
we've
just
barely
scratched
the
service.
I've
listed
several
projects
up
here
that
will
provide
observability
that
are
providing
observability.
A
I
would
say:
let's
watch
this
space
very
very
closely.
Not
all
of
these
tools
provide
the
same
visibility.
We
have
tools
like
pixi,
focusing
really
on
application
monitoring
and
application
visibility,
also,
for
example,
performance
troubleshooting.
Where
is
my
memory
used?
Where
does
my?
Where
is
my
cpu
being
used
up
bcc?
It's
a
bit
more
focused
on
the
system
level,
visibility,
what
files
are
being
opened?
What
connections
are
happening?
Who
is
doing
right
system
calls,
and
so
on?
A
Cubecontrol
trace
is
bringing
bcc
tooling
to
a
kubernetes
specific
environment,
because
bcc
is
not
kubernetes
aware,
apply,
tracing
or
function
tracing
tetragon,
we'll
talk
in
a
bit
inspector
gadget,
similar
to
control,
trace,
making
visibility
and
ebpf
very
approachable
on
kubernetes
parker.
I
think
similar
to
pixi
a
new
company
created
parka.
Definitely
something
to
watch
out
for
tracy
from
aqua
system
called
tracing,
so
I
think
there
will
be
more
and
more
toolings
here
and
I
think
we'll
see
this
space
explode
in
terms
of
observability
that
is
being
provided
with
low
overhead
all
right.
A
So
to
me,
what
I'm
most
excited
about
in
terms
of
ebp
future
is
actually
the
security
angle,
because
I
feel
we
can
make
together
a
huge
leap
forward
in
terms
of
security.
So
you
will
see
an
announcement
go
out
later
today
about
tetragon
tetragon
is
the
knowledge
of
the
psyllium
team
of
like
ebpf
creators
of
salem
craters,
apply
that
same
knowledge
to
the
to
the
runtime
security
and
security
observability
space,
so
chaturgaon
will
provide
both
observability
and
runtime
enforcement.
A
The
name
spacing
like
what
sort
of
privilege
escalations
are
going
on
what
processes
are
breaking
out
of
their
namespaces
file,
access,
tcp
the
whole
stack,
for
example,
tracing
tcp
sequence
numbers.
Obviously,
the
system
call
layer
system
calls
that
are
being
done,
process
execution
and
all
the
way
into
the
application.
Bpf
can
do
function.
Tracing
bpf
can
generate
stack.
Traces
bpf
can
even
monitor
the
code
that
is
being
executed.
A
When
we
look
at
enforcement
today,
typically
enforcement
either
happens
strictly
at
the
system.
Call.
This
would
be
something
like
seccomp,
where
you
are
allowed
to
do.
This
system
called
yes
or
no
right
like
you,
this
application
can
do
this
system
called
yes
or
no,
and
that's
great.
It
reduces
the
attack
surface
or
we
have
security
systems
that,
for
example,
use
ptrace
or
a
kernel.
A
Module
extract
visibility
and
then
asynchronously
kill
the
process
if
something
bad
happens,
with
tetragon
we're
using
ebpf
enforcement
inside
of
the
kernel,
which
means
that
instead
of
enforcing
just
at
the
system,
call
level
where
we
need
to
understand
what
is
good
and
bad.
We
can
monitor
what
are
undesirable
outcomes
such
as
I
don't
want
this
container
to
ever
break
its
namespace
boundary
when
that
ever
happens,
kill
it
immediately,
don't
allow
it
to
run.
If
you
look
at
the
upper
right,
you
see
exactly
this.
A
What's
even
more
fascinating
is
that
is
extremely
in
line
with
bpf
lsm.
So
I'm
eighty
percent
excited
for
try
to
go
on,
and
twenty
percent
excited
for
lsm
lsm
bpf
lsm
is
the
security
modular
framework
inside
of
the
linux
kernel,
it's
giving
an
opportunity
to
load
ppf
programs
at
the
variety
of
hook,
points
as
well
and
encode
logic.
In
there.
The
downside
of
ls
and
bpf
is
that
it
requires
relatively
recent
kernels
with
five
seven
tetragon
runs
on
all
the
curls.
That's
why
I'm
slightly
more
excited
about
tetragon,
but
that's
gonna
need
lsm.
A
So
with
that,
I'm
super
excited
for
the
future
of
ebpf.
I
think
most
of
the
the
real
innovation
is
yet
to
happen,
even
though
we
have
done
a
lot
of
in
a
lot
of
innovation
together
and
before
I
open
it
up
for
questions.
I
do
want
to
call
out.
We
have
formed
an
ebpf
foundation
together
with
many
players
in
industry.
If
you
want
to
collaborate,
feel
free
approach.
The
linux
foundation,
evpl
foundation
is
up
for
membership.
A
C
A
Yeah,
it's
a
fantastic
question.
I
think
that's
exactly
what's
happening,
but
it's
obviously
unrealistic
to
rewrite
the
entire
linux
kernel,
which
means
that
ebpf
actually
gives
us
a
practical
approach
to
achieving
this.
We
can
replace
certain
parts
of
the
kernel
with
vbpf
logic.
We
can
take
side
cuts
or
shortcuts.
A
A
They
are
essentially
have
started
writing
new
code
in
ebpf
instead
of
natively,
because
it's
more
secure
because
it
is
subject
to
the
very
fire,
and
so
so
I
think
it's
what
you
said
is
absolutely
correct.
I
think
we
are
moving
to
a
more
micro
kernel
based
architecture,
but
rewriting
the
kernel
is
not
an
option,
so
I
think
ebpf
gives,
and
it
gives
away
to
essentially
work
around
that
and
start
to
refactor
without
rewriting.
A
Very
good
question
yeah,
so
hubble
is
strictly
dependent
on
psyllium,
so
it's
visibility
based
on
the
networking
layer
of
psyllium
tetragon
is
a
hundred
percent
independent.
It
does
not
require
psyllium
in
any
way
it
uses.
You
don't
need
to
run.
Psyllium
tetragon
is
is
independent
and
obviously
the
focus
point
is
a
little
bit
different
as
well.
It's
primarily
about
runtime,
but
it
can
actually
also
extract
network
visibility,
but
it
does
that
we're
now
parsing
that
or
packets,
for
example,
so
difference
essentially.
B
E
E
A
That
exists
so
there's
actually
ebpf
for
ebpf.
There
was
a
fantastic
talk
last
in
the
last
ebpf
summit
last
summer,
where
you
can
use
ebpf
profiling
technology
to
actually
understand
how
much
memory
which
parts
of
the
vpf
program
are
consuming,
how
much
cpu
and
you
can
essentially
create
flame
graphs
of
which
part
of
my
epa
program,
consume
memory
and
so
on,
and
also,
I
think,
in
terms
of
other
consumption.
A
A
So
there
is
evpf
offload
to
a
variety
of
smartnic
providers,
I'm
assuming
you're
talking
primarily
networking
use
case.
Given
the
tls
context,
it
does
exist.
I
cannot
tell
you
exactly
which
hardware
vendors
offer
the
tls
offload
part,
but
because
ebpf
is
super
generic
for
kernel
level
technology,
the
offload
is
possible
and
all
the
big
smallnic
providers
have
announced
and
will
tell
you
that
they
offer
ebpf
offload,
but
for
more
specific
questions.
You'll
have
to
ask
them.
A
Excellent,
yes,
there
is
lots
of
opportunity
right,
so
I
think
we
often
quickly
talk
super
low
level
because
it's
super
fascinating,
but
it's
actually
a
great
opportunity
to
contribute
to
ebpf
on
all
levels.
I
would
start
with
go
to
ebpf.io
that
has,
I
think,
first
of
all,
material
to
get
started.
It
also
has
a
link
to
a
slack
channel,
which
has
I
think
by
now
over
8
000
people
involved
from
all
sorts
of
projects,
and
you
can
quickly
find
an
ebpf
project
to
collaborate
on.
A
A
Now
you
can
essentially
work
on
end
user
projects
or
you
can
work
on
one
of
the
libraries,
for
example.
So
let's
say
you
are
a
go
developer
or
you
can
do
grost
all
these
projects.
They
need
help,
they
need
collaboration
and
you
will
learn
about
ebpf,
as
you
call
as
you
contribute
to
these
to
these
libraries,
but
I
think
even
the
understanding
of
that
other
language
is
sometimes
and
even
more
important
than
having
in-depth
ebpf
knowledge.
A
H
A
I
think
what
usually
works
well
is,
I
think,
let's
give
a
fantastic
talk
around
what
is
ebpf
last
year.
I
think
sharing
that
link
is
is
an
awesome
first
step
because
it
gives
this
entry
level.
What
is
evpf,
what
are
the?
What
are
the
benefits
for
everybody,
and
then,
I
think
quickly,
going
to
the
projects
that
bring
value
so
instead
of
talking
like
ebpf,
is
fascinating
on
how
it's
being
done,
but
the
actual
value
is
in
the
projects
themselves,
so
in
what
they
provide.
A
Oh
here
is
how
we
could
do
like
if
you
talk
to
your
to
your
app
teams,
look
at
this
demo
on
how
you
can
figure
out
which
parts
of
your
app
are
using,
how
much
cpu
and
their
eyes
will
lighten
up,
but
when
they
see
a
flame
graph
for
the
first
time
they
will
say.
Oh
that's
great.
I
want
that
and
then
ebpf
is
more
of
an
implementation
detail
that
allows
this
or
hey.
We
need
to
run
networking
at
scale.
We
can
point
them
at
psyllium
or
you
can
say
I
want
visibility.
A
I
Yeah,
thank
you.
Actually,
I
have
a
couple
of
questions.
The
first
one
is
that,
like
I
had
the
feeling
when
you
talked
about
the
proxies
in
front
of
applications,
I
thought
directly
about
sto
and
I
thought
is,
is
going
to
be
replaced
entirely
like
using
some
custom
scripts
with
ebpf
sdks
like
golang
or
rust.
This
is
the
first
one
and
the
second
one.
Perhaps
I
I
will
let
you.
A
Yeah,
let's
do
the
first
one
and
then
you
can
ask
the
second
one.
So,
unfortunately,
I
cannot
cover
everything
in
this
talk
around
service
mesh,
but
in
a
nutshell,
we
don't
want
to
replace
service
mesh
control
planes.
We
are
data
path,
people
so,
and
we
also
don't
want
to
replace
envoy
there's
actually
we'll
have
a
talk
tomorrow
at
servicemeshcon.
If
you're
there
with
a
bit
more
details,
we
want
for
users
to
be
able
to
pick
whatever
control
plane.
They
want
whether
this
is
istio,
whether
it
is
smi
whether
this
is
just
kubernetes.
A
Oh,
I
just
want
to
use
ingress
or
gateway
api
doesn't
matter,
and
then
we
provide
the
most
efficient
data
path
with
a
combination
of
envoy
and
ebpf
together
and
move
that
into
the
kernel
and
essentially
get
rid
of
psi
cards.
That's
our
motivation.
We're
not
interested
to
create
yet
another
set
of
crds
that
make
the
whole
space
more
complicated.
I
Yeah
crystal
clear,
the
second
question
is
like
I'm
more
like
a
devsecops
and
I'm
so
interested
in
these
scripts
using
rust,
and
I
was
wondering
how
do
you
do
the
ci
stuff
like
make
it
deterministic
like
using
ci
pipelines
and
making
sure
that
it
works
with
different
kernels
and
so
on?
Is
there
some
good
practices,
some
recommendations
to
follow
yeah.