►
From YouTube: Beyond the Buzzword: BPF’s Unexpected Role in Kubernetes - Andrew Randall & Alban Crequy, Kinvolk
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2021 Virtual from May 4–7, 2021. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Beyond the Buzzword: BPF’s Unexpected Role in Kubernetes - Andrew Randall & Alban Crequy, Kinvolk
Increasingly, cloud native tools are leveraging the Linux kernel’s Berkeley Packet Filter (BPF) capabilities for a range of applications, such as networking, security, observability, and troubleshooting. In recent Linux kernel releases, BPF has been significantly enhanced, resulting in this proliferation of BPF-powered cloud native projects. In this talk, we will go beyond the BPF buzzword, review the latest relevant developments in the Linux kernel and the rich ecosystem of cloud native tooling built on it. With live demos we will show how easy it is to deploy these BPF-based tools into your Kubernetes cluster, gaining greater visibility and control at the pod level of granularity.
https://sched.co/ekDR
A
A
First
of
all,
it
allows
the
user
to
write
programs
that
run
in
kernel
space
now,
as
brendan
greg
of
netflix
put
it.
This
is
a
fundamental
change
to
a
50
year
old
kernel
model,
because
up
until
now,
you've
been
used
to
writing
programs
that
run
in
user
space.
Now
you
can
actually
run
them
in
the
context
of
the
kernel.
A
Bpf
gives
you
some
capabilities
that
really
help
when
it
comes
to
monitoring
applications
and
also
particularly
monitoring
them
from
a
security
perspective,
to
check
that
you
haven't,
got
attacks
in
your
cluster,
so
these
are
all
pretty
good
reasons
to
care
about
bpf.
I
think
so
at
this
point,
I'm
going
to
hand
over
to
alban
to
talk
to
you
a
little
bit
about
the
history
of
ebpf
and
how
we
got
to
where
we
are
today.
B
So
it
started
in
1997
where
we
had
the
first
version
of
bpf.
It
was
used
for
tcp
dump
so
for
network
capturing,
and
that
was
pretty
much
it
until
much
later
in
2014,
where
we
have
the
new
ebpf
for
extended
ebpf,
so
alexey
starov
come
up
comes
up
with
the
id
of
a
universal
internal
virtual
machine
where
we
can
have
different
ebpf
programs
running
together
for
a
variety
of
applications.
B
A
bit
later
in
2015
we
have
the
iovisor
project
established
that
comes
initially
from
a
plum
grid
and
technology
from
plum
green
and
then
it
get
moved
into
the
under
the
linux
foundation
umbrella
arbitrator
in
2016.
We
have
xdp
for
express
data
path
and
that's
the
idea
to
apply
the
bpf
program
very
early
in
the
network
stack
so
that
right
in
munich.
Ideally,
we
can
filter
packets
on
a
block,
unwanted
traffic,
so
it
make
it
very
fast.
B
B
That's
used
for
intra-host
network
to
make
it
very
fast
by
bypassing
the
network
stack,
that's
used
in
psyllium
on
istio
when
we
use
easter
the
service
mesh
together
with
selium,
for
example,
in
2018
facebook
announced
the
project
catron,
that's
a
bpf
load
balancer
on
facebook
and
also
the
world
that
they
are
using.
That's
on
their
networks,
2018.
B
B
We
have
a
new
kind
of
bpf
ring
buffer.
That's
a
communication
mechanism
between
the
bpf
programming
kernel
to
user
space
as
a
new
ring
buffer.
That
is
more
much
more
efficient
memory
wise.
So
I
picked
a
few
of
those
items
to
show
how
it's
relevant
to
kubernetes.
For
example,
with
acilium
or
istio
or
tracing
tools
on
kubernetes.
A
So,
let's
take
these
capabilities
that
adolf
talked
about
and
look
at
the
landscape
of
tools
that
are
out
there
that
you
can
use.
So
the
the
first
layer
that
I
want
to
talk
about
is
there's
a
set
of
low-level
tools
which
a
lot
of
them
have
been
around
for
quite
a
while,
where
you
get
very
low
level,
access
to
bpf
capabilities,
typically
on
an
individual
host
at
a
process
level,
something
like
bpf
tool
which
actually
ships
with
the
linux
kernel
or
the
bpf
compiler
collection,
which
we'll
talk
about
in
a
minute.
A
If
you're
coming
at
bps
from
a
programmer
perspective-
and
you
want
to
be
able
to
write
programs
to
inject
into
the
kernel,
then
there
are
libraries
available
that
you
can
use
that
shortcut
a
lot
of
these
capabilities
to
make
it
a
lot
simpler,
whether
you're
running
in
go
or
c-plus,
plus
or
rust,
or
python.
There's,
probably
a
good
library
for
you
out
there.
A
When
you
we
get
the
next
layer
up.
We
start
to
look
at
more
complete
solutions
and
less
kind
of
individual
apis
and
libraries
and
in
the
security
and
networking
space
is
one
of
the
main
areas
for
the
bpf
is
used.
Psyllium
was
one
of
the
first
bpf
enabled
networking
solutions
in
kubernetes,
but
calico
has
added
it
as
well.
Recently,
catran
alma
mentioned
falco
is
used
in
for
security
applications
from
sysdigs.
So
a
lot
of
things
you
can
plug
into
kubernetes
there,
then
the
last
layer
in
visibility.
A
This
is,
I
think
you
know
something
that
anyone
from
an
operations
perspective
should
be
looking
at
these
tools
to
say
you
know.
How
can
this
give
me
greater
insight
to
what's
running
in
my
cluster,
so
things
like
weave
scope
or
hubble
actually
provide
the
capability
to
see
in
real
time
what
what
network
connectivity
is
happening?
How
how
things
are
connecting
in
the
cluster
there's
a
few
other
tools
as
well,
which
don't
fit
neatly
into
any
of
these
categories.
A
Things
like
the
cloudflare,
ebpf
exporter,
project
or
net
cost,
which
actually
analyzes
your
network
traffic.
To
give
you
some
idea
of
what
it's
going
to
be
costing
you
to
run
that
in
the
cloud
as
an
example
of
one
of
these
tools,
hubble
is
from
psyllium
folks,
sits
on
top
of
the
psyllium
cni
plug-in
and
surfaces
up,
metrics
and
and
also
analyzes
kind
of
a
graph
view
of
your
traffic
in
the
hubble
ui,
and
looks
something
like
this.
You
know
nicely
displayed.
A
You
know
in
this
graph
here
you
can
see
which
pods
talking
to
which
pod
all
thanks
to
bpf
one
of
the
nice
things
about
this
is
as
a
part
of
this
project.
We
developed
a
library
called
tcp,
tracer
bpf
and
you
know
that's
available.
So
if
you
wanted
to
build
something
similar,
a
lot
of
the
base
capabilities
are
there
for
you
to
build
on
the
bps
compiler
collection.
A
A
You
know
really
quite
easy
to
get
up
and
running
and
using
any
of
these
tools-
and
I
recommend
you
check
those
out
if
you
haven't
already
bpf
trace,
is
kind
of
the
cousin
to
bcc,
maybe
less
well
known
and
less
frequently
used,
because
it
does
have
a
bit
of
a
learning
curve.
But
it's
great
for
quickly
writing
custom
bpf
functions.
It's
got
a
lot
of
flexibility
for
tracing
and
debugging
your
linux,
apps.
B
B
So
what
do
we
need
to
make
a
bpf
tool
community
somewhere?
We
need
a
few
things.
First,
the
granularity
that
we
care
about.
We
don't
want
to
trace
at
the
pid
level
process
level.
We
don't
necessarily
want
to
trace
all
the
process
on
the
system.
We
only
want
to
trace
a
process
from
a
specific
pods.
Usually
that's
not
something
so
easy
to
do,
because
a
bpf
program
is
running
in
the
canal.
They
don't
know
anything
about
humanity,
spots
or
kubernetes
levels,
so
we
need
to
do
a
bit
of
work
there.
B
Another
thing
we
want
to
do
is
to
aggregate
the
information
by
labels,
so
we
don't
want
to
see
the
piece
or
to
select
individual
process
but
aggregate
using
kubernetes
labels,
and
the
last
thing
I
want
is
to
have
a
cup
city
like
experience,
so
developers
should
not
need
to
ssh
on
a
specific
node.
They
should
not
need
to
know
in
advance.
This
is
the
node
where
my
product
is
running.
B
B
It
doesn't
ssh-20
node,
it
use
kubernetes
native
concepts
like
pods
on
demand
set
and
so
on
from
there
inspector
gadget
deploys
a
board
on
each
node.
That's
a
demand
set,
and
each
of
them
will
be
able
to
execute
the
different
gadgets
and
it
will
execute,
for
example,
trace
loop
or
those
from
bcc,
and
those
will
install
a
bpf
program
in
the
kernel
and
the
kernel
will
be
able
to
gather
the
information
and
send
it
back
and
send
it
back
up
to
the
cli
to
keep
ctl
gadgets.
B
So
what
do
we
have
today?
As
gadget
in
inspector
gadget?
The
first
one
here
is
capabilities.
It's
a
way
to
get
information
about
capabilities
that
are
exercised
in
your
body.
The
use
case
is,
for
example,
you
have
a
part
that
needs
some
privilege.
Maybe
you
need
cabinet,
I
mean,
can
season
me,
but
it's
quite
difficult
to
know
if
you
are
not
deep
into
this
level
of
kernel
knowledge.
B
So
what
often
people
do
is
just
to
give
all
the
capabilities,
but
that's
not
so
great
security-wise,
instead
of
just
giving
what
you
need
this
tool
is,
allows
you
to
run
the
pod
and
see
what
actual
capabilities
are
exercised
by
the
product,
and
then
you
can
pick
that
information
to
write
your
prospect
on
psps
other
gadgets
in
inspector
gadget
are
open.
Snoop,
exact,
snoop
bands,
loop.
That
tells
you
they
tell
you
when
a
file
is
open
when
a
new
binary
is
executed
or
when
there
is
a
new
tcp
port
open
on
by
your
parts.
B
So
you
can
see
this
information
in
real
time.
As
it
happens,
other
tools
are
tcp
top
on
tcp
tracer.
So
that
gives
you
the
network
traffic
happening
inside
the
pod,
and
the
watch
is
the
most
voluminous
traffic
with
tcp
top
on
another
is
tcp
tracer,
where
you
can
see
one
new
line
for
each
new
tcp
connections
to
see
what's
happening
in
your
pods
profile.
Is
a
cpu
profiler
that
this
one
comes
directly
from
vcc
as
one
it's
useful
to
see
why
something
is
so?
B
For
some
reason,
there
was
a
bug
in
a
version
of
humanities,
and
then
this
generate
the
network
traffic
to
be
slow,
but
that
was
not
so
easy
to
debug.
With
this
tool
we
could
see
what
are
the
network.
Sorry,
the
kernel
stacks
on
user
space
stack
that
are
mostly
most
often
called,
and
then
I
can
see
what's
happening
there,
a
trace
loop
is,
I
call
it
a
flight
flyer
recorder.
B
B
B
Let's
see
the
first
tool
I
want
to
show
you
today
is
exact
snoop,
so
that's
a
inspector
gadget
tool
that
allows
me
to
see
what
kind
of
program
are
executed
inside
each
pod.
So,
let's
get
started
cube.
Ctl
gadget
has
to
run
inspector
gadget,
choose
the
tool,
exact
snoop,
and
then
I
don't
want
to
filter.
I
don't
want
to
get
information
about
all
the
parts,
but
only
a
specific
one.
I
will
select
the
namespace
default
and
then
I
will
use
a
kubernetes
label
selector
to
choose
only
the
ports
with
the
label
run
equal
cooking.
B
B
B
B
B
B
A
B
B
And
the
first
step
is
to
use
this
subcommand
monitor
to
record
the
network
traffic,
and
I
will
only
select
a
list
of
namespace,
so
in
this
case
only
one
and
I
will
output
the
result
in
this
file
network
trace,
don't
log
okay,
so
what
it's
doing
is
installing
the
bpf
filter
and
every
time
there
is
a
new
connection,
a
new
network
traffic.
In
that
name
space.
It
will
record
something
in
that
file.
B
B
B
B
B
So
attack
this
network
trace
file
that
I
just
generated,
and
then
I
will
create
network
prolicy,
dot,
jml,
okay.
So
now
I
can
look
into
this
network
policy
ml
and
that's
it
contains
just
the
list
of
network
policy
that
has
been
generated
by
inspector
gadget.
So
I
can
see,
for
example,
that
the
card
service
is
about
to
talk
to
radius
on
this
tcp
port
and
so
on.