►
Description
For more great content, visit https://solocon.io
SoloCon2022:
From the Cloud to the Kernel: Powering Observability with eBPF
Speakers:
Eitan Yarmush
Architect in the Office of the CTO, Solo.io
Lawrence Gadban
Field Engineer, Solo.io
Session Abstract:
Have you tried to create and run your own eBPF programs? There are several valid approaches for getting started with eBPF, but they can be intimidating to a newcomer. Most of these mechanisms have advanced toolchains to configure, require writing boilerplate code, require jumping between programming languages, in addition to other challenges. In this session, we will present our new open-source project BumbleBee, which intends to enable you to quickly and easily create, run, and share your eBPF programs with a low barrier of entry. Additionally, with a live demo, we will explore what is happening behind the scenes to give you a high-level understanding of eBPF and how to use BumbleBee to get started with writing and sharing your own programs.
Track: Community and Open Source
A
A
What
you
really
are
doing
is
writing
a
packet
filter
and
that's
what
classic
vpf
allowed
you
to
do
extended
berkeley
packet
filter
is
kind
of
the
the
evolution
of
that
which
enables
all
sorts
of
really
interesting
technologies
to
to
come
about.
A
quick
note
is
that
evpf
and
bpf
will
be
used
interchangeably
going
forward.
A
lot
of
the
pioneers
of
ebpf
at
this
point
have
kind
of
started,
saying
that
evpf
is
no
longer
an
acronym.
A
A
The
programs
that
are
written
in
ebpf
are
event
based,
and
what
you
can
do
is
attach
your
program
to
various
hook,
points
that
then,
will
be
triggered
whenever
those
events
are
are
hit.
So
one
example
is
something
that's
called
a
k
probe.
A
k-probe
program
is
a
is
a
program,
that's
attached
to
essentially
any
kernel
function,
and
then,
whenever
that
function
is
called
your
epf
program,
that's
attached
to
it
will
be
executed.
A
Another
really
important
part
about
vpf
programs
is
that
they
are
verified
to
be
safe.
What
that
means
is
that
there
is
a
verifier
that
lives
in
the
kernel.
When
you
load
a
bpf
program
in
it's,
it
will
be
verified
that
it
won't
crash
the
kernel
it's
guaranteed
to
return.
In
other
words,
you
can't
write
infinite
loops,
there's
only
specific
sections
of
memory
that
are
available
to
programs
and
so
on.
A
So
really
you
get
a
guarantee
that
the
programs
that
are
being
loaded
into
the
kernel
are
safe,
and
this
gives
you
a
lot
of
confidence
going
forward
because
you
know
before
bpf
you
can
do
things
like
write,
kernel
modules,
which,
if
there
is
a
a
bad
kernel
module
it
can
bring
down
your
entire
system.
Bpf
works
around
that
by
verifying
the
programs
before
they
are
accepted,
so
an
actual
ebpf
program.
Let's
talk
about
that
for
a
little
bit.
A
The
program
itself
is,
as
outlined
here,
a
sequence
of
64-bit
instructions
that
comes
from
kind
of
the
unofficial,
ebpf
docs,
which
are
linked.
Essentially,
it's
it's
a
sequence
of
bytecode
or
assembly,
and
it
bp
bbpf,
has
its
own
instruction
set
so
that
assembly
that
by
code
stan,
it
is
its
own
kind
of
assembly
that
byte
code
is
then
interpreted
in
kernel,
there's
an
in
kernel
virtual
machine
which
can
understand
those
instructions
and
actually
execute
the
the
program.
A
That
being
said
on
most
architectures,
that
bytecode
will
actually
be
just
in
time
compiled,
and
that
means
that
the
bytecode
evpf
by
code
will
then
be
compiled
into
native
machine
code,
which
means
that
an
evpf
program
on
most
architectures
can
run
just
as
fast
as
native
code,
which
is
really
important
for
when
you're
trying
to
do
processing.
You
know
several
events
in
the
kernel
and
kind
of.
A
As
far
as
writing,
evpf
programs
go.
There
are
several
frameworks
or
libraries
out
there
that
help
with
this
bcc
or
the
bpf.
Compiler
collection
is
kind
of
the
original
one.
There's
tons
and
tons
of
tools
and
examples
using
bcc
and
essentially
what
you're
doing
is
it's
a
front
end
with
python
or
lua.
That
will
take
bp
bbpf
code
in
that
c,
like
format
and
then
use
python
to
help
provide
the
front
end,
and
then
you
will
deploy
your
scripts
or
your
programs
and
then
on.
A
A
So
it's
really
useful
and
really
quick
to
get
started,
but
you
do
have
some.
There
are
some
implications
for
the
host
that
you
want
to
run
on
lib
bpf,
which
is
second,
is
kind
of
a
more
modern
take
on
that
which
it
tries
to
abstract
away
the
the
actual
the
actual
functionality
that
needs
to
be
provided
by
a
front
by
a
library,
and
it
gives
you
kind
of
a
generic
library
that
you
can
use
in
order
to
make
all
the
system
calls
and
so
on
that
are
necessary
for
running
your
bpf
program.
A
There's
also
a
really
important
part
of
living
pf,
which
is
that
second
line
there,
which
talks
about
bbpf
portability
using
btf,
which
stands
for
bpf
type
format
and
co-re
or
compile,
runs,
run
everywhere.
Kind
of
the
goal
here
is
that,
with
the
the
existing
model,
when
you
are
using
ebpf,
if
you
compile
only
for
a
specific
version,
that
means
that
your
code
can
only
run
on
the
specific
version
of
the
kernel.
A
That
means
that
it's
hard
to
distribute
these
programs
to
be
able
to
run
on
more
than
one
host,
and
so
by
using
portability
and
codery
and
btf
and
so
on.
Lidbpf
enables
you
to
write
programs
that
can
be
used
on
multiple
versions
of
the
kernel
kind
of
really
opens
up
the
opportunity
for
distribution
of
those
programs.
A
We
also
call
it
bpf
trace,
which
is
a
kind
of
higher
level
scripting
type
language
that
will
allow
you
to
utilize
vpf,
but
not
actually
write
the
code
directly
so
kind
of
at
a
high
level.
What
it
looks
like
is
you
write
some
ebpf
code
in
c
in
a
c
like
format,
and
then
you
compile
that
using
clang,
and
you
end
up
with
an
elf
file
which,
if
you're,
if
you're
familiar
with
linux,
is
kind
of
the
standard
execution
binary
format.
A
You
get
an
l
file.
That
represents
your
ebpf
programs
in
that
edpf,
assembly
or
bytecode,
and
then,
when
you
actually
want
to
run
that
that
bp,
that
bbpf
program
contains
kind
of
all
the
data
that
you
need
to,
for
example,
create
maps
which
is
the
data
structure
that
edpf
uses
to
where
your
programs
can
write
data
to
which
can
then
be
retrieved
later
by
say,
your
user
space
component,
so
you'll
create
maps.
Your
loader
will
create
maps
based
off
the
definitions.
A
In
that
file,
you'll
also
actually
load
the
programs
itself
into
the
kernel,
and
then
you
have
to
attach
it
right.
So
in
this
example
on
the
left,
there
are
some
kernel
functions
that
we
may
want
to
attach
to.
In
this
case,
we
want
to
attach
the
tcp
v4
connect
function,
we'll
go
ahead
and
attach
our
program
to
that
kernel
function.
A
In
other
words,
this
is
a
k
probe
and
then
every
time
that
that
kernel
function
is
called
that
event
will
essentially
trigger
our
program,
which
will
get
the
more
or
less
the
data
that
the
connect
function
received.
So
we
can
do
some
inspection
or
whatever
there
update
our
map
and
then,
whatever
our
actual
program
needs
to
do
say.
We
just
want
to
output
some
data
output,
some
metrics.
We
can
read
from
the
map
that
contains
that
data
and
that's
now
available
in
user
space
by
the
load.
A
So
when
you
want
to
write
these
programs
at
least
kind
of
the
standard
way,
is
you
have
a
kernel
space
which
is
the
bbpf
code?
And
then
you
have
a
user
space
component.
This
is
kind
of
an
example.
We
won't
go
too
much
into,
but
this
is
kind
of
a
sample
of
what
the
code
could
look
like.
So
here
on,
the
left
is
the
bbpf
map,
and
this
is
really
again,
as
we
mentioned
a
little
bit
earlier.
This
is
really
a
data
structure.
A
That's
used
by
the
program
to
say,
load,
load
data
that
can
be
retrieved
by
user
space,
or
maybe
you
need
to
pass
it
to
another
bbpf
program,
but
either
way.
This
is
what
it
kind
of
looks
like
where
you're
defining
instructing
c,
and
then
there
are
some
special
bbpf
conventions
around
that
on.
The
right
is
the
actual
bbpf
code
that
will
in
this
case
this
is
what's
counting
tcp
v4
connect
function
calls
so
really.
A
This
is
essentially
taking
the
address
that
we're
seeing
in
the
connect
and
adding
that
to
our
map
so
that
we
can
track
those
connection
counts.
Essentially,
this
is
the
user
space
of
it.
We
don't
have
to
go
into.
We
won't
go
into
the
details
here,
but
really
what
what
this
is
trying
to
show
is
that
writing
the
bbpf
code
is
not
enough.
You
also
need
to
in
user
space
load
that
attach
it
run
it
and
then
clean
up
when
you're
done
and
on
the
right
side.
A
What
this
is
showing
is
that
you
need
to
be
able
to
take
that
data
out
of
the
map.
So
this
is
a
bunch
of
code
here,
but
really
what
it's
doing
is
writing
out
whatever.
So
in
this
case,
we
just
want
to.
We
just
want
to
output
some
data
of
the
counts
of
connections
and
on
this.
What
we're
doing
here
is
we're
printing
out
kind
of
a
nice
column
formatted
table
and
then
we're
actually
using
a
helper
method
to
actually
read
the
data
from
the
map.
A
B
So
I
don't
know
about
y'all,
but
that
was
a
lot
of
information
and
I
think
that's.
The
point
is
an
incredibly
dense
technology,
it's
very
powerful,
but
can
also
be
hard
to
jump
into,
and
so
that's
why
we
created
bumblebee
here
at
here.
At
solo,
bumblebee
helps
to
build,
run
and
distribute
ebpf
programs
using
oci
images.
B
It
allows
you
to
focus
on
writing
evpf
code
while
taking
care
of
the
user
space
components
automatically,
exposing
your
data
as
metrics
or
logs.
So
let's
break
that
sentence
down
really
quick,
so
bumblebee
helps
to
build,
run
and
distribute
ebpf
programs
using
oci
images.
So
what
is
an
oci
image?
An
oci
image
is
essentially
a
docker
image.
B
So
when
we
thought
about
building
bumblebee,
we
took
a
lot
of
inspiration
from
docker,
because
we
feel
that
the
user
experience
there
is
really
great.
Essentially
you
build
an
entire
application,
locally
push
it
to
a
registry
and
then
can
run
it
from
pretty
much
anywhere
else.
So
we
thought
why
can't
we
do
that
with
ebpf
and
it
turns
out
using
the
open
container
initiative
spec.
We
can
so
and
we'll
talk
about
later,
a
bit
more.
B
How
this
works,
but
essentially
we
allow
you
to
build
your
ebtf
programs
in
one
place,
store
them
in
a
registry
and
then
run
them
somewhere
else.
Part
two.
It
allows
you
to
focus
on
writing
evpf
code,
while
taking
care
of
the
user
space
components.
Again,
we
lawrence
went
into
more
detail
earlier
about
the
different
aspects
of
an
ebpf
program
and
we
thought
to
ourselves.
How
can
we
simplify
that?
B
B
Expose
that
data
as
either
metrics
or
logs,
so
we
also
allow
you
to
write
lid
bpf,
compatible
ppf
code
and
only
that
so
again,
no
need
to
focus
on
on
user
space
code
and
the
build
environment
is
completely
containerized.
So
you
don't
need
to
worry
about
running
in
a
specific
version
of
linux,
with
the
all
the
specific
depths
and
everything
the
llvm
tool
chain
can
be
complicated.
A
B
B
So
how
does
building
with
bumblebee
work?
Well,
as
mentioned,
we
create
a
pre-built
containerized
build
environment,
so
we
take.
You
know
llvm
clang,
that
whole
tool
chain,
as
well
as
the
bpf
headers,
and
all
the
information
that
we
that
is
needed
to
compile
your
program
into
bpf
code
and
that
is
stored
in
our
registry
at
github
registry.
Slash
solo.
B
So
how
does
build
work
well,
b?
Build
we,
we
modeled
it
as
again
after
docker
build,
so
we
thought
how
what's
the
simplest
experience
to
build
a
bpf
program
such
that
it
can
be
packaged
and
saved
to
a
registry.
Well
b,
here
comes
b
build,
so
we
pass
in
bpf.c.
So
that's
the
program
that
we're
trying
to
to
compile
and
then
the
in
then
in
the
tag
and
the
tag
just
like
a
docker
image
is,
is
where
this
will
be
stored.
So
we
build
the
bpf
code
into
an
elf
file.
B
So
that's
our
object
file,
that's
the
thing
that
we'll
actually
get
run
and
then
we
take
that
object,
file
and
package
it
up
into
an
oci
compliant
package
so
that
it
can
be
saved
to
a
registry
later,
and
we
can
do
that
with
the
push.
So
what
is
b
push
do
again
very
similar
to
docker
push.
You
can
take
your
your
built
image
and
send
it
to
any
oci
compliant
registry.
Certain
registries
will
only
accept
docker
images
up
top.
B
B
So
once
we've
pushed
it,
we
can
actually
go
ahead
and
run
it,
and
in
this
example,
it
shows
it
running
from
the
same
computer
that
we
have
built
it,
but
part
of
the
magic
here
is
that
it
can
be
from
anywhere
it
can
be
from
a
kubernetes
cluster
halfway
across
the
world.
The
only
limitation
is
that
it
needs
to
have
a
newer
version
of
the
linux
kernel
with
btf
enabled
and
we'll
go
into
more
detail
about
that
later.
B
B
Well,
essentially,
when
you
run
the
the
program
we'll
take
that
packaged
up
bpf
program
and
use
some
very
specific
metadata
from
it
to
load
that
bpf
program
into
the
kernel
create
some
maps
that
have
been
specified
and
then
take
data
out
of
those
maps
so
well,
once
the
vpf
program
is
running,
we
watch
those
desired
maps
and
output
that
that
data
as
either
logs
or
a
metric,
and
we
do
this
again
by
certain
metadata
provided
to
us
in
the
kernel
program
itself.
So
there's
no!
B
A
A
A
So
the
example
that
we've
been
using
this
point-
is
you
know,
tracking
connections
and
counting
connections,
so
this
map
will
be
used
to
count
each
connection
that
happens
from
the
kernel
to
a
given
address,
we'll
keep
track
of
that
and
count
it
and
we're
using
kind
of
a
struct
here
called
dimensions,
underscore
t
to
track
the
source
and
destination
address
and
then
on
the
right.
This
is
the
actual
bbpf
code.
That's
updating
the
map
based
off
the
data
that
we're
getting
so
again.
This
will
fire
more.
A
A
Another
interesting
thing
that
we
can
do
is
if
you
were
looking
closely
at
the
example
earlier
what
you
define
when
you
define
a
bbpf
map,
you
typically
are
going
to
use
this
section
functionality
to
determine
which
section
in
the
elf
file
your
your
code
gets
output
to,
and
so
the
dot
maps
is
kind
of
a
convention
that
says
it's
a
bbpf
map.
What
we've
done
is
added
on
to
that
and
we've
added
some
functionality
where
you
can
see
this.maps.counter
suffix.
A
That
also
allows
us
to
do
some
cool
stuff.
So,
in
this
case
we're
dynamically
determining
that
the
count
the
the
value
of
this
hash
map
is
a
counter
and
we
can
dynamically
track
the
value
here,
which
is
corresponding
to
the
value
field
in
your
in
your
hash
map,
and
we
can
automatically
output
that
and
update
it
based
off
of
just
by
you
adding
this.maps.counter,
and
so
it
really
allows
you
to
kind
of
write
your
bb
bbpf
code
and
then
basically
using
some
conventions.
Dynamically
get.
A
A
Speaking
of
metrics.
We
also
can
take
that
data
and
output
it
as
prometheus
metrics
by
default,
so
going
back
really
quickly,
since
this
is
a
counter,
we're
also
saying
that
we
want
to
not
only
output
it
when
we
run
it
here
on
the
screen,
but
we
also
want
to
treat
this
as
a
counter
in
prometheus
style
metrics.
A
B
So
without
further
ado,
let's
get
started
on
this
demo,
so
we've
been
talking
a
lot
about
bumblebee,
so
let's
actually
see
it
in
action.
So
the
first
thing
we're
going
to
do
is
we're
going
to
install
the
latest
version
of
bumblebee
and
to
do
that.
We're
going
to
use
the
install
script,
so
I'm
just
going
to
quickly
search
the
history
through
the
install
script.
We're
just
going
to
run
that
right
here
in
the
in
the
terminal.
B
B
So
I'll
do
that
now
and
I'll
do
that
in
both
of
my
windows
and
it's
important
to
know
that
when
running
bumblebee
there
are
a
couple
of
options
for
doing
this,
but
it
needs
to
be
run
with
sudo
and
the
reason
for
that
is
that,
in
order
to
increase
the
the
memory
limit
for
the
maps,
you
need
to
be
running
as
root.
So,
from
now
on,
when
we
run
these
commands,
we're
going
to
be
running
as
root
and
I'll
show
you
how
to
do
that
in
just
a
second.
B
So
what
I'm
going
to
go
ahead
and
do
is
very
first
thing:
I'm
going
to
use
init
now
what
is
init
well
init
is
going
to
initialize
a
sample
bpf
program
for
us
so
that
we
can
go
ahead
and
start
writing
it.
So
I'm
going
to
go
ahead
and
I'm
going
to
say
b
and
in
this
case
I'm
just
going
to
use
b,
because
I
don't
need
sudo
b
init
and
then
it's
going
to
run
me
through
a
bunch
of
interactive
steps
and
starting
my
own
new
ebpf
program.
B
B
And
the
last
thing
we
have
to
pick
is:
what
kind
of
output
do
we
want?
So,
as
we've
been
discussing
right
by
the
the
main
purpose
of
bumblebee
is
to
run
applications
which
surface
metrics,
and
by
by
doing
that,
we
allow
we
abstract
again
the
user
space
code
away,
so
we're
just
going
to
quickly
decide
what
kind
of
metrics
we
want
to
expose
from
this
program.
In
this
case
we're
going
to
pick
counter
now.
I
just
have
to
pick
a
file
location,
so
I'm
going
to
do.
B
Solocon.C
boom
successfully
wrote
our
skeleton
bpf
program.
So
let's
go
ahead
and
take
a
look
at
what
that
is,
so
I'm
going
to
open
up
silicon.c
and
when
I
open
up
silicon.c,
I
see
a
lot
of
of
information.
So,
let's
unpack
all
this
really
quick.
First
of
all,
we've
got
our
license
very
important.
Every
ebpf
program
needs
a
license.
B
B
So,
as
you
add
more,
you
add
them
to
this
struct
here
then
we
have
our
map,
for
which
actually
contains
the
the
data.
So
this
is
one
of
the
neat
things
that
bumblebee
has
has
added.
Is
this
you
see
here
dot
maps
dot
counter
so
typically
in
an
abpf
program,
when
you
specify
something
is
a
map
you
put
dot
maps
so
in
bumble,
bee
we've
added
another
thing:
you
another
layer
on
that:
that's
that's
the
type
of
output
that
you
want
from
that
map,
so
it
could
be
dot
counter
dot,
gage,
dot,
log
right.
B
These
are
the
different
types
of
outputs
and
the
next
thing
is
our
is
our
k
probe
itself
and
in
this
case
it's
going
to
be
tcp
v4.
So
those
are
ipv4
connections
and
again
here
we
just
have
some
boilerplate
code
to
start
you
to
start
off
on
the
on
the
right
foot,
so
we
create
a
key
for
our
dimension
or
our
statistic,
and
then
we
initialize
all
the
all
the
values
that
we
need
to
actually
emit
a
stat
for
each
connection.
B
B
Similarly,
we
have
our
our
dimensions
here
now.
In
this
case,
our
dimensions
are
source
address
and
destination
address,
to
show
off,
essentially
the
ability
to
capture
all
traffic
from
one
address
to
another
and
that
could
be
filtered
down
to
a
single
address
or
just
to
capture
across
all
addresses
and
see
where
traffic
is
flowing
in
our
network
and
in
a
minute
we're
going
to
get
to
show
that
off.
Here.
B
So
now
that
we've
taken
a
look
at
the
program,
we
can
actually
go
ahead
and
build
it.
So
what
does
it
mean
to
actually
build
this
program?
Well
to
do
that?
Let's
go
back
to
the
terminal
over
here.
The
first
thing
we're
going
to
do
is
we're
going
to
go
and
check
out
our
command
list
again.
So
when
you
want
to
run
one
of
these
programs,
the
first
thing
you're
you
need
to
do
is
build
it.
B
So,
let's
build
it
so
we're
going
to
run
sudo
b,
build
and
then
we're
going
to
see
what
build
wants
us
to
pass.
So
build
wants
two
args
and
the
the
args
that
it
needs
are
going
to
be
the
name
of
the
file,
so
the
program
that
you're
building
to
and
then
the
tag
so
exactly
like
docker,
so
I'm
going
to
pass
in
b
build
and
then
I'm
going
to
pass
the
name
of
the
the
name
of
the
program
that
I
want
to
build.
B
So
in
this
case
it's
going
to
be
tcp,
connect,
dot
c
and
then
I'm
going
to
pass
the
image-
and
in
this
case
it's
going
to
be
my
image
colon
latest,
and
this
can
be
anything
again.
This
can
be
a
docker
image
right.
This
can
be
pushed
to
any
registry
for
the
sake
of
this
demo.
I'm
just
gonna
push
it
to
a
a
local
registry,
so
I'm
gonna
call
it.
My
image
latest
apologies
for
the
brief
technical
issue.
B
And,
as
we
can
see,
we
have
successfully
created
our
new
image.
So
what
does
this
actually
do?
Well,
the
first
thing
it
does:
is
it's
going
to
write
our
image
to
tcp
connect.o
so
that
that's
our
object
file
that
we're
that
we're
actually
writing
to,
and
you
can
in
inspect
that
if
you
want
to
see
more
details
about
the
program,
the
other
thing
it's
going
to
do
is
create
our
our
latest
image.
So
it's
going
to
take
our
ebpf
program,
put
it
in
an
oci
image.
B
So
now
that
we've
done
that
we
can
actually
go
ahead
and
list
all
of
our
images.
We'll
see
that
look
at
that
we've
got
our
image
is
listed,
so
we
can
see
that
we
have
built
an
image
for
linux
on
this.
Is
our
operating
system
version,
so
it's
our
it's
our
kernel
version
5.10
and
then
our
architecture,
we
support
arm
and
yeah.
So
those
are
the
different
options
that
you
would
see
there.
B
So
now
that
I
I
can
see
our
image,
let's
go
ahead
and
run
it
so
in
order
to
run
it
now,
all
we
have
to
do
is
do
b,
run
and
pass
in
our
image.
That's
it,
and
now
we're
greeted
with
this
really
cool
terminal.
Ui.
Thank
you
to
lawrence
for
doing
a
ton
of
work
on
this.
So
now,
we've
got
our
terminal
ui.
B
And
we
see
that
it
shows
up
right
there,
so
so
what's
actually
happening
so
when
when
we
make
our
curl
hbbin.org,
our
program
is
recording
our.
So
the
program
that
we
ran
using
b
is
recording
the
source,
address
and
destination
address
to
a
metric
to
a
hash
map
which
then
bumblebee
is
picking
up
and
putting
in
a
metric
or
right
now
it's
printing
it
to
the
screen.
B
B
So
I'm
I'm
gonna
run
a
local
prometheus
server
here,
I'm
gonna
jump
and
then
I'm
gonna
curl
hb
bin
a
few
more
times
just
to
you
know
just
to
get
some
data
because
we
really
want
to
see
what
this
looks
like.
So
I'm
just
going
to
do
that
a
few
times
alrighty
and
then
I'm
going
to
jump
over
to
our
browser
and
I'm
going
to
reload
with
prometheus
and,
as
you
can
see
here,
boom
I've
already
got
it.
B
I've
got
our
metrics
they're
flowing
through
all
the
way
to
prometheus
from
bumblebee
and,
as
you
can
see
here,
I
can
actually
track
on
individual
destination
addresses
and
see
all
using
bumblebee.
So
that's
the
power
of
the
tool
that
we
created.
We
hope
you
enjoyed
the
demo
as
mentioned.
Bumblebee
is
a
totally
open
source
project.
Go
check
it
out,
we'd
love
to
hear
your
feedback.
We
love
prs.