►
From YouTube: Porting eBPF applications to BumbleBee
Description
In this livestream, Krisztian will join Lin to discuss why you may want to port your eBPF applications to BumbleBee and how you can take a BPF CO-RE libbpf script, and porting it to Bumblebee to solve the user-space, distribution, and integration challenges.
#ebpf #bumblebee #observability
00:06 welcome + speakers intro
1:14 hoot updates, service mesh, Cilium, eBPF news
4:25 general discussion around BumbleBee, what is NEW and its roadmap
11:15 oomkills and experience of porting oomkills to bumblebee
18:10 cloud flare exporter vs bumblebee
20:05 live DEMO from Krisztian about porting oomkill and run it on bumblebee
42:30 live DEMO from prometheus about oomkill events
51:15 wrap up
A
A
B
A
Awesome
very
nice
to
have
you
and
even
nice
this
time
that
you
work
for
solo.
Last
time
you
were
back,
you
were
on
the
hoot.
Actually,
it's
the
very
first
food
that
I
run.
I
host
is
episode.
20
one
click
istio
install
using
help,
and
I
believe
you
joined
solo
somewhere
in
april,
so
we're
so
excited
to
have
you
back
on
the
first
before
we
get
to
the
episode.
Let's
talk
about
news
quickly,
so
the
first
thing
I
want
to
share
last
a
week
the
hood
episode
28.
A
The
steps
are
published
now.
So,
if
you
are
interested
in
the
25
plus
minutes
demo,
that
facili-
and
I
shared
please
check
out
our
steps
here-
about
workflow
selector,
credential
name
in
destination-
rule
auto
sni,
telemetry,
minimum
tls
telemetry
new
additions
to
the
telemetry
api
so
definitely
check
that
out
and
follow
along.
A
The
next
news
I
want
to
share
is,
if
I
can
get,
the
the
thing
out
is
sd
time
release
their
100
best
in
show
in
software
development
and
solo.
The
io
made
to
the
list
of
the
category
of
api
and
integration
for
2022
so
very
exciting,
along
with
many
other
companies
like
typical
kong,
you
know
very
exciting
to
be
on
the
list.
A
Willie
morgan
has
published
a
very
interesting
article
about
his
thoughts
on
ebpf
psycho
and
the
future
of
service
mesh.
If
you
haven't
checked
it
out,
I
definitely
encourage
you
to
check
out
I'm
actually
a
big
fan
of
william
when
he
writes
a
technical
sub
leadership
blocks,
because
he
has
that
skill
of
explaining
things
really
really
understandable
for
folks
reading
his
mutual
ties.
Kubernetes
guide,
you
know
it's
just
a
great
blog,
so
it's
really
cool
that
in
this
blog
he
talked
about
ebpf.
A
You
know
how
you
should
understand
the
ebpf
and
think
about
in
the
context
of
multi-tenancy.
You
know
how
that's
becoming
a
part
of
the
problem
for
evpf.
What
are
the
areas
ebpf
is
limited
and
how
ebpf
will
intersect
with
service
mesh?
So
please
check
it
out.
I
think
it's
a
really
good
blog
with
that.
Let's
get
to
the
episodes
of
today,
protein
ebpf
applications
to
bumblebee
so
christian
for
folks
who
are
not
familiar
with
bumblebee.
B
Yes,
basically
bumblebee
is
it's
a
project
by
so
that
io
it's
it's
a
way
to
distribute
your
ebpf
modules
scripts.
Basically,
you
can
think
of
it
as
a
docker
for
ebpf,
basically,
because
it
solves
the
distributional
challenges
and
also
the
integration
integration
challenges
of
distributing
your
little
ebpf
scripts.
A
Yeah
definitely
I
launched
the
bumblebee
website
on
the
side,
so
people
can
take
a
look
too
so
bumblebee
really
focus
on.
You
know,
building,
publish
and
wrong
experience
to
help
you
get
started
with
ebpf.
In
fact,
it's
one
way
for
me
personally
to
learn
evpf
as
well,
so
you
know
we're
very
excited
about
bumblebee.
A
B
If,
if
a
project
has
many
examples,
people
are
people
can
start
exploring
these
in
their
own
environment
and
and
play
play
with
these
clips.
So
I
take
a
look
at
the
extra
slips
that
was
already
there
and
started
my
ebpf
journey
with
bumblebee
as
well
and
ported.
I
think
two
scripts
to
the
repository,
which
was
a
great
way
to
wait
for
me
to
get
get
familiar
with
the
core
concepts
of
ebpf.
A
A
B
B
This
can
be.
This
can
cover
distribution,
and
this
can
cover
integration
with
other
parts
of
the
infrastructure
as
well.
We
can
think
about
observability,
for
example,
because
there
are
some
interesting
challenges
that
that
can
be
solved
with
ebpf.
Some
of
this
cannot
really
be
solved
otherwise
or
or
cannot
really
be
solved.
The
way
that's
efficient,
so
integration
with
these
applications
and
technologies,
I
think,
is
pretty
important.
B
Bumblebee
can
integrate
with
prometheus,
for
example.
Currently
bumblebee
only
supports
the
gauge
and
the
counter
matrix
type,
but,
for
example,
on
the
roadmap.
We
have
the
support
for
the
histogram
type,
which
would
be
a
great
addition,
because
there
are
some
already
proven
applications,
abpf
applications
that
are
relying
on
the
data
by
data
metric
type,
and
once
we
have
it,
we
are
ready
to
visualize
all
these
new
kind
of
observability
challenges
with
the
help
of
bumblebee.
B
And
apart
from
this,
we
also
have
multiple
other
items.
For
example,
we
are
working
on
adding
new
languages,
for
example,
rust
and
go
and
there's
another
request
from
the
community,
for
example,
to
have
support
for
user
space
probes
as
well.
That's
that's
an
issue.
That's
that's!
A
frequent
ask
for
many
applications
working
in
the
ebbs
space.
Currently
then
there's
the
usual
multi-urge
beards
and
some
quality
of
life
improvements
on
the
b
cli.
We
will
explore
the
cli
at
the
demo
in
this
session
on
this
session.
A
A
For
example,
a
few
a
couple
of
us
were
traveling
to
venezia
for
evpf
day
as
part
of
kubecon
europe,
and
I
I
was
presenting
about
evpf
and
bumblebee
to
ebpf
day
one
of
the
user
in
the
audience
the
attendee
in
the
audience
asked.
Do
we
support
cosign
to
bumblebee
right
for
bumblebee
images
and
etan,
who
is
the
maintainer
on
the
bumblebee
actually
just
implemented
added
that
feature
to
the
bumblebee
and
even
provided
with
me
on
how
to
use
that
right.
So
our
roadmap
is
depends
on
you
to
help
us
shaping
it
too.
A
A
B
The
tool
itself
is
about
catching
wound
kills.
You
are
most
probably
familiar
with
with
the
unkiller
in
in
linux
system
and
even
in
kubernetes,
and
if
you
are
trying,
if
you
have
experience
trying
to
catch
these
room
kills
for
your
applications,
memory
leaks
in
production
systems.
It
might
not
be
that
easy.
For
example,
from
my
time
as
an
sre
devops
engineer,
I
was
having
a
harsh
time
height.
B
A
A
B
As
I
mentioned,
I
used
a
script
that
was,
that
was
already
written,
handy
and
it's
available
in
an
upstream
github
repo.
But
if
you
want
to
take
that
script
into
production,
you
will
have
to
work
around
many
rough
edges.
You
have
to
distribute
that
image.
You
have
to
it's.
It's
it's
not
real
to
get
this
into
an
image,
and
that
might
be
a
requirement
for
former
security
team
to
to
have
information
and
and,
for
example,
to
verify
the
origin
of
your
of
your
script
security
device.
B
Also,
the
integration
challenges
are
quite
rough
currently.
So,
for
example,
if
you
want
to
let's
say
you
are
using
the
usual
promotion
stack
on
top
of
kubernetes,
and
you
want
to
have
an
alert
for
your
room
event.
Then
your
pure
existing
upstream
unkill
script
will
not
get
you
there
by
default
out
of
box.
You
have
to
create
promise
metrics
from
that.
A
A
B
B
So
your
so
your
code
that
you
found
on
the
internet
might
not
work
or
might
not
be,
might
not
have
the
best
performance,
and
these
are.
These
are
very
important
aspects
to
pay
attention,
so
I
had
to
jump
around
a
lot
check,
check,
documentations
and
existing
blog
posts
and
the
and
the
debugging
experience
was
was
not
that
not
that
great
as
well.
B
So
there
were
some
glitches,
but
the
experience
is
getting
better
and
better
and
soon
we
will
publish
a
blog
on
on
this
actual
porting
exercise,
which
will
go
into
much
more
details
about
the
technical
choices
and
the
technical
limitation
of
each
data
type
and
and
all
these
technical
things.
So
I
recommend
checking
that
post
out
when
it's
when
it's
out.
A
C
A
B
B
A
B
B
So
if
you
are
loading
evps
scripts
that
are
relying
on
that
data
structure
that
can
be
handled
already
but
yeah,
I
think
the
the
core
focus
of
the
two
projects
are
a
bit
different,
because,
while
the
cloudflare
exporter
is
basically
dbpf,
exporter
is
basically
just
an
exporter
focusing
on
exporting
ebpf
metrics
in
a
primitives
format.
The
wb
project
is
more
like
about
providing
a
runtime
for
your
scripts
and
about
distributing
these
modules
in
a
in
a
very
efficient
and
convenient
way.
A
B
B
B
B
B
B
A
A
B
B
B
So,
yes,
I
got
rid
of
the
this
header,
because
this
header
only
included
the
this
truck
that
defined
the
structure
of
our
data.
We
are
basically
interested
in
the
process,
that's
getting
room
killed
and
the
process.
That's
that's
currently
running
at
the
time
of
the
unkill,
so
we
will
you
we
will
need
the
the
name
of
these
two
commands.
B
C
B
Difference
is
having
a
different
map.
There
are
various
reasons
for
this.
The
first
and
the
detriment,
reason
is
that
bumblebee
currently
only
supports
two
map
types.
These
are
hashmap
and
ring
buff
ring
buffer.
As
you
can
see,
the
original
code
is
using
a
different
type.
That's
the
let's
say:
legacy
map
type,
it's
called
perf
buffer,
so
we
are
migrating
to
ring
buff
from
perth
buff,
because
bumblebee
only
can
handle
ring
buffer.
B
There
are
some.
There
are
some
trivial
changes
because
of
this
you
don't
need
to
specify
the
key
size.
You
just
need
to
define
the
max
entries
that
can
that
can
be
used
in
this
map.
You
can
think
of
this
map
if
you
are
not
familiar
with
the
with
this
bppf
concept
as
a
queue
as
a
high
performance.
Cue
that
you
can
write
into
from
the
kernel
space
and
access
all
these
data,
all
these
events
or
these
fields
at
the
at
the
application
side.
That's
basically
the
the
structure
structures
goal.
B
Yes
for
this
type,
we
are
using.
The
data
underscore
t
that
we
referenced
earlier
and
we
are
using
a
different
name.
The
original
code
called
it
events
and
we
are
calling
it
wound-
kills
that's
just
a
minor
detail,
but
it
will
be
important
when
we
are
taking
a
look
at
the
promise
integration,
because
we
are
exposing
all
this
data
as
a
primitives
metric.
B
B
B
These
are.
These
are
room,
kill
events.
You
are
free
to
choose
any
name,
but
I
think
using
the
proper
name
can
make
make
your
life
easier
in
the
long
run.
Yeah,
and
I
go
into
more
details
on
the
blog
about
this
primitives
integration
logic.
But
basically
the
nice
thing
with
bumblebee
is
that
it's
taking
care
of
all
these
metrics
explosions.
So,
basically
you,
you
can
just
add
the
dot
counter
suffix
to
your
map.
C
B
Basically,
that's
it
after
this
point.
If
the
image
is
built
and
deployed,
you
can
scrape
all
the
content
of
your
map
as
a
parameters
matrix.
We
will
see
that
at
the
at
the
end
of
the
demo,
and
after
this
point
we
have
the
core
function,
the
core
bpf
logic.
This
logic
will
be
triggered
each
time
this
kernel
probe
is
executed
in
your
system.
B
So
basically
your
your
kernel
has
this
k
probe
and
every
time
it's
triggered,
you
can
execute
a
little
script
and
the
content
of
the
script
will
populate
this
data
type,
and
these
events
with
this
data
type
will
go
into
your
little
queue.
I
think
that's
the
simplest
way
of
explaining
the
process
of
a
simple
ebbf
script.
A
B
C
A
B
Yes,
exactly
the
blog
is
going
more
details
about
each
of
these
instructions,
but
basically,
as
I
said,
we
are
interesting
in
we
are
interested
in
these
details.
The
process
that's
getting
room
killed,
that's
the
most
important
one
and
we
can
additionally
get
the
process
id
and
we
are.
We
can
also
get
the
process,
that's
being
run
at
the
time
of
the
room,
kill
which
might
be
which
might
help
to
pinpoint
which
process
caused
the
actual
wound
kill.
B
There
are
some
small
changes
at
the
beginning
at
the
end,
and
that's
because
our
migration
to
during
buffer,
because
ring
buffers
I
mentioned,
is
about
providing
an
more
a
more
performant
way
to
handle
these
events
in
the
ring
buffer.
You
can
basically
reserve
in
reserve
the
space
in
memory
for
your
events
and,
if
you
can,
if
you
could
preserve
it
and
populate
all
the
data
that
you
wanted,
you
can
submit
it
and
commit
it.
That's
a
simplified
api
on
top
of
the
existing
logic.
B
You
can
get
rid
of
insufficient
memory
copies
and
there
will
be
no
ordering
issues
if
you
are
running
on
multiple
cpus
and
the
events
are
arriving
at
a
therapeutic
pace
and,
as
you
can
see,
the
the
line
chain,
the
changes
are
are
not
that
complex.
Basically,
we
are
just
preserving
preserving
the
space
and
at
the
end,
submitting
submitting
this
and
and
that's
that's
about
it-.
B
Okay,
so
basically
that's
the
difference
between
the
two
codes,
as
I
mentioned
check
out
the
blog
for
further
details.
So
once
we
have
the
code,
let's
see
bumblebee
in
action
and
package
this
into
an
oci
image.
To
this
we
will
first
download
bumblebee
currently
beyond
dot
12
version,
and
it's
already
installed.
B
B
B
Yes,
exactly
if
we
try
to
run
this
image
locally,
we
will
see
if
there's
an
issue
with
the
code.
You
can.
You
can
test
it
locally,
for
example,
if
the
if
the
kernel
verifier
find
something,
that's
that
not
should
be
there
some
issue
with
with
your
code
regarding
memory,
access
or
performance,
you
will
be
notified
and
you
won't
be
able
to
load
it
into
your
kernel.
So
it's
a
pretty
safe
way
to
to
play
with
this.
I'm
only
using
this
lab
setup
because
catching
gunk
is
in
your
local
system
can
cause
issues.
C
B
B
If
we
would
have
any
issues
with
our
code,
we
wouldn't
see
the
interface
of
of
bumblebee.
We
would
be
able
to
get
an
error
of
of
some
sort.
So
if
we
see
this
actual
interface,
we
are
good
to
move
forward.
We
don't
see
any
events
here
because
we
don't
have
any
room
keys
in
our
system.
Fortunately,
but
we
will
simulate
this
after
a
couple
of
minutes.
B
B
So
we
can
use
this
quick
and
dirty
cube.
Ctr,
manifest
q
kubernetes
manifest
to
deploy
this
as
a
demo
set.
If
you
deploy
it
as
a
demonstrate,
you
will
have
bumble
bee
pots
on
your
across
all
of
your
notes.
So
if
you,
if
you
experience
any
memory
related
issue
in
any
of
those,
you
will
have
the
data.
B
As
you
can
see,
the
interesting
thing
here
is
that
we
have
a
single
container.
It's
called
bumblebee.
We
are
using
a
bumblebee
image
and
we
are
executing
the
b
command,
which
is
the
command
of
the
cli,
and
we
are
passing
a
few
arguments
like
run
and
we
are
referencing
the
image
that
we
just
pushed
to
our
local
repository
and
we
are
exposing
the
ports.
So
we
can
access
this
with
promedus.
B
C
B
C
C
B
Okay,
so
let's
deploy
promote
use,
we
are
using
the
q
promoter
stack
to
deploy
everything.
We
will
just
use
the
primitives
interface,
but
this
will
deploy
nearly
everything
we
are
putting
this
into
the
monitoring
namespace
that
we
are
creating.
Now
it
should
take
a
couple
of
seconds,
but
not
that
not
that
many
okay,
we
already
have
it.
B
C
B
B
B
C
B
A
B
B
C
B
And
if
you
execute
this
query,
you
will
see
that
all
these
data
that
we
wanted
to
extract
from
from
our
events
are
basically
there
and
we
have
an
one
as
a
value,
because
we
had
we
care
because
we
had
a
long
gear
for
these
events.
As
you
can
see,
that's
the
t.
Com
dt
cam
is
the
name
of
the
process.
That's
getting
cool
queued
and
the
f
com,
the
name
of
the
process
that
was
running
at
the
time
of
the
unkill.
B
One
thing
that
that
can
be
a
bit
misleading
is
that
we
are
currently
running
primitives
stack
on
top
of
kubernetes
and
if
you
are
running
that
you
most
probably
are
running
that
you
will
have
these
default.
Kubernetes
labels
container
endpoint
job
and
pod,
and
actually,
if
you
check
these,
you
will
see
that,
for
you
have
the
bumble
bee
board
seo
as
the
value
for
your
poor
label
for
this
unkill
matrix,
which
can
be
misleading
because
not
your
actual
bumblebee
pods
getting
room
killed.
That's
not
the
not
the
port,
that's
causing
the
room
here.
B
B
So
yes,
currently
with
this
quick
proof
of
concept,
you
can
just
have
the
actual
name
of
the
process,
that's
getting
greater
than
the
nd
process
id,
but
in
this
case
it's
called
stress.
But,
for
example,
if
you
are
running
let's
say,
node.js
application
and
you
see
node.js
sd
killed
process,
then
you
can
probably
easily
pinpoint
that
your
front
ends
are
having
trouble
or
that
might
not
be
the
best
example.
A
B
That's
another
thing
that
can
be
a
bit
misleading.
The
instance
is
the
actual
node.
So,
for
example,
we
are
running
this
as
a
demon
set,
so
you
would
have
all
the
all
the
all
the
nodes
here
as
the
instance
and
the
port
is
the
is
the
port.
That's
that
bumblebee
exposes
in
this
example.
Actually
you
can
only
see
one
of
the
instances
and
that's
because
the
port.
B
A
B
B
I'm
I'm
not
sure
the
original
upstream
script
was
written
to
focus
on
kubernetes
deployments,
so
this
data
can
make
more
sense
in
a,
for
example,
in
a
in
a
in
a
bare
metal
environment
where,
where
everything
is,
is
running
on
the
the
actual
host.
In
that
case,
it
can
provide
better
data,
but
you
can
fine-tune
db
ppf
the
ebpf
logic
to
get
rid
of
this
noise.
B
That's
the
that's
being
generated,
and-
and
that's
another
good
thing
here-
that
with
bumblebee
you
can
focus
on
iterating
your
kernel
space
code
and
the
application,
and
you
would
not
need
to
touch
the
the
user
land
bits
for
for
this
experimentation.
In
this
example,
that's
a
fine
example
that
you
have
some
data.
It's
working,
you
can
find
the
actual
process
that's
getting
killed.
B
A
Yeah,
it's
so
much
better
than
not
having
this
data
right.
It's
at
least
tell
you
what
process
and
the
name-
and
you
know
the
process
id
so
and
also
it
tells
you
the
the
ip
address
which
you
can
find
out
on
which
kubernetes
vm
right.
So
that's
helpful
too
yeah
all
right!
This
is
really
cool.
I
want
to
ask
you:
is
there
anything
else
you
want
to
show
or
add
you
mentioned
a
workshop.
B
A
That's
great
so
once
you
have
the
your
blog
published
and
the
ebpf
program,
which
I
I
think
is
an
intermediate
program
built
on
top
of
a
fundamental
for
ebpf
by
solo
io,
where
were
published
in
the
comments
of
the
youtube,
so
that
you
guys
can
have
access,
so
this
has
been
wonderful.
I
feel
I
learned
a
lot
more
about
ebpf
and
bumblebee
myself
today.
So
christian,
I
want
to
take
a
minute
to
thank
you
for
your
time
to
you
know,
tell
us
the
story
of
you.
Protein
almond
kill
application.