►
From YouTube: Bpftrace Meets Pixie: Dynamic Monitoring of K8s Clusters Unleashed - Omid Azizi, New Relic (Pixie)
Description
Bpftrace Meets Pixie: Dynamic Monitoring of K8s Clusters Unleashed - Omid Azizi, New Relic (Pixie)
Bpftrace is an essential tool for developers investigating the workings and performance of applications on Linux systems; Pixie is an eBPF-based observability platform for real-time troubleshooting of applications on Kubernetes. What if you could bring these two open-source projects together and combine the power of bpftrace with Pixie's approach to monitoring Kubernetes? This session presents Pixie's bpftrace integration and how it enables dynamic monitoring of Kubernetes clusters. This talk will show how Pixie can deploy a bpftrace program across all the nodes of your cluster, and make the collected data available for querying and visualization. Topics include (1) an overview of how the Pixie bpftrace integration works, (2) how to import existing bpftrace scripts (or write new ones) into Pixie, and (3) how to use the Pixie's query language to perform real-time debugging of Kubernetes applications. The talk will include a number of live demonstrations, including how bpftrace + Pixie can identify TCP issues, and even how to discover patterns of unauthorized bitcoin mining in your K8s cluster.
A
So
I'm
happy
to
be
here
today
to
talk
about
bpf,
trace
and
pixie.
Bpf,
trace
and
pixie
are
both
open
source
projects.
We
love
them,
both
I'm
from
pixie,
so
my
omid,
my
colleague,
yeshuang,
is
sitting
over
there.
We
worked
on
this
stuff
together,
and
so
we
wanted
to
present
today
how
we
kind
of
brought
these
two
open
source
projects
together
to
make
them
better
together.
A
So
just
starting
off
real-time
debugging
of
kubernetes
is
is
challenging.
We
all
know
that
right,
debugging,
we
get
frustrated
when
things
go
wrong
and
we
need
to
have
the
right
tools
available
to
debug
things
on
the
fly
and
so
for
us
we're
always
looking
at
how
we
can
make
observability
easier
right.
How
can
we
do
that
and
ebpf
has
really
been
a
game
changer
in
that
space?
A
You
know
you're,
bringing
the
ability
to
essentially
monitor
parts
of
your
application
without
requiring
any
instrumentation
without
requiring
any
redeployments,
no
code,
modifications,
it's
been
fantastic
and
we
we
really
really
love
that,
and
I
mean
the
proof,
is
in
the
pudding:
we're
really
seeing
a
lot
of
different
ebpf
based
observability
projects
take
off
and
it's
fantastic
to
see
like
we're,
really
thrilled
about
all
the
different
tools
that
are
like
out
there
in
the
ecosystem.
Now
it's
just
great
to
have
all
the
support.
A
So,
as
I
said
this,
so
I'm
going
to
talk
about
in
this
talk
two
particular
tools:
one
is
the
open
source
bpf
trace
project,
the
other
one
was
pixie
and
then
how
we
bring
them
together
to
make
something
even
better
right.
So
first
we're
going
to
start
with
bpf
trace.
So
let's
meet
bpf
trace
in
case
you
haven't
met
it
before.
I
assume
a
lot
of
you
have
because
it's
a
fairly
popular
project
and
I'm
going
to
keep
this
brief,
but
bpf
trace
it's
a
high
level
tracing
language
for
ebpf.
A
It
makes
it
really
easy
to
write
ebpf
programs
right.
So
as
a
very
simple
example,
if
you
wanted
to
trace
every
open
syscall
in
your
in
your
system
on
your
on
your
host,
it's
essentially
a
one
liner.
I've
broken
it
down
to
three,
but
this
is
from
bpf
trace's
own
docs
right.
So
you
just
kind
of
specify.
You
say
you
know
the
trace
point:
syscall
sysenteropen!
A
You
say
I
want
to
trace
the
open
calls,
and
then
you
say
what
you
want
to
do
when
whenever
that
event
occurs
in
the
system,
and
so
we're
going
to
say
here,
printf
and
then
we're
going
to
print
out
the
command
that
triggered
that
syscall
and
we're
going
to
also
capture
the
argument
to
get
the
the
file
name
of
what
file
was
being
opened
right.
And
if
you
run
this
on
your
on
your
terminal,
right,
you'll,
just
it'll,
start
printing
out
every
single
syscall
open
that
you
see
right
so
really
powerful.
A
There's
great
community
support
for
bpf
trace.
So
if
you
go
to
bpf
traces,
repo
you'll
see
that
they
have
a
bunch
of
built-in
tools
already
like
these
are
bpf
tray
scripts
that
have
already
been
written
for
you,
they're
at
your
fingertips.
We
counted.
There
was
35,
there's
all
sorts
of
stuff
from
what's
happening
at
the
I
o
level
to
tcp
to
like
kills
like
tracing
all
the
kills
that
happen
in
your
system.
A
So
anything
you
can
kind
of
think
of
there's
kind
of
there's
there's
something
out
there
to
do
to
kind
of
monitor
that
with
bpf
trace,
and
so
it's
again
really
powerful
stuff.
The
other
thing
about
bpf
trace
is
it's
really
useful
because
you
can
write
your
own
right.
It
was
so
easy
to
write
it's
a
high
level.
Like
writing.
We
do
a
lot
of
writing
bpf
like
low-level
bpf
code.
It's
it's
hard
right,
but
when
you
write
with
bpf
trace,
it's
so
much
easier,
so
you
can
get
so
much
more
done
quickly.
A
A
What
you
get
out
of
the
box
with
pixi.
Is
you
get
kind
of
basic
stats
like
cpu
networking,
io
metrics,
all
those
sort
of
table
stake
stuff,
but
in
addition,
using
ebpf,
we
trace
all
the
messages
going
through
your
system.
So
whenever
one
pod
talks
to
another
pod
sends,
for
example,
an
http
message,
we'll
trace
that
and
we'll
pull
it
into
pixie
so
that
you
can
kind
of
query
against
all
the
messages
that
were
in
your
system
and
figure
out
what's
going
on.
A
Our
kind
of
primary
characteristics
are
we're.
We
really
want
to
keep
things
zero
instrumentation,
that's
where
the
ebpf
part
comes
in
also
pixi
has
a
distributed
architecture.
So
we'll
take
a
look
at
that
in
a
second
and
there's
essentially
in-memory
data
tables
that
store
all
the
trace
data
and
then
finally,
there's
a
scriptable
interface.
That's
based
on
python
pandas
to
let
you
query
all
the
data
that's
being
collected,
so
you
can
kind
of
real
time
debug.
What's
going
on
with
your
system.
A
From
an
architecture
perspective,
the
way
that
pixi
works
is
that
there's
a
module
called
the
pixi
edge
module
and
that
gets
deployed.
There's
one
copy
of
that
per
node
in
your
kubernetes
cluster
and
the
job
of
the
pixi
edge
module
is
foremost
to
monitor
all
the
stuff.
That's
happening
in
the
node,
so
it's
monitoring
all
the
pods
that
are
in
the
node,
if
they're
again
sending
messages
to
each
other,
it's
going
to
trace
them.
A
A
Drilling
a
little
bit
deeper
into
that
sterling
component.
The
data
collector
the
architecture
internally,
looks
as
follows.
So
we
have
sterling
internally.
We
have
these
different
components,
these
different
extensible
components
that
are
delivering
the
different
sorts
of
data
that
you're
looking
for.
So
we
have
one
that's
collecting
kind
of
your
basic
process:
stats,
cpu
memory,
networking
I
o
all
the
basic
stuff.
Then
we
have
the
protocol
tracer,
which
is
collecting
the
traces
that's
ebpf
based.
A
A
And
then
we
also
said
out
of
the
box:
you'll
gonna
get
these
profiles
right
and
this
you're
looking
for
wide
bars
here.
If
you
see
a
wide
bar,
that
means
you're
spending
your
applications
spending
a
lot
of
time
in
one
particular
function,
and
that
might
be
an
area
of
concern
or
somewhere
to
optimize.
A
So
we
want
to
leverage
that
part
and
there's
already
a
kind
of
large
collection
of
community
scripts
that
we
would
be
able
to
access
if
we
were
able
to
do
that,
so
it
would
be
really
cool
to
see
that,
on
the
other
hand,
pixi
provides
kind
of
the
easy
deployment
to
kubernetes.
It's
managing
all
of
that,
and
it
has
all
the
data
collection
from
the
entire
cluster,
all
the
nodes
in
your
cluster
and
it's
putting
all
that
data
into
structured
tables
so
that
you
can
query
it
right.
A
So
the
idea
was
that
we
would
want
to
modify
a
slightly
modified
bpf
trace
so
that,
instead
of
outputting
events
to
the
screen,
which
is
what
it
normally
does
when
you
write
a
printf
in
bpf,
trace,
it'll
just
print
it
out
to
standard
out.
We
want
those
events
to
kind
of
get
sent
to
tables
within
pxe
so
that
we
can
query
them,
and
so
we
focused
on
those
bpf
trace
scripts
that
exist
today
that
primarily
output
things
based
on
printf,
and
we
said:
okay,
we'll
take
pixie
sterling.
A
You
know
the
user
somehow
sends
in
a
bpf
trace
script,
we'll
get
that
we
will
send
it
to
bpf,
trace
to
compile
it
and
then
deploy
it
to
the
kernel
and
then,
as
the
events
are
coming
back
from
the
kernel,
they
go
back
to
bpf,
trace
and
we'll
kind
of
intercept
the
printf
callbacks
right
so
that
they
never
go
to
standard
out.
They
come
back
to
pixie
and
we're
not
really
just
parsing
kind
of
like
a
string.
A
So
conceptually
like
I
mentioned,
we
want
to
capture
the
data,
that's
coming
out
from
these
printfs
in
bpf
trace.
So
let's
look
at
an
example
right,
so
we
have
a
printf
here,
it's
a
simple
one
that
says
like
time
and
count
and
command.
So
there's
three
arguments
to
this
printf,
and
so,
when
you
have
this
sort
of
statement
in
in
a
bpf
trace
script
that
pixie
is
taking
in
what
we're
going
to
infer
from
this
is
that
there
should
be
three
columns.
There
were
three
arguments
to
this
printf.
A
So
this
we
should
create
a
table
with
three
columns
and
then,
if
you
put
the
optional
kind
of
labels
in
front
of
the
format,
specifiers,
then
we'll
automatically
also
label
the
columns
for
you.
So
we'll
say
the
first
column
was
the
time
the
second
column
was
account
and
the
third
column
was
a
was
a
command
and
every
time
the
printf
triggers
this.
These
events
will
get
sent
to
pixi
and
will
populate
the
will
populate
the
table.
That
was
automatically
created.
A
So
what
does
that
mean
in
practice?
So
here's
a
simplified
version
of
an
existing
community
script.
This
is
a
script
that
collects
or
traces
tcp
drops
in
your
system.
It's
a
little
bit
simplified.
So
you
can
you
can
take
a
look
at
the
the
real
one
at
the
link
below.
But
what
happens
in
the
original
script?
Is
it
actually
has
essentially
three
printfs?
It
prints
the
time
and
then
it
has
two
printfs,
but
we
want
to
put
this
in
a
structured
format
so
to
get
this
into
pixie.
A
For
those
folks
who
are
familiar
with
bpf
trace
and
are
thinking
to
yourselves,
what
about
those
aggregation
style
scripts,
so
bpf
trace,
has
scripts
where,
when
you
run
them,
they
just
silently
are
running
they're,
not
outputting.
Anything,
and
then
you
hit
control
c
and
it's
been
aggregating,
a
whole
bunch
of
data
and
dumping,
and
it
dumps
it
out
at
the
end,
so
that
doesn't
really
naturally
fit
kind
of
with
the
time
series
based
approach
that
pixi
takes,
and
so
you'll
need
to
do
a
little
bit
more
kind
of
working
with
that
script.
A
Okay,
now,
let's
put
an
example
together:
pixie
has
this
notion
of
a
pixel
script.
It
is
what
drives
the
kind
of
ui
within
pxe
and
it's
where
we
can
do
our
querying
within
pxe,
and
so,
if
we
want
to
get
a
bpf
script
into
pixy,
we
have
to
put
it
first
into
a
pixel
script.
So
there's
really
three
components
to
this.
To
this
example.
Here
we
have
a
trace
point.
We
want
to
trace
all
the
exec
calls,
so
we
see
it
talk
at
the
top.
A
The
bpf
trace
script
that
we
want
to
use
and
that's
kind
of
embedded
in
our
pixel
script
right.
It's
just
put
there
into
a
string
and
we're
saying
this
is
what
we
want
this
we
could
have.
If
this
was
a
community
script,
it's
essentially
what
we
copy
over
from
the
community,
and
then
we
have
the
second
section,
which
is
telling
pixi
how
to
deploy
this
bpf
trace
script.
So
we're
saying
upsert,
tracepoint
upsert
is
terminology
using
databases
which
says
insert
this
trace
point.
If
it
doesn't
already
exist,
if
it
does
already
exist,
update
it.
A
So
it's
a
combination
of
update
and
insert-
and
we
say,
give
the
give
this
this
probe
a
name.
What
we're
about
to
deploy
give
this
this
entity
a
name,
specify
the
output
table,
name
where
the
results
will
be
put
out
into,
and
the
third
argument
is
the
bpf
trace
program
that
powers
it
all
and
then
the
final
argument
is
a
duration.
We
say
once
you
deploy
this
leave
it
there
for,
in
this
case,
we're
saying
five
minutes
and
just
keep
collecting
data
for
the
next
five
minutes.
A
A
That
second
section
is
what
actually
deploys
your
embedded
bpf
trace
script
into
pixi
and
then
the
last
part
is
kind
of
the
query,
part
so
we'll
say
df
data
frame
that
stands
for
data
frame,
so
we
get
to
say,
get
a
data
frame
from
the
table
and
we're
going
to
specify
the
bpf
trace
table
that
we
just
created
where
the
results
are
going
out
to
and
then
we're
going
to
say.
For
example,
this
is
an
exec
tracer,
so
it's
searching
for
all
the
exact
calls.
A
So
maybe,
in
this
case
we're
saying
hey,
look
for
anything
that
has
sudo
in
it.
Maybe
we're
trying
to
monitor
dynamically
for
anything
with
sudo
any
pseudo
command,
so
we'll
we'll
catch
those
and
then
the
last
line
says
display
the
results
onto
the
screen
all
right,
so
you
could
change
it.
You
could
filter.
However,
you
want
that
part
at
the
bottom
is
kind
of
dynamically
running.
When
you
run
your
query,
you
could
look
for
any
sorts
of
commands.
Any
sorts
of
exec
calls
something
you're
interested
in
and
that's
the
part
of
pixi's
dynamic
ability
right.
A
A
Right,
so
if
you
come
to
pixi,
this
is
the
kind
of
main
page
that
you
arrive
at
right.
So
right
at
the
top,
you
get
a
service
map.
This
is
this
is
not
the
bp
of
trace
fed
information.
This
is
just
kind
of
what
you
see
out
at
the
outset.
You
see
who's
communicating
with
who
the
traffic
you
can
scroll
down
and
look
at
drill
down
into
different
name
spaces.
You
can
look
at
the
different
nodes.
A
A
Pixi
also
has
a
notion
of
scripts,
so
that's
kind
of
what
I
showed
on
the
previous
slide
at
the
top
left
here.
You
can
choose
different
scripts
for
for
different
things
that
you
want
to
see.
For
example,
if
you
do
px
http
data,
it
changes
the
view
now.
This
is
a.
This
is
just
kind
of
raw
data
of
showing
all
the
http
traffic
in
your
in
your
cluster,
but
we're
not
interested
in
that
one.
A
A
So,
if
I
hit
run
what's
happening
right
now,
is
it's
actually
compiling
the
bpf
trace
program
dynamically
and
deploying
it
across
all
the
nodes
in
the
cluster?
So
that's
happening
actively.
It
takes
a
few
seconds
and
then
you're
live
right
and
we
can
see
now
across
the
entire
cluster,
all
the
nodes
of
the
cluster
and
see
what
exec
calls
are
being
made.
A
We
can
see
what
commands
are
being
run.
So
we
see
a
lot
of
run
c
stuff.
We
see
docker
docker,
init
a
whole
bunch
of
commands.
The
arguments
that
are
the
first
argument,
that's
being
passed
in
is
available
as
well,
and
so
we
get
all
this
information
and
that's
the
power
of
bpf
right.
This
is
what
we
love
about.
Bpf
trace.
It's
like
these
one-liners
are
very
few
lines
of
code
and
bpf
is
like
taking
care
of
all
this.
A
All
this
for
you
and
getting
a
wealth
of
information,
and
so
then
also
in
this
script,
so
the
pixie
side
we
can
say
maybe
we
want
to
filter
for
something,
maybe
we're
interested
in
all
the
docker
commands
that
are
being
run,
and
so
we
update
that
run
again
and
it
didn't
actually
have
to
redeploy
the
second
time
it's
already
deployed.
So
it
went
much
quicker
and
now
we
see
only
commands
that
are
so.
A
The
query
actually
went
so
when
I
hit
run
it
actually
queried
through
the
distributed
system,
went
down
to
the
nodes
and
collected
all
the
commands
in
the
in
the
in-memory
database
in
the
in
the
tables
that
run
have
the
docker
command
running
and
it
just
pulled
those
ones
up
back
to
the
ui.
And
so
now
we
see
just
those
docker
commands
running.
A
A
Deploying
deploying
deploying
okay-
and
here
we
are,
and
now
this
one
ran,
and
so
this
one
actually
shows
another
kind
of
cool
part
of
the
the
pixie
part.
So
so
what
happened
here
is
the
bpf
trace
probe
that
goes
throughout
your
cluster
and
it
starts
monitoring
for
all
the
tcp
drops
in
your
cluster
right.
It's
doing
that
and
we
can
collect
the
raw
data,
but
with
pixi.
A
Right,
you
can
kind
of
see
the
different
things
and
where
the
drops
are,
and
if
you
hover
over
a
particular
connection,
you'll
see
how
many
drops
actually
happened
on
that
connection
and
whether
it
should
be
a
concern
or
not.
So
if
you're
trying
to
troubleshoot,
you
know
tcp
sort
of
related
issues.
This
could
be
very
useful
for
you
now
you
might
look
at
this
and
say
whoa
whoa.
This
is
so
much
data.
How
can
I
make
any
sense
of
this
and
so
I'll
show
off
one
other
feature
here.
A
Right,
okay,
so
what
I
did
is
I
actually
pulled
up
the
the
pixel
script.
That's
driving
this
whole
page
here
and
it's
the
source
code.
You
can
actually
just
pull
it
up,
live
in
pixie
and
we
can
go
through
here
and
I
can
say
you
can
actually
see.
What's
powering
it,
you
can
see
the
embedded
bpf
trace
script
that
we
got
from
the
community.
You
can
see
there.
A
You
know
you
can
actually
even
see
the
copyright.
Dale
hamel
wrote
this
initially
right
and
then
we
go
down
and
we
have
the
the
kind
of
pixel
part
of
it,
which
is
taking
that
bpf
trace
script,
deploying
it
and
augmenting
it,
and
we
have
a
line
here
that
says
set
a
threshold
to
display
like
we
want
to
say
how
many
drops
right.
So
maybe
we
want
to
change
our
query
now
because
we're
we're
hunting
for
something,
and
so
let's
say
I'm
only
interested.
A
I
was
having
wi-fi
issues
earlier,
so
it
could
be
that
so.
But
the
point
was
that
you
know
you
can
change
the
kind
of
script
on
the
fly
and
kind
of
what
would
have
happened
is
the
visualization.
You
would
have
just
seen
those
kind
of
connections
where
there's
a
higher
amount
of
tcp
drops
and
you
can
kind
of
debug
in
real
time
in
this
way,
because
you
can
pull
up
the
editor.
A
A
You
get
a
lot
of
powerful
stuff
out
of
the
box
right
there,
but
there
is
future
work
and
so
right
now
we
only
support,
k,
probes
and
trace
points
and
in
the
future,
we're
you
know,
looking
at
at:
u
probes
due
probes,
future
work
and
then
also
kind
of
we
have
this
restriction
because
we
force
everything
into
a
structured
table.
You
can't
have
these
random
printfs
all
over
the
place.
A
We
say
you
can
have
multiple
printouts,
but
they
should
all
output
to
a
single
table,
and
so
they
should
all
be
have
the
same
number
of
columns
right,
but
these
are
kind
of
future
work
things
that
we
can
change,
help
better
support
the
aggregation
style
scripts
that
I
was
mentioning.
So
all
these
sort
of
things
we're
going
to
be
looking
into.
A
B
B
A
C
A
Yeah
anytime,
I
think
it
was
mentioned
before
too,
like
with
bpf
right
anytime.
There's
that
question
comes
up
of
security
and
it
is
a
legitimate
concern
right
even
before
the
bpf
trace
stuff,
just
like
with
everything
else
that
we
do
like
tracing
all
the
traffic.
It's
like
a
question
like.
Can
we
limit
that?
And
so
we
have
some
security
modes
to
like
obfuscate,
the
payloads
of
http
traffic
of
other
fields
when
you
want
to
be
in
in
the
secure
mode
and
so
for
bpf
trace
as
well?
Well,
we
haven't
done
anything
specific
there.