►
From YouTube: eCHO episode 34: Parca
Description
Liz is joined by Frederic Branczyk to talk about Parca, a project that continuously profiles CPU & memory usage using eBPF
Show notes:
A
Hello
and
welcome
to
echo
this
is,
I
believe,
episode
number
34,
so
as
always,
we'd
love
to
hear
from
you
hear
where
you're
watching
us
from
do
say
hello
in
the
comments
we've
got
a
ton
of
news
today.
Let
me
just
bring
up
the
headlines
loads
of
articles
to
share
this
week,
so
I'm
hoping
some
of
you
who've
written.
A
Talking
about
how
we
can
use
bpf
to
mitigate
transient
execution
attacks,
like
spectre,
I
haven't
seen
this
particular
talk.
Yet
I
have
flicked
through
the
slides.
It's
really
really
interesting
stuff,
and
I
have
seen
daniel
talk
a
bit
about
this.
I
think
it
was
at
ebpf
summit
as
well,
so
it's
fascinating
stuff
and,
if
you're
interested
in
how
that
those
kind
of
speculative
execution
attacks
work.
This
is
really
eye-opening
stuff,
hello
to
assadisman,
from
assad,
in
fact,
from
bangladesh
all
the
m's
in
chat
yeah.
In
fact,
we've
got
another
m.
A
We've
got
michael
from
bavaria
as
well,
hi
to
christian
from
peru
great
to
see
you
with
us
all
right
and
then
there's
another
couple
of
talks
and
I'm
afraid
they've
got
me
in
them.
So
here
is
one
that
myself
and
my
colleague
christopher
did
for
our
friends
at
container
solutions,
and
we
talked
about
what
ebpf
and
psyllium
are.
A
A
Then,
if
you've
still
not
heard
enough
from
me,
I
was
on
a
podcast.
Now
I
actually
recorded
this
last
year,
but
it
came
out.
I
saw
it
a
bit
on
the
the
twitter
sphere
this
week,
so
I
I
thought
I'd
include
it
today,
where
I
had
a
really
interesting
and
fun
discussion
with
guy
from
snick
about
security.
A
Ebpf
came
into
it.
I
I
enjoyed
that
conversation,
so
I
hope
you'll
enjoy
that
one
as
well
and
then
another
one
I
haven't
listened
to
yet,
but
I
hear
is
really
interesting
with
a
past
guest
from
from
echo
dave,
tucker
and
brent
salisbury,
who
were
featured
on
this
episode
of
heavy
networking
talking
about
ebpf
and
cloud
native
networking.
So
I
can't
wait
to
check
that
out.
I
hear
it's
good,
so
that's
lots
of
things
to
watch
lots
of
things
to
listen
to.
We've
got
a
few
things
for
you
to
read
as
well.
A
This
is
a
really
good
article
from
arthur
ciao.
I
haven't
worked
through
all
the
examples
here,
but
it's
exactly
the
kind
of
thing
that
that
I
love
it's
talking
about
how
something
works
and
showing
you
it
in
code.
So
what
he's
done
here
is
implementing
kievaneti's
network
policy
and
building
an
example
of
how
that
would
work
in
ebpf.
So
that
seems
like
a
really
fun
thing
to
check
out.
A
A
A
Okay,
that's
some
things
to
read,
and
then
maybe
there
are
some
things
that
you
might
like
to
try
out.
There
is
an
eb
ebpf,
debugger
called
edb.
I
think
this
is
very
new
and
it
looks
really
interesting.
I
think
the
idea
here
is
that
you
can
run
your
ebpf
code
in
user
space
to
debug
them
more
easily
than
you
could
if
they
were
actually
running
within
the
kernel.
So
if
anybody's
tried
that
out,
let
us
know,
I
think
that
could
be
a
really
interesting
thing
to
explore
on
a
future
episode
of
echo.
A
Another
new
tool
that
we've
seen
is
from
rancher
they've
released
something
called
lock
c,
which
uses
ebpf
and
the
linux
security
modules
interface.
So
if
you're
a
regular
viewer,
you
might
remember
we
had
kp
singh
on.
I
think
it
was
in
december
talking
about
how
that
lsm
interface
sits
in
the
kernel
and
how
we
can
use
use
it
to
trigger
ebpf
programs
to
provide
more
flexible
and
bespoke
security
policies,
but
using
that
lsm
interface
and
this
lock
c
project
looks
to
be
doing
exactly
that.
So
another
really
interesting
development
there.
A
That
would
be
fun
to
try
out
and
the
last
item.
I've
got
here
under
tools:
it's
not
a
new
tool,
it's
a
new
location
for
the
source
code
for
bpf
tool,
so
you
will
have
seen
we
had
quentin
on
back
last
year,
talking
about
bpf
tool
and
showing
us
all
the
amazing
things
we
can
do
with
bpf
tool.
He's
got
an
incredible
twitter
thread
of
I
don't
know,
50
odd,
tweets
with
different
items
that
you
can
use
bpf
tool
for.
A
If
you
want
to
build
the
code
or
look
at
the
code,
you
can
now
find
a
copy
of
that
on
github.
So
the
main
development
is
still
going
to
go
on
on
the
kernel
mailing
lists,
but
it's
a
much
easier
way
to
check
out
the
the
source
code
for
that
all
right.
Yes,
mercedes,
saying:
let's
bring
up
this
comment
so
much
good
content.
I
really
do
think
it's
been
a
busy
week
or
two
in
the
world
of
evpf,
which
is
fantastic
to
see
all
right.
A
A
B
Yeah,
of
course,
so
so
parker
is
a
continuous
profiling
project
and
continuous
profiling
is
still
kind
of
an
emerging
area
in
the
observability
world.
So
I
think
it's
worth
also
spending
a
couple
of
sentences
just
on
the
methodology
itself,
without
before
diving
into
the
tool
itself.
So
continuous
profiling
is
in
essence,
just
profiling.
B
All
the
time
right,
we
profiling,
has
kind
of
been
a
part
of
the
developer
toolbox.
Ever
since
programming
has
existed
right
like
I
I
actually
when
I
started
the
company,
I
did
some
research
and
basically,
since
the
six
1960s
we've
been
doing
profiling
because
we
always
needed
to
understand
what
our
running
programs
were
doing.
B
Yeah,
exactly
theoretically,
profiling
can
be
anything
where
you
have
a
stack
trace
and
a
number
attached
to
that
stack
trace.
So
we
we've
seen
network.
I
o
profilers,
we've
seen
block
device
profilers,
so
really
anything
any
resource.
Usually
you
can
you
can
associate,
but
sometimes
there
are
even
profilers
that
are
specific
to
sp
to
certain
runtimes.
For
example,
the
go
runtime
has
a
special
profiler
for
understanding
go
routines.
B
Those
are
never
going
to
be,
let's
say
in
a
generic
profiler
like
linux
perf,
because
the
operating
system
just
doesn't
know
about
concepts
like
that,
so
there
are
definitely
also
profilers
that
are
useful
for
your
specific
language
and
runtime,
but
coming
back
to
like
continuous
profiling
when
we
think
about
profiling
most
of
the
time.
The
first
thing
that
kind
of
pops
into
our
head
when
we
think
about
always
doing
this
is
like
this
is
like
that
sounds
expensive
right,
like
profiling
is
kind
of.
B
Has
this
association
with
having
overhead
and
they're
kind
of
there
there's.
I
guess
two
major
reasons
why
we
can
do
continuous
profiling,
so
the
the
first
and
foremost
is
that
we
use
a
technique
of
of
profiling
called
sampling
profilers.
There
are
kind
of
a
couple
of
different
kind
of
profilers,
and
talking
about
all
of
them
is
maybe
a
bit
too
too
much
for
today.
So
I'm
going
to
focus
just
on
sampling
profilers
today,
but
xamarin
profilers
are
essentially
that
we
look
at
a
stack
trace
in
the
case
of
cpu.
B
B
100
samples
per
second
over
time
actually
accumulates
and
we
get
a
statistically
significant
representation
of
our
program,
and
so
that's
why
we
can
always
do
this
and
then
why
we're
here
to
talk
about
evpf,
right,
ebpf
kind
of
put
that
put
sampling
profilers
on
steroids
in
terms
of
overhead
lowering
overhead?
I
guess
because
we
can
capture
exactly
the
data
that
we
want
in
exactly
the
format
that
we
want
and
only
export
it
every
couple
of
seconds
as
opposed
to
capturing
every
single
sample.
B
A
B
Right
right,
so
it's
it's
even
so.
We
use
the
the
like
perf
subsystem
for
this
as
well,
but
in
instead
of
doing
what,
like
the
canonical
perf
tool
does,
which
which
just
captures
a
whole
lot
more
data
than
what
we
actually
need.
We,
we
really
only
capture
the
stack,
trace
and
capture.
How
often
have
we
seen
that
stack
trace
and
that's
it
and
then
every
10
seconds,
our
user
space
program
kind
of
takes
everything
that
we've
saved
in
our
ebpf
map
like
resets
it
and
in
like
after
another
10
seconds.
A
Right,
okay,
and
when
you
were
thinking
about
the
idea
for
parker
was,
did
you
have
the
we
want
to
do
continuous
profiling?
First
or
you
know,
how
can
we
do
that
or
were
you
thinking
with
this
ebpf
thing
we
can?
This
is
something
we
could
do
continuous
profiling
with
you
know,
was
it
evp,
first
or
continuous
profiling
that
you
wanted
to
yeah.
B
It's
a
it's
actually,
it's
funny
that
you
ask
it
in
that
way,
because
when
I,
when
I
started
the
company,
my
my
first
reaction
actually
was,
I
don't
want
to
concern
myself
with
the
collection
of
profiling
data
at
all.
I
felt
like
the
harder
problem
was
storing
it
and
analyzing
it
in
a
useful
way,
and
you
know
I'm
I'm
the
last
couple
of
years.
B
So
we
were
trying
to
find
a
way
to
mitigate
this
problem,
and
actually,
since
we've
mentioned
it
a
couple
of
times
today,
our
very
first
proof
of
concept
we're
actually
just
using
linux
perf
and
so
after
getting
some
like
good
successes
with
that
we
were
like
okay.
How
can
we
do?
How
can
we
do
something
even
better
or
can
we
do
something
even
better
and
that's
kind
of
how
we
ultimately
stumbled
into
evpf
world.
B
Yeah,
that's
a
that's
a
great
question,
so
best
supported
are
compiled
languages
today.
So
basically
like
go
c,
plus,
plus
c
rust,
haskell
anything
that
compiles
to
you
know
machine
executable
code.
There
is
kind
of
a
middle
ground
that
also
works
quite
well,
but
doesn't
quite
work
in
the
zero
configuration
way
just
yet.
B
So
whenever
we
deal
with
languages
that
or
runtimes
that
have
a
just-in-time
compiler,
they
actually
eventually
do
produce
machine
readable
like
executable
code
right.
So
it
takes
a
bit
more
effort,
but
most
of
those
jit
runtimes
usually
write
the
kind
of
mapping
of
memory
addresses
to
symbol
data
that
we
humans
understand
to
some
standardized
place,
and
then
we
can
grab
them
there
and
that
parka
does
have
support
for
as
well
so
like
node.js
and
java
erlang.
B
All
of
these
have
support
for
this,
but
they
do
require
you
to
pass
a
flag
to
your
to
your
command.
So
it's
not
quite
zero
instrumentation
or
not.
It's
not
quite
zero
changes.
It
is
zero
instrumentation!
You
don't
need
to
do
anything
to
your
code,
but
in
the
node.js
case,
for
example,
I
think
it's
like
dash
dash
prof
perf
map.
I
think,
or
something
like
that,
that.
B
Yes
exactly
right,
however,
that's
really
just
an
intermediate
state.
The
whole
purpose
of
the
parka
project
is
that
it's
entirely
hands-off.
You
only
need
to
deploy
the
parka
agent
and
you
get
whole
system
visibility.
So
that's
that's
just
you
know.
The
perf
map
interface
is
something
that
the
linux
kernel
had
standardized
for
perf,
and
so
that's
something
that
exists
that
we
can
already
take
advantage
of,
but
we
want
that
experience
to
be
even
simpler
than
that
right.
As
I
said
it,
it's
supposed
to
be
100.
You
don't
have
to
do
anything.
A
B
Code,
but
in
cases
where
we
or
we
don't
have
a
jit
like
let's
say,
ruby
or
python,
it's
exactly
like
you
say
and
those
we
also
need
specific
integrations
for
which
we
are
working
on,
where
we
actually
inspect
the
python
or
ruby
runtime
memory,
because
at
the
end
of
the
day
they
build
kind
of
virtual
processors.
You
can,
you
can
think
of
it
as
within
the
the
interpreter,
and
so
we
just
need
to
read
the
frames
that
they
built
in
their
memory
as
opposed
to
on
operating
system
stack.
B
A
B
I
I've
already
got
it
set
up,
because
it's
really
just
these
three
commands,
but
what
I'm
I'm
gonna
show
it
on
a
kubernetes
cluster
and
as
I
mentioned,
we
have
kind
of
two
components:
we
have
the
parka
server,
which
is
the
thing
that
stores
the
the
data
and
allows
you
to
query
the
data
with
a
built-in
ui
and
then
the
thing
that
actually
does
the
collection
and
the
actual
profiling
is
the
parka
agent.
So
they
are
distinct
components,
and
so
you
can.
B
B
B
As
I
said,
every
10
seconds,
it
grabs
the
profiling
data
from
every
container
running
on
that
on
that
node
and
then
compiles
that
in
the
open
standard
pprof,
which
is
a
an
open
standard
for
representing
profiling
data
that
google
developed
and
then
the
parker
agent
takes
that
and
sends
that
to
the
to
the
parker
server
and
that's
where
we
can
then
analyze
it.
And
let
me
pull
that
up.
B
B
B
In
this
case,
the
parka
agent,
actually
today
only
supports
cpu
profiling,
but
that's
that's
just
a
matter
of
starting
with
something
and
cpu
tends
to
be
the
most
expensive
resource
that
we
have
in
in
terms
of
cloud
resources
and
that's
why
most
people
are
most
interested
in
optimizing
their
cpu
cpu
time.
So
that's
why
we
kind
of
started
with
this
one.
We
definitely
want
to
add.
You
know,
allocation
profiling,
network
profiling,
disk
profiling.
B
A
B
Yeah
yeah
absolutely
that's
kind
of
the
classic
use
case
for
for
profiling
right,
but
we
find,
since
we
have
this
whole
system
overview,
we're
profiling,
absolutely
everything
all
the
time.
We
actually
get
a
really
fantastic
view
of
what
is
my
biggest
cpu
offender
in
my
entire
infrastructure
and
that
then
very
very
quickly
becomes
a
tool
for
cost
optimizations,
because
we
can
actually
statistically
significantly
optimize
our
infrastructure
right,
not
like
individual
points
in
time.
A
We've
got
a
question
here
that
I
think,
since
we're
just
talking
about
running
on
on
kind
of
managed
cloud
services
door
is
asking
whether
or
not
you've
run
parker
on
a
variety
of
different
managed.
Kubernetes
services.
Do
you
think
there's
any
problems
running
on
those
platforms,
and
I
guess
the
question
there
is:
can
you
load
the
ebpf
programs
into
the
kernel
on
all
those
platforms.
B
B
We
know
of
reports
that
eks
works
and
aks,
but
I
I'm
not
sure
about
the
rancher
or
kubernetes
engine,
but
I'm,
as
you
said.
Basically
all
of
this
revolves
around
whether
we
can
load
the
evpf
program
and
whether
we
have
access
to
the
kubernetes
container
runtime
interface
and
we
do
support
we
support
docker.
B
We
support
cryo
and
we
support
container
d
for
kind
of
discovering
the
the
c
groups
to
attach
our
profilers
to
for
for
the
individual
container,
so
that
already
kind
of
brings
us
to
to
the
topic
that
you
know
if
we
have
a
cri
runtime.
That,
let's
say,
is
a
virtual
machine
that
wouldn't
work.
I
think
there
are
like
a
handful
of
providers
that
do
something
like
that.
A
B
B
If
you,
if
you
know
me,
this
may
not
be
super
surprising,
because
I
happen
to
be
the
maintainer
of
the
kubernetes
service
discovery
in
prometheus
and
I'm
generally
a
prometheus
maintainer,
but
essentially
how
it
works.
Is
each
parka
agent
on
every
node
basically
asks
the
the
kubernetes
api
to
tell
it?
B
What
are
the
pods
and
containers
running
on
the
host
that
I
am
on
and
whenever
a
new
one
gets
created
or
deleted,
we
discover
the
c
group
c
groups
associated
with
those
containers
and
then
attach
our
ebpf
programs
using
the
perf
subsystem
to
that
c
group,
and
that
way,
basically,
every
time.
Basically,
the
the
perf
subsystem
is
kind
of
buffer
based.
No,
it's
overflow
based
so
at
most
we
would
get
a
hundred
samples
per
second
meaning,
like
at
most.
A
A
B
So
yeah,
let's,
let's
continue
the
demo
here
so,
as
I
said,
the
very
first
thing
you
would
do
is
you
would
select
the
type
of
profile
and,
as
I
said,
this
is
all
completely
generic
and
based
on
the
pprof
format.
So
if
you
have
prof
formatted
profiled,
let's
say
for
memory
you
can
send
those
to
parker
as
well,
and
the
parka
server
will
understand
it.
It's
just
that
the
evpf
program
doesn't
support
creating
those
today
so
yeah
you
can.
B
You
can
first
kind
of
look
at
the
cumulative
metrics
of
each
of
the
processes
and
if
there
is
one
particular
one,
let's
say
we
had
a
spike
here
right.
Maybe
this
one
is
one
that
we're
particularly
interested
in
understanding
what's
happening
right,
and
we
can
then
browse
this
particular
profile
that
was
taken
at
that
point
in
time
and
see
where
the
cpu
time
was
spent.
B
So
at
this
point
it's
some
sometimes
people
haven't
seen
these
types
of
visualizations
yet
so
this
is
what's
called
a
flame
graph
or
actually
in
this
representation
it's
called
an
icicle
graph,
a
funny
association
with
our
brand.
That
was
totally
not
intentional.
You
know
polar
signals
in
terms
of
icicles.
B
B
Exactly
yeah
yeah
yeah
yeah
anyway.
So
the
way
the
way
you
read
these
essentially,
is
that
the
very
top
one,
the
root
kind
of
represents
all
cpu
time
like
of
this
profile
and
then
the
further
down
we
go.
The
spans
are
always
relative
to
that
root.
So
if
we
let's
say
hover
over
this
one,
we
can
see
runtime
that
go
exit.
All
the
stacks
under
this
particular
function
in
cumulative
make
up
96
of
our
cpu
cpu
time,
and
then,
if
we
go
further
down
it,
it
goes
down.
B
But
basically
what
this
means
is
that
the
if
there's
a
span,
let's
say,
for
example,
this
one
that
has
an
overlap
and
not
necessarily
all
children
make
up
all
the
cpu
time
of
that
span.
Then
that
means
this.
This
kind
of
gap
in
between
here
that's
optimization
potential,
because
that's
something
that
truly
this
function
is
doing
as
opposed
to
what
the
what
the
children
are
doing
here
or
if
it's
a
leaf.
If
we
can
optimize
this
leaf,
then
it's
going
to
have
exactly
that
effect
on
the
cumulative
cpu
time
of
our
process.
B
As
I
said
earlier,
what
we
can
do
is
we
can
not
only
look
at
profiling
data
as
a
point
in
time,
but
we
can
look
at
all
the
cpu
time
our
process
has
spent
over
the
lifetime
of
it
of
this
process
right.
So
here
I've
filtered
down
all
of
our
data
just
to
seeing
the
parka
container
so
parka
profiling
itself.
B
We've
got
not
only
a
representation
of
this
process
at
a
point
in
time,
but
over
the
lifetime
of
the
process,
and
if
we
can
optimize
something
in
this
report,
it
means
because
this
is
kind
of
statistically
significant
data.
It
means
it
will
actually
translate
into
a
cost
saving
of
our
of
our
infrastructure.
B
So
that's
that's
why
continuous
profiling,
or
it's
one
of
the
reasons
why
continuous
profiling
is
so
interesting,
and
there
are
a
couple
of
other
things
that
are
just
possible
because
we're
capturing
this
data
over
time
earlier
we
were
looking
at
this
peak
right,
and
so
another
thing-
that's
really
interesting
to
do
is
look
at
differences
of
certain
points
in
time
or
even
entire
versions
of
software
right.
So
it's
kind
of
this
question:
that's
as
old
as
software
engineering,
which
is
like.
B
Why
was
my
process
spending
like
having
the
cpu
spike
here,
but
not
here
right,
and
we
can
actually
do
that.
We
can
select
a
point
in
time
here.
We
can
select
the
time
here
and
it
will
tell
us
okay
in
this
case
it
was
pretty
dramatic.
Almost
everything
was
worse,
but
we
can
see
kind
of
shades
of
red
where
things
were
particularly
bad
right.
B
So
here
this
is
the
difference,
as
we
can
see
in
the
like
second
line
in
the
hover
bubble,
it
was
plus
infinity
which
me,
which
means
that
this
entire
span
was
not
there
at
all
in
the
first
in
the
in
the
first
profile
that
we
selected
so
looking
at
all
of
the
spans
here.
Basically,
what
we
can
infer
from
that
is
garbage
collection
was
happening
at
this
point
in
time
and
that's
pretty
typical
to
see
in
a
garbage
collected
language.
Let
go
but
it's.
B
B
A
A
B
B
B
B
I
I'm
guessing
that
it's
been
discussed
numerous
times
here,
but
just
for
for
folks
that
it
might
be
kind
of
new,
so
the
bpf
kind
of
introduced
and
I'm
by
far
no
expert.
So
if
there's
someone
who's
more
knowledgeable
and
I'm
saying
something
stupid,
please
correct
me,
but
basically
it
kind
of
started
around
this.
Compile
ones
run
everywhere:
initiative
because
it
with
like
the
bcc
tool
chain.
A
B
So
I'm
we
have
a
bunch
of
things
in
progress
and
with
some
of
those
things.
That
may
not
be
the
case
anymore,
because
you
know
this.
This
ecosystem
is
work
moving
so
quickly
that
some
of
the
things
we
may
be
using
may
not
be
supported
in
rel.
Eight,
I
don't
think
actually
go.
117
is
necessarily
supported.
B
I
think
it's
just
what
we
happen
to
use
when
we
did
the
like
go
mod
initialization,
so
I
think
that's
something
that
we
should
check
out,
but
we're
doing
our
best
to
keep
the
kernel
version
as
low
as
possible,
but
unfortunately,
for
like
baseline,
compile
ones
run
everywhere.
You
need
a
5.2
kernel,
I'm
not
100
sure
which
rail
8
kernel
equivalent.
That
is
yeah
fine.
So
with.
B
Very
long
intro,
that's
how
we
write
our
write,
our
ebpf
programs,
which
is
why
you
don't
see-
let's
say
a
string
replacement
in
here
or
something
it's
truly
just
the
c
code,
and
we
compile
it
like
that,
so
we
with
epf.
This
is
also
a
little
bit
different
with
the
bcc
tool
chain.
We
actually
are
even
more
restricted
than
what
it
was
with
bcc,
so
we
need
to
pre-declare
a
bunch
more
things.
So
in
this
case
we
say
also
how
many,
how
many
addresses
can
be
in
a
single?
B
B
Then
we've
got
a
couple
of
ebpf
maps
and
the
interesting
ones
are
which
one
was
what
I'm
kind
of
reading
this
right
now
as
well,
but
we've
got
conceptually.
We've
got
one
that
has
a
mapping
of
a
stack
id
to
the
actual
stack
right,
and
then
we
have
another
one
that
says
the
stack
id
to
the
observations.
B
So
how
often
have
we
seen
this
particular
stack,
because
that's
how
we
are
gonna,
ultimately
infer
the
cpu
time
right,
because
we're
saying
at
most
we're
doing
a
hundred
observations
per
second
and
the
way
that
we're
doing
that
is
by
saying
we've
seen
this
stack
with
this
id,
let's
say
five
times
and
that's
how
we're
inferring
the
cpu
time
from
from
there.
That's
essentially
how
any
sampling
profile
ever
works.
A
B
But
because
we're
we're
doing
it
like
this
and
storing
it
in
the
ppf
map,
we
can
actually
only
every
once
in
a
while
read
these
these
maps
from
user
space
and
all
the
other
time
we
can
just
completely
leave
it.
Leave
it
running
right.
Just
whenever
the
perf
sub
subsystem
decides,
this
ebpf
program
should
be
run,
it
runs
and
inserts
or
increases.
This
particular
stack
trace
that
that
we're
seeing
at
that
point
in
time.
So
these
are
the
definitions.
B
This
is
this
one's
also
important.
That's
what's
actually
already
used
above
the
way
that
we
actually
identify
the
the
stack
that
we're
seeing
with
the
with
the
counter
is
that
we
use
the
process
id
the
id
of
the
user
space
stack
and
the
kernel
stack.
So
that
way,
we're
kind
of
uniquely
identifying
the
stack
trace
of
a
particular
process.
A
B
This
is
the
the
actual
program,
so
the
very
first
thing
is
we
we
check.
Is
this
even
a
pro
process
that
we
are
remotely
interested
in?
I
think
zero
is
that
this
is
some
kernel
thing.
That's
happening,
so
definitely
not
a
user
space
program
that
we're
interested
in
right
now,
the
next
and
then
we
start
building
our
stack
key,
that
we
were
defining
above
right
and
then
we
use
a
couple
of
bpf
helper
methods.
So
these
are
we
didn't
write
these
the
bpf
get
stack
id.
B
This
is
something
that
we
get
from
the
helpers
and
here
we're
saying
that
we
want
the
user
stack.
So
the
what
this
does
essentially
is
it
walks
the
frame
pointers,
and
we
can
talk
about
that
in
a
bit
more
detail
in
a
bit
because
I
actually
have
we've
got
some
kind
of
new
things
that
we're
working
on
in
this
this
area.
So
I've
got
some
things
that
I
can
show
about
that
because,
like
some
visualizations
but
vaguely
speaking,
we
can.
B
A
B
B
100
yeah
yeah.
That
said,
we
we
do
put
a
great
deal
of
care
into
making
sure
that
giving
us
that
privilege
is,
you
know,
respected
as
much
as
we
can
possibly
do
that
and
one
of
the
things
I
think
that
is
kind
of
cool.
Since
I
do
already
have
my
editor
here,
we
put
a
lot
of
effort
into
actually
doing
byte
by
byte
reproducible
builds.
A
We're
getting
fairly
close
to
the
end
of
the
hour,
and
I
know
you
have
a
few
things
to
to
talk
about
around
sort
of
what's
coming
next
and
william
has
asked
he's
hoping
that
in
the
near
future,
there'll
be
things
like
memory
discs
and
and
networking
support.
So
do
you
want
to
tell
us
a
bit
about
what
the
future
holds
for
parker
and
what
you
have
in
class.
B
Yeah
absolutely
so,
there
are
a
lot
of
things
happening
and
I'm
going
to
keep
it
focused
on,
what's
hopefully
most
relevant
to
the
to
this
audience.
We
have
an
entire
company
working
on
this
kind
of
umbrella
project
right.
So
there
are
a
lot
of
things
happening,
but
specifically
in
the
bpf
space.
Let
me
share
some
slides.
B
B
That
is
using
something
called
frame
pointers
you
can
think
of
it
as
when,
when
our
operating
system
or
our
compiler
really
builds
our
stack
frames
of
our
compiled
programs,
it
reserves
one
register
to
say
where
the
upper
frame
essentially
is
located
or
where
it
ends,
I'm
not
exactly
sure
of
the
the
exact
semantics,
but
it
doesn't
really
matter.
The
point
is
that
we
can
use
the
frame
pointers
to
kind
of
walk,
a
linked
list
to
get
the
entire
stack
representation,
and
this
is
really
great,
because
walking
a
linked
list
is
super
fast.
B
B
That
said,
unfortunately,
and
brandon
greg
comments
on
this
in
on
his
website
as
well.
Unfortunately,
there's
a
very
evil.
Compiler
optimization
that
kind
of
ignores
frame,
pointers
and
reuses
that
register
that
frame
pointers
are
supposed
to
be
written
into
as
an
additional
register.
For
you
know,
performance
reasons
I.
B
B
B
B
You
know
without
needing
anything
else,
because
the
alternative-
and
this
is
what
perth
does
today-
is
we
kind
of
take
an
entire
snapshot
of
the
whole
stack,
put
it
into
user
yeah
into
user
space
and
then
unwind
it
from
there,
because
we
can
kind
of
do
that
offline.
But
that
has
a
couple
of
drawbacks.
First
and
foremost,
we
need
to
kind
of
copy
that
stack
every
single
time.
B
We
capture
a
a
sample,
as
opposed
to
just
walking
a
stack,
but
also
it
means
that
we're
potentially
grabbing
some
super
sensitive
data
from
a
process's
memory
right
and
then
at
least
somewhat
persistently
putting
it
into
user
space.
And
so
ideally,
we
would
be
able
to
do
this
at
collection
time
right,
and
so
I'm
I'm
really
excited
about
this,
because
it's
kind
of
a
novel
thing.
We
haven't
really
seen
this
in
other
profilers
before
so
yeah.
We're
not
even
sure
whether
this
is
gonna
work
out,
but
if
it
does.
A
A
And
a
really
interesting
point
that
you
know
what
is
intended
as
an
optimization
could
result
in
you
know
if
everybody
wanted
to
profile
their
applications,
they
might
have
to.
You
know,
find
a
way
around
that
optimization.
Maybe
maybe
they
could
just
turn
off
that
optimization,
but
it's
pretty
cool
that
you'll
have
a
sort
of
zero.
A
B
That
that
said,
please,
if
you
have
the
possibility,
please
keep
frame
pointers
if,
if,
if,
if
like
someone,
anyone
who
watches
if
you're
walking
away
with
anything
after
this
episode
today
then
plea
that
it
please
let
it
be,
keep
your
frame
pointers.
It
just
makes
life
so
hard.
It
makes
debugging
and
production
completely
impossible.
It
makes
capturing
this
kind
of
data
impossible
and,
as
a
matter
of
fact,
we've
been
speaking
to
several
engineers.
B
Who've
built
similar
systems
at
you
know,
hyperscaler
that
you're
thinking
of
right
now
and
all
of
the
hyperscalers
have
essentially
had
internal
conversations,
and
they
all
came
to
the
conclusion.
It's
not
worth
omitting
them.
Just
just
keep
frame
pointers.
The
the
tiny
performance
gain
that
you're
getting
is
not
worth
not
being
able
to
understand
your
systems.