►
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Do it Live: Measuring your Applications in Production - Jason Keene, Pivotal
Ever had a bug that only shows its head in production? Ever wanted to ask arbitrary questions about your software while it is running with minimal impact on performance? In this talk, we go beyond pre-defined instrumentation and look at various techniques of gathering information about your application to help you solve your production woes. Topics include: - How to ask good questions, and get good answers. - Securely running a container in your cluster for probing your application. - How to use eBPF/BCC tools to gather data without impacting performance. - Techniques for using ptrace/debuggers when all else fails. - Knowing when to *not* use these tools and how to better instrument your code in advance of a production issue. With these tools and techniques you will be better equipped to understand your software and solve the problems you'll face in production.
To learn more: https://sched.co/GrXR
A
So
today,
I
wanted
to
talk
about
measuring
your
applications
in
the
production
environments
that
they're
running
in
and
the
reason
I
wanted
to
talk
to
you
about
this
is
I,
feel
it's
invaluable
from
a
development
perspective.
To
be
able
to
do
this,
oftentimes
I
feel
the
developers.
Don't
have
the
tools
to
be
able
to
ask
the
questions
they
want
to
ask
so
I
feel
this
is
something
that
can
provide
a
lot
of
value
to
developers.
A
What
you
know,
one
of
them
is
well
you
get
to
see
your
software
under
actual
load,
so
you
get
to
see
its
failures
when
it's
under
load,
you
could
see
faults
as
they
occur.
You
can
discover
patterns
of
usage
of
your
users,
which
can
inform
you
know
future
features,
but
primarily
you
really
want
to
be
able
to
make
measurements
in
production
to
be
able
to
debug
this.
A
Your
system
when
problems
occur
sometimes
reproducing
the
problem
in
your
production
environments,
reproducing
that
in
an
artificial
environment
is
too
time-consuming
or
too
difficult,
and
and
sometimes
simply
like
being
able
to
reproduce
a
problem.
It's
not
something
you
know
how
to
do
so.
You
have
to
collect
data
from
your
production
environment
just
to
seed
analysis
in
a
different
environment.
A
Ultimately,
production
is
reality.
Everything
else
is
at
best
of
proximity,
so
we
need
to
be
able
to
measure
in
production
to
be
able
to
debug
problems
as
they're
occurring,
but,
more
importantly,
I
feel.
Understanding
the
character
of
your
workloads
is
critical
to
their
successful
operations.
So
being
able
to
make
measurements
in
production
allows
you
to
have
an
understanding
of
your
workloads.
A
A
So
we
want
to
be
able
to
solve
problems
and
we
want
to
be
able
to
understand
our
systems.
So
today's
talk
is
put
into
three
different
sections.
First,
I'm
gonna
go
over
a
method
that
I
like
to
keep
in
mind
when
I'm
debugging
issues
I
think
it's
incredibly
viable.
So
I
wanted
to
share
that
with
you
today.
Next
we're
gonna
go
into
some
tools.
There's
been
some
recent
developments
in
the
past
few
years
in
the
Linux
kernel
and
some
user
space
tools
that
have
recently
been
released.
A
So,
let's
start
off
with
the
method.
So
the
method
I
like
to
call
ask
questions,
get
answers.
So
you
start
off,
say
you
have
a
problem
in
front
of
you
start
off
with
a
space
and
possible
cause
it
just
visualized
as
an
area,
and
the
function
of
a
question
is
to
put
pressure
on
the
area
to
divide
it
into
different
sections.
A
So
in
this
case
our
question,
if
it's
true
the
results
in
the
category,
a
will
remain
if
it's
false
the
categories
and
category
B
will
remain
and
then
oftentimes
asking
a
question
might
not
differentiate
between
certain
possibilities,
so
you
might
have
an
area
in
between
where
there's
overlap.
Where
you
either
result
of
the
question
that
area
will
remain
so
an
answer.
A
The
function
of
the
answer
is
to
cut
off
and
eliminate
sections
of
the
possibility
space
and
that
reduces
your
possibility
space
by
that
amount,
and
so
you
iterate,
you,
ask
another
question:
get
another
answer:
eliminate
part
of
the
possibility
space
do
that
multiple
times
until
your
possibility
space
is
extremely
small
and
you've
narrowed
in
on
the
cause
of
your
problem.
So
I've
used
this
technique.
A
lot
in
my
career
and
I
find
it
to
be
very
valuable
when
you're
using
this
method.
A
It's
critical
that
you
are
confident
in
your
answers
and
I'll
demonstrate
so
say
you
ask
a
question
and
you
get
your
answer
back,
that
you're
85%,
confident
in
the
result,
so
you
might
say:
well,
eighty-five
percent
confidence.
That's
that's
kind
of
high,
but
if
you
applied
this
method,
you're
it's
an
iterative
method,
so
as
you
iterate,
if
you're
only
85%,
confident
and
your
answers
after
five
iterations
you're
down
to
forty
four
percent
confidence
and
your
overall
analysis.
A
So
what
makes
this
even
worse
is,
after
that,
even
if
you're,
a
hundred
percent,
confident
your
later
questions
about
the
answers
that
you
get
to
your
later
questions,
you
can't
regain
that
uncertainty
that
uncertainty
has
crept
into
your
analysis
and
you'll
never
get
it
back.
So
you
really
want
to
be
extremely
confident
in
the
answers
that
you
get
to
the
questions
that
you
ask
so
just
kind
of
visualize
that
here
so
you
might
think
the
problem
is
in
that
narrow
space
that
you
narrowed
in
on,
but
in
actuality
you
eliminated
and
a
previous
step.
A
So
that's
what
that
looks
like
in
practice
and
I
can
speak
from
experience.
It's
extremely
frustrating
when
that
happens,
so
just
asking
a
question
doesn't
help
if
you
can't
get
an
answer
to
your
question,
not
having
the
right
tools
constrains
the
space
of
possible
questions
that
we
can
ask
so
I
like
to
use
the
streetlight
effect
analogy
here.
A
So
if
you're
not
familiar
with
a
streetlight
effect,
there's
a
story,
a
gentleman
is
looking
for
his
wallet,
he
lost
his
wallet
and
a
police
officer
walks
up
to
him
and
says:
well
is
the
wallet
under
the
streetlight
where
you're
looking
and
he
says
no
I
actually
lost
it
in
the
park.
But
this
is
where
the
light
is.
A
So
that's
where
I'm
looking
so
you
might
have
experienced
this
effect
and
using
tools
and
trying
to
debug
problems
yourself,
where
your
tool
kind
of
just
illuminates
a
small
area
of
possibilities,
and
so
that's
where
you
look,
but
your
problem
isn't
the
problem
isn't
actually
where
you're
looking.
So
what
we
want
is
a
flashlight
where
we
can
walk
around
wherever
we
want
and
be
able
to
shine
the
light
on
whatever
we
want
to
be
able
to
find
the
wallet
in
the
park,
not
in
under
the
street.
A
Lamp,
so
we
need
tools
that
can
give
us
answers
to
any
arbitrary
question.
We
want
to
ask
so
yeah
we're
going
to
talk
about
some
of
those
tools
now
to
start
off
with
I
kind
of
just
want
to
frame
the
discussion
around
the
different
types
of
instrumentation.
So
I
like
to
view
these
as
a
hierarchy
at
the
top
of
the
hierarchy
is
static,
always
on
instrumentation.
A
These
are
things
like
logs
metrics
distributed
tracing
the
three
pillars
so
to
speak
of
observability,
there's
an
incredible
amount
of
value
that
you
can
get
out
of
these,
but
the
problem
of
these
is
that
you're
paying
the
performance
penalty
at
all
the
times.
So
you
can't
cover
every
aspect
of
your
system.
Using
these
techniques,
the
performance
penalty
will
be
too
high,
so
incredibly
useful.
There
should
be
the
first
place
that
you
look
to
start
debugging
problem,
but
they
don't
give
you
every.
A
You
know
every
bit
of
coverage
that
you
would
need
after
that
is
static
instrumentation
that
requires
activation.
So
these
are
technologies
like
USD,
T
and
trace
points.
These
are
things
that
a
developer
would
instrument
their
code
with
in
advance
and
when
that
codes
running
in
production,
you're
not
paying
any
performance
penalty
for
that
instrumentation.
So
you
can
kind
of
add
more
of
this
into
your
program
and
then,
when
someone
wants
to
actually
instrument
your
program,
they
can
activate
that
instrumentation.
A
A
So
we
want
tools
that
can
answer
these
arbitrary
questions
about
our
software.
So
we
want
to
be
able
to
intercept
any
point
of
execution
without
restarting
the
process.
We
want
to
be
able
to
read
from
memory
and
registers,
collect
data
across
multiple
processes
and
the
kernel,
ideally
with
low
overhead,
and
to
do
it
all
safely.
A
A
So
yeah
debuggers
used
a
system
called
called
P
trace
process
trace
short
for
cross
trace.
It
allows
a
trace
or
program
to
control
the
execution
of
a
Tracey,
so
I.
Imagine
most
folks
to
promote
with
you.
Buggers
debuggers
can
intercept
Siskel's
signals
that
the
Tracey
is
receiving
can
read
and
write
to
the
Tracey's
register
memory,
including
its
text
section.
You
can
single
step
through
the
Tracey
can
write
breakpoints.
It
has
a
lot
of
capabilities.
A
One
of
the
problems
with
P
trace,
though,
is
that
when
the
tracer
is
running
typically
the
Tracey's
execution
is
suspended
and
most
times
that
means
all
threads
in
that
process
are
suspended.
So
what
does
that
look
like?
So
in
this
case
we
have
a
Tracey
on
the
right
and
the
trace
on
the
left
is
a
tracer.
Just
you
just
think
debugger,
so
the
Tracey
there's
a
breakpoint
set
in
the
Tracey,
so
that
once
that
trap
is
hit,
the
Tracey
is
suspended.
A
A
A
So
I
actually
used
a
technique
that
we'll
discuss
in
a
moment
here
where
I
replace
the
user
with
a
controller
process
so
that
eliminated
some
of
the
you
know,
slowness
from
me
typing
on
the
keyboard,
so
actually
use
that
to
debug
a
problem
that
I
had
in
a
logging
agent.
So
I
mentioned
I
work
in
observability.
We
have
an
agent
that
runs
in
all
of
our
systems,
collecting
logs
from
the
system
and
forwarding
them
out
via
G
RPC.
So
looks
something
like
this
so
periodically.
A
The
agent
would
just
stop
sending
any
data
for
a
matter
of
minutes
and
then
all
of
a
sudden
it
would
just
start
resuming
sending
data.
So
this
is
really
perplexing
to
us.
We
had
the
these
like
reports
from
people
running
our
software,
and
we
didn't
really
have
any
visibility
into
like
why
this
problem
is
occurring.
So
just
a
little
bit
background
on
this
problem,
so
the
agents
uses
a
ring
buffer
internally
to
keep
track
of
messages
and
just
a
little
bit
of
background
about
ring
buffers.
A
You
typically
have
a
read
and
write
in
to
see
in
a
ring
buffer
and
the
write.
Nc
will
always
be
ahead
of
the
read
in
de
see
the
read
and
to
see
you
will
not
advance
past
the
right
end
to
see
it
will
block
until
new
data
is
available.
So
just
just
a
little
bit
of
background
they'll
be
relevant
in
a
moment
here.
So
the
question
that
I
want
to
ask
is
when
the
agent
is
in
this
state,
when
it's
not
sending
any
data,
what
is
the
state
of
these
read
and
write
indices?
A
So
the
bug
that
we
saw
on
the
logging
agent
so
I
just
create
a
ring
buffer
and
that's
spin
up
a
thread
of
execution
to
do,
writes
and
then
another
thread
to
do
reads
so,
in
this
case
I'm
going
to
instruct
del,
which
is
the
go
debugger
to
attach
to
that
process.
I
then
capture
the
standard
in
and
standard
out
of
the
debugger
I
tell
the
debugger
to
start
and
then
I'm
gonna
bugger
attaches.
It
typically
will
suspend
the
program.
A
So,
on
the
left
hand
side,
you
can
see
what
that
sampling
the
inner
body.
That
loop
looks
like
so
we
just
go
to
sleep
for
a
second
and
then
we
send
a
signal
to
the
debugger
that
wakes,
the
debugger
up,
suspends
the
process,
and
then
we,
if
it's
time
to
exit,
we
just
tell
the
debugger
to
clean
up,
and
if
it's
not
time
to
exit,
then
we
tell
it
to
sample
some
data
and
the
commands
that
were
sending
the
debugger
on
the
right
hand
side
here.
So
it's
a
sample
data.
A
We
just
set
a
breakpoint
at
the
function
that
were
interested
in
tracing.
We
then
continue
to
hit
that
break
point,
and
then
we
print
out
a
delta
between
the
right,
indicee
and
the
redundancy,
and
then
we
clear
our
breakpoints
and
continue.
So
that's
a
you
know
pretty
much
all
there
is
to
this.
So
I'm
gonna
run
that
Tracy
program
and
a
left,
and
you
can
see
we
get
about.
4
million,
writes
and
seven
ish
million
reads
per
second
and
then
I
add
my
instrumentation
and
you
can
see.
There's
no
performance
overhead
and
doing
this.
A
A
So
this
is
great
we
can
sample
his
data
without.
We
have
very
low
performance
overhead.
However,
this
only
works
for
low
frequency
events,
so
sampling
at
a
low
frequency.
If
I
use
the
same
technique
to
instrument
every
single
read
and
write
operation
that
the
ring
buffer
is
doing,
you
can
see
the
performance
overhead
of
that
so
same
performance
as
before.
We
then
run
the
trace
program
and
it
turns
seven
million
operations
a
second
into
several
hundred,
so
this
isn't
suitable
for
doing
high
frequency
sampling.
A
So
why
is
that?
Well
the
controller
process
that
I
have
driving
the
debugger
that
has
a
lot
of
I/o
to
share
data
between
the
debugger
and
the
controller.
In
addition,
there's
a
lot
of
overhead
and
jumping
in
out
of
the
kernel
there's
context
switches
whenever
you
jump
in
out
of
the
kernel,
so
that
contributes
to
the
overhead.
So
is
there
a
better
way
to
do
this?
What
if
we
could
move
all
this
user
land
code,
doing
I/o
and
doing
context
switches,
there's
no
rule
necessity
for
that
to
live
in
user
land?
A
What
if
we
could
just
move
all
that
into
the
kernel
itself,
then
you
have
a
traci
just
hitting
trap
instructions
jumping
into
the
kernel.
The
kernel
does
some
work
to
instrument
the
code
and
then
jumps
back
into
the
tray,
see
that
would
be
ideal,
so
this
is
actually
something
that
a
technology
called
BPF
gonna
accomplish
for
us.
So
what
is
BPF
just
real
briefly.
Bpf
is
a
custom
instruction
set
that
you
can
write
in
and
build
programs
that
compile
to
it,
and
it
allows
you
to
inject
programs
into
the
kernel.
A
The
kernel
takes
its
BPF
bytecode.
It
validates
it
make
sure
it's
safe
to
run,
and
then
it
will
compile
it
to
make
sure
it
runs
fast.
On
your
native
architecture,
one
of
the
things
you
can
do
with
the
BPI
programs
is,
you
can
attach
various
events
sources
to
it.
So
we'll
see
that
later,
BPF
was
originally
created
for
packet
filtering
in
kernel
packet
filtering.
So
that's
where
it
gets
its
name.
A
Berkeley
packet
filter
and,
if
you
ever
use
TCP
dump
the
expression
that
you
give
TCP
dump
actually
compiles
into
BPF
bytecode,
so
you're
running
a
BPI
program
whenever
you
use
TCP
Dom.
So
what
can
you
BPI
program
do?
Well,
you
do
a
lot
of
things
that
you
would
expect
out
of
a
normal
instruction
set
one
of
the
things.
One
of
limitations
is
around
memory
access.
You
don't
have
full
access
to
the
kernel
memory,
one
of
the
capabilities
that
BPF
that's
really
important.
A
Is
these
helper
functions
so
the
kernel
exposes
of
various
amount
of
functions
that
can
do
really
useful
things
that
are
useful
for
tracing
so
reading
stacks
from
kernel
or
user
land
aggregating
data
and
in
kernel
data
structures,
manipulating
packets,
there's
a
whole
list
of
capabilities
that
you
can
call
into
to
do
useful
things.
There's
hundreds
of
them,
so
I
didn't
list
them
all
here,
but
there's
a
lot
of
useful
functionality
there.
So
what
can
be
PF
programs?
Not
to
so
your
program
must
have
finite
execution.
A
That's
a
big
big
one
like
you
can't
go
into
infinite
loops.
I
can't
go
into
loops
at
all
because
of
the
probability
that
you
might
go
into
infinite
loop,
so
you
can't
do
anything.
They
can
hang
the
kernel.
You
can't
acquire
locks
that
could
potentially
hang
the
kernel
like
I
mentioned
before.
Access
to
memory
is
a
little
bit
restricted,
so
yeah
there's
some
restrictions
on
what
you
can
do.
A
It
do
BPF
program
just
important
to
keep
in
mind
so
here's
the
kind
of
example
of
what
we
want
to
achieve
with
our
new
program,
so
we
have
our
traci
and
that
we
have
a
little
user
space
program
called
the
tracer
that
it's
doing
a
little
tiny
bit
of
work
and
user
land,
but
not
nearly
as
much
as
previously.
So
what
happens
this
situation?
Well?
The
tracer
is
going
to
compile
our
BPF
program
for
us.
It's
then
going
to
hand
that,
off
to
the
colonel,
the
colonel
is
going
to
verify
and
compile
it's.
A
The
BPI
program
is
then
going
to
be
attached
to
event
sources,
so
the
tracer
will
tell
the
colonel
hey
this
program.
I
just
gave
you
attach
these
event
sources
to
it
when
those
events
occur.
So,
for
instance,
the
Tracie
hits
a
trap
instruction.
Then
the
BPA
program
will
get
called
and
it'll
handle
the
event.
And
then
the
program
can
officer
optionally
write
to
some
internal
memory
that
then
gets
shared
with
the
tracer.
A
So
I
mentioned
the
event
sources.
The
ones
that
we're
going
to
be
discussing
today
are
you
probes,
primarily
there's
also
something
called
USCT
probes.
These
are
statically
defined
probes
on
Linux
they're
actually
implemented
using
new
probes.
So
it's
kind
of
using
the
same
technology
there,
but
yeah
there's
other
event:
sources.
Some
kernel
dynamic
and
static,
instrumentation,
networking
stuff,
so
yeah
there's
various
events
sources.
You
can
touch
your
program
to
do
so.
Yeah,
you
probes,
is
really
awesome.
It's
like
one
of
my
favorite
technologies.
A
It
allows
you
to
trace
any
instruction
and
use
your
land
with
a
lot
less
overhead
than
Petrie's.
So
here's
an
example
of
how
you
probes
works.
So
you
have
your
program
down
at
the
bottom
here.
Each
one
of
these
blue
boxes
is
a
instruction,
so
a
variable
with
instructions,
something
like
x86
64.
So
in
the
middle
here
we
want
to
instrument
the
instruction
in
the
middle.
So
what
the
kernel
will
do
will
copy
that
instruction
into
a
buffer
and
kernel
memory
called
the
single
step
out
of
line
buffer.
A
It
will
then
write
a
trap
instruction
over
on
over
on
top
of
that
instruction,
and
then,
when
that
trap
hits,
the
kernel
will
take
over
run
the
handler
that
we
register
with
it
after
the
handler
PPF
code
is
done
running
then
it'll
single-step,
the
one
instruction
out
of
that
buffer
and
then
after
that's
done,
you'll
then
jump
into
the
programs
text
just
immediately
after
that
instruction.
So
we
were
able
to
add
this
instrumentation
without
our
program,
knowing
every
instruction
that
our
program
wanted
to
execute
was
able
to
execute.
A
So
that's
you,
probes,
USCT
I,
mentioned
for
their
static
trace
points
that
are
defined
by
the
developer.
At
you
know
the
defined
in
source
code
I
like
to
think
of
them,
as
kind
of
like
a
tracing
API
you're,
exposing
these
kind
of
landmarks
that
folks
can
then
instrument
later
in
case.
They
are
interested
in
debugging
problems
with
your
software,
so
they're
really
useful.
A
You
can
report
arbitrary
data
when
the
USCT
probe
fires
so
they're
kind
of
like
structured
login.
You
can
touch
whatever
data
you
want
to
them,
but
you're
not
paying
the
performance
costs
that
you
would
normally
pay
with
logging.
That's
enabled
you
only
pay
the
pro
ins
costs
when
you
enable
these
probes
there's
some
probes
that
are
built
into
language
runtimes,
so
python
knowed,
Ruby
Java.
They
all
have
USC
probes
that
you
can
either
compile
in
or
enable,
and
those
ECT
probes
will
allow
you
to
instrument,
functions
and
methods
that
are
being
called
in
those
runtimes.
A
So
you
can
get
access
to
language
native
functions.
So
how
do
we
write
these
BPF
programs?
Well,
there's
two
technologies
that
I've
used
to
write
these
programs.
The
first
is
BCC
BCC
has
been
around
for
a
while,
and
it's
really
mature
has
a
lot
of
capability,
but
it
has
a
little
bit
of
a
failure
around
complexity
and
writing.
These
programs,
BPF
trace
on
the
other
hands,
is
very
new
is
released
a
few
months
ago.
A
It
exposes
like
a
really
nice
language.
Everyone
at
eclis
simplifies
writing
these
programs,
but
it
doesn't
have
all
the
capabilities
that
bcc
has
so
I
think
currently
there's
a
place
for
both
of
these
tools.
So
bcc
is
that
BPF
compiler
collection,
so
you
write
your
BPI
program
in
a
version
of
C
and
BCC
will
compile
that
for
you
and
hand
it
off
to
the
kernel.
It
also
assists
with
you
attaching
events
and
interacting
with
kernel
memory.
It's
implemented
as
a
shared
library,
so
BCSO,
there's
bindings
and
Python
Lua
and
third-party
go
bindings
exists.
A
A
So
let's
do
a?
U
probe
demo,
so
we're
going
to
use
BCC
to
instrument
some
code
using?
U
probes.
So
this
is
the
code
that
we
want
instrument.
There's
no
instrumentation
added
to
this
in
advance,
just
an
infant
for
loop,
calling
a
function
f
and
then
that
function
is
just
doing
a
atomic
increment
on
a
counter.
So
the
BCC
code
that
we
had
write
is
is
here
so
up
at
the
top.
We
have
a
C
program,
that's
going
to
increment
a
counter
in
kernel
memory
and
then
below.
A
We
just
have
some
code,
that
kind
of
wires
that
up
to
the
main
dot
F
function,
which
is
what
we're
interested
in
instrumenting.
We
then
have
accounts,
which
we
use
to
then
read
kernel
memory
and
display
that.
So
this
is
what
that
looks
like
when
it
runs
so
we're
getting
around
125
million
operations,
a
second
on
a
single
thread,
and
then,
when
we
run
our
instrumentation
code,
you
can
see
that
drops
to
about
three
hundred
and
ninety
thousand
in
operations
a
second.
So
you
know
it
definitely
has
an
overhead.
You
probes
are
not
free.
A
Well,
one
thing
to
notice
is
when
you
detach
from
the
instrumentation,
you
actually
regain
all
that
performance.
The
other
thing
to
keep
in
mind
is
this
is
on
a
single
thread,
so
a
lot
of
workloads,
I,
would
venture
to
guess,
aren't
doing
more
than
10
to
20
thousand
operations.
Second,
on
a
single
thread:
if
you
are,
then
that's
when
you
want
to
consider
using
something
other
than
you
probes,
maybe
some
user
space
instrumentation,
but
if
you're
doing
less
than
that,
then
you
probes
as
fine
to
use.
A
So
that's
you
probes,
see
a
demo
with
us
CT.
So
we
have
just
some
go
code
here
that
is
instrumented
using
us
CT.
So
the
main
important
bits
is
that
the
function
now
is
instead
of
doing
a
counter.
It's
actually
doing
some
work
so
reaching
out
to
Google
and
just
doing
a
random
search,
and
then,
when
we
enter
the
function,
we
fire
a
entry
probe
and
we
exits.
We
fire
an
exit
probe.
So
this
will
allow
us
to
measure
the
overall
latency
of
communicating
with
Google.
A
So
I'm
not
gonna,
go
in
detail
of
how
this
works,
but
we
basically
have
just
two
functions
that
keep
track
of
the
timestamps
for
when
these
events
occur
and
correlate
them
and
store
them
in
a
histogram.
That's
the
BP
BCC
program,
the
BPF
part
of
it.
This
is
the
Python
part,
which
kind
of
wires
everything
together
and
displays
the
data,
that's
very
similar
to
what
we
saw
before
with
you
probes,
when
we
hit
control-c
here,
just
displays
out
the
histogram.
A
So
this
is
that
running.
So,
on
the
left
hand,
side,
you
have
all
the
queries
going
to
Google
and
then
we
have
our
instrumentation
running
and
when
I
hit
control.
C
here
you'll
be
able
to
see
the
distribution
of
latency
is
for
all
those
requests.
So
yeah
that's
a
kind
of
an
example
of
using
yooper
or
using
USCT
to
instrument
your
code
in
advance,
so
all
that
was
using
BCC.
So
what
about
BPF
trace
I
mentioned?
That's
like
the
new
tool,
so
yeah
it
dramatically
simplifies
writing
these
programs.
A
A
Here's
the
example
of
doing
the
latency
calculation
and
printing
out
a
histogram,
just
a
handful
lines
of
code
versus
you
know:
20
30
lines
of
code.
So
a
lot
less
complexity.
It's
a
lot
easier
to
use
so
to
just
take
a
step
back
when
I
first
started
getting
into
Linux
tracing
the
field
was
like
pretty
messy.
There's
a
lot
of
different
options:
a
lot
of
different
technologies
to
look
into
investigate
a
lot
of
different
tracers.
A
It's
kind
of
intimidating
a
little
bit
and
I
feel
what
BCC
and
BPF
trace.
They
dramatically
simplify
the
landscape
of
tools.
You
get
all
the
capabilities.
You
really
want
with
just
a
few.
You
know
a
few
tools
to
use
so
similar
to
how
docker
took
the
kernel
technologies
of
cgroups
namespaces
in
second
I
feel
like
BPF.
Trace
has
a
potential
to
be
the
one
tool
that
really
exposes
and
makes
generally
usable
technologies
like
you,
probes,
k,
probes,
BPF,
trace
points,
etc.
A
So
yeah
we
kind
of
got
what
we
wanted.
We
are
able
to
intercept
any
point
of
execution
without
restarting
the
process
with
as
low
overhead
as
possible.
We're
able
to
read
it
from
memory
and
registers,
collect
data
across
multiple
processes
and
the
kernel
and
we're
able
to
do
it
all
safely.
So
that's
great!
So
how
do
we
put
this
into
practice?
We
need
to
deploy
a
container
to
probe
our
applications,
so
I
actually
put
together
a
repo
that
can
assist
you.
If
you
want
to
play
around
with
it.
A
It's
called
towel,
just
Jason,
King,
/,
towel
and
github,
so
that
composes,
a
composed
of
a
docker
image,
daemon
set
and
a
coop
CTL
plugin,
so
I'll
go
into
what
those
are
used
for.
So
first,
the
daemon
set
some
things
to
note
about
it
is
that
it
runs
your
this
container
as
root.
So
obviously,
like
you
know,
that
comes
with
a
lot
of
caveats
running
as
root,
it's
price,
something
that
you
want
to
make
sure
you're
doing
correctly.
A
Additionally,
we
share
the
host
pit
namespace,
that's
just
so
that
we
don't
have
to
do
pit
mapping
and
enter
into
other
paid
namespaces.
We
also
volume
mount
a
whole
bunch
of
stuff
in
order
to
have
easy
access
to
docker
and
to
have
access
to
file
system
in
the
containers.
So
this
gives
you
a
lot
of
power,
but
with
a
great
power
comes
great
responsibility
and
you
have
to
use
it
wisely.
A
This
is
an
example
of
using
the
towel,
so
I'm
gonna
use
the
first
I'm
going
to
get
the
pods
just
to
show
that
nginx
is
running
here
and
then
I'm
gonna
use
the
cube.
Ctl
plugin
towel
apply
just
to
apply
the
daemon
set.
So
now
that's
running,
so
each
node
gets
its
own
own
pod
and
then
I
use
coop,
CTL
Tao
exec,
which
we'll
check
into
the
towel
pod.
That's
co-located
with
the
same
pod
that
I
give
it
so
I
gave
it
a
label
selector
there.
A
You
can
then
see
that
it
put
me
in
the
directory
where
the
container
is
mounted
on
the
host.
So
that's
useful
I
then
use
open,
snoop,
just
a
snoop
in
on
what
engine
is
opening
and
then
every
time
I
curl
it
you
can
see
it's
hitting
or
it's
opening
up
the
index.html
file,
see
ya
and
then
I
used,
read
L
feel
just
to
dump
out
the
symbol.
Table
for
nginx
was
interested
in
instrumenting.
The
request
response
cycle
for
nginx,
so
yeah
I
found
a
symbol
here.
Http
process
requests.
They
look
pretty
interesting.
A
A
So
then,
whenever
I
curl
it,
you
can
see
the
hello
come
message
shows
up,
so
that's
not
terribly
useful
per
se.
So
let's
do
something
a
little
bit
useful,
so
say:
I
want
to
measure
the
latency
of
that
one
function,
not
the
entire
request
response
cycle,
but
I'm
interested
just
in
measuring
that
one
function.
A
So
I
have
a
program
here
that
does
that
just
a
few
lines
of
code
and
then
I'm
gonna
apply
some
load
on
to
nginx,
using
Siege
and
then,
whenever
I
control
C
this,
you
can
kind
of
see
the
distribution
of
requests
that
came
into
that
one
function,
not
necessarily
the
entire
request
response
cycle
and
then
afterwards
you
can
just
use
coop
C
Hotel
des
bleed
to
clean
up
after
yourself.
You
don't
want
these
pods
running
in
a
production
environment
for
longer
than
they
need
to
be
cool.
A
So,
like
I
mentioned,
this
runs
this
route
so
make
sure
to
delete
the
daemon
said
when
it's
no
longer
needed
also
put
the
daemon
set
in
a
namespace,
that's
restricted.
So
you
can
use
just
this
role
here.
There's
a
resource,
pods
exec
that
allows
you
to
just
control.
Who
has
access
to
exec
on
to
that
pod?
So
that's
useful
if
you
want
to
control
that
so
how'd
it
get
started,
I
have
a
getting
started
guide
up
in
the
top
of
the
readme
of
the
towel
repo.