►
From YouTube: Debugging Live Applications the Kubernetes Way: From a Sidecar - Joe Elliott, Grafana Labs
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Debugging Live Applications the Kubernetes Way: From a Sidecar - Joe Elliott, Grafana Labs
Linux features a number of powerful debugging tools that give us insight into how our applications run in a real environment. Through live demonstration this session will present a straightforward way to begin debugging applications in a Kubernetes native way: from a sidecar. Sidecars offer a low impact way of profiling applications without installing packages or making messy changes to your nodes. The techniques demonstrated will include recording LTTng events, cpu profiling, generating Flame Graphs and dynamic tracing with BCC. These techniques will be performed against a .NET Core sample application, but that will not be the focus of the session.
https://sched.co/UaXU
A
A
This
presentation
is
kind
of
information,
dense,
there's
a
lot
of
slides
that
have
a
lot
of
details
on
them.
I'd
recommend,
instead
of
maybe
trying
to
take
pictures
or
jot
things
down,
they
can
get
the
slides
both
in
the
coop
con
schedule,
as
well
as
from
this
github
repo,
and
it
might
help
you
kind
of
keep
track
of
sections
that
you
are
review
it
later
to
understand.
Maybe
how
to
do
some
of
these
techniques.
A
Also,
this
repo
is
built
around
net
core
I'm,
not
going
to
be
doing
a
lot
of
net
core
specific
things
or
trying
not
to
I'll
mention
it
some.
But
these
all
these
techniques
are
about
to
look
at
our
kind
of
framework
agnostic
there
Linux
Pacific
and
they
can
be
run
against
all
kinds
of
runtimes
and
frameworks,
not
just
on
that
core.
A
Ok,
so
what
are
we
talking
about
today?
This
presentation
is
primarily
divided
into
two
major
parts
in
the
first
part,
we're
going
to
talk
about
kind
of
building,
a
toolset
container,
a
sidecar
that
we
can
pack
full
of
a
lot
of
tools
that
we're
gonna
use
to
debug
our
applications.
We're
talk
about
some
of
the
challenges
there,
some
of
the
things
that
I
have
come
across
and
I've
been
working
on
my
toolset
container
for
net
core
and
then
the
second
half
of
the
presentation.
A
We
are
going
to
kind
of
demo
three
different
techniques,
we're
gonna,
look
at
perf
for
CPU
profiling,
Ltd
ng,
for
user
space,
static,
trace
points
and
BCC
for
you,
probes,
dynamic
tracing,
like
I,
said
these
three
techniques
can
be
performed
against
all
kinds
of
runtimes
and
frameworks.
So
don't
think
this
is
net
core
specific
yeah.
Also
we're
going
to
do
this
in
a
way
that
we
can
do
in
production.
The
idea
here
is
we're
going
to
increase
knowledge
of
our
applications
in
the
production
environment,
whether
they're
going
through
some
negative
moment
like
they're.
A
You
know
they're
not
responding
to
requests
correctly,
we're
trying
to
figure
that
out
or
they're
performing
well,
and
we
just
want
to
understand
more
about
what
it
means
to
be
performing
it
load
or
at
scale.
We
can
use
these
techniques
to
learn
more
about
our
application
under
those
circumstances.
So
all
these
are
very
low
impact
and
all
of
these
are
designed
to
be
run
at
scale
under
load,
I'm
and
then
finally
kind
of
a
theme
of
the
presentation
here.
A
If
you,
if
you
are
interested
in
this
field,
I
highly
recommend
his
blog
also
Sacha
Goldstein
did
a
lot
of
work
on
dynamic
tracing
for
net
Corwin.
Oh
and
my
work
in
net
core
is
kind
of
an
extension
of
that
to
get
that
to
work,
I
mean
there's
two
other
really
good
links
there.
If
you're
kind
of
new
to
this
world,
those
links
are
really
good
overviews
of
just
Linux
debugging
systems
in
general,
it's
kind
of
a
fractured
world.
A
If
there's
lots
of
tools,
there's
lots
of
terms
and
technologies,
and
when
you
first
get
involved
in
it,
it
can
be
confusing
about,
what's
happening,
all
the
different
pieces.
These
are
two
kind
of
good
overviews
that
touch
on
a
lot
of
the
major
areas
and
ideas
and
what's
going
on
and
then
finally,
my
repo
like
I,
said
the
slides.
Are
there
so
like
this
slide?
A
A
When
I
first
began
performing
some
of
this,
we
had
some
application
issues
in
production
and
I
would
SSH
into
a
node.
It
was
kind
of
a
naive
approach
here,
like
I
I,
wanted
to
gain
more
information
about
a
process
that
was
running
on
my
linux
instance.
Sorry
ssh,
din
and
iran.
Perf
I
pulled
some
tools.
A
I
was
trying
some
of
these
techniques
and
I
and
I
did
it
helped
I
learned
a
lot
about
the
application
of
what
it
was
doing,
but
when
I
was
done
with
this
I
kind
of
had
a
mess
like
got
all
these
tools
installed.
Suddenly
I
had
one
node
with
different
packages
installed,
and
you
know
some
garbage
stuff
I'd
left
around
like
little
notes
or,
like
you
know,
files
that
output
and
I
didn't
like
that.
I
didn't
like
that.
I
was
kind
of
taking
this
world
where
I
had
a
bunch
of
consistent
notes.
A
A
The
sidecar
is
a
really
neat
technique,
because
you
can
put
all
of
your
tools
and
those
are
all
of
your
tools
in
this
container,
that
you
run
alongside
your
application
container,
and
when
you
delete
the
pod,
then
it
all
goes
away
and
it
cleans
up
after
itself,
and
so
you
don't
have
this
kind
of
concern
about
impacting
the
node
and
maybe
what
I
could
do
or
an
impact
I
could
have
on
other
processes
on
the
node
and
it
doesn't
require
hosts
access.
So
every
organization
is
kind
of
different.
A
You
know
who's
allowed
on
a
node
or
not.
Maybe
everybody
is
maybe
just
your
infrastructure
people,
maybe
your
devs
are,
or
maybe
they're
not,
but
the
people
who
would
normally
be
doing
this
kind
of
analysis
are
probably
developers
and,
if
they're
not
allowed
on
the
node,
then
this
kind
of
sidecar
technique
will
open
up
some
of
these
tools.
To
your
developers
give
them
some
observability
give
them
some
opportunity
to
see
their
applications
better
without
having
to
give
them
SSH
access
directly
into
the
node.
A
It's
also
easy
easy
we'll
talk
about
why
it's
not
easy
in
a
second,
but
it's
easy
to
build
a
toolset
container
that
that
supports
this,
that
that
does
the
that
has
your
debugging
tools
in
it
and
what
I
mean
by
easy
here.
Is
it
it?
It
obeys
a
lot
of
the
same
rules
that
our
application
containers
do.
You
can
iterate
on
it,
so
you
can
build
a
container
if
it
just
has
perf
and
just
does
some
CPU
profiling
and
get
that
working,
and
you
have
a
new
way
to
see
your.
A
You
have
a
new
way
to
see
your
application
and
then,
a
couple
months
later,
you
have
some
other
problem
going
on
a
person
not
helping.
So
maybe
you
learn
a
new
technique
or
a
new
tool,
and
you
keep
iterating
and
building
on
this
tool
sigh
container
over
time
and
you
can
deploy
it
and
to
anywhere.
You
want
also
like
your
application
containers.
It's
immutable.
It
can
be
dropped
in
Undead
that
can
be
dropped
into
QA.
A
It
could
be
drop
into
prod,
it
could
be
dropping
to
our
desktops,
and
it
has
this
like
portability
aspect
to
it,
and
so
it
so
it
has.
These
kind
of
neat
features
that
are
applications
do
as
well.
Also,
once
you
are
in
the
container
and
you're
kind
of
messing
around
with
things
you'll
often
find
oh
I
didn't
think
to
bring
this
tool
and
I
didn't
have
to
think
to
bring
this
this
this
thing
in
the
script
or
whatever
you
can
write
scripts
on-the-fly.
A
You
can
and
pull
new
tools,
kerlor
apt,
get
them,
and
then
again,
when
you
delete
your
pod,
it's
all
gone.
You
don't
have
to
worry
about
the
changes
you
made.
You
don't
have
to
worry
about
what
you
may
have
done
dynamically
inside
the
container.
Finally,
I
feel
like
this
also
supports
development
diversity.
A
If
you
have
a
lot
of
teams,
and
these
teams
have
different
technologies,
like
you
got
your
node
team
and
your
Python
team
and
your
net
core
team,
and
then
each
team
can
how
to
build
their
own
containers
instead
of
like
trying
to
have
a
node
and
that
node
has
tools
for
everybody.
You
can
instead
have
each
team
who's,
building
their
own
containers
to
debug
their
own
applications
and
they
have
developed
their
own
techniques
internally
and
they
can
kind
of
maintain
that
on
their
own
okay.
Why
is
this
not
easy?
What
are
some
of
the
challenge?
A
What
are
some
of
the
problems
with
trying
this
I'd
say
the
number
one
issue,
the
most
difficult
thing
I
ran
across
while
I
was
working
on.
This
was
when
you
install
some
applications
like
perf.
They
want
to
be
installed
for
the
kernel
that
you
are
installing
them
on.
There
are
kernel
specific
installs.
If
you
apt-get
perf
right
now,
it
will
be
a
version
of
perf
that
is
built
for
the
kernel
that
you
are
running.
A
A
So
some
of
the
options
there
we
can
bake
our
tooling
in
some
of
us
are
lucky
enough
and
we
were
lucky
enough
that
all
of
our
nodes
are
going
to
be
the
exact
same
kernel
version.
We
have
this
kind
of
like
very
homogeneous
architecture
to
deploy
to
and
if
you're
in
this
world,
that's
fantastic,
baked
everything
in
and
make
it
exactly
what
you
need
for
the
kernels
you're
running
and
you'll
be
in
a
good
spot.
Other
things
we
can
attempt
to
do.
A
We
can
mount
stuff
from
host
and
we're
gonna
do
that
a
little
bit
and
that's
kind
of
where
we're
gonna
cheat
some,
because
it's
kind
of
by
mounting
resources
from
the
host
we
kind
of
blur
the
line
between
using
host
access
or
not.
And
finally,
another
option
is
pulling
this.
After
deployment
you
deploy
this
and
you
try
to
pull
in
the
tools,
then
once
you're
on
the
actual
machine
and
the
kernel
is
known,
the
kernel
version
is
known
at
that
point.
It's
that
that
approach
is
primarily
how
I
have
worked
with
my
dotnet
core
container.
A
My
sidecar,
with
my
tools,
is
pulling
these
things
after
a
deployment,
and
while
it
is
kind
of
flexible
and
has
given
me
the
ability
to
deploy
to
a
lot
of
environments,
I've
gotten
it
working
in
like
AWS
cops
with
a
cop's
cluster
on
Debian
gke
abun
I've
got
it
running
locally.
Of
course,
it's
been
very
difficult
to
keep
that
working,
and
it's
been
a
lot
of
overhead
to
keep
it
working.
My
advice
would
be
to
remain
on
the
top
line.
A
Hopefully
you
can
just
bake
everything
in
it'll
work
nicely
for
you,
I'm
also
slide
carry
images
can
be
very
large.
This
kind
of
doesn't
matter,
and
it
kind
of
does
I
mean
we
don't
want
large
container
images,
but
you're
also
not
pulling
this
all
the
time.
This
isn't
a
normal
application,
you're
going
to
pull
it
at
specialized
moments
right,
like
I
did
today,
I
want
to
do
some
profiling
in
production,
so
I'm
gonna
do
it
and
that
you
do
it
that
one
time
and
a
month
later
you
do
it
again.
A
So
if
these
container
images
are
a
little
large
and
they
require
a
little
more
time
to
pull
I
think
it
doesn't
matter
as
much,
but
they
are,
they
can
get
very
large
for
sure
and
they
can't
be
added
dynamically.
They
kind
of
can,
but
they
kind
of
can't-
and
this
is
another
cheat
moment-
Cooper
may
in
a
kubernetes
native
way-
they
cannot
be
added
dynamically.
A
A
pod
is
immutable
and
you
can't
inject
a
container
at
runtime,
so
these
techniques
are
kind
of
more
useful
in
signature
situations,
where
you
have
a
known
kind
of
like
problem
and
application,
it's
in
a
minute.
You
can't
reproduce
it
anywhere
else.
It
happens
in
production
like
twice
a
day
or
three
times
a
day.
You
don't
understand
that
you
can't
reproduce
it,
and
you
know
this
is
happening,
so
you
can
then
deploy
a
version
of
that
application
with
your
sidecar
container
and
wait
for
the
issue
to
come
up
and
then
start
doing
your
profiling.
A
It
doesn't
really
work
for
like
responding
to
an
incident
if
an
incidents
in
progress
you
have
to
deploy
this
container
or
this
pod
with
the
sidecar.
Now
you
can't
actually
using
docker
inject
this
container
in
the
same
process
namespace
as
another
container,
so
it
is
possible
to
pull
this
off.
If
you
break
the
SSH
and
node
specific
rules,
there
was
a
group.
I
was
working
with
who
wanted
to
run
perf
once
an
hour,
and
so
they
set
up
a
cron
job
every
hour.
They
would
use
a
dr.
A
command
on
the
host
to
inject
the
sidecar
they'd
run
perfect
gentle
flame
graph,
offload
that
and
store
that-
and
they
kind
of
had
this
like
regular
record
of
how
their
container
was
doing
what
the
performance
was.
So
that
was
pretty
cool,
but
it
was
only
breaking
the
kubernetes
native
kind
of
rules.
A
A
If
you
set
this
to
true,
then
all
processes
and
all
containers
can
see
each
other,
which
is
a
requirement
for
kind
of
doing.
A
lot
of
this
analysis,
where,
like
your
debugging
tools,
are
gonna
reach
over
and
start
looking
at
your
application,
your
actual
application
and
we're
also
going
to
share
mounted
volumes
between
our
sidecar
and
our
application
container.
A
lot
of
these
tools
want
to
communicate
through
a
folder
through
a
directory
on
on
the
machine,
and
so
we
need
to
hook
mount
the
right.
Hosts
are
the
right
paths
I'm.
A
Sorry,
we
need
to
mount
the
mount
the
right
paths
in
a
empty
dirt
and
we
need
to
share
that
between
the
two
containers.
We'll
look
at
that
and
then
we're
gonna
mount
host
pass
and
that's
kind
of
the
the
cheat
I
was
talking
about,
so
we're
gonna
bring
in
some
resources
from
the
host
I'm
gonna
talk
about
why
I've
chosen
to
do
that
in
this
case,
I
mean.
Finally,
a
lot
of
these
techniques
are
like
asking
the
kernel
for
information
about
another
process,
and
that
certainly
is
a
security
concern
right.
A
A
First,
we
look
at
CPU
profiling.
This
is
a
great
technique
for
understanding
like
which
functions.
Your
application
is
spending
time,
and
this
is
extremely
powerful
if
you,
your
application,
is
behaving
strangely,
you
don't
understand
it
if
you
were
to,
if
you
were
to
run
this
technique
and
find
that
a
specific
function
we're
eating
up
like
60%
of
the
CPU
time,
it
was
on
the
stack
constantly
and
you
didn't
expect
that
and
that's
a
great
place
to
start
looking
at.
Maybe
why
that
should
be
true.
A
A
So
this
is
a
sampling
kind
of
kind
of
approach
when
it
does
that
it's
going
to
record
those
and
then
we're
gonna
use
a
tool
called
flame
graphs,
I
believe
Brendan,
Gregg
wrote
this
and
we're
used
flame
graphs
to
generate
this
pretty
picture
here,
I'm,
so
we're
using
peripheral
flame
graphs
over
here
on
the
side.
I
have
the
pod
spec.
Hopefully
that's
readable
like
way
back
there.
A
Hopefully,
that's
readable
if
it's
not
like
I
said
grab.
The
slides
I
knew
that
this
is
gonna,
be
a
lot
of
detail,
share
processes,
namespace
true,
that's
the
most
important
one.
It's
gonna
be
on
all
these
we're
going
to
also
mount
temp
and
we'll
look
at
that
in
a
second,
so
I
noticed,
I
have
an
empty
door
named
temp
and
I'm,
mounting
that
between
my
sidecar
and
my
application
container
and
perf
is
going
to
use
that
to
communicate
some
specific
information
and
then
I
also
have
my
security
context,
privilege,
true,
okay,
so
let's
get
lucky.
A
This
is
a
quick
script
that
is
going
to
exercise
our
application.
This
script
is
going
to
have
two
endpoints
and
wait
a
second,
so
extremely
low
load
just
to
cross.
The
second
one
is
going
to
hit
endpoint
to
calculate
a
Fibonacci
value
and
the
other
one
is
going
to
hit
endpoint
that
just
echos
the
string
back
directly
and
will
look
kind
of
a
second.
Why
I
chose
to
do
two
different
things
like
that,
so
we'll
keep
going
on
this
side
over
here
we'll
keep
our
our
our
process.
A
Our
exercise
script
on
this
side,
I
have
three
containers
so
I'm.
Sorry,
three
pods
each
pod
has
two
containers.
One
is
my
primary
application
container
and
one
of
those
is
my
toolset
sidecar.
So
we're
going
to
exact
into
our
sidecars
we're
going
to
run
some
clever
scripts
and
we're
going
to
get
some
good
information
about
our
process.
A
Yeah
so
we've
exact
into
our
sidecar
I'm,
running,
Bosch,
etc.
I've
just
kind
of
scripted.
These
out,
don't
worry
about
the
details
of
these
you
can.
This
is
nothing
spectacular
or
interesting
if
you
just
art
googling
this.
This
is
like
the
third
hit
you're
gonna
get.
So
there
is
no
secret
sauce
in
this
script
right
here,
I'm
gonna
run
the
script.
It's
going
to
use
perf
to
again
sample
the
stack
of
my
main
application.
It's
going
to
run
for
a
couple
seconds
there
five
seconds
and
then
it's
going
to
generate
a
flame
graph.
A
Also
notice
I
have
to
pass
a
PID
somewhere
on
PSX
and
I
can
see
that
my
main
application
is
number
six.
It's
a
pin,
six
dotnet
sample
net
core
after
yellow.
If
you
did
not
set
share
name
process,
namespace
true
and
you
ran
this
PSX
command,
you
would
not
be
able
to
see
your
primary
process
and
you
would
not
be
able
to
perform
these
techniques.
Okay.
So
let's
run
this
guy
and
it
work.
A
Okay,
so
I
should
have
a
little
or
a
little
guy
called
flame
graph
SVG
here
there.
It
is
right
there,
so
we've
generated
an
SVG
file
and
it
is
now
that
flame
graph.
It
is
a
record
of
our
stack
during
that
five
second
period,
two
quickly
before
we
look
at
the
actual,
you
know,
output
to
look
at
temp.
A
The
reason
we
mapped
or
with
the
reason
we
mounted
temp-
is
because
perf
wants
to
see
a
perf
map
and
temp,
and
the
perf
map
includes
a
mapping
from
a
instruction
pointer
or
a
place
in
memory
to
a
actual
function
named
Oh
yikes
yeah
that
really
fast
and
it's
gonna
use
this
to
Reese,
but
Reese
symbolic,
8r
spec.
So
you
don't
want
to
stack
with
no
symbols
right,
that's
annoying!
It's
a
bunch
of
stuff!
You
don't
understand!
A
This
is
going
to
give
nice
pretty
names
to
every
position
in
our
stack,
and
it
helps
us
understand
like
what's
going
on.
A
lot
of
different
frameworks.
Have
support
for
this
I
know
immediately
that
node
does,
for
instance,
a
lot
of
different
application
frameworks
and
platforms
have
perf
map
support
and
you
can
perform
this
analysis
with
them.
A
So
I'm
going
to
cheat
a
little
bit
and
just
open
this
flame
grab,
our
flame
graph
SVG
already
have,
instead
of
pull
that
one
from
the
container,
and
so
this
is
what
looks
like.
If
you
were
to
open
this
in
a
browser,
it
would
actually
be
interactable.
You
can
like
control,
F
and
search
it
and
you
can
kind
of
zoom
into
different
parts,
but
we're
not
going
to
get
into
all
that.
What
this
kind
of
means
is.
This
is
our
stack
kind
of
smeared
out
over
time
here.
A
I
can
see
this
kind
of
column
here,
and
this
is
maybe
30
or
so
percent.
So
that
means
like
30
percent
of
the
time
the
stack
was
sampled.
It
was
ed
in
this
position
in
the
stack
and
it
helps
us
understand
what
our
application
is
kind
of
using
its
time
to
do.
I
dunno,
just
because
I've
spent
time
doing
this,
that
this
column
way
over
here
on
the
right.
A
That
goes
real
high
is
my
actual
HTTP
pipeline,
and
that's
where
my
requests
are
being
serviced
if
I
were
to
see
this
thing,
take
like
99%
of
the
whole
thing,
and
some
some
call
at
the
very
top
was
a
huge
amount
of
time.
I
might
want
to
investigate
that
call.
Perhaps
that
call
is
taking
way
too
long
and
I
need
to
understand
why.
A
A
A
I
want
to
record,
but
static,
trace
points
are
frameworks
that
are
designed
for
very,
very
high
volume,
things
that
you
can
write
in
the
critical
path,
and
you
can
record
this
information
and
have
low
to
no
impact
on
your
application
I'm
if
you're
to
log
like
a
million
times
a
second,
you
would
crush
your
application
and
spend
all
this
time
logging.
You
want
to
do
that
right,
but
static,
trace
points
are
designed
for
these
very
high
volume
kind
of
events.
A
These
high
volume
logging
events
it
does
require
pre
instrumentation,
though,
so
you
have
to
spend
some
time.
You
know
building
these
into
your
application.
I'm
going
to
use
ltte,
ng
and
babel
trace,
primarily
LT
and
G
is
a
framework
for
collecting,
trace
points
and
specifically
or
we're
going
to
be
doing
user
space
trace
points.
So
our
application
in
user
space
is
going
to
create
events
and
pass
it
to
our
LT
ng
daemon
in
user
space.
This
particular
this
particular
technique
doesn't
require
kernel
access
at
all
and
you'll
notice.
A
There's
no
security
context
privilege
for
this
one,
because
it's
just
application,
data
and
user
space
being
passed
back
and
forth
babel
trace
is
a
good
tool
to
kind
of
dump.
The
results
of
this.
The
events
and
trace
compass
is
a
cool
UI,
but
we
won't
get
into
that.
So
you
can
actually
pull
all
these
events
and
kind
of
nice
UI
on
your
desktop
and
kind
of
work
with
that
with
trace
compass,
so
share
process
namespace.
True,
it's
gonna
be
on
all
of
them.
A
Okay,
so
this
is
the
script
gonna
run
this
time,
so
these
are
a
bunch
of
Ltd
and
G
and
command
or
gonna.
Like
start
a
session,
we're
gonna
give
it
specific
information
about
the
application.
We
want
it
to
record
events
from
and
then
we're
gonna
collect
those
events
for
a
few
seconds,
then
we're
going
to
shut
it
all
down
and
we're
gonna
look
at
the
events
that
came
out
of
it,
like
the
other
one
I
need
the
PID.
This
will
be
6
as
well,
probably
yep.
There's.
A
The
reason
we're
mounting
VAR
run
Ltd
ng
is
because
Ltd
and
G
wants
to
communicate
through
a
UNIX
socket
and
by
default
it
puts
it
in
far
run.
L
ki
ng,
so
for
this
to
work
at
all,
both
of
the
applications
need
access
to
the
same
folder.
Essentially,
and
so
that's
why
we
mounted
this
one
in
an
empty
day
and
then
I
can
run
Babel,
trace
and
kind
of.
Just
you
know.
Throw
these
events
out.
A
This
is
obvious,
is
not
super
useful
right
like
what
can
I
learn
from
just
blowing
those
events
out
to
the
console.
Not
a
lot
but
Babel
trace,
does
support
Python
extensions,
so
you
can
kind
of
write,
Python,
scripts
too,
and
now
analyze
this
and
then
also
like
I
said
you
can
use
trace
compass,
so
you
could
bring
all
these
back
to
your
desktop.
You
can
use
this
trace
compass
utility
to
kind
of
see
all
kinds
of
interesting
metrics
about
your
events
like
how
many
per
event
type
occurred
like
when
did
they
occur.
A
Also
dotnet
core
specifically
has
a
pre
instrumentation
for
LT
dmg,
which
is
why
I'm
using
it
the
framework
is,
the
framework
itself
is
generates
healthy,
TMT
events.
In
fact,
if
you
look
at
these,
these
are
not
application
events.
These
are
like
garbage
collection
thread,
creation,
thread,
destruction
for
thread
weights
or
deadlocks.
It'll
show
up,
and
these.
A
Finally,
dynamic
tracing
is
a
the
newest
of
all
these
techniques.
Dynamic
tracing
is
a
really
cool
trick
to
have
the
kernel
call
back
to
you
and
tell
you
at
any
point
when
a
function
is
called
an
application.
It's
particularly
interesting
because
it
requires
no
instrumentation,
so
I
don't
have
to
do
any
work.
I
can
take
a
binary
that
I
compiled
a
month
ago
and
I
can
attach
myself
to
any
function
in
that
application
know
when
it
was
called
I
know
the
parameters
that
are
passed
to
it
and
know
that
the
return
value
from
it.
A
This
is
really
powerful.
What
we're
about
to
see
is
just
the
most
basic
demonstration
that
does
exactly
what
I
said,
but
if
you
do
get
BCC
working
and
become
invested
in
BCC
or
BPF
the
bottom
link,
there
has
all
kinds
of
tools
out
of
the
box
that
are
gonna.
Let
you
like
analyze,
network,
throughput,
analyze,
disk
throughput
and
understand
other
details
about
your
application
than
just
when
a
function
is
called.
A
This
one
is
crazy,
all
kinds
of
stuff
you
got
to
do
here.
This
is
the
hardest
one
to
pull
off
at
the
top
I'm
running.
This
script
called
run
native
and
then
I'm
sharing
all
this
stuff
and
that's
all
related
to
a
specific
requirement
of
you
probes,
which
is
what
BCC
is
using
and
that
specific
requirement,
and
unfortunately,
this
specific
parm
is
not
met
by
all
frameworks.
A
Okay,
also
we're
mounting
this
is
this
is
the
cheap
mode
here
we're
mounting
user
source
and
Lib
modules
and
that's
to
get
Linux
headers
so
to
compile
compile
these
BPF
programs,
we
need
Linux
headers
and
it
can
be
difficult
to
pull
the
exact
Linux
headers
that
your
kernel
has
compiled
into
a
container
running
on.
The
kernel
are
running
it
I'm,
sorry
running
in
kubernetes,
so
I'm
cheating
a
bit
here,
I'm
actually
mounting
the
Linux
headers
from
the
host
itself,
and
that
makes
this
just
work
really
nicely.
A
A
Cool,
so
the
first
thing
we
have
to
do
here
is
regenerate
these
offsets
and
I'm
going
to
get
into
that,
because
that's
very
specific,
but
you
have
to
know,
is
the
exact
offset
that
your
function
call
begins
in
in
the
binary
as
it
sits
on
the
disk,
and
so
that
can
be
very
difficult
to
learn.
You'll
find
when
you
start
digging
into
it.
Some
some
frameworks
nicely
give
this
to
you.
A
These
in
this
case,
specifically
we're
gonna
trace
that
one,
a
5o
is
the
offset
of
the
fibonacci
the
function
that
calculates
the
Fibonacci
values,
so
we're
gonna.
First
we're
gonna
trace
that
function.
Every
time
it's
got
called
we're
gonna
know
the
endured
integer
parameter
that
was
passed
in
then
we're
gonna
trace
the
return
value.
A
So,
every
time
it's
called
we're
gonna
the
integer
value
that
was
returned
on
that
second
line
and
in
the
final
tracing
the
final
trace
line
is
actually
going
to
trace
the
the
function
that
is
calculating
the
echo
value
so
calculating
the
echo
value
is
literally
returning
the
parameters
passed
down,
but
we're
gonna
trace
that
function.
We're
gonna
see
the
string
that
comes
out
of
it.
If
we're
looking.
A
A
So
cancel
that
one
we're
gonna
start
the
next
one,
this
one
traces,
the
return
value,
so
in
this
case
you'll
actually
see
it
match
up
exactly
because
that's
the
response
from
the
HTTP
endpoint.
So
every
time
you
see
this
one
it
again,
it's
the
same
function
being
hit,
but
this
time
we're
actually
tracing
the
return
value
from
that
function.
Dynamically
and
then
to
be
clear.
There's
no
special
instrumentation
on
this
application.
I
didn't
write
any
special
code,
I
just
attached
to
a
random
function
in
the
application.
A
Finally,
we're
going
to
do
strings
and
this
kind
of
shows
off
the
power
of
BP
F.
So
what
BP
F
is
is
a
technology
to
write
a
subset
of
C
that
is
compiled
to
a
bytecode
that
is
run
in
the
kernel,
so
it
makes
it
very
fast
and
it
also
makes
it
it
makes
it
very
fast
and
very
performant,
but
it's
only
a
subset
of
C
can
be
kind
of
rough
to
work
with
sometimes
in
this
case
we're
attaching
it
to
a
you
probe
and
we
are
able
to
take
the
dotnet
core
internal
string.
A
Representation
which
is
I,
think
is
utf-16
a
Mauthausen
sure
and
then
extract
that
string
and
displayed
on
the
console.
Perf
also
supports
dynamic
tracing,
but
it
only
does
it
and
it
supports
strings,
but
it
only
supports
a
null
terminated
strings.
So
you
can't
actually
do
this
with
perf,
which
is
kind
of
it
shows
off
the
flexibility
of
BPF.
Basically,
every
time
these
functions
are
called
I'm
running
a
little
program
that
I
wrote
I'm
telling
the
kernel.
Please
run
this
program
every
time
as
it
hits,
and
you
can
do
all
kinds
of
neat
things
with
that.
A
How
am
I
doing
on
time?
Oh
pretty
good,
cool,
okay,
yeah!
Thank
you
all
for
listening
to
me.
That
was
cool.
If
you
have
any
questions.
Let's
do
that
part.
That's
Gotham!
If
you
know
Gotham
he's
really
cool,
he
works
for
get
low
and
if
you
don't
get
to
your
questions
now,
please
visit
me
at
any
moment
at
the
booth
I'll
be
over
there
a
lot
tomorrow
afternoon.
Specifically,
you
can
ask
me
questions
in
T.
A
B
A
Can
but
you
have
to
cheat
and
do
it
from
docker
on
the
on
the
node,
so
docker
has
a
specific
command
line,
parameter
of
it
says
inject
into
a
process
namespace
of
an
existing
container.
So
you
can
do
it
if
you
let
yourself
break
the
rules
and
run
from
the
node,
but
other
than
that.
Kubernetes
pods
are
immutable
and
you
can't.
Although
I
heard
somebody
say
that
they're
kind
of
working
on
something
I
don't
swear
to
that.
I
didn't
say
that
nevermind.