►
From YouTube: When the Logs Just Don’t Cut It: Root-Causing Incidents Without Re-Deploying Prod- Phillip Kuznetsov
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe in Amsterdam, The Netherlands from April 17-21, 2023. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
When the Logs Just Don’t Cut It: Root-Causing Incidents Without Re-Deploying Prod - Phillip Kuznetsov, New Relic
Speakers: Phillip Kuznetsov
We’ve all been there: your pod is crash-looping, you check the logs and you realize you forgot to log something important - now you’re unable to figure out what went wrong. You try to reproduce the problem locally with no luck: it only seems to happen in production. What do you do? Do you re-deploy to production with more print statements? You could burn hours doing that while you risk more problems. What if you could instead get that same data without the headache of restarting prod? In this talk, I’ll show you how to magically collect this data using bpftrace. Bpftrace lets you capture lots of useful data (function arguments, return values, latencies of individual functions - just to name a few) without re-deploying pods. Bpftrace is very powerful, but can be complex to work with, especially in multi-node environments like a Kubernetes cluster. I’ll show you how to cut past these problems by walking through a demo incident. I’ll show you some tips and tricks for working with bpftrace on Kubernetes, including how to leverage Pixie to easily deploy and collect data from bpftrace scripts.
A
Hello,
everyone,
my
name,
is
Philip
kuznetsov
and
I
will
be
presenting
a
root,
causing
incidents
without
redeploying
prod
a
little
bit
about
me.
I
am
a
software
engineer
at
New
Relic,
where
I
spend
most
of
my
time
building
pixie,
let's
just
jump
right
into
it.
So
here's
the
situation.
Imagine
we're
working
at
an
e-commerce
company
called
online
boutique
we're
selling
a
bunch
of
hip
trendy
items,
and
things
have
been
chugging
on
chugging
along
pretty
well
recently,
we've
had
no
problems.
Code
is
shipping
everything's
great.
A
Until
today
the
front-end
service
is
panicking.
Something
is
happening
there
that
we
don't
really
know.
What's
going
on
we
what's
worse.
Is
we
haven't
released
code
to
the
front-end
service
in
weeks,
so
there's
no
way
that
any
changes
in
the
front-end
service
are
responsible.
This
is
a
classic
problem.
In
microservices
you
have
dependency
trees
across
multiple
different
services
that
when
one
thing
changes
in
one
service
causes
a
break
in
another
service.
A
A
So
we're
looking
at
the
logs-
and
we
see
this
one
log
with
the
Panic
that
says
one
of
the
specified
money
values
is
invalid.
It's
a
bit
weird
when
our
money
values
is
weird
Okay.
We
keep
seeing
this
error
and
it
keeps
getting
returned
by
this
sum
function
before
I
dive
a
little
bit
deeper
into
what
the
sum
function
is.
You
should
understand
why
we
need
to
have
a
sum
function
in
the
first
place.
Why
don't?
We
just
add
these
together.
The
way
money
is
represented
inside
of
our
service.
Let
me
jump
ahead.
A
A
A
Unfortunately,
it's
not
that
easy
to
tell
what
happened
in
our
environment
like,
ideally,
we
just
go
and
add
a
log
in
here
and
just
print
it
out,
but
in
many
different
environments,
especially
production
environments,
that's
not
really
an
option.
We
might
try
to
reproduce
this
locally,
but
sometimes
a
bug
doesn't
really
appear
locally.
It's
hard
to
reproduce
and
again
we're
searching
for
that
root.
Cause
here,
sometimes
deploying
to
probably
just
takes
too
long.
A
There
is
a
situation
where
you
might
just
go
and
add
this
log
in
and
hope
that
you
kind
of
discover
what's
going
wrong.
I
guess
the
severity
of
this
incident
is
rather
low,
so
it
would
be
okay
to
go
and
add
this
log
in,
but
that
also
brings
in
its
own
risks
as
well.
So
maybe,
while
we're
adding
this
this
log
in,
we
sneak
in
some
code
that
actually
breaks
things
such
as
some
buggy
code
or
something
like
that,
and
then
often
for
many
of
our
production
environments.
A
If
only
there
was
another
way,
I
wonder
where
that
way
is
going
to
come.
Okay,
so
I'm
going
to
show
you
BPF
Trace
is
a
great
tool.
That's
going
to
help
us
add
these
logs
in
production.
Bpf
Trace
is
a
way
to
get
kernel
level
visibility
inside
of
your
running
applications
as
it
mentions
here.
It's
also
a
high
level
tracing
language
for
the
Linux
ebpf
and
ebpf
is
really
just
a
way
to
run
sandbox
programs
inside
of
your
kernel,
sandbox
being
something
that
guarantees
Safety
and
Security
for
your
executing
programs.
A
So,
in
our
case,
the
arguments
to
this
is
valid
function.
You
can
think
of
this
analogous
as
analogous
to
a
debugger,
a
specifically
a
break
point
in
a
debugger.
The
break
point.
You
add
the
breakpoint
to
your
code.
Your
code
runs
until
that
breakpoint
pauses.
You
can
go
poke
around
the
variables
get
data
whatever
you
want
and
then,
when
you
want,
you
resume
the
execution
and
it
continues
onwards.
So
BPF,
trace
and
ebpf
are
very
similar
in
that
regard.
Instead
of
you
poking
around,
though
we
have
a
program
that
specifies
the
poking
around
I
guess.
A
So
here's
an
example
of
a
you
probe.
Oh
sorry,
a
BPF
Trace
script
specifically
on
this
is
valid
function
that
we
want
to
go
and
add
a
log
to
we
specify
the
type
of
probe
u-probe.
We
specify
where
we
want
to
go,
collect
this
data
on
the
symbol,
and
then
we
have
this
body
of
the
function
which
goes
and
figures
out
how
to
grab
this
data
that
we
want,
and
then
we
print
it
out
now.
A
Typically,
when
you
deploy
BPF
Trace,
you
go
SSH
into
a
node,
and
you
run
this
BPF
Trace
script
with
a
BPF
Trace
CLI,
but
in
a
kubernetes
environment,
where
you
have
many
nodes,
this
can
be
really
difficult,
or
at
least
just
tedious.
You
have
to
go
and
find
this
specific
node
that
is
running
the
specific
pod
that
you
want
to
go
Trace.
You
have
to
find
the
binary
path
for
that
specific
pods
server
that
you're
running,
specify
that
and
then
you
can
deploy
your
BPF
Trace
probe
at
Pixi.
A
Actually,
before
I
do
that,
so
this
is
online
boutique.
You
can
see
kind
of
what's
going
on
here,
it's
just
e-commerce
site
and
then
what
I'm
going
to
do
is
I'm
going
to
add
this
BPF
Trace
program
to
go
and
check
our
is
valid
call
and
so
I'll
show
you
that
actually
before
I
do
that
here's
the
data
we
get
out.
So
if
you
remember
that
structure
I
mentioned
earlier
that
split
up
the
float
value
for
money
into
two
different
integer
values.
We
have
our
units
value
here.
A
We
have
our
Nano
Nanos
values
here,
and
so
you
can
see
actually
right
at
the
top
here.
We
have
99
items
up
here,
so
that
is
the
sticker.
So
that
looks
good.
That
means
actually,
when
we
put
in
the
99
item,
things
are
working.
So
probably
is
not
just
from
that.
Like
entry
point,
so
something
in
our
code
is
probably
manipulating
it
to
cause
some
problems
and
okay,
so
I
have
to
go
trigger
the
error,
and
so
here's
what
happens
when
you
go
and
try
to
add
the
sticker?
A
A
A
So
this
is
really
bad,
but
what's
really
good
is
this
actually
gives
us
a
big
clue
onto
what's
happening
now.
I
mentioned
that
Nanos
are
one
billion.
Nanos
are
equal
to
one
unit.
So
if
you
convert
this
negative
10
million
into
units,
that's
negative
one
cent,
so
you
have
one
minus
one
here:
one
minus
one:
dollar
minus
one
cent,
that's
99
cents,
so
I
think
I
have
a
better
idea
on
what's
actually
happening
here.
A
This
99
value
is
entering
into
some
of
our
sum
functions
and
it's
being
converted
in
a
really
wrong
way,
I'm
trying
to
think
of
where
we
actually
call
this,
but
maybe
I
won't
show
you
that
actually
so
in
our
sum
function,
we
call
sum
a
number
of
times
I'm
going
to
skip
ahead
a
little
bit
here.
So
I've
seen
that
sum
is
called
a
number
of
times
one
time
it
sees
a
zero
and
the
next
time
and
one
sorry,
one
of
the
arguments
is
a
zero
value.
A
The
next
argument
is
that
99
value
that
we've
added
and
then
when
I
was
digging
into
this
earlier
I
noticed
that
this
condition
right
here,
which
is
basically
another
validity
check
after
we've,
summed
values
together.
This
validity
check
is
actually
incorrect
when
the
units
value
is
zero,
but
the
Nano's
value
is
greater
than
zero.
We
actually
don't
pass
into
this
if
statement
here,
even
though
that
value,
which
would
be
0
and
99
Cents,
would
actually
be
correct.
A
A
I've
shown
you
one
example
of
where
BPF
Trace
can
be
used
so
specifically
we're
using
a
u-probe,
but
there
are
many
opportunity
which
stands
for
user
space
probe,
but
there
are
other
places
where
you
can
probe
as
well.
So
such
as
the
kernel,
a
bunch
of
static,
Trace
points
that
kernel
developers
added
as
well
as
Library
developers,
have
added
and
there's
just
many
great
tools
that
have
been
already
listed
out
here
that
you
can
go
try
today.
A
A
The
evpf
landscape
is
pretty
interesting.
The
ebpf
application
landscape
is
pretty
interesting,
there's
a
lot
of
great
tools
from
networking
to
observability
to
security,
and
you
should
also
check
these
out
as
well.
They
give
you
this
like
nice.
Access
to
this
kernel
level,
visibility
that
evpf
gives
you
and
add
a
bunch
of
great
features.
On
top
of
that,.
A
Today,
I
showed
you
some
go
code
which
has
some
challenges
without
there's
some
challenges
with
writing
BPF
Trace
with
go,
but
it
is
completely
over
something
you
can
overcome
and
it's
it's
very
valuable
once
you
have
this
tool
in
your
tool,
set
I,
guess
with
that
I'll
be
at
the
pixie
booth
in
the
project
Pavilion
and
you
can
come
and
talk
to
us
talk
to
me
and
my
colleagues
more
about
what
pixie
does
what
BPF
Trace
does
and
I
think.
That's!
That's
all.
D
B
A
F
C
A
C
A
I
I
know
Java
is
a
bit
more
challenging.
There
is
some
benefit.
There's
some
benefits
that
the
programs
are
jitted
as
well,
but
I
don't
know.
Actually
the
details
on
what
you
need
to
do
for
Java.
We
have
actually
implemented
a
separate
feature,
not
not
the
BPF
Trace
feature,
but
a
profiling
feature
in
Java
and
so
there's
an
option
to
go.
Look
at
our
commits
there
and
I.
We
have
a
slack
Channel.
C
A
F
A
A
A
So
we
have
this
u-prote
here
and
Pixi
doesn't
require
this,
but
let's
say
vanilla,
BPF
Trace
does
you
would
typically
add
like
path
to
app
path.
Basically,
and
then
you
would
have
to
go
SSH
into
your
node,
make
sure
BPF
Trace
is
installed
and
then
go
and
add.
This
then
run
the
script.
Basically,
so
I
guess
yeah
I'd
be
running
that
I'll
just
save
it
is
valid.bt.
H
A
D
A
A
I
A
D
A
J
A
You
can
you
could
deploy
the
Damon
set
after
you've
run
into
the
issue.
It's
not
requiring
it
doesn't
require
you
to
go
and
run
it
continuously.
So
if
you
want
to
add
it
afterwards,
you
can
totally
do
that
and
we've
made
our
deploy
process
pretty
fast,
so
you
can
deploy
and
get
data
within
I,
think
three
or
so
minutes
and
or
deploy
and,
let's
say,
deploy
pixie
and
get
BPF
Trace
data
in
three
minutes
plus
the
time
it
takes
to
write
the
ppf
trace
script
yeah.
This
is.
B
G
I
J
A
A
As
long
as
your
pods
have
symbols,
both
Pixi
and
BPF
Trace
should
work,
and
there
might
be
a
few
more
caveats
on
on
that
as
well.
It's
in
our
docs,
but
I
know
that
symbols
are
important
and,
to
some
degree,
dwarf
info
is
helpful,
but
I
don't
think
actually
dwarf
info
is
a
hard
requirement,
so
yeah,
oh,
and
to
be
clear
if
you're
running
go
or
something
like
that,
your
symbol
should
already
be
included.
A
Your
learning
purposes
we
we
have
not
designed
anything
like
that
before,
but
we've
we've
thought
about
it
and
I
think
that's
like
a
really
cool
idea,
so
it
might
be
something.
So
actually
we
have
Pete
back
here.
He's
he's
somebody
who
works
closely
on
this
and
he
can.
He
can
talk
to
you
about
this
afterwards.
Oh
I
think
there's
a
question
back
there.
B
Hey
so,
if
I
understand
it
correctly,
if
you
have
access
to
the
like
pixie
agent,
or
at
least
that
UI,
you
can
run
like
any
code,
you
want
I
guess.
I
was
just
curious
how
you
control
access
to
the
to
those
agents
and
like
search
it
or
what?
What
I
guess
yeah?
What's
the
security
stuff
for
that
yeah.
A
There's
there's
a
few
different
options,
so
there
is
this
data
collection
side
where
we
offer
the
ability
to
redact
key
pii
containing
information,
so
you
can
run
in
restricted
mode
with
regards
to
BPF
Trace,
specifically,
we
have
adding
more
access
controls
on
the
roadmap,
so
I
think
a
temporary
solution
for
say
like
getting
the
observability
power
of
Pixie.
While
we
are
working
on
the
access
control
side
of
things,
you
can
go
and
pipe
that
data
into
some
open,
Telemetry
collector
and
use
create
a
view
on
that
side.
A
Basically,
so
I
guess
you
can
pipe
that
data
to
an
open,
Telemetry
collector,
send
it
to
Prometheus
and
then
use
Prometheus
views
or
something
like
that
or
a
grafana
or
some
other
tool
like
that.
And
that
way
you
can
get
that
data
and
everything
like
that,
while
preserving
the
permission
layer,
I
guess
you
can
also
just
remove
certain
users,
I
guess
from
your
system.
K
Okay,
I
can
respond
about
symbol
stores.
We
are
I'm
also
on
the
pixie
team,
so
kind
of
back
to
the
question
about
that.
We
are
aware
that
there
are
already
existing
simple
storage
Technologies
and
we
are.
That
is
not
on
the
roadmap
yet,
but
it
would
be
a
really
really
neat
thing
to
enable
both
for
BPF
trace
and
also
for
the
profiler
and
in
general,
it's
it
would
be
super
useful
I
can
see
that
so
should
anybody,
you
know,
join
us
on
slack
and
want
to
discuss
simple
stores
where
we
are
absolutely.
A
A
Yeah
yeah,
so
we
have
this
full
symbol,
name,
first
of
all,
so
I
guess
it's
a
little
bit
more
specific,
but
yeah.
You
can
specify
the
exact
pods
that
you
want
to
attach
to
there
is
this
label
selector
down
here
we
have
our
label
name
and
then
the
label
value
so
app
is
equal
to
front
end.
So
it's
only
attaching
to
the
front-end
service.
C
C
C
A
I've
we've
never
done
a
like
toe
for
toe
comparison
between
the
two.
As
far
as
I'm
aware
but
I
imagine
you
can
continually
optimize,
say
the
bytecode
injection
and
stuff
like
that.
But
the
same
way
you
can
continually
optimize
the
ebpf
injection,
but
at
the
end
of
the
day,
you're
kind
of
doing
the
same
operation
so
I
imagine
that
the
performance
is
comparable
right.
So.