►
From YouTube: Keynote: Debugging in Prod Without (Lots Of) Pain, Zain Asgar, GVP/GM - Pixie, & Natalie Serrino
Description
Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Keynote: Debugging in Prod Without (Lots Of) Pain, Zain Asgar, GVP/GM - Pixie, & Natalie Serrino, Principal Engineer- Pixie- New Relic
Ever had an issue in production that you couldn't reproduce in your local environment? Join this session to learn more about how we are tackling this problem with Pixie, a new CNCF sandbox project. We will also discuss high-level trends in observability, and the impact of Kubernetes.
A
Process,
at
least
to
me,
is
usually
pretty
scary.
There's
lots
of
issues
getting
you
know,
access
to
the
right
production
data,
and
usually
when
this
is
happening.
You're
under
lots
of
pressure
to
you
know,
fix
the
you
know
thing
causing
the
outage.
You
usually
have
to
send
out
updates
like
every
20
minutes
or
30
minutes
or
whatever
the
slas
are
and
you're
under
immense
pressure
without
access
to
a
lot
of
data,
and
it's
just
very,
very
challenging
as
a
developer.
A
So
to
start
off
with
that,
let's
first
take
a
look
at
a
monolithic
application
in
a
monolithic
application.
You're
trying
to
you
know,
deploy
primarily
one
large
service,
which
may
be
talking
to
some
databases
and
some
other
back
back
end
systems
and
generally
you're
trying
to
look
at
things
like.
Oh,
I
forgot
to
add
some
some
particular
logging
or
metric.
A
A
A
Talk
to
each
other,
that
was
a
lot
louder
and
you
know,
did
an
upstream
api
change,
caused
some
bucks
to
happen
and
which
services
are
making
this
transaction
slow.
And
you
know
this
isn't
meant
to
be
a
comprehensive
list,
but
it's
just
supposed
to
kind
of
show
you
the
breadth
of
the
types
of
problems
you're
going
to
run
into
you
know.
One
of
the
one
of
the
cool
and
interesting
things
is
as
kubernetes
is,
making
the
deployment
of
microservices
a
lot
easier.
A
We
actually
see
the
number
of
services
is
exploding
right
in
this
diagram
you're,
seeing
how
this
problem
works
for
two
services
in
reality.
What
we
actually
see
is
something
like
this.
This
is
actually
an
actual
service
diagram.
We
pulled
from
a
you
know,
an
actual
cluster
running
a
workload,
and
if
you
go
take
a
look
at
you
know
popular
other
services
like
netflix
and
others.
You'll
see
like
there
are
hundreds
of
services
that
are
talking
to
each
other.
So
if
you're
trying
to
debug
this,
it
can
be
a
pretty
big
challenge.
A
So
I
went
through
a
lot
of
information
over
there,
but
let's
summarize
them
with
some
of
the
the
key
problems
and
kind
of
bucket
them
loosely.
So
you
can
understand
understand
what
those
look
like.
So
one
of
the
problems
is
like
collecting
the
right
data
and
an
example
of
this
is
getting
the
right
log
lines
and
metrics
and
figuring
out
why
two
pods
can't
talk
to
each
other.
A
Then
there
is
the
flexibility,
or
rather
the
inflexibility
of
analysis,
which
is
you
know,
your
program
just
died,
and
you
really
want
to
understand
why
and
you
want
to
be
able
to
analyze
that
and
then
obviously
you
know
the
classic
problem
doesn't
happen
in
my
dev
environment
and
then
there's
also
just
contextualizing
the
information
and
understanding
why
this
problem
is
happening
in
the
context
of
where
it's
happening-
and
you
know
this
is
like.
Oh,
I
deployed
my
service
and
something
broke.
A
Did
it
actually
actually
break
because
of
a
change
in
the
different
service
so
like
contextualizing
it
with
other
updates
in
your
api
and
also
figuring
out
which
of
these
services
is
making
like
a
particular
transaction?
Slow
is
another
another
part
of
contextualizing
information
and
with
that,
let's
quickly
summarize
some
potential
solutions-
and
this
is
by
no
means
a
comprehensive
list,
and
there
are
many
different
ways
to
solve
these
problems.
A
It's
trying
to
create,
like
scriptable
and
api
driven
interfaces
to
give
the
level
of
flexibility
that's
necessary,
and
the
third
thing
is
to
try
to
build
a
more
kubernetes
native
system
that
can
understand
actual
entities
and
kubernetes
so
that
we
can
actually
build
an
entire
experience
for
debugging.
With
that,
I
will
move
over
to.
You
know
quickly
talk
about
pixi,
which
is
a
project
that
natalie
and
I
work
on.
It's.
B
Thanks
zane
yeah,
so
we're
going
to
do
a
lightning
demo
here
and
if
anyone's
interested
in
learning
more
about
pixi's
capabilities,
you
should
also
come
to
our
talk
on
friday
at
11
and
right
now
we're
just
getting
the
display
set
up
so
that
I
can
actually
see
what
I'm
doing
when
I'm
demoing
sorry
about
that.
No
we're
good
yeah
but
yeah.
As
being
said,
I'm
also
an
engineer
working
on
pixie
and
really
excited
to
be
here
today.
B
So,
while
we
sped
up
this
video
a
little
bit,
it
usually
takes
about
two
or
three
minutes
to
deploy
pixi
onto
your
cluster.
I
think
that
the
key
thing
here
is
that
in
just
one
command,
we're
automatically
deploying
all
these
ebpf
probes
to
your
cluster
to
collect
a
rich
set
of
telemetry
data
which
we'll
then
I'll
show
in
the
ui
demo.
Next.
B
And
you
know
kind
of
interesting
fact
about
this-
is
that
you
know
pixie.
The
pixie
team
was
at
kubecon
in
2018
and
linker
d,
which
is
another
really
awesome.
Project
gave
a
demo
that
heavily
inspired
this
deployment
flow
and
we
hope
that
deployments
of
you
know
observability
and
other
types
of
infrastructure
and
kubernetes
can
get
a
lot
simpler.
So
we're
deploying.
B
B
B
We
can
list
out
the
various
nodes
in
my
cluster
and
we
can
see
things
like
the
namespaces
that
exist,
and
this
is
where
the
kubernetes
native
part
of
this,
these
systems
that
we
want
to
build
come
in
where,
if
you
know
the
system
you're
running
on
like
kubernetes,
you
can
attach
much
more
helpful
context
to
debugging
kubernetes
specific
problems.
So
that's
why
we
want
to
reason
about
entities
like
pod
or
service,
and
so
kubernetes
helps
us
do
that.
B
So
when
we
cluster
kind
of
our
service
graph
here,
we
can
kind
of
see
that
there's
two
main
apps
running
on
here
and
I
heard
in
a
bug
report
that
customers
are
finding
my
online
boutique
to
be
slow.
So
let's
dig
into
why
that
might
be
we're
gonna
go
into
our
services
and
just
widening
this
for
visibility.
B
B
So
I
can
see
that
this
checkout
service
right
here
is
taking
almost
a
second.
You
know
in
some
of
its
requests
and
that
is
heavily
unexpected
to
me
as
a
developer.
So
I'm
going
to
click
into
this
and
try
to
figure
out
more
about.
What's
going
on,
I
can
see
these
http
requests
and
errors
in
latency
and
all
of
these
things
I
didn't
have
to
instrument
my
system
manually,
it's
just
being
collected
through
the
ebpf
probes
that
were
deployed
via
that
simple
px
deploy
command.
B
I
can
see
the
inbound
traffic.
I
can
see
that
you
know
the
traffic
coming
from
this.
Ip
is
actually
a
lot
slower
on
average
than
the
traffic
coming
from
the
other
ip
there's
only
one
pod
here.
So
let's
drill
down
into
this
pod
and
see
if
we
can
figure
out
more
about
what's
going
on-
and
this
is
a
lightning
demo,
so
we're
not
actually
going
to
solve
this
just
trying
to
show
off
what
I
might
do
in
a
true
incident.
B
So
we
can
see
lots
of
both
kind
of
infrastructure
level
metrics,
as
well
as
application
level
metrics.
In
this
view,
and
one
of
the
things
that
we're
most
excited
about
is
that
we
recently
added
continuous
profiling
to
pixi,
so
you
can
actually
see
performance
flame
graphs
of
you
know
not
just
like
the
fact
that
something's
slow,
but
which
function
is
slow,
which
function
is
the
problem.
B
B
One
point
that
zane
was
talking
about
is
the
idea
that
we
want
things
to
be
api
driven.
We
want
things
to
be
scriptable.
We
want
to
be
able
to
compose
data
pipelines
together
of
different
pieces
and
we
don't
want
it
to
be
in
some
walled
garden.
That's
like
very
hard
to
access,
so
one
thing
that
we
did
when
we
built
pixie
was
we
made
sure
that
everything
that
you
see
in
all
the
views
that
I
showed
is
generated
by
a
script,
and
we
have
client
apis
for
this.
B
B
B
A
B
B
The
thing
is,
we,
you
know
are
so
excited
to
be
part
of
the
cncf
because
of
the
quality
of
projects
that
the
cncf
has
we're
thrilled
that
you
know
like
these
projects
like
prometheus
and
open
tracing
like
now.
We
get
to
be
part
of
the
cncf
with
them
and
they
are
also
solving
these
problems
in
really
important
ways.
You
know
we
have
open
tracing,
which
is
making
a
standard
for
data
that
people
can
use
to
query
and
analyze
all
of
their
data
together.
That's
huge
huge
for
flexibility
of
analysis.
B
You
know
huge
for
default
default
observability
on
kubernetes,
it's
so
much
easier
to
get
in
all
this
information
today
when
you
have
an
incident
than
it
used
to
be
linker
d,
as
I
mentioned,
you
know
they
directly
inspired
a
large
part
of
our
architecture.
They
an
envoy
you
know
with
the
service
mesh
approach.
It
has
huge
impact
for
observability.
B
B
Also.
You
know
here
are
some
talks
that
we
are
gonna
be
attending.
We're
gonna
be
attending
way
more
than
just
this,
but
here
are
some
of
them
that
we're
really
excited
about,
as
mentioned
before,
you
know
we're
actually
giving
a
new
a
different
talk
on
friday
at
11,
so
you
know
we
would
love
to
see
you
there
and.
Finally,
if
you
want
to
check
out
the
source
code,
pixi's
fully
open
source
like
please
check
out
our
github,
and
we
would
love
to
you,
know,
see
your
issues
or
feature
requests.