►
From YouTube: Pythia: An Automated, Cross-layer Instrumentation Framework for Diagnosing Performanc... Emre Ates
Description
Join us for Kubernetes Forums Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Pythia: An Automated, Cross-layer Instrumentation Framework for Diagnosing Performance Problems in Distributed Applications - Emre Ates, Boston University
It is extremely difficult to understand where to enable instrumentation a priori to help diagnose problems that may occur in the future. We present Pythia, an automated cross-layer instrumentation framework, which explores the space of possible instrumentation choices and enables instrumentation needed to diagnose a newly-observed problem in production systems. Pythia builds on distributed tracing and uses statistical techniques to identify where instrumentation is needed. This talk will discuss 1) the scalable design of Pythia 2) our progress on identifying promising data structures to represent the instrumentation search space across multiple data center stack layers (e.g., application and kernel). These structures must trade-off between compactness, exhaustiveness, and accuracy. 3) Creating algorithms to search this space quickly while staying under a specific instrumentation budget.
A
So
debugging
distributed
systems
is
something
that's
really
challenging,
which
a
lot
of
the
previous
talks
have
focused
on.
So
one
of
the
reasons
for
this
challenge
is
knowing
where
the
problem
is
in
our
distributed
system,
it
could
be
in
any
one
of
the
many
components
that
comprise
the
distributed
system.
It
could
be
in
one
of
the
several
stack
layers
like
that.
The
components
are
hosted
on
like
virtual
machine,
the
kernel
or
the
network
layer.
It
could
also
be
a
result
of
emergent
behavior.
That
only
happens
when
you
put
all
of
your
system
together.
A
So
today,
people
use
instrumentation
and
also
boating
methods
to
debug
distributed
systems.
They
basically
add
instrumentation
points
to
different
components,
and
one
problem
with
this
is
that
different
problems
benefit
from
different
instrumentation
points,
and
if
you
want
to
instrument
everything
and
add
instrumentation
points
for
every
problem
that
may
have
have
happen
in
the
fiercest
'im,
then
basically,
you
will
be
drowning,
an
instrumentation
data
and
your
system
would
be
collecting
more
instrumentation
data
rather
than
doing
useful
work
so
and
today's
debugging
happens
in
air
cycle.
A
Basically,
we
first
gathered
data
from
current
instrumentation,
and
then
we
try
to
find
a
problem
using
this
instrumentation.
If
we
can't,
we
add
more
instrumentation
and
then
we
collect
more
data
onto
and
we
continue
with
the
cycle
until
we
can
localize
the
performance
enough
to
find
come
up
with
a
solution
so
right
now,
this
process
involves
many
iterations
of
guess
and
check.
It
increases
the
downtime
and
it
also
in
cost
money
both
in
terms
of
dual
overtime
and
downtime.
A
So
our
research
question
is
whether
we
can
automate
this
cycle
and
if
we
can
create
an
automated
framework,
that's
continuously
running
alongside
production
distributed
systems
and
it
enables
and
disables
instrumentation
in
response
to
newly
happening
performance
problems
and
only
reporting
to
the
developers
once
it's
certain
that
there
is
a
problem
and
it
has
localized
it
enough
so
and
we
are
using
several
key
observations
that
let
us
construct
such
a
system,
one
of
them,
is
that
similar
workflows
should
perform.
Similarly,
for
example,
you
can
see
a
workflow
diagram
in
the
slide
here.
A
This
is
a
little
bit
different
from
the
open
tracing
expand
style.
It's
basically
made
up
of
events
as
the
nodes,
and
the
edges
indicate
happens
before
relationships,
and
here
you
can
see
a
read
request
from
storage
which
starts
from
the
client
and
then
there's
a
load
balancer.
You
do
a
metadata
operation,
you
read
it
and
then
you
return
to
the
client.
A
So
if
this
request
is
issued
frequently
in
the
system,
then
you
would
probably
see
many
of
these
similar
workflows
and
if
one
of
them
is
suddenly
looks
like
that,
then
this
means
that
there
is
a
problem
in
this
request.
Most
probably,
and
also
you
can
use
this
trace
to
immediately
vocalize
this
problem
to
the
read
component.
So
a
key
enabler
for
this
is
request:
centric
tracing
or
workflow
centric
tracing
or
the
series
of
tracing
which
basically
lets
us
get.
A
These
work
flow
graphs
from
production
distributed
systems,
so
our
contributions
are
Pythia,
which
is
an
initial
approach
for
an
automated
cross
layer,
instrumentation
framework
and
also
an
architecture
for
enabling
this
approach
and
a
method
for
enables
ending
this
cross
layer,
instrumentation
in
a
single
search
space
and
some
initial
heuristics
for
exploring
the
search
space
to
locate
problems.
I
will
also
talk
about
our
initial
experiences
using
Pythia.
A
Here's
an
overview
of
Pythia
I
will
go
up
through
every
step
of
Pythia
and
then
I'll
focus
on
the
interesting
components
later
Pythia
begins
by
gathering
data
on
the
top
left.
It
gathers
work
for
skeletons
which
are
basically
traces
with
most
of
the
instrumentation
that's
disabled,
so
we
can
only
collect
like
very
simple
info
information
about
the
traces
and
from
these
traces.
We
extract
critical
paths
and
we
group
them
based
on
performance
expectations.
We
group
this
pour
the
requests
that
are
expected
to
perform.
A
Similarly,
after
that,
we
look
at
all
of
the
groups
and
we
identify
which
groups
are
problematic.
These
can
be
groups
that
have
high
performance,
variation
or
groups
that
are
consistently
slow.
Then
we
determine
where
to
enable
instrumentation
within
these
groups,
for
example,
between
which
trace
points,
and
also
after
that,
we
look
at
our
search
space
and
search
strategies
to
find
out
what
instrumentation
we
can
enable
provided
that
we
have
log.
We
have
localized
the
problem
into
a
single
edge
after
enabling
this
additional
instrumentation.
We
collect
data
again,
and
we
begin
this
cycle
again
now.
A
I
will
focus
on
how
we
collect
this
data
and
group
the
workforce,
so
entry
and
tracing
is
something
that
most
of
you
are
familiar
with.
Basically,
it
preserves
the
traces
of
request
workflows
by
propagating
a
unique
context
like
a
request
ID
with
each
request.
Afterwards,
these
trace
points
are
stitched
together
and
we
get
traces.
So,
for
example,
from
this
distributed
system,
which
is
OpenStack,
you
can
collect
a
trace.
A
That
looks
like,
on
the
right
hand,
side
where
it
begins
with
Nova,
and
then
it
does
authentication
using
Keystone
and
then
maybe
those
two
requests
in
parallel
and
then
collects
results.
So
each
edge
indicates
the
latency
of
the
of
that
time
period
and
the
nodes
indicate
the
different
events,
and
we
can
call
the
start
and
end
nodes
that
match.
We
can
refer
to
them
as
spans,
which
is
pretty
standard.
A
After
collecting
these
workflows,
we
extract
the
critical
paths
because
we
only
care
about
the
components
of
the
workflows
that
affect
overall
request
latency,
after
that
we
group
the
workflows
by
expectation.
So
one
initial
expectation
can
be
that
we
expect
all
of
the
requests
of
the
same
type
that
X
access
the
same
distributed
system
components
to
perform.
Similarly,
and
there
can
be
other
types
other
ways
to
express
expectations.
For
example,
we
can
look
at
the
request.
Types
like
read:
request,
write
requests.
A
A
So
if
you
remember
these
traces
from
the
beginning
of
the
presentation,
basically,
we
can
have
multiple
different
workflows
that
have
similar
structure
that
are
expected
to
perform.
Similarly,
but
some
component
of
them
is
actually
taking
a
longer
time.
So
we
for
this,
we
actually
can
look
at
different
metrics
to
measure
the
variation
of
the
requests.
We
could
look
at
standard
deviation
and
have
an
acceptable
threshold
for
request
latency
standard
deviation.
A
We
can
also
look
for
multimodal
distributions,
which
usually
means
that
there
are
multiple
kind
of
modes
of
operations
like
hash
widths
and
cache
misses
that
result
in
different
performance
characteristics
after
localizing
the
performance.
We
determine
what
we
can
enable
next
to
bring
us
closer
to
the
source
of
the
problem,
and
for
this
we
use
our
search
space
and
search
strategies
which
I
will
come
to
later
after
we
enable
more
instrumentation,
we
collect
the
data
again
and
we
then
group
our
request,
workflows
that
have
a
little
bit
more
instrumentation
and
then
we
begin.
A
Thus
we
enable
more
instrumentation.
So
now
I'll
talk
about
the
architecture
of
Pythia.
We
decided
to
separate
this
as
the
instrumentation
plane
and
the
control
plane.
This
helps
us
separate
the
concerns
and
simplify
the
designs
of
each
of
each
parts
of
the
architecture.
So,
on
the
top
we
have
the
control
plane,
which
is
basically
the
workflow
that
I
just
described
earlier,
and
this
this
can
be
implemented
as
a
Big
Data
job
that's
running
periodically
in
some
day
Center
and
the
instrumentation
plane
is
basically
your
instrumentation
framework
and
your
distributed
applications.
A
Now
we
decided
to
status
requirements
from
the
instrumentation
plane
instead
providing
a
single
implementation
for
the
instrumentation
plane,
because
a
lot
of
the
instrumentation
implementations
are
really
tightly
coupled
with
the
application,
so
we
would
prefer
to
adapt
bethere
to
different
ones,
so
it
will
be
compatible
with
many.
One
requirement
is
always
on
context
propagation.
So,
regardless
of
if
you
are,
if
you
have
enabled
some
trace
points
or
disabled
them,
the
context
propagation
should
always
happen
to
be
able
to
collect
a
request
in
their
entirety.
A
One
of
the
most
important
ones
is
controllable,
trace
points
because
PCA
works
by
enabling
and
disabling
trace
points.
We
need
to
have
trace
points
that
can
be
enabled
and
disabled
now.
This
is
currently
not
supported
or
not
supplied
by
many
of
the
instrumentation
frameworks
like
Yaeger
or
open
telemetry,
based
ones,
but
we
have
actually
implemented
it
for
open,
Stax
tracing
for
infrastructure
and
it
took
us
only
80
lines
of
Python.
A
So
it's
not
a
not
impossible
to
implement
and
we
also
need
fast,
restore
construction
for
Pythia
to
be
able
to
quickly
quickly
respond
to
ongoing
performance
problems.
Now,
I'll
talk
about
the
search
space.
The
search
space
is
basically
a
manifest
of
all
of
the
instrumentation
that
piteå
can
control
and
it's
used
to
find
the
relevant
next
trace
points
to
enable
after
the
problem
is
narrowed
down
to
a
heart
of
a
request.
A
So
I'll
talk
about
one
approach
for
a
search
space.
That's
the
current
one
that
we
are
exploring.
That's
based
on
calling
context,
trees.
A
calling
context
tree
is
basically
a
call
graph
which
has
an
individual
node
for
each
unique
stack
trace
and
it
leverages
the
hierarchy
of
distributed
system
architecture,
which
is
inherently
hierarchical.
So
if
we
have
collected
the
trace
that
looks
like
this
during
the
offline
profiling
stage
and
the
search
space
would
be
looking
like
the
tree
on
the
right
hand,
side.
A
So
the
topmost
span
would
be
Nova
because
it
encompasses
all
of
the
request.
Duration
and
his
children
would
be
Neutron,
Keystone
and
glance
and
Keystone
would
happier
appear
again
after
glance,
because
that's
actually
another
calling
context
and
there
could
be
different
problems,
manifesting
based
on
whether
Keystone
is
called
by
Nova,
or
it's
called
by
glance
for
the
different
data.
A
Centers
stack
layers,
for
example,
for
the
names
colonel
we
plan
to
have
a
separate
search
space
and
link
them
together
when
there
are
system
calls
now
search
strategies
are
methods
to
explore
this
search
space
and
the
goal
of
the
search
strategies
is
to
quickly
converge
on
different
problems,
then
also
to
keep
the
instrumentation
overhead
low.
Good
search
strategies
would
reduce
the
time
to
solution.
A
So
I'll
talk
about
hierarchical
search,
which
is
one
of
the
many
choices.
It
basically
searches,
the
trace
point
choices
in
a
top-down
manner
and
it's
very
compatible
with
calling
context
trees
because
both
of
them
leveraged
the
hierarchy.
So
it
would
start
by
just
enabling
Nova
and
collecting
the
request
latency
for
Nova
requests,
and
if
there's
a
problem,
we
didn't
or
a
request.
We
would
then
enable
the
children
one
by
one,
for
example.
A
First
Neutron
then
Keystone
then
glance
the
order
of
enabling
children
can
be
based
on
the
variance
information
that
p3
has
collected
before
and
maybe
if
the
problem
is,
for
example,
narrowed
down
to
glance,
then
we
would
go
on
to
look
at
the
children
of
clans,
which
is
only
Keystone
in
this
simple
search
space.
There
can
also
be
other
search
strategies
or
optimizations
of
search
strategies
across
their
search
strategy
could
skip
to
the
kernel
layer
when
we
narrow
the
problem
down
to
only
a
single
district
system.
A
Now
I'll
talk
about
our
experiences
with
PPA
and
have
the
validate
our
approach.
We
basically
implemented
Pythia
using
OpenStack
use
by
tracing
OpenStack.
So
OpenStack
is
an
open
source
cloud
platform.
That's
written
in
Python.
It's
used
for
managing
clouds
like
spinning
up
VMs
and
allocating
storage.
End-To-End
tracing
of
OpenStack
is
done
using
OS
profiler,
which
is
specifically
written
just
for
OpenStack.
We
augmented
OS
profiler
by
adding
controllable,
trace
points,
and
we
also
stored
more
very
variable
values
such
as
the
queue
lengths
within
the
open,
stack
traces.
A
We
run
our
experiments
on
the
message,
that's
open
cloud
and
we
have
a
synthetic
workload
that
come
that's
composed
of
nine
different
request
types
that
creates,
deletes
and
lists
the
virtual
machines
floating
IPs
and
volumes.
We
simultaneously
execute
20
workloads
with
different
combinations
of
these
request
types,
and
then
we
collect
the
request.
Latencies
oops.
A
Ok
yeah,
so
so
we
collect
the
latest
values
for
each
request
and
in
the
plot
you
can
see
the
request
latency
on
the
x-axis
and
the
group
request
types
on
the
y-axis,
and
these
are
different
box
plots.
So
you
can
see
that
the
off
most
of
the
requests
perform
kind
of.
Similarly,
some
of
them
have
higher
variance.
A
For
example,
all
of
these
requests
have
authentication
requests,
but
we
wouldn't
want
to
trace
authentication
for
everywhere,
but
only
for
the
server
create
requests
because
that's
was
later
seized.
We
care
about
so
I'll
skip
to
the
highest
variance
edge.
That
pthey
has
found
after
enabling
instrumentation
for
this
request
group:
here's
a
Python
code,
snippet
from
OpenStack.
Basically,
the
synchronized
function
in
the
top
line
and
the
do
build
and
run
instance
function.
A
The
bottom
line
are
traced
and
the
highest
various
age
was
the
edge
between
the
end
of
synchronized
to
the
beginning
of
do
build
and
run
instance,
and
there's
actually
only
one
line
of
code.
That
does
anything
in
that
middle
place
and
that's
basically,
where
our
variance
was
coming
from.
So
then
we
can
look
at
what
self
that
build.
Semaphore
is
to
further
understand
what
this
problem
was,
and
if
you
look
at
the
initialization
of
this
semaphore,
there's
basically
a
configuration
option.
That's
limiting
the
maximum
number
of
concurrent
VM
builds
to
ten.
A
So
this
could
be
simple
when
I
present
it
like
this,
but
remember
that
Nova
has
actually
thousands
of
configuration
options
and
Nova
is
just
one
of
the
micro
services
that
make
up
OpenStack,
so
without
PTR.
Finding
this
from
the
traces
would
be
challenged
after
finding
yourself.
We
can
actually
add
more
instrumentation
and
see
what
Pythium
could
have
done
if
this
was
already
in
instrumented.
A
So
after
this
we
can
do
you
use
a
method
called
canonical
correlation
analysis
or
CCA.
Cca
can
be
used
to
find
the
important
variables
in
the
traces,
so
it
finds
the
variables
that
are
high,
the
correlated
with
highest
with
two
over
latency,
and
it
works
by
solving
that
equation
on
the
right.
Basically,
X
is
the
different
variables
that
we
have
collected
and
Y
is
the
request
durations,
so
it
finds
a
coefficient
matrix,
a
that
when
multiplied
with
X
have
provides
the
highest
highest
correlation.
A
So
basically,
variable
variables
that
have
a
high
coefficient
would
be
highly
correlated
and
variables
that
have
a
low
coefficient
would
have
no
effect
on
request
latency.
So
when
we
do
memory
instrument
the
queue
length
and
use
CCA,
we
find
that
the
that
variable
value
has
a
0.85
correlation
with
request
x,
with
request
latencies.
A
So
after
adding
the
queue
length
variable
into
the
traces,
we
can
also
use
it
to
wear
during
grouping,
and
you
can
see
on
the
before
side,
though,
is
the
left-hand
side.
The
server
create
is
the
topmost
boxplot,
with
the
high
variance
and,
on
the
right
hand,
side.
It's
actually
splits
into
three
different
groups,
but
very
small
variance
because
they
are
just
based
on
different
queue
lengths.
A
So
if
there's
nobody
waiting
for
the
semaphore,
when
we
arrive,
then
we
get
the
third
box
plot
and
then
the
second
box
plot
is
basically
with
one
two
three
waiters
and
the
top
one
is
the
more
than
three
waiters
for
the
same
request.
So
I
take
away
from
this
is
that
PCA
can
correctly
identify
sources
of
variation
in
a
in
a
running
distributed
system
and
also
find
like
which
variables
are
measuring
these.
So
what
about
the
other
trace
points?
Could
we
have
done
this
without
pity
as
well?
Here's
a
plot
of
the
request,
latency.
A
If
we
enabled
everything
for
the
problematic
group
there
are
about
350,
trace
points
and
95%
of
the
variance
of
this
request
is
measured
by
just
six
trace
points.
So
if
we
didn't
use
Pythia,
we
would
be
collecting
350
trace
points
for
every
request
and
350
like
49
of
them
would
basically
be
not
used
and
nobody
would
be
looking
at
them,
and
Pythia
can
actually
correctly
identify
the
highest
variance
edge
without
ever
enabling
all
of
these
trace
points.
A
So
the
takeaway
for
this
is
that,
for
a
given
problem,
meaning
of
the
trace
is
collected
are
not
useful.
Then
there
is
only
a
subset
of
trace
points
that
can
be
used
to
diagnose
the
problem.
So
our
next
steps
with
Pythia
is
to
try
it
out,
try
it
out
on
other
distributed
systems
right
now,
it's
just
for
OpenStack,
but
we
plan
to
use
it
for
HDFS
and
Ceph.
We
also
want
to
integrate
kernel
instrumentation
with
Pythia
and
also
understanding,
pts
overhead
and
finding
out
search
strategies
that
actively
limited
is
another
next
step.
A
We
also
have
some
open
questions.
For
example,
what
is
the
ideal
structure
of
the
search
space
and
what
possible
search
strategies
are
there
and
are
there
different
trade-offs
that
are
suitable
for
different
kinds
of
problems,
and
how
can
we
represent
events
like
context,
switches
and
page
faults
in
the
search
space,
because
the
search
space
relies
on
the
structure
of
the
traces
for
something
like
a
context
switch
which
can
happen
anywhere
in
the
request?
A
We
can't
really
place
it
in
our
call
in
context
G,
but
calling,
but
things
like
context,
switches
and
garbage
collections
can
cause
performance
problems,
so
it
would
be
useful
if
PTI
can
actually
explore
those
instrumentation
choices
as
well.
Another
question
is
how
we
can
formulate
and
choose
an
instrumentation
budget
that
would
keep
pts
tracing
overhead
law.
A
Another
open
question
is
how
granular
the
expert
formas
expectations
should
be.
Does
the
developer
have
to
specify
just
request
types,
or
do
we
have
to
take
into
account
other
behavior
like
caches
and
cache
misses
so
in
conclusion,
it's
very
difficult
to
debug
distributed
systems
and
if
we
automate
instrumentation
choice,
this
is
a
promising
solution
to
overcome
this
difficulty.
If
you're
interested
about
this
more,
you
can
look
at
our
associate
19
paper.
This
is
gonna,
be
like
this
week
later
this
week,
or
it's
already
on
the
website.
Yeah.
B
C
A
C
A
C
B
A
That
it's
an
interesting
question
because
we
get
different
results
based
on
what
we
use.
For
example,
we
could
use
coefficient
of
variance,
which
is
the
variance
divided
by
the
mean
to,
for
example,
decide
if
a
request
is
problematic
or
not,
but
that
kind
of
meant
tries
to
find
short
to
request
more
problematic
because
may
their
meat
is
much
lower.