►
Description
MLExray: Observability for Machine Learning on the Edge - Michelle Nguyen, Stanford
Anyone who’s ever deployed on the edge has had this hope before: “It ran perfectly on my cloud environment, it’ll surely work when I deploy it across these other different environments”. Unfortunately, much of the time, this hope falls flat. This is frustratingly true for those deploying machine learning models on the edge. These models are often painstakingly trained and fine-tuned over months and days to achieve those extra few percentage points of accuracy… Only to see performance drop by over 10% once deployed to an edge device. This session will cover common problems encountered when deploying machine learning models on the edge, and how MLExray, an open-source observability framework created at Stanford, can be used to help debug these issues when they inevitably occur. *MLExray has been accepted into MLSys 2022: https://arxiv.org/pdf/2111.04779.pdf
A
A
A
A
Am
I
about
to
hit
something
an
obstacle
and
it
collects
all
that
information
and
does
it
locally
so
it
stores
it
locally
and
then
runs
some
inference
on
it
to
figure
out
what
action
it
should
do
next
and
then,
similarly,
you
know
you
have
the
amazon
echo
and
that
does
the
same
thing.
It
collects
input
from
the
environment.
You
know
what
whatever
you're
saying
at
the
time
in
your
home
and
tries
to
figure
out,
should
I
do
something
with
this
information.
A
So
there's
a
lot
of
benefits
to
this
right.
You
can
see
immediately
here
from
this
picture.
You
get
a
lot
less
network
bandwidth
and
egress,
especially
if
your
sensor
is
collecting
tons
of
information
now
you're
not
sending
out
to
some
other
cloud
that
you
potentially
don't
even
own,
and
so
that's
really
nice
because
you
get
the
data
and
you
can
immediately
start
processing
running
your
models
on
it
and
then,
second
of
all
the
nice
thing
about
it.
Is
you
get
some
really
good
security
privacy
benefits
right
so
think
about
the
amazon.
A
Echo
case
your
echo
sitting
on
your
your
counter,
and
it
feels
a
little
bit
weird
to
be
like.
Oh,
it's
listening
to
me
and
like
sending
all
like,
whatever
I'm
saying,
all
the
way
directly
to
amazon
right,
it's
nice
to
think
about,
like
okay,
it's
actually
doing
edge
compute,
whatever
I'm
saying
it's,
storing
it
locally.
It's
probably
expiring
it
out
after
some
point,
because
that's
a
ton
of
data
that
it's
collecting
and
all
of
it
is
just
stored
within
the
edge
device
itself
and
then
finally,
there's
scalability
so
before.
A
If
you
have
millions
of
connected
devices
and
all
these
are
streaming
their
data
to
your
cloud,
you're
going
to
have
to
be
able
to
handle
all
this
input
from
a
bunch
of
these
different
devices.
But
now
all
these
models
are
running
individually
on
the
device
itself
and
so
they're
just
responsible
for
dealing
with
their
own
set
of
inferences.
A
So
how
does
this
actually
look
like?
How
do
you
start
go
to
go
and
deploy
your
models
to
your
edge
devices
and
how
it
usually
happens
is
that
the
first
part
is
pretty
much
the
same
right.
You
need
to
go
and
you
train
your
model.
You
have
a
bunch
of
training
data,
you
go
and
you
make
sure
you
tune
all
your
parameters
and
make
sure
your
accuracy
is
correct,
like
you
have
an
image
classification
you're,
trying
to
classify
what
kind
of
dog
breed
is
this?
A
You
train
this
in
the
cloud
and
you
ensure
it's
like
okay,
any
input
I
give
it.
It
looks
about
you
know
about
right,
and
then
you
go
and
you
can
deploy
these
to
all
these
different
edge
devices.
So
I
wanted
to
make
it
very
clear
that
all
these
are
just
like,
potentially
different,
hardwares
environments.
So
I
just
made
these
boxes
of
the
edge
devices
different
colors,
because
you
know
one
could
be
an
iphone.
One
could
be
an
android
device.
A
So
it's
just
like
very
clear
that
these
models
are
now
going
to
a
very
different
set
of
environments
and
what
actually
can
happen
here?
What
could
go
wrong
right?
It
works
perfectly
in
the
cloud
and
keep
in
mind
that
your
cloud,
it's
running.
It
has
as
many
resources
as
you
want
to
give
it
right.
You
can
give
it
as
much
memory
you
want.
A
You
give
it
as
much
cpu
that
you
want
and
as
long
as
you're
willing
to
pay,
but
then
now
you're
going
to
these
edge
devices
and
like
I
mentioned
before,
you
have
phones,
but
you
might
be
deploying
to
just
like
you
know
some
little
thing
running
a
sensor
or
something
and
you're
going
to
be
limited
by
your
resource
requirements.
Your
your
memory
requirements,
so
a
bunch
of
things
could
go
wrong.
So
the
first
case
up
on
top,
is
you've
deployed
to
your
edge
device.
A
And
then
you
know
things
seem
to
run,
but
then
you
get
the
completely
wrong
results.
You're
noticing
a
huge
drop
in
accuracy.
So
before,
maybe
when
you
had
80
90
accuracy,
you're,
seeing
that
your
model
that's
been
deployed
somewhere
else
is
giving
you
like,
maybe
even
30
accuracy,
and
the
second
case
where
something
could
go
wrong,
is
your
model
seems
to
run
fine?
It's
giving
you
the
right
inferences,
but
it's
taking
a
really
really
long
time,
and
so
it's
like
okay,
it
used
to
work.
No,
it
doesn't.
A
A
I'm
gonna
go
and
just
like
put
a
bunch
of
print
statements
and
just
figure
out
where
exactly
in
this
pipeline
are
things
going
wrong
and
obviously
that's
a
very
tedious
problem,
because
there's
especially
in
a
like
a
deep
learning
model,
there's
a
lot
of
stuff
going
on
in
there
right
you
you're,
going
to
like
print
a
bunch
of
things.
You
have
to
make
sense
of
all
that
information,
and
that
is
where
ml
x-ray
comes
in
so
ml.
A
X-Ray
is
a
project
that
came
out
of
a
stanford
research
project,
so
huge
credits
goes
to
all
these
people
that
I've
listed
on
the
bottom,
who
came
up
with
the
original
idea
for
mlx
rate
and
all
the
great
work
that
they've
done
so
far
on
it,
and
essentially
what
ml
x-ray
is.
Is
that
it's
an
end-to-end
framework
for
debugging,
your
cloud-to-edge
deployment
and,
let's
dive
a
little
bit
into
more
about
what
that
actually
means.
A
So,
first
mlrxa
gives
you
this
api
and
you
basically
invoke
that
api
and
it
starts
collecting
a
ton
of
information
for
you
as
you
run
your
model
so
up
on
top.
We
have
an
example
of
the
python
api.
You
start
the
the
inference.
You
stop
the
inference
using
the
mlx
ray
api
and
during
each
part
in
your
model
pipeline.
It
just
records
tons
of
information.
So
it's
like
the
pre-processing
step,
the
individual
layers
down
to
the
final
output
that
it
gets
in
the
end.
So
you
don't
have
to
go
in
and
actually
add.
A
So
what
kind
of
information
does
it
get?
It
really
gets
a
bunch
of
things.
So,
just
like
a
bunch
of
baseline
debug
data
so
like
what
are
the
inputs
and
outputs
of
your
model
and
then
for
each
individual
layer
inside
your
model
pipeline.
It
also
gets
the
input
and
output
of
that,
and
you
know
for
the
case
where
it's
like
things
start
running
very
slowly.
It
will
track
your
end
to
end
latency
for
running
the
entire
inference,
but
also
in
between
each
layer
itself
and
just
resource
information.
A
So
it's
like
what
was
the
memory
when
my
model
was
running
and
for
devices
like
android.
It
also
collects
just
other
sensor
information.
So
it's
like
your
android
knows
what
how
it's
rotated,
what
the
lighting
conditions
are
like
and
that
just
helps
give
more
context
to
just
how
the
model
is
running
on
your
device
and
you
can
go
ahead
and
use
the
mlx
rate
api
to
just
add
whatever
other
fields
you
might
want
to
log,
so
more
information
that
you
might
want
to
collect
that
are
specific
to
your
model
itself.
A
A
What
am
I
supposed
to
do
now,
and
so
the
idea
behind
ml
x-ray
is
that
you
compare
this
to
a
reference
pipeline,
so
you
have
your
model
that
you've
trained
in
the
cloud.
You
know
this
is
your
like
baseline
model.
This
thing
runs
the
best
that
you
can
imagine
the
best
accuracy
that
you
can
get.
It
runs
at
the
speed
that
you
want,
and
so
what
you
do
here
is
you
collect
logs
from
that
baseline
model,
and
then
you
have
the
you
know
the
one
that
you've
deployed
to
your
edge
device.
A
You
collect
the
logs
from
that
one,
and
essentially
you
combine
them
together
to
figure
out.
You
know
what
are
the
differences
here
and
that's
what
we
call
a
debug
report,
so
you
compare
the
differences
and
try
to
figure
out
where
exactly
in
my
model
are
things
going
wrong
and
what
does
this
flow
kind
of
look
like
when
you're
debugging,
so
the
first
is
in
accuracy?
Validation
is
the
model
that's
deployed
on
your
edge
device.
A
Is
it
doing
as
well
as
the
cloud
model
that
you
originally
trained,
and
so,
if
it
is
that's
great,
maybe
you
know
they
have
comparable
performance
and
you're
you're
done.
It's
like
all
right,
great
I've
deployed
to
this
android,
I'm
good
everything
looks
good.
I
can
move
on
to
you
know
whatever
the
next
device
device
I
want
to
deploy
to,
but
a
lot
of
the
times
it's
gonna
be
like
okay.
A
No,
my
accuracy
has
dropped
a
lot,
and
so
that's
when
you
look
into
the
accuracy
or
the
output
between
each
layer
and
what
you
might
see
here
is
like
oh,
okay,
this
particular
layer.
The
output
is
very,
very
different
from
the
output
that
I
was
getting
in
my
in
my
baseline
reference
model,
and
so
then
there
you
can
go
and
look
into
okay,
my
maybe
I
have
some
pre-processing
issues,
maybe
there's
something
wrong
with
the
weights
and
biases
in
this
model.
A
When
you
know
when
you
try
to
quantize
it
so
that
it
runs
better
on
an
edge
device.
So
that
gives
you
indication
about
just
which
layer
you
should
actually
look
into
and
then
finally,
mlx
ray
has
this
thing
called
custom
search
or
custom
assertion
checks,
and
so
what
actually
happens
there
is
that
you
can
tell
mlx-ray
as
you're
running
through
my
model
and
collecting
all
these
logs
just
run
some
assertions
that
I
know
like
always
need
to
be
the
case
so
that,
if
it
fails,
I
know
there's
already
something
in
my
pipeline.
A
That
is
wrong,
and
so
an
example
of
this
is
going
back
to
the
lane
detection
case,
like
you
know
that
when
it
detects
a
lane
in
a
picture,
it
has
to
be
this
wide,
for
example,
and
so
you
can
add
an
assertion
to
be
like
whenever
you
detect
a
lane,
always
make
sure
it's
this
width
and
then,
if
it
isn't,
then
it
will
trigger
a
failure
and
can
be
like
okay,
so
there's
something
wrong.
Potentially
here.
A
And
so
how
natalie
I
kind
of
got
involved
in
mlx
right.
So
all
the
work
I
just
presented
before
was
work
that
was
done
by
those
people
I
mentioned
at
the
stanford
research
group
was
that
we
got
in
contact
with
these
people
and
we
found
there's
a
lot
of
limitations
with
mlx
ray
that
pixie,
which
we'll
talk
about
about
in
a
little
bit
could
potentially
solve
so.
The
first
is
you
have
to
make
code
changes
to
enable
this
instrumentation?
A
It's
not
a
lot
of
code
right,
you
go
and
you
start
your
inference.
You
stop
your
inference.
You
invoke
the
mlx
ray
logging
and
it's
not
that
bad,
but
it
can
be
cumbersome.
So
it's
like
you
have
to
remember
to
go
and
add
that
in
you
have
to
remember
to
take
it
out,
because
you
know
whenever
you're
ready
to
finally
deploy
to
prod.
A
You
don't
want
to
keep
just
logging
all
this
information
all
of
the
time,
and
it
has
a
slight
performance
impact,
sometimes
as
well
plus
memory,
because
now
you're
keeping
all
this
data
in
a
log
stored
on
your
device.
So
you
can
pull
it
out
later
to
analyze
it
and
the
third
thing
which
I
didn't
show
as
much
but
there's
just
not
a
lot
of
ways
to
visualize
this
data,
so
mlx
rate.
It
gives
you
an
api
for
collecting
the
data
and
parsing
it,
and
it
gives
you
some
simple
operations.
So
it's
like
okay.
A
B
All
right
thanks
michelle.
So,
as
michelle
said,
you
know
as
contributors
to
the
pixie
project,
we
were
really
inspired
when
we
saw
what
mlxray
could
do
and
what
we
wanted
to
do
is
try
to
take
some
of
its
concepts
and
see
how
we
could
use
pixi,
which
is
an
observability
platform
for
kubernetes,
to
apply
some
of
these
concepts
in
a
kubernetes
native
environment.
B
B
You
know
off
option
for
edge-based
deployments,
because
the
compute
and
the
data
collection
and
all
that
stuff
actually
lives
on
the
node
on
which
it's
collected
and
then
finally,
pixi
has
a
very
customizable
scriptable
interface,
so
that
allows
us
to
build
custom
dashboards
for
specific
use
cases
that
we
have
and,
as
she
said
before,
pixi
is
a
cncf
sandbox
project,
totally
open
source.
So
anyone
can
use
it.
B
So
I
think
the
way
we
think
about
pixi
is
progressive
instrumentation,
which
means
you
start
with
a
baseline
of
a
certain
amount
of
visibility
into
your
system,
and
you
can
extend
that
visibility
in
areas
that
you're
interested
in
so
pixi
right
out
of
the
gate.
Has
things
like
resource
utilization
flame
graph
showing
which
functions
are
taking
a
long
time
in
your
application.
B
You
can
write
custom,
scripts
and
custom
dashboards
and
soon
it's
going
to
support
things
like
supporting
arbitrary
data
sources
from
open,
telemetry,
ingest
and
things
like
that
in
the
diagram
below.
We
just
have
an
example
of
how
a
upro
works
in
your
system,
so
you
can
say
I'm
going
to
trace
this
particular
function.
B
B
B
B
I
have
that
application
running
on
my
kubernetes
cluster.
So
this
is
the
pixie
ui.
You
can
see
that
it's
showing
me
the
traffic
between
different.
You
know
applications
in
my
cluster
things
like
the
nodes
and
the
name
spaces,
and
specifically,
what
I'm
interested
in
is
this
model
server,
which
is
running
my
tensorflow
application,
which
is
an
image
classifier,
so
right
off
the
bat
we
can
see
things
like.
What's
the
cpu
usage,
what
are
the
containers
that
are
running?
What
are
the
processes
bites
in
and
out
things
like
that?
B
So
what
I
want
to
do
is
I
want
to
trigger
some
load
on
this
application,
and
what
I
want
to
do
is
I'm
going
to
apply
custom?
U
probes,
using
a
pixel
script
and
that's
going
to
allow
me
to
trace
model
execution
in
terms
of
the
number
of
times
that
it's
run
and
also
the
latency
of
the
model
execution.
B
B
B
B
Okay,
great
so,
as
you
can
see,
kind
of
the
same
thing
that
we
showed
earlier,
I'm
just
going
to
jump
ahead
to
the
the
part
where
I'm
going
to
deploy
the
pro.
So
basically,
as
I
said,
we're
basically
going
to
instrument
this
application
using
a
custom
script.
This
is
deploying
you
probes
that
I
have
written
in
bpf
trace
and
it's
basically
going
to
show
me
the
latency
and
the
number
of
times
this
model
was
executed
over
time
in
my
cluster,
without
any
redeploys.
B
B
B
So
you
might
think
okay,
it's
taking
about
a
second
to
execute.
That
might
be
a
little
bit
long.
I
need
more
information
about
what's
happening.
I
need
to
debug
this
and
figure
out
where
the
bottleneck
is
so.
I
mentioned
before
that.
Pixi
has
flame
graphs
that
it
captures
about
the
you
know,
most
cpu
intensive
functions
that
your
application's
running.
B
It
might
be
a
little
hard
to
see
here,
but
you
can
basically
see
the
entire
call
stack
of
what's
running
and
what
I'm
going
to
discover
looking
at
this
is
that
most
of
the
time
here
is
spent
parsing,
json
and
serializing
json.
It's
not
actually
spent
on
the
model
itself,
and
this
is
a
really
common
problem.
Without
the
ability
to
attribute
the
latency,
you
actually
might
blame
the
model
when
it
isn't
actually
the
model's
fault.
B
B
So
it's
kind
of
a
surprise
that
this
would
even
show
up
at
all,
especially
in
comparison
to
the
other
ops,
but
as
as
we
establish,
we
have
the
ability
to
do
custom
tracing
of
whatever
function
that
we
want.
So
what
I'm
going
to
do
here
in
the
last
part
of
the
video
is
basically
trace
arguments
to
this
non-max
suppression
operation
and
try
to
figure
out
based
on
the
arguments
that
it's
receiving,
why
it
might
be
taking
a
long
time.
B
Okay,
it's
a
little
hard
to
see
here
because
it's
small
text,
but
what
we
see
is
that
there
are
actually
thousands
of
bounding
boxes
being
passed
into
this
function,
2
000.
In
fact
that
is
a
lot
higher
than
I
might
expect
from
my
reference
pipeline.
So
this
has
allowed
me
to
get
to
the
part
of
the
issue
which
is:
why
is
this
particular
layer
of
my
model?
Slow?
It's
because
it's
receiving
an
outsized
input,
it's
an
input
that
we
wouldn't
actually
expect.
B
Okay,
it
looks
like
michelle's
mic
is
off
now,
so
I
guess
you
know
resources
the
ml
x-ray
repo
it's
on
github.
You
can
just
search
for
it
and
see
kind
of
what
they've
been
able
to
do.
It's
very
well
targeted
for
mobile
deployments,
the
ml
x-ray
paper,
which
has
been
accepted
into
ml
mlsys.
So
if
you
are
kind
of
part
of
both
worlds,
you
can
check
them
out
at
mlsys
and
then
finally,
you
can
check
out.
You
know:
pixi's
github
repo,
it's
pixi
dash.