►
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon North America 2021 in Los Angeles, CA from October 12-15. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
AI inference on the Edge cloud using WebAssembly - Michael Yuan, Second State
Michael Yuan will discuss options to run Tensorflow model inference on WebAssembly. Using a Mobilenet image classification task as an example, he will cover the plain JS approach, Tensorflow.js, pure Rust libraries for Tensorflow, and WASI-like Tensorflow extensions that run on specialized inference chips. The session will go through the journey of 60,000x performance gain over the evolution of WebAssembly. He will also discuss what’s the future for WebAssembly-based AI on the edge cloud.
A
Hello,
everyone
welcome
to
the
cloud
native
webassembly
day
in
this
talk.
We're
gonna
talk
about
ai
inference.
You
know
how
to
do
artificial
intelligence
with
web
assembly,
and
where
do
we
use
it?
It's
probably
on
the
edge
of
the
cloud
you
know,
so
we've
got
ai
inference
on
the
edge
cloud,
so
my
name
is
michael
yang
and
I'm
from
a
company
called
second
state.
A
So
this
talk
is
about
how
webassembly
make
it
easy
to
run
tensorflow
inference,
functions
in
production
on
edge
devices
and
platforms?
Okay.
Well,
so
before
we
start
getting
to
the
the
details
of
webassembly,
let's
talk.
Let's
spend
a
couple
minutes
to
talk
about
ai.
You
know
that's
deep
learning
and
ais
hot
topics
today
right
so
when
people
think
about
ai
people,
think
about
the
data,
the
algorithms
and
all
that
you
know,
that's
how
everyone
loves
to
train
the
greatest
ai.
A
However,
you
know,
after
years
of
development,
with
deep
learning,
I
think
you
know.
Most
people
in
the
industry
has
realized
that
training.
The
ai
is
not
the
difficult
part.
Now
you
know
it's
it's
pretty
much
a
low
code
work.
You
know
you
don't
even
need
to
be
a
programmer.
You
just
need
to
know
how
to
run
a
python
script
right.
You
know
how
to
collect
those
state,
collect
those
data
and
how
to
label
those
data
and
then
feed
that
into
a
computer
algorithm.
A
However,
most
of
the
work
is
actually
most
of
the
engineering
workouts
at
least,
is
actually
on
the
inference
side
is
to
when
you
have
a
ai
model.
It's
that's
that's
trained
and
when
you
use
it
in
your
production
system
to
say
to
recognize
part
of
speech
or
recognize
image
recognize
the
face.
You
know
whatever
you
train
the
the
algorithm.
For
you
know,
that's
is
the
bulk
of
the
consumer.
Part
of
the
computing
part
that's
associated
with
ai
today.
A
So,
if
you
look
at
this
graph,
you
know
that's
estimated
from
2019
and
the
training
only
takes
like
five
percent
of
the
computing
pattern.
The
influence
is
95.
So
in
order
to
deploy
your
ai
system
or
deep
learning
system
in
production
and
to
do
it
efficiently,
we
have
to
optimize
for
that.
95
percent
the
influence
part.
A
However,
there's
inference
is
very
important:
it's
95
of
the
work,
but
it's
also
difficult
to
optimize,
and
if
you
look
at
this
graph,
you
know
there's
each
dot.
Each
of
the
blue
dots
represents
a
trained
ai
model
for
image
recognition
and
on
the
xxl.
It's
the
time.
It's
required
to
run
this
model
and
on
the
y-axis
is
the
is
the
accuracy
of
this
model.
So
you
can
see
the
accuracy
goes
from
70
to
85.
A
It's
about
20
improvements
for
the
foot
accuracy,
but
the
time
needed
to
run
ranges
from
a
couple
milliseconds
to
over
30
milliseconds.
So
in
order
to
get
20
improvements
in
accuracy,
the
model
has
to
run
seven
times
eight
times
as
long.
So
that
means
it's
difficult
to
scale
those
models
right.
You
know.
A
So
when
you
have
those
models,
if
you
have
a
slightly
more
accurate
model,
you
need
a
lot
more
computing
power
so
to
save
computing
power
to
make
the
models
run
as
efficient
as
possible
becomes
a
major
issue
in
deploying
those
models
in
production.
You
know
when
you
need
to
deploy
those
models
in
production.
You
really
do
need
to
pay
attention
how
efficient
they
are,
how
efficient
you
can
make
them
wrong
right.
A
So
you
know
that's.
So,
let's
look
into
several
ways:
that's
how
people,
you
know,
run
their
tensorflow
models
for
inference
so
here.
So
the
title
of
this
slide
is
why
not
python
right,
you
know
so
the
official
tensorflow
serving
is
a
application.
The
python
application,
because
python
is
easy
to
use
a
lot
of
people,
use
it
to
train
the
model.
So
it
certainly
stands
to
reason.
People
use
it
for
inference
as
well.
A
Once
you
train
the
model
in
python,
you
can
see
the
modeling
action
in
python,
then
perhaps
you
can,
you
can
use
it
in
production
as
well.
However,
python
as
a
language
is
very
inefficient.
You
know
there
has
been
a
study.
That's
you
know,
there's
it's
a
famous
study
that
published
on
the
journal
science.
It's
you
know.
The
authors
took
a
machine
learning
algorithm
and
the
implemented
in
python
and
then
c.
A
A
As
a
result,
you
know
when
you
use,
when
you
say
I'm
doing
machine
learning
or
tensorflow
in
python
it's
actually
it
doesn't
actually
do
both
of
the
box
of
work
in
python.
You
know
it's
under
the
hood.
The
python
functions
actually
cause
the
c
libraries.
This
tensorflow
library
is
written
in
c
right,
so
the
the
workload
is
actually
done
in
native
code
reading
c.
A
So
you
know
that's,
that's
sort
of
you
know
how
people
get
reasonable
performance
autopilot.
You
know.
Python
is
mostly
just
the
interface
language.
You
know.
That's
makes
the
system
easy
to
program
and
easy
to
use
by
you
know
by
the
developers
right,
you
know
that's
and
then,
when
you
really
need
heavy
lifting,
when
you
really
need
computation,
you
goes
down
to
the
c
libraries.
It
goes
down
to
the
native
library,
tensorflow
libraries
reading
c
right.
However,
even
then
the
data
pre-processing
and
post-processing
are
still
written
in
python
right.
A
You
know
because
when
you
read
say,
if
you
have
a
you
know
image
recognition
model,
that's
you
have
to
read
the
image
and
resize
it
and
flatten
it
to
make
it
appear
to
be
the
size
of
the
tensor.
You
want
and
then
feed
that
to
the
tensorflow
model
and
then
to
get
the
the
result
from
the
model
and
then
map
the
model
result
back
to
a
metadata
file
or
label
file
to
figure
out.
A
What's
the
english
name
for
the
objects
that
are
that's
on
that
image
right
so
that
process,
only
10
of
the
work
is
actually
done.
Running
the
tensorflow
model
itself
in
negative
code
or
in
c
library,
and
the
most
of
the
data
pre-processing
and
post
processing
are
still
written
in
python.
You
know
if
you,
if
you
are
you,
if
you
are
writing
this
entire
application
in
python,
so
ninety
percent
will
work,
is
slow
and
can
be
heavily
optimized.
A
That's
standing
in
front
of
your
door.
It
would
be
able
to
tell
whether
it's
it's
a
person
in
the
household
or
it's
somewhere
else
right.
You
know
you
can
unlock
the
door
or
you
know,
do
you
can
do
things
like
that
and
for
things
like
that,
you
want
it
to
run
to
you.
You
want
the
ai
model
to
run
either
on
the
camera
or
in
the
house
on
the
router
or
in
the
server
in
the
house.
You
know
something
of
that
nature.
A
You
don't
want
it
to
go
all
the
way
back
to
the
internet
and
not
only
because
the
performance
would
be.
You
know
to
run
on
the
remote
server
say
in
new
york
right.
You
know
that
would
be.
You
know,
you
would
see
a
big
performance
impact,
but
also
because
you
don't
want
the
data
to
leave
your
house
because
for
privacy
reasons-
and
of
course
you
know
another-
you
know
very
important
use
case
is
self-driving
cars
right.
A
You
have
cameras
and
all
that
and
you
need
those
those
data
to
be
processed
in
real
time
on
the
edge
inside
of
the
car
not
go
back
to
somewhere,
not
not
not
go
to
the
cloud
and
process
and
then
return
to
the
internet.
So
in
in
many
of
those
cases,
python
doesn't
run
any
of
those
devices.
You
know
it
doesn't
run
very
small
devices,
it
doesn't
run
a
very
old
linux
operating
system.
A
It
doesn't
run
on
real-time
operating
system,
you
know,
so
there
are
lots
of
issues
and
python,
of
course,
offers
limited
support
in
web
and
in
web
and
application
frameworks.
You
know,
there's
you
know,
there's
only
certain
frameworks
that
works
with
python
and
there's
many
others.
That's
that's.
You
know,
for
instance,
in
node.js
it
would
be
difficult
to
call
python
program
from
node.js
it's
possible,
but
it's
not
as
easy.
A
So
the
little
table
I
have
here
shows
the
performance
characteristics
of
different
programming
languages
and
you
can
see
c
plus
plus,
but
of
course,
by
far
is
the
best.
The
ross
will
come
to
a
very
close
second
in
java.
You
know
it
runs
fast
enough,
but
it
consumes
a
huge
amount
of
memory
100
times
more
memory
than
everyone
else
and
for
python
it
runs
very
slow,
it's
100
times
slower
than
everyone
else
and
consumer
reasonable
amount
of
memory.
A
A
A
Where
you
have
you
know,
you
may
have
a
variety
of
different
cpus
and
and
operating
systems
on
different
devices
right
and
it
cannot
be
managed
and
orchestrated
because
it's
a
native
application
just
to
start
it
and
it
could
crash
and
it
cannot
be
managed
and
stopped.
You
know
that's
it's
difficult
to
do
that
and
it's
it's
not
safe.
It's
crashes
from
bugs
or
attacks.
You
know,
that's
it
could
you
know
you
could
make
a
mistake
and
it
crashed
the
whole
system
right.
It's
costly
grain!
There's!
A
No,
you
know
the
access
control
is
nice,
the
operating
system
level.
You
know
so
it's
whoever
the
user,
the
the
you,
the
the
user,
who
runs
the
application
with
the
the
application,
will
have
the
same
privilege
and
permission
as
a
user
who
runs
it
right.
You
know.
So
if
you
have
root
user
response
application,
you
have
access
to
the
entire
system
and
which
is
generally
a
bad
thing.
So
the
native
applications
there's
a
lot
of
issues
with
native
applications.
A
That's
why
you
know
20
years
ago
java
gained
popularity
because
of
jvm
and
after
that
you
know,
docker
gained
popularity
as
well.
You
know
that's
because
it
provides
isolated
around
time
and
portable
around
that
for
developers
to
run
their
applications,
so
we
don't
want
just
native
applications.
We
want
something
that
is,
that
is
more
modern,
but
also
have
high
performance,
not
slow
performance
like
python
right
so
whereby
somebody
to
the
rescue
you
know
that's,
so
we
have
two
conflicting
goals.
A
A
So
whether
somebody
has
some
really
interesting
advantages
over
you
know
over
both
fully
managed
application
and
also
over
native
applications.
The
first
is
high
performance,
it's
much
more
performant
than
say
python
or
node.js
or
javascript
or
or
even
java,
in
the
jvm,
and
this
has
sandbox
safety.
It's
a
it's!
It's
a
sandbox
that
contains
the
the
the
native
application.
So
it
has
safety.
It's
got,
capability-based
security,
meaning
that's!
When
you
start
a
web
representative
vm,
you
can
give
it.
You
know
a
set
of
policies
or
rules.
A
What
kind
of
system
resources
you
can
access.
So
it's
that
allows
the
webassembly
runtime
to
have
a
different
set
of
security
permissions
than
users
who
run
it
right
and
it's
language
agnostic
in
in
a
way.
So
you
know
that's
meaning
that
any
front-end
language
and
chemi
program
that
can
be
compiled
to
lvm
is
going
to
run
on
webassembly.
Although
language
that
requires
a
large
runtime
become
somewhat
impractical
to
run
on
web
assembly,
for
instance,
you
know
it's
difficult
to
get
java
or
python
running
on
it.
A
However,
for
rust
c
swift
tiny
go,
and
you
know
languages
like
that.
You
know
that's
it's.
You
can
compile
all
those
languages
to
wherever
somebody
and
just
run
them
in
the
virtual
machine.
Once
a
virtual
machine
and
the
last
thing
is,
we
see
a
product
community
fit
here.
You
know,
meaning
that
you
know
there's
a
large
already
a
large
community
for
for
web
assembly
products
and
there
are
lots
of
people
contributing
which
we
thought
it's
a
it's
a
crucial
factor
for
success
of
technology
right.
A
So
to
sum
it
up,
you
know,
from
our
point
of
view,
we
see
the
raster
programming
language
on
the
front
end
and
web
assembly
on
the
on
the
back
end
roughly
equivalent
to
the
java
programming
language
plus
the
jvm
about
20
years
ago.
So
we
believe
maybe
this
thing
can
can
be
as
important
or
as
a
group
as
big
as
java.
A
So
this
is
the
web
assembly
runtime
that
we
have
donated
to
to
the
cncf.
You
know
if
you're
interested,
you
know
just
like
it
on
github
or
you
know,
fork
it
or
check
out
just
generally
check
it
out
and
see.
You
know
this
is
a
a
popular
web
assembly
vm
space,
specifically
optimized
for
high
performance
applications
outside
of
the
browser.
So
there's
a
bunch
of
web
assembly
vms,
that's
designed
for
the
broader
use
case,
but
this
is
not.
This
is
designed
for
the
server
for
the
use
case
outside
of
the
bar.
A
A
So
in
the
next
couple
of
slides,
I
want
to
talk.
You
know
using
welcome
edge
and
ss
vms
examples
to
talk
to
you
to
to
discuss
whatever
somebody's
approaches
to
tensorflow
inference.
How
do
you
do
tensorflow
influencers
well
assembly?
We
all
know
how
to
do
it
in
cc,
plus
plus.
We
also
know
how
to
do
it
in
python,
but
how
to
do
that
in
webassembly
and
what
are
the
implications?
A
So
the
first
easiest
one
to
for
developers
to
understand
is
to
just
run
web
assembly
interpreted
on
the
cpu.
You
know
so
the
tensorflow
operators
are
implemented
in
rust
or
c
plus
plus.
It
is
then
compiled
the
web
assembly
and
then
on
the
host
machines
cpu.
We
execute
this
program
and
read
a
tensorflow
model,
and
then
you
know
execute
those.
Those
operators-
and
here
is
example-
you
know-
there's
we
have
a.
We
have
an
example
of
this
approach
this
in
our
repository.
A
It
does
image
recognition,
so
it
takes
about
10
minutes
to
do
to
recognize
to
do
an
image
recognition
on
a
modern
cpu.
So
it's
a
very
long
time.
You
know
the
performance
is
obviously
not
acceptable,
but
you
know
that's,
but
that's
that's,
perhaps
the
most
straightforward
way
to
do
it
right.
You
know.
So
you
know
you
provide
the
implementation
of
the
tensorflow
operators
in
new
web
assembly
and
then
you
know
use
the
interpreter
to
to
to
run
them.
You
know,
that's
the
that's!
Our
benchmark
takes
about
10
minutes.
A
A
much
better
way
is
to
do
the
just
in
time
compilation.
So
it's
the
same
web
as
a
web
assembly
application,
but
it
runs
in
a
jit
runtimes
like
v8,
for
the
gi.
The
way
git
runtime
works
is
that
it's
no
longer
interprets
web
assembly
instructions
step
by
step.
It
tries
to
compile
it
as
it's
run.
It
right,
you
know.
So
most
of
the
execution
happens
in
the
native
environment,
so
in
that
case
we
immediately
see
a
300
to
500
times
boosting
performance.
You
know
the
same
image.
A
A
Of
course,
we
can
go
one
step
further
to
say
you
know
the
jit
compiler
compiles
things
as
it
goes
right.
You
know
so
as
a
vm
and
executes
the
program
it
compiles
it,
we
can
compile
ai
models
directly
into
webassembly
bytecode
programs
that
allows
us
to
save
the
time
to
need
it
to
map
tensorflow
operations
to
web
assembly
operations
at
runtime
right.
You
know,
because
tensorflow
model
consists
of
a
series
of
tensorflow
operations,
the
the
web
assembly
program.
What
it
does
is
read
the
tensorflow
model
and
then
execute
those
those
those
those
operations.
A
A
So
in
that
case
the
image
recognition
task
takes
one
to
two
seconds
which
I
I
have
to
say
it's
more
or
less
acceptable
for
internet
applications.
Now
you
know,
that's
if
you
have
something
in
the
web
page
on
a
web
page
that
it
can
recognize
a
face
in
one
to
two
seconds.
I
think
that's
for
most
people.
It's
acceptable
right
so
representative.
Examples
of
those
is
the
tbm
project,
apache
tbm
project
which
compiles
tensorflow
models
directly
into
webassembly
and
on
the
second
stage,
qualcomm
work
with
for
the
onyx
comparator
for
wasm
compiles.
A
You
know,
onyx
model
ai
model
directly
into
webassembly
and
runs
on
qualcomm
hardware.
Let's
take
it
one
step
further,
you
know
because,
let's
recall
how
python
does
it
python
does?
You
know,
does
not
call
tensorflow
library
in
the
interpreter
in
the
interpreter
or
in
a
jet
compiler.
What
it
does
is
that
when
the
python
program
needs
to
run
a
tensorflow
model,
it
drops
out
of
the
python
and
into
the
native
c,
plus
plus
you
receive
environments
and
runs
the
tensorflow
model
there,
and
then
it
gets
a
result
and
it
goes
back
to
the
python
environment.
A
We
can
do
exactly
the
same
with
the
web
assembly
as
well,
so
we
can
call
native
web
summary
libraries
from
we
can
call
native
tensorflow
libraries
from
webassembly.
You
know
we.
We
provide
a
roster
api
that
compiles
down
to
tensorflow
cost
and
then
your
then
your
image,
then
your
data,
preparation
and
the
post
processing
work
can
still
be
written
in
the
height
in
a
in
a
highly
optimized
and
high
performance
language
like
rust.
So
to
do
so,
we
can
show
that's
the
image.
A
Recognition
task
now
takes
half
a
second,
that's
2000
times
improvement
from
the
original
interpreted
mode
right.
You
know.
So
an
example
of
this
is
the
ssvm
with
wazi
tensorflow.
It's
a
rust
sdk
and
the
web
assembly
extension
that
we
wrote
for
the
for
for
the
ssvm,
which
we
donated
to
cncf,
of
course.
A
Now,
because
the
tensorflow
model
is
wrong
in
the
native
library
right
in
written
by
reading
cc,
plus
that
particular
native
library
can
now
take
advantage
of
the
the
hardware
features
on
that
particular
device
right.
So
we
can
use
you
know
a
head
of
compilation,
techniques,
aot
compilers,
to
optimize
for
data
preparation
tasks
that
seem
wasn't.
We
can
run
the
tensorflow
library
as
specialized
hardwares
such
as
you
know,
ai
chips
or
gpus,
and
you
know
things
like
that,
and
the
image
recognition
task
now
takes
sees
a
another
10
volt
improvements.
A
It
only
takes
0.05
seconds
to
recognize
the
image
right.
You
know
that's
in
that
case
we
can
do
20
image
recognitions
per
second.
The
full
video
speed
is
30
frames
per
second
right,
so
it's
close
to
a
to
real-time
video.
So
if
we
have
a
real-time
video
feed,
we
will
be
able
to
run
a
model
on
it
and
see
what's
in
in
each
of
the
each
of
the
video
frames,
so
the
ssvm
on
tencent
serverless.
It
shows
an
example
of
this.
It's
a
it's
a
it's
a
very
nice
example.
A
A
A
I
have
just
discussed
as
a
rosi
tensorflow
approach
right,
you
know
so
there's
a
function
in
that's
written
in
rust,
it's
called
infer
and
the
info
function
takes
a
data
array
that
is
that
represents
the
image
and
come
back
with
a
string
which
is
a
text
description
of
what's
in
the
image
right.
So
that's
the
message,
signature
and
in
this
screen
you
can
see
it's
read
the
image
and
then
it's
resize
the
image
and
then
flatten
the
image
into
a
predefined
format.
That's
can
be
consumed
by
the
by
the
tensorflow
model.
A
Now
in
the
next.
In
this
slide
we
read
the
tensorflow
model
and
we
read
the
labels
associated
with
the
model,
meaning
you
know,
there's
there's
you
know
when
we
tell
we
try
to.
We
try
to
train
the
model
for
what's
on
the
picture,
we
give
the
model
labels
right.
You
know
this
picture
has
caps
this
picture
stock.
A
You
know
things
like
that,
so
those
labels
are
in
this
label
file
and
then
we
have
a
really
simple
api
to
run
to
our
tensorflow
session,
and
then
we
gave
the
flattened
image
as
input
and
then
gets
the
output,
the
predictions
output
from
the
from
the
tensorflow
model.
So
in
in
this
process
we
drop
to
the
operating
system
and
run
this
model
and
get
the
result
back
now
we
get
the
result
back.
The
result
is
in
the
array.
A
It's
in
array
of
probabilities,
you
know,
say
if
the
if
the
data
has,
if
the
model
has
a
hundred
associated
labels,
you're
gonna
get
a
hundred
numbers
and
each
number
corresponding
to
the
probability
of
that
label.
On
that
image
right,
you
know.
So,
if
the,
if
the
label
corresponding
to
cats
is,
has
probability
of
0.8,
that
means
there's
80
percent
of
chance.
This
this
picture
contains
a
cat
right.
A
So
here
we
just
sort
the
return,
tensor
the
return
array
and
find
the
object
with
the
largest
probability
find
the
the
index
with
the
largest
probability
and
then
from
there.
We
we
go
to
the
the
labels
file.
The
metadata
file
to
find
out,
what's
that's
largest
probability,
correspond
to
and
then
generate
an
english
description
for
the
model
right
you
know.
A
A
So
what's
next
there's
there
are
lots
of
things
that
the
field
is
still
young
and
but
I
think
it
solves
a
real
big
problem.
You
know
it's
how
to
deploy
production,
ready,
ai
inference
application
right.
You
know
in
the
manner
that
is,
that
is
secure.
That
is
safe.
That
is,
you
know,
developer
friendly
right.
You
know
so
there's
many
things
that
we
can
do
to
improve
what
we
have
done
so
far
right.
A
First,
we
can
support
the
was
in
neural
network
specification,
which
is,
which
is
a
specification
developed
by
the
community
to
unify
you
know
the
different
ai
mod
different
ai
model
approaches
right.
So
the
approach
we
took
is
very
so
far
it's
very
tensorflow
specific.
It's
called
wazi
tensorflow
right,
so
the
washi
n
specification
can
accommodate
more
more
ai
frameworks
and
we
are
committed
to
work
with
people
in
that
specification
group
and
provide
more
sdks
for
front-end
languages,
which
we
just
saw.
A
We
have
a
rust
sdk
and
we
call
those
api
singles
sdks
to
tensorflow
inference,
but
we
can
do
that
for
for
other
languages
that
supported
our
website
as
well,
and,
on
the
other
hand,
other
end
is
to
support
more
ai
frameworks
on
the
back
end
of
tensorflow
beyond
the
tensorflow
right.
So
there's
onyx
models,
pedal
panel,
tangent,
mk
net
and
mxnext.
A
You
know
things
like
that.
You
know
those
you
know
you
can
just
the
idea
is
that
you
can
just
plug
in
a
different
model
and
then
run
the
exact
same
code,
and
then
you
would
be
able
to
utilize
that
model
right
more
ai
hardware.
You
know,
that's,
that's!
Could
be
supported
by
those
very
much
so
I
think
my
time's
up
and
thank
you
very
much
for
watching-
and
I
have
another
talk
later
in
the
afternoon
to
talk
about
how
web
assembly
can
become.
A
It
can
be
used
as
a
serverless
runtime,
and
a
lot
of
this
that
use
case
is
already
illustrated
here.
When
we
develop
a
ai
inference
function,
we
actually
deploy
it
as
a
serverless
function,
either
our
own
infrastructure
or
in
tencent
serverless
or
aws
lambda.
You
know
that
allows
application
developers
to
just
write
code,
upload
code
and
then
boom
have
the
ai
influence
functions
in
production
ready
for
everyone
to
use.
So
I
hope
you
enjoyed
this
talk.
Go
to
our
github
repository
on
github.com,
second
dash
state
ssvm.