►
From YouTube: WebAssembly Based AI as a Service on the Edge with Kubernetes - Rishit Dagli & Shivay Lamba
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2023 in Amsterdam, The Netherlands from April 17-21. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
WebAssembly Based AI as a Service on the Edge with Kubernetes - Rishit Dagli, Narayana Junior College; Incoming University of Toronto & Shivay Lamba, Meilisearch
WebAssembly (WASM) is being adopted at an increasing rate for edge applications. That allows WASM runtimes, such as WasmEdge (a lightweight and high-performance runtime for cloud-native, edge, and decentralized devices), to run serverless functions on the edge. Following the large-scale adoption and benefits of serverless computing, we focus on deploying these as a Function-as-a-service on edge devices. Machine Learning inference is often a computationally intensive task and edge applications could greatly benefit from the speed of WebAssembly. Unfortunately, Linux containers end up being too heavy for such tasks. Demonstrating Machine Learning deployments in such a fashion, another problem we face is that the standard WebAssembly provides very limited access to the native OS and hardware, such as multi-core CPUs, GPUs, or TPUs which is not ideal for the systems we target. The talk also shows how one could use the WebAssembly System Interface (WASI) to get security, portability, and native speed for ML models. To top it off this talk ends with a demo of deploying a Machine learning model as a serverless function using WASM deployed on an edge device.
A
Okay,
we'll
get
started
hello
and
in
the
next
30
minutes,
we'll
be
talking
about
web
assembly
based
AI
as
a
service
on
the
edge
with
kubernetes
I'm
rishat
and
I
am
an
I'm
an
under
a
CS
undergrad
student
at
the
University
of
Toronto
and
I'm
and
I
also
work
on
the
machine.
Learning
team
on
Finch
at
SpaceX
should
I.
B
Yeah
hi
I'm
ashuvai
and
I'm
a
contributor
and
maintainer
at
layer,
5
the
service
mesh
Community
within
the
cncf
and
mostly
developer
relations,
engineer,
admin,
research,
it's
a
open
source,
raw
space
search
engine.
So
really
looking
forward
to
be
sharing
with
all
of
you.
How
we
can
leverage
the
use
of
webassembly
to
enable
very
easy
access
to
functions
as
service
within
the
edge.
A
So
I
want
to
start
out
with
this
meme
when
I
start
talking
about
webassembly,
which
is
that
webassemblies
neither
web
nor
assembly-
and
this
is
an
interesting
me
now
Meme-
and
this
also
depends
on
what
you
consider
the
definition
of
web
and
we
want
to
run
it
on
more
than
just
web.
We
want
to
run
it
on
edge
devices,
that's
what
we
are
excited
about.
A
So
what
is
webassembly
if
it's
not
web
and
not
assembly,
and
why
should
you
care
about
it
if
you
are
deploying
Edge
applications,
especially
machine
learning,
applications
which
are
quite
compute
intensive?
So
that's
what
we
want
to
talk
about
so
simply
put
it's
a
binary
instruction
format
for
a
stack
based
virtual
machine
and
what
that
means
is
it's
analog
is
to
machine
code,
a
higher
level
than
machine
code,
and
it's
essentially
designed
as
a
compilation
Target.
So
you
would
have
you'd
probably
have
some
JavaScript
functions.
A
Some
Rush
functions
then
use
webassembly
to
make
to
make
a
compilation
Target
out
of
it.
So
so
then
you
would
probably
use
webassembly
make
a
DOT
make
a
webassembly
module
out
of
it
and
then
run
it
with
webassembly.
So
yeah,
you
can
essentially
take
take
your
code
or
take
your
functions.
Take
your
modules
from
other
languages
and
compile
them
down
to
webassembly,
which
is
the
idea
and
and.
B
The
idea
is
that
you
can
see
that
it's
not
just
your
functional
programming
languages,
but
also
scripting
programming
languages
like
python
JavaScript
that
can
be
compiled
into
webassembly.
So
it's
kind
of
a
polygote
environment
where
you
can
use
any
of
your
standard.
Like
you
know,
languages
and
that
can
be
compiled
into
a
web
assembly.
Target.
A
C
A
It's
also
open
and
debuggable
webassemblies
open
source,
which
is
why
we
are
presenting
about
some
of
the
work
we
have
been
doing
with
webassembly,
and
it
can
also
run
on
non-web
platforms,
which
is
kind
of
why
you
should
think
about
using
it
for
your
Edge
applications
and
it's
also
based
on
the
open
web
platform.
So
you
can
also
use
it
on
the
web
for
Edge
use
cases
where
you
might
have
not
a
lot
of
bandwidth,
not
a
lot
of
network.
A
You
can
also
use
it
for
that,
and
the
webassembly
security
model
is
something
really
interesting
as
well.
It's
not
just
simple
virtualization
and
you
get.
You
tend
to
get
a
lot
of
security
with
webassembly
too
so
I
want
to
talk
about
how
so
Shiva.
Would
you
like
to
talk
about
how
awesome
comes
into
the
picture.
B
Sure
so,
as
we
kind
of
recap
over
the
past
few
slides,
what
we
have
seen
is
that
you're
primarily
getting
a
lot
of
high
performance
byte
code.
And
why
is
that
the
case?
Because
what
we're
doing
is
that
we
are
taking
your
high
performance
functions,
written
in
languages
like
rust
and
they're,
getting
directly
compiled
to
these
byte
codes.
So
one
of
the
common
use
case
where
how
web
assembly
started
was
on
the
web,
where
JavaScript
functions
that
couldn't
be
that
performant.
B
B
We
are
also
expanding
it
to
server
side
and
on
the
edge
as
well
and
you're,
still
doing
that
that
we
can
take
your
high
performance
functions,
written
very
popular
languages
like
C,
plus,
plus
or
rust,
and
convert
them
into
this
byte
code
and
then
run
them
because
of
the
small
size
that
comes
with
this
web
assembly
module
and
as
we
mentioned,
that
it's
compatible
with
all
of
these
various
programming
languages.
And
since
it's
a
binary
instruction
format.
So,
like
you
know,
destroying
it
and
initializing
it
at
the
lowest
level
is
very
easy
to
do.
B
A
So
on
this
point,
I
would
also
like
to
add
that
so,
if
you
think
about
something
running
at
Native
speed,
the
first
thing
you
think
is
it's
not.
You
cannot
put
the
same
thing
on
multiple
platforms
and
well
that's
right
to
a
certain
extent,
especially
in
the
case
of
ahead
of
time
compiling.
So
if
you
take
a
take
one
of
the
popular
version,
runtimes
like
washroom
Edge
and
try
to
ahead
of
time,
compile
it
into
a
DOT
SE
file,
which
brings
a
lot
of
the
Native
execution
speed.
A
It's
also,
then
tied
to
the
machine
type.
So
webassembly
in
general
is
not
not
tied
to
what
machine
it
is
running
on
it's
just
an
instruction
format.
You
can
run
it.
If
you
try
to
do
something
like
ahead
of
time,
compile
it
down,
you
would
essentially
get
native
speed,
but
it
would
be
tied
to
what
kind
of
machines
it
can
run.
B
So
one
of
the
other
benefits
that
we
get
is
that
with
webassembly
we
are
primarily
looking
at
again
the
portability
aspect
of
being
able
to
run
these
awesome
byte
codes
anywhere
that
we
want,
because,
usually,
if
you're
using
let's
say
a
python
based
or
a
JavaScript
based
function,
it
will
be
more
kind
of
limited
and
tied
with
the
dependencies
on
the
specific
platform
that
you
are
running
it
on.
So
the
platform
dependency
is
also
something
that
improves
with
using
bosom
and,
of
course.
B
Finally,
when
we're
talking
about
machine
learning,
we
know
that
it's
very
competition,
intensive
mathematical
combination
intensive,
so
especially
if
you're
running
it
on
edge
devices.
You
want
high
speed,
given
the
lower
constraint,
like
give
the
constraints
that
you
have
with
the
size
and
also
with
the
competition
that
you
get
with
running
a
edge
device.
So
it's
really
good
for
being
able
to
do
model
inference,
conversion
and
deployments
as
well
on
the
edge
and
that
also
very
quickly,
and
that
brings
us
to
water
mix.
Would
you
reset.
A
So
a
washroom
image
is
a
popular
runtime
to
allow
you
to
easily
run
version
for
Edge
devices
and
which
is
also
what
we
will
be
showing
in
the
demos
and
talking
a
bit
about
today,
and
so
we
want
to
talk
a
bit
a
bit
more
about
how
does
version
run
on
edge
devices,
which
is
very
interesting
because
we
talked
about
all
of
these
benefits.
But
we
didn't
really
think
talk
more
about.
How
does
it
essentially
run
the
same
code
on
all
devices?
A
How
does
it
do
this?
So
what
so?
What
it
allows
you
to
do
is
have
a
fast
scalable,
secure
way
to
run
your
sim
code
and
by
safe
and
portable.
We
also
mean
that
users
and
programs
can
only
access
what
they
have
the
right
to
access,
and
we
also
mean
that
this
that
one
process
should
not
create
problems
for
other
processes.
A
So
essentially
we
want
standard
platform,
independent
methods,
so
we
want
to
have
our
system
called
standardized
and
these
should
be
independent
so
which
is
where
so,
which
is
where
webassembly
system
interface
comes
in
was
he
was
he
which
solves
a
lot
of
these
problems?
And
so
what
was
he
does?
Is
it
allows
you
to
have
standard
platform?
A
Independent
system
calls-
and
this
is
a
very
popular
tweet
from
the
creator
of
Docker,
and
he
says
that,
if
version
plus
was
he
existed
in
2008,
there
would
have
been
no
need
to
create
Docker,
and
so
could
you
go
to
the
next
one?
So
so
I
want
to
motivate
this
and
talk
more
about
waji
by
starting
with
a
statement
which
is
wrong
that
she
directly
gives
you
access
to
all
system
resources
which
it
does
not
because
it's
far
too
important
for
stability
and
security.
A
So
so
the
way
version
works
is
I've.
Taken
this
image
from
Lynn,
Clark
and
the
way
and
and
the
way
all
applications
work
is
the
RC
kernel.
Can
I
do
this
task
and
to
which
the
so
to
which,
instead
of
having
all
of
these
platform
specific
ways
to
do
system
calls
was
he
allows
you
to
have
standard
libraries?
A
B
Absolutely
so
so
viscussed
is
the
application
of
how
was
image
as
a
popular
runtime.
With
the
idea
of
time.
Comparison
can
be
used
to
very
quickly
do
things
like
modern
difference,
especially
from
a
machine
learning
perspective,
but
also
you
can
do
it
for
other
high
conditional
tasks.
So,
let's
kind
of
understand,
where
does
communities
come
into
the
picture?
B
So
you
can
use
the
system,
resources
and
a
bunch
of
different
libraries
and
functions
that
you
can
directly
get
from
your
Docker
containers,
whereas
with
webassembly
you
get
very
fast
execution
and
fast
load
times,
especially
if
you're
having
heavier
Docker
containers.
They
can
take
a
lot
of
time
to
load
up
initially
when
you're,
like
you
know,
in
the
warm-up
stage.
So
that's
the
benefit
that
you
get
with
the
awesome
containers
as
well,
because
they
are
smaller
in
size
and
they
have
a
much
faster
like
a
load
of
speed.
B
So
the
idea
is
that
we
are
supposed
to.
We
can
basically
run
these
side
by
side
in
the
entire
Community
stack,
and
this
is
how
it
the
goal.
Like
you
know
it's
that
if
you
take
a
look
at
the
entire
kubernetes
landscape,
you
want
to
be
able
to
run
you.
You
want
to
be
have
to
be
able
to
have
all
of
your
different
communities
applications
and
then
the
high
level
and
the
low
level
container
runtimes
and
run
wasm
as
part
of
this
entire
ecosystem,
to
be
able
to
complement
each
other.
B
A
A
As
well
as
as
well
as
running
tensorflow
Lite
models
with
volume,
as
well
as
running
tensorflow,
Lite
modules,
with
volume
and
iot
compilation,
if
that's
what
you're
excited
and
will
not
go
very
deep
into
those
benchmarks
right
now,
but
if
that's
what
you
are
excited
and
we
have
a
GitHub
repository
and
you
will
find
all
about
the
differences
in
time.
It
takes
another
thing.
I
just
wanted
to
quickly
mention
before
we
go
forward
is
when
we
were
talking
about,
was
image
and
kubernetes
or
version
and
kubernetes.
A
We
particularly
want
to
share
that.
The
idea
of
having
Linux
containers
on
volume,
containers
side
by
side
is
a
great
one,
because
we
want
you
to
take
a
look
at
running
your
Edge
applications
with
volume,
but
we
don't
want
to
to
run
all
the
processes
on
version
and
I
would
again
like
to
take
this
by
an
analogy
and
say
CPU,
GPU
and
tpus.
If
you
are
running
a
YouTube
video,
you
probably
don't
want
to
use
a
TPU
there,
which
is
the
same
case
over
here
too.
A
A
So
I'll
start
out
with
a
demo
about
so
I'll
start
out
with
a
demo
about
so
I'll
start
out
with
a
demo
about
running
was
on
my
jobs
and
we
have
a
few
demos
over
here.
I'll
start
out
with
a
demo.
I
had
been
preparing
just
yesterday,
and
this
is
running
a
mobilenet
V2
model
running
a
mobile
net.
V2
model
with
watch
image
and
I
want
to
first
show
how
to
run
this
app
locally.
A
So
we'll
do
that.
This
is
also
in
a
GitHub
repository.
I
have
spent
some
time
writing
some
docs,
so
people
can
understand
and
yeah
that's
what
we'll
be
doing
today.
So
I'll
start
out,
so
this
is
actually
a
rest
application
and
let's
go
I
I
open
in
what
my
main
dot
RS
looks
like.
So
we
we
are
essentially
using
track
tensorflow,
which
is
a
pretty
popular,
which
is
a
pretty
popular.
A
A
way
to
oh
yeah.
I.
Would
like
to
zoom
in
a
bit
yes,
so
we
are
actually
I'm
actually
using
track
tensorflow
over
here,
which
is
a
pretty
popular
way
to
run
machine
learning,
on
machine
learning
with
rust.
It's
a
pretty
popular
library
and
it's
actually
multi-threaded
version
doesn't
support
multi-threads.
So
we
have
made
some
changes
into
how
track
is
loaded?
How
truck
does
some
stuff
to
make
to
still
make
it
work,
but
essentially
what
I
want
to
quickly
show
over
here
is
that
we
have.
A
We
have
a
tensorflow
model
over
here
which
is
being
loaded.
We
are
doing
some
pre-processing
just
ideal
stuff,
putting
in
an
image,
and
then
we
are
just
running
the
model.run,
so
the
t-wake
is
something
I
also
want
to
talk
about,
which
is
how
how
version,
how
version
interacts
with
the
state
and
yeah
so
I'm,
simply
running
a
model
inference
over
here
interest,
and
what
I
want
to
do
now
is
show
running
this
locally
to
do
that.
A
I'll
first
start
out
with
building
with
building
this
rust
application
as
a
as
a
web
assembly
module,
and
to
do
that,
I'll
specify
the
I'll
specify
my
target
to
be
volume
30
to
minus
4C.
You
do
have
to
add
this
install
the
version
32
minus
wasi
Target,
which
I've
already
done
so
I'll
just
start
I'll
just
start
with
the
building.
I'll
just
start
with
building
this
rust
application
and
at
the
Target
to
be
version
32
minus
4C.
So
it
knows
that
it
needs
to.
B
It's
going
to
set
the
stone
like
over
here
that
this
is
where
this
is
Step,
where
we're
taking
a
rust
function,
and
this
could
be
used
with
any
other
programming
language
as
well,
for,
for
example,
with
C
plus
plus
you'll,
be
using
M
script
and,
if
you're,
using
like
tiny,
go
or
any
other
language.
This
is
the
step
where
we're
generating
the
byte
code.
The
awesome
bytecode
that
can
be
used
anywhere.
A
A
Okay,
so,
regardless
of
that,
what
I
did
want
to
show
you
was,
you
could
so
I
seem
to
have
some
dependency
error,
but
I
seem
to
have
some
dependency
error,
but
the
GitHub
repository
does
have
the
right
one,
at
least
okay.
So
what
I
wanted
to
show?
You
is
how
you
could
easily
make
it
as
a
compile
it
down
as
a
web
assembly.
So
what
you'd
get
is
a
DOT
was
awesome
file.
A
Now
the
dud
wasn't
file
can
run
anywhere
and
I
also
wanted
to
show
you
aot
compilation,
which
you
can
also
do
with
washroom
time
and
watch
image.
We
are
just
showing
an
example
of
aot
compiling
with
version
time
here
and
so
that
that
would
compile
it
down
to
a
DOT
C
awesome
file.
You
can
also
get
the
Linux
shared
Library
format
to
aot,
compile
this
and
then
I
wanted
to
run
it
locally.
A
I
also
wanted
to
talk
a
bit
about
the
webassembly
text
format,
so
the
web
assembly
text
format
allows
you
to
get
your
webassembly
module
as
a
bit
readable
code,
which
is
pretty
helpful
for
debugging,
so
I
just
wanted
to
show
that
you
can
get
it
dot
w80
file
as
well.
All
of
this
is
up
on
a
GitHub
repository,
so
I
mean
feel
free
to
try
it
out
for
yourself.
I,
unfortunately,
could
ensure
it
here.
So
I'll
go
back.
B
All
right,
and
also
what
we
want
to
showcase
is
the
future
of
service
functions
right.
So,
as
we
move
ahead
from
writing
functions
and
having
to
use
managed,
Hardware
or,
like
you
know,
resources
to
be
able
to
run
these
Services,
we
are
of
course
moving
towards
serverless
platform,
which
inherently
has
a
number
of
benefits
as
compared
to
your
standard.
Like
you
know,
Linux
containers
or
services
that
you
might
be
using.
B
So
if
you,
instead
of
like
the
standard
virtual
machines,
because
you
get
on-demand
service
and
then,
of
course,
it's
very
easy
to
scale
it
up
and
down
as
compared
to
your
standard
virtual
machines.
So
that's
where
webassembly
is
also
moving
to
be
a
really
popular,
functional,
Service
platform
as
well,
and
that's
what
we
also
want
to
showcase
with
Edge
devices
as
well,
specifically
because,
as
we
mentioned
earlier,
that
you're
some
of
your
popular
programming
languages
that
are
typically
used
to
write
these
functional
service
calls
such
as
python
or
JavaScript,
are
great.
B
But
then
they
do
come
with
resource
limitations
and
like
in
a
lot
of
dependencies
that
might
not
run
or
on
various
platforms.
And
that's
where
webassembly
comes
in
pixel
and
the
additional
benefit
that
you
also
get
is
the
Securities
unboxing.
That
is
typical
of
webassembly,
and
it
provides
a
lot
more
isolation
to
your
functions
that
are
being
written
with
this
entire
infrastructure
of
being
able
to
write
them
in
webassembly.
So
those
are
some
of
the,
like.
B
A
Start
by
showing
a
demo
of
using
kubernetes
and
to
manage
your
webassembly
modules
and
the
main
role
that
the
main
thing
we
want
to
talk
about
when
using
kubernetes
is
so
we
so
recently.
A
very
interesting
talk
right
here
at
kubecon
yesterday
was
the
docker
was
in
preview,
which
is
pretty
interesting.
They
just
announced
it
yesterday,
so
they
had
essentially
written
one
of
so
they
had
essentially
written
a
shim
in
volume
which
allows
you
to
to
have
containers
and
directly
run
them
with
kubernetes
as
well.
A
So
what
we
want
to
also
show
over
here
is
is
something
we
had
done
to
run
it
with
kubernet
manage
it
with
kubernetes,
and
especially,
we
want
to
show
how
the
shim
layer
works.
So
you
can
so
you
can
run
your
web
assemblies.
I
I
have
a
kubernetes
cluster
here
in
Azure
and
I
also
have
two
node
pools
here.
A
So
if
I
just
go
to
node
pools
and
show
you
that
I
essentially
have
two
note
pools
called
pool,
one
which
is
the
three
nodes
for
system
and
the
mywasi
pool,
which
is
a
single
node,
and
this
is
a
Washi
node
pool.
So
this
is
kind
of
the
idea
we
want
to
talk
about.
You
have
a
process.
You
have
multiple
pods.
A
You
know
on
the
bossy,
node
pool,
which
is
what
I
do
with,
which
is
what
I
do
with
this
runtime
class
you
you
use
it
with
schedule.
It
schedule
my
pouch
to
be
run
on
the
watching
output
and
what
I
also
use
here
so
I've.
Also,
given
the
configuration
for
spin,
which
is
actually
taken
from
the
official
spin
repository,
but
that's
again,
if
you
want
to
go
beyond
Edge
use
cases,
I
will
not
really
talk
about
that
in
this
demo.
A
We'll
just
show
how
you
can
use
slight
as
your
Handler,
but
I
also
just
put
in
the
official
code
from
the
spin
Repository.
So
with
that
we
also
have.
We
also
have
our
container
running
over
here.
This
is
essentially
a
web
assembly
converted
to
a
container,
and
you
can
do
that
using
volume
to
oci
so
Volume
2
oci
is
a
pretty
popular
tool
which
allows
you
to
take
your
web
assembly
modules
and
convert
them
to
and
convert
them
to
oci
compliant
containers
and
convert
them
to
oci
compliant
containers.
A
A
So
this
is
a
very
simple
example
to
show
webassembly
modules
running
in
what
my
are
you
managing
it
with
kubernetes
and
if
I
just
call
this
I,
if
I
just
call
the
a
external
API
and
appended
or
hello
to
it,
I
should
just
probably
get
a
Hello.
How,
which
is
what
this
container?
Does
it
just
prints
out
hello,
but
it
uses
was
him
and
was
he
under
the
hood
to
do
this
to
get
to
get
the
system
calls
and
so
on.
A
C
D
You
were
saying
before
that
you
don't
think
we
should
run
all
the
workloads
as
random
modules.
There
are
some
that
are
better
fit
for
a
regular
container
and
others
that
are
probably
more
suitable
for
wasm,
which
workloads
do
you
see
wasn't
shine
like
when?
Would
you
say
yes,
this
is
a
good
candidate
to
use
for
what,
as
a
awesome
container
instead
as
a
regular
one
sure.
B
So
as
we
gave
an
example
for
machine
learning
over
here
right,
so
when
we're
doing
machine
learning
inference,
we
need
to
be
very
quick
now
such
kind
of
more
High
computational
tasks
where
you
would
need,
because
it
will
take
time
right
to
compute
to
do
the
inference.
So
those
are
kind
of
tasks
where
you
can
use
wasn't
because
of
the
faster
load
time
for
actually
spinning
up
a
wasn't,
a
cluster
and
being
able
to
run
a
wasm
container
because
of
the
small
size
and
quick
inference
right.
B
You'll
want
to
do
all
of
those
such
kind
of
tasks
using
your
version
container,
whereas
with
Docker
you
could
run
all
of
your
system
files
and
all
the
system
calls
using
your
Docker
container,
because
it's
well
supported
because
of
the
entire
ecosystem
of
Docker
containers
and,
like
you
know,
being
able
to
make
system
calls
that's
where
you're
going
to
be
using
the
interface
between
both
Docker
and
wussy.
But
specifically,
if
you're
asking
for,
like
the
use
case
for
wasm.
It's
mainly
for
doing
all
these
high
inference
tasks.
E
E
B
A
I
mean
volume
would
allow
you
to
reach
the
bar,
taking
even
more
larger
models
and
what
something
like
tensorflow
Lite
would
allow
you
to
do,
and
I've
been
one
of
the
contributors
on
tensorflow
Lite
myself,
but
volume
definitely
raises
the
bar
when
used
with
tensorflow
Lite,
but
for
very
large
models.
I
I
think
one
of
the
right
ways
would
be
to
work
on
optimizing
the
model
itself,
if
it's
a
pretty
large
model
in
terms
of
wasm
and
the
stock.