►
Description
Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Distributed Computation with WASM and WASI - Bailey Hayes & Carl Sverre, SingleStore
Distributed computation is a necessary paradigm for modern analytics. WASI-data is a new proposal for a WASI interface-type API that enables portable, host-, and language-independent ecosystems of composable WASM modules to support distributed algorithms. In this talk, we present the WASI-data interface design and demonstrate a reference implementation written in Rust. We will discuss WASI-data's potential to eliminate data lakes and how WASM could become the primary unit of compute for data processing systems.
A
Hey
everyone
today
we're
going
to
be
talking
about
distributed
computation
with
wasm
and
wazi.
My
name
is
bailey
hayes,
I'm
a
software
engineer
at
single
store.
I've
been
working
with
webassembly
for
several
years
now
at
my
last
company,
I
worked
on
the
more
traditional
use
case
for
webassembly,
which
was
working
on
a
data
visualization
library
running
in
the
browser
carl
hi.
My.
B
Name
is
carl:
I
have
been
working
on
databases
for
the
last
eight
and
a
half
years
at
a
company
called
single
store.
I
work
with
bailey
we're
both
software
engineers
there
and
over
the
last
couple
months,
I've
been
deep,
diving
into
the
world
of
wazom
with
bailey,
and
it's
been
a
lot
of
fun
and
yeah.
We're
really
excited
to
show
sort
of
share
our
vision
of
distributed
computation
with
wasm
and
wazi,
especially
with
related
relation
to
data.
So
that's
why
we're
here?
B
So,
let's
talk
about
whoops.
Let's
talk
about
apps,
really
quick.
We
just
had
a
great
presentation
from
liam
that
was
really
interesting
about
capabilities
and
how
applications
can
become
extracted
from
their
requirements
a
little
bit
more.
I
think
that
we
actually
look
at
this.
Also
from
state
applications
have
state.
We
all
know
that
and
applications
start
off
with
usually
very
little
state
when
you're
in
development
on
your
computer.
B
That
gets
a
lot
more
interesting
as
apps
grow.
As
we
all
know,
applications
tend
to
have
a
huge
explosion
of
data,
especially
modern
applications.
We're
seeing
a
huge
like
surge
of
data,
intensive
applications
and
data
intensive
applications
use
a
lot
of
state
they.
If
you
look
at
any
modern
application,
you
have
things
like
databases,
you
know
blob
storage,
caching,
layers,
cues,
all
sorts
of
data
everywhere,
and
the
thing
that's
interesting
about
data
everywhere
is
that
it
leads
to
data
moving
everywhere.
B
B
We
see
this
weird
thing
where
the
database
doesn't
always
isn't
always
able
to
run
the
right
type
of
computation
and
so
we're
starting
to
see.
Apps
move
not
just
like
an
answer
to
a
question,
but
the
actual
data
required
to
answer
that
question
into
the
application
where
they're
running
sort
of
custom,
processing
and
custom
code.
B
B
A
Yeah
well
we're
all
here,
so
we
already
know
a
lot
of
the
benefits
of
vlasm,
but
for
us
we're
a
multi-tenant
managed
service
database.
It's
running
across
the
clouds
and
it's
distributed,
which
means
the
secure
by
design
feature
of
wasm
is
probably
the
most
important
component
and
you
wouldn't
want
your
data
to
leak
to
somebody
else
like
from
one
bank
to
another.
A
A
And
the
our
goal
with
wazi
data,
which
I
should
just
say,
introducing
blazidata,
that's
our
proposal
for
how
to
do
this.
I
want
to
be
able
to
take
any
language
like
roster
python
and
say
I'm
using
wazidata,
and
that
gave
me
the
framework
for
being
able
to
distribute
my
computation
and
push
that
compute
down
into
a
distributed
database
or
some
other
type
of
distributed.
Runtime.
B
So
yeah,
so
wazi
data
is
partially
something
that
we
have
built
and
are
working
on
actively
and
partially
a
proposal
for
where
we
see
this
going
so
we've
sort
of
broken
up
into
four
stages
in
stage
one
and
two
we're
focusing
on
building
the
sort
of
building
blocks
of
computation.
So
in
stage
one
we
basically
just
want
lambda
functions.
These
are
just
functions
that
map
from
a
single
piece
of
data
to
potentially
another
single
piece
of
data.
In
stage
two,
we
want
procedural
extensions.
B
We
want
to
be
able
to
define
a
procedural
multi-stage,
algorithm
or
data
application.
We'll
give
you
an
example
of
that
in
a
little
bit
stage,
three
is
going
to
be
about
relational
extensions,
it's
about
using
wasm
to
actually
extend
and
augment
relational
query
execution
systems.
So
inside
of
a
database,
you
have
things
like
logical
query
plans
which
actually
describes
the
the
way
that
the
data
is
computed
or
the
answer
is
computed
over
a
distributed
system.
We
want
to
be
able
to
extend
that
with
wasm
and
then
in
stage
four.
B
A
Let's
start
with
the
simplest
one:
let's
define
functions,
you
have
things
like
scalars
value
in
value
out.
We
want
to
operate
on
records
for
us.
That
might
mean
a
row
in
a
database,
but
that
could
also
be
structs
in
your
language
and
then
be
able
to
work
on
batches
and
vectors
of
these,
and
so
this
is
the
simplest
unit
of
compute.
Basically,
that
we
can
start
with.
A
A
A
We
need
to
be
able
to
find
functions,
create
a
plan
for
us
we're
just
passing
in
a
string
right
now.
This
is
definitely
something
we
want
to
make
better
in
the
later
stages,
so
our
plan
string
in
our
case
is
going
to
be
like
a
sql
query
for
our
database
single
store,
but
could
also
be
like
a
mongodb
query.
A
B
Coming
up
with
examples
for
this
sometimes
is
surprisingly
hard.
That
was
a
very
contrived
example,
but
we
just
wanted
to
show
that
you
know
we
want
to
be
able
to
do
procedural,
define
a
procedural
work,
a
set
of
steps
in
wasm
and
run
that
then
it
gets
really
interesting.
So
so,
once
we
have
the
basic
foundation,
we
have
the
basic
lambda
function,
the
thing
that
can
work
on
a
unit
of
compute
and
we
have
the
higher
idea
of
having
a
wasm
module,
be
like
an
orchestration
of
of
distributed
computation.
B
B
So
for
anyone
who
isn't
super
into
databases,
most
databases,
especially
relational
databases,
use
relational
algebra
and
they
have
things
called
logical
query,
plans
and
physical
query
plans.
Logical
query
plans
basically
define
a
distributed
computation,
so
if,
in
your
sql
engine,
if
you
say
like
select
star
from
foo,
join
bar,
something
like
that
internally,
it
might
look
like
this
in
in
the
engine.
It
might
have
some
type
of
tree,
but
sometimes
these
things
are
limited.
You
might
want
to
do
a
distributed
operation.
B
We
propose
that
do
the
opposite,
actually
allow
you
to
augment
relational
query
plans
using
wasm
and
push
that
actual
plan
directly
into
the
engine,
so
we're
looking
at
ways
to
do
this
and
we're
really
excited
about
exploring
this
with
the
community.
So
this
is
definitely
future
work
and
similarly,
I
mentioned
earlier-
we
want
to
be
able
to
represent
computational
dags,
so
just
arbitrary,
directic,
acyclic
directed
acyclic
graphs
of
computation.
B
That
would
allow
you
to
do
really
complex
workflows.
If
you
could
imagine
defining
this
workflow
entirely
in
wasm
from
the
actual
coordination
aspect
of
the
workflow,
all
the
way
down
to
the
individual
components
in
that
workflow.
That
would
be
amazing
and
if
you
could
do
that
in
an
abstract
way,
that
could
run
on
many
different
distributed
systems.
That
would
even
be
more
more
amazing,
and
that
goes
back
to.
I
think
what
liam
was
saying
about.
B
B
So
yeah
there's
there's
a
million
workflow
engines,
there's
a
million
ways
to
do
all
this
stuff,
but
none
of
them
allow
you
to
push
the
computation
all
the
way
into
the
database
and,
as
you
said
at
the
beginning,
wasm
is
a
fantastic
way
of
sort
of
representing
computation
and
we
really
believe
that
by
defining
these
things
inside
of
a
wazon
module
and
pushing
that
into
the
engine
into
these
distributed
databases
and
distributed
computation
systems,
that's
where
we're
going
to
see
a
huge
win,
so
cool
all
right.
So
now
a
demo!
A
So
remember
we
keep
going
back
to
that
sentiment.
Analysis
example.
This
is
our
rest
code
for
doing
that
sentiment,
analysis,
we're
building
right
now,
just
on
whitex
bindgen
and
the
interface
types
proposal,
so
not
a
ton
that
we
had
to
do
unique
to
us
for
being
able
to
run
this
in
a
distributed
way.
Carl
wrote
this
macro
the
swazi
interface
macro
and,
and
that
basically
creates
the
bindings
that
we
need.
So
let's
run
it
like
a
data
scientist
which
I
am
not.
A
I've
got
a
python
notebook
here
and
I'm
using
a
framework
called
dask
if
you're
familiar
with
spark.
This
is
kind
of
like
the
python
equivalent.
The
data
that
we're
operating
on
looks
a
little
bit
like
this.
It's
a
table
with
comments
from
stack
overflow.
We
wanted
to
show
you
something
with
real
data
and
maybe
even
do
a
real
type
of
analysis
on
that
data.
A
So
I've
got
my
text.
I
can
see
how
large
my
data
frame
is
and
you
can
kind
of
think
of
a
data
frame
as
almost
like
a
table.
I've
got
a
thousand
rows
and
that's
also
really
common.
You
pull
a
small
set
of
the
data
iterate
on
it
locally
on
your
host,
like
which
might
be
windows
or
mac
or
linux,
and
come
up
with
kind
of
some
of
your
ideas.
A
A
A
I
think
that
on
stack,
overflow,
neutral
statements
because
stack
overflow
is
supposed
to
be
giving
you
answers.
Those
answers
should
be
technical
and
neutral.
Those
are
going
to
have
the
highest
score.
Now.
You
probably
are
familiar
with
upvoting
and
down
voting,
maybe
on
reddit
or
even
stack
overflow
on
posts.
But
comments
also
have
upvoting
and
down
voting.
A
Because
I
want
to
operate
on
the
full
side
of
the
data
now
I'm
going
to
connect
to
our
database,
which
we've
also
added
wasm
support
and
that
full
set
of
data
is
82
million
rows
and
like
before
I'm
going
to
take
the
exact
same
waza
module
that
I
was
running
in
dask.
I'm
going
to
run
it
in
single
store.
A
A
A
Now,
let's
do
something
really
cool,
I
want
to
run
distributed
computation.
That
was
the
promise
of
the
whole
talk
and
that's
what
this
query
does.
Our
database
is
distributed,
we're
taking
our
wasa
module
and
running
it
across
all
of
our
nodes
across
82
million
rows
of
data,
and
we
get
back
very
quickly
and
a
query
that
uses
analytics.
Now
it's
pretty
simple
analytics
here,
but
the
idea
is
that
we
want
to
know
by
bucketing
our
our
ro,
our
all
the
rows
and
we
group
by
them
based
on
their
score.
A
So
if
the
score
ranges
from
say
10
to
20,
that's
one
bucket,
and
when
we
look
at
this,
we
see
that
these
scores
are
both
very
negative
and
very
positive.
So
just
a
quick
peek
at
the
data,
maybe
I'm
not
able
to
draw
any
types
of
conclusions.
So
the
next
thing
you
do
is
plot
it
we're
going
to
render
a
really
simple
graph
that
has
a
trend
line
using
ordinary
least
squares.
A
When
we
graph
this
with
the
trend
line,
we
see
that
first
it
supports
my
hypothesis
that
higher
scores
will
be
more
neutral,
but
something
else
interesting
came
out,
which
is
they
also
seem
to
be
more
leaning
towards
more
neutral
positive.
Perhaps
negative
comments
are
more
likely
to
get
down
voted,
so
I
think
that's
pretty
cool
and
kind
of
shows
a
little
bit
of
the
power
of
being
able
to
run
wasm
and
two
totally
different
frameworks,
and
I
would
love
to
see
it
show
up
in
others,
so
yeah.
C
B
And
just
something
to
point
out
about
that
graph.
That
graph
is
generated
by
pushing
the
sentiment
code,
which
we
compiled
from
wasm
into
a
distributed
system
running
in
google
cloud
right.
So,
instead
of
pulling
82
million
comments
down
to
our
laptop
to
generate
that
graph,
we
are
pushing
the
compute
to
google
cloud
to
actually
generate
this
graph.
A
B
Sweet
so
yeah,
so
in
conclusion,
wasm
and
distributed
is
a
fantastic
thing.
I
mean,
as
liam
said,
it's
the
future
and
we
really
really
believe
that,
but
we
believe
that
data
has
to
be
involved,
and
so
really
wazi
data
is
really
just
a
proposal.
It's
an
idea.
It's
an
idea
that
we
could
actually
push
down
computation
into
distributed
systems
and
databases
and
data
systems
so
that
data
apps
can
stop
pulling
data
out,
put
keep
the
data
where
it
is,
keep
it
efficient
and
leverage
wasm
for
its
superpowers.
B
C
But
I
hadn't
seen
any
questions
during
the
conversation,
because
I
think
it
was
so
interesting,
but
let
me
check
again
so
if
there
any
questions
appear
in
the
slack,
the
slack
is
available
all
week.
So
if
you
have
questions
while
you're
chewing
on
everything,
you
saw
right,
carl
and
bailey
will
drop
into
the
slack,
and
you
can
ask
the
questions
there.
If
right
now
isn't
there's
a
question
over
there
go
ahead
and
say
your
question,
we'll
repeat
it.
D
B
A
That
one
yeah,
I
love
that
question
crypto
has
a
really
similar
problem
where
they
want
to
compute
the
gas
or
like
what
the
you
know,
how
many
cycles
are
going
to
run
on
the
cpu
and
yeah.
That's
super
important
for
us,
because
you're
running
in
a
managed
service,
just
like
maybe
even
a
serverless
function,
a
lot
of
if
you
like,
squint
and
tilt
your
head,
a
lot
of
the
use
cases
are
really
the
same,
and
yes
I
would
we
haven't
done
anything
yet.
A
We
would
love
to
take
advantage
of
the
things
that
crypto
is
doing
in
that
space,
something
that
I
haven't
seen
yet
is
throttling,
which
is
something
else
that
I
would
really
like
to
check
out
yeah
if
we
could
find
a
way
to
both
predict
like
what
the
performance
is.
A
lot
of
these
machine
learning
operations,
they're
long
running
like
they're
just
always
running,
perhaps-
and
perhaps
you
only
want
to
run
them
when
there's
down
time
and
so
being
able
to
say,
okay,
I
know
what
my
gas
is
going
to
be.
A
C
B
Yeah,
just
the
quick
answer
is
with
oh
yeah:
sorry,
we're
good,
okay
cool,
so
the
answer
is
basically
in
the
current
approach
to
what
we
have
inside
of
single
store
and
inside
of
dask.
So
like
what
we've
implemented,
you
can
actually
build
mapreduce
using
a
combination
of
your
functions,
which
cannot
act
like
a
mapper
and
and
the
actual
frameworks
themselves.
B
So
single,
sour
and
das
both
provide
sort
of
solutions
to
be
able
to
do
reductions
in
groups
and
shuffles
and
stuff
like
that,
that
you
need
to
be
able
to
run
large-scale
map-produce
operations.
So
in
our
example
of
sentiment
analysis
we
were
using
it
like
a
mapreduce
operation.
We
were
mapping
the
data
over
from
from
each
of
the
comments.
We
were
mapping
to
the
like
sentiment
records
which
contains
the
red
sentiment
scores
and
we
were
reducing
by
aggregating.
Those
sentiment
scores
right,
so
you
could
think
about
that
as
a
mapreduce.
C
D
B
D
D
B
You
just
showed
off
like
get
rid
of
a
lot
of
that,
so
this
code
leverages
widex
and
widex
by
gen
100.
So
essentially,
what
this
does
is
this.
This
macro
takes
a
mod
written
in
rust.
The
goal
is
ergonomic
simplicity
for
people
who
might
not
be
they
might
just
want
to
write
a
simple
rust
function.
They
don't
want
to
necessarily
have
to
learn
anything
more
than
that,
so
we
want
to
simplify
it
to
literally
import
wadzi
interface
annotate.
B
I
think
it's
up
on
the
screen
annotate
your
your
rust
mod,
with,
like
some
basic
struts
and
stuff,
like
that.
This
macro
will
spider
this
mod
and
export
all
the
functions
using
the
canonical
api
per
the
same
as
widex
bind
gen.
What
this
gives
you
the
advantage
of
is
on
the
host
side
like
when
we
implemented
the
das
side.
For
example,
all
we
had
to
do
is
just
take
the
widex
existing
widex
bind
gen
python
module
and
essentially
load
the
widex.
B
What
we're
exploring
is
ways
of
eliminating
the
widex
stage,
because
we
really
want
to
like
create
ergonomic
simplicity,
that's
not
entirely
related
to
wazi
data,
that's
more
just
like
general.
For
the
ecosystem,
we
want
to
be
able
to
write
like
a
simple
rust
function
and
just
like
deploy
it
with
good
structure,
structural
types
and
real
types
and.
C
B
C
We
do
have
just
a
couple
more
minutes,
so
if
you
want
to
launch
on
another
question,
we
are
watching
the
slack
and
and
the
meeting
play
so
there's
a
question
online.
How
would
you
handle
failure
failure
recovery
during
a
batch
process?
Would
the
state
of
the
process
be
stored
somewhere
to
be
able
to
restart?
How
are
you
thinking
about
that?
That's.
A
A
great
question,
one
of
the
cool
things
about
wasm,
is
that
I'm
able
to
inspect
things
before
I
ever
actually
actually
run
it,
which
is
kind
of
how
crypto
computes,
perhaps
gas,
ahead
of
time
like
with
ethereum
for
us.
I
could
do
things
like
saying
I'm
about
to
execute
a
transaction.
I
can
annotate
that
before
I
execute
it,
wasm
is
basically
a
stack
based
machine.
I
could
probably
replay
things
if
it
fails.
I
can
also
freeze
my
wasm
module
where
it's
at
say.
I
have
some
kind
of
network
or
hardware
failure
happening
on
a
node.