►
Description
Building WASM Powered Distributed Stream Platform - Sehyo Chang, InfinyOn
A modern data stack is critical for building data-driven organizations. Yet, technologies available today call for separation between data and compute results in a costly and inefficient data stack. A WASM-based in-line computation removes the barrier between data and computes, opening the door for new types of data stack that can deploy anywhere with hyper-efficiency. This talk will show how the Fluvio open-source project leveraged WASM technology to build a distributed stream platform. The session will discuss the unique challenges of integrating the SmartModule, a WASM-based module system, in a distributed data stack and its benefit in solving various use cases. Sehyo Chang is a creator of the Fluvio project and co-founder of InfinyOn. He has over 30 years of industry experience working on networking, distributed systems, microservices, and zero-trust security frameworks. Before InfinyOn, Sehyo was responsible for the service mesh architecture at NGINX, where he created nginMesh, an NGINX compatibility layer for Istio, and built RUST based module system for NGINX.
A
A
A
We
open
source
in
2019
fluvio
is
retaining
rust
and
designed
as
a
cloud-native
platform
from
the
ground.
Up
we
have
basically
the
platform
does.
Is
it
collects
events
from
consumers
and
the
producer,
and
then
it
stores
and
processed
to
the
consumers
and
connectors
we
have?
The
producer
can
be
written
in
many
languages
such
as
javascript
and
python
and
rus.
Of
course,
the
connectors
that
we
have
can
collect
the
data
source
from
other
database.
A
A
A
Okay,
so
what
are
the
challenges
of
trying
to
connect
this
data?
The
biggest
issue
is
existing
data
processing
paradigm,
we're
using
request
and
response
model
in
that
model.
You
basically
persist
the
data
into
some
kind
of
storage
devices
or
database,
and
you
make
requested
response
from
the
application.
A
A
A
The
the
best
approach
for
data
emotion
is
it's
done
to
what's
called
stream
processing
and
one
way
to
visualize.
This
is
just
like
flowing
the
stream
of
water,
where
the
data
kind
of
represents
the
water,
and
we
have
a
processing
steps
in
that
can
transform
data
in
any
way.
That
should
we
need
to
do.
A
A
A
A
Now
it
gets
worse
as
you
transfer
to
h
to
n80
gateway
or
nab
and
two
various
different
proxies,
so
this
can
be
very
expensive,
and
if
you
don't
carefully
model
your
costs
on
your
microservices
or
the
databases,
it
can
be
a
very
difference
between
successful
project
or
or
you're.
Shutting
down
your
business.
A
A
A
A
A
So
so,
basically,
solution
is
to
actually
reduce
the
data
movement,
and
one
way
to
do
is
that,
instead
of
moving
the
data,
we
reduce
the
data
by
moving
a
compute2
near
to
the
store
as
much
as
possible
or
to
the
source
of
data,
and
we
leveraged
power
compute
to
reduce
the
data
to
small
as
possible
and
the
most
low
with
the
powerful
muscle,
low,
the
computer
and
computer
is
getting
faster.
While
your
network
is
not
getting
there
much
faster.
A
Okay.
So
now,
let's
go
back
to
this
previous
example
and
see
how
we
saw
this.
The
problems
we
encountered
previously
so,
instead
of
sending
string
data
to
another
another
node,
we
can
actually
combine
into
a
single
in
a
node
where
we
process
the
data
and
and
only
send
out
the
data
that
makes
sense
for
the
downstream
stages.
A
A
A
A
Next,
you
need
to
be
efficient
to
minimize
energy
to
run
small
devices.
Third,
you
need
to
be
able
to
sandbox
such
that
it
can
be
run
in
any
environment
and
the.
Lastly,
you
need
to
support
different
languages.
It's
no
longer
too
sufficient
to
just
support
a
single
language
or
single
set
of
set
of
language
related
families.
A
The
the
containers
are
much
better,
but
it
doesn't
provide
a
true
isolation,
because
there's
a
lot
of
money
spending
on
trying
to
lock
up
the
containers,
the
other.
The
traditional
approach
of
moving
to
the
jar
offers
portability,
but
you
know
very
tight
to
java
virtual
machine
and
it
doesn't
provide
actually
the
true
isolation.
A
With
webassembly
now
we
have
opportunity
to
move
compute
to
anywhere,
and
this
is
the
new
type
of
infrastructure
that
we
can
advantage
to
to
move
everything
to
the
data.
Emotion,
okay,
the
bigger
picture
is
that
the
portability
and
isolation
provide
to
allow
us
to
reshape
our
compute
infrastructure
and
and
data
stack
by
flattening
into
vertical
unified
stack.
A
A
A
It
can
be
a
file,
it
can
be
ebs
or
s3
or
it
can
be
a
storage
array
the
regardless
of
the
storage
devices,
the
store,
binding,
basically
abstract,
out
into
a
unified
api,
and
then
we
layer
on
top
of
that
ross
api
to,
of
course,
to
give
us
the
api
in
terms
of
web
assembly
module
and
then
and
then
we
have.
We
are
working
on
different
language
bindings
to
expose
that
into
different
language,
such
as
assembly
script
or
python
and
other
possible
languages
in
the
future.
A
And
this
is
a
simple
example
of
filter
that
filters
out
the
records
based
on
whether
contains
you
know
letter
a
okay.
Now,
the
we,
the
interesting
part,
is
that
we're
using
the
the
roses
language
constructs
called
procedure
macro
to
basically
annotate
them
to
indicate
that
what
kind
of
binding
we
want
to
perform.
So
in
this
case,
of
course,
the
filter.
A
Okay,
the
next.
This
is
the
rust
api
signature
and
it
provides
the
filter,
and
then
it
has
only
one
single
argument.
The
record
so
directory
is
the
basically
the
primitive.
The
basic
constructs
are
the
storage,
the
bindings
it
can
either
return
boolean,
true
or
false.
If
it's
satisfied
criteria
or
it
can
the
exception,
the
the
result
type
is
the
rust
equivalent
to
indicate
that
it
does.
Exception
occurs
okay
and
then
so
this
allows
to
provide
a
same
api
for
the
same
storage
bindings.
A
Now
note
that
the
user
doesn't
have
to
worry
about
how
to
encode
and
decode
is
basically
is
done
through
this
wrapper
that
procedure
macro
that
generates
all
the
glue
routines.
It's
done
by
a
smart
modules
framework.
Okay,
now,
let's
move
further
go
down
into
how
actually
binding
their
stores
in
with
the
smart
module.
A
These
records
are
grouping
of
batches,
so
when
we
actually
send
these
records
to
fluvial
smart
modules,
the
entire
the
records
group
records
is
read
in
a
single
batch
and
this
in
this
case,
the
wreck.
The
group,
the
batch
group
starts
at
five
offset
one
thousand
and
two
five
pointer,
twelve
hundred
so
in
one
single
shot,
you'll
read
those
file
contents
and
and
and
send
over
to
those
smart
modules.