►
Description
Data is the lifeblood of decision making and many enterprises have been trying to evolve into data-driven organizations. Despite allocating significant resources to achieve transformations to data-driven concerns, progress has been limited. Outdated notions of centralized data storage and ill-fitting tooling such as traditional ETL, ELT and data pipelines in between are generally recognized as the root impediment to achieving the desired transformation. Contemporary notions such as (decentralized) Data Mesh and Data Product have been at the forefront of achieving rewards associated with data-driven transformations.
A revised notion and implementation of data pipelines plays a critical role in enabling the Data Mesh. We introduce a model for processing potentially large data streams in near real-time with distributed and eventually decentralized data pipelines. Data pipelines in our model are intended to verifiably integrate data from different sources and provide data preparation, transformation, integration and analytics suitable across data ownership boundaries. We provide an implementation example of our model and propose that modern, decentralized data pipeline models play an integral part in enabling (trustless) privacy-preserving technologies, such as FHE, at scale.
A
All
right
so
I'm
Bernhard
with
fluence
Labs
I
got
Alex
a
with
me
and
we
talk
about
screen
processing
at
scale
with
decentralized
data.
Pipelines
I
would
be
remiss
not
to
also
bombard
you
with
a
data
Deluge
slide
high
volume,
High,
Velocity,
High,
veracity
and
basically
I
mean
David,
already
addressed
it,
but
there's
so
much
data
coming
in
that
we
don't
even
know
what
to
do
with
it.
A
Very
different
and
data
could
be
managed
with
cool
things
like
SPSS
and
SAS
that
changed
Big
Data
evolved
and
we're
really
sort
of
driving
these
four
v's
in
the
Big
Data
paradigm
and
most
people
look
at
I,
don't
know,
maybe
you
don't,
but
a
lot
of
people
look
at
Big
Data,
predominantly
as
batch
processing
driving
the
training
of
ml
models,
basically
using
data
at
rest,
usually
High
Gravity
to
to
do
prediction,
analysis
and
all
that
good
stuff.
However,
a
very
critical
aspect
of
big
data
is
actually
stream
processing.
A
It
used
to
be
called
real-time,
Analytics
and
in
stream
processing
you
operate
on
data
in
flight
that
eventually
leads
to
the
augmentation
of
existing
batch
blobs,
wherever
they
are
falcon,
ipvs
doesn't
matter,
and
the
major
V's
of
of
stream
processing
are
around
usually
volume
and
velocity.
You
kind
of
know
what
structure
you
get
structured,
unstructured
hybrid
structure,
you're,
pretty
cognizant
of
that
what's
coming
in
in
streams
because
you
control
them
to
a
certain
degree
and
and
some
of
the
improvements
in
stream
processing
are
around
in-memory
processing.
Ram's
gotten
really
really
cheap.
A
So
that
really
really
changes
everything,
because
in
many
respects
for
spring
processing
you
you
often
want
to
operate
in
the
millisecond
window
mini
batch.
If
you
will
up
to
seconds
so,
Ram
really
makes
a
difference
on
how
to
do
this
and
then,
of
course,
cloud
and
and
increasingly
serverless
are
are
drivers
of
modern
stream
processing.
So
if
you
look
at
the
Confluence
around
Kafka,
for
example,
they
are
now
advocating
very
very
strongly
using
a
function
as
a
service
to
do
intimate
stateless
processing.
A
So
those
are
sort
of
certain
attributes
right
now,
modern
stream
processing.
In
order
to
process
streams,
you
kind
of
move
data
in
order
to
move
data,
you
need
pipelines
and
also
there
have
been
a
lot
of
improvements
and
some
of
the
key
attributes
of
modern
data
pipelines
around
distributed.
Architectures
particularly
fault
tolerance
and
failovers,
and
the
way
you
handle
checkpoints
and
exactly
once
processing
is
is
one
of
the
the
most
important
checkpoints
right
now.
A
On
top
of
it,
you
have
sort
of
a
governance
in
monitoring
about
what
data
data
security
sort
of
the
the
data
engineering
aspect
that
is,
is
very
closely
tied
to
data
Pipelines,
which
also
separates
the
data
Engineers
from
the
data
scientists
for
a
very
interesting
debates,
most
of
the
time,
but
moving
on
so
big
data
is
Big
Business.
There's
lots
of
data
Big
Data
Paradigm
is
huge
and
there
is
pretty
much
solutions
for
everything.
A
A
A
So
why
talk
about
decentralized
stream
processing
in
pipelines,
I
mean
it
looks
like
everything
is
working,
looks
like
there
is
a
huge
ecosystem.
A
lot
of
it
is
open
source
and
even
if
you
have
a
content,
addressable
storage
like
Falcon,
why
not
just
build
on
on?
You
know:
centralized
Cloud
providers
offerings
to
to
utilize
the
the
cheaper,
better
content,
addressable
storage,
and
if
you
do
it,
how
do
you
do
it
so
before
I
answer
those
questions
about
four
years
ago,
I'm,
probably
gonna
I
always
manual
her
name
but
degani.
A
She
came
up
with
the
concept
of
data
mesh,
it's
not
quite
in
the
in
the
in
the
same
league
as
data
warehouses,
data
Lakes
lake
houses,
but
it's
It's
associated
technological
construct.
It's
a
process
construct
that
has
identified
a
lot
of
attributes.
Were
data
management,
particularly
big
data
management
processes,
tooling,
and
the
use
of
the
outputs
fails
massively
within
organizations,
and
if
you
look
at
the
attributes
she
identified
and
the
transformation
she's
been
proposing
a
lot
of
these
attributes
and
Transformations
align
very
very
closely
with
web3.
A
We
look
at
decentralized
ownership,
so
the
domain
domain
ownership,
in
her
case
is,
is
only
the
people
who
are
skilled
enough
and
knowledgeable
enough
about.
The
data
should
be
involved
in
handling
the
data,
but
it
comes
back
to
a
little
bit
further
down
from
architecture
distributed.
Technically,
you
want
data
encoders
one
unit.
That's
a
really
really
important
one
you,
because
from
a
verifiability
perspective
and
auditability
perspective
and
and
on
it
goes
anyway.
A
So
there
there's
a
lot
of
I,
don't
know
controversy,
but
a
lot
of
discussions
about
the
the
Merit
of
data
meshes,
but
data
as
a
service
and
data
as
a
product
with
with
ownership
within
Enterprises
or
or
organizations
is,
is
certainly
for
real,
and
this
is
one
of
the
the
answers
or
attempted
answers
to
it.
What
matters
more
for
this
talk
about
our
discussions
is
that
a
lot
of
the
pain
points
in
organizations
handling
data
are
aligned
very,
very
well
with
Solutions
web3
has
to
propose
and
decentralized
solutions
have
proposed.
A
However,
if
you
are
in
this
in
in
in
the
age
of
cancellation,
it's
very
easy
to
to
to
violate
a
business
terms
of
service.
Anti-Platforming
has
been
taking
place
at
a
much
faster
rate
than
it
has
in
the
past
due
to
changes
in
in
our
culture
and
that's
very,
very
expensive.
If
you
have
petabytes
of
data-
and
you
have
till
the
end
of
business
day
to
move
it,
then
that's
a
pretty
high
chance
that
your
business
is
done
and
that's
an
enormous
risk.
A
Other
risks,
Associated
continuous
vendor,
lock
in
and,
of
course,
exit
barriers.
Again,
it's
just
something
very
difficult
to
price
and,
in
my
opinion,
as
well
as
Gartner's
forcers
and
others,
it's
massively
underpriced.
So
a
lot
of
the
cloud
pricing
is
actually
deceptive
in
the
sense
that
it's
not
fully
priced.
So
it's
actually
a
lot
more
expensive
on
the
tail
than
you
actually
would
expect,
and
that's
actually
very,
very
significant,
because
it's
one
of
those
intangibles
that
are
just
not
fully
actualized
in
in
a
lot
of
decision
making.
A
So
how
do
we
go
from
so
back
to
stream
processing?
This
is
where
we
come
from,
so
we
had
the
the
little
journey
here.
Amazon,
for
example,
has
a
has
a
a
stream
process
that
includes
serverless,
and
they
use
Kinesis
stream
plus
some
other
stuff
to
to
deal
with
the
actual
streams,
the
acquisition
and
as
an
intake
sink,
and
then
they
use
lambdas,
which
is
their
function
as
a
service
to
some
processing
and
putting
into
dynamodb
well
on
the
decentralized
approach,
we
can
actually
counter
that
offering
to
a
very
high
degree
today.
A
So,
for
example,
ceramic
streams,
David
I,
just
talked
about
ceramic
they're,
a
great
partner
built
on
ipfs
is
is,
is
one
way
to
look
at
replacing
mqtt,
Kinesis
or
even
Kafka.
To
a
certain
extent
on
the
on
the
key
value.
Storage
again
from
ceramic
composed
to
be,
for
example,
is,
is
an
increasingly
strong,
Contender
and
beta
now,
so
it's
starting
to
look
really
really
good
and,
of
course,
the
server
list.
This
is
where
fluence
comes
in.
A
That's
us,
and
now
I
want
to
talk
a
little
bit
about
fluence
fluence
is
a
decentralized
serverless,
stateless
peer-to-peer,
compute
protocol.
So
basically
you
want
to
decentralized
Lambda
the
function
as
a
service
and
it's
a
an
open,
permissionless
peer-to-peer
protocol.
It's
a
decentralized
capacity
Marketplace,
which
is
on
chain,
and
it's
basically
function
as
a
service
off
chain,
and
it's
built
around
webassembly,
which
gives
you
a
high
portability.
A
We
also
have
almost
done
proof
of
execution
and
validity,
which
is
an
Unchained
proof
right
now
and
have
a
bunch
of
tooling
and
a
dow
coming.
So
basically,
in
a
nutshell,
we
what
you
do.
Is
you
take
your
business
logic?
You
express
it
in
webassembly.
Currently,
it's
in
Rust,
Python
and
JavaScript
are
coming,
and
that
business
logic
is
then
compiled
to
webassembly
those
webassembly
modules.
We
call
now
service
can
be
deployed
to
one
or
more
peers
in
the
peer-to-peer
Network,
which
is
open
and
permissionless,
and
we
have
a
general
purpose.
A
Web
assembly
runtime
called
Marine.
That's
part
of
a
reference
peer
in
the
network
in
order
to
interact
with
those
Services
which
are
not
HTTP,
accessible,
they're,
only
peer-to-peer
accessible
for
a
variety
of
reasons,
including
our
strong
Embrace
of
the
web
3
ether.
So
we
really
don't
want
to
deal
with
the
censorship
risks
that
come
with
DNS.
A
You
need
something
you
need.
You
need
a
workflow
engine
and
we
wrote
our
own.
It's
called
aqua,
it's
built
on
the
bias
calculator
derived
from
PI
calculus
and
that
lets
you
choreograph
and
compose
these
distributed
Services
into
workflow.
So,
basically,
if,
depending
on
your
intent
and
capability,
you
can
build
a
distributor
in
decentralized
applications
and
protocols.
A
So
going
back
to
what
we
want
to
do
is
we
want
to
do
this
serverless
component
with
fluence
and
the
way
we
can
do
it
today
is
we
can
compile
some
of
the
Apache
error
tools
to
webassembly
I?
Have
it
in
a
little
less
transparent,
because
I
didn't
we
didn't
quite
to
get
it
done.
Docdb
is
also
something
we
have
almost
ready
for
webassembly
and
then
we
write
was
a
custom
webassembly
modules
to
to
interact
with
other
pieces,
including
ceramic
stream,
and
composed
beam
as
so
yeah.
Okay.
A
So
how
do
we
go
and
do
it
now
so
want
to
do
some
examples?
So,
let's
have
some
devices
could
be
peer?
Logs
could
be.
Temperature
sensors
doesn't
really
matter
dumping
data
into
ceramic
streams
instead
of
Kafka,
for
example,
and
then
you
use
aqua
and
and
marine
services
these
webassembly
modules
to
build
crawlers.
So
this
is
actually
very
close
to
the
Amazon
glue,
analogy,
Amazon
glue
being
an
integral
part
part
in
their
serverless
stream
processing
offering
and
one
thing
you
can
do
with
aqua
is
you
can
use
one-track
clients.
A
So
you
have
this
polling
capability
to
deploy,
crawlers
that
now
look
for
data
changes
updates
in
the
database
in
streams
Etc
and
then
push
this
into
into
other
logic,
like
stream,
processors,
for
example,
and
the
stream
processors
can
actually
end
up
in
two
pipelines,
for
example
an
event
trigger.
So,
for
example,
you
you
have
data
Windows.
You
compare
two
threshold
values
that
then
trigger
events
and,
of
course,
driving
that
data
into
your
your
persistence.
A
Syncs
such
as
filecoin,
for
example,
and
one
of
one
there's
something
very,
very
cool
and
I,
don't
know
if
we
have
enough
time
to
go
through
it,
but
Sergey
from
camo
addressed
this.
In
a
very,
very
good
talk
last
year,
so
if
anybody
is
interested
in
in
data
data
meshes
and
particularly
Apache,
parquet
file
format-
you
really
should
listen
to
the
lecture
from
last
year-
is-
is
driving
parquet
files
and
we'll
get
into
that
in
a
little
bit.
A
But
what
we
basically
can
do
is
we
can
build
highly
decentralized
data
Lakes
on
top
of
ipfs
or
firecoin
in
an
entirely
server,
an
entirely
decentralized
with
a
highly
decentralized
serverless
component.
Today,
it's
it's
absolutely
possible
by
by
choosing
existing
file
from
interoperable
file
formats
and
and
tooling.
B
Thank
you,
hello,
hello.
So,
as
Bernard
said,
we
have
this
thing
called
spells
that
are
basically
a
recurring
orchestration
action.
So
it's
a
workflow
that
takes
a
bunch
of
things,
glues
them
together
and
says
you
do
that
you
do
that.
You
do
that
and
it's
distributed,
so
it
can
like
tell
what
to
do
to
different
nodes
to
different
peers
across
the
network
and
different
data.
Centers,
all
that.
B
So
we
have
a
language
for
all
expressing
all
those
distributed,
workflows,
it's
called
aqua
and
it
has
a
nice
property
of
being
like
it's
when
you
design
something
in
aqua.
It's
very
clear
from
a
system
design
perspective
what's
happening.
What
components
are
involved
because
it's
very
simple
and
it
works
with
interfaces
mostly
so
here
we're
talking
about
crawlers,
so
we
need
to
read
from
ceramic
stream
or
whatever
Source
you
have
right
now.
It's
ceramic
stream.
So
here
are
the
components
that
we
will
use
to
do
that.
B
So
we
have
a
service
called
ceramic
stream
that
just
basically
gives
us
CID
of
the
mutable
stream,
the
tip
of
the
stream,
the
the
actual
value
of
the
latest
one.
After
applying
all
the
commits
that
ceramic
does
internally,
we
have
a
service
that
is
able
to
work
with
files
locally
on
local
peer.
We
have
a
parquet
service
that
also
locally
able
to
compress
one
file
to
and
provide
a
new
path
that
returns
your
parquet
compressed
file
and
we
have
Estuary
to
upload
to
filecoin,
assuming
all
the
keys
are
inside
and
managed
by
the
compute
providers.
B
So
we
have
that,
and
here
is
how
the
crawler
looks
like
it's
pretty
simple.
What
would
you
expect?
The
word
spell
basically
means
here
that
this
is
a
recurring
execution.
So
what
we
do
here
is
we
read
from
ceramic
on
line
27,
then
we
take
from
local
key
value
storage.
We
we
take
the
last
tip
that
we
saw
on
previous
run.
So
you
can
imagine.
B
This
function
runs
every
three
seconds,
every
five
minutes,
whatever
you
configure,
and
so
we
have
to
see
if
there
were
changes
and
if
tip
of
the
stream
changed,
then
now
we
can
download
the
actual
content
and
put
it
to
a
file
and
once
a
file
reaches
certain
size
on
line
32,
we
can
go
and
compress
it
through
parquet
and
say
Estuary,
please
upload
it
to
filecoin
and
provide
a
CID
back
and
what
we
do
here.
You
need,
like
you,
upload
files
to
filecoin
for
some
reason,
so
you
need
to
be
able
to
read
them
back.
B
So
you
need
to
be
able
to
maintain
that
the
cages
where
you
put
the
files
on
file
coin
to
retrieve
them
back,
that's
exactly
what
we
do
on
line
35.
We
just
put
you
to
local
key
storage
and
say
here
our
list
of
all
the
parquet
files
that
we
uploaded
and
whatever
your
business
logic
is,
after
that
you
can
just
take
the
cids
for,
for
example,
a
function,
Falcon
files
just
take
the
cids
and
you
can
do
whatever
you
want
with
them.
B
Maybe
you
want
to
orchestrate
some
other
data
Pipeline
with
some
other
spell,
but
that's
like
a
very
simple
example
of
glue,
a
bunch
of
different
Technologies
together
through
a
simple
script,
and
it
right
here
it
happens
on
a
single
peer.
But
since
it's
a
recurring
task,
it
actually
can
involve
more
peers
into
it.
And
you
can
imagine
this
just
scaling
to
something
we
call
subnet.
B
So
you
pay
a
bunch
of
money
to
blockchain
and
say
I
want
this
happening
on
bunch
of
peers,
reading
from
a
bunch
of
streams
and
all
put
into
filecoin,
and
then
I
will
do
something
with
those
files.
So
that's
basically
what's
happening
here.
That's
how
you
can
build
simple
crawlers
with
fluence
and
that's
how
we
orchestrate
different
services.
B
I,
guess,
that's
it
so
yeah
as
I
said
it's
possible
to
scale
this
orchestration
process
to
something
called
subnet,
and
once
you
have
that
you
have
a
certain
problem
of
service.
Discovery
right
you
when
you
have
a
single
peer-
and
you
know
it's
addressed,
it's
public
key.
You
can
find
it
in
the
network.
But
when
you
have
a
dynamic
sub
network
of
different
peers
running
this
orchestration
flows,
you
have
a
bunch
of
them.
Some
of
them
are
dying.
Some
of
them
are
returning
back
someone
just
usual
failover
happens.
B
You
have
to
have
some
service
Discovery
for
that,
and
we
have
this
pattern
when
we
have
a
thing
called
registry.
That's
basically
a
distributed
hash
table
where
you
can
put
when,
where
different
workers
register
themselves
by
the
deal
ID
delayed
is
the
kind
of
name
of
the
contract
on
the
blockchain,
where
there
are
mining
paid
for
that's
basically
paid
job.
So
we
have
this
ID
of
paid
job
and
everyone
who
does
that
job.
B
They
register
themselves
like
I'm,
doing
this
job
and
you're
able
to
reach
them
through
this
resolve
sub
Network
that
internally
use
Academia
and
all
that
things
that
you
probably
know
about,
and
it
gives
you
back
a
bunch
of
workers
and
you
can
go
to
all
of
them
and
say
hey,
give
me
what
kind
of
what
Falcon
files
you
have
uploaded
so
far,
you
can
ask
for
different
logs
States.
What
how
many
errors
happen?
How
many
executions
happened?
What
are
the
metrics?
A
A
I
did
I
managed
to
compile
Apache
Arrow,
a
Polaris
which
is
a
basically
a
panda
type
library
in
in
Rust
and
and
parquet
create
as
well.
So,
if
anybody's
interested
in
how
to
get
this
into
webassembly,
it's
not
entirely
straightforward,
but
it's
not
super
difficult.
So
I
probably
can
accelerate
your
journey
there.
If
that's
something
you
want
to
do
and
as
I
said,
we
have
a
docdb
coming
and
if
you
look
on
the
right,
this
is
your
Marine.
This
is
your
rust
code
right
now
to
to
handle
the
file
writer.
A
Everything
is
stateless,
so
how
do
you
deal
with
state?
So
what
we
do
in
handling
the
state
we
integrated
with
ipfs,
so
we
have
an
entire
Library
ready
for
you
to
use
that
basically
allows
you
to
use
ipfs
through
Aqua,
so
you're,
basically
you're
bridging
two
different
peer-to-peer
networks
with
one
orchestration
tool
and
in
in
the
right.
Basically
what
we're
doing
and
is
yeah
you
get
the
file
name
and
you
build
this
path,
and
then
you
start
just
basically.
A
This
is
this-
is
just
normal
Arrow
Apache
error
file
writing
into
a
parquet
file
with
all
the
metadata
and
all
this
good
stuff
on
the
left.
You
have
your
Aqua
that
actually
orchestrates
that
Marine
Service
to
do
that
and
what
we
do
here
is
we.
Basically
we
set
things
up.
We
we
get
our
topology
Discovery
in
place.
A
and
that's
actually
a
highly
efficient
way
now.
The
other
part
to
consider
is
webassembly
is
single
threaded.
You
may
want
to
write
streams,
multiple
streams
to
multiple
files
or
or
or
can
or
funnel
multiple
streams
into
a
single
right
and
in
in
aqua.
This
you
have
parallelization
capabilities,
so
you
can
run
you
can
basically
Fork
your
your
workflow
run
it
across
a
whole
bunch
of
different
peers.
A
So
if
you
wanted
to
have
10
different
file
writers,
you
could
do
so
and
then
you
can
bring
it
back
at
the
end
and
that's
actually
extremely
powerful,
and
if
you
look
at
it,
I
mean
the
right
side.
This
is
all
all
there's
to
it.
I
mean
this
is
just
how
a
battery
error
writes
files.
This
is
what
you
need
to
do
and
on
the
left.
A
It
irritates
the
bleep
out
of
me
and
I
I
really
want
to
see
this.
A
lot
of
this
moved
into
a
decentralized
environment
and
parquet
is
is,
is
a
key
to
that.
It's
not
the
solution,
but
it's
a
key
to
it
and
in
part,
because
it's
not
aside
from
the
fact
that
it's
columnar
blah
blah
It's,
actually
an
immutable
file
and
the
key
immutable
is
that
it's
accepted
in
the
industry
as
as
a
standard
to
be
immutable.
A
Now,
just
because
it's
mutable
doesn't
mean
it's
content,
addressable
and
it
doesn't
have
any
hashes
yet
and
Order
matters.
So
if
you
have
your
data
in
different
order,
you
get
different
immutable
parquet
files.
However
Sergey
from
kamu
is
really
on
to
this,
and
he's
already
has
a
crate
which
actually
compiles
the
web
assembly.
A
A
So
there's
a
lot
possible
for
us
to
to
become
involved
with
you
and
help
you
to
achieve
your
goals
if
you're
interested
in
content,
the
rest
of
parquet
and
decentralized
data,
Lakes
love
to
talk
to
you
one
way
or
the
other,
and
if
you're
interested
in
that
to
be
for
in-memory
processing,
it's
actually
really
cool.
If
you
haven't
looked
at
it,
if
you
don't
know
it,
you
should
look
at
it.
It's
it's
part
of.
What's
increasingly
termed
the
modern
stack
and
please
reach
out,
and
how
do
you
do
it?
A
A
C
Thanks
I
wanted
to
ask
a
question
about
Aqua
the
language.
It
is
suspiciously
similar
to
Apache
beam
kind
of
inspired
thing.
Isn't
it
well,
and
the
question
then
would
be
was
a
deliberate
choice
to
kind
of
have
a
language
and
not
a
doctor,
similar
SDK,
implemented
in
like
what
people
are
used
to
like
I
mean
Google
cloud
data
flow
and
things
like
that,
and
then
you
would
basically
get
all
the
users
that
code
and
that
stuff
yeah.
A
A
As
a
foundation,
though,
it's
it's
rigorously
built
on
Pi
calculus,
so
every
operation
is
fully
representable
and
traceable
in
pi
calculus.
That's
already
a
big
deviation
and
it's
really
really
important,
because
if
you
want
off
chain
verification
auditability,
you
have
to
have
that
one
second
Aqua
doesn't
just
manage
the
workflow.
A
It
injects
a
massive
massive
abstraction
layer
between
Bare
Metal
lip
P2P,
so
peer
Discovery
content,
routing.
All
this
stuff
is
highly
highly
abstracted
from
you
for
you
and
that's
an
an
incredible
component
of
the
implementation
of
Aqua,
because
Aqua
so
Aqua
I
should
have
been
more
explicit.
Aqua
is
a
language.
A
A
C
C
A
B
I
can
show
later
something
we
can
hit
some
yeah
grafana
graphs,
so
the
web
assembly
runtime.
It
runs
like
on
Watson
time
and
it's
rather
close
to
bare
metal.
In
that
sense,
it's
like
two
2x
or
3x
slower
or
something
and
since
Aqua
is
just
telling
lip
P2P
send
this
debt
and
this
to
other
peers,
just
like
the
network.
Latency
is
what
you
get
there,
there's
more
or
less
no
overhead
to
in
principle
to
that.
So
there
is
there
certain
overhead
go
into
webassembly
in
and
out
in
and
out,
but
that's
optimizable.
B
Yeah
so
well,
there's
a
this
fundamental
problem
when
you
scale
your
Discovery
networks
like
kademlya
to
to
a
certain
degree
and
like
I
guess,
ipfs
folks
can
speak
about
that
at
length.
There's
a
certain
problem
that
the
high
churn
and,
in
effect
the
academic,
becomes
slower
and
slower,
and
it
could
take
days
to
resolve
something.
So
our
approach
since
Aqua
allows
you
gives
you
really
powerful
tools
for
system
design
and
our
approach
right
now.
B
What
we're
trying
to
do
internally
is
to
have
hierarchical
Academia
when
you
have
one
big,
cadem
layer
that
that
is
not
actually
address
all
the
discoveries,
but
instead
it
manages
a
lot
of
small
cademless
that
are
kind
of
targeted
to
single
tasks.
So
when
you
have
a
lot
of
peer
doing
similar
jobs,
it's
in
a
small
Academia,
and
this
way
you
can
have
one
layer
of
hierarchy
on
you
or
have
we
can
have
n
layers,
and
this
way
you
kind
of
solve
the
this
scaling
problem
of
kademly.
At
least
we
hope
so.
A
Okay,
the
white
paper
is,
is
in
progress,
but
basically
so,
if
you're
familiar
with
the
lottery
system
used
by
life
peer,
for
example,
or
the
golden
ticket,
whatever
they're
called
samori
or
whatever
the
blockchain
is
anyway,
it's
it's
a
lottery
system
and
you
basically,
you
identify
I,
have
to
step
back
pure
webassembly
modules
are
deterministic,
and
and
based
on
that,
you
can
make
certain
assumptions
you
get
the
trace
and
all
you
need
from
the
trace
is
is
so
it's
a
very
long
answer,
part
of
it.
So
it's.
A
A
You
package
this
at
the
very
beginning,
so
you
have
your
Genesis
data.
So
if
you
have
a
service,
hello
parameter,
your
parameter,
hello,
Cod
Summit
is
your
the
only
piece
of
data
that
goes
with
that
packet.
You
have
a
client,
the
client
could
be
short-lived
long
and
it
doesn't
matter
flings
this
literally
the
smart
packet
onto
a
relay.
The
relay
starts
looking
at
that
packet.
What
needs
to
be
done?
Do
I
have
the
service.
No.
A
Now
now
this
whole
Academia
work
starts
kicking
in
route,
Discovery
content
driving
blah
blah
blah
into
the
next
peer,
where,
hopefully,
we
have
the
service
on
that
service.
Aqua
virtual
machine
starts
looking
at
the
script
what's
been
executed,
but
it
hasn't
been
executed
and
starts,
instructing
basically
the
web
assembly
to
to
run
those
webassembly
modules.
All
along
the
steps
you
get
signatures
from
the
participating
peers,
Ingress
and
egress.
Hopefully
it's
going
to
be
a
BLS
where
you,
basically
you
get
constant
size,
blah
blah
blah
and
that's
very,
very
viable
at
the
content.
A
So
what
you
basically
have
you
have
this
this
design
custodial
chain?
If
you
will
that
can
be
expressed
in
hashing
and
if
there's
a
break
in
the
hashes,
you
know
that
something
hasn't
been
executed
properly.
So
when
you
start
sampling
probabilistically
as
long
as
the
hash
change,
it's
it's,
it's
not
entirely
dissimilar
to
how
the
Merkle
threes
work
in
in
blockchains.
A
You
can
do
probabilistic,
sampling
and
and
then
rerun
these
pieces,
and
if
the
pieces
are
successfully
re-run
in
terms
of
signature
matching
blah
blah
blah
all
this
stuff,
then
then
you
get
a
hit
as
I
said
it's
probabilistic,
but
we
think
we
get
it.
We
can
get
it
to
a
really
really
high
probability.
So
that's
roughly
how
it
works.