►
Description
Visit the https://archive.devcon.org/ to gain access to the entire library of Devcon talks with the ease of filtering, playlists, personalized suggestions, decentralized access on Swarm, IPFS and more.
https://archive.devcon.org/archive/watch/6/introducing-the-graph-substreams-for-high-performance-indexing/
Substreams is a powerful, parallelizable, blockchain data indexing technology, developed within The Graph.
It enables truly composable indexing, with shared modules written in Rust, both in batch and in streaming. It reaches unbeforeseen performances by virtue of its parallel design that supports both mappers and stateful store modules.
In this session, we will see how Substreams solve indexing for all blockchain protocols.
Speaker(s): Alexandre Bourget
Skill level: Intermediate
Track: Developer Infrastructure
Keywords: thegraph,data,indexing,subgraphs,substreams
Follow us: https://twitter.com/efdevcon, https://twitter.com/ethereum
Learn more about devcon: https://www.devcon.org/
Learn more about ethereum: https://ethereum.org/
Devcon is the Ethereum conference for developers, researchers, thinkers, and makers.
Devcon 6 was held in Bogotá, Colombia on Oct 11 - 14, 2022.
Devcon is organized and presented by the Ethereum Foundation, with the support of our sponsors. To find out more, please visit https://ethereum.foundation/
A
A
They
don't
understand
what
happened
but
anyway,
we
said
thanks
and
goodbye
to
our
VCS
and
shifted
our
Focus
to
make
the
graph
the
greatest
data
platform
on
Earth.
So
today,
I'm
here
to
introduce
substreams,
which
is
a
powerful
new
paralyzed
engine
to
process
blockchain
data
and
before
I,
can
do
that.
I
just
want
to
set
a
bit
of
context.
Perhaps
you
can
raise
your
hand
if
you
know
what
sub
graphs
are.
Raise
your
hand
if
you
know
you're,
good,
okay,
so
slip
graphs.
A
Can
be
thought
of
an
ETL
process,
right,
extract,
transform
and
load
and
subgraphs
add
that
little
Q
there
that
graphql
layer
to
it
and
some
grass
today
provide
that
sort
of
simple
approachable,
end-to-end
solution
to
blockchain
indexing
and
the
graph
node
is
responsible
for
all
of
these
components.
Right.
The
extraction
is
done
through
hitting
Json
RPC
nodes
and
then
transformation.
You
provide
some
assembly
script.
You
guys
know
that
composite
wasm
running
in
a
distributed
environment,
and
then
you
have
the
load
aspect
which
the
graph
node
does
puts
that
into
postgres
and
offers
you
a
rich.
A
You
know
and
beautiful
graphql
interface.
On
top
and
one
of
the
reason,
one
of
the
reasons
we
were
brought
in
was
that
so
we
could
push
the
graph
to
New
Height
in
terms
of
performances.
So
to
do
that,
we
brought
first
thing
the
fire
hose
something
at
the
extraction
layer
to
our
take
to
boosting
performance
by
one
two,
three
orders
of
magnitude:
the
first
layer
extraction.
It's
the
method
of
extract
attracting
data
from
blockchain
nodes.
Imagine
prying
an
egg
open
where
the
data
is
exfiltrated
as
fast
as
possible,
and
all
the
juicy
data
gets.
A
Then
thrown
in
a
grpc
stream
as
well
as
into
flat
files,
and
you
can
think
of
that
as
sort
of
a
the
bin
log
replication
stream
for
blockchains,
where
you'd
find
in
a
Master
Slave
replication
engine,
like
in
databases
so
we'll
get
back
to
fire
hose
in
a
minute.
Then
sub
streams
is
sort
of
a
reasoning
of
this
rethinking
of
the
second
box,
the
transformation
layer.
A
Here,
instead
of
the
traditional
sub
graph
handlers
and
assembly
script,
you
will
write
substreams
modules
in
Rust
and
those
can
be
executed
in
real
time
as
well
as
in
parallel
with
unprecedented
performance.
So
let
me
give
you
first
a
primer
on
fire
hose
because
there's
a
lot
of
benefits
of
substreams
that
come
directly
from
the
fire
hose.
So
a
streaming
fast
for
many
years.
We've
been
thinking
hard
about
all
these
indexing
problems
from
first
principles
and
we
needed
a
first,
a
robust
extraction
layer.
A
We
wanted
something
that
is
creamy,
but
that
was
extremely
low,
latency,
so
I'm
thinking
that
would
push
data
out
the
moment
the
transaction
was
executed
within
a
block
within
a
blockchain
node
Json
RPC
was
not
going
to
cut
it
and
we
didn't
want
to
have
to
deal
with
those
large
bulky
nodes.
Right,
hang
on
a
thread
occupied
due
to
with
managing
high
right
throughput
I
kept
everything
in
a
key
Value
Store
behind
a
you
know,
a
Json
RPC
requests
and
it
was
really
heavy
in
Ram
and
CPU,
and
you
needed
super
optimize
ssds.
A
This
is
really
annoying
and
all
these
things
are
much
costlier
than
what's
needed
when
our
goal
was
to
get
to
the
data
inside.
So
we
also
wanted
proper
decoupling
between
the
processes
producing
the
data,
so
the
blockchain
knows
and
its
intricacies
and
it's
it's
request
response
bottle
and
they're
all
different
and
the
data
itself.
A
We
wanted
the
data
to
be
the
interface
and
we
wanted
something,
also
extremely
reliable
in
the
sense
that
we
could
avoid
hitting
load,
balanced
nodes
that
had
all
sorts
of
different
views
of
the
world
and
that
we
need
to
have
like
client
code
to
like
latency
inducing
code
to
to
resolve,
what's
happening
there.
If
there's
another.
Fourth,
you
need
to
query
nodes
again
and-
and
you
know,
for
reorganization
heuristics,
for
example,
but
also
you
know,
we
wanted
something
better
than
even
the
websocket
streams
that
pretend
to
be
sort
of.
A
A
After
the
facts,
we
wanted
to
be
able
to
have
massive
parallelization
like
there
was
no
other
way
to
have
reliable
and
durable
in
performance
without
parallelization.
So
our
solution
was
the
fire
hose
and
the
fire
host
of
all
of
these
issues
in
a
radical
way.
We
took
a
radical
approach
because
we
wanted
to
solve
those
problems
definitively
like
meaning
that
there
would
be
no
further
optimization
possible,
except
attempting
to
bend
sort
of
space-time
Continuum
itself
right.
So
with
streaming,
we
with
even
multiple
nodes
pushing
out
data.
Multiple
nodes
are
actually
racing
to
push
the
data.
A
The
first,
through
a
consuming
process,
gets
the
first
to
get
out
like
you
can't
really
add,
remove
more
latency
there,
and
there
can
be
nothing
faster
than
immediately
when
the
transaction
has
just
executed
from
your
node
and
then
like
regarding
the
staple
processes
and
cost
flat
files
flat
fast
for
the
wind.
We
have
a
hashtag
for
that
right.
Flat
files
are
the
cheapest
much
cheaper
than
processes,
they're
easier
to
work
with
there's
nothing
simpler,
nor
cheaper
in
terms
of
computing
resources.
A
You
can
imagine
the
best
data
model,
the
most
complete
and
that's
what
he
done
for
fire
for
for
ethereum,
for
example,
that
both
the
the
data
model
for
ethereum
within
fire
hose
is
the
richest.
There
is
like
you
have
in
there
the
full
call
tree
internal
transactions,
you
have
the
inputs
and
outputs
as
raw
bytes.
You
have
the
logs.
Obviously
you
have
the
state
changes.
Like
you
see,
on
etherscan
down
to
the
internal
transaction
level.
You
have
balance,
changes
the
same
way
with
the
prior
value
and
the
next
value.
A
So
when
you're
doing
like
navigation
backwards
or
forward
you
get,
you
have
the
data
you
need.
You
have
also
gas
costs
at
different
places
and
there's
that
important
notion
of
total
ordering
between
things
happening
within
the
logs
State
changes
and
calls
all
of
these
things
happening
during
execution
are
totally
ordered.
So
you
get
in
there.
Everything
parity
traces
would
give
you
and
more
and
everything
you
would
need
to
rebuild
a
full
archive
node
from
Flat
files,
and
everything
there
is
goes
to
the
transaction
level,
not
rounded
at
the
Block
Level,
which
is
crucial.
A
If
you
want
to
index
with
Precision
right
it
doesn't
rounding
of
blockchain
information
at
the
Block.
Level
is
sort
of
a
was
meant
for
helping
in
consensus
right,
but
it
doesn't
mean
that
what
happens
mid
block
is
of
less
value
than
what
happens
at
the
boundaries.
So,
okay,
so
that's
very
interesting
and
now
regarding
reliability,
whoops,
no,
not
too
fast!
Regarding
reliability,
the
fire
hose
erpc
stream
provides
reorg
messages
like
new
block
or
undo
this
block,
or
this
block
is
now
final,
accompanied
by
a
precious
cursor.
I.
Think
that's
really
key
here
with
each
message.
A
So,
if
you
get
disconnected
and
upon
reconnection,
you
give
back
that
cursor
you'll
continue
exactly
where
you
left
off
and
potentially
receiving
that
undo,
say
signal
that
you
would
not
have
seen.
Were
you
disconnected
right?
So
you
will
get
it
so
with
the
guarantees
or
linearity
of
the
stream.
So
no
websocket
implementation
would
do
that
because
it
doesn't
make
sense
for
a
single
node
to
track
all
the
force
possible
even
two
days
after
the
fact
and
undue
messages
come
with
full
payloads.
A
So
you
get
all
the
Delta,
so
you
can
just
turn
around
to
your
database
and
apply
the
reverse
or
you
know,
pluck
again
in
the
the
full
payload
of
what
happened
in
the
block
and
just
like
what
you
do
is
so
it
doesn't
pose
on
the
reader
to
store
what
happened
like
at
that
previous
block.
If
the
signal
was
just
remove,
block
7000
right,
okay
and
when
you
commit
that
cursor
to
your
transfer
database,
well,
you
get
through
that.
You
know.
Finally,
some
consistency
guarantees
within
your.
A
You
know
your
your
back
end,
so
some
of
our
users
told
us
they
could
cut
ninety
percent
nine
zero
of
their
code
reading
the
chain
because
they
were
relying
on
some
that
reliable
stream
and
okay-
and
it
also
lays
lies
down
the
foundation
for
massively
paralyzed
operation
files
plus
the
stream.
And
so
this
is
the
future
of
the
graph
unbeatable
performance
and
it's
core
to
our
multi-chain
strategy,
because
you
know
any
blockchain
can
have
that
data
model.
A
Now,
let's
dig
into
sub
streams.
Substreams
is
a
powerful
clustered
engine
to
process.
Blockchain
data,
it's
a
streaming
first
engine
and
it's
powered
by
the
fire
hose
underneath
and
its
data
models
of
the
chain.
So
let's
dig
in
here
are
a
few
quick
facts.
It's
invoked
as
a
single
Jace,
grpc
call
and
within
the
request
we
provide
all
the
transformation
code
like
you'll
have
in
there.
Oh,
it's
too
low
you'll
have
in
there
the
code
from
some
wasm
modules
relationships
within
the
modules,
and
you
know
all
the
transaction,
the
Transformations.
Within
the
request.
A
It's
not
a
long
running
process,
except
if
you
run
it
for
long.
It's
not
a
service.
You
you
spin
up
right
and
the
backing
nodes
are
stateless
which
provide
nice
scalability
properties.
Modules
for
Transformations
are
written
in
Rust.
They
compile
to
wasm
and
they're
running
a
secure
sandbox
on
on
the
infrastructure,
they're
similar
to
the
sub
graphs
and
the
ultimate
data
source,
being
the
blockchain
data
being
deterministic
all
the
transformation.
Outputs
are
also
deterministic
and
the
request.
A
If
the
request
you
send
involves
process
prior
history,
even
if
it's
15
million
blocks
well,
the
substream's
runtime
will
then
turn
around
and
orchestrate
the
execution
of
a
multitude
of
smaller
jobs.
In
parallel
fuse,
the
is
lost
on
the
Fly
for
you
and
aggregate
the
results
to
simulate
a
linear
execution.
A
So
you
would
see
a
dime
in
the
difference
and
all
the
results
are
streamed
back
to
you
as
fast
as
possible,
with
same
guarantees
provided
by
the
fire
hose
with
a
block
per
block
cursor
and
a
transparent
handoff
from
batching
into
a
historical
processing
to
the
real-time
low
latency
rewards
aware
stream
of
the
head
of
the
chain.
So
let
me
show
you,
if
you're
interested,
how
we
create
one
of
these
things
raise
your
hand
if
you're
curious,
okay,
you're,
good
okay.
So
let's
start
we
start
with
a
manifest
like
that.
A
A
Contracts
between
modules
are
about
data,
so
there
are
protobuf
models
similar
to
the
protobuf
models
of
the
root
chain
of
the
chains,
the
layer
ones,
and
you
have
pointers
to
the
binary
that
you're
working
on
your
drive
and
all
that
and
you
have
Imports
and
imports
are
actually
very
interesting
because
you
can
import
third-party
substreams
packages
and
these
yamo
can
be
packaged,
and
so
you
can
import
from
someone
else's
package.
You
can
write
your
own
or
combine
both.
A
That
means
substreams
enables
composition
at
transformation
time,
which
I
think
is
pretty
unique
in
a
pretty
Game
Changer
and
then
follow
up,
and
there
you
have
the
module
sections
which
defines
the
relation
between
the
different
modules,
and
you
see
it
defines
it
directed
a
cyclic
graph.
You
have
modules
that
slowly
refine
the
data
and
so
there's
two
types
of
modules:
one
the
mapper,
the
first
up
there
map
pools
and
this
one
takes
inputs,
does
transformation
and
outputs.
It's
parallelizable
down
to
its
core
block
wise,
though
massively
paralyzable
and
then
there's
the
store,
input.
I.
A
Think
it's
awesome.
This
one
takes
any
inputs
and
outputs
a
key
value
store
that
or
something
accumulated
in
a
stateful
way
and
stores
can
then
be
queried
by
Downstream
modules
and
okay,
so
we'll
see
a
bit
more
after
then,
the
name
corresponds
to
the
function
in
the
wasm
code
and
the
inputs
can
be
of
a
few
things:
either
the
raw
fire
hose
feed
so,
for
example,
The
Source.
Here
that
means
the
block
with
all
transactions
you
know
for,
for
that
particular
block,
and
it
can
be
the
output
of
another
module.
A
Like
you
see
down
here,
the
input
of
map
map
pools
so
you'll
get
the
data
as
bytes,
and
it
also
be
a
store
which
would
be
a
reference
we'll
see
in
the
next
slide
there
and
on
the
store
pools.
Here
you
see,
there's
an
update
policy
which
sets
constraints
on
what
you
can
do
with
the
store
and
it
defines
a
merged
strategy
for
when
you're
running
paralyzed
operation.
A
Okay,
I'll
get
to
that
also
a
little
later,
and
the
value
type
field
will
help
anyone
decoding
understand
what
bytes
there
is
in
that
store,
so
you're
going
to
UI,
you
can
jsonify
them
and
your
code
consuming
can
automatically.
You
know,
decode
them
with
protobuf
all
languages
supported.
Otherwise
the
key
value
is
just
Keys
as
strings
bytes
values.
A
Very
simple-
and
one
thing
to
note
here-
is
that
because
it
has
deterministic
input,
it's
possible
to
Hash
a
module
like
the
kind
and
all
of
its
inputs
and
the
pointers
to
its
parents
and
including
the
initial
block.
So
you
have
a
fully
determined
and
hashable,
let's
say
cache
location
for
all
of
the
history,
similar
to
git
right
all
the
history
of
data
produced
by
by
and
you'd
hash,
also
the
wasm
code
right.
So
it
makes
it
for
an
extremely
cachable
system
and
highly
shareable
and
cross
verifiable
output
of
modules,
which
opens
really
interesting
possibilities.
A
A
Okay,
you
see
the
relation
there,
so
this
gets
piped
to
that
and
if
we
add
another
module
here,
you
see
how
the
graph
comes
together.
This
one
computes
the
price.
This
is
unit
swap
V3
thing,
it
computes
the
price,
but
it
you
want
to
get
them
for
certain
pools,
because
maybe
you
want
to
use
the
decimal
placements
in
the
pool,
we'll
see
a
little
bit
more
there
and
when
you're
running,
let's
say
you're
running
that
at
block
15
million
well
you're
guaranteed
the
runtime
guarantees
that
the
store
you'll
have
to
execute
code
at
block.
A
50
million
will
have
been
synced
linearly
or
in
parallel,
but
you
wouldn't
know,
but
it'll
give
you
a
full
in-memory
store,
eventually
backed
by
some
disk
but
whatever,
and
you
can
query
the
key
value
store
at
each
block.
It's
guaranteed
to
be
synced
for
you,
that's
exciting,
no!
Okay,
so
you
see
the
dags
fully
being
built
right,
so
the
dependency.
A
So
now,
let's
let's
this
leads
us
to
composability
see
each
color
here
means
a
different
author
and
modules
written
by
different
people,
ideally
the
most
competent
for
each
right
like
we
would
hope
they
would
crop.
You
know
analyze
what's
on
chain
and
refine
the
data
and
Abstract
it
to
new
heights,
and
the
contract
between
the
handoffs
is
always
data.
A
While
someone
wants
to
build
on
top
of
it,
something
like
that,
if
someone
wants
to
compute,
you
know
the
USD
denominated
volumes,
aggregation
of
nft
sales
on
openc,
you
know,
you'll
take
some
sales,
you
merge
it
with
the
price
and
we
see
here
that
little
Trader
Inc.
He
seems
he.
Maybe
he
wants
to
feed
that
into
his
trading
bot
because
it's
a
streaming
engine
we're
not
storing
that
in
the
database
yet
right.
But
this
begs
the
question:
where
is
that
all
that
beautiful
data
at
land?
Where
does
it
get
piped?
A
That's
where
it
thinks
head
up
like
substreams,
being
limited
to
the
transformation
stage
of
the
ETL
analogy?
Remember
it
doesn't
really
care
where
you
load
it
and
that
could
be
anywhere.
These
are
just
a
few
examples.
You
can
load
that
in
databases.
We
already
have
a
sync
for
postgres
and
you
hook
to
sub
streams
and
it
just
loads
it
into
postgres,
with
a
data
model
that
we've
agreed
upon
right.
A
If
you
write
it
in
a
certain
way,
it
just
thinks
over
there
and
or
message
cues
or
whatever
you
know:
data
lakes
or
some
Bots
or
some
trading
already.
You
know
so
I,
don't
know
some
whale
detector.
You
want
to
hook
directly
on
the
stream
or
also
so
something
I
think
big
for
doing
some
ad
hoc
data
science,
because
you
know
you
have
a
really
fast
engine
allows
you
to
process
the
whole
history
in
like
it
can
take
a
few
minutes
to
process
the
whole
of
it
here,
image
to
go
and
pluck
some
new
insights.
A
A
Well,
so
we're
working
to
make
substreams
feed
directly
into
into
graph
node
to
then
provide
the
same
loading
experience
and
then
querying
experience
that
you've
come
to
know
and
love
and
you'll
be
able
to
deploy
a
sub
grasp
this
time,
not
containing
assembly
script,
but
a
stud
streams
package
with
an
entry
point
and
would
process
the
history
in
parallel
and
load
that
in
your
database
in
Crazy
speeds,
so
stay
tuned
for
that
that's
not
out
yet,
but
you
know
soon.
Okay,
and
so
this
is
a
simple
example
in
Python,
it's
not
really
longer
than
that.
A
You
have
two
one
or
two
dependencies
like
grpc,
so
that
you
can
use
the
query.
So
we're
leveraging
a
lot
there
and
you
can.
You
know,
see
that
spkg
there
we
can
use
that
to
code,
gen,
python
classes
and
helpers,
and
all
of
that
because
it
turns
out
that
the
Manifest
through
the
spkg
there
is
for
those
who
know
protobuf,
is
they
file
descriptor
set.
It
contains
all
of
the
things
all
the
protobuf
definitions,
so
the
spkg
also
contains
all
the
wasm
code,
the
module
graph
information.
A
You
know
the
dependencies,
the
inputs
and
all
that
and
even
some
documentation
everything
is
needed
is
in
there,
so
you
can
pass
it
down
to
the
you
know:
modules
you
take
it
from
from
the
disk
and
boom.
You
send
the
request
to
the
server
and
it's
running,
so
you
can
deploy
packages
also
very
easily
and
consume
them.
Very
simply
this
way.
There's
a
few
Imports
we've
omitted
there,
but
it's
simple.
This
is
the
show.
Okay
and
let's
look
at
a
simplified
data
model
for
unionist
swap
V3
and
I'll
show
some
code
making
use
of
it.
A
Okay,
so
this
here
the
pool
is
a
list
of
pool.
This
is
actually
what
gets
handed
off
from.
You
know
our
mapper,
which
finds
the
pool
that
were
created
down
to
the
store
pool
which
we're
going
to
look
also,
and
so
it
has
a
list
of
tools
and
the
tools
you
can
imagine
an
address
and
the
two
tokens
that
are
concerned
here
and
we
have
a
reference
to
the
Token,
also
which
is
going
to
be
very
useful,
to
enrich
the
data
down.
A
So,
let's
see
what
happens
in
the
mappers,
so
this
is
a
sample
rust
code.
Raise
your
hand
if
you
love
rust.
Where's
your
hand.
If
you
no
rust,
okay,
it's
very
simple
here:
I'm
gonna
go
through
you.
Have
the
map
pools
function
corresponding
to
the
Manifest?
There
has
one
input
the
block.
This
is
the
fire
hose
block
with
all
transactions,
all
logs,
All
State
chains.
You
can
craft
your
own
triggers
in
there
as
you
wish,
but
we
have
a
simple
Voyager
and
see
that
that
line
there
blocks
events.
A
You
have
a
thing
that
goes
through
transactions
and
it's
going
to
trig
on
pool
created
and
that
pool
created.
Object
in
Rust
was
actually
code
gen
from
the
Json
API.
So
you
can
just
give
the
instruction
and
we're
going
to
filter
it
for
only
the
V3
swap
Factory,
and
then
that
beautiful
filter
map
will
give
us
the
log.
Okay
and
then
we'll
we'll
output,
we're
going
to
collect
some
of
these
things
into
one
list
of
pools
and
it's
assigned
to
the
pools,
object
there
and
and
notice
that
little
thing
here.
A
This
is
the
RPC
create
unit
Swap
tokensing.
This
actually
hits.
You
know
an
e-call
on
a
node
behind
similar
to
what
we
have
in
sub
graphs.
That's
actually
very
important.
It
means
that
once
we've
processed
this
layer
once
and
we've
done
it
for
the
history,
it
can
be
cached
very
efficiently,
so
anyone
relying
on
that
thing
will
never
need
to
reprocess
it
again.
You
can
give
the
package
to
someone
and
they
can
access
the
stores.
A
That's
been
cached
by
other
people
immediately,
so
you
could
go
to
block
50
million
and
you'll
have
the
list
of
all
pool
created
that
you
can
query
super
fast.
You
can
depend
on
it.
Also,
so
I
think
that's
pretty
cool,
and
here
that's
the
store
modules.
The
store
module
is
pretty
simple
here.
It
receives
C
the
pools
from
the
output
of
the
prior
module
and
it
does
it
Loops
through
the
little
pools
there
and
calls
output
set
and
see
the
key.
A
There
is
pool
colon,
the
address
is
going
to
store
the
Proto
buff
encoded
stuff
of
the
pool
with
the
token
decimals,
for
both
right
did.
I
say
that
to
be
constrained,
so
the
store
here
is
constrained
in
two
ways.
In
order
to
preserve
parallelized
ability,
the
stores
are
right
only
you
cannot
read
your
rights,
otherwise
that
would
make
some
potentially
cyclic
and
they
expose
only
the
function
defined
by
the
update
policy
in
this
case
it's
set.
A
So
let's
see
what
happens
if
we
run
things
in
parallel,
so
here
we
have
two
jobs
covering
two
segments
of
the
chain.
You
know
one
million
block
each
and
to
see
those
ugly
arrows
there.
They
correspond
to
a
pool
created
event,
and
so,
in
our
code
you've
seen
we
would
write
a
key
for
each
of
them
and
so
in
the
first
partial
run,
we'd
have
a
what
we
call
a
partial
store
with
four
keys
and
the
next
one.
A
If
you
do
that,
you
can
paralyze
endlessly
so
we'd
have
here
a
complete
store
with
six
keys
and
now
at
that
place
we
have
a
snapshot,
we
can
have
periodic
snapshots,
and
so,
if
you
want
to
go
and
explore
the
chain
at
any
point
in
time,
you
have
a
snapshot
plus
a
little
partial,
you
can
have
the
state
synced
at
any
block
height.
So
so
this
one,
you
know,
has
the
last
key
win
policy,
but
you
have
a
few
others
like
min
max
ad
and
another
one
like
first
key
wins.
A
So
if
you
merge
them,
you
have
set
if
not
exist,
and
then
that
that
allows
us
to
build
different
aggregations
right,
you'd
like
to
see
that
running,
live
every
few
minutes
now,
I
need
to
bring
that
other
window
up.
So
you
see
that
okay,
that's
good
enough!
Okay,
so
let's
imagine
we
want
to
see
that
pool
pools
created
thing.
Okay,
do
you
see
that
I
want
to
see
the
output
I'm
going
to
run
that
I
hope
everything
is
good?
A
You
know
the
demo
of
gods
are
connecting
okay,
okay,
whoa,
not
too
fast,
so
this
is
going
through
starting
at
the
beginning,
and
we
have
there
a
pool
credit
event
and
see
that
we
have
everything
decoded,
because
it's
the
protobuf
thing
we
have
the
thing
to
decode
it.
We
could
feed
that
it
arrives
on
the
wire
as
bytes
and
it
properly
serialized
bytes,
and
then
we
see
that
the
token
address
is
there.
We
have
a
decimal,
we
have
the
address,
and
so
what
that
means
is
pretty
crazy.
Already.
A
Oh,
do
I
see
that
okay,
what
that
means
is
that
you
can
inspect
the
chain
with
your
code
at
any
place
in
time
for
a
mapper,
especially
you
can
go
there
and
I
could
run
it
again
here
and
say
and
say:
I
want
to
run
the
mapper.
Let's
say
a
block
I,
don't
know
something
more
recent.
Give
me
a
recent
block.
What's
the
block
yesterday,
fifteen
seven
million
I
know
something
like
that.
Okay
and
see.
Is
there
anything
recent,
so
there's
some
stuff
right?
Some
things
are
recent.
A
Can
I
see
that
someone
still
created
a
new
pool
and
I
can
inspect
my
code
to
make
sure
it
works?
Where
is
that
come
on
right?
This
one
was
rap
ether
and
infinity.
They
were
just
created
that
address
as
a
new
pool,
so
you
can
go
and
test
your
code
everywhere.
What's
that's
done
well,
you're
set
right.
You
can
go
then
to
the
next
data
dependency.
So
this
really
changes
Dynamics
for
debugging,
and
this
can
also
work
for
stores
they
can.
A
You
can
ask
for
a
store
and
it's
going
to
process
it
in
parallel,
and
then
you
can
inspect
all
the
keys
that
exist
there
or
see
the
Deltas
coming
through.
Okay,
and
let
me
show
you
something
running
in
parallel,
uh-huh
graph
out
now.
This
is
very
interesting
because,
oh
no,
let's
start
it
at
15
million.
Let's
say
I
want
graph
out
at
15
million.
I
didn't
run
it
again
before
and
I
want.
So
let's
write
it.
So
this
starts
a
whole
bunch
of
parallel
processes
and
you
see
up
there.
A
The
number
of
blocks
per
second
yesterday
I
had
about
8
000
on
Solana
blocks.
I
had
16
000
depends
on
the
power
you
put
behind
there,
but
this
all,
like
you,
said
the
pool
counts
as
a
dependency
on
the
pools.
The
pools
is
further
down,
so
we're
able
to
schedule
things
and
all
that
just
massively
parallel
and
once
that's
ready.
Let's
say
everything
was
done,
I
would
start
streaming
and
get
all
the
content.
And
so
let
me
show
you
the
graph
out.
A
It's
very
interesting
because,
because
graph
out
graph
out
has
refined
the
data
up
to
entities
now
we're
talking
about
database
tables
and
Fields
and
you
get
and
you
get
out
of
that.
Do
you
not
see
it?
Okay,
that
wait
a
second
here,
so,
let's
imagine
see
we
have
token
and
an
update
and
you
have
the
field
derived
if
and
you
have
the
old
value
and
the
new
value
I
don't
know
if
you've
seen
this
thing
in
the
data
Science
World.
A
This
looks
very
much
like
change
change,
data
capture,
cdcs
that
can
power
a
lot
of
large-scale
systems
and
and
you
have
the
prior
after
so
you
can
feed
that
to
your
let's
say:
postgres
apply
the
changes
and
when
you
have
an
undo
signal
it
gives
you
back
that
payload.
You
then
just
flip
everything
and
you
have
guaranteed
linearities
to
your
store
with
a
cursor
and
it
is
Flawless.
It's
just
extremely
simple:
to
keep
your
things
in
sync.
A
You
also
want
to
have
a
slack
body
can
have
it
undo
message,
remove
the
message
if
you
have
a
thing
coming
through
right,
so
I
think
that's
pretty
cool.
What
do
you
think?
Okay,
okay,
that's
cool!
Can
I
shut
that
down
here.
Do
I
have
another
window,
I'm
close
to
them.
Prepare
your
question.
Please
ask
ask
them
succinctly:
we
have
a
half
a
minute.
I
just
wanted
to
have
a
final
final
note
there.
A
Therefore,
introducing
new
value
flows
and
also
I'm,
seeing
new
products
new
Services
being
offered
directly
on
the
network.
You
know
to
satisfy
some
needs
that
perhaps
couldn't
be
addressed
before
and
I
mean
there's
a
place
for
you
in
there
as
a
developer
as
an
indexer
as
someone
who
realizes
the
radical
benefits
of
such
a
platform
and
who,
who
builds
on
it
and
promotes
it.
So
my
ask
to
you:
is
you
go
to
you?
Go
try
substreams,
put
pressure
on
your
favorite
layer
ones
so
that
they
do
integrate
the
fire
hose
natively.
That's
pristine's.
A
Aptos
has
done
that
recently.
Some
others
start
wear
I,
think
so
that
makes
it
everything
we've
seen
today
becomes
immediately
available
to
them.
Sell
them
on
the
goodies
so
also
join
a
Discord
I,
love,
follow-up,
follow-up
questions
and
come
see
me
afterwards.
I
love
feedbacks
on
these
sort
of
things.
All
of
this
is
open
source.
So
let's
dig
and
build
something
of
the
like
together
the
biggest
blockchain
platform
on
Earth.
Thank
you
for
your
time.
Today.
B
Hey
so
one
question,
modularity
and
composability
of
these
substreams
is
super
super
powerful,
but
still,
if
I
look
at
this
compared
to
SQL
and
like
DVT
models
right,
it's
a
lot
more
complex.
So
how
can
we
enable
people
to
really
kind
of
learn
this
and
like
build
these
kind
of
hyper
modular
data
streams
so.
A
Sql
layer
like
this
is
powering
going
through
history.
It's
an
adult,
transform
with
stateful
storage,
but
you
would
pipelive
into
SQL
store
to
do
other
things
right.
You
would
have
refinement.
You
have
Knowledge
from
the
community
as
to
how
to
analyze
this
and
that
protocol
ever
ever.
You
know
increasing
refinements,
but
then
you
might
store
that
in
your
store
with
off-chain
data
and
maybe
that's
best
fit
for
you.
Maybe
you
feed
it
into
a
subgraph.
That's
what
you
need!
You
had
a
total
decentralized
solution
and
you
don't
need
to
host
anything.
A
C
A
It
is
not
the
same
Paradigm
to
enable
parallelization
Nation.
You
need
to
distinguish
the
data
dependencies
and
that
infers
the
number
of
stage
of
parallelization.
That
is
needed.
It's
not
easy
at
all.
Actually,
it's
pretty
crazy
to
try
to
parallelize
the
subgraphs.
We
try
that
that's
what
yield
us
to
design
substreams
by
cutting.
You
know
uni
swap
stuff,
so
you
will
want
to
go
and
write
in
Rust
modules
and
it's
a
different
Paradigm.
So
it's
not
just
an
easy
switch
I
admit,
but
it
brings
us
to
the
next
stage,
an
evolution.