►
From YouTube: The Graph's Core Devs Meeting #5
Description
The Graph’s Core Devs Meeting #5
This video was recorded: Tuesday, July 6 @ 8am PST, 2021.
A
Alrighty
welcome
everyone
to
the
fifth
protocol,
town
hall,
now
being
converted
to
the
core
devs
call,
so
thank
you
for
joining
us
just
wanted
to
say
a
few
words
about
this
transition.
So
we
noticed
over
the
last
few
months
we
see
a
lot
of
engagement
from
indexers
developers
that
are
working.
You
know
on
graph
node
or
you
know
very
closely
to
the
protocol
and
we're
ready
to
create
a
more
focused
group
around.
A
You
know
core
dev
processes,
doing
a
lot
more
ideation
and
building
in
public
and
meanwhile
transitioning
to
a
bit
more
of
a
community
call
cadence,
so
delegators
curators
sub
graph
developers
can
have
their
own
forum
for
their
own
discussions
and
challenges.
I
would
also
like
to
welcome
streaming
fast
to
the
core,
devs
call
and
the
graph
community.
This
will
be
the
first
call
they've
joined,
I'm
so
excited
to
see
what
they
come
up
with
in
their
research
and
working
with
you
know,
edunote
and
other
teams.
A
I
also
would
like
to
welcome
oliver
zerhuzen,
who
has
joined
the
foundation
to
work
on
governance,
so
you'll
be
seeing
him
around
and
a
lot
more
frequently
in
the
core,
dev
and
community
calls
and,
lastly,
wanted
to
welcome
semiotics
they'll
be
presenting
a
little
later,
but
excited
to
have
him
lean
and
more
with
our
community
as
well
with
that
arielle.
Do
you
want
to
take
it
away.
B
We
already
talked
about
the
reasons
for
having
this
gip,
but
what
I
want
to
like
update
is
that
these
are
already
approved
in
the
in
the
ggps
three
and
four
by
the
council,
and
these
are
undergoing
like
a
boat
in
the
an
unchained
boat,
so
the
upgrade
happens.
Okay,
so
we
already
have
a
four
out
of
six
votes,
and
it's
it's
probably
will
happen
today
tomorrow
or
or
the
next
day.
B
The
there's
some
delegation
working
group
that
is
being
organized
by
oliver
he's
collecting
some
feedback
about
the
community.
From
from
all
the
all
the
delegation
feedback,
so
core
devs
can
be
connected
and
we
work
on
the
on
on.
We
know
the
right
focus
to
to
give
to
the
work,
but
he's
going
to
talk
more
about
that.
So
oliver,
if
you
want
to
take
from
here.
C
Now,
since
january,
we
have
in
fact
seen
more
more
items
come
through
and,
as
a
matter
of
fact,
it's
quite
a
bit.
If
you
actually
screen
through
the
forum
posts,
you
find
probably
about
a
couple
of
dozen
different
form
threads
that
touch
on
a
number
of
different
things,
really
in
addition
to
what
brandon
had
collected.
C
So
these
are
the
items
here
that
that
you're,
seeing
and
we've
really
formed
a
cross
network
group
where
we
have
aerials
in
there
chris
russell's
in
there
from
the
indexer
community.
We
also
have
chris
ewing
in
there,
from
from
the
delegator
community,
providing
us
sort
of
guidance
and
feedback.
You
know
from
that
end
and
we
are
really
calibrating
on
what
are
the
most
important
issues
to
the
delegators.
C
And
how
should
we
prioritize
them-
and
we
just
had
just
coming
out
of
a
meeting
last
last
hour,
had
a
very
interesting
discussion
around
sort
of
the
dynamics,
some
of
the
constraints
that
we
have
within
the
network.
So
we're
really
having
deep
discussions
so
that
we
come
up
with
you
know,
solutions
that
are
both
meeting
what
the
delegators
want,
but
that
also
fit
well
within
our
protocol.
C
The
goal
really
is
to
present
back
to
the
community
when
we're
done
with
our
discussions,
a
sort
of
roadmap
around
delegator,
you
know
process
and
parameter
improvements,
and
that
is
something
that
you
know
we're
going
to
have
posted
in
the
forum
and
that
we're
looking
to
get
feedback
on
in
essence,
what
we
want
to
come
up
with
is
you
know,
guidance
for
the
devops
team
that
they
know
out
of
the
couple
dozen
forum
threats
that
are
out
there.
You
know
what
should
they
be
focusing
on
next,
as
they
do
protocol
enhancements.
C
C
So
when
you
have
seen
the
you
know,
the
post
being
on
the
roadmap
being
posted
in
a
couple
of
weeks,
make
sure
you
give
us
your
feedback,
how
you
think
about
it,
whether
you
feel
that
we
got
it
right
or
whether
we're
going
to
fine-tune
some
of
that
now.
Arielle
has
already
worked
on
a
gip
draft.
That
already
is
essentially
addressing
some
of
the
items
that
you've
seen
in
the
couple
of
slides
ago,
and
this
is
something
I
really
encourage
everyone
to
go
into
the
forum
post.
C
It's
actually
labeled
under
proposal
to
change,
indexer,
cut
mechanism.
It's
a
very
interesting
proposal
that
addresses
a
few
pain
points
that
includes
the
transparency
around
understanding,
how
indexers
set
their
reward
cut
percentages,
and
it
also
makes
it
actually
easier
for
indexers
to
maintain
their
cut
percentages
without
having
the
need
to
readjust
every
time.
Delegation
amount
changes
and
it
also
actually
addresses
you
know
one
specific
instance
that
we've
seen
coming
up,
which
we
call
sandwich
attack.
C
Where
allocations
can
be,
you
know
changed
I
mean
the
reward.
Cuts
can
be
changed.
You
know
right
before
and
after
allocations
that
have,
you
know,
presented
themselves
as
a
pain
point,
and
that
is
also
a
component
that
is
being
addressed
so
make
sure
you
get
into
the
forum
provide
your
feedback
there.
We
are
currently
fine-tuning
that
arialis
and
you
know,
are
looking
to
get
a
poll
posted
here
fairly
soon,
and
that
was
my.
B
B
I
I
encourage
any
like
indexer
delegators
to
participate
and
add
feedback
and
and
see
like
everything,
makes
sense
and
and
it's
aligned
with
the
with
a
solution
for
for
this.
This
is
the
current
rewards
of
distribution.
When,
when
you
have
indexers
is
taking
30
percent
of
the
stake
and
delegates
are
70
percent,
let's
say
we
have
11
rewards
and
the
index
forecast
is
ten
percent
indexes
will
get,
it
will
get
the
the
ten
percent
of
the
whole
rewards
and
the
relationships
the
the
rest.
B
B
So
it's
not
really
like
it's
not
that
the
rewards
are
accruing
for
about.
For
the
amount
of
delegation
it's
being
like,
distributed
from
the
absolute
amount
of
rewards
and
the
new
calculation,
this
is
the
new
calculation
and
changing
the
the
slide
for
the
same
mistake.
What
it's
going
to
do
first,
is
to
assign
rewards
based
on
the
on
how
much
the
indexes
or
the
legislators
are
staking.
B
B
I
posted
like
a
gip
about
this
and
there's
some
as
a
link
to
spreadsheet,
where
you
can
see
like
how
the
delegator
yield
change
and
the
indexer
yield
change,
based
on
the
on
different
index,
indexer
to
the
legit
ratios
and-
and
you
can
see
basically
this
this
formula
in
in
in
movement
based
on
unchanged.
So
I
encourage
that
you
take
a
look
at
the
spreadsheet
and
provide
some
feedback.
B
This
candidate
implementation
had
a
change
where
we
take
a
snapshot
of
the
indexercats
when
the
allocation
is
created
and
that's
its
use.
So
multiple
different
allocations
can
have
a
different
extra
cut
based
on
when
it
was
created.
Okay
and
the
reason
for
that
is
to
avoid
last
minute
condition
change.
B
B
So
the
the
the
resolution
by
the
alliteration
following
the
the
habitation
charter
was
to
draw
the
disputes,
so
the
funds
were
returned
to
the
the
the
deposit
were
returned
to
the
fishermen
and-
and
it
was
not
slashed
okay,
so
that
that's
the
main
thing
the
as
a
next
step,
the
the
team
is
going
to
is
working
on
reproducing
the
the
wrong.
If
call
now,
the
investigation
is
to
find
out.
This
was
a
particular
condition
on
one
of
the
of
the
theme
notes
or
or
something
else.
B
So
the
the
the
current
line
of
work
is
setting
up
a
weight.
So
we
can
reproduce
the
the
determining
mission,
okay
and
we
will
post
in
the
forum
more
details
about
that.
One
more
thing
to
mention
about
disputes.
Is
the
team
published
this
cli
that
you
can
use
and
install
easily
on
your
on
your
terminal?
B
It's
it's
a
cli
that
lets.
You
manipulate
all
the
manage
all
your
disputes.
You
can
like
list
open
disputes.
You
can
create
index
in
this
field.
You
can
create
query
disputes
by
presenting
at
the
station.
So
this
is
like
a
first
version.
I
think,
will
help
a
lot
and
will
be
keep
on
improving
and
open
for
feedback.
D
Great:
hey
everyone,
I'm
adam,
I'm
product
manager,
working
at
edge
and
nodes
on
the
graph
node.
Let
me
find
yeah,
someone
talked
about
two
things,
two
things
today,
you
see
my
screen
yeah,
so,
firstly,
it's
about
core
processes.
Let
me
actually
just
narrow,
so
I
can
see
anything
so
obviously,
this
is
like
this
is
a
very
exciting
time
for
the
protocol.
D
These
are
the
two
sort
of
important
components
of
versioning,
so
the
spec
version,
which
is
the
version
of
the
subgraph,
manifest
what
that
looks
like
then
the
api
version,
which
is
the
version
of
the
mappings
api
so
and,
and
that's
going
to
be
particularly
relevant.
I
guess
with
an
upcoming
release,
which
I'll
talk
about
in
a
second
and
so
essentially,
I've
just
got
a
couple
of
updates
and
requests
for
people,
so
the
first
like.
Firstly,
we
really
want
to
move
to
more
frequent
requests
for
graph
node.
D
So,
firstly,
I
need
to
find
essentially
more
people
to
help
beta
tests
so
indexes
for
beta
testing.
So
we've
got
a
couple
of
people
who
we've
we've
got
a
couple
of
people
who
we
have
been
working
with
for
the
last
release.
We
want
to
essentially
move
to
doing
that
with
more
people
in
particular.
If
you've
got
software,
you
write
which
in
which
is
used
by
others.
So
we
know
that
with
the
0.23
they're,
quite
a
lot
of
migrations,
quite
a
lot
of
changes.
D
So
if
you
maintain
grafana
dashboards,
that
used
by
other
people,
then
we
want
to,
we
want
to
sort
of
know
that
we
don't
need
to
be
taken
by
surprise.
So
please
please
get
in
touch
either
in
the
forum
on
discord,
then
that
would
be
great
and
we
also
want
to
have
more
contributions
to
graph
cli
from
sort
of
called
from
outside
the
core
development
teams.
And
so
this
comes
just
comes
back
to
the
line
chain.
D
Assembly
script
work
unit,
testing
works
so
they're
essentially
test
creating
a
framework
which
you
want
to
like,
ideally
build
in,
and
so
we
wanna
like.
We
want
this
to
be
an
ongoing
thing
where
it
happens
more
and
more
often,
and
so
we
want
to
essentially
have
a
bit
of
a
working
group.
So
yeah
I'd,
like
I
think
petco
is
here
so
he's
it'd
be
great.
D
If
you
could
be
part
of
this,
but
if
you're
interested
in
being
part
of
this
and
working
how
we
could
build
more
things
into
the
cli,
then
that'd
be
really
great
and
please
get
in
touch.
And
then
the
last
piece
is,
I
guess,
a
call
to
subgroup
developers.
So
our
grading
assembly
script-
the
latest
version-
is
something
we're
currently
working
on
or
credit
to.
D
Otavio
he's
been
working
working
on
this
and
this
introduces
new
functions,
the
sort
of
new
latest
version
across
graph
node
graph,
cli
and
graph
ts,
and
so
we'll.
Obviously,
when
we
ask
we're
doing
sort
of
thera
testing
on
this,
but
we
obviously
want
to
make
sure
that
we
understand
all
the
changes,
because
there
are
some
breaking
changes
in
the
in
the
essentially
latest
version
assembly
script.
D
So
if
you're
interested
in
testing
the
latest
version
with
us,
then
then
again,
please
get
in
touch
that'd,
be
great,
so
I'll
pause,
I'll
pause
there.
That's
obviously
only
sort
of
a
light
touch
thing
on
releases,
just
in
case
there
are
any
questions
about
any
of
that
or
or
anything
in
general,
about
like
how
how
we're
managing
things
at
the
moment.
F
D
Official
good
good
question,
so
I
think
yeah,
I
guess
a
an
overarching
thing
is
that
graph
note
itself
is
hasn't
sort
of
had
its
1.0
version
yet
and
and
and
we're
working
our
way
towards
towards
that.
As
I
guess,
there's
a
major,
a
major
version
bump
and
I'll,
be,
I
guess,
a
bit
of
a
milestone,
because
when
there's
yeah
so
I'm
talking
about
what
we
want
to
fit
into
that.
But
I
think
in
general
it
will
probably
be
sort
of
minor
versions.
D
So
I
guess
I
like
it
like
every
couple
weeks
and
I
think
that
wouldn't
necessarily
like,
I
think
we
still
need
to
work
out
how
that
fits
into
the
protocol,
because
I
think
there's
there
are
sometimes
some
reasons
and
like
like
like
at
the
moment.
There's
a
push,
I
guess
to
move
to
the
latest
version.
So
two
three
one,
essentially
because
it
allows
us
to
upgrade
the
ipfs
yeah
ivfs
node
that
everyone's
using
because
currently
the
0.22
they
support
sort
of
up
to
0.4.
D
So
there'll
be
some
instances
where
we'll,
I
guess,
encourage
like
like
folks
to
upgrade,
but
I
think
in
general
it
probably
won't
be
every
every
two
to
four
weeks
but
again
like
there
might
be
some
instances
where
we've
introduced
new
things
and
again
that
comes
down
to
the
sort
of
the
versioning
strategy
that
that
we,
I
think,
we're
still
kind
of
working
out
full
disclosure.
But
I
think
that
yeah,
hopefully
by
releasing
frequently
we
can,
we
can
again,
as
I
say,
build
that
muscle.
D
D
Them
off
off
now,
I'd
miss
some
I'm.
Essentially
it's
going
to
be
pretty
well
well
documented
and
because
it's
tied
to
that
api
version,
you'll
be
able
to
essentially
use
the
an
older
api
like
the
current
api
version
and
and
that
will
also
work.
Fine
and
graph
node
will
essentially
support
both
in
parallel
for
a
while,
and
so,
as
someone
says
that
there's
a
detailed
upgrade
guide
to
help
you
get
through
that.
D
Yes,
yes,
so
that's
on
yeah!
That
request
certainly
come
through,
like
I
wouldn't
say
that,
there's
a
specific
time
frame
but
like
I
think,
adding
some
more
control
in
terms
of
block
handlers
and
and
other
filters
has
certainly
come
up
quite
a
lot
and
hopefully
we'll
be
able
to
get
some
of
those
kind
of
improvements.
D
But
yeah
like
I
think
I
think
I
think
one
thing
slightly
off
topic,
but
we
want
to,
in
general
in,
like,
in
general,
start
to
be
a
bit
more
open
about
what
we're
sort
of
working
on
some
of
the
like,
like
ongoing
projects
and
also
things
that
we're
sort
of
discovering
and
so
yeah.
I
guess,
watch
this
space
on
on
on
that
front,
particularly
as
we
start
to
collaborate
more
with
streaming
fast
and
work
out
what
we're
all
respectfully
working
on.
D
D
D
I
think
there
is
an
indexing
reward
sort
of
flag
which
is
set
by
the
sub
graph
oracle,
but
then
there's
also
like,
I
think
we
need
to
establish
like
where
the
looks
control
is
for,
like
where
an
index
would
say
that
they
do
want
to
index
these
kind
of
subgraphs
or
make
those
kind
of
decisions
so
very
interested
to
talk
to
indexers
about
how
to
best
surface
thing.
This
config
configuration
on
the
network
so
yeah
I'll
be
reaching
out.
G
G
G
For
example,
indexers
need
to
optimize
pricing
and
consumers
need
to
bid
efficiently
to
get
the
best
service
and
we've
been
investigating
automating
query
negotiations
with
deep
reinforcement
learning
and
towards
that
long-term
goal.
Today,
we
would
like
to
share
some
initial
steps,
which
have
probably
some
more
short-term
impact
to
the
protocol,
and
this
is
about
again
indexers
query
price,
optimization,
because
this
is
an
essential
task.
G
Some
queries
cost
a
lot
more
in
terms
of
infrastructure
costs
and
latency
can
be
used
as
a
metric
to
measure
this
complexity.
However,
predicting
latency
given
a
graphql
query
is
a
very
hard
problem
and
we
started
tackling
this
issue
using
ai
and
I
will
let
matt
to
describe
what
we've
been
doing
so
far.
H
The
indexer
is
then
passed
with
pricing
that
query
and
figuring
out
how
expensive
it
will
be
to
run
it.
So
currently,
this
is
done
with
agora,
which
lets
you
lends
you
a
lot
of
flexibility
to
specify
how
exactly
you
want
to
price
a
query,
but
it
turns
out
that
actually
figuring
out
how
to
write
simple
rules
to
price
queries
does
not
quite
catch
all
of
these
very
expensive
queries,
which
is
a
very
hard
problem,
as
was
explored
in
this
ibm
cloud
research
paper
that
you
see
at
the
bottom
here.
H
So
what
we
are
doing
is
developing
a
deep
learning
model
to
predict
the
latency
from
a
raw
graphql
query,
so
this
is
actually
done
in
a
few
steps.
First,
we
take
the
raw
graphql
query
string
and
we
actually
need
to
turn
it
into
a
more
digestible
form
to
the
neural
network.
So
we
actually
take
each
individual
word
and
piece
of
syntax
in
the
query
and
we
turn
that
into
a
categorical
token.
H
So
to
solve
that
we
actually
moved
on
to
solving
query
generation.
Being
able
to
generate
queries
comes
with
many
advantages
number
one.
We
have
unlimited
data,
which
means
that
we
can
train
the
model
until
it
is
able
to
actually
learn
the
syntactical
qualities
of
graphql
and
actually
predict
the
latency
of
completely
new
queries.
H
There's
also
the
added
benefit
of
not
needing
needing
any
historical
occurring
data
at
all.
So
one
thing
we
could
do
is
we
could
pre-train
a
model
before
a
sub-graph
is
even
deployed
so
before
any
queries
are
even
indexed
for
a
sub-graph.
We
could
actually
already
have
a
way
to
predict
how
much
those
queries
would
cost
to
run.
H
So
this
is
all
great,
but
it
comes
with
one
very
large
challenge
and
that's
that
gendering
generating
queries
is
very
hard.
So
the
subset
of
random
strings
that
actually
form
a
valid
query
is
extremely
small.
So
what
we
have
done
is
we
have
built
a
tool
that
can
shrink
this
subset
of
this
set
of
random
strings
down
to
the
set
of
random
queries.
H
So
now
that
we
have
a
query
generator,
we
can
actually
plug
it
into
our
latency
prediction,
training
pipeline,
which
will
look
something
like
this.
So
now
we
will
have
a
query
generator
generating
a
query.
Our
latency
prediction
model
will
predict
the
model,
the
latency.
For
that
query,
we
will
also
feed
that
query
through
our
local
indexer
to
get
the
true
latency.
We
can
then
compute
the
error
between
the
latency
prediction
and
the
true
latency
and
then
trained
our
aim
model
as
before,
using
us
supervised
learning.
H
So
our
next
steps
are
to
actually
introduce
an
adversarial
component
to
this
query
generation
pipeline.
So
this
is
the
same
exact
graph.
You
just
saw
with
the
added
added
component
of
a
reward
arrow
going
from
that
supervised
learning
error
to
the
query
generator,
so
the
query
generator
will
be
trained
via
reinforcement,
learning
for
making
the
cost
prediction
model
mess
up.
So
now
it
will
be
incentivized
to
come
up
with
queries
that
are
interesting.
F
Here
I
can
share
some
comments
just
on
why
I
think
this
is
super
cool.
You
know,
so
a
common
challenge
that
you
see
in
a
lot
of
you
know.
Blockchain
protocols
is
figuring
out
how
to
cost.
You
know
different
operations,
usually
compute
and
storage,
and
you
know
so.
The
approach
that
you've
seen
taken
almost
every
protocol
to
date
has
been
to
kind
of
set
this
at
the
protocol
level
right.
F
You
know
came
to
a
similar
question
which
is
like:
how
do
we
price
or
how
do
we
cost
queries,
and
one
of
the
challenges
here
is
that
configurations
are
so
much
more
heterogeneous
than
you
would
see
with
like
node
operators
running
like
a
blockchain
node,
for
example,
you
know
the
the
cost
of
executing
a
query.
Is
you
know
very
contingent
on?
You
know
the
caching
that
you've
set
up
both
in
graph
node?
Caching,
that
happens
in
postgres
the
indexes
that
you've
created.
You
know
how
horizontally
scaled.
F
Your
setup
is,
so
there's
really
no
way
that
we
ever
could
have
done
like
you
know
the
equivalent
of
ethereum
gas
costs.
You
know
as
a
protocol
wide
set
of
set
of
costs,
and
so,
instead
we
ended
up
with
you
know,
agora,
which
you
know
zach's
presented
in
past,
calls
which
gives
indexers
a
way
to
flexibly,
communicate
their
own
costs.
F
F
You
know
that
are
idiosyncratic
to
my
specific
configuration,
and
actually,
how
might
I
change
that
configuration
in
response
to
what
those
costs
are
or
in
response
to
you
know
the
market
opportunity
of
the
types
of
queries
that
are
coming
in
you
know,
and
so
this
is
kind
of
the
first
step
in
really
making
that
a
first
class
experience.
You
know
so
the
tools
that
that
semiotics
and
the
models
that
somatics
are
learning
is
really
making
this
kind
of
peer-to-peer
pricing
price
discovery
and,
like
cost
communication,
really
efficient
and
so
yeah.
E
H
H
But
actually,
when
we
look
at
the
data,
we
find
that
a
very
large
majority
of
the
queries
are
actually
extremely
similar
to
one
another,
because
they're
generated
by
scripts
that
are
just
sort
of
filling
in
variables
or
doing
something
similar.
So
even
though
the
model
is
able
to
perform
well
within
this
small
data
set,
it's
not
expected
to
actually
perform
well
on
completely
new
queries,
because
the
set
of
unique
queries
that
the
hosted
service
is
getting
is
just
a
bit
too
small.
H
C
H
C
H
Yeah
so
originally
we
were
training
on
the
full
hosted
data
set
that
we
got
and
that
took
a
very
long
time.
We
actually
didn't
even
get
through
the
whole
data
set.
But,
as
I
said,
we
were
actually
training
on
lots
of
duplicates.
So
when
we
actually
processed
the
data
set
further
and
deduplicated
it,
we
got
a
data
set
of
unique
queries
and
we
were
able
to
train
on
that
in
a
few
hours.
I
think-
and
that's
that's
including
a
few
times
through
actually
so
yeah.
E
F
Yeah,
I
mean
just-
I
guess,
generally
I'd
love
to
figure
out
a
way
to
make
these
more
interactive.
You
know
I
love,
I
love
the
presentation
format
for
the
you
know
for
the
start
of
it,
but
now
that
we
do
have
multiple
core
dev
teams,
it
would
be
great
to
discuss.
You
know,
I
know,
there's
lots
of
other
meetings
happening.
You
know
where,
where
some
of
these
ideas
are
being
discussed
but
it'd
be
great
to
you
know
even
discuss
some
partly
formed
ideas
on
these
calls.
F
You
know
to
give
people
kind
of
a
a
taste
of
work
in
progress.
You
know.
One
thing
I
think
that
came
up
in
the
index
for
office
hours
with
streaming
fast
last
week
was
discussions
around
like
changes
to
the
you
know,
information
model
of
subgraphs.
You
know
so
the
idea
of
actually
breaking
subgraph
mappings
into
like
multiple
stages
that
that
can
be
run
concurrently.
I
So
I
think
brandon
you're
talking
about
the
parallelism
right.
You
want
me
to
go
through
a
little
bit
of
that.
The
story,
the
pancake
swap
story-
is
that
the
right
place
to
do
that
know
some
of
the
elements
that
we
we
thought
of
and
I'm
we're
writing
an
rfc
on.
That's
what
we're
doing
do
you
hear
me?
Well
yeah
yeah,
yeah
yeah.
I
think
that
sounds
great,
so
I
did
it.
I
did
a
demo
to
chris
from
graphos.
I
If
you
had
processed
the
full
history
linearly
and
the
thing
that
was
taking
two
months
right
linearly
through
mappings
and
querying
yeah.
So
that's
sort
of
a
blend
of
the
streaming
fast
technology,
because
it's
based
on
files,
opening
files,
processing
data
linearly
in
chunks.
But
the
fact
that
you
can
parallelize
that,
and
so
that's
what
some
of
the
thinking
we
want
to
bring
there
or
at
least
a
proposition.
We're
are
explaining
our
way
of
how
we,
how
we
thought
of
it,
how
we
implemented
it
and
see
if
that
resonates.
I
F
I
So
I
don't
know
if
that's
right,
okay,
so
look
the
streaming
fast.
Tech
started
from
the
principle
that
we
want
to
take
a
blockchain
and
extract
all
the
data.
There
is
all
of
the
data
there
is
so
that
we
theoretically
would
be
able
to
rebuild
a
node
of
the
concerned
blockchain,
because
we
would
have
all
the
data
all
the
data
right.
So
all
of
the
accounts
not
change
all
the
state
changes
all
of
the
whatever
you
find
in
the
storage
of
a
blockchain.
I
We
want
to
extract
and
put
into
files
so
that
we
can
process
them
and
and
files
have
great
advantages
as
opposed
to
processes
that
are,
for
example,
an
ethereum
node,
an
archive
node
right,
there's,
cpu
memory,
overhead,
there's
disk
access,
and
you
have
large
systems,
I'm
thinking
of
an
archive
node
here
large.
You
know
this
needs
to
be
managed.
You
want
a
copy
of
that
to
scale
it
out.
I
If
you
have
things
in
files
like
in
gcp,
we're
using
that
there's
not
even
a
cost
for
downloading
the
file,
there's
not
an
egress
cost,
everything
is
internal,
so
it's
extremely
cheap,
and
so,
when
we're
practicing,
I'm
telling
you
processing
the
the
full
history
of
that
section.
I'll
tell
you
we
didn't.
We
processed
the
whole
chain,
all
transactions
unfiltered
right,
and
we
did
that
four
times
and
it
still
took
like
for
that
part,
maybe
30
minutes,
because
that's
negligible
right,
we're
dealing
with
files,
it
unpacks
very
fast.
So
so
that's
our
core
principle.
I
We
take
the
chains
extract
the
data
put
them.
We
wanted
to
have
the
most
complete
description
of
all
the
protocols
we
handled,
so
we
defined
protobufs.
That's
what
we're
using.
We
wanted
that
to
be
easily
consumed
by
a
lot
of
different
languages.
So
that's
the
first
delineation
of
I
would
say
a
contract
between
you
know:
data
consumers
and
data
producers.
Any
chain
would
produce
protobufs
in
chunks
of
100
blocks.
I
These
are
stored
in
the
files
and
then
we
have
what
we
call
the
fire
hose,
which
is
being
integrated
by
the
team
right
now
that
would
bridge
files
and
real-time
streams.
So
the
nodes
that
produce
those
those
streaming
blocks
and
the
fire
hose
knows
to
go
to
files
knows
to
to
now
switch
to
the
live
stream,
ensures
contiguity
navigates
force
for
you,
that's
sort
of
a
you
know
that
the
fusing
of
those
things-
and
this
is
chain
agnostic,
and
so
where
am
I
with
the
stage
with
the
that?
Does
that
answer
your
question?
F
I
So
so
what
happens
is
that
that
is
for
specifically
the
parallelized
version
right.
Of
course
we
have
we
for
the
pancake
swap
story.
We
built
the
linear
version
that
just
goes
linearly.
This
one
already
went
faster
because
it
was
using
files
and
the
linear
storage,
not
relying
on
hitting
the
node
right,
because
there's
a
little
bit
more
data
in
there
we
tried
to
leverage
it
more
and
and
with
and
avoid
hitting
the
node
for
information.
I
So
that's
one
thing,
but
so
the
parallel
version
yes,
runs
the
first
stage
and
then
drops
the
entities
to
disk
and
then
the
second
stage
starts
and
picks
back
up
the
entities
that
would
exist
if
it
had
run
linearly,
and
that's
why
you
can
imagine.
You
can't
run
a
unit
swap
without
first
knowing
the
pairs
so
stage.
One
goes
and
extracts
the
pairs
for
the
full
history
drops
out
on
disk.
I
The
next
stage
starts
with
you
know,
knowing
the
pairs,
so
it
loads
the
pairs,
and
now
it
can
do
a
little
bit
more
work
computing,
whatever
price
average,
and
then
you
have
another
value
that
relies
on
the
price
average.
So
you
flush
the
prices.
First
ones
and
then
you
restart
process,
and
you
continue
from
the
pairs
plus
the
average
on
which
you
can
build
the
next
stage.
I
Then
you
write
a
little
bit
more
and
all
these
things
produce
very
quickly
and
thinking,
like
maybe
half
an
hour,
so
you
have
json
files
that
are
extremely
easy
to
introspect
and
you
have
full
history
so
for
iterating,
that's
also
one
of
the
core
principles.
If
you
want
data
agility,
the
processes
need
to
be
extremely
fast.
That's
why
we
always
strive
for
parallelization,
because
if
you
can't
do
that
in
in
blockchain
world
you're,
just
like
you're
gonna
take
more
and
more
time
ever
you
can't
you
can't.
I
We
can't
do
that
right,
we'll
have
more
computers,
we
can
spin
up
more
computers,
but
we
we
won't
be
able
to
extend
the
space
time
to
go
more
and
more
linearly.
So
that's
why,
in
our
opinion,
parallelism
is
important,
but
it
also
gives
you
a
lot
of
agility,
so
you
can
inspect
the
latest
bits
of
the
data
before
they're
hitting
postgres
and
that's
the
second
stage
in
our
operation.
I
We
took
those
files
and
then
generated
huge
csv
imports
and
were
using
sort
for
the
parallel
injection,
the
fastest
method
there
is
to
injecting
postgres
loading,
csvs,
removing
indexes
and
then
adding
indexes
after
really
to
have
an
optimal
injection
that
takes
how
long
three
four
hours
with
the
rebuild
of
the
index.
So
that's
how
we
managed
to
you
know,
shrink
the
full
history
in
six
hours.
Is
that
interesting?
Is
that
fun?
F
Yeah,
I
think
I
think
one
thing
that's
really
exciting
to
me
about
this
kind
of
model
that
you
guys
are
proposing
is
that
it
really
maps
a
lot
more
to.
I
think
what
a
subgraph
is
doing.
You
know
this
kind
of
the
pipeline.
Nature
of
you
know
the
data.
One
of
the
features
that
you
know
I
know
adam's
been
thinking
about.
A
lot
is
like
what
does
it
look
like
to
have
more,
you
know,
analytics
style
functionality
as
a
first
class.
F
You
know
you
know
concept
in
subgraphs
and
we
were
seeing
this
with.
I
believe
the
uniswap
subgraph,
which
pancake
swap
initially
you
know
forked,
which
was
that
a
lot
of
the
a
lot
of
the
overhead
was
actually
within
the
mappings
them
kind
of
you
know,
imperatively,
doing
these
sort
of
like
aggregations,
and
you
know
the
types
of
things
that
really
could
be
declared
declaratively
and
be
maybe
staged
as
like
a
downstream
part
of
the
pipeline,
but
because
we
didn't
give
you
that
facility,
it
was
like
okay.
F
How
do
we
put
everything
that
we'll
ever
need
inside
this?
One
mapping
function
that
you
know
runs:
you
know
sort
of
sequentially,
but
you
know
once
we
start
breaking
things
out
within
the
subgraph
and
of
these
pipelines.
You
know
it
really
starts
opening
up.
You
know
possibilities
for
you
know
for
extending
subgraph
functionality,
but
also
you
know,
on
the
the
performance
side
as
well.
D
Yeah-
and
I
think
I
think
one
of
the
interesting
things
that
interesting
challenges
there
is
that,
like
I
think
at
the
moment,
writing
subgraphs
is
like
conceptually.
It
just
runs
from
top
like
top
to
bottom
and
so
like,
like
it's
a
relatively
simple
mental
model,
whereas
obviously
this
is
making
much
more
powerful
much
like
much
faster
like
it's
definitely
the
way
we
want
to
go.
But
it's
like
thinking
about
how
we
surface
that
to
subwoofer
developers
in
a
way
that,
like
is
easy
to
build
the
powerful
subgraphs
that
they
want
to
build.
F
One
one
question
I
have
is
just
sort
of
an
open
question,
as
this
work
is
developing.
It's
just
what
are
the
impacts
on
determinism
from
the
file
approach
and
any
impacts
on
like
being
able
to
bring
along
the
the
proofs?
You
know
the
the
sort
of
provenance
of
the
data
along
with
you
know
the
data
that's
getting
added
to
the
files.
So
not
just
you
know
not
just
the
the
raw
data
itself,
but
hey
the
the
proof
that
this
data
came
from.
You
know,
xyz
block.
I
I
guess
there's
there's
two
aspects
to
that.
You
know
there's!
Okay,
so
you
know
that's
in
the
work,
the
notion
perhaps
or
the
proposition
we're
gonna
write
an
rxt
about
that
to
have
sort
of
a
network
sub
graph
that
fire
hose.
I'm
talking
about
is
raw
data
unmishmashed.
You
know,
there's
no
mappings
there.
The
blockchain
produces
a
lot
and
there's
data
in
there.
That
should
be
deterministic
and
one
way
and
we've
seen
with
the
work
in
aragon,
they're
they're
trying
to
have
another.
You
know
sort
of
parity
traces
and
they
figure
out.
I
There's
small
inconsistencies
between
the
implementation,
so
we
did
a
lot
of
work
between
different
clients,
output,
the
same
thing,
so
protobuf
is
the
same
and
the
content
there
could
eventually
be
sort
of
hashed
cross
verified
in
a
way,
and
that
could
become
that's
one
of
our
thinking
here.
What
we're
thinking
we're
going
to
write
rfc
about
that
have
yet
another
service
on
the
graph?
Perhaps
right
that
would
be
that.
I
Would
not
you
don't
query
it
you
don't
you
know
you
shape
it
up
to
just
to
have
you
know
the
information
you
want,
but
it's
not
you
know.
It's
raw
raw
unmapped
and
then
that
could
be
used
if
it's
composed
by
the
the
other
mappers.
So
you'd
have
yes
provenance.
Maybe
that
really
it
touches
on
a
composability
of
a
grass
and
it
starts
making
sense
when
you
have
thing
that
is
more
raw,
anything
that
is
less
raw
because
there's
refinement.
I
F
Right,
it's
a
lot
easier
to
compare
files
than
it
is
to
compare
the
sort
of
like
runtime
behavior
of
these,
these
different
nodes
right
right,
yeah,
the
the
network,
subgraph
idea-
is
really
fascinating.
You
know
there
were
some
projects
a
few
years
ago
that
were
kind
of
you
know.
I
think
after
the
graph
came
out
trying
to
apply
the
you
know
the
graphql
concept
to
blockchain.
F
You
know
semantics
of
querying
that
blockchain
data
or
you
know
the
way
that
that
it
was
extracted
from
the
chain,
and
so
you
know,
I'm
excited
not
only
about
the
performance
gains
of
exposing
us
through
the
graph,
but
actually
providing.
Potentially,
you
know
a
maybe
a
more
definitive
or
useful
set
of
semantics
on
top
of
these
networks
that
you
know
might
even
get
more
usage
than
the
sort
of
native
you
know
like
json
rpc
style.
You
know
apis.
I
I
think
there's
tremendous
value
in
that
labor
flat
layer
because
it
empowers
the
rest,
but
also
even
for
those
communities
like
if
such
a
radical
approach
is
brought
to
their
chain.
This
brings
a
lot
of
value
to
their
data
right.
It's
always
been
about
valorising
or
putting
value
to
the
data,
making
it
valuable
and
useful.
So.