►
From YouTube: Blockchain Analytics 101: Building an Indexer from Scratch by Benjamin Memisevic | Devcon Bogotá
Description
Visit the https://archive.devcon.org/ to gain access to the entire library of Devcon talks with the ease of filtering, playlists, personalized suggestions, decentralized access on Swarm, IPFS and more.
https://archive.devcon.org/archive/watch/6/blockchain-analytics-101-building-an-indexer-from-scratch/
A workshop which goes through how to build an EVM observability application and how to interpret the data collected. This workshop aims to empower users to venture into blockchain analytics to identify arbitrage opportunities, high value projects and the centralization of protocol/DAO voting power in a protocol.
Speaker(s): Benjamin Memisevic
Track: Developer Infrastructure
Keywords: Indexing,MEV,Geth
Follow us: https://twitter.com/efdevcon, https://twitter.com/ethereum
Learn more about devcon: https://www.devcon.org/
Learn more about ethereum: https://ethereum.org/
Devcon is the Ethereum conference for developers, researchers, thinkers, and makers.
Devcon 6 was held in Bogotá, Colombia on Oct 11 - 14, 2022.
Devcon is organized and presented by the Ethereum Foundation, with the support of our sponsors. To find out more, please visit https://ethereum.foundation/
A
A
Contract
engineer
being
in
the
space
since
about
2018.
today,
I'm,
basically
going
to
do
a
tutorial
where
we'll
build
through
a
simple
blockchain,
indexer
and
just
sort
of
explain
how
indexes
work
different
pipelines,
you
can
design
and
make,
and
in
general,
just
interacting
with
Geth
on
a
sort
of
obviously
level.
A
Okay,
just
some
background
quickly
about
me
and
how
I
would
love
to
run
the
presentation
in
general
yeah.
So,
as
I
mentioned
before,
I've
been
in
the
space
since
2018
I
used
to
work
at
a
consultant
in
labyrus,
which
is
an
Australian
smart
contract,
consultancy,
and
then
I
worked
as
an
engineer
at
Tracer
Dao
in
mycelium,
which
is
Perpetual
swaps.
A
Protocol
on
Arbitron
yeah
I
was
the
team
lead
at
reputation
dial,
which
was
a
project
at
mycelium,
where
we
focused
on
indexing
and
monitoring
article
systems
on
chain
to
make
sure
Oracle
was
made
well
essentially
held
accountable
and
also
that
they're
functioning
properly
and
accurately
so
sort
of
just
detecting
errors.
Sort
of
in
that
pipeline?
A
A
Okay
ether
scan
either
scan
is
probably
the
most
popular
indexing
tool.
All
around
I've
spent
a
lot
of
time
on
an
etherscan.
It's
basically
a
hobby
of
mine,
I'll
click
through
random
blocks
and
see,
if
there's
something
fun,
if
you've
ever
accidentally
sent
eth
to
a
contract
instead
of
calling
the
function-
and
you
think
nobody
saw
it-
I
saw
it.
A
So
this
is
one
Trends.
Oh
there's
a
screen
here.
This
is
just
one
address
that
I
found
this
morning
in
a
block
and
it's
essentially
an
eoa
that
seems
to
be
Fanning
out
Eve
to
a
bunch
of
addresses
extremely
quickly.
So
it's
sending
out
full
five
transactions
trying
to
spread
out
I
think
about
0.8
eth
yeah,
but
it's
really
worthwhile.
A
Just
as
a
question
does
anybody
have
any
pros
that
they
think
having
transparent
transactions
have
over
ZK
yeah,
no
corruption,
exactly
nice
and
I'm,
not
trying
to
just
Shield
transparency?
Don't
worry,
I,
don't
work
for
the
SEC
or
anything
like
that,
I'm,
not
after
the
taxes,
but
it's
still
highly
important
in
certain
systems.
Oracles,
which
I
work
for
don't
really
function
on
zero
knowledge.
You
have
to
know
that
your
oracles
are
operating
properly
and
you
have
to
know
what
each
Oracle
is
submitting.
A
A
This
is
actually
one
example
here.
Also
from
this
morning,
so
mango
markets
lost
100
million
dollars
last
night
because
of
Oracle
manipulation,
so
making
sure
that
you
can
observe
what's
happening
and
also
the
exposure
that
those
oracles
have
onto
the
each
market
is
very
important
and
something
where
transparency
and
building
an
indexer
is
highly
useful.
A
Yeah
sorry,
I'm
just
run
into
here,
so
why
would
I
actually
want
to
store
it
differently
than
on
chain?
Because
you
know
the
blockchain
is
a
database
in
and
of
itself
all
the
data
is
always
there.
The
problem
becomes,
though,
if
I
want
to
retrieve
specific
data
or
a
range
of
data
or
God
forbid,
create
some
sort
of
average
across
it.
So,
if
I
wanted
to
do
that,
I
might
have
to
make
hundreds
of
equals
to
process
one
transaction
or
one
event
that
I
defined
on
chain.
A
Another
really
good
reason
is
non-effemeral
data
on
the
chain,
so
mempool
data.
Anything
that
you
run
through
as
a
test
is
an
eth
call,
doesn't
stay
on
the
blockchain
forever.
Mempool
data
in
particular
is
highly
useful
for
arbitrage
strategy,
but
there's
actually
no
consistent.
History
of
the
mempool.
A
Obviously
manpul
also
differs
from
node
to
node,
but
having
a
history
from
a
range
of
nodes
is
highly
useful.
If
you
want
to
sort
of
background
strategies
that
you
could
have
done,
if
you're
an
arbitrator,
yeah
also
a
gas
estimation
as
well,
if
you're,
making
eth
calls
that
don't
actually
go
through
chain,
you
want
to
have
a
history
of
how
your
contracts
will
look
when
they're
finally
deployed
creating
a
mini
indexer
that
runs
off
your
localhost.
Smart
contract
known,
is
absolutely
a
valid
option.
A
So
this
is
the
decoded
version.
I'll
show
you
the
encoded
version
a
little
bit
later
on,
but
even
here
not
everything
is
super
obvious
to
the
eye
right.
So
you
have
the
answer
at
the
top,
which
is
obviously
the
new
value
of
that
price
feed
will
have
for
those
curious.
This
is
the
price
of
one
inch,
but
there's
other
points
that
are
really
confusing
right.
A
So
sort
of
on
that
talk
on
that
point,
sorry
I'll
just
quickly
explain
how
OCR
works.
That
was
the
chain
link
log
that
we
just
looked
at
previously,
so
how
chain
link
operates
nowadays.
Is
that
also
I?
Don't
work
for
chain
link
I
just
want
to
say
how
trainlink
operates
now
is
that
most
of
the
aggregation
and
collection
of
data
happens
off
chain,
so
a
price
deviation
will
happen.
A
All
these
nodes
will
look
at
their
different
apis
and
they'll
send
all
their
answers
to
a
single
node,
and
then
this
node
will
submit
all
of
these
answers
and
the
final
aggregated
answer
on
chain
on
behalf
of
everybody
else,
who's
supporting
that
price
for
you
so
sort
of
knowing
that
I'll
also
just
go
through
a
simple
timeline.
So,
as
an
example
say,
I
have
a
chain
link
feed,
that's
monitoring
the
price
of
ethereum
when
the
price
of
ethereum
dips
by
certain
percentage,
all
the
chain
link
nodes
will
know
and
they'll
start
a
new
round.
A
Every
every
single
chain
link
node
will
then
send
an
answer
that
they've
collected
from
their
apis
to
a
leader
that
leader
will
aggregate
that
answer
normally
taking
a
medium
of
some
sort.
He'll
pass
this
along
to
the
elected
transmitter
for
that
round
and
then
that
transmitter
will
send
off
a
message
on
chain
containing
all
the
information
that
everybody
submitted
individually,
as
well
as
the
final
answer,
and
why
I
sort
of
went
through
this
is
by
knowing
this
information
knowing
how
the
system
works.
We
can
now
make
sense
of
the
log
that
we
looked
at
earlier.
A
Okay,
so
believe
it
or
not,
this
observers,
array
right
here,
byte
array-
is
actually
a
list
of
the
addresses
that
each
of
these
observations
come
from.
So
if
you
all
those
numbers,
there's
a
corresponding
Oracle
address
of
the
Oracle
that
submitted
that
answer,
and
that
answer
is
the
number
that
they
thought
is
true.
Another
thing
to
note
is
that
number
is
very
large
yeah.
If
that
number
was
exactly
how
it
is,
I
would
be
retired,
as
holding
like
51
inch
tokens,
but
here
I
am
so.
A
This
is
sort
of
another
step
we
have
to
overcome,
so
we're
sort
of
missing
more
information,
but
we
know
it's
available
somewhere
within
the
contract,
so
this
information
is
available
through
a
view
function
in
the
chain
link,
aggregator
contract
called
transmitters,
which
returns
a
list
which
returns
a
list
of
the
addresses
of
each
article
that
is
supplying
that
contract
with
data
and
there's
another
function
called
decimals,
which
tells
us
by
how
much
I
should
Define
divide
those
numbers.
Those
observations
to
get
the
accurate
like
human,
readable
price
of
one
inch
in
this
case.
A
The
problem,
though,
is
that
these
variables
can
change
block
by
block
these
are
adjustable
variables.
Transmitter
has
changed
all
the
time,
not
only
for
security
reasons,
but
sometimes
certain
articles
are
better
at
performing
on
certain
feeds
than
others,
and
decimals
can
change
when
different
markets
require
more
Precision
out
of
a
price
feed.
A
Okay,
so
this
is
sort
of
how
we
break
it
down
so
I
had
my
observations,
arrayed
there
and
the
way
to
break
it
down,
and
you
can
find
this
out
by
sort
of
reading
the
contract
and
then
doing
a
bit
of
guesswork.
Is
that
if
I
look
at
that
byte
array
and
I
split
it
apart
into
pairs,
each
of
those
pairs
can
be
decoded
from
hex
into
a
number.
This
number
represents
the
index
of
which
that
observation
in
order
corresponds
to
which
item.
A
In
that
transmitters
array,
the
observation
was
from
so
in
this
case.
The
second
observation
here
was
from
the
second
article,
so
we're
indexing
from
Counting
from
zero,
so
zero
X
zero
three
one,
two
e,
a
dot
dot
dot
responded
with
four
three
nine:
three,
nine
nine,
so
already
we're
sort
of
getting
kind
of
complex
and
how
we're
just
deciphering
this
one
single
look,
sir:
we'll
go
through
the
steps
and
code
as
well.
A
How
we
would
do
all
the
different
calls,
but
essentially
at
the
minimum,
what
we
have
to
do
to
fetch
this
data
is
first
we
have
to
encounter
this
chain
link
log.
Then
we
have
to
pass
the
data
back
into
regular
data
types.
Take
out,
transmitters,
buy
it
array,
break
it
up
into
pairs,
convert
that
into
hex
to
get
a
number
call.
The
transmitters
function
with
an
eth
call
and
then
pass
out
which
address
is
with
which
address
now
I'm
doing
this
for
One
log,
not
too
bad
I,
probably
wouldn't
want
to
do
it
by
hand.
A
But
imagine
if
you
wanted
some
sort
of
aggregate
data.
Imagine
if
I
told
you
okay,
tell
me
how
accurate
this
Oracle
was
on
every
single
Wednesday
for
the
past
six
months
now
you're
passing
hundreds
of
logs
you're,
making
thousands
of
calls,
and
if
you
want
this
to
be
done
on
demand,
it's
going
to
be
incredibly
slow,
it
might
take
minutes
on
the
other
side.
On
the
other
hand,
sorry
I
could
make
like
an
SQL
statement
like
select
average,
where
Oracle
equals
this,
where
time
equals
this,
and
this
I'm
converting
hundreds
of
lines
of
code.
A
A
A
A
A
So
you
can
already
select
these
from
the
node
relatively
quickly
and
they
each
have
a
relationship
with
each
other.
If
I
have
the
log
I
can
retrieve
the
contract
address
that
that
log
was
emitted
from
and
I
can
also
retrieve
the
transaction
that
that
log
came
from
and
then
through.
That
I
can
also
retrieve
the
block
number
that
that
transaction
happened
on
and
so
on
and
receipts
as
well,
so
transaction
receipts.
So
you
can
see
how
much
gas
was
paid
for
that
receipt.
A
So
dissecting
a
log
I'm
just
going
to
go
over
the
structure
of
logs
and
for
those
curious,
the
reason
why
I'm
concentrating
on
logs
is
that
there's
quite
a
lot
of
functionality
within
the
get
client
with
blockchain
clients
in
general,
for
creating
these
indexers
based
on
logs,
creating
them
based
on
transactions
is
also
possible,
but
a
lot
more
manual.
You
have
less
sort
of
filtering
power
on
that
first
step,
so
how
log
works
so
logs
are
constructed
out
of
topics
and
data.
Essentially,
so
topics
are
also
data
topics.
A
One
two
three
are
data
types
that
you
can
emit
in
your
event
and
mark
them
as
indexed
and
why
this
exists
is
that,
instead
of
building
an
indexer,
I
could
potentially
just
use
the
blockchain
client
as
well
to
look
for
cases
where
the
data
is
equal
to
a
particular
amount.
So
I
could
tell
Geth.
Tell
me
when
topic:
zero
is
equal
to
X
and
topic.
One
is
equal
to
this
and
retrieve
all
those
cases.
For
me
within
some
block
range
topic,
zero
I'll
go
over
in
a
moment.
The
data.
A
Sorry,
something
to
also
known
on
topics
is
that
you
can
only
index
up
to
three
fields:
data
though
you
can
throw
as
much
stuff
as
you
want
in
there,
and
this
is
like
highly
valuable
and
a
good
way
to
think
of
logs,
and
data
in
general
is
sort
of
like
a
print
statement
for
your
smart
contracts.
So,
if
you've
ever
like
debugging-
and
you
know
we're
all
programmers
here
and
we
hate
using
the
actual
debug
tools,
we
just
print
hello.
A
This
also
removed,
which
is
like
a
bull
that
just
signifies
if
our
log
was
removed
from
the
canonical
chain
due
to
a
reorganization
topic,
zero
topic,
zero
is
highly
useful
topic.
Zero
defines
the
log
I
would
say
so
what
topic
zero
is
is
I,
take
my
event,
definition
so
say:
I
have
an
event
called
transfer
and
it
takes
a
uint
as
an
argument.
A
I
want
to
look
at
every
single
instance
on
the
blockchain,
where
this
transfer
event
happened
and
to
do
that.
I
have
to
calculate
topic
zero
and
to
do
that.
I
take
the
kikak
of
my
event,
name
so
transfer
and
then
each
data
type
that
is
within
that
event
definition.
So
here
it
would
be.
You
know,
un256
or
whatever
some
important
things
to
note.
A
So
something
to
note
as
well
is
that
this
topic,
zero
is
unique
for
every
single
contract,
that
is
to
say
that
you
can
only
have
one
event
with
one
exact
hash
definition
per
contract,
but
this
is
not
stipulated
across
the
whole
chain,
so
I
could
potentially
say
look
for
every
single
log
with
topic.
Zero,
but
I
might
actually
collect
data
for
four
or
five
contracts
that
are
doing
a
completely
separate
thing.
That's
just
something
to
be
wary
of.
A
Storage,
another
really
cool
thing
that
you
can
do
when
you're
building
an
indexer
is
accessing
the
actual
contract
storage,
so
private
variables
now
become
completely
accessible
to
you.
This
is
really
good.
One
of
the
main
use
cases
that
I've
certainly
used
this,
for
is
when
a
contract
is
implementing
EIP
1967,
which
is
the
proxy
pattern.
So
you
have
the
contract,
that's
implementing
all
the
calls
from
another
contract
and
that
contract's
address
that's
actually
got.
The
implementation
is
always
in
a
specific
storage
slot.
A
A
Okay,
infrastructure
design,
so
this
is
a
bit
of
a
web
2
thing,
but
it's
important
to
think
about
how
you
want
your
indexing
application
to
run.
You
can
make
it
as
complex
or
as
simple
as
you
want.
So
if
it's
just
a
regular
ETL,
you
can
have
your
program
connected
to
a
node
and
then
you're
inserting
into
a
database,
but
you
can
do
so
much
more.
A
You
can
create
a
hackathon
submission
that
is
looking
for
events
on
lens,
for
example,
and
then
I'm
sending
out
alerts
through
epns
every
single
time
someone's
profile
gets
locked
I
could
take.
You
know,
I
could
scale
my
application.
I
could
have
several
nodes,
have
a
load
balancer
between
it,
so
I'm
not
overwhelming
any
single
mode.
I
could
be
sending
off
my
messages
after
I've
dealt
with
them
to
an
AI
analysis
and
I
could
be
storing
them
across
several
database
nodes.
A
Stuff
I've
worked
with
personally.
That
I
recommend
is
a
sort
of
microservice
architecture.
You
can
have
Kafka
sort
of
in
the
middle
as
a
message
broker
and
pause
information
between
all
your
services.
All
your
different
storage
points,
maybe
you're
an
arbitrager,
and
you
want
to
do
sort
of
an
analysis
through
that
analysis
inside
in-memory
database,
like
readers
for
quit
act
for
quick
access,
while
simultane
simultaneously
throwing
any
archive
data
into
your
own
sort
of
postgres
database.
A
Database
options,
so
indexing
is
really
powerful
because
you
can
choose
how
quickly
you
want
to
access
what
subset
of
data,
so
these
are
just
like
a
few
I
threw
up
so
time.
Scale.
Gb
is
extremely
interesting,
but
any
time
series
database
sort
of
proves
this
point.
The
blockchain
doesn't
give
you
an
option
directly
to
look
through
transactions
within
time
range.
You
can
specify
a
block
range,
but
that's
not
really
always
ideal
blocks
take
a
different
amount
of
time
to
create
over
time,
and
you
have
to
do
sort
of
extra
leg
work
there.
A
If
you
want
to
convert
from
block
to
time
time
scale
off
is
something
really
awesome,
especially
when
you're
looking
at
a
large
data
set
like
a
blockchain
and
that's
continuous
Aggregates,
and
these
are
materialized
views
that
you
can
create
for
aggregate
data.
So,
for
example,
I
can
calculate
the
average
every
single
week
of
how
much
gas
is
spent
on
ethereum
and
I
can
permanently
store
that
data
in
a
view
and
access
it
instantly,
as
opposed
to
having
to
recalculate
that
data
kdb
is
very
popular
within
sort
of
the
Mev
Community.
A
From
what
I've
seen
it's
a
completely
in-memory
database,
it's
extremely
fast,
so
similar
along
the
lines
of
readers
but
far
more
performance.
It's
used
a
lot
by
Quant
firms,
Arbitrage
and
also
Formula
One,
like
racing,
so
they
use
it
for
in
race,
analysis
and
stuff,
like
that,
postgres,
absolute
classic,
completely
free,
perfect
audio
system.
Why
would
you
use
anything
else?
A
Okay,
now,
I
just
want
to
go
through
a
quick,
Code
walkthrough
of
a
simple
application,
so
we
can
actually
build
these
index
areas
in
about
150
lines
of
go
code.
Easy,
okay!
So
first
I
want
to
explain
my
language
Choice,
because
I've
been
contested
on
it
in
the
past,
I
really
like
using
go
for
the
back
end
and
interacting
directly
with
Geth
and
with
the
Geth
Library
like
first
of
all,
Geth
is
a
go,
is
fast.
Geth
is
probably
the
most
well
maintained
Library.
A
It
obviously
doesn't
have
features
of
certain
other
blockchain
ethereum
clients,
but
it
is
definitely
the
most
well
maintained
and
it
always
fits
through
the
specs.
The
yellow
paper
specifications
go,
has
a
huge
bonus
and
that
parallelization
is
directly
built
into
the
language.
So
we're
going
to
be
doing
a
lot
of
calls
through
web
circuits
or
through
HTTP,
and
we
want
to
paralyze
these,
and
we
don't
have
to
worry
about
race
conditions
when
we're
using
go
yeah.
A
A
So
we
can
Port
our
application,
make
it
as
generic
as
possible,
and
we
can
start
looking
at
events
on
polygon,
polygon,
ethereum
optimism,
arbitrum,
whatever
you
want
without
having
to
do
any
extra
legwork.
It's
like
deploying
a
smart
contract
to
multiple
chains,
and
the
reason
this
is
is
all
these
RPC
calls
that
we're
making
are
specified
within
the
yellow
paper.
So
any
evm
client
that
is
being
created
is
going
to
have
to
fit
through
these
specs.
A
Okay,
creating
a
client.
This
is
like
pretty
basic
code,
so
I'm
just
creating
a
HTTP
client
I'm
feeding
in
my
Alchemy
sort
of
key.
That's
what
the
RPC
thing
is
there?
It's
just
a
string
I'm
doing
a
bit
of
error
handling
here,
making
sure
that
I
am
in
fact
connected
and,
in
fact
connecting-
and
that's
really
it
for
this-
there's
really
not
too
much
to
worry
about
here.
Yes,.
A
Okay,
that's
a
good
question,
I'll,
just
repeat
it.
So
the
question
was:
is:
should
you
run
your
own
node
infrastructure
or
use
an
outsourced
node
like
Alchemy?
So
there's
several
advantages
to
running
your
own
and
a
few
disadvantages
running
your
own
node
Beyond.
You
just
have
direct
access
and
you
can
manage
all
your
load
yourself.
A
A
Then
I
would
run
my
own
node
I
see
some
like
node
operators
here
and
I
feel
that
but
full
hackathon
projects
absolutely
use
alchemy,
fantastic
solution,
not
just
Alchemy
and
any
inferior,
whatever
I'm,
not
sponsored,
guys
I'm
sorry
I
keep
using
the
same
names
websockets
versus
HTTP,
so
most
providers
will
allow
you
the
option
of
accessing
their
service
to
a
websocket
or
HTTP
I'm.
Just
going
to
go
over
this
quickly
because
it's
more
of
a
web
2
thing
HTTP
is
normally
how
your
wallet
connects
to
the
chain.
A
A
It
takes
an
array
of
addresses
from
which
I
want
to
collect
logs
from,
and
it
takes
a
list
of
topics
so
before
we
talked
about
topics
and
how
you
can
index
up
to
three
and
you
have
topic
zero,
so
you
can
specify
here,
I
want
to
collect
data
from
which
topic
zero
and
which
other
topics
I
want
to
be
equal
to
some
hex
value
and
I
sort
of
just
declare
this
object,
and
that's
all
you
really
need
from
here.
We
haven't
actually
made
the
call.
Yet
this
is
just
constructing
the
query.
A
This
also
takes
a
block
range
which
I
didn't
include
for
some
reason:
next,
okay,
so
now
we're
actually
making
the
query-
and
this
is
where
block
range
becomes
very
important.
So
there's
two
ways
in
which
I
can
request
data
from
a
blockchain
node
I
can
do
subscribe,
filter
logs,
which
is
I'm
establishing
a
connection
to
the
node
and
I'm
telling
the
node.
Okay.
Here
are
my
parameters
whenever
this
happens
on
chain.
A
Send
me
a
message,
so
this
is
really
useful,
because
now
I
can
sort
of
distribute
my
load
from
the
node,
because
I
get
these
messages
one
at
a
time.
I
can
process
them
and
as
long
as
my
application
is
at
least
decently
efficient
I'll
process
everything
without
any
lag.
Now
this
always
isn't
an
option,
because
one
thing
about
throw
subscribe,
filter
logs
is
that
I
can't
request
historical
information.
I
will
only
be
receiving
data
from
the
point.
I've
subscribed
so
from
the
next
block.
That's
coming
now.
A
A
Okay
channels,
this
is
a
go
thing,
but
it's
also
extremely
important
to
know,
and
it's
one
of
the
reasons
that
I
suggest
making
these
index
applications
and
go
so
what
a
channel
is
and
goes
is
essentially
like
a
pipe.
So
if
I'm,
sending
data
from
a
process
to
B
process,
I
use
a
pipe,
but
there's
certain
issues
associated
with
doing
this
right.
So
you
have
to
deal
with
race
conditions
for
one
which
is
a
huge
headache.
A
Also
piping
can
be
quite
messy,
I,
don't
know
how
you're
doing
it,
but
you
can
do
it
like
through
a
bash
script.
It
can
get
pretty
weird.
What
go
offers
is
channels
and
channels
are
naturally
blocking
so
I
don't
have
to
worry
about
message,
one
getting
there
before
message,
three
and
screwing
up
all
my
analysis.
A
How
do
I
actually
process
data
so
when
I
request
data
from
the
blockchain
and
I
get
logs
back,
I
actually
get
back
a
pretty
messy
data
structure,
not
messy
but
not
human
readable.
So
what
I'm
going
to
get
back
is
logs
data
structure
which
we
went
over
earlier
I'm,
going
to
have
these
topics
and
I'm
going
to
have
the
data
field
inside
the
log
and,
if
I'm
pulling
the
transaction.
Similarly,
I
will
have
the
data
field
within
the
transaction
field,
which
is
identical
to
the
sort
of
log
data.
A
Now
passing
this
across
is
pretty
complex,
so
there's
natural
pot,
padding
that
happens
from
rlp
encoded
values
and
I
also
could
convert
from
hex
back
into
sort
of
regular
values,
at
least
for
go
to
interpret,
but
also
so
humans
can
interpret
it.
I,
don't
really
know
what's
happening
when
somebody
says.
Oh,
the
value
came
back
as
like
0x,
60,
zeros
and
then
a
three.
A
So
what
we
can
do
instead
is
we
can
generate
an
ABI,
and
this
is
exactly
how
etherscan
does
it
and
for
those
not
familiar
and
ABI,
is
a
specification
of
all
the
functions
and
all
the
events
that
happen
on
a
particular
contract
and
I
can
generate
this
by
going
to
a
contract
and
using
cell
C.
So
that's
the
command
at
the
top,
so
I'll
see
ABI
and
the
name
of
the
file
or
the
path.
A
But
the
API
is
highly
useful,
but
something
to
note
is
that
it
does
not
directly
appear
on
chain
on
chain
is
only
sort
of
the
compiled
like
bytecode,
so
the
ABI
you
have
to
get
by
having
access
to
the
source
code,
or
sometimes
people
upload
it
to
ether's
Camp
and
what
the
API
allows
us
to
do
is
the
ABI
knows
as
I
described
before
with
the
CAC.
It
knows
all
the
input,
types
for
all
the
functions
and
all
the
events
that
are
defined
within
that
contract.
A
It
should
come
back
up
in
a
moment:
okay,
oh
it
is
back
fantastic,
so
I
can
use
this
API
and
use
the
evm
to
decompile.
All
of
this
hex
data
that
I'm
going
to
get
blurted
out
back
into
regular
data
types,
so
creating
an
API
object
similar
to
the
JavaScript
concept,
but
I'm
going
to
call
this
API
function,
which
is
also
a
API
Library.
A
Go
will
tell
me
that
I
made
a
mistake
if
I
try
to
assert
wrongly
that
it
enters
a
string.
Go
will
tell
me
I'll.
Actually
you
want
to
type
assert
to
a
string
I,
don't
know
why
I
can't
just
do
it
automatically,
but
life
can't
be
that
easy,
okay
working
with
a
database,
okay.
So
now,
essentially
what
we've
constructed
is.
We
have
a
system,
that's
requesting
data
from
the
node
either
live
or
through
through
a
historical
query.
A
I've
collected
that
data
I
now
have
a
method
to
unpack
that
data
into
regular
go
data,
types
and
I
can
convert
between
them
as
I
wish.
Now
I
want
to
insert
that
data
somewhere.
So
this
is
a
pretty
standard
way
of
just
interacting
with
a
database
and
go
completely
fault
proof
and
it's
using
a
library
called
Gom.
So
go
RM.
It's
a
used
to
be
sponsored
by
chain
link
actually
as
well
pretty
fantastic
Library
highly
recommended.
A
So
it's
very
easy
for
what
I
have
to
do:
I
just
Define,
a
struct
which
has
what
my
tape?
What
I
want
my
table
to
look
like
I,
throw
on
some
strings
where
I
want
my
primary
keys
or
foreign
keys
or
indexes
and
I'll
just
leave
that
in
the
module
by
itself
and
then
what
I'll
do
is
I'll
initiate
a
connection
to
my
database
and
it's
also
just
this
one
line
code.
A
A
Inserting
so
we're
pretty
much
like
done
here
guys
all
I
have
to
do
now
is
use
the
structs
that
I
previously
defined
pass
in
my
decoded
values
and
then
I
just
have
this
one
line
insert
or
update
on
conflict.
So
what
this
is
essentially
doing
is
I'm,
sending
a
message
and
it'll
be
acid,
depending
on
your
database
of
choice,
I
guess
out
to
the
database
and
I'm
telling
it
okay.
A
If
there's
no
entry
inserted,
if
there
is
updated
and
updating
actually
absolutely
can
happen,
although
the
blockchain
is
immutable,
remember
that
the
node
is
getting
new
data,
all
the
time,
reorgs
do
happen,
so
something
you
want
to
watch
out
for
is
your
block
hash,
changing
and
also
your
block
times
down
profile
idb?
Oh
sorry,
we
just
have
like
type
assertion
here,
so
you
can
sort
of
see
how
these
are
getting
paused
back
into
the
struct.
A
So
I
have
my
profile
ID,
and
this
is
an
example
that
I
did
based
off
lens
so
profile.
Ig
and
lens
is
an
integer
which
represents
your
user
on
lens
and
then
also
nft,
which
is
an
nft
that's
generated
on
lens.
Whenever
you
follow
somebody
and
I
just
passed
these
into
that
struct
I'm
calling
my
sorry
database
dot,
Clauses,
update
function
and
I'm
passing
in
a
pointer
to
the
struct
okay.
A
A
Oh
okay,
so
EIP
1967
basically
says
it's
a
proxy
pattern,
so
say
I
want
to
make
changes
to
a
Smart
contract.
You
obviously
can't
really
do
that.
So
what
I
can
do
is
I
can
define
a
vip1967
contract
and
tell
it
do
all
the
functions
of
another
smart
contract
and
I
can
change
which
smart
contract
is,
which
my
contract
logic
is
actually
being
changed
by
just
changing
that
variable.
A
But
the
problem
is:
is
that
there's
no,
like
view
function
for
the
contract
address
that's
on
that
1967
contract,
but
the
storage
slot,
where
that
address
is
stored,
is
always
the
same
across
any
EIP
1967
contract.
So
I
can
retrieve
this
address
and
then
I
can
query
that
contract
for
any
information
that
I
want.
Does
that
help?
Okay?
A
A
Yeah
I'll,
just
repeat
the
question,
but
the
question
is:
how
many
block
confirmations?
Do
you
wait
before
inserting
the
data
into
the
database
or
how
do
you
deal
with
updates?
So
the
great
thing
about
the
system
is
that
you
don't
have
to
wait
at
all
every
time.
There's
a
new
reorganization,
the
logs
will
be
like
re
put
through
the
node
and
then
the
indexer
will
automatically
update
that
column
for
you.
So
the
data
you
have
in
your
database
will
always
be
the
most
accurate
that
it
can
be
and
most
up-to-date.
A
Why
is
it?
Volunteering
I
mean
it
costs
gas
to
emit
events?
Why
isn't
it
standardized?
I
mean
it
is
standardized
in
development.
It's
not
like.
You
have
to
deploy
a
contract
with
events,
and
if
there
is
contracts
deployed
with
that
events,
you
can
also
create
an
indexer
based
on
that
I.
Think
that's
what
you're
getting
at
or
yeah.
A
A
A
Oh,
can
someone
help
me
their
logos,
like
June
analytics?
Thank
you
like
June
analytics,
where
I
can
do
an
SQL
query
on
absolutely
anything,
but
in
terms
of
why
there's
so
many
events
happening.
A
lot
of
these
events
are
just
shut
out
for
debugging
purposes
and
they're
left
in
the
contract,
and
you
sort
of
have
to
decipher
as
you
go
any
other
questions.
A
A
Contract
state,
yeah
yeah
so
as
before,
as
I
went
like
through
storage.
So
if
you
wanna,
if
you
want
to
get
contracts,
staged,
there's
three
main
things
that
you
can
do
so
there's
access
through
variables,
so
every
single
public
variable
on
a
contract
automatically
has
a
getter
assigned
to
it.
So
that's
one
way
where
you
can
access
State.
The
second
one
is
through
storage,
as
I
mentioned
before,
so
it's
kind
of
tricky
to
find
out
exactly
where
a
variable
is
in
storage.
A
A
A
A
So
yeah
that
is
actually
important
to
note.
So
if
somebody
like
destructs
a
contract-
yes,
okay,
if
somebody
self-destructure
contract,
I'll,
just
repeat
the
question:
if
somebody
self-destructs
a
contract,
I
believe
it's
possible
for
another
contract
to
appear
on
that
identical
address
and
that
can
have
different
cards,
so
it
can
potentially
break
your
indexer.
A
What
you
can
do
there
to
solve
that
system
is
you
can
have
a
micro
service
looking
for
Destruction
transactions
and
matching
them
up
doing
a
query
in
your
database
to
see
if
that's
affecting
one
of
your
indexers
and
then
making
changes
to
that
indexer
live
so
I'll
normally
do
that
through
a
microservice
architecture,
so
I'd
have
one
looking
for
destructs.
If
it
destructs
happen,
send
a
message
adjust
or
cancel
my
indexer
for
a
period.
The
great
thing
about
the
blockchain
is
the
dot
is
always
there.
A
A
Oh
okay,
I
mean
okay,
so
number
one
I
would
say
in
general
is
latency.
You
don't
have
to
wait
on
any
graph
note
operators
to
fill
up
the
graph
for
you,
second
of
all,
I
can
create
a
graph.
That
is
what
a
graph
sorry
a
data
set.
That's
infinitely
more
complex
than
what
I
could
do.
I
mean
I
can
theoretically
do
anything
on
the
graph,
but
here
I
have
very
close
knit
control
of
exactly
what
my
code
is
doing
and
what
I'm
going?
A
A
Fourth,
you
don't
have
to
pay
it's
free,
yeah
I!
Guess
that's
like
some
of
them,
but
really
my
main
pitch,
for
it
is
that
this
isn't
that
much
code,
it's
not
expensive
to
run.
It's
mainly
really
just
doing
HTTP
calls.
You
can
run
this
for
free
on
any
cloud
provider
on
like
the
E2
micro
or
whatever
the
AWS
equivalent
is
you
can
run
it
on
the
free
tier
Super.
A
A
So
yes,
there
is
a
way
to
handle
this
and
you
can
subscribe
new.
So
there's
a
function
called
subscribe,
new
head
inside
the
get
client
and
every
single
time.
There's
a
new
block
header
or
an
uncle
you'll,
get
a
message
there.
So
you
can
make
a
service
where
you're
looking
for
uncles
it'll
give
you
the
information
you
need,
and
then
you
can
look
at
what
that
Uncle
did
and
it
should
produce
new
events,
and
it
should
tell
you
like
the
removed
tag
for
the
event
for
the
lot.
A
So
you
can
retrieve
so
when
your
block
hash
gets
Uncle
like,
let's
just
assume
that
you
detected
properly
through
subscribe
new
head,
you
can
re-request
that
block
through
the
block
hash
and
then
it'll
give
you
inside
the
log
field,
a
removed
field,
which
is
a
ball
to
tell
you.
If
that
log
has
been
deleted
or
not,
and
if
it
has
been
deleted,
then
you
can
mock
that
in
your
database.
You
can
delete
the
row
or
you
could.
It
depends
on
your
private
key
sorry,
not
private,
key,
your
primary
key
setup.
A
A
Sort
of
Hardware
that
you
need,
it's
so
I'll
say
like
for
a
couple
different
components
from
a
database
point
of
view.
Absolutely
it
depends
what
sort
of
service
you're
running,
how
quickly
you
want
to
retrieve
that
data,
but
in
terms
of
storage,
I,
don't
know,
I
would
say
maybe
six
seven
hundred
gigs
for
the
database.
The
more
difficult
component
is,
if
you
want
to
do
that
historically,
for
every
single
block
number
you're
going
to
run
into
hiccups
with
having
nodes
requesting
that
data
from
nodes.
A
So
that's
going
to
be
very
expensive
if
you're
doing
that
through
Alchemy
or
inferior
or
one
of
those
node
subscribers,
where
you're
paying
per
call
and
if
you're
running
your
own
node
I
would
say
one
isn't
enough.
If
you
wanted
to
finish
like
this
Century,
so
what
I
would
do
is
run.
You
know
whatever
20
nodes
put
hook
them
all
up
through
historical
data,
so
you
can
use
snapshots
now.
A
So
it's
nice
and
easy
to
set
that
up
and
then
have
a
load,
balancer
and
distribute
out
your
calls
into
the
different
notes,
and
that's
probably
the
quickest
way
to
do
it
in
terms
of
the
actual
indexing.
Pretty
lightweight
I
wouldn't
really
worry
about
it.
The
way
I
normally
run
it
is
have
a
kubernetes
cluster
and
then
just
Auto
scale
as
I
need
it.
A
A
A
My
database
for
those
who
are
not
familiar
partitioning
is
sort
of
imagine
you
have
a
CSV
and
you
have
a
huge
Excel
file
now,
instead
of
having
this
huge
Excel
file
that
I
have
to
search
through
I
break
out
the
Excel
file
into
like
five
different
files
and
I
Define
them
by
a
range
and
now
I
can
just
search
through
that
range
for
a
particular
transaction.
So
there's
lots
of
time,
partition,
databases,
but
I
would
time
partition
your
data
if
that's
relevant
and
then
create
indexes
on
everything.
A
A
A
Oh
the
other
one
I
didn't
mention
is
graph
databases.
I
haven't
seen
that
much
use
for
them.
I
haven't
used
them
myself,
sorry,
but
that's
also
an
extremely
powerful
way
to
store
your
blockchain
data.
So
I
can
see
the
relationships
between
different
addresses
or
I
can
see.
The
relationship
between,
for
example,
like
oracles
and
the
Ave
Market.
A
A
Sing
for
indexing,
so
the
different
channels-
I
mean
it's
paralyzation,
but
essentially
I
don't
have
to
wait
for
one
request
to
get
through
before
I
do
the
other
one.
So
it's
very,
very
useful,
so
I'd
say
if
you're
doing
the
Subscribe
filter
logs
method
when
you're
getting
one
event
at
a
time,
you're-
probably
absolutely
fine
without
using
a
routine,
but
if
I'm
doing
the
filter,
log
method
and
I'm
requesting
a
huge
amount
of
logs
at
the
same
time
and
I
have
additional
calls
to
do
on
each
of
those.
A
Yeah
you
can
insert
more
than
one
row
at
the
same
time.
I
could
run
multiple
index
programs,
for
example,
from
like
block
range,
zero
to
100
100
to
200
300
to
400,
and
they
can
all
insert
separately
so
I'm
finishing
that
index
far
quicker
than
if
I
had
to
go
from
zero
to
four
hundred
on
one
routine
or
one
program.
A
A
Yeah
I
mean
look,
node
providers
are
fantastic
and
highly
professional,
so
you're
going
to
get
amazing,
Service
and
you're
not
going
to
really
experience
much
delays
there,
but
you
can
set
up
your
node
in
gcp
and
you
should
not
really
have
too
much
more
of
a
delay.
The
bigger
risk
of
running
your
own
node
is,
if
you
make
a
mistake
or
you're
missing
a
Geth
update.
A
lot
of
the
manual
stuff
you
have
to
do
around
managing
your
node
is
automatically
done
by
node
providers.
A
Okay,
one
thing
that
you
can
do
is
that
you
can
run
multiple
nodes
in
different
regions,
because
the
mempool
is
not
necessarily
synced
between
all
nodes.
At
the
same
time,
so
a
popular
setup
is
to
run
three
or
four
nodes
in
different
regions
and
they'll
have
different
peers,
and
then
you
can
congregate
that
data
yourself,
like
through
another
process
and
that
way,
you'll
get
a
much
more
complete
view
of
the
mempool
at
like
block
execution
time.
Yes,.