►
Description
In this video we walk through how to analyse Ethereum data on Bacalhau.
This example aims to show how easy and efficient it is to analyse large datasets with Bacalhau.
We'll do this by:
- Running a local Analysis of a subset of ethereum data
- Using Bacalhau to analyse this same data subset
- Using Bacalhau for batch processing the entire ethereum dataset captured and plotting the transaction volumes against block datetimes.
Full documentation for this example can be found here:
👉 https://docs.bacalhau.org/examples/data-engineering/blockchain-etl/
00:00 Intro to Bacalhau
02:50 Analysing Ethereum Data with Bacalhau
06:00 Local Analysis of a subset of Ethereum Data
09:50 Using Bacalhau to analyse the same subset of Ethereum Data
16:30 Massive Scale Analysis of the full Ethereum Dataset with Bacalhau
============================================================
About Bacalhau:
Currently Filecoin is known for being a decentralised storage and retrieval network, providing over 17 EiB of distributed data storage capacity to the world through thousands of Storage providers globally.
And we can think of this distributed storage network as being the layer zero of the Filecoin Architecture Network because ..
Storage is really only the start!
Bacalhau is aiming to revolutionise the way we compute and analyse over and with the data on both the Filecoin network and elsewhere by providing a platform for public, transparent, and optionally verifiable computation.
It enables users to run arbitrary Docker containers and WebAssembly (wasm) images as tasks against data stored in the InterPlanetary File System (IPFS)
It operates as a peer-to-peer network of nodes where each node has both a requestor and compute component
What will you build??
Links:
🌐 See more examples on the docs: https://docs.bacalhau.org
🦄 Follow our twitter @BachalauProject
👨👩👦👦 Join the chat: #bacalhau on the Filecoin Slack Channel (visit filecoin.io to join)
A
Hello,
devs
and
fielders
I
am
developer.
Ali,
I'm,
developer
relations,
engineer
here
at
the
filecoin
foundation
and
today
I
want
to
chat
to
you
and
do
a
real
life
example
of
how
you
can
use
an
awesome
project
called
battle
which
is
open
compute
for
data.
As
we
know
in
today's
world,
data
is
everywhere
and
it's
absolutely
essential
that
we
can
analyze
that
and
analyze
the
business
logic.
But
what
is
bakuyao?
A
Well,
it's
a
network
of
open
compute
resources
which
is
available
to
serve
any
data
processing
workload
without
having
to
upload
it
to
a
server
or
anywhere
else.
And
it's
super
simple
to
use.
You
don't
need
an
AI
degree.
I
don't
have
one.
It
requires
minimal
operational
overhead
or
setup,
it's
decentralized,
first
or
Edge
first
principle,
and
it
aims
to
provide
efficient
distributed.
Efficients
I
should
say
distributed
computation.
A
It's
peer-to-peer
compute
over
data,
as
I
mentioned,
open
collaboration
on
data
sets,
is
absolutely
essential
in
today's
Digital
World,
especially
with
the
amount
of
public
information
that
is
available,
we
can
just
make
way
more
sense
out
of
it.
If
we're
all,
sharing
those
kind
of
analysis,
I
like
open
AI,
for
example,
anyway,
the
baculao
platform.
How
does
it
work
this?
It
provides
a
platform
for
public,
transparent
and
optional
optionally,
verifiable
data.
A
It
enables
you
to
run
arbitrary,
Docker
containers
and
webassembly
images
as
tasks
against
data
stored
in
the
interplanetary
file
system
or
ipfs,
so
it
operates
as
a
peer-to-peer
network
of
nodes
where
each
node
has
both
a
requester
and
a
compute
component.
Now
don't
worry
if
that
all
kind
of
went
over
your
head,
it
did
the
first
time,
I
read
it
as
well.
You
can
have
a
look
at
the
docs
and
really
dig
into
exactly
how
this
works.
A
If
you
want
to
know
more
about
the
architecture,
but
in
the
meantime,
I
just
want
to
get
my
hands
dirty
and
show
you
and
get
your
hands
dirty
and
show
you
how
it
works.
So,
let's
get
on
to
the
example,
so
the
example
I
want
to
share
with
you
today
was
written
by
Phil
Winder
who's
with
winder.ai.
By
the
way
you
can
see
this
example
in
in
action
on
the
backlier
docks
as
well.
A
So
you
can
there's
plenty
of
examples
here
on
the
backyard
docks
I'm
going
to
go
through
this
ethereum
blockchain
analysis
example
anyway.
So
it
aims
to
show
you
how
easy
it
is
to
analyze.
Large
data
sets,
in
this
case
the
full
ethereum
blockchain
well
up
to
the
point
of
saving
to
filecoin
anyway,
when
this
example
was
written
with
bacliao.
A
So,
to
make
this
example,
Phil
ran
an
ethereum
node
for
over
a
week
to
allow
it
to
synchronize
the
chain
and,
as
we
all
know,
ethereum
has
been
around
for
a
while
now
and
there
is
a
mammoth
amount
of
data
stored
on
the
ethereum
network.
So
this
information
was
then
extracted
to
file
coin
using
s3.tech
by
way
of
a
tool
called
ethereum.
A
Etl
again,
you
can
dig
into
the
docs
if
you
want
to
know
more
about
that
tool.
So
this
allows
public
access
to
the
data
without
the
need
to
run
your
own
ethereum
node.
But
if
you
are
interested
in
how
to
run
it
in
ethereum
node,
you
can
go
ahead
and
have
a
look
at
the
appendix
here
on
the
docks
that
shows
you
how
to
get
up
and
start
it
and
run
that
ethereum
node
and
then
extract
the
data
yourself
as
well,
so
feel
free
to
dig
into
that.
A
But
generally,
this
amount
of
data
analysis
would
require
a
lot
of
repeating
and
refining.
So
this
is
where
something
like
the
decentralized
back
of
your
platform
comes
in.
It
gives
the
ability
to
process
the
data
in
a
scalable
and
batchable
manner.
Let's
see
what
I
mean
here,
though,
by
showing
me
the
code
now
in
this
example,
I
am
going
to
go
through
a
few
things.
First,
the
first
thing
I
want
to
do
is
show
you
how
to
run
a
small.
A
How
this
example
would
work
locally,
so
I'm
going
to
do
a
quick
local
analysis,
then
I'm
going
to
go
ahead
and
use
back
allowed
to
do
an
analysis
on
that
same
segment
of
data,
so
I'm
not
going
to
use
the
full
ethereum
chain
in
the
local
analysis.
We'd,
be
here
all
day,
I'm
going
to
use
that
small
chunk
of
data
as
well
on
back
of
the
eye
and
then
I'm
going
to
use
backward
down
to
process
the
entirety
of
the
ethereum
chain
that
was
saved
from
that
running,
ethereum
node.
A
A
Here,
curl
and
install
super
easy
check
out
the
docks,
if
you
need
any
more
help
with
how
to
install
that
I'm
just
going
to
verify
here
that
I
have
all
of
those
prerequisites,
so
I
can
do
a
Python
3
dash
dash
version
to
check
that
I
have
python
I
do
and
then
I
can
go
back.
Hold
y'all
version
to
check
that
I
have
bakl
yeah,
so
I
can
see.
I
do
and
my
client
and
server
version
are
also
the
same.
A
So
that's
something
you
do
want
to
keep
an
eye
on
too
all
right,
let's
get
on
with
the
local
analysis.
So
what
I'm
going
to
do
is
download
a
segment
of
this
data
from
one
of
the
ipfs
hashes
that
it's
been
saved
to,
and
there
is
an
appendix
in
the
docs
with
all
of
those
hashes
of
the
ethereum
data,
so
I'm,
just
choosing
one
of
these
and
I'm
going
to
go
through
an
ipfs,
HTTP
gateway
to
fetch
this
data
and
then
unzip
it
because
it's
a
tar
file.
A
So
let's
go
ahead
and
run
this
locally
and
we'll
see
what
happens
so,
as
you
can
see
my
example
folder
up
the
top
here
is
empty
at
the
moment.
I'm
going
to
run
this
grab
the
file
and
unzip
it
so
we'll
see
a
couple
of
things
happen
here:
here's
the
file.tar.gz
and
then
it
is
now
currently
unzipping.
It's
a
big
file.
So
it's
taking
a
bit
of
time
to
do
there.
We
go
so
we've
unzipped
that
file.
A
Let's
take
a
look
at
what
is
in
this
file,
so
we've
got
blocks
contracts
logs,
receipts,
token
transfers,
tokens
transactions,
I
Mammoth
amount
of
data,
even
in
this
little
chunk,
which
is
about
a
hundred
thousand
blocks
of
each
data
so
for
reference,
I
think
in
the
moment
on
the
ethereum
blockchain
we're
up
to
about
16
million
blocks
or
something
like
that.
So
imagine
trying
to
run
this
locally
over
16
million
blocks.
A
So,
let's
look
we
looked
at
that
file
directory.
We
could
see
a
bunch
of
stuff
happening
things
you'd
expect
on
the
ethereum
network.
We
need
to
make
some
sense
of
it
now,
though,
so,
let's
use
pandas,
which
is
a
open
source
data
analysis
tool,
analysis
tool
built
on
top
of
the
Python
programming
language,
we're
going
to
use
it
here,
just
to
clean
up
the
ethereum
data
from
the
CSV
file
found
in
our
output
directory
and
then
we're
going
to
sorry.
Sorry,
we're
going
to
read
the
CSV
file.
A
First,
we're
going
to
create
some
columns
out
of
this
data
and
then
we're
going
to
plot
these
columns
onto
a
graph
using
pandas
and
matplot.
So
let's
go
ahead
and
try
this
I
actually
have
this
file
already
saved.
We
have
a
look
up
the
top
here
and
I've
called
it
local.pi.
So
this
is
the
same
file
here:
I'm
just
going
to
go
ahead
and
run
that
I
will
just
clear
my
console
here
of
that
output.
A
So
you
can
either
do
this.
You
can
do
this
two
ways
you
can
either
do
it
straight
in
your
terminal,
without
the
need
for
a
script
already
saved,
so
you
could
just
run
python3
and
then
copy
paste
that
script
in
here,
but
I
am
going
to
just
run
it
from
my
local
files
since
I
have
it
saved
here.
So
if
I
go
Python,
3
I
just
need
to
go
up
to
my
files
directory
and
run
local
dot,
pi
alrighty,
so
we've
this
has
run
and
you
can
see
the
data
columns.
It
has
output.
A
So
we've
now
got
like
some
nicely
formatted
data
and
on
my
other
screen,
we've
actually
also
got
a
plot
here.
So
look
at
this
plot.
What
is
it?
Well,
it's
a
block
date
time
against
ethereum
transaction
data.
So
that's
that's
what
you're
seeing
on
this
plot
here,
it's
cool,
but
it's
only
inspected
the
daily
trading
volume
of
ethereum
for
that
single
chunk,
that
hundred
thousand
blocks
of
data-
and
we
can
do
better
I
think
with
Baku
yeah.
A
So,
to
do
this,
all
we
need
to
do
is
create
the
code
for
the
analysis
the
same
way
as
we
did
locally
using
pandas
before
then
we're
going
to
package
that
code
in
a
Docker
image
to
run
on
battle
Yeah
and
then
we're
going
to
run
that
docile
Docker
image
using
through
the
bachelor
clean,
so
we're
going
to
run
it
on
the
back
of
our
Network
through
the
back
of
your
clay.
Now,
don't
worry
you
don't
need
to
learn
Docker
all
of
a
sudden,
I
wasn't
lying
when
I
said,
there's
no
prereqs.
A
For
this
video
or
example.
We
already
have
a
Docker
container
image
in
the
docker
registry,
so
you'll
just
be
able
to
follow
along
with
the
code
and
use
that
push
Docker
image
that
it
contains
this
analysis
code
there.
So
here's
the
analysis
code
that
we've
pushed
up
into
that
Docker
image.
Basically,
all
this
script
does
is
passes
the
incoming
data
and
produces
a
CSV
file
with
the
daily
trailing
trading
volume
of
ethereum.
A
Now,
as
you
can
see,
there's
no
for
Loops
here,
so
we're
just
going
to
run
this
on
one
part.
So
one
chunk
of
data
that
a
hundred
thousand
approximately
blocks
of
ethereum
data
side-
note
if
you
did
want
to
play
around
with
creating
your
own
Docker
image
in
pushing
it
up
to
the
registry
here,
is
how
you
would
do
that.
Here's,
the
docker
file
that
we've
used
to
run
our
analysis
script.
A
So,
let's
try
it
out.
Shall
we
I
am
going
to
move
back
over
to
my
terminal
here
now.
I
already
have
a
file
so
called
main.pi.
This
is
the
analysis
code
here.
So
let's
just
check
that
that's
working
by
running
it
I'm
gonna,
run
this
example
first
by
python
three:
where
is
it
located
in
files
main
dot,
pi
and
then
I
am
going
to
get
the
outputs
into
a
new
directory
which
I'm
going
to
call
output?
A
So
that's
the
end
of
the
code
there
so,
as
you
can
see
it
loads
the
files.
This
is
what
the
script's
doing
loading
the
files
and
then
processing.
So
it's
more
than
a
hundred
thousand
blocks.
Actually
it's
115
517,
blocksworth
or
11
days
worth
of
records,
and
this
is
saved
to
a
CSV
file
which
you
can
see
here.
So
that's!
A
Basically,
what
that
script
does
and
next
we're
going
to
check
for
the
we're
going
to
run
this
on
Battle
Yale,
so
let's
go
ahead
and
run
this
example
on
back
all
the
hour.
I'll
just
clear
that,
and
all
we're
going
to
do
is
call
Docker
run
I'm
not
actually
going
to
type
this
out
I'm
just
going
to
copy
faces
from
the
example
bakugou
Docker
run,
it's
ID
only
input
volumes,
here's
the
input
volume.
So
this
is
the
docker
image.
A
Sorry,
the
input
volume
that
we're
going
to
use
this
ipfs
hash
from
before
from
the
local
example
and
here's
the
docker
image
here:
ghcr
dot,
IO
Buckle
out
Etc,
but
let's
hit
enter
on
this
and
run
it.
Let's
run
back,
we
are,
as
you
can
see
here,
we
get
a
job
ID
back
and
we're
still
waiting
for
it
to
process
okay.
So
it's
now
finished,
so
we're
just
going
to
copy
that
battle.
A
Id
and
check
if
this
job
has
actually
completed,
am
I
going
to
do
that
by
doing
a
back
of
your
list,
I'm
just
going
to
filter
by
the
ID
of
the
job
here,
so
I
don't
get
some
random
data
from
earlier
when
I
was
playing
around
with
this
okay,
so
here's
the
job
you
can
see.
It's
come
from
this
Docker
image,
the
one
that
I
mentioned
were
pushed.
It's
got
an
ID
and
its
state
is
completed,
and
it's
also
published
the
results
to
this
ipfs
hash
here.
A
A
And
use
the
job
ID
that
I
want
in
results
there.
So
this
is
all
I
have
to
do
using
the
back
of
your
clay
to
get
the
results
of
this
job
alrighty.
So
we
can
see
that
our
results.
Folder
now
has
some
combined
results
per
Shard
raw
data
and
yeah
lots
of
lots
of
information
in
it.
What
can
we
do
with
all
this
information?
So
let's
do
what
we
did
earlier
and
plot
the
results.
So
I've
actually
also
got
a
script
for
that
here.
A
So
I'm
going
to
run
this
script,
which
is
fairly
simple,
it's
just
going
to
read
that
CSV
file
from
earlier
and
then
it's
going
to
create
a
plot
of
it.
So,
let's
see
if
it's
the
same
as
what
happened
on
our
local
analysis
from
earlier.
So
let's
run
it
and
check
it
out.
So
I
am
just
gonna
run
it
from
my
plot
file
script
here
so
up
in
files,
plot
Main
and
let's
see
if
we
get
the
same
graph
as
that
we
had
earlier,
and
it
looks
pretty
similar
to
the
last
one.
A
We
saw
from
running
it
locally
as
it
should,
because
it's
the
same
data
great
so
we've
achieved
the
same
thing.
What's
the
point
of
that?
Well,
the
point
is
that
really
we
want
to
iterate
over
the
entire
entirety
of
the
ethereum
data
and
not
just
this
tiny
sample
of
it.
So
let's
check
out
how
we
would
do
massive
scale
analysis
using
battle
on
the
entire
data
set
we
have
of
ethereum,
so
the
first
thing
we're
going
to
need
to
do
that
is
these
hashes.
A
So
these
are
the
hashes
from
running
the
ethereum
node.
For
a
little
over
a
week,
there
looks
like
11
days
we
are
going
to
use
our
analysis.
Script
that's
stored
in
Docker
and
iterate
through
all
of
those
in
a
full
for
Loop
and
run
them
on
baccalia.
We're
then
going
to
Output
that
bakalia
I,
sorry,
the
job
ID
that
comes
from
running
each
of
those
Loops
each
of
those
ID
each
of
these
hashes
here
and
output
this
to
a
text
file.
A
A
Run
jobs
in
batches
like
this
over
massive
scales
of
data,
so
really
kind
of
your
imagination
is
the
only
limit
to
what
kind
of
data
you
could
use
for
this,
but
you've
seen
how
easy
it
is
really
just
to
upload
data
and
to
bacalia
and
then
use
it
as
like
kind
of
a
cloud
computing
Center,
but
it's
a
peer-to-peer
distributed
Network,
so
really
cool.
It
looks
like
this
is
finished,
which
is
awesome,
so
we
can
download
and
merge
all
of
these
results
into
a
single
directory.
So
I'm
just
going
to
go
ahead.
A
I'll
clear
this
I'm
just
going
to
go
ahead
and
do
this
so
4
ID
in
the
jobs
iD
Tech
text
that
we
have
just
created.
There
are
now
looking
at
just
remove
any
old
ones
and
then
make
a
new
results
directory
and
save
that.
So
here
we
go
is
the
main
thing
here:
buckle
your
get
output
directory
results,
idid,
and
that
will
be
the
main
thing
that
will
merge
these
download
and
merge
these
results
into
a
single
directory.
A
So,
let's
just
go
ahead
and
run
that
and
we'll
see
that
we
get
all
of
these
folders
so
for
each
hash,
so
we've
got
probably
54
of
them.
We
get
a
results.
Folder
come
through
so
now
that
that's
had
a
chance
to
run.
Let's
plot
the
results
and
see
what
we
get
so
again,
I'm
just
going
to
use
this
script,
which
is
really
basic.
A
So
let's
have
a
look
at
ethereum
transaction
data
over
all
of
those
years
by
running
this
script
here
so
five
and
three
files,
massive
scale
plots
slot
tie
let's
go
ahead
and
see
what
happens
here
and
we
run.
We
get
all
of
the
data
plotted
onto
a
new
graph,
which
is
amazing,
and
it
was
super
simple
now.
A
If
I
hadn't
tried
to
run
this
on
my
local
Mac
I'm,
pretty
sure
it
would
have
crashed.
So
it's
pretty
amazing
that
I
can
just
without
having
any
of
this
data
on
my
computer,
not
taking
up
any
hard
drive
space.
It's
pretty
amazing
that
I
can
just
like
do
all
this
in
the
cloud
I've
got
all
the
data
saved
up
in
my
bakala
Edge
nodes
or
ipfs.
Really
not
the
cloud,
and
then
it
is
running
those
jobs
for
us,
so
there's
so
many
more
options
that
you
can
do
with
this
data.
A
You
could
use
this
same
data
set
and
do
I.
Don't
know
your
imagination
is
really
the
limit
and
I'd
love
to
see
what
you
can
make
of
it.
Please
feel
free
to
share
whatever
you
do
make
with
this
example
or
with
other
examples
in
our
docs,
by
getting
involved
in
the
future
of
data
joining
the
discussion
on
Twitter
on
YouTube,
on
our
slack
so
we're
on
the
filecoin
project,
slacking
the
channel
backward
yeah
and
take
a
look
at
our
GitHub
or
see
more
examples
on
our
backlier
docs
site
as
I
I
love.