►
Description
@andrewxhill will discuss how to use Textile Powergate, a library that manages storage and retrieval across the Filecoin and IPFS networks.
Read more about Andrew and Textile on the Filecoin blog. https://filecoin.io/blog/community-andrew-hill-textile/
Stay connected on Filecoin Slack:
https://app.slack.com/client/TEHTVS1L6
And dive into the Textile docs:
https://docs.textile.io/
A
Okay,
what
do
you
say
we
get
started,
so
I
can't
I
I
so
I'm
gonna
be
just
sharing
my
screen
for
a
little
bit
here,
so
I
won't
be
able
to
oop
my
zoom
says:
I'm
okay,
there
we
go
so
I
won't
be
able
to
see
any
questions
for
a
moment.
So
let
me
just
explain
a
little
bit
about
kind
of
today's
agenda
for
the
master
class
and
what
to
expect
so.
A
You
can
snooze
off
through
during
the
parts
that
you're
not
interested
in,
but
basically
I'll
spend
the
first
period,
I'm
just
going
to
give
a
handful
of
slides
to
help
orient
everybody
about
what
powergate
is
and
how
to
think
about
using
it.
Following
that,
I'm
gonna
do
a
kind
of
hands-on
demonstration
of
using
powergate
from
the
very
beginning
of
installing
the
tooling
to
connecting
to
the
network
to
actually
storing
data
on
filecoin
and
I'm
going
to
use
a
really
useful
tool
there,
so
that
we
can
see
things
actually
happen.
A
A
A
My
name
is
andrew
hill.
I'm
one
of
the
co-founders
at
textile.
I
actually
come
from
a
pretty
varied
background.
I
did
my
phd
in
biology.
I
studied
biodiversity
informatics
and
at
the
time
we
were
very
interested
in.
How
do
you
take
these
large
data,
sets
that
were
being
digitized
in
natural
history,
museums
and
take
the
data
that
was
coming
from
real
biological
specimens?
A
I
also
worked
on
the
study
of
viral
evolution.
Actually,
another
big
data
topic,
and
one
thing
that
I
was
always
very
interested
in-
is
how
you
communicate
and
give
people
access
to
data
outside
of
the
scientific
community
that
led
me
into
mapping.
I
spent
six
years
at
a
startup
building
a
mapping,
an
open
source
mapping
stack,
you
can
still
find
the
company,
it's
called
carto
it's
at
carto.com.
A
A
You
can
actually
use
any
kind
of
layer
in
this
stack
below,
and
you
can
also
swap
these
layers
with
your
own
implementations
as
long
as
they
fulfill
the
needs
of
the
powergate
technology,
and
so
the
stack
at
a
high
level.
The
stack
is,
is
kind
of
made
up
of
these
five
parts.
There's
the
file
coin
implementation,
the
lotus
node,
and
so
that's
that's
at
the
bottom.
A
A
A
And
so
then,
finally,
the
clients.
So
we
maintain
a
command
line,
client
and
we'll
use
that
today
and
I'll
show
you
how
that
runs,
and
you
can
use
that
interactive
or
we've
tried
to
expose
a
lot
of
it.
So
you
can
actually
build
it
into
scripts
and
pipelines,
and
we
also
maintain
a
javascript
client
and
the
javascript
client
is
we'll
we'll
show
a
few
examples
of
using
it.
A
The
other
one
we'll
use
today
is
a
project
that
came
out
of
hack
fs,
which
is
a
python
a
set
of
python
bindings
called
pygate
so
that
you
can
use
the
powergate
apis
from
python
building.
This
workshop
was
my
first
deep
dive
into
that
client,
which
is
really
fun
and
easy
to
use
it's
a
little
bit
behind
our
current
apis.
So
there's
some
limits
on
what
it
can
do
today,
but
it
if
somebody's
eager
to
use
that
it's
actually
just
a
couple
pull
requests
from
being
up
to
date.
A
So
but
I'll
show
you
that
so
powergate
has
strengths
and
weaknesses,
and
I
want
to
point
those
out
to
everybody
who's
planning
to
use
powergate
so
that
you
kind
of
know
you
kind
of
know
where
it
should
fit
in
your
development
stack
some
strengths
just
to
go
over.
These
is
providing
these
apis
and
these
libraries
to
integrate
file
file
coin
storage
into
your
existing
systems.
So
when
we
were
planning
powergate,
it
was
real.
A
It
also
provides
a
number
of
analytical
tools
about
the
network,
and
so
every
30
minutes
powergate
will
sort
of
query
the
network
and
get
information
about
how
miners
are
behaving.
What
kind
of
what
kind
of
prices
that
they're,
quoting
what
kind
of
deal
flow
is
happening
and
it
uses
that
it
exposes
that
information
as
indices
that
you
could
use
in
your
app.
If
you
use
nothing
else
from
powergate,
you
could
use
those
apis
to
inform
your
application
or
your
system,
but
it
also
uses
those
that
information
itself.
A
A
A
You
can
stand
up
the
powergate
stack
in
just
a
few
minutes
and
we'll
do
that
today
and
already
be
developing
against
the
apis,
so
that
you
can
figure
out
how
to
use
it.
You
can
test
it
in
your
idea.
You
can
test
it
in
your
pipeline
or
in
your
system
and
and
debug,
and
improve
and
then
finally
deploy
on
a
real
network.
A
Where
there's
real
file
coin
being
exchanged-
or
in
this
case
test
net
file
coin
being
exchanged,
but
but
when
mainnet
launches,
that
will
be
very
real,
and
so
you
want
to
already
have
done.
The
development
already
done.
The
testing
before
you
join
that
network
and
and
powerget
makes
that
very
easy.
A
A
They
come
with
their
own
idea
about
how
to
manage,
who
is
using
their
apis,
and
so
powergate
didn't
want
to
put
a
strong,
a
strong
opinion
on
how
that
was
done
or
force
you
to
adopt
a
new
user
model
or
a
new
set
of
authentication.
So
it
has
a
very
lightweight
authentication
model,
meaning
it's
also.
Let
me
jump
to
0.3
here.
It's
also
not
really
meant
to
provide
a
web
ready,
api
or
environment
that
would
be
publicly
available.
A
A
A
They
can
fire
requests
at
the
power
gate
supplying
their
token
and
therefore
using
their
wallet
address
to
make
deals
or
request
requesting
lists
of
information
about
what
has
already
been
done
from
that
wallet
address
in
the
powergate
number
two
is
that
the
powergate
doesn't
really
care
about
your
data
security
model,
so
it
doesn't
care
if
you're,
if
you're
it
doesn't
care
what
the
data
looks
like
when
you're
creating
deals.
It
just
it
just
cares
that
there
is
data
and
that
data
has
a
hash
cid
on
ipfs
other
than
that.
A
It's
not
going
to
reject
data.
It's
not
going
to
protect
your
data,
so
you
need
to
choose
what
level
of
encryption
or
security.
You
also
need
to
determine
what
format
and
architecture
of
data
you
want
to
be
on
the
network
and
do
all
of
that
before
you
push
that
data
to
to
file
coin
and
use
powergate
to
get
it
there
and
actually,
in
the
workshop
today.
A
I
keep
mentioning
that.
It
moves
data
between
file
coin
and
and
ipfs.
We
use
turn.
We
use
these
two
terms
quite
a
bit
and
you'll
see
them
in
the
configuration
for
your
storage.
Deals
that
there's
hot
storage
and
there's
cold
storage,
managed
by
powergate
and
hot
storage,
means
that
it's
live
on
the
ipfs
network
and
if
you
push
a
new
storage
deal
or
a
new
storage
request
to
the
power
gate,
and
you
request
that
it
remains
in
hot
storage.
A
Similarly,
there's
configurations
for
cold
storage
and
in
this,
in
this
case,
cold
storage
means
on
the
file
coin
network,
and
so
all
the
configuration
around
cold
storage
is
about
the
redundancy
for
how
many
miners
you
want
to
store
a
store
with
how
long
you
want
the
deals
to
be
made.
What
regions?
A
So
I
mentioned
this
caveat
before
and
it's
just
that
the
cold
storage
configurations
are
pretty
well
set
right
now
in
the
power
gate
repo
to
work
for
the
slingshot
competition.
So
if
you're
taking
part
in
the
slingshot
competition,
look
at
those
configs
and
try
to
understand
them,
but
you
may
not
want
to.
A
You-
may
not
want
to
change
them
right
now,
because
it
means
that
you
might
be.
You
might
be
trying
to
store
with
miners
that
aren't
participating
in
the
competition
yet
so
they
might
not
store
your
deal
you
or
or
you
might.
You
might
run
into
connectivity
issues,
whereas
right
now
it
should.
It
should
work
by
default
for
the
competition.
A
It's
meant
to
live
on
that
side
of
your
application
and
any
perhaps
public
users
that
you
might
have,
and
so
they
should
be
using
your
application
to
interact
with
the
ffs
instances
in
your
powergate,
but
they
can
still
use
the
ipfs
network
to
be
requesting
data
that
is
maintained,
live
on
the
network.
So
if
you
have
data
that
is
on
the
in
the
hot
layer
and
and
pinned
to
the
ipfs
network,
they
can
still
use
ipfs
to
retrieve
it
without
an
api
that
goes
directly
to
powergate.
A
However,
or
no
in
addition,
actually,
you
can.
You
can
have
data,
that's
not
pinned
to
the
hot
layer
but
maintained
elsewhere,
and
it
would
still
be
available
over
ipfs
and
powergate
would
also
find
it
there
to
say
that,
essentially
that
that
data
could
be
fulfilled
if
requested,
without
having
to
go
back
to
retrieval
deals.
A
Here,
it's
just
a
little
representation
of
the
ffs
instances.
I
mentioned
where
every
ffs
instance
has
its
own
token
and
when
you're
talking
to
the
api,
you
would
supply
that
token
so
that
the
correct
wallet
addresses
are
available
with
the
correct
file.
Coin
balances
and
deals
that
are
created
or
retrievals
that
are
started
would
come
from,
would
be
paid
from
those
balances.
A
So
the
there's
pretty
broad
language
and
platform
support,
because
it
is
all
api
driven.
There's
there
are
builds
of
the
cli
and
the
powergate
daemon
for
all
major
platforms.
You
can
go
to
our
github
releases
page
and
download
those
and
run
them
on
your
computer,
whether
linux
mac
windows
and
then
the
api
clients.
We
have
support
in
javascript
and
python.
There
is
a
golang
client
as
well.
A
You
know.
So
if
you
want
to,
if
you
want,
if
you
want
to
store
new
data
with
or
track
the
storage
of,
new
data,
on
your
powergate
instance,
the
way
that
you
do,
that
is
you
actually
just
push
a
new
storage
config
and
that
storage
config
defines
what
powergate
should
be
doing
to
data
that
lives
at
a
cid
cid
being
a
hash
of
data
on
ipfs.
So
it's
the
if
you
if
you
haven't
used
powergate,
if
you
haven't
used
ipfs,
all
data
has
a
fingerprint
that
is
referred
to.
A
It's
referred
to
as
its
cid
the
content
identifier,
and
so
any
new
data
that
you
want
stored
on
the
network.
You
will
get
its
cid,
you
will
make
sure
it's
available
on
ipfs
and
then
you
will
define
a
config
for
how
you
want
that
data
managed
in
powergate.
There's
a
default
config-
and
I
mentioned
before
that.
The
default
should
be
pretty
good
for
everybody
participating
in
the
competition.
A
A
A
If
you
want
to
create
deals
and
then
with
minors,
it's
going
to
take
time
and
then
after
a
deal
exists,
it's
going
to
exist
for
a
long
period
of
time,
and
so
I
believe
the
default
in
powergate
is
currently
a
year
for
storage
deals,
and
so
it
will
monitor
those
deals
for
here
for
you,
and
so
that's
that's
pretty,
oh
and
and
we'll
do
it
best
to
keep
your
storage
up
up
to
date
for
that
year.
So
if
you,
if
you're,
storing
in
replicas
so
storage
updates.
A
You
can
update
your
configuration
and
tell
powergate
to
to
change
the
way
that
it's
storing
and
tracking
your
data,
and
it
will
do
that
it
will
move
data
from
one
minor
to
to
a
sec
deal
for
that
data
and
then
finally,
it's
slingshot
ready,
so
everything
that's
currently
published
in
the
latest
release
and
available
on
master
is,
is
really
tuned
for
you
to
take
part
in
this
competition.
So
it's
a
great
starting
point
for
you.
A
So
last
thing
I'll
mention
before
we
jump
into
the
kind
of
hands-on
tour
here,
is
that
if
you're
not
already
join
the
slack
channel
in
the
filecoin
slack
called
powergate
users,
there's
a
lot
of
really
helpful
information
flowing
there.
People
hitting
different
areas
where
they're,
confused
or
new
questions
or
error
codes
that
people
hit
that
they
want
to
know
more
about,
and
my
colleague,
ignacio
actually
just
grabbed,
some
of
the
more
more
common
ones
and
just
published
a
new
faq
document
where
the
link
is
in
that
channel
as
well.
A
Where
you
can,
you
can
kind
of
if
you're,
if
you're
like,
if
you're
just
using
power
again,
you
have
questions
that
would
be
a
good
place
for
you
to
land
and
figure
out
if
other
people
have
had
those
questions
before
whether
that's
the
faq
document
or
just
scrolling
through
the
slack
users,
users
channel
all
right.
So
with
that,
let
me
see.
A
A
A
A
If
you
drill
in
here-
and
I
go
to
the
powergate
pages,
you
can
drill
in
here
and
and
learn
kind
of
the
thinking
behind
powergate
how
to
run
different
instances
of
powergate.
How
to
think
about
storing
data,
learn
to
learn
more
about
the
ffs
instances
and
all
of
that
good.
All
that
good
information
there's
also
the
github
repo.
A
So
if
we
go
and
check
that
out
there's
this
is
a
lot
there's
a
lot
more
kind
of
deep
technical
information
in
here,
even
just
starting
with
the
readme,
an
overview
of
some
of
of
of
that
information,
and
so,
if
you're
running
your
own
powergate
or
you're
extending
on
this
library
at
all.
This
is
a
really
great
place
to
start
so.
For
today,
though,
I'm
going
to
show
you
how
to
run
powergate
in
a
particular
way
so
that
you
can
start
developing
very
quickly
and
the
way
that
we're
going
gonna
run.
A
It
is
using
docker,
so
I'm
gonna
just
open,
I'm
just
gonna
fire
up
docker
here.
If
you
don't
have
docker
installed,
you
can
just
go
to
the
docker
website
and
download
docker
desktop,
which
will
get
it
running
on
your
machine
pretty
quickly
and
that
will
allow
you
to
run
the
containerized
version
of
the
powergate
stack
with
a
pretty
minimal
work
and
pretty
minimal
setup.
A
Okay,
so
the
way
that
we're
gonna
run
powergate
is
the
variant
called
localnet
and
you
can
find
it
here
in
our
documentation
as
well.
The
purpose
of
localnet
is
two
parts,
one.
It's
an
excellent
way
for
you
again
to
spin
up
the
entire
powergate
stack
without
having
to
connect
to
the
full
network
of
miners
and
and
then
be
able
to
create
deals
without
having
to
wait
for
the
full
for
sort
of
the
slower
speed
of
the
real-time
network.
A
We'll
look
at
here
is
the
big
sectors:
option
which
will
let
you
store
big
files
versus
not,
which
is
for
which
it
will
run
faster
and
is
more
tuned
for
small
files,
where
really
you're,
just
testing
sort
of
the
api
and
the
responses
for
apis
and
that
that
sort
of
way
of
running
no
local
net
is
excellent.
For
is
excellent
for
running
in
continuous
integration,
for
example,
and
we
have
examples
of
that
all
over
so
again,
there's
there's
a
few
ways
to
install
this.
You
can
just
go.
A
A
A
Let's
just
jump
in
there
and
the
most
helpful
pieces
here
are
that
they're
there's
two
different
docker
compose
files
and
so
docker
compose
files.
Allow
you
to
or
sort
of
define
a
full
set
of
commands
to
tell
docker
how
to
run
a
system
instead
of
instead
of
sort
of
firing
up
one
at
a
time
with
all
the
configurations.
So
it's
a
recipe,
and
so
there's
two
here
one
is
just
the
docker
compose.
A
The
other
is
docker
compose
localnet,
and
if
we
want
to
peek
in
here,
you
can
see
you
can
see
how
this
stack
is
configured
for
localnet.
It's
going
to
run
three
things:
it's
going
to
run
an
ipfs
node,
a
powergate
node
and
a
lotus
node,
and
then
here
you
can
see
this.
Is
that
big
sector's
flag
that
I
mentioned
before
and.
A
A
So
if
we,
I
don't
actually
recall
this
might
be
fixed
now.
But
yesterday,
when
I
was
yesterday
when
I
was
testing
there's
one
thing
I
ran
into
that,
I
wanted
to
use
as
an
opportunity
to
show
you
you
can
actually
edit
these
docker
files,
there's
no
reason
why
you
can't
tweak
this
to
run
a
different
way
than
it's
defined.
A
Yesterday,
when
I
ran
this,
it
had
a
configuration
that
was
just
ready
for
ready
for
running
against
the
test
net
and
when
I'm
running
localnet,
I
want
it
to
always
just
select
the
best
miner,
because
there's
only
going
to
be
one,
this
one
mocked
miner,
so
powergate
has
a
ton
of
different
commands
that
you
can
read.
You
can
go
into
the
documentation
and
see
these
different
flags
that
you
can
set,
but
I'm
just
going
to
set
this
one
here
as
an
example
of
the
fact
that
I
can.
A
A
But
this
takes
time,
because
if
you
do
this
for
the
first
time
it's
gonna,
the
lotus
note
is
gonna-
have
to
sync
to
the
network
and
catch
up.
So
what
we're
gonna
do
is
run
the
localnet
and
you
can
see
the
default
now
for
localnet
is
to
run
this
with
big
sectors
on.
So
we
don't
need
to
set
that
flag
here,
but
I
can
so.
I
can
just
run
make
localnet,
and
this
will
start
the
work
of
firing
up
these
three
different
containers
and
all
the
and
all
the
things
it
needs
to
do.
A
A
While
it
well,
it's
doing
all
of
that.
The
next
thing
that
we
want
to
do
is
oh
yeah,
and
then
you
know
after
we
do
all
this
testing.
I
think
one
point
is
that
you,
if
like,
if,
when
you're
ready
to
move
to
production,
this
is
where
this
is
a
great
way
to
get
started.
If
you,
if
you
can
run
it
in
a
docker
environment,
where
you
just
run
makeup
and
get
it,
get
it
synced
to
the
chain
and
start
having
your
apis
connected
to
testnet.
A
So
the
next
thing
that
we
want
is
we
want
the
cli
and
so
there's
again
there's
two
ways
to
get
the
latest
cli
and
install
it
on
your
machine.
One
is
just
go
to
the
releases
and
and
download
the
the
pow.
So
there's
three
three
four
builds
in
here:
one
is
the
benchmarking
library.
The
other
is
the
daemon
for
running
the
actual
powergate
api
stack.
The
third
is
the
docker
builds
and
the
fourth
just
pow,
that's
the
client.
Pow
is
the
name
of
the
client,
and
so
you
can
just
download
the
one.
A
A
If
I
do
all
right,
so
I'm
just
going
to
remove
this,
so
we
have
a
nice,
a
nice,
fresh
computer
with
no
pow
command
and
so
to
build
it
from
source
requires
that
you
have
go
set
up.
This
is
this,
is
all
built
and
go
and
so
to
to-
or
this
is
all
written
and
go
so
to
build
it.
You
need
to
have
your
go
environment
set
up.
A
So
if
you
don't
or
you're
not
familiar
with
go,
I
definitely
recommend
going
to
download
the
build
from
our
releases
page
and
install
it
that
way.
So
next,
I
can
again
there's
there's
a
make
file
in
here,
and
you
can
see
it
has
a
bunch
of
good
stuff
in
here,
but
the
most
important
one
is
just
the
install
command,
and
so
we
can
just
make
install.
A
A
I
can
run
the
health
command
and
if
it
gives
me
a
success
it
it's
good.
I
think
that
degraded
I
I
saw
this
running
localnet
before
and
I
hadn't
noticed
it
before.
I
think
it's
just
because
we
are
running
in
in
local
net.
It's
not
a
real
network,
but
otherwise
our
api
is
is
healthy
and
running,
and
so
I
took
this
as
a
good
sign
and
I
continued.
I
continued
yesterday
with
development,
the
the
whole
powergate
api
is
sort
of
split
into
two
parts.
A
There
are
admin
methods,
and
then
there
are
ffs
methods
and
the
admin
methods
you
can
run
here
and
then
you
can
create
ffs
instances
and
get
tokens
and
only
ffs
token
holders
can
then
run
ffs
commands.
So,
oh,
if
you
run
the
help
command,
you
can
see
a
bunch
of
different
things
available
for
you
to
run.
A
A
lot
of
these
might
be
a
little
bit
boring
to
start.
If
I
run
the,
if
I,
if
I
make
a
request
to
get
all
the
asks,
so
this
is
one
of
the
indexes
that
has
been
built
up.
It's
not
going
to
be
very
exciting
on
localnet,
because
there's
only
one
mocked
miner.
If
you
ran
this
on
test
net,
it
would
be
much
more
interesting.
A
You'd
have
a
bunch
of
different
miners
that
you
could
page
through
and
you
it
would
give
you
a
lot
of
information
about
what
they're
asking
for
storing
data
same
thing
with
all
the
all
of
the
indices.
You'll
just
see
on
localnet
it
just
has
it
just
has
a
single
minor
there
and
again,
your
peer
network
on
local
net
is
going
to
be
a
little
bit
a
little
bit
boring.
Perhaps
but
but
when
you're
running
on
test
net
you,
you
should
see
a
lot
more.
So
let's
go
here.
A
A
And,
like
I
mentioned,
pygate
wasn't
developed
by
textile.
It
was
developed
by
a
team
at
hack,
hackfest
and
but
it's
a
really
cool
little
library
that
will
allow
us
to
write
some
things
in
python
and
then
I'm
going
to
show
you
in
the
cli
doing
more
or
less
the
same
things.
So
to
run
this
to
run,
pygate
you'll
just
need
to
install
it
with
python.
A
A
A
A
In
the
same
way,
I
can
do
this
in
the
cli,
so
if
I
do
pow
fbs
create
you
can
see,
it
came
back
with
the
same
token,
and
this
is
something
that
you
want
to
keep
kind
of
secure
in
your
system.
Again,
this
should
map
to
a
user
or
a
system
process
that
really
is
going
to
control
this
wallet
or
set
of
wallets
and
make
requests
for
new
new
deals
and
pay
for
them
out
of
the
wallets.
Behind
this
ffs
instance,.
A
A
The
cool
thing
about
powergate
powergate
instances
that
we're
doing
is
we're
spinning
up
power
gates
for
a
lot
of
teams
right
now
and
we're
auto
funding
that
master
address
and
already
setting
it
up
to
fund
new
ffs
instance
addresses.
So
every
ffs
instance
should
actually
have
a
balance
already
and
so.
A
A
Yep
and
cool,
so
you
see
it's
already,
it's
already
generated
the
first
address
for
my
my
ffs
instance
behind
this
token,
and
it
should
already
have
funded
it
from
that
from
that
master
address.
If
there
were
funds
in
the
master
address,
if
you're
running
your
own
powergate
instance,
you'll
need
to
actually
request
funds
for
your
master
address
and
set
up
that
that
set
it
up
to
fund
your
ffs
instances.
Alternatively,
you
don't
have
to
do
that
and
you
can.
You
can
actually
fund
every
ffs
instance
independently.
A
A
And
again,
I
need
to
tell
it
my
token,
so
I
can
figure
out
that
default
config
before
I
start
doing
things
and
you
could
see
it
has
some
good
settings
already.
So
it's
saying
when
I
push
a
new
cid
and
I
tell
it
to
use
the
default,
pin
that
data
to
pin
that
data
to
ipfs
and
also
create
a
new
deal
for
it
with
a
replication
factor
of
one
on
file
coin.
So
that
means
it's
gonna
store.
A
One
deal
with
one
miner
on
filecoin
same
thing:
I
can,
I
can
do
ffs
config
defaults
here
and
you
can
see
it
it
more
or
less
says
the
same.
Some
of
these
are
technically
nulls
that
it
prints
nicely
here
and
I
think
python
just
ignores
them,
but
you
can
see
that
it
has
a
has.
The
full
has
the
full
data
of
zeros
and
nulls
here,
so
a
bit
more
complete
to
understand,
what's
happening
there.
A
Okay,
so
for
an
example,
I'm
gonna,
I'm
gonna
blast
through
this
and
kind
of
probably
eat
up
a
bit
of
my
office
hours
as
well,
but
I
wanna
show
you
this
example.
So
I
went
and
found
a
data
set
there's
a
really
great
newsletter.
Called
data
is
plural.
I
recommend
subscribing
to
the
newsletter
if
you
like
data
sets
in
general,
but
they
also
maintain-
and
I
shared
this
in
the
slingshot
slack
channel-
they
maintain
a
table
of
tons
and
tons
of
different
data
sets.
A
So
every
every
edition
of
the
newsletter
is
just
a
list
of
really
interesting
open
data
sets
for
people
to
explore,
and
so
I
found
this
one
on
there.
It's
it's
simply
a
collection
of
images
and
classifications
of
shells
seashells,
and
so
I
thought
this
was
a
really
interesting
data
set
just
to
play
with
the
problem
that
I
had
was.
A
When
I
downloaded
this
data,
it
was
not
necessarily
organized
in
the
easiest
way
to
query
it
or
build
it
into
any
kind
of
visualization
or
user
interface
to
to
mess
with
it
later
and
as
part
of
the
slingshot
competition.
That's
one
of
the
requirements
is
that
you
will
want
to
not
only
push
data
to
the
file
point
network,
but
you'll
want
to
give
ways
for
people
to
discover
that
data
or
use
it
in
other
applications
more
easily.
A
It's
a
pretty
basic
python
script
that
just
goes
through
breaks
each
of
these
files
by
the
delimiter
reorganizes
them
into
into
a
folder
called
structured
and
also
builds
up
these
a
few
different
indices
that
it
will
store
in
json
files
and,
what's
nice
about
that,
is
then
I
can
build
a
pretty
simple
web
explorer
for
this
data.
That's
just
going
to
pull
the
data,
the
json
files
populate
some
user
interface
and
know
where
to
find
each
image
based
on
these.
A
A
B
A
And
so
then,
what
we
can
do
with
that
is
is
actually
build.
A
little
web
application
around
it
for
for
visualizing
I'll
show
that
at
kind
of
the
end,
but
the
next
thing
that
we
want
to
do
is
we
want
to
start
pushing
this
to
file
coin,
and
so
we
can
go
back
to
here
and
start
pushing
data
and
so
to
push
data
in
python.
The
pygate
library
includes
a
couple
of
different
helpers
that
you
can
use
to
get
data
into
your
powergate
instance.
A
So,
like
I
mentioned
before,
you
have
to
stage
this
data
on
the
ipfs
network
and
the
the
powergate
stack.
If
it's
running
ipfs
for
you
or
or
if
you're
running
in
production-
and
you
have
your
own
ipfs
node,
you
can
use
your
ipfs
node.
However,
you
want,
but
powergate
also
has
some
helpful
tools
for
you
to
stage
data
for
creating
deals.
But
to
do
it
you
have
to
break
up
local
files
into
bytes
and
byte
chunks,
because
it's
going
to
use
those
chunks
to
stream
that
data
instead
of
moving
giant
blocks
at
a
time.
A
A
Class
and
then
that
new
data
that
I
just
created
or
I've
been
creating
in
the
structured
folder
and
why
don't
I
push
one
of
the
indices
so
and
if
I
look
at
this
images,
json
it's
going
to
be
12
megabytes
so
because
I'm
running
my
local
net
with
that
big
sectors
flag,
I
can
store
a
file,
that's
this
size
and
see
what
happens
and
so.
First
of
all,
let
me
make
sure
that
works.
A
And
then
let's
go
ahead
and
push
it,
so
I'm
going
to
stage
this
file
first
so
again
stage
just
means
move
it
to
the
ipfs
node
connected
to
the
powergate
instance
and
hold
it
there
cache
it
there
until
I'm
done
making
deals
with
it.
So
I'm
going
to
next
I'm
going
to
break
this
file
up
into
chunks,
so
these
are
just
chunks
that
are
the
correct
size
to
stream
over
the
api
and
as
always,
I'm
going
to
create
the
ffs
token
we
generated
above.
A
A
And
you
should
see
that
because
these
are
immutable
ids
that
are
the
exact
fingerprint
of
the
data
that
the
one
that
I
pushed
from
this
ffs
instance
is
the
same
as
the
one
pushed
from
this
ffs
instance.
Remember.
I
made
two
different
ones
at
the
beginning
here:
okay,
so
this
data
is
staged
but
not
stored.
So
now
we
want
to
push
this
data
or
we
want
to
push
the
storage
config
associated
with
this
cid
and
powergate
will
start
doing
work
to
get
it
stored.
A
So,
like
I
keep
mentioning
we're
just
going
to
use
the
default
storage
config.
So
why
don't
I
go
ahead
and
call
client
config,
so
this
is
referring
to
the
ffs
config
sorry
cut
is
actually
wrong
in
the
cli
will
match
that
a
bit
more.
But
so
what
I
want
to
do
is
push
this
cid
or
yeah.
Sorry
we're
going
to
push
the
config
for
this
cid
is
what
what's
really
going
to
happen
and
we're
going
to
again
give
it
our
token.
A
Now.
I
want
to
do
a
few
things
here
along
the
way,
because
this
is
not
an
immediate
process.
So
in
the
cli
I
would
actually
supply
a
flag
here
to
wat
just
watch
the
output
and
it
would
print
to
the
terminal
and
I'll
show
you
that
as
well.
But
here
I
can
just
call
it
and
then
it's
going
to
return
before
everything
is
done,
but
at
the
same
time
I
can
call
the
logs
command
and
call
the
law
or
request
the
logs
associated
with
that
cid.
A
A
A
A
A
A
A
A
A
And
so
we
can
see,
we
can
see
how
this
played
out
now
and
it's
been
pretty
quick
because
it's
running
on
localnet
this
happens
blazing
fast,
whereas
on
testnet
it
will
take
hours
for
the
deal
to
conclude,
but
we
can
see
the
exact
process
that
happened.
We
pushed
the
new
storage
configuration
here
just
using
the
default
associated
with
the
cid
it
stored
that
and
it
started
executing
a
job
with
a
new
id,
and
so
we
can
use
that
id
for
later
checking
that
particular
job
status.
A
When
it
was
so
hot
storage
it
it
realized,
wasn't
fulfilled.
So
it
went
and
ran
that
it
was
happened
really
quick,
because
the
data
was
already
on
ipfs,
so
it
just
pinned
it.
It
ensured
that
cold
storage
was
satisfied
which
it
wasn't
because
the
replication
of
that
data
on
filecoin
was
currently
zero,
but
we
told
it
by
the
default.
We
tell
it
to
replicate
at
least
one
time.
So
then
it
goes
through
the
sort
of
dance
of
finding
the
right
minor.
A
A
Cool
so
then,
finally,
we
can
actually
just
ensure
that
this
is
really
existing
on
the
network
and
we
can
do
that
by
requesting
data
back
from
the
file
point
or
from
the
powergate.
And
when
I
do
this
request,
if
I
had
disabled
hot
storage
and
the
powergate
couldn't
find
the
data
on
ipfs
elsewhere
or
it
wasn't
still
cached
there's
a
flag
in
your
configuration
that
will
allow
powergate
to
make
a
new
retrieval
request
and
that's
called
that's.
A
A
A
A
Great
and
now,
if
we
cat
that
you
can
see,
it
has
our
huge
index
that
we
built
when
I
was
organizing
the
data.
So
here
I
had
the
names
index
was
just
was
just
tracking
the
genus
species
for
every
file
and
what
file
it
was
stored
in,
and
so
ultimately
we
end
up
with
with
this
structured.
This
structured
data
down
in
here,
and
let
me
just
show
that
really
quick,
because
it's
pretty
it's
pretty.
A
A
Before
I
show
you
this,
let
me
just
show
you
what
I've
done.
The
data
that's
actually
intended
to
be
pushed
is
not
the
single
index
the
point,
but
it
would
take
too
long
because
this
is
half
a
gigabyte,
so
I
don't
want
to
do
it
on
the
call
call
right
now,
but
the
idea
here
is
to
actually
push
entire
structured
data
folder
and
I
had
to
recover
it
from
the.
A
In
that
folder,
more
than
my
structured
data,
which
are
those
files
I
mentioned
before,
but
I've
included
pre-pop
indices
and
I've
included
this
index.html
file
and
the
index.html
file
is
designed
so
that
it
could
run
directly
on
the
cid.
So
if
somebody
pulls
it
from
ipfs
or
opens
in
a
gateway,
they
could
actually
use
this
index.html
to
explore
the
data,
that's
in
files
and
in
my
indices
and
if
we
think
about
the
limits
of
filecoin.
B
B
A
Called
data
table:
if
you
go
to
datatable.net,
you
can
check
that
out
and
I'm
going
to
just
create
a
table
of
all
the
data
and
allow
the
user
to
to
explore
it.
And
so,
if
I
go
to
here,
you
can
see,
I
have
it
running
locally.
It's
going
to
do
its
best
to
pull
all
this
up,
and
here
are
all
the
images.
A
And
I
can
do
a
little
search
here
and
it
will
filter
of
the
60
000
records.
It
will
filter
and
find
me
the
images
I
want.
And
finally
I
can
go
and
pull
them
up
here,
and
so,
when
I
store
this
in
using
powergate,
it
will
exist
on
filecoin.
I
can
also
mirror
this
to
ipfs.
I
can
also
do
some
really
clever
things
where
potentially
data
that's
not
available.
A
When
the
user
clicks
on
it,
it's
not
available
over
ipfs,
you
could
actually
go
back
to
powergate
and
pull
it
back
out
and
then,
finally,
finally
I'll
I'll
use
the
same
data
set
in
two
days
to
show
you
how
to
do
really
similar
things
with
textile
buckets.
So
we'll
go
a
little
deeper
on
the
example
and
using
a
different
way
to
get
data
on
file
point,
and
so
with
that,
I'm
a
bit
over
on
my
on
my
workshop.
A
So
I'm
going
to
actually
stop
here
and
open
it
up
for
last
few
questions
and
everything,
but
that
should
really.
I
should
really
help
you
get
started,
building
with
filecoin
using
testnet
using
the
the
docker
instances
and
then
finally,
finally,
we
need.
I
just
want
to
show
you
on
the
textile
blog.
There
is
a
page
for
accessing
powergate
instances,
and
so
what
we're
doing
is
teams
that
are
taking
part
in
the
in
the
competition.
A
If
you
want,
you
can
make
requests
to
to
get
access
to
a
hosted
powergate
instance,
which
textile
will
run
for
you,
give
you
private
access
to,
and
you
can
use
that
full
suite
of
apis.
You
know
you
know
in
a
test
net
ready
environment,
we
definitely
encourage
if
you're
running
giant
data
sets
or
you're
really
proficient
at
standing
up
up
infrastructure
you,
you
really
should
consider
rolling
your
own
and
optimizing
and
actually
even
helping
us
with
feedback
on
on
ways
to
run
it
in
in
those
kind
of
environments.
A
A
One
thing
that
I
noticed
that
the
I
mentioned
before
that
the
pygate,
the
pygate
client
is
a
little
out
of
date.
So
it
wasn't.
I
wasn't
able
to
do
things
like
create
an
override
flag
and
some
some
of
the
some
of
the
methods
are
kind
of
missing
options,
and
so
you'll
just
have
to
work
around
those
or
send
a
pull
request
to
update
them.
B
A
B
A
A
Okay
with
that-
and
I
definitely
recommend,
if
you
get
in
there-
it's
a
it's-
a
really
pretty
straightforward,
client
library,
the
python
one-
and
it's
just
it's
like
really
just
a
pull
request
or
two
away
from
having
the
full
set
of
apis
available
there
with
all
the
options.
And
so
you
could
there's
kind
of
it's
kind
of
a
two-step
process.
There's
just
some
new
grpc
definitions,
you'd
need
to
drop
in
and
then
update
in
the
client
methods
to
to
enable
all
the
all
the
suited
tools,
but
already
it
works.
It
works.
A
A
So,
like
I
mentioned,
we'll
have
another
another
master
class
on
wednesday
morning
and
that
one
we're
gonna
focus
on
textile
buckets
which
are
dynamic,
folders
managed
over
ipfs
and
textile
has
an
api
called
the
textile
hub
through
which
you
can
push
your
buckets
for
remote,
pinning
on
ipfs,
and
also
we
expose
a
number
of
apis.
That
will
let
you
move
your
move.
Snapshots
of
buckets
to
filepoint
through.
B
A
So
with
that
thanks
everybody
for
spending
your
morning
or
evening
or
or
afternoon
with
me
and
we'll
chat
more
over
slack,
like
I
said,
powergate
users,
you
can
find
us
all
and
last
question
there.
The
answer
is
yep
I'll
share
links
to
today's
code
from
from
the
tutorial
I'll,
just
kind
of
clean
that
up
and
probably
get
it
up
there
sometime
this
afternoon.