►
From YouTube: [Estuary] Ecosystem February 2022 Deep Dive: Command Line Programs to Manage Files on Cloud Storage
Description
The Ecosystem WG is a collection of 80+ people across 5+ organizations that are passionate about the long-term adoption of Web3 and dedicated to growing the Filecoin, IPFS, and libp2p ecosystems.
We regularly host all hands meetings to check in on progress and showcase the growth of various developers, applications, storage providers, clients, and communities around the world.
This deep dive features Jimmy Lee and Marshall Culpepper simplifying RClone’s support for Estuary, you’ll learn how you can use Estuary and popular file system management tools such as RClone to upload files even faster and easier to Filecoin.
Check out more about Estuary and Rclone here:
https://estuary.tech
https://rclone.org/
If you, your team, or organization want to be a part of our working groups or would like to present and be featured in our next public All Hands (March 24), please reach out to: ateam@protocol.ai.
View this months slides here:
https://docs.google.com/presentation/d/1RvWuQB9LOac3nIsaJ_vjEW2LOwwM3H6dCc3mMr6qRP8/edit?usp=sharing
A
Hey
everyone
thanks
for
having
me,
so
I'm
really
lucky
today
to
kind
of
present
marshall's
work
that
he's
done
recently.
It's
absolutely
huge.
He
added
article
and
support
for
estuary.
So
for
those
of
you
who
don't
know
our
clone,
is
it's
a
command
line
program
that
helps
you
manage
files
on
cloud
storage?
A
Many
more
petabytes
of
storage,
potentially
and
currently
right
now
estrella,
is
being
used
by
textile
weaponry,
storage,
nft
storage,
zora
chainsafe,
and
many
other
collaborators
too.
So
we're
hoping
that
this
will
open
the
funnel
for
more
data
for
those
of
you,
who've
never
used
asteroid
before
you
know
here's
the
website
we'll
go
through
how
to
upload
data
without
our
clone
and
how
tedious
that
can
feel,
and
then
we'll
show
you
how
what
the
difference
is
when
you
use
our
clone.
A
You
know
you
can
create
an
account
simple
sign
in
form
and
then,
if
you
need
to
upload
data,
you
need
to
have
an
api
key,
and
so
next
one
here
is
a
view
of
your
files
and
data.
So
this
is
my
account
and
I
have
around
233
gigabytes,
and
that
means
that
that
data
gets
replicated
six
times.
So
it's
around
one
terabyte
of
data
in
total
and
anyone
with
an
account
can
use
this
page
to
check
here
is
the
api
key
page.
You
can't
actually
see
these
api
keys.
A
I've
revoked
all
of
them,
but
you
can
generate
a
key
and
once
you
generate
a
key,
you
can
go
and
upload
data
using
any
script
that
you
might
write
and
then
here's
an
example
of
that
script.
So
you
have
an
api
key.
If
you
were
to
upload
one
file
at
a
time,
you
can
use
28
lines
of
code
and
you
can
use
this
example.
A
You
can
either
use
the
one
that
we
have
on
github
or
write
one
yourself.
It's
pretty
simple.
What
you'll
see
is
results.
So
if
you
fail
and
use
the
wrong
api
key
won't
upload
anything.
You
won't
get
any
response.
It'll
say
error
invalid
auth,
but
if
you
use
an
api
key,
that's
valid
you'll
see
the
cid,
the
local
s3
id,
which
is
our
own
index
and
then
the
providers
that
have
it
so
it's
as
ipfs
as
it
gets
from
here.
Marshall
will
take
over
and
show
you
how
our
clone
makes
a
huge
difference.
B
Thanks
jimmy
yeah,
so,
as
you
said,
estuary
is
really
great
for
basically
automating
a
large
data
set
onboarding
into
the
filecoin
ecosystem.
It
does
all
the
automated
deal
making
and
it
also
does
pending
and
ipfs,
which
are
really
you
know,
challenging
to
do
from
a
client's
perspective,
but
we
still
need
a
really
good
client
front-end
to
kind
of
manage
the
uploads
and
sort
of
file
system
import
into
that
into
that
ecosystem,
and
so
what
our
clone
gives
us
is
two
really
really
big
sort
of
value
adds.
One
is.
B
That
means
that
we
can
move
data
and
mass
from
storage
providers
like
google
drive
or
dropbox
or
s3,
or
so
many
other
different
examples
that
you
can
see
on
their
website
and
so
basically
to
get
started
using
our
clone.
It's
really
straightforward,
you
just
create
a
configuration
file
and
there
is
a
tool
that
automates
this
for
you.
B
All
you
really
need
to
have
in
that
configuration
file
is
the
fact
that
you're
using
estuary
as
your
backend
and
the
token
that
you
get
from
the
the
site
that
jimmy
just
showed
and
once
you've
got
that
configured.
You
can
start
using
the
estuary
back
end
as
if
it's
a
file
system
on
your
local
computer.
B
So
just
a
couple
of
examples
to
show
like
the
power
of
like
how
simple
this
can
be
to
use.
So,
for
example,
I
have
a
directory
in
my
google
drive
called
rclone
test2,
which
has
like
another
directory
and
some
files
under
it.
I
can
just
like
list
that
directory
with
our
clone
and
sort
of
see
it
like
a
normal
file
system,
but
what
I
can
also
do
using
the
copy
command
is.
B
I
can
recursively
copy
that
entire
file
system
into
estuary
and
if
you
notice
in
the
second
invocation
here
on
my
command,
it's
our
my
command
line.
It's
our
clone
copy
and
there's
a
there's
one
kind
of
slight
difference
between
normal
file
systems,
which
is
that
you
use
the
name
of
the
cloud
provider
as
a
prefix
to
the
path
so
g
drive
colon
means.
I'm
using
google
drive
estuary
colon
means
I'm
using
estuary.
Those
are
the
names
I
gave
them.
Those
are
totally
arbitrary.
B
Obviously,
these
names
are
generated
for
testing
purposes,
but
I
think
it's
pretty
straightforward,
and
so
what
estuary
does
in
the
background-
and
this
is
not
an
rclone
feature-
this
is
an
estuary
feature-
it's
it's
actually
pinning
that
content
onto
ipfs
and
it's
automating
the
process
of
doing
deal
making
and
getting
that
storage
into
the
file
point
network,
and
so
because
now
this
content
is
in
the
estuary
backend.
I
can
start
doing
things
like
seeing
the
cids
for
content
immediately
and
so
there's
another
command
that
I
added
to
arclund.
B
That
specifically
supports
a
directory
listing
of
cids
based
on
content
that
you
have
in
your
directory
structure.
I
just
called
it
lscid
that
seemed
right
and
what
it
lets
you
do.
Is
you
see
every
file
in
that
directory,
along
with
the
cid
as
a
column,
in
the
listing?
This
actually
supports?
Not
just
the
cid
is
like
a
raw
text
string
but
formatting
as
gateway
urls
and
a
few
other
nice
little
features
so
that
you
can
do
like
automated
clicking
and
like
get
get.
B
B
This
is
like
a
really
good,
immediate
feedback
way
to
make
sure
that
content
is
getting
into
estuary
and
then
like
what
what
sort
of
data
has
been
pinned
and
like
where,
where
it
will
be
pinned-
and
you
can
actually
use
these
cids
to
retrieve
content,
usually
using
like
dweb.link
or
ipf
icio
after,
like
usually
a
short
delay
after
initial
pinning,
and
so
that's
that's
sort
of
like
the
the
tl,
dr
r
r
clone.
You
know.
B
The
purpose
of
this
is
like
really
to
help
these
large
data
migrations
get
into
estuary
and
again,
most
of
the
smarts
of
what's
happening
here
are
not
in
the
client
they're.
In
the
back
end
of
what
estuary
does
a
day-to-day,
which
is
you
know,
up,
take
that
uploaded
content
and
then
go
automate.
The
deal
the
deal
making
and
the
the
management
of
the
the
trusted,
my
the
trusted,
storage
providers,
etc,
etc,
etc.
C
Marshall
cake,
thank
you
so
much
what's
not
represented
on
the
screen
here
is
how
much
the
chat
is
blowing
up
on
the
value
and
simplicity
of
this
kind
of
art.
Loan
support
breast
story
so
like
it
is
folks,
are
really
really
excited
and
I'll
quote
cake
here
that
there's
no
excuse
not
to
for
large
data
migrations
to
not
pass
billions
of
objects
onto
filecoin
now
that
the
support's
available.
So
thank
you
so
much
and
thank
you
for
supporting
estuary.