►
From YouTube: ESPA Module 4D - Data Onboarding Process
Description
Stu Berman (CTO) with PiKNiK discusses the data onboarding process at the Enterprise Storage Provider Accelerator (ESPA) bootcamp week that took place in February 2022.
Data is growing at an incredible speed and much of this data is archived and/or simply lost by enterprises. Our program will accelerate net-new Enterprise Storage Providers into the ecosystem, using a Web3 protocol with an impressive incentive model called Filecoin.
Learn more at:
Sign-up: https://web3espa.io
Landing Page: https://m.fil.org/espa-bootcamp
Follow ESPA:
Twitter: https://twitter.com/web3ESPA
LinkedIn: https://www.linkedin.com/company/web3espa/
A
A
So
we'll
talk
briefly
about
how
that
works
again,
all
the
detailed
documents
will
really
help
give
you
the
the
specifics
of
how
that's
done
now.
In
order
for
storage
provider
to
store
deals,
the
data
has
to
force
first
be
transformed
into
car
files.
A
A
A
A
Now
there
are
brokers
that
can
handle
deal
ingestion
and
transforming
the
data
into
the
appropriate
car
file.
So
this
diagram
really
shows
three
variants
of
this.
The
first,
the
first
road
on
top,
shows
the
customer
data
in
its
raw
format
coming
in
and
then
the
lotus
client
has
to
process
that
into
a
car
file
again
in
multiple
two,
then
that
gets
transferred
as
a
car
file
to
the
lotus
market
as
part
of
a
deal
making
process
and
goes
into
the
storage
provider
for
long-term
storage.
A
The
second
row
is
a
broker
such
as
estuary.tech
and
they'll
accept
data
really
in
any
size
and
within
estuary.
They
will
then
chop
it
up
into
smaller
pieces,
convert
it
to
a
car
so
that
that's
handled
by
estuary
and
not
the
lotus
client
directly
gloss
client
is
a
process
within
estuary
and
then
they
send
those
deals
to
lotus
markets
online
as
well.
A
Offline
deals
require
a
deal
proposal
like
all
deals
from
the
client,
but
then
the
data
transfer
is
handled
by
the
storage
provider,
so
the
example
above
shows
bid
bot.
It's
an
auction
system
for
clients,
the
storage
provider
configures
the
parameters
for
their
bids,
how
much
they're
willing
to
pay
what
size?
All
those
parameters
that
you
have
to
have
and
then,
when
the
storage
provider
wins
an
auction,
the
bidbot
triggers
a
deal
proposal
and
that's
followed
by
an
automated
download
of
the
bidbot
client,
and
so
that's
shown
with
those
two
sets
of
arrows
going
both
directions.
A
A
That's
when
the
the
proposal
starts
from
the
lotus
client
within
bidbot,
hosted
by
textil
to
the
lotus
market,
bidbot
client,
upon
winning
the
bid
and
accepting
the
proposal.
The
process
that's
running
on
lotus
markets,
then
triggers
an
automated
download
through
their
software.
So
that's
handled
automatically
for
you.
A
You're
not
limited
to
these
tools,
there's
other
tools
in
the
ecosystem,
but
these
are
three
good
examples
of
what's
available
until
filecoin,
virtual
machine
or
fvm
is
available
for
storage.
Deal.
Renewals
estuary
today
can
handle
the
renewal
process
for
deals
in
the
future,
that'll
be
on
chain
through
fvm,
and
then
the
various
clients
can
actually
specify
deal.
Renewals
and
payments
for
that
today,
the
deals
have
a
an
expiration
date
that
has
to
be
manually
renewed
in
the
case
of
textile.
A
So
in
this
case
the
storage
provider
is
actually
acting
on
behalf
of
a
customer
and
sending
additional
deals
out
to
other
storage
providers.
Now
that
they've
got
the
first
copy
and
in
conjunction
with
that,
we'll
use
a
tool
like
powergate
from
textile
dot
io
as
well,
and
this
is
really
a
powerful
extension
to
the
basic
native
lotus
client.
However,
it's
got
some
api
integration
and
ability
to
integrate
into
our
system
so
that
we
can
do
automate
automation
on
the
car
file
creation
when
the
storage
provider
is
acting
also
as
a
broker
for
customers.
A
Next,
we'll
talk
about
deal
flow
management,
that's
particularly
important
storage
and
retrieval
deals
require
capacity
management
as
well
as
other
factors
such
as
variable
pricing.
Now
the
most
rudimentary
tool
we
have
in
lotus
is
the
config.toml
that's
stored
in
the
lotus
markets
directory
and
there
you
can
script
responses
to
deal
requests.
You
can
use
perl,
you
can
use
python,
but
the
logic
becomes
overwhelming
if
you
have
sophisticated
deals.
A
So
many
of
us
have
looked
at
third
party
tools,
such
as
cid
gravity,
which
allows
fine-grained
detailed
deal
logic
to
be
performed
with
a
very
simple
user
interface.
I
can
set
specific
clients
to
get
specific
prices
based
on
deal.
Duration
deal
size.
The
deal
type
as
well
as
I
can
write
rate
limit
the
deal.
The
data
flow
that
they
send
to
me.
So
it's
very
helpful
to
have
all
these
different
controls.
A
A
This
start
epoch
can
vary
from
one
day
to
two
weeks.
Each
deal
is
required
to
have
a
start
epoch,
and
that
has
to
be
taken
into
account
with
the
time
it
takes
to
download
the
data
to
import
it
into
lotus,
publish
your
storage,
deal,
messages
and
seal
them
and
get
them
proving
in
addition
to
deal
capacity.
Your
storage,
repo
is
also
limited
in
capacity,
and
you
never
want
your
discs
to
reach
100
capacity.
A
A
A
A
A
Now
there
are
four
basic
messages
that
we
see
on
a
regular
basis,
publish
storage
deal
pre-commit,
commit
and
then
every
day,
a
daily
window
post.
There
are
also
other
gas
fees
associated
with
sending
fill
from
one
wallet
to
another
wallet,
as
well
as
early
termination
and
penalties
for
faults.
As
an
enterprise
storage
provider,
you
may
also
provide
additional
value-add
services
that
are
not
part
of
the
core
lotus
features
currently
available,
such
as
encryption
or
file
chunking.