►
From YouTube: ESPA Module 3G - Scaling Large Filecoin Implementations
Description
Tao Tao (CEO) with Acrontech discusses scaling large Filecoin implementations at the Enterprise Storage Provider Accelerator (ESPA) bootcamp week that took place in February 2022. This discussion includes expansion of architecture through virtualization.
Data is growing at an incredible speed and much of this data is archived and/or simply lost by enterprises. Our program will accelerate net-new Enterprise Storage Providers into the ecosystem, using a Web3 protocol with an impressive incentive model called Filecoin.
Learn more at:
Sign-up: https://web3espa.io
Landing Page: https://m.fil.org/espa-bootcamp
Follow ESPA:
Twitter: https://twitter.com/web3ESPA
LinkedIn: https://www.linkedin.com/company/web3espa/
A
So
today's
topic,
I
want
to
talk
a
little
bit
about
scalability,
so
it's
everyone's
concerned
when
we're
starting
to
do
the
firecore
mining
right
once
you
have
your
first
5p
10
petabytes
of
storage.
How
do
you
scale
that
200
to
1000
right
and
go
from
there?
So
I
want
to
talk
a
little
bit
about
the
challenge.
We
think
the
industry
potentially
have
both
from
infrastructure
perspective
from
architecture
operational
and
what
we
believe
that
we
can.
Our
solution
can
solve
some
of
the
scalability
issue.
A
So,
let's
look
at
an
average
miner
right.
If
you
have
a
miner,
you've
been
doing
this
for
a
while.
You
have
about
a
hundred
petabytes
of
capacity,
maybe
500
kb,
of
ceiling
power
right.
If
you
look
at
your
infrastructure,
footprint
depends
it's
somewhere
between.
You
know.
You
will
have
hundreds
of
servers,
hundreds
of
a
storage
device
thousand,
if
not
a
tens
of
thousands
of
hard
drives.
You
know
hundreds
of
switches
anywhere
between
depends
on
your
power
density
50
to
200
cabinets
right,
sometimes
maybe
multiple
data
centers.
A
If
you
look
at
that,
that's
pretty
darn,
big
stuff,
you
know
mark
and
I
have
been
doing
infrastructure
for
big
enterprise
fortune
100
company.
This
stuff
can
support,
maybe
a
fortune
not
necessarily
fortune
100
but
fortune
500
company
for
sure
in
mid-size
enterprise,
compute
footprint
right
here.
So
now
the
challenge
comes
with
this.
Is
we
really
need
to
run
this
at
a
very,
very
most
efficient
way
right,
because
we
also
try
to
rapidly
expanding,
because
100
petabytes
for
enterprise
may
be
a
lot
of
data
and
for
far
core?
It's
very,
very
small
right.
A
So
the
challenge
comes
with
this
is
how
can
you
further
expand
your
capacity
right,
because
your
capacity
can
either
expand
linearly
or
exponentially
depends
on
the
investment
or
the
way
you
you're
doing
business?
But
how
can
you
ensure
that
your
operational
expenses
not
go
faster
than
your
infrastructure
footprint.
A
So
the
first
challenge
is
you're,
not
getting
all
those
infrastructure
overnight,
you're
buying
them
over
time
right
every
month
you
buy
something
every
quarter
you
buy
something.
So
you
end
up
with
inconsistent
architecture.
Sometimes
you
have
a
new
family
of
a
cpu.
You
have
a
different
number
of
cores.
Sometimes
you
know
dell
is
running
out
of
are
short
on
stuff.
You
get
enough
by
hp
or
super
macro.
What
have
you
right
and
gpu
model
also
have
its
own
roadmap,
so
you
have
a
portfolio
of
your
gpu
portfolio
of
your
ssd.
A
You
all
know
what
I'm
talking
about
you
end
up
with
different
sets
of
things
that
you,
you
have
to
concurrently
run
same
thing
with
the
storage
on
the
storage
side.
You
know
you
have
a
high
tier
required
that
you
know
need
a
very,
very
high,
I
o
and
performance.
You
also
need
a
code
here
where
you
know
you
need
to
be
very
cost
effective
because
you
know
nobody
has
unlimited
resource
here
right.
A
A
Now
the
next
challenge
will
be
the
architecture
if
you
stuck
with
the
traditional
bell
metal
architecture.
The
moment
you
buy
your
hardware.
This
is
what
you
get
it's
very
difficult
to
change,
cpu
to
change
amount
of
memory.
You
have
to
shut
down
the
machine
to
open
it
up.
You
know
it's
very
rigid
in
terms
of
architecture
and
because
of
that
the
average
utilization
is
kind
of
low,
because
you
can
hard
to
match
your
cpu
power
to
your
memory
to
your
gpu.
A
There's
always
some
of
the
component.
It's
not
fully
utilized,
especially
you
know.
Maybe
last
year
you
are
selling
32
gig
of
sector.
Now
you
are
selling
64.
and
the
compute
requirement
profile,
for
that
is
a
little
bit
different
right
or
there's
a
new
version
released
and
someone
optimized
some
algorithm.
A
And
everything
is
there
you
keep
running?
You
need
to
worry
about
trying
to
get
the
next
steps
expand
your
capacity,
but
you
inevitably
is
going
to
spend
a
lot
of
time
and
money
and
resource
on
what
I
call
keep
the
lights
on.
Why
you
need
to
keep
the
lights
on
because
the
stuff
you
bought
two
years
ago,
18
months
ago,
need
a
new
driver
need
a
new
firmware
update.
The
os
needs
to
be
patched
right
when
you
have
a
single
server
or
single
switch,
no
problem.
You
can
do
that.
A
That's
why
you
see
the
little
bit
of
messy
mesh
down
here
when
you
combine
all
the
technology
component
when
you
try
to
keep
them
all
up
to
date.
You
cannot
afford
it
not
to
do
that,
because
there
is
a
security
vulnerability
involved,
so
patching
doesn't
generate
revenue.
Patching
barely
keeps
the
lights
on,
but
you
still
have
to
do
it.
That's
become
a
higher
higher
operational
overhead,
so
you
want
to
get
all
these
things
taken
care
of
without
occupying
your
precious
resource,
so
that
comes
to
a
concept
of
total
cost
of
ownership.
A
Also
from
the
technology
perspective,
you
invest
millions
of
dollars.
Have
the
technology
stacked
there?
What
if
next
year,
the
container
technology
comes
around
the
corner?
Are
you
ready
to
do
that
right?
How
flexible
is
your
system
to
be
future
proof,
so
those
are
the
things
that
at
least
to
keep
us
weak
at
night
until
we
have
a
solution.
A
So,
fortunately,
this
type
of
issue
has
been
for
the
enterprise
I.t
for
last
two
or
three
decades
right
because
back
then
you
know
in
a
1980s
when
I
first
started
companies
proud
of
themselves
have
like
you
know,
one
system
editing
taking
care
of
a
50
machine,
and
today
you
know
at
fortune
100
company.
You
have
one
system,
admin,
probably
taking
care
of
5000,
if
not
10
machine,
and
how
do
they
do
that
right?
A
A
It
will
give
you
a
software
defined
infrastructure,
so
your
infrastructure
is
no
longer
on
bell
metal,
but
because
of
that
virtualization
everything
is
software
defined
because
of
everything
software
defined.
It
will
allow
us
to
do
automation
to
do
infrastructure.
As
a
code
I
can
assign,
I
can
define
the
size
of
my
server.
I
can
real
time
readjust
the
memory
I
allocate
to
that
particular
server
or
I
can
split
the
gpu
into
one
or
one
quarter
of
that
gpu
to
whatever
I
need.
So
I
have
that
flexibility
and
also
because
of
software
defined
infrastructure.
A
It's
given
me
the
automation.
I
can
define
the
policy
based
action
if
this
server
goes
down
a
machine
or
an
algorithm
can
automatically
take
it
away
out
of
the
pool
and
spawn
up
the
new
one.
To
put
in
that
place,
and
also
last
but
not
least,
is
container
technologies
has
to
be
future
ready
right
because
containers
comes
along
and
then
what
do
we
do?.
A
A
We
look
at
it
from
the
angles
like
how
do
we
running
this
in
the
most
efficient
way,
because
we
both
have
done
enough
enterprise
architecture
in
companies
like
hp
and
dell
and
vmware
right?
So
we
look
at
this
compute
profile,
both
from
computing
storage
perspective
like
this
is
what
a
high
performance
cluster
would
typically
do
right.
You
need
have
hot,
very
high
hardware
utilization.
A
A
You
need
have
a
consistent
operation,
no
matter
where
you
are,
what
kind
of
underlying
infrastructure
stack
you?
Have
you
always
consistently
deploy
your
lotus
node
the
same
way
you
manage
your
miner,
you
manage
your
worker
the
same
way
and,
most
importantly,
during
its
entire
life
cycle.
You
maintain
it
in
such
ways.
A
B
B
In
order
for
us
to
really
utilizing
majority
of
the
resources,
we
also
need
to
really
squeeze
the
last
bit
of
the
memory
and
therefore
memories
also
relocate
dynamically
on
the
flight
virtualized
as
well,
and
the
other
part
is
from
the
gpu
standpoint,
because
last
year
I
think
most
of
us
who
has
been
through
this
business.
We
all
have
supply
chain
issues
and
because
of
supply
chain
issues
and
also
the
skyrocket
of
the
price.
B
The
coupled
driver,
it
means
utilizing
a
single
driver,
stable
driver.
You
have
the
abilities
to
using
that
particular
driver
for
all
your
systems
and
by
doing
that
alone,
think
of
how
much
obturation
headache
you're
minimizing
and
the
other
part
is
in
the
last
part-
is
also
we
are
virtualizing
the
high
performance
and
yummy
drives.
B
So
even
though
there's
a
single
drive,
they
can
providing
lots
of
iaps
lots
of
throughput,
but
when
we
are
dealing
with
the
ceiling
power,
you
know
by
the
by
the
commit
by
the
pomp
tools.
It's
not
enough.
The
single
drive
is
not
enough.
Even
aggregate
drive
on
a
single
server
still
not
enough.
So
what
we
did
was
we
kind
of
virtualized
the
entire
hardware.
From
the
storage
architecture
standpoint
we
aggregate
all
the
drives
from
all
the
server
and
therefore
we'll
be
able
to
achieve
some
phenomenal
numbers
next
page.
Please.
A
B
Exactly
by
the
way,
this
is
very
good
question,
and
that
is
you
know.
Initially,
we
did
have
some
lots
of
debate
from
architecture
standpoint,
but
let's
put
this
way,
you
know
if
you,
this
is
a
switch
data
center,
which
is
a
phenomenal
environment,
but
by
default,
if
you're
looking
at
most
of
physical
server,
not
by
one
day
but
by
365
through
one
year
period,
the
utilization
surprisingly
you'll
be
surprised,
probably
between
20
to
30
max
max
and
also
that's
including
reboot.
B
And
also
the
other
point
is
the
syncopation.
You
do
any
maintenance
instead
of
the
entire
single
server
physical
server
goes
down,
only
the
vm
goes
down
and
because
we
decouple
the
entire
driver
stack,
then
suddenly
you're
not
going
to
experiencing,
let's
say
10
15
minutes,
reboot
time
everything
comes
back
within
10
to
15
seconds.
A
B
So
this
one
here
is
so
we
kind
of
you
know.
After
our
initial
poc
pilot,
probably
two
to
three
months,
we
decided
to
go
to
almost
like
a
three
tier
on
storage
architecture.
Standpoint,
three
tier
it
doesn't
mean
the
traditional.
You
know
the
storage
fabric,
you
guide,
the
you
know
the
traditional
router.
It's
usually
utilizing
a
three
type
of
performance
storage.
Go
to
the
previous
slide.
Please
yes,
right
here!
So
the
first
year
we
are
having
is
the
hot
here
the
heart
here.
B
This
is
where
all
the
ceiling
and
happened
referring
to
as
the
old
nvme
drives
the
way
we
are
putting
all
the
vme
drives.
We
kind
of
return
to
the
enterprise.
It's
you
know,
because
in
the
past
five
to
eight
years
enterprise
they
really
develop
a
very
mature,
hybrid
converging
fan,
structure.
Hyper
converge
means
you
know
they.
Instead
of
using,
let's
say
the
traditional.
You
know
what
we
typically
single
data
center
is
blade
system
attached
to
a
sense
storage,
maybe
a
pure,
maybe
something
else.
B
Such
as
the
mouse
system,
you
connect
to
the
the
fabric,
switch
you
think
of
in
the
back
and
fiber:
that's
either
8
gig
or
16
gig,
but
guess
what?
When
most
systems
2
to
it?
Let's
say:
16,
node
or
32
node.
Suddenly,
each
node
may
only
have
one
giga
2
gig,
that's
not
enough.
So
the
way
we
did
hyper
converge
is
pretty
much
we're
using
pizzabox
pizzabox.
B
Is
it's
also
two
tiered
within
the
hot
air
means
we're
actually
using
somewhat
of
the
very
based
intel
obtained
cash
drives
and
from
the
income
cash
drive
at
the
first
right
with
multiple
groups
and
then,
when
the
iphone
go
to
the
also
the
high
performance
nvme
capacity
drive.
By
doing
that
alone,
we
are
serving
two
purpose.
The
first
purpose
we
are
have
the
abilities
to
utilizing
the
full
potential
of
all
the
intel
nvme
drive.
Let's
say
the
current
one
we
have
in
the
production
is
the
intel
5800.
B
B
On
top
of
that,
we
also
also
offer
multiple
capacity
drives
in
there
and
think
this
way,
because
you
know
a
good
question:
asking
is
okay:
what's
the
endurance
for
the
drives
they're
very
expensive,
and
also
if
the
way
how
the
falcons
keep
writing
on
them,
you
can
easily
easily
wipe
the
entire
drive.
You
know
within
a
couple
weeks
or
a
couple
months.
B
It
depends
on
endurance
of
the
drive
and
then
one
of
the
key
thing
is
using
this
particular
you
know
the
intel
obtained
and
then
to
the
high
performance,
nvme
capacity
drive,
and
by
doing
this
a
particular
approach,
remember
all
the
right
actually
go
to
the
intel
opening.
First,
your
intel
obtained.
You
know
the
one
we
have
is
kind
of
right
up
to
100
petabytes
per
day,
100
no
100
by
that's
the
life
endurance.
B
You
know
so,
but
you
know
we
have
a
few
of
them,
so
they
write
to
the
intel
first
and
then
from
intel
to
all
the
capacity
drive.
So
depending
on
how
many
capacity
drive
we
have
then
suddenly,
instead
of
each
capacity
drive,
let's
say
we're
using
the
sum
of
them.
We
have.
Is
the
micron
9300
max
also
phenomenal
drives,
so
instead
of
each
drive
may
taking
the
actual
right?
B
Maybe
let's
say
you
know,
500
terabyte
to
one
petabyte
per
day,
but
using
this
one-time
wrap
a
day
using
this
particular
approach,
actually
only
one
tenth
or
twenty
percent
of
the
actual
right.
But
what
happens
the
drives?
I
go
to
the
intel
opting
first
and
then
from
the
intel
obtain
go
to
the
supermax,
go
to
the
micron.
It's
only
a
very
small
percentage
10
to
20,
then
suddenly
the
drives
you're
going
to
increase
the
drive
life
endurance
by
five
to
seven
x
per
drive.
B
So,
even
though
we
are,
we
did
spending
some
money
on
the
hardware
I
mean
on
the
software,
but
then
we
start
to
calculating
the
amount
of
capacity
investment.
We
are
saving
on
actual
drive.
It's
actually
a
much
better
approach
doing
that.
So
that's
what
we
have
on
the
heart
here
and
then
initially,
our
code
here
is
directly
with
cfs
and
then
you
know
from
our
learning.
What
we
identified
cfx
is
a
phenomenal
storage,
especially
from
the
the
cost
per
terabyte
standpoint,
and
also
reliability
with
the
data
integrity.
B
B
So
that's
why,
in
the
middle,
we
put
the
middle
one
here:
they're,
not
nvme
ssd,
but
their
regular,
sata,
ssds
ssd,
and
then
they
be
to
almost
like
a
absorption
like
a
buffer
zone
between
the
very
fast
high
performance
lvme
to
the
lower
cost
on
all
the
cfs.
So
that's
kind
of
what
we
have
on
the
storage
architecture
standpoint.
Any
questions.
A
E
B
The
scheduling
standpoint
on
the
heart
here
because
we're
taking
approach
of
the
hyper-converged.
What
does
it
really
mean
is
that
there
are
two
ways
we
can
scaling
the
solutions?
The
first
way
is,
we
call
them
a
scaling
up
means
if
we
have
the
2u
server
and
by
default
most
of
the
time
we'll
be
seeing
today,
the
tool
server
has
24
base.
We
don't
have
to
populate
all
the
base.
You
know
fully
max
up
front,
but
it
depends
on
the
workload
we
only
populate
depends
on
how
much
we
actually
need
to
write
to
it.
B
Let's
say
only
populate,
maybe
eight
drive
out
of
the
24
and
then
later
on.
When
we
realized,
we
need
either
more
performance
or
capacity.
We
simply
add
in
the
drives
to
all
the
servers.
So
this
is
one
of
the
ways
for
us
to
scanning
the
capacities
from
the
scaling
up
standpoint.
That
means
we
haven't
really
reached
the
limitation
on
the
cpu.
All
the
memory
and
the
other
way
we
can
scaling
out
you
know
means
if
the
cpu
we
do
have
the
limitations.
We
also
reach
limitation
on
the
memory.
B
Let's
say
two
terabytes,
we
simply
needing
add
more
low
power
or
even
gpus,
and
by
doing
that,
we
adding
the
identical
servers
or
similar
servers
to
the
cluster,
and
by
doing
that,
we
can
simply
dynamically
increase
the
storage
and
performance
at
the
same
time
on
the
fly.
So
this
is
another
way
for
us
to
scaling
the
solutions
on
screening
out.
Okay,.
B
And
also
from
the
security
standpoint,
you
know
one
of
the
key
thing
we
kind
of
look
into
this.
You
know
we
are
treating
this
as
a
high
performance,
computing
environment
and
also
we
are
treating
this
as
a
true
pci,
compliancy
environment.
Part
of
the
reason
is,
you
know,
even
though
right
now
from
the
pure
data
standpoint,
you
know
it's
not
as
sensitive,
but
we
are
ready
to
take
in
some
of
the
customers
live
production
data
and
also
we
are
in
the
process.
Really.
This
is
a
you
know,
producing
the
crypto.
B
So
by
doing
that,
we
are
borrowing
some
of
the
best
practice
from
the
pci
compliancy,
and
by
doing
that
we
are
doing
the
full
end-to-end
access
management
from
the
auditing
standpoint.
You
know
if
we
need
to
make
a
change
on
the
firewall.
Everything
is
software
defined
using
a
single
policy
push.
Then
we
know
exactly
what's
going
on,
even
our
developers
or
the
operation,
the
administrators
who
access
the
environment,
we
have
multiply
of
multi-factor
authentication
building.
Everything
is
fully
out
automated
everything's
fully
audited.
So
why
we're
you
know
we're
going
to
taking
additional.
B
You
know
really
taking
the
platform
to
the
next
level.
You
know
why
there's
the
pci
compliancy,
if
that's
required,
or
whether
it's
in
the
back
end
we
to
make
sure
all
the
people's
access
fully
audited.
Then
we
multiply
that
right
away.
So
that's
why
security
is
another
key
things
within
our
design.
B
And
then
the
next
thing
is
really
container.
Writing
you
know,
because
personally
I
had
the
you
know.
I
had
a
lot
of
fun
back
then,
10
years
ago
we
initially
docker
come
out.
You
know
with
lots
of
other
flavors
of
minings
and
with
oddly
you
know,
when
you
know
back
then
was
darker
then
later
on
become
kubernetes
with
lots
of
other
cryptos.
You
know
people
utilizing
container
base,
it's
so
much
easier.
You
know
easy
to
deploy
easy
to
maintain
a
single
image
you
can
deploy
anywhere
and
then
that's
one
of
the
disadvantage.
B
Utilizing
a
file
concurrently
because
of
the
complex
of
the
environment,
complex
of
the
architecture.
It's
not
quite
ready,
but
then
you
know
we
do
know
you
know
on
the
roadmap
or
by
the
community's
demand.
It
just
could
be
a
matter
of
time.
So
that's
why
the
platform
we
are
choosing
is
also,
you
know,
really
a
container
writing
kubernetes
in
day
one.
So
why
were
the
code
is
ready?
We'd
be
able
to
pushing
that
you
know
from
there
right
away
and
currently
our
to
go.
A
So
we
talk
about
the
consistent
infrastructure
right.
Let's
talk
about
the
consistent
operation,
so
what
the
software
defined
infrastructure
allow
us
to
do
is
large-scale
deployment,
because
we
can
script
everything
we
have.
You
know
everything
is
software
defined
right
and
we
talk
about
the
full
lifecycle
management
that
we
can
apply
patches
and
the
drivers
and,
more
importantly,
like
mark
point
earlier
that
the
hardware,
the
device
level
driver
is
decoupled
from
my
actual
servers,
because
everything
running
as
a
virtual
server.
A
That
way
I
can
maintain
the
entire
stack
my
average
from
bare
metal
server
perspective.
My
average
utilization
is
above
70
75,
so
server
it's
running
working
force
to
its
you
know
those
are
expensive
equipment,
so
anyone
sign
the
purchase.
Order
knows
that,
so
we
want
to
squeeze
the
last
bit
out
of
them.
B
B
E
A
Did
our
analysis
of
the
size
we're
not
there
yet
we're
not
a
thousand
petabytes,
but
that's
the
size
we
want
to
be
like
end
of
this
year
so
last
year
we
did
the
full
total
cost
of
ownership
analysis.
If
we
go
bare
metal,
you're.
Looking
on
the
right
side
left
side
right,
you
have
your
hardware
cost.
Your
software
is
pretty
much
free.
A
You
don't
have
a
whole
lot
of
software
cost,
but
your
label
cost
is
right
up
there
for
a
three
year,
total
cost
of
ownership.
But
if
you
invest
some
money
in
this
case
about
nine
million
dollars
on
the
vmware
software
license,
this
is
a
tag
price
right
after
negotiation
will
be
done
a
little.
But
as
a
result,
you
have
much
higher
hardware
utilization.
A
So
you
do
not
need
to
buy
that
many
hardware
and
because
of
the
software
defined
infrastructure,
you
have
all
kinds
of
automation.
Your
availability
is
also
higher
right,
so
you
drastically
reduce
your
labor
cost
your
opex
so
as
a
result,
you're
saving
24
million,
with
the
invest
of
nine
million.
So
you
spend
money
to
save
money.
If
you
look
at
the
total
cost
of
ownership
over
three
years,
so
instead
of
three
spending
60
million,
we
almost
cut
that
in
half.
A
B
And
also
from
our
typical
deployment
standpoint,
we
have
pretty
much
two
layers.
One
of
them
is
hardware.
Another
one
is
the
os
or
radio
application
on
the
lotus
from
all
the
server
deployment,
since
we
are
standardized
within
the
virtualization
so
think
of
typically,
even
though
from
the
physical
server
there's
different
tools
you
can
use,
but
the
reality
is
different.
B
Servers
also
have
different
hardware,
how
different
drivers
different
bios,
different
configurations
and
also,
if
you
multiply
that
by
10
by
hundreds
of
thousands,
then
definitely
require
loads
of
overhead
ruler,
getting
system
ready
and
because
we
virtualize
everything
from
os
standpoint.
Then
suddenly,
for
us
is
a
template,
a
template.
We
deploy
a
template,
we
configure
everything
pre-configured,
you
know
we
talk
about
within
10.
Actually
within
minutes
we
can
guiding
the
template
ready
and
once
the
template
is
ready,
then
we
are
utilizing
the
automation
tools.
You
know
one
of
them.
B
A
I
want
to
throw
one
example.
Last
week
our
head
developer
teams
come
to
us
see,
you
know
what
the
nine
terabytes
per
vms
you
you
assigned
earlier
is
proved
a
little
bit
shy.
Can
you
increase
the
server
to
12
petabytes
per
server?
Again,
those
are
all
virtual
server
right.
Take
like
what
two
hours
push
out.
You
know
hundreds
of
vms
sell
down
all
these
stickers.
Just
one
reboot
and
I
do
not
have
to
stop.
You
know
once
that
vm's
done
with
this
job,
reboot
that
one
the
rest
of
the
vm
still
get
the
fire
code.
B
This
particular
way
along
doing
that.
Think
of
you
know
if
using
the
traditional
approach
do
that
on
the
physical
server,
how
much
downtime
you're
going
to
have,
and
even
though
you
can
deploy
some
but
more
likely
even
using
proxy
or
something
else,
something
might
happen
will
happen
which
therefore
directly
reduce
your
utilization.
So
using
our
approach,
that's
why
ours,
our
approach
is
so
efficient
because
of
automation.
B
And
also
from
the
monitoring
standpoint,
you
know,
I
think
this
is
another
challenge.
Lots
of
people
run
into
means,
there's
different
hardwares,
there's
a
different
storage,
utilizations
network
utilizations,
you
know,
there's
a
miners,
there's
workers,
everything
else.
So
that's
why
we
do
have
a
sophisticated
monitoring
solutions
really
be
able
to
monitor
the
end
to
end
monitor
so
of
every
single
component,
every
single
sensor.
We
know
exactly
what's
going
on
at
what
time
and
also
you
know,
crosstalk
multiple
data
centers,
and
that
is
essential
for
us
to
perform
our
operations.
A
A
You
have
your
physical
infrastructure
right.
You
need
to
set
up,
install
provision,
you
know
all
the
all
the
all
the
physical
stuff,
and
then
on
top
of
that
you
have
your
os
layer
where
you
configure
your
miners
and
the
workers
and
all
that
and
of
course,
on
top
of
that
you'll
have
your
daily
mining
operation
and
distribution,
monitoring,
alertings
and
dealing
with
the
issues
you
have
so
as
a
company,
we
can
really
be
very
flexible
on
any
of
that
right.
A
We
can
do
everything
you
know
from
the
invest
perspective,
all
the
all
they
want.
Is
you
know
I'm
looking
good
for
I'm
looking.
I
I
think
the
firecoin
is
promising.
I
want
to
get
into
there,
so
I
don't
need
to
own
anything.
We
will
figure
out
how
to
buy
what
kind
of
equipment
we
need
to
buy.
We'll
set
it
up,
you
know
we'll
allocate
part
of
the
investment
to
the
pledge
and
will
go
from
there
right.
That's
one
model
or
you
know
some.
Some
investor
may
want
to
be
more
in
control
right.
A
They
can
have
their
own
pledge,
they
can
own
the
equipment,
but
we
can
still
come
in
to
set
up
the
software
defined
infrastructure.
The
whole
technology
stack
get
the
operation
going.
We
can
even
do
the
operation
for
you
and
of
course
there
is
you
know
whatever
you
want
saying.
As
a
professional
service,
we
can
look
at
your
existing
operation
based
on
what
we
have
done
right.
How
can
you
be?
Where
is
your
low-hanging
fruit?
How
can
you
be
more
efficient
in
your
current
operation
model
without
through
everything
out
of
the
window?
A
That's
you
know
once
one
size
doesn't
fit
all,
maybe
you
already
have
a
thing
going?
You
have
a
team
going.
You
have
existing
skill,
we
can
come
in
to
look
at
your
stack
and
say:
maybe
you
do
a
little
bit
here
and
there
fine
tuning
and
get
more
band
out
of
your
back
and
also
from
scalability
and
from
architecture
perspective.
A
So
pretty
much
this
is,
you
know,
catch-all
type
of
thing
that
anything
that
we
can
offer
as
a
service,
and
last
we
also
working
with
our
partners
to
get
some
real
data
in
our
data
center
right
now.
We
all
know
that
you
know
that's
a
new
trend
and
we
want
to
jump
on
that
train
as
well.
So
we
have
some
real
data's
already
in
china
that
we're
piloting,
but
we're
open
to
get
some
of
them
in
u.s
operation
as
well.