►
From YouTube: ESPA Module 4B - Servicing and Maintenance
Description
James Hoang (Sr. Production Engineer) with PiKNiK discusses servicing and maintenance at the Enterprise Storage Provider Accelerator (ESPA) bootcamp week that took place in February 2022. This discussion includes monitoring tools, tooling/debugging, chain maintenance, proof of space time (PoSTs), and block rewards.
Data is growing at an incredible speed and much of this data is archived and/or simply lost by enterprises. Our program will accelerate net-new Enterprise Storage Providers into the ecosystem, using a Web3 protocol with an impressive incentive model called Filecoin.
Learn more at:
Sign-up: https://web3espa.io
Landing Page: https://m.fil.org/espa-bootcamp
Follow ESPA:
Twitter: https://twitter.com/web3ESPA
LinkedIn: https://www.linkedin.com/company/web3espa/
A
Hello
and
welcome
to
the
servicing
and
maintenance
module
for
this
module
today,
I'll
be
going
over
some
of
the
tools
that
are
available
to
run
and
maintain
your
storage
node,
we'll
start
with
talking
about
some
of
the
monitoring
tools
available.
So
why
do
we
even
need
monitoring
tools?
Well,
as
with
anything
in
life,
nothing
ever
goes
smoothly
right.
A
There
are
going
to
be
issues
that
you
run
into
or
you
over
provision
on
hardware
and
need
to
re-optimize,
and
you
can't
re-optimize,
if
you
don't
have
any
metrics
or
statistics
to
track
the
many
phases
of
sector
sealing
involved
to
file
coin,
which
leads
us
into
the
next
section
of
what
do
you
even
need
to
monitor?
Well,
as
I
just
mentioned,
the
sector
ceiling
pipeline,
for
instance,
you
can't
re-optimize
or
troubleshoot.
A
A
When
and
not
if
one
of
your
servers
has
a
process
that
kills
itself
you'll
be
able
to
quickly
detect
that
it
simply
ran
out
of
ram
instead
of
spending
days
to
figure
out
that
otherwise.
Finally,
it
is
extremely
important
that
your
wallet
balances
are
tracked
and
not
just
for
accountability
reasons,
but
also
in
case
your
control
wallet.
Balance
starts
to
go
low.
If
one
of
these
control
wallets
ever
completely
runs
out
of
fill
your
entire
sector
ceiling
pipeline
comes
to
a
grinding
halt.
A
There
will
be
more
on
the
different
types
of
wallets
in
the
upcoming
wallet
setup.
Module
here
are
some
of
the
general
options
that
are
available
to
help
monitor
your
storage
node.
You
can
use
a
blockchain
explorer,
which
is
just
a
website
that
pulls
information
from
the
blockchain.
Some
examples
of
this
are
filefox
fieldscan
or
phil
scout.
There
are
also
the
traditionally
used
open
source
software
applications
such
as
grafana,
which
is
a
great
dashboard
application
or
prometheus,
which
is
a
time
series
database
great
for
tracking
information
over
time.
A
Since
the
filecoin
community
has
such
a
large
software
development
backing,
there
are
a
lot
of
open
source
applications
available.
Such
as
farcaster
a
pre-built
grafana
dashboard
with
all
of
the
metrics
and
information
gathering
already
taken
care
of
or
said,
gravity
a
deal
filter
that
can
help
manage
your
storage
and
retrieval
deal,
pricing
and
other
configurations.
A
Another
consideration
is
that
you
can
always
have
your
software
development
team
make
their
own
scripts
and
automation
processes.
This
will
allow
you
to
create
processes
that
monitor
or
track
exactly
what
you
need
to
next
up
is
ipmi
the
intelligent
platform
management
interface,
which
comes
built-in
with
any
enterprise-grade
hardware.
And
finally,
you
can
always
use
general
linux
cli
commands,
which
can
be
extremely
useful
to
monitor
your
storage
node
in
this
section
I'll
be
discussing
the
ipmi
tools
in
greater
detail.
A
The
ipmi
port
will
also
need
its
own
cable
and
ip
address
in
order
for
you
to
access
its
interface,
usually
the
form
in
the
web
gui.
There
are
numerous
features
that
the
ipmi
tool
provides
and
I'll
be
going
over
some
of
them
now.
The
first
feature
that
I
want
to
talk
about
is
the
ability
to
monitor
your
hardware's
temperature
in
the
screenshot
on
the
left.
A
The
temperatures
of
the
different
ram
slots
and
cpu
are
being
recorded
and
there
is
a
line
graph
of
any
changes
on
the
right
to
indicate
if
there
are
any
sudden
spikes
in
heat.
It's
important
to
monitor
temperatures
because,
as
with
any
server
high,
temperatures
will
destroy
your
servers
and
filecoin
sector
sealing
is
a
very
computationally
demanding
process.
One
of
the
features
that
ipmi
can
support
to
combat
high
temperatures
is
by
controlling
the
fan
speeds
on
the
server
in
this
sample
screenshot.
A
The
line
graph
shows
your
fan
utilization
percent
in
correlation
with
a
specific
temperature
which
is
configured
on
the
right,
for
example,
as
the
heat
of
your
server
increases
from
40
degrees
celsius
to
60
degrees
celsius,
the
fan
utilization
will
automatically
rise
from
50
percent
to
90
percent
in
order
to
combat
the
increase
in
heat.
If
the
temperature
rises
too
high
for
even
the
fans
to
handle,
you
can
just
shut
off
the
servo
with
the
ipmi
tool.
A
I'm
mostly
kidding
on
this
in
this
info
scenario,
it
would
be
better
to
stop
the
processes
that
could
be
making
your
server
overheat
and
powering
it
off
should
be
a
last
resort.
The
ipmi
tool
has
the
functionality
to
remotely
power
off
a
server
which
is
extremely
convenient
if
the
terminal
or
os
is
unresponsive
or
if
the
server
is
physically
hundreds
of
miles
away
from
you
in
another
state
or
country.
A
Even
it
is
important
to
note
that
this
is
different
from
powering
down
a
server
from
the
command
line,
which
is
often
safer
as
the
command
line
shutdown
allows
for
processes
to
come
to
a
stop
properly,
whereas
the
ipmi
is
a
forced,
immediate
shutoff.
On
the
topic
of
remote
leak
accessing
the
server,
the
ipmi
tool
usually
has
the
feature
to
completely
control
the
server
remotely.
A
Besides
some
latency
from
connecting
over
the
network,
you
are
able
to
access
the
server
as
if
you
are
physically
connected
a
monitor,
which
can
be
extremely
useful,
for
example,
when
troubleshooting
processes
and
applications
that
are
not
command
line.
Heavy
and
finally,
the
last
feature
I
want
to
mention
is
the
event
log
which
just
tracks
every
single
event,
error
or
alert.
For
example,
it
will
track
anything
from
removing
the
lid
from
the
server
chassis
to
the
times.
A
One
of
the
redundant
power
supplies
went
offline
for
any
reason,
and
even
each
time
a
ram
stick
makes
a
calculation
error
now
onwards
to
some
linux.
Cli
commands
in
the
next
few
slides
I'll
briefly
go
over
some
examples
of
commands.
You
can
use
to
check
your
hardware
status,
the
free
command
checks,
the
amount
of
registered
ram
and
how
it's
being
used.
The
cad
command
displays
the
content
of
a
file
and
in
the
sample
screenshot.
A
Next
we
have
ls
cpu,
which
displays
the
generic
cpu
info,
such
as
the
model
or
the
cpu
core
information,
and
we
have
the
htop
command,
which
is
useful
for
tracking
all
of
the
ongoing
processes
that
are
running
on
a
server
and
how
they
are
impacting.
The
cpu
htop
can
be
useful
in
determining
if
you
have
too
many
or
too
little
sectors
running
in
parallel
on
a
server
there's.
A
Also,
the
demede
code
command,
which
has
its
own
suite
of
commands
to
check
many
different
aspects
of
a
server
such
as
the
memory
and
processor,
just
to
name
a
few.
It
is
important
to
note
that
this
command
will
cover
different
information
than
the
commands
covered
for
ram
and
cpu
in
the
previous
slides
and,
finally,
the
I
o
top
command
checks.
All
of
the
ongoing,
writes
and
reads
occurring
on
a
server's
drive
which
is
useful
to
see
if
your
drives
are
keeping
up
with
the
amount
of
data
that
they
are
expected
to
process.
A
The
tail
command
is
useful
to
monitor
any
log
files
that
you
might
have
set
up
to
lock.
The
output
of
your
lotus
processes
and
the
watch
command
in
conjunction
with
some
of
the
lotus
commands
can
be
extremely
useful
to
monitoring
your
storage
provider
node
speaking
of
lotus
commands
in
this
final
section,
I
will
briefly
cover
a
few
lotus
cli
commands
that
can
be
used
to
help
monitor
your
storage
node.
One
of
the
most
important
commands
is
the
lotus
miner
info,
which
displays
a
ton
of
information
about
your
storage
provider
such
as
your
power
on
chain.
A
Combining
this
command
with
the
wash
command
mentioned
previously
is
a
simple
but
effective
way
to
start
off,
with
monitoring
your
storage
node,
the
combination
of
lotus
miner,
proving
info
and
proving
deadlines
can
be
used
in
conjunction
to
display
upcoming
window
post
deadlines,
which
is
useful
to
prevent
any
other
processes
from
competing
for
hardware
resources
during
this
crucial
period
of
time.
And
finally,
there
are
plethora
of
other
commands
like
the
wallet
list
and
the
actor
control
list
that
displays
information
about
parts
of
your
storage.