►
From YouTube: Rapid Processing of NASA Satellite Data Stored with Ceph - Kevin Hrpcek & Steve Dutcher
Description
Rapid Processing of NASA Satellite Data Stored with Ceph - Kevin Hrpcek & Steve Dutcher, University of Wisconsin Space Science and Engineering Center
The NASA VIIRS Atmosphere SIPS, located at the University of Wisconsin, is responsible for assisting the Science Team in algorithm development and production of VIIRS Level-2 Cloud and Aerosol products. To facilitate algorithm development, the SIPS requires access to multiple years of satellite data occupying petabytes of space. Being able to reprocess the entire mission and provide validation results back to the Science Team in a rapid fashion is critical for algorithm development. To accomplish this task the Atmosphere SIPS has deployed a six petabyte Ceph cluster employing numerous different components such as librados, EC-Pools, RBD, and CephFS. This talk will discuss choices we made to optimize the system allowing for rapid reprocessing of years of satellite data.
About Steve Dutcher
University of Wisconsin Space Science and Engineering Center
Data Scientist
Steve Dutcher is a Data Scientist working at the University of Wisconsin-Madison. He graduated in 2000 with a B.S in computer science and began his career at University of Wisconsin, Space Science & Engineering Center. In the early years he worked on airborne instruments flying aboard NASA aircraft such as the DC-8, ER-2, WB-57 and the Global Hawk UAV. The field experiments would fly under satellite overpasses and use the data for validation. He then moved on to working with satellite ground processing systems which involved receiving level-0 data from polar orbiting satellites and processing through level-2. Finally he transitioned to his current role working on a NASA contract as part of the VIIRS Atmosphere SIPS. The Atmosphere SIPS receives global data from multiple satellites and is responsible for producing the operational VIIRS Atmosphere products. This work entails reprocessing petabytes of data, collocating measurements with other satellites and validating results. All of that needs to be done in a rapid fashion in order to provide feedback to the science team aiding them in their development. This has drawn him to constantly be in search for new advancements in the world of high throughput computing in order to help advance Atmospheric Science
About Kevin Hrpcek
UW-Madison Space Science & Engineering Center
Systems Administrator
Kevin is a technology and science enthusiast who is a Systems Administrator for the Space Science and Engineering Center at the University of Wisconsin - Madison. He joined SSEC in 2015 at the beginning of the NASA Atmosphere SIPS contract. Working with a small team of developers he has helped design and build a high throughput computing system for satellite data with Ceph as a core component. This system specializes in the rapid reliable processing and storing petabytes of data for multiple satellites and sensors. The Atmosphere SIPS works closely with NASA funded science teams to aide in the development of algorithms that focus on cloud and aerosol properties. Kevin is also part of the data processing team for MIT Lincoln Laboratory's NASA funded TROPICS project which will consist of 6 cubesats. These projects have given Kevin the opportunity to play a small role in supporting the advancement of science and understanding of the world.
A
All
right,
everybody
hear
me
good,
looks
like
the
people
coming
in
and
slow
down
a
bit
so
we'll
get
started,
so
we're
going
to
talk
about
rapid
processing
of
NASA
satellite
data
and
basically
using
staff,
as
our
back-end
accomplish
all
this
I'm
Kevin.
This
is
Steve.
We
work
on
the
same
team
at
SSE
C
at
the
University
of
Wisconsin,
someone
to
start
off
with.
Nobody
really
knows
who
we
are
where
we're
from,
but
about
SSE
C
and
then
we'll
get
into
the.
What
we're
actually
doing
so.
A
The
uw-madison
Space,
Science
and
Engineering
Center,
founded
well
over
50
years
ago,
is
one
of
the
pioneering
centers
and
satellite
remote
sensing
technology,
so
imaging
the
space
from
Earth
through
low-earth
orbit
or
geostationary
satellites.
A
lot
of
the
original
hardware
on
some
of
the
first
stuff
up.
There
came
from
us,
so
the
Explorer
7
mission
spin
scan
camera
the
first
goals
to
have
light,
and
we
not
me
but
the
high
speed
photometer,
that
was
a
an
instrument
on
the
Hubble,
was
actually
developed
at
SDCC
other
than
that
pretty
much
everything
worked
on.
A
The
building
has
something
to
do
with
like
geosciences
or
atmospheric
research,
so
there's
data
validation
and
calibration
for
the
sensors
numerical
weather
prediction
modeling,
the
crop
co-operative
Institute
of
meteorological
satellite
studies
funded
by
NOAA,
is
located
there
and
there's
also
ice
core
drilling
which
at
first
may
not
sound
like
it,
has
to
do
with
that
mysterious
search.
But
you
can
look
at
the
depth
of
the
ice
and
see
details
of
what
that
I
am.
The
sphere
was
doing
at
a
time
to
go
back.
A
Our
main
goal
is
we
generate
level
2
and
level
3
NASA
products,
and
we
work
with
the
NASA
science
teams
to
get
their
algorithms
to
run
years
and
years
of
data
efficiently
and
in
a
timely
manner.
We
then
just
send
these
products
back
to
NASA
for
distribution
to
the
world
and
the
science
communities
the
primary
instrument
we
work.
Roofs
is
veers
right
now
there
are
two
of
them
in
orbit.
One
is
on
s
NPP,
which
has
been
up
there
since
2012
right,
yeah,
2012.
A
A
So
our
data
format
is
the
it's
time
based
into
six
minute,
become
granules.
So
it's
basically
what
the
satellite
has
seen
in
a
six
minute
chunk
of
time.
The
pretty
picture
up
here
is
from
obviously
in
Spain
from
last
week.
So
it's
about
a
thirty
two
hundred
kilometer
swath,
which
is
the
width
of
it
and
just
in
six
minutes,
is
actually
3,200
kilometers
in
length.
So
it
actually
comes
out
to
a
very
nice
timeframe
and
six
divides
nicely
into
a
day.
A
When
you
break
it
up,
we
get
orbit
files
from
NASA,
they
come
into
our
orbit
files.
We
break
them
up
and
see
six
minute
granules
and
within
each
of
these
files,
there's
a
data
for
22,
different
channels,
imagery
based
and
visible,
and
to
the
near-infrared
and
I
think
there's
one
infrared
band
in
there
as
well.
So
all
this
data
comes
in
and
we
have
lots
and
lots
of
objects.
So
this
is
almost
eighty
eight
thousand
objects
per
year
per
instrument
per
product,
there's
two
instruments,
and
probably
15
or
20
products,
at
least
for
every
granule.
A
That
comes
in
so
there's
many
millions
of
objects
per
year
of
this,
and
then
there's
also
different
versions.
When
tell
the
algorithm
developers
send
us
their
code,
so
you
know
rated
one
versus
version
1.2
and
you
have
to
have
both
at
the
same
time,
for
a
certain
amount
of
time.
What
I'm
getting
at
is
lots
of
data
all
the
time.
A
A
This
isn't
real
color.
This
is
showing
a
cloud
classification
also
kind
of
ice
versus
water,
vapor
clouds,
and
so
it
just
kind
of
slowly
paints
aboard
the
earth.
It
image,
isn't
the
entire
Earth
every
day
other
than
the
single
viewers
instrument.
We
actually
deal
with
a
lot
of
other
satellite
data
as
well,
because
you're
constantly
validating
calibrating
against
other
sensors.
A
The
big
one
is
the
MODIS
instrument
on
the
Terra
and
Aqua
satellites,
which
are
from
about
99
and
2000
2001
I
believe
so
those
records
are
15
years,
long
of
basically
non-stop
data
that
we
have
to
occasionally
process
and
keep
on
hand
at
all
times
for
comparisons,
because
the
kind
of
modus
is
seen
as
like
the
gold
standard.
It's
been
around,
it's
been
calibrated,
so
it's
really
really
like.
Okay
does
what
this
new
sensor
match?
What
modus
has
been
telling
us
and
these
records
can
be?
A
You
know
100
plus
terabytes
of
data
easily
other
project
we
work
on
is
a
tropic,
so
it's
a
collaboration
with
MIT
some
other
universities,
it's
funded
by
NASA
as
well.
This
one's
pretty
cool,
it's
just
kind
of
getting
off
the
ground.
They
are
not
actually
in
orbit
yeah,
it's
a
constellation
of
six
cube
sets
which
will
be
like
get
the
picture
going
there.
We
go
constantly
orbiting
in
a
30-degree.
A
Pro-Grade
orbit
around
the
Earth
tropics,
it's
a
tropical
region,
so
this
is
just
another
cool
thing
that
we'll
be
dealing
with
and
having
to
collect
data
for
six
different
cube
stats.
You
know
24/7
365,
doing
very
similar
work
for
the
steps,
so
even
more
objects,
even
more
data,
but
these
ones
do
actually
have
much
layer
data,
lower
data
volume
due
to
the
type
of
satellite
that
they
are
or
the
sensor
actually.
C
This
one
on
okay
yeah,
so
in
addition
to
the
poor
orbiting
satellites,
there
are
also
geostationary
satellites.
This
imagery
right
here
is
about
five
thousand
pixels
by
five
thousand
pixels
taken
every
10
minutes,
and
this
is
just
another
data
source
that,
like
more
and
more
geostationary,
satellites
are
going
up
and
the
time
frequency
of
the
measurements
is
going
up,
and
so
we
need
lots
more
storage,
space,
scalable.
C
So
when
we
were
just
deciding
like
you
know,
how
are
we
gonna
accomplish
all
this?
We
said.
Well,
what
are
storage
requirements?
Well,
we're
gonna
like
have
to
grow
our
cluster
by
a
petabyte
per
year,
and
we
need
good
data
replication
and
we're
gonna
have
hundreds
of
millions
of
objects,
something
that
works
really
well
for
our
high
throughput
computing,
because
that's
kind
of
what
we
do
needs
to
be
affordable
work
with
our
BMS.
C
C
So
this
is
just
a
rough
number
on
our
data
ingest,
we're
ingesting
600
gigabytes
per
day
in
producing
our
products
that
we're
delivering
to
NASA
were
producing
another
500
gigs
per
day.
Then
there's
another
700
gigs
of
other
products
being
created
per
day,
30,000
objects
per
day,
and
when
we
really
get
the
system
going
and
we're
producing
producing
products
and
looking
back
at
historical
records,
we
could
easily
be
doing
10
terabytes
a
day.
C
So
now,
I'm
gonna
just
kind
of
go
a
little
bit
through
how
data
flows
through
our
system,
so
Ness
is
collecting
all
this
data
for
us
and
we're
basically
sitting
there
listening
listing
files
on
NASA
server
and
saying
oh
there's
a
file
that
we're
interested
in.
We
use
rabbitmq
before
a
lot
of
work,
so
we'll
list
the
files
we'll
send
a
message
to
RabbitMQ
and
a
worker
will
pick
that
off
the
message
say:
hey
I'm,
gonna
download
this
file
and
he
just
goes
and
stores
it
in
our
stuff
cluster
we
I
should
mention.
C
C
It's
the
sensor
satellite
instrument
put
an
entry
in
our
database.
We
use
that
for
routine
check,
summing
of
miles
just
to
make
sure
Integrity's,
because
we
came
from
Gluster
we
get.
We
were
used
to
having
a
check
on
all
of
our
files.
Make
sure
that
they're
not
getting
corrupted
some
hasn't
really
done
that,
but
we
still
do
the
checks
anyways.
C
This
is
kind
of
an
interesting
little
tool.
We
wrote,
we
call
it
PG
track.
What
it
does
is.
It
says:
hey
database,
give
me
your
state
right
now,
so
it
takes
that
transaction
ID
and
that
it
says
what
has
happened
since
that
last
ran,
and
so
every
time
we
insert
a
file
into
the
database,
rabbit
PG
track
will
take
that
make
us
up.
Urls
send
that
off
to
work
use,
and
we
have
numerous
downstream
products.
Listening
to
that.
C
This
is
another
interesting
tool
that
we
developed
for
in-house.
We
call
it
rado
sleight
it's
a
Python
tool,
it
pretty
much
all
of
our
tools
use
access
it.
So
one
of
the
things
it
does
is
it
just
provides,
and
so
we
have
kind
of
a
sepia
RL
here
which
has
Poole
named
arch,
we're
using
namespaces
and
file
names,
and
so
it's
kind
of
a
nice
little
object
that
we
just
pass
around
in
our
rabbit
messages
like
hey.
This
object
go
get
this
and
do
whatever
operation
you
want
to
do
on
it.
A
So,
like
you
said,
we
mostly
use
liberators
and
wave
not
way
that
one.
Maybe
two
years
ago
we
were
migrating
from
k4
and
one
pool
to
a
k6m
two
pool,
and
it
turned
out
that
I
think
it's
by
coincidence
that
the
the
pool
alignment,
because
we're
doing
everything
to
liberate
us,
we
kind
of
actually
have
to
worry
about
that.
The
pool
alignment
how
it
was
calculated
is
changed
between
the
original
pools
that
this
cluster
was
built
with
way
back
on
my
camera
and
then
when
the
new
pool
was
created,
probably
on
Luminess
I.
A
Believe
so
like
we
made
this
new
pool
and
we
actually
couldn't
write
a
single
object
into
it
or
like
it,
only
wrote
up
to
the
like
a
first
4k
or
something
of
the
object.
So
we
actually
ended
up
figuring
that
one
out
and
pushing
a
patch
upstream
for
the
Python
bindings
other
than
that
using
straight
web
rails
and
Python
for
everything.
That's
worth
really
well
for
us.
Yeah.
C
So
our
rate
of
slight,
you
know
we're
demonstrating
at
stand
here,
but
you
know,
get
put
and
get
these
bytes
out
of
this
file,
which
leads
me
to
my
next
slide.
So
we
developed
this
application
called
PDS
serve.
So
our
data,
our
bra
level,
zero
data
comes
in
five
gigabyte
files,
and
we
because
this
is
like
the
official
level
zero
product
we
we
don't
want
to
really
chop
it
up
into
small
chunks
and
refactor
it.
So
what
we
do
is
we,
when
the
first
things
we
do
when
a
pile
comes
in?
C
Is
we
index
where
every
one
minute
block
of
that
data
starts,
so
that
when
our
application
goes
to
download
the
file,
it
can
just
go
grab
those
bytes?
What
we
had
seen
with
this
is
you
know
you
get
a
file
and
because,
in
that
two-hour
block,
there's
there's
20
different
six-minute
chunks
for
our
six
minute
granules
and
when
all
20
compute
nodes
grabbed
that
exact
same
pile.
The
the
time
to
download
that
really
went
up
and
we're
like
shouldn't,
take
that
long,
but
it
was
sort
of
kind
of
cereal.
C
So
so
the
first
thing
we
do
want
to
get
those
files
we
index
it.
We
throw
them
to
in
a
relational
database
and
now,
when
our
code
runs
on
our
compute
nodes,
it
can
it
calls
our
rate.
Osler
function,
says
rate,
oh
slight,
yet
from
this
byte
to
this
byte
downloads,
just
that
chunk
of
data,
it
needs
added
a
lot
of
fishin,
see
yeah.
A
So
it
might
be
worth
mentioning
quick,
it
seems
I'm
talking
to
people
yesterday
and
whatnot.
We
seem
to
be
a
very
unique
outlier
and
actually
using
libera
DOS,
everybody
seems
to
be
using
Safa
fast
or
array
dos
gateway,
even
in
the
scientific
community.
So
we're
brave
or
I,
don't
know
what
idiots
yeah.
C
No
I
would
agree
with
that
statement.
Abed
libera,
if
you
haven't,
checked
all
the
burritos
I
personally,
like
initially
when
we
started
out.
We
just
call
shell
out
ray
dos,
get
this
file
to
the
local
node,
but
since
we're
a
Python
shop
and
everything
we
do
is
I
thought
why
not
just
have
used
labret
dose
and
get
that
file
directly
and
that's
worked
out
really
well
for
us.
C
The
other
part
of
our
system
is
we
rely
on
kubernetes
heavily
for
routine
services
like
that
PDS
serve.
That
was
just
talking
about
for
us.
A
a
container.
Often
time
consists
of
sitting
there
listening
for
work,
you
picking
a
message
off
of
that
queue,
pulling
that
by
all
to
SEF,
storing
some
metadata
back
in
a
relational
database,
whether
it
be
hey
this,
this
granule
has
this
geolocation.
C
I
said
we
primarily
do
Rados,
but
there
are.
There
are
times
where
we
want
to
have
where
we
need
POSIX,
and
so
for
this
we
use
some
of
us.
So
this
is
an
instance
of
NASA
world
view.
Our
scientists
use
it
to
look
at
their
products
and
this
demo
is
going
to
go
through
its
overlaying
like
deep,
blue
aerosol
oceans
on
this,
and
they
can
scan
through
their
data.
C
C
I'd
be
remiss
if
I
didn't
mention
HT
Condor,
because
it
was
developed
at
uw-madison.
We
use
HTC
Condor
for
all
of
our
computing
and
I
like
to
say.
Hd
Condor
process
is
driving
science
forward,
and
by
that
I
mean
we.
Our
primary
goal
is
to
support
our
science
team.
They
are
developing
algorithms,
like
that
deep
blue,
aerosol
or
those
cloud
retrievals,
and
in
order
to
do
that-
and
you
know
so-
they'll
look
at
a
grand
you'll
be
like
oh
there's.
C
Tell
today,
so
we
accomplished
this
by
running
a
cluster,
so
they'll
submit
their
version
will
send
a
note
to
our
cluster
reprocess
years
of
data,
boil
it
down
to
a
simple
level,
three
file
that
they
can
quickly
open
in
in
MATLAB
or
whatever,
and
see
the
impact
of
their
algorithm.
What
got
better,
what
kind
of
works
our
hte
Condor
cluster
consists
of
roughly
a
hundred
nodes,
2,400
cores
it
all,
has
one
gigabit
networking
a
lot
of
RAM
per
Corp
just
because
of
the
processing
needs,
but
then
we
have
our
SEF
cluster,
which
is
48
OS.
C
C
So
this
slide
is
just
a
few
of
our
graphs
for
showing
like
at
the
bottom
right.
Here
is
just
heavy
level
3
processing,
it's
a
lot
of
just
pulled
on
this
granule
grab
the
grab,
the
data
that
the
scientist
is
interested
in
grid
it
up
and
put
it
in
level
3
file
and
that's
kind
of
the
product.
They're
going
to
look
at.
A
Top
ones,
just
this-
it's
good
to
mention
these
testing
after
you
come
out
of
it
came
out
a
single
chat
that
we
use
with
staff,
but
since
these
ones
would
take
and
I
found
me
screenshots
for
a
while
ago,
like
we
started
using
the
staff
manager,
Prometheus
exporter,
it's
awesome.
I
definitely
recommend
it
to
people.
A
If
you
go
through
the
stuff,
github
there's
somewhere,
you
a
folder
called
Grif
fauna,
and
they
have
a
bunch
of
dashboards
in
there
that
you
just
export
and
import
directly
into
a
griffon
instance,
and
once
you
point
your
soft
cluster
to
it,
it
kind
of
mostly
worked
magically.
You
had
to
reload
tweaking
here
and
there
yeah.
C
C
Every
scientist
wants
to
see
their
data
process
on
the
whole
record,
but
really
what
they
want
is
yeah
I
want
to
see
my
new
algorithm
processed
over
just
this
one
little
site
here
and
just
give
me
every
granule
and
can
I
have
that
in
the
next
hour,
and
if
we
can
fill
their
needs
of
giving
this
data
back
to
them
and
now
we're
later
while
the
algorithm
is
still
in
their
head
and
their
changes
are
fresh
in
their
mind.
They
can
actually
like
in
my
mind
it
helps
them,
do
science
and
do
stuff
better.
C
C
C
They
said
you
could,
but
so
what
we
did
is
found
a
way
to
glue
that
all
together
and
make
it
useful,
because
we're
putting
all
of
our
objects
in
Raynaud's
and
our
science
team
doesn't
really
know
what
an
object
stories
we
wanted
a
way
to
provide
them
POSIX
access
to
all
these
files,
so
we
developed
this
fuse
layer
which
has
a
stuff
out.
So
the
stuff
out
is
shown
right
here
and
all
it
says
is
hey.
This
file
is
at
this
archive.
C
C
You
have
a
and
they're
actually
reading
data
straight
off
of
labret
DOS
and
they
have
no
clue
they're
doing
it
so
I,
don't
know
that
I
would
recommend
like
back
in
the
day,
we
were
planning
to
make
this
a
tool
that,
like
other
people,
the
stuff
community,
would
use
with
the
improvements
to
set
off
s.
I'm,
not
sure
that's
needed
anymore,
but
it
was
an
interesting
tool
that
we
made
use
of
and.
A
A
You
cannot
just
point
it
at
a
pool
and
it
does
an
LS
on
it
and
for
our
main
pool
it
takes
10
hours
to
list
the
you
know
almost
90
million
objects,
but
then
it
just
stores
that,
with
some
metadata
and
I
think
that
yeah,
the
picture
shows
a
basic
metadata
of
the
object
in
the
cluster
namespace
pool
size,
basically
what
you
get
back
from
a
radial
staff.
So
it's
constantly
running
in
the
background
on
some
random
server
and
it
just
constantly
updates
the
metadata.
A
It
keeps
running
information
in
there
and
we
it's
kind
of
important
to
have
this
completely
separate
from
everything
else,
because
all
of
our
other
databases
like
every
file
and
staff
is
in
at
least
three
databases,
but
most
of
the
other
ones,
except
this
one
I
linked
through
that
PG
trap
tool.
So
if
you
delete
it
from
you
know
our
ingest
database
that
actually
gets
deleted
from
the
main
database
as
well
and
oops.
A
Growing
pains
we've
been
started
along
not
too
long
ago
for
four
years
ago,
2015,
maybe
coming
from
the
Gloucester
world,
not
much
CPU
and
memory
was
really
used
first
server,
so
our
original
servers
were
terribly
under
sized
the
cluster.
We
have
now
grew
from
six
or
Steve
servers
up
to
the
48.
There
currently
is,
you
know,
like
we
said,
adding
about
a
petabyte
a
year,
or
so
we
had
a
few
full
cluster
outages,
unfortunately,
which
has
led
us
into
some
different
design
choices.
A
Even
recently,
the
mobile
ones
are
a
few
years
ago,
I
did
a
small
reweighed
by
utilization
on
a
Friday
afternoon
and
the
whole
cluster.
Just
panicked
and
everything
started
getting
out
of
sync.
Everything
was
starting
and
crashing.
The
OS
teams
were
flap
flapping
for
hours
and
just
some
seemed
like
a
matter
of
the
Mons
we're
having
problems,
keeping
up
with
the
changes
that
the
concept
was
making
in
the
OSD
map,
and
that
is
a
case
where,
just
you
know
you
sent
all
the
know
out
flakes.
A
So
all
the
changes
stopped
waited
half
a
day,
maybe
a
full
day,
and
you
know
eventually
the
cluster
got
healthy
and
you
carefully
removed
those
no
off
changes
to
put
it
back
in
operation
and
hope
it
didn't
freak
out
again,
the
other
one
which
was
way
worse,
actually
was
no
September
last
year
or
so
was
mid
mimic
upgrade,
which
made
it
even
funner,
because
I
couldn't
figure
out
if
it's
the
upgrade
that
caused
it
or
something
else.
I
wouldn't
worry
too
much
about
upgrading
the
mimic
from
Luminos
I'm,
pretty
positive.
It
wasn't.
A
That
is
more
of
the
fact
of
the
scale,
my
cluster
to
know
it
just
keeps
growing
and
growing,
and
at
the
time
we've
800,
OS,
beads
or
whatever
and
six
petabytes,
and
what
ended
up
happening
is
really
similar.
The
first
one,
no
everything
was
flapping
crashing
and
just
the
cluster
was
just
thrashing
itself.
Updating
the
OSD
maps
super
frequently
the
Mons
couldn't
keep
up
and
what
it
seemed
like
is.
A
The
Mons
were
just
spinning
on
their
I'm
reading
the
OSD
map
cuz
it,
and
then
they
would
time
out
on
the
leases,
so
they
would
have
to
constantly
reelecting
a
quorum.
So
it's
just
constantly
doing
that
and
that
Matt
and
what
what
really
ended
up
getting
a
goring
was
increasing
the
the
least
time
probably
helped,
and
some
very
insane
aggressive
caching
settings
on
the
Mons
to
get
them
to
stop
having
to
read
the
OSD
map
off
of
disk
every
time
they
reelect
at
a
forum,
a
big
thanks
to
the
suffuses.
A
A
A
So
for
scaling,
the
lesson
learned
here
is
pay
attention
to
some
of
your
settings.
Once
you
hit
a
certain
spot,
they
you
need
to
be
careful,
so
our
architecture,
as
I
mentioned
we've
been
growing
constantly.
Over
the
last
five
years
we
got
four
or
six
eight,
ten,
twelve
and
terabyte
Nearline
sass
and
sometime
KAS.
We
have
two
clusters
and
due
to
this
last
big
outage
problem,
we
used
to
have
this
large
arc
cluster
archive
everything.
A
Was
there
we
decided
to
split
off
a
small
five
node
cluster,
so
all
of
our
processing
happens
on
that
one
and
all
the
ingest
and
then
after
30
days
it
all
kind
of
just
moves
to
the
other
one
through
some
scripting
that
we
have,
but
mostly
a
dull
shop.
24
OST
is
an
old,
usually
and
20
gigabit
networking
we
don't
have
any
SSDs
or
government
NASA
contracts.
A
So
this
one's
pretty
simple,
we're
just
running
a
basic
algorithm
on
a
granule.
Usually
we
pull
the
object
into
dev,
SH
M
and
it's
two
different
methods.
One
is
directly
ones
using
the
PDS
circuit
to
get
that
chunk
of
data
and
setteth
us
was
actually
faster
than
copying
into
dev,
sh
m
and
that's
during
the
reading
and
writing
directly
office
ffs,
but
that's
only
a
single
granule.
So
what
happens
when
you
scale
up
so
then
you
do
100
granules
48
at
a
time
on
a
server
and
south
of
s
is
still
doing
great.
A
A
Besides
the
outlier,
we
kind
of
discovered
there's
a
problem
with
our
PDS
method
of
getting
data
that
we
have
to
figure
out,
but
once
again
stuff.
If
s
is
comparable
as
well
and
that's
an
entire
cluster,
the
source
data
was
about
350,
400
megabytes
and
the
output
data
is
about
a
gigabyte
and
that's
1200
cores
doing
this.
A
A
Luckily,
or
maybe
not
luckily,
but
just
a
few
days
ago,
I
had
a
server
fail,
so
I
decided
to
throw
and
the
rebalance
graph
just
because
this
was
our
one
of
our
oldest
for
a
terabyte
hard
drive
nodes.
So
it's
only
about
forty
four
terabytes
of
raw
disk
and
within,
and
you
can
see
for
hours.
Basically,
the
entire
cluster
recovered
and
peaked
at
around
twenty
eight
gigabytes,
a
second
during
that
recovery
and
they
didn't
actually
hit
too
much
CPU
load
to
handle.
That
future
would
like
to
move
the
blue
store
someday.
A
A
One
other
notable
thing
is
our
big
cluster
is
stuck
on
the
bob-tailed
tunable.
Actually,
it
was
built
with
hammer
so
I,
don't
know
why
it's
not
using
later
tunable
z'.
Maybe
there
was
a
bug
and
what
enables
was
set
on
that
it
should
have
been
using
Firefly
at
at
least
and
on
a
test
cluster
when
I
moved
that
up
to
the
latest
optimal
tunable.
A
Obviously,
everything
moved
I
am
slightly
scared
to
do
that
on
the
seven
petabyte
cluster,
but
from
the
certain
presentation
yesterday
they
were
talking
about
that
cool
up
map
feature
to
use
it
and
utilize
it
to
slow
down
the
tunable
's.
When
you
make
changes
like
that,
so
the
cluster
doesn't
go
into
a
complete
frenzy
of
remapping
everything,
so
that
might
actually
be
I
path
forward
for
that
technologies
used
a
a
whole
lot
of
open
source,
open
source
communities.
Awesome.
Thank
you
to
everybody
that
contributes
there
and
maybe
one
or
two
questions.