►
Description
MeerKAT Astronomy Data Store Deployment and Operations - Martin Slabber, SARAO
MeerKAT, inaugurated on the 13’th of July 2018 under the auspices of SARAO, is a radio telescope consisting of 64 antennas which has been built in the Northern Cape, South Africa.
In this talk, Martin will describe the hardware, software stack, deployment and operation tools used on the MeerKAT data archive.
MeerKAT currently has two Ceph clusters, one on-site at the telescope in the arid Karoo and the second larger cluster in Cape Town.
The Cape Town cluster consists out of 2640 hard drives and 55 NVMe devices. The Karoo cluster consists out 240 hard drives and 480 solid state disks.
The clusters were built by a small team of engineers and as much as possible of the deployment and operation has been automated.
About Martin Slabber
Engineer/DevOps, SARAO
Martin Slabber is an Electronics Engineer turned Software Engineer turned DevOps for the Science Data processing team at SARAO. SARAO recently completed the construction of MeerKAT the largest radio telescope in the world.
A
Good
morning
everybody,
my
name
is
Martin
slower
I'm
from
answer
al
or
SK
in
South
Africa
and
we're
busy
building
a
radio
telescope
called
meerkat
telescope.
I'm
gonna
tell
you
a
bit
more
about
how
we
use
safe
in
this
telescope,
but
I
thought
it
would
be
interesting
to
to
look
back
into
kind
of
the
history
of
how
astronomy
is
used.
Data
storage.
It
turned
out
to
be
a
very,
very
interesting
topic
and
there
I
would
say
like
a
topic
for
a
book
one
day,
and
it
actually
caused
me
to
spend
more
time
in
this
slide.
A
Then
I
really
should
have
and
him's
the
first
part
of
the
slides
are
like
nice
pictures
and
in
the
the
quality
drops
quickly
because
I
had
to
do
it
at
two
o'clock
last
night,
I'll
give
you
a
little
bit
of
an
overview
of
our
safe
clusters.
We
have,
we
always
know
we
have
multiple,
safe,
glasses,
I'll,
just
focus
on
those
that
that's
in
production.
A
Then
we've
done
a
few
things
special
with
our
hardware
and
I'll
talk
through
those
and
at
the
end.
I
just
want
to
give
a
few
screenshots
of
all
of
the
tools
we
use.
So
astronomy
has
always
been
one
of
those
things
that
humans
have
been
absolutely
fascinated
about.
Unfortunately,
we've
always
tended
to
record
data
for
that.
Astronomy
has
always
been
like
a
heavy
user
of
data.
A
Recording
I
still
want
to
work
out
the
exact
bytes
required
for
each
of
these
observations
and
compare
that
to
what
we
need
nowadays,
but
a
good
example
is
of
17,000
years
ago-
and
this
is
just
500
kilometers
north
of
here
and
in
front
where
they
were
already
observing
stars
and
logging,
the
positions
of
stars
by
cave
paintings.
They
got
a
little
bit
more
advanced.
This
is
an
example
of
in
California
and
cheedo
Canyon,
where
you
use
petroglyphs,
where
they
basically
carve
out
astronomy
symbols.
A
This
was
a
huge
advancement,
I
would
say,
like
the
invention
of
SSDs,
nearly
where
we
could
make
our
own
clay
tablets
and
the
astronomers
have
been
punching
their
little
holes.
This
is
from
the
Babylonian
times.
What's
interesting
thing
here
is
the
360
de
year
was
already
established
and
the
twelve
zodiac
symbols
was
already
recorded
here.
This
specific
tablet
is
about
6,000
BC,
but
it's
obviously
been
once
much
much
earlier.
A
Here's
the
first
example
of
storing
astronomy
data
on
spinning
Rast.
It
was
very
popular.
This
one
is
actually
from
close
bias
in
Germany
and
no
net
side
and
sage
mentioned
user
interface,
so
I
want
to
give
him
some
encouragement.
That
is
a
calendar.
I
would
not
call
that
a
very
good
user
interface,
one
of
the
first
uses
of
astronomy
data,
recording
that
actually
became
very
useful
in
modern
times,
was
from
the
Chinese
recording
data
onto
soap.
A
This
particular
one
is
of
comment
and,
for
example,
Halley's
comments,
trajectory
and
cycle
needed
such
a
long
period
of
time.
We
need
its
data
over
such
a
long
period
of
time
to
accurately
calculate
it
that
this
piece
of
silk
was
actually
used
to
work
out.
How
Halley's
Comet
actually
moves
it
gets
a
bit
better.
A
A
If
you
read
the
history
now
and
I'm,
no
historian,
the
politics
starts
to
play
here
and
it
becomes
really
interesting
stories.
Very
famous
one
was
the
the
rural
fine
journals.
These
were
developed
by
Kepler
and
again
it
was.
It
was
still
paper,
an
interesting
alternative.
Medium
name
came
in
and
this
was
photographic
plates.
A
A
Each
one
of
these
is
very
interesting.
Here
is
a
I
would
say
a
kind
of
an
expected
outcome.
A
sponge
cards
became
very
popular,
then
at
some
stage
for
VLBI
the
very
long
baseline
interferometry
people
turned
to
very
interesting
and
alternative
ways
of
storing
data.
Vhf
types
was
actually
used,
but
the
types
weren't
used
in
the
typical
digital
fashion,
as
you
know
now,
where
we
stored,
but
it
was
like
literally
analog
data
stored
on
two
types.
A
Things
do
go
wrong,
so,
even
nowadays,
when
things
go
wrong
in
your
cluster,
just
think
back
I
think
we've
got
much
better
recovery
tools,
then
pasting.
This
rolls
of
type
together,
then
jump
back
into
paper.
I
think
this
needs
no
explanation,
but
this
is
the
WoW
signal
that
basically
was
the
7th
7th
second
signal
from
the
city
project.
The
search
for
extraterrestrial
intelligent.
A
I'll,
stop
there
the
history
lesson
over
just
brief
intro
to
history
right
there.
This
just
brings
us
to
where
we
are
right
now,
at
the
moment,
type
and
disk
with
discuss
a
whole
region
of
SSDs
as
well
and
then
and
then
of
course.
The
reason
we
here
is
the
future
of
data
storage
in
astronomy
is
definitely
safe.
It
served
us
well,
we
requite
keen
we
very
young
team,
as
in
like
very
new
to
safe,
so
we've
still
got
a
lot
to
learn.
A
A
A
A
section
of
South
Africa
is
a
semi
desert
area
with
a
very,
very
low
population
and
because
of
the
low
population,
there's
very
little
interference
from
human
from
humans
and
our
activities
like
telephones
and
computers
and
cars
and
microwaves,
and
so
this
was
an
ideal
place
for
us
to
build
a
radio
telescope.
It's
also
fairly
flat
ish,
so
it
helps
a
little
bit,
but
mostly
it
was
was
the
lack
of
humans
in
that
area.
A
A
A
A
It'll
spent
stretch
art
quite
high
up
in
Africa,
so
it's
quite
a
challenging
project,
and
so
it
was
decided
from
the
South
African
point
of
view
that
we
are
gonna,
try
and
bolt
some
smaller
telesco
and
work
our
way
up
to
this
64
dis
telescope.
So
what
we
did
was
not
going
work
Oh,
another
history
lesson,
but
the
previous
one
we
built
was
cat.
7
and
the
cat
stands
for
kuru
right
telescope,
and
then
we
got
to
me
a
cat.
A
We
decided
to
just
call
it
mirror
of
cat
7,
so
more
of
cat
Simon
yeah,
now
the
size,
the
longest
baselines
is
at
the
moment
8
kilometers.
So
this
is
the
farthest
two
dishes
from
one
another,
and
then
70%
of
the
dishes
are
all
within
a
core
of
one
kilometer,
so
really
tightly
packed
in
the
middle
lots
of
dishes
and
in
Dallas
are
spread
out.
A
So
no
the
previous
line
died.
There
was
a
number
there
about
37
gigabits
per
second
coming
from
each
of
these
dishes.
There's
64
of
them,
so
it's
a
huge
volume
of
data
that
we
receive
per
second.
Fortunately,
we
don't
have
to
store.
All
of
that.
What
happens
is
in
astronomy.
You
you
throw
away
a
lot
and
then
you
a
breach
more
and
so
what
we
have
is
we've
got
correlator
and
this.
A
This
slide
itself
is
a
very
interesting
and
fun
talk,
but
the
correlator
allows
us
to
reduce
the
data
to
a
very
manageable
data
science.
At
the
moment,
we
doing
at
the
peak
will
be
something
like
24
28
or
something
terabytes
for
8
for
an
8
hour
observation,
so
yeah
I
think
it's
about
3
terabytes,
a
few
terabytes
three
terabytes
a
second
to
something.
So
it's
very
manageable
data
rates
that
we
have
at
the
moment.
As
our
higher
resolution
systems
come
on
law
online,
we
will
have
some
more
some
more
challenging.
A
A
Just
to
give
you
an
introduction
or
an
idea
of
of
the
data
flow
we
use
and
kind
of
we're
safe,
fits
in
for
us.
So,
like
I,
said
the
data
flows
from
each
of
these
dishes
into
our
correlator
and
out
of
the
correlative.
It
comes
a
scientific,
worthy
stream
of
data.
Before
then,
it's
it's
basically
signals
and
it
needs
to
be
correlated
and
faulted.
A
What
we
didn't
do
is
on
this
side.
We
have
our
fatty
Nakuru
and
on
that
side
we
have
the
site
in
Capetown
and
in
between
is
our
transfer
processing
out
of
a
correlated
goes
into
the
science
data
processing
subsystem
and
assigned
data
processing
does
a
lot
of
the
calibration
the
images
and
some
early
stage
imaging,
and
these
results
are
stored
into
a
safe
cluster.
A
We
also
have
a
small
compute
cluster
of
our
own
for
our
internal
science
teams
and
what
comes
out
is
the
best
known
picture
of
the
center
of
our
of
our
galaxy
and
that's
at
the
moment,
our
claim
to
fame
for
producing
that
it's
an
enormous
picture
by
the
way,
that's
just
a
little
snip
snippet
of
it.
So
yeah
all
of
the
green
lines
that
I'd
put
in
there
as
our
interfaces
to
sip
and
from
from
a
science
point
of
view,
we
do
all
of
that
with
radio
Skateway.
A
A
Our
transfer,
demon
or
transfer
processes
is
all
custom
in-house
software
that
we've
developed
to
fulfill
our
needs,
and
that
is
to
move
our
data
from
the
site
to
the
career
and
the
reason
we've
done
it.
Ourselves
is
because
we've
got
certain
tokens
that
we
will
look
at
in
the
data
and
decide
when
to
move
which
observation
we
also
because
the
site
is
quite
far
away
from
Cape
Town
and
it's
it's
got
kind
of
an
unreliable
fiber
link.
We
get
a
few
breaks
a
year.
A
Our
transfer
demon
knows
what
observations
to
move
to
Cape,
Town,
first
and
and
what's
needed
and
fills
out
policies
that
way
our
big
cluster
in
Cape
Town
is
fortunately,
very
simple.
It's
just
a
big
cluster
using
hard
drives
and
I'll
show
you
some
slides,
more
I'll
put
in,
especially
towards
the
end
a
lot
of
these
screenshots
about
our
user
interface,
because
it
ends
up
this.
One
particularly,
is
what
the
users
see.
Is
this
and
I
don't
really
care
about
how
many
sweat
and
tears
goes
into
your
safe
cluster?
A
Bit
more
about
our
team,
we're
quite
a
small
team.
Our
whole
safe
team
is
here
at
the
moment
if
he
sits
in
front
here
and
it's
Thomas,
and
we
recently
had
who
have
shown.
February
join
us
myself,
I'm
in
the
infrastructure
concept,
team
infrastructure
person,
so
every
now
and
then
I
get
the
opportunity
to
play
with
Seth,
but
most
of
the
time
I
deal
with
the
hardware
and
networking
stuff
and
also
leaning
up
I
must
tell
I'm
no
historian
or
an
astronomer.
A
So
if
you
have
like
really
tough
questions
of
on
those
two
out,
you
can
ask
me,
but
I'll
have
to
come
back
to
you
so
earlier.
I
said:
I
want
to
tell
you
about
the
hard
way,
because
that's
kind
of
a
very
important
part
about
our
story,
and
we
had
to
make
a
few
very
crucial
decisions
and
one
of
them
that
we
did
was
we
decided
to
develop
our
own
and
storage
pods.
It
was
a
lot
of
options
on
the
market
and
at
the
moment
this
is
even
better
and
more
selection.
A
A
Each
of
our
brands
was
not
a
lot
of
dollars,
so
we
had
to
like
really
assess
what
our
value
proposition
is
within
our
budget,
and
it
turned
out
for
us
that
if
we
would
take
your
typical
storage
appliance-
and
we
will
just
carve
away
everything
we
didn't
need,
we
ended
up
with
a
very
basic
piece
of
hardware.
We
tailored
this
piece
of
hardware
specifically
to
our
safe
needs.
Okay,
so
it's
probably
not
the
best
for
your
general
storage
I.
Don't
think
this
would
be
like
your
OpenStack
solution
of
of
note,
but
for
our
storage
requirements.
A
Here's
a
photo
of
it.
You
can
see
how
our
typical
safety,
where
we
have
in
our
data
center,
so
we
we
teamed
up
with
a
company
and
in
Cape
Town
called
parallax.
They
specialize
in
building
or
providing
solutions
to
the
military
field,
so
they're
really
really
good.
At
discussing
the
problem
with
you
and
working
through
all
of
the
solutions,
one
little
quirk
that
we
we
had
at
that
stage
was
we
wanted
these
storage
spot
to
have
a
low
RFI.
A
No
RFI
is
radio
frequency
interference
and
because
we've
decided
to
go
so
far
away
from
civilization
to
build
the
telescope
to
avoid
RFI.
We
didn't
want
to
now
take
our
storage
pods
up
into
the
Karoo,
put
them
close
to
the
telescope
and
then
basically
cause
RFI.
So
we
have
a
very,
very
particular
data
center
there.
It's
all
metal
cladded
with
like
cool
space-age
doors,
but
every
component
you
take
there.
You
want
to
be
really
sure
that
it
doesn't
add
noise,
so
the
storage
pods
besides
having
good
airflow
and
performance
or
specific
performance
for
us.
A
It
also
has
a
very
low
RFI
footprint.
Okay,
just
the
I
mean
you
can
probably
read
the
stats,
but
I'm
just
gonna
give
it
an
idea.
We've
got
48
three
and
a
half
inch
drives
at
the
moment.
We
use
the
a
terabyte
and
Seagate
drives.
We
have
one
nvme
in
each
of
our
boxes.
We
do
we
have
a
boot
drive,
we've
decided
to
make
that
boot
drive
an
SSD.
A
Fortunately,
for
me,
a
lot
of
these
stumbling
blocks
arrived
just
before.
I
joined
the
storage
team,
I
I
joined
kind
of
in
the
same
time
as
these
boxes,
starting
to
arrive.
So
it's
been
like
a
Christmas
ride.
For
me
all
the
time.
One
of
the
things
that
happened
is,
as
these
boxes
were
about
to
go
into
production.
The
red.
A
Then
I
show
here
and
we
had
a
safe
cluster
up
and
running
and
it
all
looks
good
and
at
some
stage
it
started
flapping.
No,
his
things
just
started
flapping
and
things
went
horribly
wrong
and,
being
very
inexperienced
was
safe.
We
tried
very
hard
to
figure
it
out,
but,
like
the
guys
from
switched
mentioned
earlier,
we
we
find
up
the
the
guys
from
CERN
and
Dan,
helped
us
and
quickly
pointed
out
that
the
the
memory
on
these
boxes
were
way
to
love.
That
was
okay.
That
was
fortunately
easier
than
the
RAID
controller.
A
So
we
could
easily
procure
memory
and
an
app
that
more
recently
and
I
must
actually
state
right
now,
a
large
portion
of
our
clusters
is
down
and
working
for
us
to
to
get
home
it's
more
recently.
We
chose
a
very
bad.
This
is
the
because
you
know
we
were
trying
to
save
money
and
we
read
the
forums
and
got
the
best.
We
thought
would
work,
but
so
yeah.
A
So
recently
we
just
decided
to
do
to
buy
much
better
OS
drives
and
just
before
we
came
here,
we
basically
went
out
and
replaced
all
of
their
OS
drives
and
so
we'll
get
a
bit
better,
not
performance,
but
yeah,
less
breakage.
Out
of
our
class,
though
we
have
some
other
nodes
that
form
part
of
our
storage
cluster,
and
these
are.
A
These
boxes
are
mostly
there
for
processing
purposes,
but
we
also
let
them
serve
as
our
storage
nodes.
So
each
of
these
boxes
have
4gb
use
undertrained
a
keeps
Fram
two
biggest
CPUs
and
twenty
one
terabyte
SSDs
knows
SSDs.
We
all
pull
into
our
safe
cluster,
but
these
ones
are
the
specific
mod
boxes
are
all
on-site,
we've
got
24
of
them
and
we
use
them
to
do
the
Astronomy
processing
and
then
right
back
to
the
SSDs.
Another
process
picks
up.
This
is
all
run
on
a
mesos
cluster
with
very
custom
framework.
A
A
We've,
since
the
the
main
cluster
in
the
crew
is
called
Chaika,
a
maquette
we've
dubbed
the
one
in
Cape
Town,
which
is
close
to
the
ocean
with
dub
dead
and
siaka
on-site
we've
got
about
at
that
site.
It's
got
about
20
petabytes.
That
would
be
the
role
capacity
of
storage
capabilities.
That's
comprised
out
of
55
of
these
storage
boxes.
That
gives
us
nearly
3,000
OS
these
at
that
site.
So
the
previous
speakers
talk
is
going
to
be
very
relevant
for
us.
A
A
We've
played
around
with
very
many
networking
options
and
had
a
couple
of
suppliers
bring
us
units
to
taste
with
and
at
the
end
we
went
for
the
Mellanox
switches.
We
found
them
to
work
very
well.
We've
also
opted
for
twenty
five
gigabit
into
each
of
our
boxes.
We
did
not
split
our
safe
network
from
our
production
network
for
reasons
to
keep
it
just
simple.
We
initially
planned
to
have
210
gig
networks,
and
it
just
worked
out
to
be
more
cost-effective
for
us
to
have
a
single
25,
gigabit
interface.
A
A
A
We
do
it
from
from
bare
metal
up
to
applications
gets
rolled
out,
we
use
ansible
very
heavily
and
we
use
the
safe
ansible
project
to
roll
out
our
safe
and
for
hardware
install
we
use
mass,
which
is
the
canonical
product
and
it's
basically
metal
as
a
service.
It's
a
very
cool
name,
but
it's
actually
super
simple
to
use
and
it
allows
us
to
install
our
OS
on
our
boxes
from
here
and
I'm.
Just
going
to
show
you
a
couple
of
screenshot
of
the
tools
we
use.
A
So
this
is
mass
and,
like
I
just
said,
mass
allows
us
to
easily
redeploy.
We've
had
a
few
hiccups
with
with
Marcin
and
if
everybody
anybody
else
considers
using
mass
for
force
F
one
of
the
the
biggest
gotchas
for
us
was
that
mass
really
wants
to
take
control
of
the
hard
way
it.
Basically,
the
hardware
is
under
its
control
and
you
really
have
to
go
and
wrangle
your
hard
way
back.
A
If
you
want
to
do
some
funny
things
now,
if
safe,
even
if
you
reinstall
the
the
OS,
the
OSD
still
contains
all
of
the
required
metadata
for
you
to
reconstruct
your
node
after
a
node,
so
our
first
rollout
of
Mars.
What
happened
is
that
when
you
reinstall
the
OS
Mars
will
actually
go
and
wipe
each
of
the
OSD
drives.
It
would
do
some
security
stuff
and
just
clean
the
drives,
and
we
couldn't
recover
this
OS
DS,
so
that
I
mean
safe,
recovers
and
everything,
but
it
kind
of
wastes
a
lot
of
time.
A
So
what
we
realized
that
it
was
very
simple
you
just
before
you
install
the
OS,
you
have
to
tell
Mass
like
not
to
manage
each
Drive
that
took
a
a
lot
of
clicking
with
the
first
few
nodes
to
test
it,
but
eventually
we're
scripted.
All
of
that,
and
since
that
it's
yeah
it's
work
very
well
for
us,
so
we
used
mas,
basically
just
to
manage,
get
the
OS
on
the
box,
and
thereafter
we
use
ansible
to
do
all
of
the
other
provisioning.
So
we
don't
at
all.
A
Mars
has
capabilities
to
install
software
and
and
loan
certain
processes.
We
don't
use
that
safe
ansible,
sometimes
a
hate
relationship,
but
most
mostly
a
loved
one.
It's
worked
well
for
us.
We
do
not
deploy
in
a
container
environment.
I
must
admit
we
are
considering
it
very
strongly,
but
at
this
stage
what
we
do
is
we
again
like
we
do.
If
our
astronomy
software,
we
just
try
and
get
as
close
to
the
hardware
and
OS
level
deployments
safe,
ansible
supports
that
without
any
hiccups.
A
We've
had
a
few
problems
here
and
there,
but
it's
mostly
been
resolved
quite
easily
one
of
the
things
that
we
we
do
struggle
with
safe
ansible
that
I
might
just
mention,
is
when
we
restore
we,
we've
picked
a
version
of
safe
that
we
want
to
restore
install
on
all
of
our
hardware
and
if
safe
moves
forward,
safe,
ansible
also
moves
forward,
and
we
found
it
difficult
to
install
our
aloha
version
and
we've
developed
a
few
workarounds
around
that
as
a
team.
We
always
like
to
have
insight
and
see
what's
what's
going
in
a
perfect
world.
A
This
is
this
is
how
we
would
like
to
see.
Our
full
infrastructure
would
like
to
gather
a
host
matrix.
There's
a
project
called
net
data.
We
were
quite
fond
of
it.
It
does
everything
we
need
to
do
and
SIF
manager
gives
us
a
lot
of
metrics
and
information.
We
gather
all
of
that
into
one
of
our
favorites
is
Prometheus.
A
We
visualize
with
crow
fauna.
We
use
the
alert
manager
and
Prometheus
just
just
enough
of
it
to
kind
of
generate
some
alerts
for
us.
There's
a
product
called
alerta'
that
gathers
everything
and
give
us
a
bit
of
a
dashboard.
I've
got
a
slide
on
each
of
these
as
quickly
and
then
we
push
it
out
into
email.
There's
some
people
who
hate
the
reading
email
like
myself,
who
wants
it
on
the
IRC
and
we
have
happy
admins
but,
as
we
all
know,
that's
not
the
case.
A
Reality
is
kind
of
more
like
press
and
we're
working
very
hard
to
get
back
to
the
previous
slide,
but
as
we
as
we
remove
one
of
the
links,
this
is
always
another
one
popping
up
anything
here,
yeah,
so
I
didn't
mention
before
we
use
elastic
stack
a
lot
of
the
time
more.
As
like
bubblegum
to
stick
holes.
We
do
a
lot
of
scripting
with
with
a
different
hardware.
A
There
is
this
so
many
ways
of
interfacing
with
them.
I
forgot
to
mention
that
we
use
Dell
hardware
for
our
months
and
mgrs,
so
we
have
quite
a
what
a
selection
of
hardware
and
then
all
of
our
other
applications.
Also
we've
kind
of
unfortunately
got
a
bit
of
everything
there.
So
we've
got
quite
a
lot
of
different
hardware
platforms
that
we
to
work
within
and
there
the
consistency
is
very
poor.
A
And
so
now,
I'm
gonna
I'm,
nearly
at
the
end
I'm
just
gonna
race
through
some
slides,
to
tell
you
about
the
products
we
use.
So
this
is
net
data.
Quite
recently,
I
mean
a
few
months
ago
shown
of
myself
found
it
and
we
just
kind
of
installed
it
on
to
our
our
local
nodes
when
we
started
kind
of
enjoying
it.
A
We,
where
scientists
come
to
us
and
telling
us
about
like
these
applications,
doc,
it's
disk
utilization
is
high
and
you
know
in
ease
faster
disks,
and
we
were
like
what
and
to
install
this
onto
that
processing
node
and
gave
him
the
interface
and
very
quickly.
He
could
like
come
back
to
us
and
tell
us
that
you
know
it's
something
different.
That
was
the
problem,
so
it's
nice
and
it
also
exports
there's
Prometheus
metrics,
which
is
very
useful
for
us.
Like
so
many
people
we
use
graph
are
now
in
our
environment.
A
A
But
the
idea
is
that
there's
no
hot
aisle
or
cold
aisle,
it's
just
one
big
mix
of
storage,
and
so
as
we
packed
our
storage
nerdy
and
it
changes
the
airflow
and
and
basically
everything
just
gets
very
messy.
So
we
had
the
meal
to
know
the
exact
temperature
of
each
not
the
need,
I.
Think
it's
more
like
an
itch.
We
had
that
we
had
to
scratch.
A
A
It's
just
so
many
of
them
that
you
can
adjust
at
some
stage
ignore
too
many
of
your
emails,
and
you
never
know
what's
important.
So
a
little
dashboard
has
actually
become
very
useful
and
then
just
the
like
a
last
mention
is
we
use
something
called
nade
marks
to
data
center
information
management
system
to
help
us
a
huge
amount
to
keep
track
of
all
of
these
nodes
and
kind
of
where
they
all
sit
and
just
basically
physical
location.
A
A
We
surf
are
quite
a
small
community
there,
but
as
these
things
goes,
I
mentioned
it
here,
because
you
know
somebody
else,
and
somebody
else
know
something
else,
and
maybe
the
word
gets
out
there,
that
there
is
a
safe
community
and
kept
in
South
Africa.
We've
had
two
meetups
in
Cape
Town.
So
far
it
was
a
handful
of
two
handfuls
of
people
there,
but
it's
all
good
at
the
moment.
It's
all
research
related.