►
Description
Testing Ceph for the Cloud, in the Cloud - Adam Wolfe Gordon, DigitalOcean
DigitalOcean, a public cloud provider, has been using Ceph to offer block storage and S3-compatible object storage for nearly three years. For most of that time DigitalOcean has used community releases of Ceph, not finding need to make any modifications. However, as the company's storage infrastructure has scaled and user workloads have changed, the DigitalOcean storage team realized they would need to start modifying and contributing to Ceph. One challenge in doing this was the need to test Ceph changes easily and efficiently, preferably without relying on external environments.
In this talk, Adam Wolfe Gordon will discuss how DigitalOcean automated configuration of cloud-based test environments for Ceph's integration testing tool, Teuthology, and challenges faced in doing so. He will demonstrate how to easily set up a Ceph "lab" on DigitalOcean using Terraform and Ansible automation.
About Adam Wolfe Gordon
DigitalOcean
Sr. Software Engineer
Adam Wolfe Gordon is a software engineer at DigitalOcean, currently working on managed Kubernetes. He previously worked on block storage at DigitalOcean and EMC, implementing everything from user-facing storage management APIs for the cloud to the i/o-path for distributed storage systems, and occasionally contributing to Ceph. Adam has previously spoken at international distributed systems and Go programming language conferences, as well as numerous meetups and company-internal venues. He likes building elegant microservices, continuous deployment, and occasional forays into lower-level software.
A
All
right,
I
guess
we'll
get
started.
Can
everyone
hear
me
sounds
good
all
right,
so
I'm
Adam,
Wolfe,
Gordon
I
work
at
digitalocean
these
days,
I
actually
work
on
our
eyes.
Kubernetes
team
I
work
on
our
managed
kubernetes
product,
so
I'll,
be
here
all
week
for
kube
con
before
I
started
working
on
the
communities
team.
Earlier
this
year,
I
worked
on
our
block
storage
team
at
digitalocean,
doing
storage
stuff
and
before
that,
I
worked
at
EMC.
A
A
Those
are
qmu
KVM,
based
block
storage,
which
you
can
attach
to
virtual
machines,
which
is
set
based,
also
s3,
compatible
object,
storage,
set
based
and
then
higher-level
offerings
like
our
managed
kubernetes
manage
databases
manage
load
balancers
that
are
built
on
top
of
those
primitives
and
all
the
network
stuff
that
goes
along
with
that
firewalls
dns
voting
ip's.
All
that
good
stuff,
like
I
mentioned
all
our
storage
products
are
based
on
Seth
we've
had
seven
production.
A
For
about
three
years
we
launched
block
storage
in
mid-2016,
which
was
shortly
after
I
started
to
the
company
and
about
a
year
later,
a
little.
Over
a
year
we
launched
our
s3
compatible
object,
storage,
it's
called
spaces.
We
launched
that
in
the
late
2017,
so
we
have
a
fairly
big
staff
deployment.
At
this
point
we
have
21
clusters
around
the
world
and
nine
different
data
centers
it's
about
40
petabytes
of
raw
capacity.
In
total,
those
numbers
might
be
a
little
bit
off.
A
When
we
started
using
stuff
and
and
actually
still
to
this
day,
we've
been
using
community
builds
of
staff
community
releases,
we're
not
haven't
needed
to
build
our
own
stuff.
We've
really
had
very
few
problems.
It
works
really
really
well
for
us,
we've
been
very
happy
with
sort
of
the
quality
of
staff
and
the
feature
set
that
it
offers,
and
for
the
first
two
years
or
so,
we
really
didn't
feel
any
need
to
develop
staff
itself.
A
A
As
we
scaled,
though,
we
did
start
to
feel
like
we
needed
to
work
on
staff
and
build
some
staff
development
capability
in
house,
in
particular
for
our
block
storage
product.
Since
we
don't
control
the
workloads
that
run
on
it,
those
are
customer
workloads.
We
really
wanted
to
have
more
quality
of
service
knobs,
be
able
to
prevent
a
customer
with
a
heavy
workload
from
saturating.
The
cluster
have
a
bit
more
control
and
offer
users
a
more
consistent
performance
experience.
So
that
was
one
thing
we
wanted
to
build.
A
On
the
object
storage
side,
we
started
hitting
some
scale
limitations
of
rgw,
some
bugs
in
our
GW
as
we
launched
and
scale
their
object
storage
offering.
So
we
realized
that
we
were
going
to
need
to
do
some
staff
development
work
and
really
bring
some
of
that
capability
to
our
storage
team
that
deal,
and
you
know
we
really
wanted
to
solve
our
own
problems.
A
That
meant
we
needed
to
do
a
few
new
things
that
we
hadn't
been
doing
before.
For
one
thing
we
needed
to
build
stuff
ourselves.
That
was
pretty
easy.
You
spin
up
a
build
VM
we
run
make
it's
really
pretty
easy
to
build
set
packages
I.
We
also
need
to
be
able
to
run
those
custom
builds
and
our
staging
clusters
are
in
production
again.
That
was
pretty
easy.
We
already
had
our
own
package
repositories
that
we
were
pulling
community
packages
into
sort
of
putting
our
own
packages.
There
was.
It
was
really
simple.
A
What
this
talk
is
about
is
how
we
set
ourselves
up
to
run
tests
on
staff
on
our
infrastructure,
so
I'm
guessing
a
lot
of
the
people
who
decided
to
come
to
this
talk
already
know
something
about
the
testing
tools.
There
are
for
Stephanie
how
you
test
stuff,
but
I'll
go
over
a
really
brief
overview
anyway,
I
first
of
all,
there's
a
framework
called
tooth
ology
and
that's
the
integration
testing
framework
for
Seth.
A
It's
used
to
build
the
integration
test,
Suites,
which
are
in
the
separate
history
and
technology,
can
do
a
whole
bunch
of
things
that
can
configure
SEF
clusters
on
a
set
of
nodes.
I've
run
IO
in
various
ways:
inject
some
faults,
I
all
that
kinds
of
things
you've
wanted
a
storage,
a
testing
framework
to
do
and
the
usual
place
where
technology
tests
get
run
is
the
sepia
lab,
and
it
see
Sage
mentioned
the
sepia
lab
in
his
keynote
this
morning.
A
At
digitalocean,
we
didn't
want
to
rely
entirely
on
the
CPA
lab
for
testing,
and
this
is
for
a
number
of
reasons.
For
one
thing,
not
everyone
has
access
to
it.
There's
a
process
you
go
through
when
you
are
becoming
a
second
tributary
to
get
access
to
the
CPA
lab.
We
didn't
want
people
at
digitalocean
to
necessarily
have
to
do
that
right
away.
We
wanted
people
to
be
able
to
make
changes
to
SEF
and
run
to
ecology
tests
on
them
right
away.
I
want
an
our
own
infrastructure.
A
The
CPA
labs
also
a
shared
resource,
so
there's
some
contention
for
it.
Your
tests
aren't
necessarily
being
able
to
run
right
away.
If
there's
lots
of
tests
running
that
felt
like
something
we
want
to
work
around
it.
In
particular,
we
wanted
to
have
a
CI
system
that
would
start
tests
in
a
lab
environment
automatically,
run
integration,
tests
on
PO
requests
or
on
branches
and
doing
that
on
a
shared
resource
didn't
feel
like
a
friendly
thing
to
do
to
the
community.
A
So
we
wanted
to
build
our
own
for
that
purpose
and
at
the
end
of
the
day
we
are
the
cloud
we
have.
Tens
of
thousands
of
servers
that
we
can
spin
up
VMs
on
all
the
time
we
didn't
feel
like
we
needed
to
take
up
a
community
resource
for
testing
our
own
changes
when
we
could
avoid
it,
there's
certain
kinds
of
testing,
maybe
that
we'd
still
want
to
do
in
the
CPA
lab
or
elsewhere,
but
for
the
most
part
we
felt
like
we
could
run
at
least
correctness
tests.
A
So
I
have
a
fairly
simple
plan.
My
plan
was
to
follow.
G
needs
nodes
to
test
on
at
the
end
of
the
day.
That's
that's.
Where
and
fastest
on
notes
and
different
tests
declare
how
many
nodes
they
need
for
their
tests
and
how
those
most
should
be
configured
so
I
felt
like
I
should
be
able
to
write
a
wrapper
for
tooth
ology,
or
maybe
a
module
for
to
follow
G.
That
would
provision
VMs
on
demand.
A
So
when
a
test
says
it
needs
three
nodes,
it
would
spin
up
three
VMs,
provide
them
to
technology
for
testing
and
then
tear
down
those
BM.
So
I
was
done
right.
This
is
a
very
cloud-like
thing
to
do
to
theology
had
a
module
called
for
they
used
Lib
cloud
IDE
to
run
tests
on
OpenStack,
so
I
live
cloud
also
supports
digitalocean
seems
like
it
should
be
relatively
easy
to
adapt
it
to
use
our
resources
instead
of
OpenStack
reverses
it
turns
out.
A
First
of
all,
to
follow,
G
is
very
intertwined
with
system
called
paddles
and
paddles
is
a
node
reservation
system
and
also
a
result
repository
for
tests
when
trichology
goes
to
run
a
test.
D
requests
nodes
from
paddles
and
paddles
tells
it
which
nodes
it's
canoes
and
then
it
walks
those
nodes
and
paddles
to
the
other
tests.
A
Kaizen,
you
can
manually
specify
nodes
to
to
use,
but
it's
still
gonna
reach
out
to
paddles
and
try
and
lock
them
that
didn't
really
work
for
the
case
of
ephemeral,
spinning
of
an
ephemeral
set
of
VMs
to
use
as
your
test
nodes,
it's
kind
of
a
mismatch,
and
it
also
turned
out
that
the
lib
cloud
module
wasn't
quite
what
I
was
hoping
for
it
did.
It
was
very
OpenStack,
specific
I
and
it
really
spun
up
an
entire
lab
environment
rather
than
sort
of
an
ephemeral
set
of
test.
Vms
I
didn't
just
didn't
quite
work.
A
The
way
I
expected
it
would
and
also
the
lib
cloud
module
didn't,
seem
to
be
very
actively
maintained
when
I
asked
about
it
on
IRC.
The
answer
I
got
was,
basically
it
sometimes
works.
Sometimes,
doesn't
it's
not
sort
of
the
most
tested
part
of
Technology,
so
I
was
a
little
bit
concerned
about
us,
relying
on
it
for
our
CI,
so
I
had
a
new
plan
which
was
to
play
SF
lab
internally
on
digital
ocean
droplets,
dissolution
teams,
the
nice
thing
about
doing
this
is
I.
A
So
the
pieces
of
ACEF
lab
I
at
the
end
of
the
day
are
the
paddles
server
which,
like
I
mentioned,
does
the
the
node
reservation
debacle
keeps
track
of
codes
and
also
records
test
results?
There's
also
a
web
UI
that
goes
along
with
paddles
capito
that
can
run
on
the
same
node
and
it
needs
a
database
to
back
Appl
spread
out
of
the
same
node.
A
You
don't
have
to
do
it
from
there,
but
can
the
third
component
is
the
testimonies
are
the
actual
machines
that
pathology
is
going
to
install
stuff
on
form
them
into
sort
of
an
ad
hoc
SEF
cluster
run
the
tests
and
then
I
return
them
to
the
pool
as
it's
just
nodes
at
ease.
So
these
can
be
pretty
much
arbitrary
machines
or
VMs.
They
just
need
some
disks,
because
these
disks.
A
So,
having
figured
out
sort
of
what
the
machines
were,
that
I
needed,
it
was
pretty
straightforward
to
automate
setting
up
a
set
lab
using
digital
ocean
droplets.
I
wrote
some
terraform
code
to
actually
create
the
different
resources.
The
droplets
and
the
disks
and
I
wrote
some
ansible
code
to
do
the
configuration
that
terraform
stuff
calls
out
to
ansible
to
do
that.
Configuration
so,
at
the
end
of
the
day,
I
have
a
single
command.
A
I
can
run
that
spins
up
a
fully
functional
lab
for
testing
stuff,
and
most
of
the
rest
of
this
talk
is
going
to
be
about
the
sort
of
road
blocks
that
I
ran
into
in
building
this
automation.
There
were
definitely
some
things
that
I
had
to
work
around
to
come
up
with
a
fully
functional
test
lab
in
an
automated
way,
I
on
vm's,
but
before
I
talk
about
those
things,
we're
gonna
attempt
a
live
demo.
A
I'm
gonna
show
how
to
use
the
automation
that
I
built
to
spin
up
a
test
lab
and
we'll
see
if
it
can
run
tests
by
the
end
of
this
time.
So
the
URL
to
the
source
code
first
of
all,
is
on
the
slide
there.
So
if
anyone
wants
to
check
that
out,
please
do
give
it
a
try
and
let
us
know
what
your
experience
is.
A
All
right
so
here
I
have
a
pretty
much
clean
check
out
of
the
digitalocean
staff
lab
repository.
You
can
see
it
as
a
bunch
of
terraform
files
of
the
dot,
t--
F
files,
and
it
has
a
directory
called
ansible,
which
holds
all
of
the
ants
of
all.
Please
and
rules
that
we're
going
to
use
to
configure
assemble
I,
won't
sort
of
dig
into
this
too
much
because
you
can
go
and
read
it
yourself,
but
it's
it's
all
just
kind
of
terrify
me
at
school
to
get
started.
A
What
we
need
to
do
is
create
a
TFRs
file
that
declares
some
parameters
for
our
lab,
so
we
give
the
lab
a
name.
This
name
is
going
to
be
used
in
the
node
names,
so
you
can
see
in
your
digital
control
panel
which
nodes
belong
to
which
lab
setup.
We
also
have
we
tell
it
which
region
we
want
to
run
in
the
the
digital
ocean
data
center.
It's
gonna
run.
A
It
I've
put
it
in
the
Amsterdam
because
we're
in
Europe
we
tell
it
which
image
to
use
for
the
test
nodes,
so
we're
gonna
use
1
2
1804
today,
and
we
tell
it
how
many
test
nodes
we
want.
I've
said
three,
and
if
we
look
at
our
VARs
file,
we
can
see
there's
a
few
other
things
you
can
customize
like
how
many
volumes
each
test
node
should
have
helping.
A
They
should
be,
and
there's
also,
these
SSH
keys
that
it
needs
and
I've
just
put
those
in
separate
files
panel,
because
they're
long
and
boring
to
look
at,
but
with
all
of
with
that
VARs
file
created,
we
just
run
a
single
command.
This
terraform
applied
command
which
I'm
writing
the
under
docker,
but
you
don't
have
to
we'll
hit
enter
and
we'll
see
it's
going
to
tell
me
which
resources
it's
gonna
create.
A
If
we
look
here,
it's
gonna
create
some
droplet,
some
test
nodes,
a
paddles,
node,
a
head
node,
and
it's
going
to
create
a
bunch
of
volumes
that
will
get
attached
to
those
test.
Nodes
and
I
will
say:
yes,
please
start
creating
them
and
we'll
see
it
start
creating
we'll
watch
it
just
for
a
minute
until
its
eye.
A
Well,
we
ran
it
to
it:
I
I'm,
using
the
same
SSH
key
twice,
so
it
ran
into
a
problem,
but
normally
it
would
continue
with
spinning
up
all
the
nodes.
I
have
a
pre
created
one
that
I
used
I
created
just
in
case.
Something
went
wrong
with
the
demo,
so
we'll
use
that
later
to
actually
run
some
tests,
but
that
is
the
single
command
to
just
be
not
by
a
classic
test.
Cluster
idea
and
I'll
go
back
to
the
slides
now.
A
A
A
So
my
kind
of
hacky
solution
was
to
install
DNS
mask
on
one
of
the
nodes,
I
just
all
into
paddles
mode,
and
we
use
that
as
an
internal
DNS
server
within
the
test
lab
environment.
So
all
the
other
all
the
test
modes.
They
had.
No,
they
all
get
configured
to
use
the
paddle
server
as
their
DNS
server
and
then
I
just
stick.
All
the
other
hosts
IPS
in
the
Etsy
hosts
file
and
DNS
mask
will
resolve
addresses
from
the
hosts
file.
When
it's
asked
for
resolution,
this
didn't
mean
in
our
ansible
playbooks.
A
A
You
know,
try
it
on
one
image
and
it
works,
and
then
another
image
it
doesn't
so
there's
a
little
bit
of
figuring
out
to
do
there,
but
at
the
end
of
the
day
it
works
pretty
well,
and
this
is
something
that,
if
we
add
additional
distributions
right
now,
this
automation
basically
only
supports
the
bun.
If
we
add
additional
ones,
we'll
have
to
add
more
conditionals
there
to
sort
of
work
around
the
differences
between
this
region's.
A
This
is
create
I,
think
in
the
CPA
lab.
This
is
created
by
the
provisioning
stuff
when
they're
provisioning,
a
new
node,
so
I
just
needed
to
create
it
from
our
our
automation.
This
was
a
relatively
easy,
but
it
well.
It
was
very
easy.
It
just
listed
the
disks
Wow.
It
just
took
us
a
bit
of
debugging.
It
was
a
little
bit
mysterious,
so
I
my
tests,
weren't
working-
and
this
is
what
it
turned
out-
was.
A
A
A
One
of
the
things
that
this
automation
does
is
try
and
reconfigure
the
networking
on
the
nodes,
and
that
doesn't
work.
If
your
nodes
are
in
the
cloud
in
particular,
it
was
trying
to
use
DHCP
to
get
addresses
and
dissolution
droplets,
don't
necessarily
use
DHCP
to
get
addresses.
They
have
they're
just
as
pre-configured
by
clouding
it
so
I.
A
Basically,
the
the
network
interfaces
on
these
droplets
would
end
up
without
addresses
could
get
to
many
more
I
was
able
to
blacklist
the
these
rules.
Are
the
ansible
rules
that
it
was
running
from
except
CI,
manciple
I
told
you
already
supported
that
extra,
so
that
was
convenient.
I
just
had
to
figure
out
exactly
which
ones
were
causing
problems
and
worked
to
blacklist
them.
So
these
black
lists
now
live
in
an
overrides
file
that
we
have
on
our
head
node
and
we
just
have
to
remember,
run
a
running
tests
to
always
include
the
overrides
file.
A
Otherwise
the
nodes
might
go
away
as
I
mentioned
earlier.
We're
running
I
are
we're
running
tests
largely
from
a
CI
environment,
so
remembering
to
include
that
overrides
file
is
easy.
If
we
were
running
them
manually,
then
it
would
just
be
something
that
everyone
has
to
sort
of
know
before
they
start
running
tests.
A
Another
related
problem
we
ran
into
is
our
automation.
With
our
tests
were
all
working
fine
for
a
long
time
and
then
all
of
a
sudden
they
started
failing.
It
turned
out
that
something
in
added
just
yet
step
see
I'm
ansible
that
would
try
and
I
basically
clear
off
disks
before
before
our
staff
started
using
them.
I
think
this
probably
happened
as
part
of
the
booster
work.
So
there
wasn't
leftover
stuff
on
the
disks
I.
A
This
play
was
trying
to
touch
a
config
drive
on
the
droplets,
because
it
wasn't
using
the
scratch
devs
file
to
figure
out
which
disks
to
use
and
I
when
I
wasn't
able
to
touch
that
the
test
would
fail.
So
we
started
just
having
this
sort
of
mysterious
failure.
Again.
The
solution
was
to
blacklist
that
roll.
It's
just
as
s
FCM
ansible
progresses.
We
have
to
just
keep
an
eye
on
what
it's
doing
so
that
we
know
which
additional
goals
we're
gonna
have
to
block
those
because
they
won't
work
there.
A
A
G
I
makes
a
lot
of
assumptions
about
where
set
packages
live
on
the
internet
like
where
it's
gonna
pull
him
in
from
and
I
it
tries
to
look
for
packages
in
a
system
called
Shaymin,
which
is
basically
the
source
of
truth
for
all
of
the
upstream
set.
Builds
that
happen
as
part
of
the
upstream
sets.
Yeah
we're
doing,
builds
internally
I,
and
we
have
our
own
internal
app
depositories
where
the
packages
end
up.
So
those
builds
aren't
in
Shaymin.
That
was
kind
of
the
goal
of
this
whole
exercise.
A
A
Related
to
that
one
tooth
ology
makes
some
assumptions
about
the
paths
that
packages
will
be
at
for
a
build.
When
you
pointed
at
an
apt
repository,
it
doesn't
just
find
the
find
them
by
version
it
finds
them
by
any
of
the
get
shell
1,
the
version
number
or
the
branch
name,
and
it
actually
will
use
all
of
these
location
methods
for
packages
within
the
same
run.
A
So
we
have
to
have
all
of
those
paths
available
for
every
build
that
wasn't
something
that
we
had
thought
of
when
we
were
figuring
out
our
build
process,
but
it
was
easy
enough
to
add
it
to
it.
We
just
have
to
create
a
bunch
of
symlinks
so
that
all
those
packages
are
available
at
the
locations
that
she
fell
and
she's
looking
for
them,
I
kind
of
expected.
A
So
those
commits
weren't
in
the
upstream
repository
they
weren't
D,
not
publicly
available
here
plus
tree,
so
I
had
to
actually
modify
truth
ology
to
just
not
run
this
check,
it
doesn't
actually
use
any
of
the
data
it
gets
back
when
it
does
this
check,
it's
just
checking
it
as
like
a
sanity
check
for.
Does
this
commit
exist
later,
it's
gonna
go
and
check
out
the
check
out
that
commit
using
git
and
not
partridge
fine.
So
we
just
had
to
add
a
flag
where
tell
told
you
not
to
double-check.
A
A
And
I'll
become
the
to
Salah
G
user
and
we
can
see
that
the
automation
automatically
checked
out
both
SEF
and
tooth
ology,
so
that
all
the
sources
available
in
the
places
where
technology
expected
OB
and
it's
also
automatically
set
up
a
Python
virtual
end,
so
that
I
can
run
all
of
the
technology
tools
like
that.
For
example,
and
if
I
run
a
suite
I
take
out
the
dry
run
flag,
we
should
see
it
schedules
and
jobs.
A
A
A
A
This
one
job
when
q3
running
at
3
waiting
and
those
have
decided
to
use,
not
I,
went
to
a
chino
forest
they're
kind
of
stuck,
but
if
we
had
a
jobs
that
we're
using
the
victoria
to
know
for
they'd
be
able
to
run.
This
is
something
I
was
trying
to
figure
out.
The
other
day
was
why
it
wasn't
filtering
properly
because
I
had
it
filter
properly
before
but
yeah.
This
is.
This
is
what
it
spun
up,
so
it
spun
up
the
entire
system.
A
It
is
usable
for
tests
and
one
piece
of
future
work
if
anyone's
looking
to
contribute
to
our
automation
would
be
being
able
to
spin
up
nodes
of
all
the
different
distributions
they
have
them
in
the
same
lab,
rather
than
just
a
single
one.
That's
for
our
purposes.
Internally.
We've
mostly
been
testing
on
a
single
distribution
at
a
time,
but
it
would
be
nice
to
be
able
to
do
more
than
one
alright.
We
can
go
back
to
slides
to
to
finish
up
here.
A
A
This
is
a
bit
of
a
barrier,
a
barrier
to
entry
for
new
stuff
developers.
If
you
want
to
run
integration
tests,
you
need
to
either
get
access
to
CPA,
lab
or
I
figure
out
all
of
these
challenges
to
make
it
run
in
a
different
environment.
So
that's
something
that
hopefully,
this
automation
that
we've
built
and
open
sourced
will
help
with
is.
If
people
want
to.
You
know
really
quickly
get
started
with
running
tests
on
stuff.
They
can
spin
up
an
environment
under
solution
and
just
start
running
their
tests.
A
Hopefully
this
is
something
that
the
community
will
find
useful,
especially
as
try
and
grow
the
contributor
base
for
Steph.
This
should
make
it
easier
for
people
to
jump
her
in
and
start
running
tests
and
start
writing
tests
and
start
making
changes
and
writing
tests
for
them,
which
is
ultimately
the
goal
like
I
said
it.
It
is
available.
I've
got
the
the
URL
there
on
the
slide
as
well
as
well
as
my
email
address.
If
you
want
to
get
in
touch
with
me,
please
do
try
it
out.
A
We
would
love
to
have
people
using
this
and
have
this
something
that
do
can
contribute
to
the
Ceph
community,
since
we
really
have
got
a
lot
of
value
out
of
Ceph.
Hopefully
this
is
something
that
will
help
the
community
in
general
start
to
run
more
tests,
bring
new
contributors
on
and
that's
all
I
have
I've
got
some
time
for
questions
and
I'm
also
around
all
week.
If
you
want
to
chat
with
me,
you
can
find
me
at
our
booth
out
here
or
our
booth
at
Keuka.
Thanks.
A
Yeah,
so
we
so
the
the
automation
that
spins
up
the
lab
environment
clones
it
and
and
does
that
we
don't
we
don't
do
that
on
every
CI
right.
We
we
have
a
single.
We
have
a
lab
that
we
just
keep
around
the
nice
thing
about
spaying
it
up
with
this
automation.
Is
that
if
I
need
to
replace
a
node,
because
something
weird
goes
wrong
with
it,
for
example,
you
know
like
the
the
problems
I
mentioned,
where
we
had
nodes
randomly
going
away.
I
can
use
terraform
to
just
taint
the
node
and
then
recreate
it
yeah.
A
A
Yeah,
so
I
have
contributed
a
little
bit.
I
contributed
the
thing
to
have
it
not
check
get
shows.
But
yes,
it
is.
It
is
something
we've
thought
about.
I'm
I
personally
am
NOT
on
the
storage
team
anymore.
So
I'm
not
touching
it
these
days,
but
I
think
as
we
you
know,
grow
our
our
Sefton
elana
capability
at
digitalocean.
That's
definitely
something
that
whole
look
at
yeah.
I.
Think
our
our
goal
is
definitely
to
make
it
better
and
make
it
easier
to
run
and
make
it
easy
to
run
in
different
laces.
A
Yes,
we
do
have
developers
making
staff
changes.
We
why
I
have
made
some
upstream
staff
changes
in
the
RBD
year
area,
but
we
also
have
people
working
actively
in
our
GW
and
some
made
our
BD
as
well.
We
we
haven't
had
a
ton
of
contributions
so
far
about
where
we're
gaining
velocity
there,
so
there'll
be
more
soon.