►
Description
Testing Ceph: Status, Development, & Opportunities - Gregory Farnum, Red Hat
Over ten years of deployments, Ceph has proven itself resilient to failures of all kinds. Much of this success can be traced to its “teuthology” automated testing system, which runs thousands of machine-hours of tests every day. This talk will describe the current status of the system and recent developments to improve its shortcomings on technical and community levels. We will also look at Ceph testing more broadly (from PR checks to the wider ecosystem) and identify opportunities to contribute in this high-impact part of the project.
About Gregory Farnum
Principal Software Engineer, Red Hat
Greg Farnum has been in the core Ceph development group since 2009. Now a Red Hat employee, Greg has done major work on all components of the Ceph ecosystem, and currently focuses on testing and the core RADOS system.
A
Alright,
alright,
well
my
mind,
will
watch
it's
time
to
start
and
I
just
lost
one
person
already,
so
we're
gonna
get
going
hi
I'm
so
excited
to
be
here
at
our
second
day
of
cephalic
on
my
name
is
Greg
and
we're
gonna
talk
about
testing
Steph
gonna
go
over
some
of
the
history
of
the
Ceph
testing
frameworks
and
look
at
what
developments
were
working
on
and
what
opportunities
there
are
for.
Other
people
like
you
to
contribute
to
the
testing
the
set
product
project,
so
in
SEF
I'm
sorry,
I
killed
my
killed.
A
One
Who
am
I
slide,
so
I'm
Greg
I've
been
working
on
SEF
since
June
1st
2009
or
there
abouts
so
I'm
just
coming
up
on
10
years,
which
is
really
exciting
and
very
strange
for
me.
I
currently
am
working
on
the
Rados
team
and
focused
on
and
doing
some
some
shepherding
of
testing
efforts
throughout
the
project
in
the
past.
I've
worked
all
over
the
place.
It's
been
a
long
time
said
anything
in
rtw
RBD,
but
back
long
ago,
I
have
I
was
the
sem
best
tech
lead
for
several
years
up
until
I.
A
Think
2015,
although
I
don't
remember
exactly
and
I'm
really
excited
to
be
here,
so
the
the
biggest
test
SEF
testing
system
we
have
is
called
to
thaw
logic.
It's
the
academic
study
of
cephalopods
and
the
guy
who
named
it
thought
it
was
fun
and
the
early
design
of
psychology
started
about
eight
years
ago.
I
think
we
needed
to
formalize
step
testing.
A
It
was
really
important
because
we
kept,
we
were
running
a
lot
of
like
Linux
FS
tests
on
our
local
development
boxes
and
seeing
if
we
passed
but
I
remember
at
least
one
time
there
was
this
bug
that
we
found
and
I
don't
know
exactly
what
it
was,
but
something
went
wrong
and
we
traced,
through
the
source
code,
to
figure
out
why
we
found
an
if
statement.
We
said,
oh,
that
if
statement
is
obviously
backwards,
so
he
swapped
it
and
the
bug
went
away
and
then
and
I
said,
and
it
was
fixed.
A
A
We
were
not
real
sophisticated
at
the
time
we
had
some
people
involved
in
the
project
who
tried
to
build
up
good
systems.
There
wasn't
really
anything
available
for
testing
distributed
systems
at
the
time,
at
least
not
that
was
accessible
to
us
on
our
budget.
As
an
open-source
project,
we
tried
using
auto
test,
which
is
involved
in
the
Linux
kernel,
a
lot,
because
a
lot
of
our
early
people
were
familiar
with
those
processes,
but
it
just
it
didn't
didn't
work
very
well
for
something
in
the
nature
of
staff.
A
A
So
a
guy
named
TV,
who
was
much
better
at
these
sorts
of
things
than
I,
am
came
up
with
a
plan.
He
wrote
this
thing
called
the
orchestra
module
in
Python.
It's
basically
just
wrapping
wrapping
SSH
up
in
a
lot
of
pretty
things
and
allows
you
to
have
a
central's
test,
Runner
and
and
and
an
SSH
out
to
those
nodes
and
run
tests
against
them
tooth
ology
itself
started
as
a
test
runner.
It
wasn't
a
big
framework
which,
like
it
is
now
in
which
we'll
get
into
it,
was
initially
just
a
test.
A
A
A
This
was
an
old
example
of
the
actual
targets
we
were
using
of
a
rolls
mapping
and
of
some
of
the
first
tasks
that
he
actually
wrote.
The
Seth
one
I
think
at
the
time
did
the
install
as
well
as
turning
on
a
cluster
K
client
would
run
a
kernel
client
on
a
kernel
client
on
all
of
those
on
all
of
the
name.
Clients
in
that
yeah.
A
All
of
the
roles
named
in
that
in
that
list,
and
then
work
unit
was
a
name
we
have
for
just
shell
scripts
that
run
tests
and
this
one,
and
so
we
have
a
task
that
runs
the
shell
scripts
on
whatever
we
name
tastin
tooth
ology.
Alright
can
actually
be
context
managers,
so
they're,
not
just
a
one-off
thing.
The
Ceph
tasks
like
turns
on
a
set
cluster
and
then
yields
to
the
next
to
the
next
task
in
the
system.
Until
it
comes
back
and
then
we
come
back
to
the
Ceph
task
it
cleans
up.
A
This
is
convenient
for
doing
things
like
turning
on
an
OSD
thrasher
that
runs
in
the
background,
while
you're
running
other
tests
that
may
have
no
awareness
of
it.
The
two
thought
s
runner
lets.
You
combine
the
ammo
fragments,
because
the
animal
is
how
we
define
all
these
things
that
you've
seen
these
in
these
thing
as
you've.
Seen
in
these
examples,
and
so
you
can
just
run
the
tooth
ology
command
line
and
give
it
a
target
ammo
which
may
change
whenever
you're
available
machines
change
and
then
a
mapping
of
the
roles
and
then
the
actual
test.
A
You
want
to
run
or
any
other
combination
it
just
glues
them
together
and
throws
the
archive
in
the
folder
you've,
given
it
in
this
case.
Something
like
this
is
much
shorter
and
simpler
than
our
jobs
tend
to
be
now,
but
as
covers
the
essence
of
them,
and
the
logging
is
really
useful
in
addition
to
the
original
starting
config
dot.
Yeah
mole,
wow
yeah,
I'm
a
nation
like
the
the
original,
like
configuration
of
the
system,
it
has
all
of
the
set
demon
logs
that
are
produced
on
all
the
remote
nodes.
A
Today,
tooth
ology
looks
a
little
bit
different.
First
of
all,
instead
of
having
like
for
developers
in
an
office
using
a
few
shared
machines
and
just
saying
hey,
I'm,
using
these
two
machines
don't
touch
them.
We
have
a
gigantic
lab.
We
call
it
the
sepia
lab,
that's
a
kind
of
kind
of
octopus,
I
think
and
it's
devoted
staff
development.
We
have
250
plus
servers
in
it.
I
think
these
numbers
are
a
little
out
of
date
down
it's
devoted
to
running
Ceph
tests
24/7.
A
We
keep
it
busy
pretty
much
all
the
time,
except
for
over
very
long
breaks
like
Christmas,
occasionally
and
SSH
access
is
granted
to
people
who
are
engaged
in
the
development
community
and
actually
and
have
a
demonstrated
commitment
to
to
building
stuff
it's
more
than
100
people.
It's
I
think
not
more
than
200
yet,
but
you
know
that's
sort
of
order
of
things.
A
Theology
has
a
system
for
letting
you
lock
the
servers,
because
we
have
300
servers.
We
have
a
hundred
people
who
are
trying
to
run
tests.
We
can't
we
don't
want
to
collide
with
each
other,
so
you
can
run
a
tooth
ology,
lock
command
and
get
a
certain
number
of
servers,
and
then
you'll
often
do
what
you
want
with
them.
A
A
A
So
we
have
a
couple
of
different
things:
different
subfolders
within
here
on
all
of
them.
We
have
a
Stefan
rate,
Oh
Scott
yeah
mo
that
do
some
config
setup.
We
have
a.
We
have
options
for
clusters
where
all
the
cluster,
all
the
jobs,
it
runs
on,
run
against
two
nodes
and
we
have
this
extra
OpenStack
that
will
file
in
case
where
you're
running
against
and
OpenStack
cluster.
We
have
two.
We
have
two
options.
A
You
can
either
run
with
the
default
thrashing
or
with
no
thrashing
a
thrashing
is
the
sort
of
chaos
monkey
we'll
talk
a
look
at
later,
and
then
we
have
a
few
different
object,
store.
Backends
I
think
this
is
also
changed.
Actually,
since
I
put
the
slide
together,
because
you
know
we're
pretty
focused
on
blue
store
now
more
than
more
than
we
were
in
the
past,
but
you
know
we
got
a
couple:
different
blue
store
example,
blue,
store
options
in
file
store
and
three
different
particular
tests.
A
A
A
What's
important
is
that
this
is
very
comprehensive,
it
covers
all
of
the
core
set
functionality
and
a
lot
of
the
extra
stuff,
and
these
Suites
include
Thrashers.
These
are
chaos
monkeys
that
we
run
while
other
tests
are
in
progress.
In
particular,
we
have
a
whole
lot
of
OSD
Thrasher's
and
some
monitor
thrashing
ones,
and
we
run
all
of
our
core
Rados
tests,
with
thrashing
enabled
to
make
sure
that
we
maintain
our
API
guarantees
under
all
kinds
of
very
ridiculous
failure
conditions.
A
A
You
can
set
up
to
email,
you
personally
and
you
can
look
and
see.
Oh
hey
I
broke
something.
Let
me
go
look
at
the
logs
that
have
been
saved
on
the
on
this
on
this
on
the
tooth:
ology
server
or
access
them
through
the
web
and
figure
out.
Why?
In
addition,
we
test
set
pull
requests.
We
accept
pull
requests
from
any
random
person
who
comes
in
and
once
detect
and
tech
leads
or
other
reviewers
look
at
them
and
say:
ok,
I
think
this
might
work.
A
It
looks
okay,
so
I'd
like
to
run
it
through
an
integration
test
and
make
sure
that
that
passes
before
we
merge
it
into
the
end
of
the
into
the
master,
build
and
I,
and
there
are
some
tools
to
help
do
that
that
we've
that
various
developers
have
built
that
called
one
called
PTL
tool
pie
where
you-
and
these
allow
us
to
do
things
like
we
just
we
have
tags
on
github
that
could
say
hey.
This
needs
to
run
through
QA
and
others.
A
That
say
like
hey,
I'm
I'm,
including
this
in
my
integration
branch
and
tools
that
will
help
you
will
like
go
and
find
okay,
although
all
the
PRS
that
are
labeled
for
this
integration
branch,
merge
them
into
a
single
branch,
push
them
to
SEP
CI.
That
get
then
schedule
a
run,
and
once
that's
done
say.
Ok,
like
this
run,
looks
good,
so
I'm
gonna
go
back
and
and
approve
all
of
the
PRS
included
in
it
for
merging
or
not.
A
It's
important
that
nothing
merges
into
the
Sephiroth
master
with
passing
these
tests
or
the
subset
that
leads
believe
are
appropriate
because
otherwise,
it's
just
it's
just
too
huge
and
complicated
a
system.
If
we,
if
we
merged
up
without
passing
tests,
it
would
break
and
on
the
very
very
rare
occasions
when
we
do
merge
something
without
patent
without
tests,
because
it's
very
urgent-
and
we
know
that
it's
a
simple
and
correct-
fix
we're
right
about
half
the
time.
A
Usually
when
that
happens
it,
it
is
immediately
obvious
because
a
build
fails
rather
than
because
we
broke
a
broken
algorithm,
which
is
reassuring,
but
still
like
every
time
that
happens.
We're
like
okay,
no,
like
really
nothing,
can
merge
without
going
through
the
tests
or
we
are
we
screwed
up
really
badly.
And
finally,
we
have
nightly
jobs
that
are
scheduled
automatically
via
crontab
the
source
file,
for
it
is
posted
on
github.
We
run
all
the
Suites
generally
from
1
to
7
times,
weekly
against
master
and
our
stable
branches,
just
to
make
sure
things
are
staying
kosher.
A
In
particular,
we
have
a
lot
of
Jenkins
based
tests
that
are
used
for
testing
the
set
danceable
and
set
volume
associated
pieces
of
pieces
of
the
overall
ecosystem.
These
are
nice
because
they're
really
easy
to
integrate
their
much
easier
right
now,
integrate
with
github.
So
if
you
run,
if
you
like,
put
in
a
PR
to
ansible
or
to
set
volume,
then
they
can
just
get
tagged
and
then
the
test
run
and
they
give
you
a
pass
or
a
fail
report.
A
That's
not
really
suitable
for
the
full
SEF
suite,
because
the
Rado
suite
requires
a
reasonable
run
of
that
requires
a
couple
thousand
machine
hours,
so
we're
not
going
to
do
that
on
every
PR
automatically.
But
in
addition,
we
have
a
make
check
target
in
the
in
the
repository
and
Jenkins
triggers
that
on
every
push,
or
at
least
every
approved,
push
to
make
sure
that
we
haven't
broken.
Something
really
really
obvious.
A
big
important
part
of
that
is
what
we
call
the
set
object.
A
Corpus
Saif
is
a
data
storage
system
and
it's
really
important
that
when
you
upgrade
your
Ceph
storage,
it
still
be
able
to
read
all
of
the
data
that
it
wrote
in
the
last
version,
and
so
we
have
a
we
tests.
We
have.
We
have
a
system
that
lets
us
run
a
tooth
ology
job
or
the
let's
have
thrown
a
tooth
ology
suite
and
collect
samples
of
all
of
the
encoded
pieces
of
data
and
save
them
into
the
SAP
object
corpus.
A
And
then
we
make
sure
that
and
we
do
and
we
do
that
on
our
major
releases.
And
then
we
make
sure,
on
the
on
the
next
major
release,
that
it
can
still
decode
all
the
data
written
by
the
previous
ones
that
we
support
upgrading
from
and
that
we,
when
we
encode
something
we
get
the
same
thing
back
when
we
decoded
but
collecting
all
that
data
for
the
subtopic
corpus
is
actually
pretty
time
intensive
and
very
manual
and
so
Keefe,
who
recently
did
it
for
but
I
did.
A
A
If
you
would
like
to
learn
how
to
maintain
these
set
object
corpus
and
improve
our
ability
to
gather
that
data
and
validate
it
on
every
push,
but
with
all
that
said,
there
are
some
test
gaps
in
the
system
or
in
the
way
we
test
stuff
right
now,
I
want
to
cover
my
buddy
bit
here,
because
and
I
don't
want
you
to
think
that
we're
bad
at
it
we're
actually
very
good
at
testing
staff.
The
functional
coverage
is
great.
A
A
So
there's
a
lot
of
things
we're
very
careful
about,
but
there
are
some
parts
that
don't
get
covered.
Thorgy
handles
two
handles
daemons
directly
by
SS
aging
into
the
node,
and
then
it
says,
and
then
it
runs
SEF
OSD
in
a
foreground
SSH
process.
So
it
does
not
test
our
unit
tests
or
our
units
for
integrating
with
system
D
or
other
init
systems.
It
does
not
test
the
integration
with
kubernetes
at
all.
Right
now,
thatha's
not
do
performance
testing.
A
We
have
a
new
purse
suite
as
of
late
last
year
that
uses
the
set
benchmark
tool
from
mark
Nelson
and
that's
been
used
by
the
performance
and
hardware
testing
communities
around
staff,
but
they
there
are.
The
number
of
tests
that
actually
runs
is
very
limited
and
there
isn't
any
analysis
of
it
built
into
any
system
right
now.
A
Scale
testing
is
not
really
suitable
within
two
thaal
expect
tests
to
run
within
an
hour
or
two
and
for
them
to
be
very
small,
so
we
can
run
lots
of
tests,
but
staff,
of
course,
is
a
scalable
distributed
system.
So
we
want
to
be
able
to
make
sure
that
we
scale
up
to
tens
of
thousands
of
nodes
and
so
that's
a
test
gap
when
the
test
gap
we
have
right
now
and
in
addition
to
the
inner
system
itself,
youth
ology
handles
all
the
seven
stalls.
A
But
we
have
some
potential
fixes
that
we'd
like
to
look
at
and
if
you
want
to
get
involved
with
any
of
these
proposed
solutions.
We've
been
talking
about
them,
some
on
the
mailing
lists
and
we
would
love
to
have
more
people
working
on
them
in
term
to
fix
tooth
ology
handling
demons
directly.
We
need
to
expand
the
internal
tooth,
ology
and
CEFCU
a
suite
api's
to
restart
in
signal
using
the
init
system
instead
of.
A
Actually,
this
is
an
actual
like
piece
of
code
out
of
ethology,
where
we
have
that,
where
we've
abstracted
out
the
monitor
remote
node,
but
we
can
just
call
this
run
command
and
we
issue
a
kill,
all
9
for
performance
testing.
We've
started
gathering
the
historical
data
on
performance
and
we
want
and
and
then
we
can
guess,
a
rough
out
outer
limit
and
fail
test
when
it
exceeds
some
some
difference.
But
I
don't
think
this
has
actually
been
done
yet
and
we
started
gathering
data
it's
eight
months
ago.
A
So
we
clearly
need
to
figure
out
a
like.
Do
that
and
come
up
with
a
better
long-term
solution.
Some
kind
of
like
I,
don't
know
how
it
work
really,
but
some
kind
of
analysis
system
that
tracks
the
performance
and
alerts
when
they're
out
of
bounds
based
off
of
the
machines
you're
running
on,
because
the
machines
might
impact
their
performance
and
the
different
versions
will
obviously
perform
differently,
hopefully
faster,
as
time
goes
on
rather
than
slower,
and
but
we
don't
want
to
them,
be
like.
A
Oh,
you
ran
Firefly
for
the
first
time
in
a
year
and
everything
slow
so
fail
in
terms
of
scale
testing.
We've
just
decided
to
follow
Jesus
and
integration
test
framework,
and
it's
not
a
good
place
for
a
long-running
test
like
this.
So
various
down
streams
have
done
some
things,
but
we
in
the
upstream
would
also
like
to
figure
out.
A
How
do
you
inject
aged
and
large-scale
data
within
short
running
tests,
by
doing
things
like
running
a
long-running
rate
of
gateway,
job
and
then
collecting
the
OMAP
inputs
and
the
file
data
and
and
then
just
figuring
out
how
to
stream
that
into
Rios
much
more
quickly
than
going
through
rate
OCW
itself
and
we'd
also
like
to
be
able
to
get
tests
that
run
with
longer-term
allocations,
either
in
a
smaller.
But
but
you
know,
still
reasonably
sized
portion
in
the
community
lab
or
in
somewhere.
A
That's
been
donated
by
some
by
some
community
member
with
resources
and
to
fix
the
tooth
and
to
get
test
coverage
of
the
install
and
cluster
config
we'd
like
to
again
write
new
internal
api's.
So
we
can
request
installs
and
farm
them
out
to
set
danceable
or
deep-sea
or
rook
or
whatever
else,
and
that
the
end
so
that
the
set
like
the
functional
tests
themselves
don't
care
who
installed
it.
They
just
know
that
they
like
that,
and
they
just
know
that
there
was
a
thing
that
ran
the
install
and
the
install
set
up.
A
There
is
one
other
problem
I
haven't
mentioned
so
far.
The
the
new
set
manager
orchestrators
that
we've
talked
about
a
few
times
and
that
Sebastian
has
been
discussing,
are
not
integrated
into
the
testing
at
all
right
now
and
that's
because
they
rely
on
unset,
danceable
or
deep-sea
or
rook
to
do
things
and
we,
except
for
the
one
that
runs
on
SSH,
but
that
hasn't
been
integrated
yet
and
since
we
aren't
using
those
at
all,
there's
no
way
to
test
the
orchestrators
within
the
tooth
ology
system
and
that's
a
pretty
big
hole.
A
A
In
addition
to
the
test
gap
coverage,
there
are
some
framework
weaknesses
within
tooth
ology,
which
you
might
have
heard
a
little
bit
about.
If
you
went
to
oh
no
I'm
going
to
forget
his
name
until
I
see
slide.
If
you
went
to
the
digital
ocean
talk
about
running
to
the
ology
yesterday,
so
tooth
ology
is
strongly
tied
to
the
CPA
community
lab
it's
supposed
to
go
anywhere,
but
we
have
a
lot
of
hard
coded
value
sneaked
in
because
we
don't
have
any
way
of
preventing
them
from
appearing.
The
documentation
is
often
unclear.
A
The
the
code
and
test
II
bug
cycle
involves,
locking
lab
machines,
doing
doing
machine
images
and
package
installs.
You
need
built
set
packages
in
order
to
actually
run
any
tests,
because
so
so
you
have
to
wait
for
some
external
package
repository
system
to
finish,
and
it
makes
it
hard
for
third
parties
to
contribute
to
technology
development
in
general.
In
addition
to
people
just
testing
stuff
themselves.
A
We
just
have
a
script
called
V
start
that
turns
on
a
couple
of
well
a
specified
number
of
OS
T's
and
monitors
and
things
on
your
local
dev
box,
and
so
you
can
run
a
lot
of
the
file
system
tests
and
some
of
the
rtw
ones.
Just
against
that,
we
have
started
doing
more
deliberate
work
to
foster
a
community
around
testing
staff.
We
run
a
weekly
meeting
on
Wednesday
mornings,
us
time
to
discuss
and
you
can
get
details
on
about
it
on
the
website.
A
We
have
started
reaching
out
to
some
of
the
known
users
and
doing
active
work
to
merge
their
enhancements
and
and
and
their
in
their
local
ports
into
the
upstream
in
ways
that
let
us
that,
let
that
make
it
easier
to
work
with
in
the
future.
We
want
to
keep
looking
at
the
localization
patches
and
replace
them
with
configuration
options
to
make
that
setup
easier
and
to
rebuild
tooth
ology
OpenStack
with
more
reliable
api's.
There
are
some
successes
coming
out
of
that
already.
A
Sousa
has
had
their
own
fork
of
tooth
ology,
of
tooth
ology
from
the
upstream
community
lab
and
from
what
Red,
Hat
use
and
they've
almost
finished,
bringing
their
their
fork
back
into
the
upstream
so
that
they
can
run
out
of
that.
So
thanks
to
kheer
and
I,
now
know
that
digital
ocean
deploys
tooth
ology
in
their
cloud,
and
you
can
do
that
too.
If
you
want
just
in
their
cloud
without
without
any
fork
at
all,
but
just
some
some
mangling
of
the
infrastructure.
A
All
right
so
another
problem
too
soggy
doesn't
have
good
trend
analysis
tools.
We
can
really
easily
tell
that
a
test
has
failed.
It
shows
up
right
in
papito,
there's
a
summary
dot,
Y
animal
that
says
failed.
We
get
core
dumps
or
or
and
and
we
have-
and
we
have
robust
mechanisms
for
for
triggering
that
failure.
If
there's
a
core
dump
that
isn't
whitelisted,
then
we
fail.
If
there's
a
warrant
or
an
error,
log
message
that
doesn't
wait-listed,
then
we
fail
if
a
test
assert
doesn't
pass,
we
fail,
but
there's
not
a
good
way
to
say.
A
Oh,
this
yam
will
fragment
like
this.
This
test
hasn't
passed
in
the
last
week
or
this
test
stopped
passing
or
has
failed
every
other
run
and
over
the
last
month,
or
this
combination
of
vml
fragments
fail.
Every
time
we
don't
have
any
way
of
saying.
Oh
eight,
a
particular
combination
of
yellow
fragments
is
now
taking
four
times
longer
than
it
used
to
to
pass
and
we
don't
have
any
way
of
tracking
performance
numbers
and
from
a
pure
functionality
perspective
like
the
test.
A
Functionally
passed:
that's
okay,
but
from
a
you
want
your
storage
system
to
behave
predictably
and
to
not
get
slower,
that's
not
so
great,
so
we
need
to
either
find
or
build
some
trend
tracking
software
that
we
can
integrate
with
into
with
compete
with
papito
and
the
wider
thorgy
frameworks
and
and
start
picking
up
on
those
things
automatically
a
minor
problem
which
we
mostly
fixes
at
Suites
explode
in
size.
They
are
combinatorial
explosion
of
every
single
yamo
fragment
and
every
time
we
add
a
new
feature,
condition
vary
or
configuration
option
that
we
want
people
to
switch.
A
The
sweet
doubles.
So
obviously
that's
bad.
If
a
key,
if
it
that
means
it
keeps
taking
longer
to
run.
In
addition,
it
means
that
it's
harder
for
newer
developers
and
third
party
people
to
know
which
tests
are
the
most
important
to
to
analyze.
If
there's
a
failure,
it
means
that
you're
using
up
more
compute
resources
that
you
might
not
have
very
many
of
so.
We
have
a
subset
functionality
that
Sam
wrote
three
years
ago.
A
I
want
to
say
that
uses
all
the
ammo
fragments,
but
not
the
total
combinatorial
explosion
and
you
can
and
you
can
set,
and
you
can
split
it
up
into
subsets
of.
However
many
you
want
and
then
iterate
through
those
subsets
like
over
time,
rather
than
on
every
single
run,
run
to
the
system,
and
so
you
can
get
it
down
from
$124,000
set
397,
that's
still
a
reasonable
cross-section
of
the
tests,
and
that
makes
sure
that
every
option
gets
turned
on
at
least
once
in
addition
to
the
subset.
A
We
also
have
filter
scheduling,
so
you
can
do
things
like
say:
hey,
I,
don't
want
to
run
anything
on
a
file
store
or
I
only
want
to
run
the
things
on
file
store.
We
when
we
have
test
failures,
we
can
say:
hey
I,
want
to
rerun
this
suite
again
or
I
want
to
rerun
all
the
jobs
that
failed
in
this
previous
run,
because
I
made
a
small
change.
That
I
think
fixed.
That
and
I
want
to
validate
that
before
I
put
the
whole
PR
through
again,
and
so
that's
mostly
under
control.
A
A
bigger
problem
is
that
scheduling
is
very
primitive.
Like
I
said
it's
Bienstock
d
and
jobs
go
in
in
the
order
you
put
them
in
and
then
you
have
like
50
processes,
racing
to
say:
hey
is
there
a
new
job,
I
can
have.
Oh
I
got
a
new
job
and
then
and
then
and
then
the
process
locks
the
number
of
nodes
it
needs,
and
then
it
runs
the
job.
But
if
that
means
that,
if
you
have
a
5
node
job,
it
never
ever
runs
because
the
process
gets
it
and
says:
hey.
A
In
addition,
scheduled
Knightley's
can
be
delayed
for
days
because
people
keep
on
manually,
scheduling,
Suites
at
a
higher
priority
and
then
and
then
a
week
later,
there's
room
and
so
now
we're
running
a
full
nightly.
Tet
Rados
test
on
a
week
hold
sha-1
and
we'd
really
rather
be
running
the
current
one.
So
we've
come
up
with
some
workaround
hacks,
like
basically
all
of
our
jobs
run
on
the
same
number
of
nodes,
mostly
to
which
we've
gotten
accustomed
to,
but
there's
also
a
problem.
A
We
have
people
who
spend
a
lot
of
time,
monitoring
the
Beanstalk
queue
and
killing
and
rescheduling
old
mighty
leaves
as
they
aged
out.
It
would
be
great
if
we
could
update
the
existing
scheduled
test
runner
so
that
we
lock
the
needed
machines
before
the
job
gets
off
the
queue
and
that
way
stuff
still
comes
off
in
order
and
gets
what
it
needs
without
without
a
big
huge
change.
A
But
another
thing
that's
being
pushed
and
that
people
are
working
on
is
just
writing.
Sef
tests
in
some
kind
of
kubernetes
friendly
framework
that
exists
sage
is
real
big
on
prowl.
There
are
several
options,
and
the
idea
here
would
be
that
we
can.
It
would
be
quick
and
easy
to
get
SEF
running
inside
of
the
system
and
we
can
write
integral
and
we
can
write
tests
around
those
integration
points.
The
concern
that
I
have
around.
A
That
is
that
we
might
miss
identify
what
the
what
the
integration
points
that
we
need
to
test
are
and
and
and
not
correctly
identify
risky
areas
that
need
coverage
with
that
and
so
and
and
we
might
just
pre
implanting
all
the
existing
tests
and
I
mean.
If
we
do
that,
then
we
now
have
all
the
tests
reimplemented
inside
of
a
modern
framework
and
that's
great,
so
we
need
to
switch
to
it,
but
we're
probably
not
ever
gonna
reach
that
point
if
we
do
it
incrementally
without
having
without
having
some
kind
of
portable
system.
A
Additionally,
within
the
to
thaw
a
framework,
the
physical
machines
are
first-class
citizens.
If
you
think
back
to
the
mo
fragment,
I
showed
you
it
was.
It
was
machines
with
an
IP
specific
machines
with
an
IP
address
and
we
do
pack
things
like
packet
installation
on
those
individual
target
nodes.
And/Or
things
like
power,
cycling
on
those
target
nodes
and
demons
like
ACF
OST,
belong
on
that
machine,
but
integrates
that's
nonsense.
You
have
pods
and
those
pods
might
be.
A
So
we
need
to
build
out
new
interfaces
for
dealing
with
stuff
like
that
and
that's
the
end
of
what
I
got
we
have
about
four
minutes
left.
If
you
got
any
questions
you
want
to
go
around,
Brad
is
graciously
volunteered
to
carry
a
microphone
so
bring
it
at
me.
Thanks
very
much
any
questions,
Wow
no
heckling
either
you
must
have
been
super
bored
well,
in
that
case,
I'll
be
around
all
week,
have
a
great
rest
of
the
day
at
cephalic
on.