►
Description
Anshu Dubey (ANL) presents: "What All Codes Should Do: Overview of Best Practices in HPC Software Development"
Abstract: Scientific code developers have increasingly been adopting software processes derived from the mainstream (non-scientific) community. Software practices are typically adopted when continuing without them becomes impractical. However, many software best practices need modification and/or customization, partly because the codes are used for research and exploration, and partly because of the combined funding and sociological challenges. This presentation will describe the lifecycle of scientific software and important ways in which it differs from other software development. We will provide a compilation of software engineering best practices that have generally been found to be useful by science communities, and we will provide guidelines for adoption of practices based on the size and the scope of the project.
A
On
to
do
Bank
to
do
Bay
is
a
currently
a
member
of
the
mathematics
and
computer
science
department
here
at
Argonne
as
she's
joined
to
the
computer
scientist.
She
spent
12
years
at
the
University
of
Chicago
in
the
astronomy
and
astrophysics
department,
as
first
a
lead
application
developer
than
the
computer
science
applications
group
leader
and
the
associate
director
of
the
flash
project,
and
this
project
was
responsible
for
developing
a
very
large
community
code,
called
flash
edition.
A
So
first,
if
you
do
have
questions,
please
enter
them
in
the
text
chat
section
and
what
I
will
do
is
we
will
collect
those
there's
about
five
different
sections:
don't
use
talk,
and
so,
when
we
reach
one
of
the
end
of
those
sections,
if
there
are
questions
we
will,
we
will
feed
those
through
to
anshu.
So
one
of
us
will
read
them
off
to
her,
but
I'll
encourage
you
to
type
the
questions
as
you
have
them,
and
and
that
way,
when
we
get
to
the
end
of
the
session,
we
will
have
questions
ready
to
go.
A
B
You
for
that
introduction
Katherine.
So
this
is
a
series
of
webinars
that
we
have
conceived
in
order
to
bring
best
practices
to
the
HPC
practitioners.
This
is
a
collaboration
among
the
ideas,
scientific
software
productivity
project,
which
is
basically
focused
in
improving
software.
Productivity
of
the
scientific
codes
are
gone
leadership,
computing
facility,
national
energy,
research,
scientific
computing,
center
book
and
Oakridge
leadership
computing
facilities.
B
What
I
will
do
is
in
the
first
section
of
my
talk.
I
will
give
you
an
overview
of
the
series,
which
is
the
objectives,
what
we
hope
to
achieve
by
through
the
series
and
the
next
few
seminars
webinars,
that
you
can
expect
to
see
in
this
series
and
then
I
will
move
on
to
the
technical
section
of
this
talk,
which
will
have
four
other
sections
but
I'll
get
to
that
later.
So
the
objective
of
this
series
is
to
bring
the
knowledge
of
useful
software
engineering
practices
to
HPC
scientific
code
developers
notice
the
emphasis
on
useful.
B
Instead,
what
we
really
want
to
do
is
to
collect
the
experiences
and
knowledge
that
is
acquired
by
various
teams
that
have
been
working
in
this
realm
and
be
as
informative
as
possible
about
these
practices
that
have
worked
for
some
of
the
projects.
So
our
emphasis
is
on
letting
you
know
the
practices
that
are
there
and
to
that,
so
that
you
can
pick
and
choose
the
practices
that
will
help
your
productivity
rather
than
put
unsustainable
burden
on
your
team.
B
We
also
will
highlight
the
current
various
customizations
that
other
teams
have
done
and
to
indicate
how
the
customizations
can
be
done
as
needed.
That
is,
and
you
can
do
that
for
yourself,
based
upon
the
information
that
we
make
available,
we
will
do
it
as
much
as
possible
through
examples
in
case
studies,
and
we
will
give
references
for
available
resources
and
suggestions
for
further
reading
the
webinars
in
the
series.
Today's
webinar
is
basically
an
overview
and
introduction
on
what
we
want
to
do
in
this
webinar
is
to
provide
the
motivation
as
to.
B
Why
should
a
computational
scientist
worry
about
software
process?
The
practices
that
many
codes
have
adopted
and
found
useful
examples
of
customizations
and
one
of
the
topics
that
will
not
be
covered
by
other
webinars,
but
we
think
is
extremely
important,
is
discussion
of
community
codes,
why
they
are
helpful
and
how
can
one
go
about
building
a
community?
B
The
next
webinar
in
the
series
will
be
two
weeks
from
today
at
the
same
time,
and
that
will
be
given
by
Barry
Smith
on
developing
configuring
building
and
deploying
HPC
software,
the
tools
and
best
practices
for
configuration
and
building
of
your
software.
Some
basic
principles
of
software
design
and
development.
B
Helpful
hints
that
will
that
can
help
developer
productivity
through
use
of
development,
environments
and
customizations
for
project
needs.
Following
that,
two
weeks
later,
there
will
be
a
talk
on
distributed
version,
control
and
continuous
integration
text
testing.
This
will
be
done
by
Jeff
Johnson,
who
will
use
git
as
the
example
platform,
and
he
will
talk
about
using
it
for
not
just
version
control,
but
also
as
a
development
platform
and
a
control
change
process
which
he
will
explain
through
the
use
of
pull
requests
and
mechanisms
for
communicating
and
tracking
progress.
B
He
will
also
talk
about
continuous
integration
with
Travis
CI.
The
fourth
webinar
in
the
series
will
be
on
testing
and
documentation
of
code.
This
will
be
given
by
Alicia
kimbugs,
because
testing
and
documentation
both
are
in
particular
practices
that
can
you
can
do
too
much
or
too
little,
both
of
which
is
too
bad.
So
we
will
hope
to
give
we
hope
to
give
guidelines
on
how
to
evaluate
your
own
team
needs
and
the
extent
of
testing
that
is
useful
for
your
team
rather
than
burdensome.
B
We
will
also
talk
about
granularity
of
testing,
whether
you
do
unit
test
integration,
level,
test
system,
level,
test,
etc
the
definitions
and
how
to
leverage
this
testing
granularity,
not
only
to
pinpoint
failure,
but
also
to
provide
code
coverage,
and,
in
addition,
it's
also
important
to
talk
about
how
to
get
buy-in
from
the
development
team
and
that
will
be
covered
in
this
draft
as
well.
Following
that
we
so
the
details
for
the
next
three
talks
will
follow
later.
B
A
B
You
so
the
technical
section
of
my
this
particular
webinar
is
divided
into
four
sections.
The
first
section
is
about
the
motivation.
The
second
section
talks
about
basically
surveys
from
various
communities
and
the
customizations
they
have
done
the
practices
they've
adopted.
The
third
section
will
be
on
best
practices
and,
finally,
I
will
talk
about
Community
Development,
the
motivation
section.
So
most,
the
most
people
that
start
out
in
computational
start
with
what
is
called
heroic
programming
and
heroic
programming
is
in
principle,
doing
whatever
it
takes
to
make
a
project
work.
B
The
text
here
has
been
picked
straight
from
this
link
that
you
can
see,
and
this
particular
link
defines
heroic
programming
as
expenditure
of
huge
amounts
of
coding,
effort
by
talented
people
to
overcome
shortcomings
in
process,
project
management,
scheduling
architecture
or
any
other
shortfalls
in
the
execution
of
software
development
project.
In
order
to
complete
it,
scientific
teams
often
resemble
heroic
programming,
and
what
is
more
troublesome
is
many.
B
Do
not
see
anything
wrong
with
that
approach
and
I
can
quote
when
anecdote
from
my
own
experience
about
a
particular
team
that
wanted
to
do
simulations
on
material
which
turn
into
solids
through
sheer
stress,
and
so
they
talked
to
us
at
the
flash
Center
about
how
to
go
about
whether
they
can
use
flash
and
how
to
go
about
doing
it.
We
realized
that
what
they
needed
to
do
would
require
quite
a
bit
of
effort,
so
we
gave
to
them
information
about
what
we
would
need
to
how
much
we
would
need
to
work
with
them.
B
And
how
would
we
go
about
doing
this
at
the
end
of
that
discussion?
I
was
told.
Oh,
we
will
just
get
a
couple
of
postdocs
and
we
will
do
it
because
this
is
how
we
do
our
science
we
do
whatever
it
takes
to
the
best
of
my
knowledge
that
project
has
not
gone
anywhere
to
emphasize
on
what
is
wrong
with
heroic
programming
see
in
the
there
are
projects.
There
are
some
results
that
can
that
you
can
do
with
one
or
two
people,
just
writing
a
simple
small
amount
of
code
and
getting
their
results
from
them.
B
But
by
and
large-
and
this
was
very
fruitful
way
to
do
things
in
the
beginning,
when
computational
science
was
just
beginning
to
get
started
as
a
valid
way
of
doing
science,
but
the
scientific
results
that
could
be
obtained
with
heroic
programming
have
pretty
much
run
their
course,
and
that
is
because,
as
you
have
better
scientific
understanding,
you
want
to
refine
your
models.
When
you
refine
your
models,
your
software
becomes
more
complex
because
you
have
more
equations
added
into
it.
You
have
more
models
added
into
it.
B
Your
numerical
methods
gets
more
sophisticated
sophisticated,
so
you
have
more
complex
software.
When
you
have
more
software
complex
software,
then
you
really
need
to
start
focusing
on
different
aspects
of
it
at
different
times.
For
example,
the
mathematical
model,
the
numerical
algorithms,
that
implement
those
mathematical
models,
the
verification
of
your
numerix
and
verification
is
not
just
correctness.
It
is
also
stability
and
conversions
and
the
performance
of
your
code.
B
What
these
need
is
to
have
people
with
different
roles
and
responsibilities
in
the
team,
and
it
is
not
a
possible
any
longer
for
a
single
person
to
take
on
all
of
these
roles,
so
it
becomes
important
to
have
to
move
away
from
the
model
of
heroic
programming
and
start
to
have
a
team.
This
becomes
an
even
bigger
problem
in
stream
scale,
science,
which
is
what
the
leadership
class
facilities
and
the
other
high
performance
supercomputing
facilities
deal
with.
It
is
because
they
are
typically
places
where
codes
that
are
aiming
for
higher
fidelity.
B
Modeling
are
operating,
they
have
multiphysics
codes.
Therefore,
their
codes
are
more
complex.
They
are
their
simulations
are
more
complex.
Even
their
analysis
is
more
complex
they're.
They
have
many
many
models
in
the
same
code
because
of
which
there
are
many
moving
parts
that
need
to
interoperate
with
one
another.
They
need
the
righty
of
expertise,
not
just
in
domain
science
because
they
have
multiphysics
within
the
code,
but
also
they
need
certain
amount
of
method,
expertise.
B
If
anything,
this
has
become
worse
with
the
onset
of
higher
platform
heterogeneity,
because
now
you
not
only
need
experts
who
know
science
and
software
there.
You
also
need
experts
that
can
understand
the
different
demands
placed
upon
the
performance
of
a
code
by
different
aspects
of
a
platform
architecture.
So
the
only
and
these
requirements
are
not
very
well
known,
so
far,
they're
still
only
unfolding,
so
no
one
has
clear
answer
as
to
how
to
design
and
implement
the
code
so
that
it
can
tackle
platform
heterogeneity.
B
So
the
only
safeguard
that
we
have
pretty
much
is
to
invest
in
flexible
design
and
robust
software
engineering
practices,
so
that
changes
and
adjustments
to
platforms
is
not
as
onerous
as
it
is.
If
you
are
starting
from
scratch-
and
there
are
other
reasons
in
other
realms
in
addition
to
the
extreme
scale
computing-
and
that
is
accretion-
leads
to
unmanageable
software
when
I
say
accretion.
B
What
it
does
is
it
increases
the
cost
of
Management
parts
of
software
may
even
become
unusable
over
time,
but
that
is
the
smaller
of
the
challenges
posed
by
a
created
software.
The
bigger
problem
is
that,
as
you
start
to
collect
more
and
more
components,
more
and
more
capabilities
in
your
software,
if
there
is
no
verification,
if
there
isn't
adequately
defined
verification
process
soon,
you
will
stop
having
confidence
in
your
own
results,
and
this
has
been
known
to
happen.
B
In
addition
to
that,
this
increases
the
ramp
on
time
for
new
developers
when
people
come
in
and
there
is
a
structured
software
in
complex
software,
it
takes
long
enough
time
for
new
developers
to
become
truly
productive
in
the
code
anyway,
when
the
software
is
also
without
any
structure,
if
they
can't
get
a
handle
on
where
various
things
in
the
software
are,
it
just
takes
much
longer
for
the
software
for
the
new
developers
to
become
productive
members
of
the
team
and
this
what
it
does
is.
It
reduces
software
and
science
productivity
both
due
to
technical
debt.
B
Technical
debt
is
a
term
that
is
the
that
is
used
to
describe
the
debt
that
is
incurred
in
quick
and
dirty
practices.
That
then
collects
interest
in
terms
of
more
effort
needed
to
add
feature
to
the
software.
So
you
pay
your
technical
debt
by
in
the
involve
indulging
in
better
design.
If
you
have
better
design,
then
you
are
pretty
much
able
to
add
features
without
too
much
effort,
but
and
as
the
design
becomes
more
and
design
becomes
less
and
less
effective.
The
technical
debt
incurred
by
adding
features
also
grows.
B
So
it's
very
important
to
keep
that
in
mind
when
design
when
making
decisions
about
the
software.
The
changes
to
the
software
and
its
design
and
I
finished
this
section.
With
the
this
quote,
it
seems
likely
that
significant
software
contributions
to
existing
scientific
software
projects
are
not
likely
to
be
rewarded
through
the
traditional
reputation
economy
of
science.
B
B
The
next
talk
will
actually
to
talks
from
today.
You
will
get
much
more
details
about
the
difference
between
distributed
and
version
controls,
but
SVN
is
SVN
and
CVS
are
examples
of
non
distributed
version
control,
get
mercurial,
etc.
Are
examples
of
distributed
version
controls.
The
bills
in
these
projects
are
equally
divided
between
new
make
and
see
make
all
of
the
projects
provide
some
form
of
documentation
with
some
some
form
of
users
guide
to
their
users.
Most
of
them
use
automated
documentation
generation
tools
in
order
to
produce
their
documentation.
B
The
big
amount
of
documentation
provided
provided
varies
from
code
to
code.
All
the
codes
have
testing
in
some
form.
The
dominated
form
is
integration,
testing,
where
various
components
are
joined
together
to
do
regression
to
verify
the
overall
correctness
of
the
code.
A
couple
of
them
do
manual
regression
testing,
but
quite
a
few
of
them
are
also
automated.
B
Roughly
half
the
projects
make
use
of
unit
testing
explicitly
and
majority
of
these
codes
are
publicly
available.
There
are
a
few
that
are
not
pertinent.
Oddity
of
them
are
publicly
available.
The
second
source
that
I'm
summarizing
from
is
from
a
workshop
on
community
codes
that
was
conducted
in
2012
at
the
University
of
Chicago
under
NSF
funding
this
one.
So
the
the
previous
survey
was
mostly
from
codes
that
are
in
the
earth
sciences
community.
This
one
with
the
community
code
workshop
had
codes
from
a
wide
variety
of
domains.
B
So
the
codes
that
participated
in
this
workshop
work
was
flash,
which
is
astrophysics
cactus,
which
is
numerical
general
relativity
ends
of
a
cosmology
psmf
for
Earth
System
lattice
QCD
code
suite
amber,
which
is
quantum
chemistry.
Tombow
is
an
EMR
package
and
whitie,
which
is
an
analysis
and
visualization
tool
that
is
mostly
used
by
the
astrophysics
community
and
more
now.
B
All
these
codes,
without
exception,
have
rigorous
auditing
plus
process
in
place,
which
means
that
they
have
processed
not
only
for
testing
but
also
gatekeeping,
for
contributions,
policies
for
contributions
from
internal
and
external
users.
The
models
vary
from
code
to
code.
All
these
codes
also
have
wide
user
communities
and
all
codes
assert
that
communities
benefit
greatly
from
having
a
common,
highly
exercised
codebase.
There
is
a
question
yeah,
so.
A
B
Would
say
that
it
is
not
impossible,
but
I
would
need
a
great
deal
of
convincing
to
believe
that
that
would
be
possible,
because
a
loan
scientist
so
I
can
see
a
lone
scientist
becoming
productive.
If
the
loan
scientist
had
a
community
code
that
they
could
customize
but
I
don't
see
a
large-scale
scientific
problem
that
you
start
from
scratch.
You
are
able
to
build
all
the
models
that
you
might
need
in
that
scientific
problem
that
it
would
actually
need
leadership,
science
class
capable
code
in
order
to
be
productive.
B
B
B
The
challenge
is
that
both
the
sources
for
our
community
codes
found
was
that
that
the
challenges
are
broadly
classified
into
two
sections:
the
technical
and
the
sociological.
The
technical
sections
are
that
all
parts
of
the
cycle
can
be
under
research,
which
means
that
your
models
are
under
research.
Your
understanding
of
the
physics
is
under
research.
The
numerical
methods
that
are
used
to
implement
the
models
are
under
research
so
and
the
requirements
from
all
of
these
aspects
of
code
development
change
throughout
the
life
cycle
as
knowledge
grows.
B
In
addition,
for
all
of
them,
verification
is
complicated
not
only
by
the
complexity
of
the
code
itself,
but
also
because
they're
all
dealing
with
floating-point
representation,
so
relying
upon
bitwise
matching
of
answers,
even
though
is
desirable,
is
not
always
available
to
them.
So
they
have
to
understand
that
error
bars
much
more.
Clearly,
real
world
is
messy,
and
so
the
software
that
deals
with
has
to
deal
will
has
to
figure
out
how
to
not
be
messy
when
the
natural
tendency
of
such
software
would
be
too
messy.
B
And
then
there
are
the
sociological
challenges,
such
as
competing
priorities
and
incentives,
because
the
science
is
always
interested
in
getting
a
quick
answer
to
whatever
is
being
developed.
They
pose
a
question.
They
want
a
quick
answer,
but
in
order
to
enable
the
capability
that
may
give
that
quick
answer,
a
code
may
need
to
take
longer
than
the
scientists
would
like.
So
there
are
all
these
competing
priorities
in
play.
B
So
many
of
them
evolved
their
own
testing
methodologies
and
they
follow
this
methods,
as
understood
by
they
do
not
follow
the
methods
as
understood
by
the
software
engineering
research
community
they
developed
according
to
their
own
needs,
and
so
in
some
sense
the
extent
and
granularity
of
their
testing
reflects
the
project
priorities
and
the
team
size.
So
if
you
look
at
a
small
team,
it
has,
it
is
much
more
informal
and
where
they're
testing
reflects
that,
and
there
are
larger
teams
that
have
much
more
formalization.
B
The
lifecycle
of
a
scientific
code
is
much
closer
to
what
I'm
going
to
show
in
the
figure
in
the
next
slide.
Most
of
this,
these
teams
have
not
adopted
any
formal
development
model,
as
outlined
by
the
software
engineering
community,
the
development
model
they
all
have
some
kind
of
development
model.
Most
of
them
are
ad
hoc
and
again
device
based
upon
their
own
understanding
in
their
own
specific
needs,
the
closest
that
any
of
the
software
engineering
development
models
come
to
what
is
adopted.
B
What
is
used
in
the
scientific
software
community
is
the
agile
model,
because,
because
there
is
this
need
to
quickly
have
the
development
available
for
users
to
use
for
their
science,
but
of
the
projects
that
we
have
talked
about
so
far.
None
of
them
follow
it
explicitly
they
much
more.
They
all
of
these
are
much
more
adapted
from
the
models
that
are
available
in
the
viral
community
and
they
much
they're
much
more
responsive
to
the
needs
of
the
lifecycle.
B
So
this
is
what
a
software
lifecycle
looks
like
well,
it
starts
out
with
a
scientists
who
want
to
model
some
phenomena
in
the
physical
world
and
when
I
mean
physical
world
I
don't
mean
just
physics.
It
also
includes
biology
and
chemistry,
but
it
is
just
physical
word.
This
particular
lifecycle
is
indicative
of
the
modeling
of
physical
world
that
is
in
that
can
be
done
with
partial
differential
equations.
So
you
start
with
formulating
a
model
which
is
in
the
form
of
equations.
These
equations
are
reflective,
also
of
the
approximations
made
in
doing
the
modeling.
B
Sometimes
you
drop
a
term
because
it's
not
well
understood.
Sometimes
you
take
an
easier
equation
to
solve,
because
the
harder
equation
is
just
intractable,
given
the
resources
etc.
So
these
are
approximation
and
up
in
the
equations.
The
equations
are
then
discretized
and
gave
rise
to
difference
equations.
B
So
when
there
is
an
improvement
in
a
numerical
method
or
a
new
numerical
method
comes
along,
there
is
this
short
circuit
which
can
modify
the
equations
that
are
used
to
model.
Similarly,
an
implementation
may
be.
This
loop
comes
into
play
when
an
implementation
is
modified
because
of
a
certain
criterion
of
accuracy
and
stability
are
managed
and
the
validation
itself
can
give
rise
to
model
fidelity
which
can
in
turn
change
the
equations.
B
So
this
is
basically
what
a
typical
computational
science
lifecycle
looks
like
and
in
the
end
that
feeds
into
what
is
the
software
productivity
cycle,
which
is
which
is
also
available
from
the
ideas
productivity
website,
which
is
you
raise
the
scientific
questions
you
do
experimental
design.
This
design
can
be
a
physical
experiment
or
it
can
be
a
computational
experiment,
and
you
can
sometimes
make
use
of
this
after
development,
even
for
experimental
design.
B
So
you
model
and
do
the
software
development,
you
do
testing
validation,
you
do
simulations
analysis
and
validation,
and
the
internal
cycle
reflects
the
software,
the
execution,
the
workflow
and
the
analysis,
and
these
in
a
cycle
ultimately
end
up
in
scientific
discovery.
Again,
this
link
is
where
more
is
the
report
from
the
software
productivity
world
with
productivity
workshop
that
gave
rise
to
this
view
of
scientific
productivity,
and
that
is
the
end
of
this
section.
Do
we
have
questions
no.
A
A
B
The
next
section
is
on
some
of
the
best
practices,
as
we
have
collected
from
the
surveys
and
experiences
of
people
who
are
participating
in
building
this.
So
the
software
process,
baseline,
is
what
all
codes
should
do
insofar
as
it's
possible
within
their
resources.
Again,
I
want
to
re-emphasize
that
the
idea
behind
the
series
of
webinars
is
not
to
dictate
to
people
what
they
should
be
doing.
B
The
idea
behind
this
webinars
is
to
make
all
the
information
that
we
have
gathered
all
of
the
acquired
wisdom
available
to
the
community,
so
so
the
community,
the
the
practitioners
in
the
community,
can
may
pick
and
choose
the
knowledge
as
it
applies
to
them
and
adopt
it
for
their
own
purposes
and
even
evolve
and
give
us
back
the
information.
So,
but
one
thing
that
people
should
do
is
invest
in
extensible
code
design,
because
it
is
very
rarely
true
that
a
code
just
serves
one
purpose
and
finishes
there.
B
Even
if
you
start
out
by
assuming
that
that's
exactly
what
you're
designing
a
code
for
typically
others
will
find
use
for
it
and
it
will
grow
so.
Giving
some
thought
to
code
design
right
up
front
is
a
useful
thing:
it's
not
just
useful
for
extensibility,
but
also,
if
you,
as
you
divide,
designing
your
code.
B
If
you
give
some
thought
to
design,
you
also
side-by-side
develop
verification
process
for
that
code,
so
that
even
that
small
little
piece
of
code,
you
will
have
greater
confidence
in
the
reliability
of
results,
whether
you
are
doing
a
single
single
small
code,
development
or
a
large
code,
I
strongly
recommend
I
personally
strongly
recommend
using
version
control.
My
personal
data
is
also
managed
under
version
control.
Once
I
got
familiar
with
it,
there
was
no
way
I
was
going
to
live
life
without
that
convenience
in
life.
B
So
earlier
in
this
talk,
I
talked
about
technical
death
version
control
and
automated
testing.
You
can
think
of
almost
as
technical
saving
of
building
up
your
resources,
because
if
you
invest
in
doing
this
right
up
front
the
amount
of
time
its
gonna
save
you
down,
the
line
is
going
to
be
huge
because
every
time
you
add
a
little
bit
to
the
code.
B
If
you
have
an
automated
testing
built
in
there,
it
will
test
right
there
and
you
will
catch
a
defect
as
soon
as
possible
without
having
to
backtrack
a
great
deal
and
in
the
flash
Center.
We
found
this
that
anybody
who
came
and
became
a
part
of
the
flash
center
got
used
to
the
automated
testing
that
we
were
doing
when
they
left
the
center.
They
built
this
earth
hose
with
them.
It
just
went
with
them
and
they
instituted
it
wherever
they
went,
they
just
refused
to
work
without
it
so
and
any
non-trivial
code.
B
Any
two
code
that
that
is
planning
to
have
any
sort
of
longer
life
should
definitely
institute
a
rigorous
verification
and
validation
regime
right
in
the
beginning.
If
there
is
the
team,
defining
coding
and
testing
standards
is
absolutely
important,
because
everybody
in
a
team
has
their
own
different
coding
style
and
their
own
different
testing
style.
If,
if
you
do
not
have-
and
also
these
codes
are
typically
developed
in
environments
like
universities
and
academic
locations,
where
the
developing
population
is
sort
of
transient,
people
come
and
go
cry.
B
Students
graduate
postdocs,
move
on
to
permanent
jobs
and
see
if
their
code
cannot
be
understood
by
someone
once
they
have
left
that
part
of
the
code
cannot
be
maintained.
So
it
is
very
important
to
have
a
uniformity
in
coding
practices.
So
some
some
minimal
degree
of
coding,
standards
and
testing
standards
are
necessary.
It
is
also
well
very
important
to
have
well
have
well-defined
policies
for
auditing
of
the
code
and
the
maintenance
of
the
code,
whether
that
auditing
requires
making
sure
that
the
coding
in
coding
standards
are
met.
What
are
the
testing
practices?
B
B
How
much
of
the
code
is
released
to
the
wider
public
and
contribution
is
whether
the
code
is
going
to
be
taking
any
contribution
from
outside
contributors
and
if
so,
what
are
the
contributions
of
contributing
possible
inform
policies
and
I
will
touch
a
little
bit
on
this
towards
the
end
of
the
talk,
and
they
there
should
also
be
clear
and
well-defined
policies
for
documentation.
Now.
B
The
next
part
is
about
the
practices
that
are
desirable,
but
are
much
harder
to
achieve
and
have
not
sort
of
penetrated
as
much
in
the
community
and
one
of
the
an
important
one
is
provenance
and
reproducibility.
So
by
now,
the
practice
of
computational
science
has
achieved
enough
of
what
I
would
call
validation
as
a
scientific
discovery
tool
that
more
and
more
the.
B
Reliability
of
results
as
it
exists
in
other
forms
of
scientific
discovery
like
experimentation
and
theory
and
provenance,
is
an
important
aspect
of
it,
because
provenance
lets
you
go
back
and
practice
and
reproduce
the
results.
This
is
reflected
actually
in
even
in
this
year's
supercomputing
called
for
papers
where
people
were
we're,
the
submitters
were
offered
a
chance
to
participate
in
a
reproducibility
drive
where
you
could
they
could
designate
if
that,
if
I
mean
they
could
indicate.
If
that
paper
was
going
to
participate
in
this,
and
there
is
a
question.
A
So
yes,
there
I
will
read
it.
A
number
of
sociological
challenges
often
set
the
tone
for
software
development.
For
example,
a
lot
of
projects
have
only
one
or
two
FTEs
for
the
software
development
and
often
split
across
multiple
people
who
are
split
on
multiple
projects
and
that
comes
from
multiple
organizations.
Many
of
the
traditional
software
engineering
practices,
such
as
agile
work
because
developers
are
a
hundred
percent
on
one
project.
Do
you
have
suggestions
on
how
to
operate
a
successful
software
engineering
team
in
such
circumstances,.
B
The
answer
is:
no
I
am
actually
personally
facing
this
issue
right
now,
in
that
I
am
on
three
different
projects,
each
one
of
which
has
their
own
agile,
like
not
quite
but
agile,
like
development
and
I'm
having
a
hard
time
juggling
it.
So
if
someone
in
the
community
has
an
answer
to
this
problem,
I'd
be
very
happy
to
hear
about
it
and
actually
I
think
it
can
benefit
the
whole
community.
B
Coding
standards,
for
example,
so
explicit
examples.
The
easiest
one
to
give
on-the-fly
for
me
is
from
the
flash
code,
because
that's
the
one
I'm
most
familiar
with
so
in
the
flash
code,
it's
basically
in
Fortran
90.
So
there
is
not
a
whole
lot
of
Lang
language
based
checking
of
things
that
you
could
do
so
what
what
we
did
in
coding
standards.
B
What
we
said
was
every
component
would
have
its
API
and
its
API
would
be
defined
in
an
interface
module
of
Fortran
90,
and
this
is
required
of
everybody,
because
then
what
happens
is
if
you,
if
so
compilers,
are
ignore
mismatch
in
arguments.
If
you
don't
have
a
an
interface
block,
but
if
you
have
an
interface
block,
the
compiler
will
catch
it.
So
this
was
one
coding
standard.
Other
coding
standards
had
to
do
with
naming
conventions
so
that,
if
you
followed
the
convention,
you
could
immediately
tell
where
a
variable
came
from
other
coding.
B
Standards
were,
let's
see,
P,
because
Fortran
allows
you
to
have
on-the-fly
definition
and
implicit
typing
of
variables.
We
adopted
the
standard
that
there
should
be
implicit
none
and
if
you
were
using
data
modules
that
you
could
only
use
them
by
with
the
only
construct
which,
if
you
are
trying
to
use
a
variable
in
a
module
that
you
haven't
explicitly
called
out
in
where
you're
using
it,
it
would
be
immediately
caught.
A
B
Is
absolutely
true?
Yes,
that
is
another
advantage
of
coding
standards,
so
I
think
I
was
done
with
this
slide
right.
Yes,
I
was
done
with
the
slide,
so
also
many
of
these
practices,
as
I
said
earlier,
will
be
covered
in
much
greater
detail
later
in
the
series,
so
I'm
not
going
to
elaborate
upon
them
a
whole
lot
more.
B
A
useful
resource
for
collect,
collect
and
for,
if
you're,
looking
for
best
practices
is
this
link
here,
which
is
the
how
to's
collected
by
the
ideas
productivity
project.
So
there
are
two
kinds
of
projects
in
this:
one
is
what
is
Doc's,
which
are
the
characterizations
of
important
topics
in
computational
science
and
engineering,
and
the
how
to
dogs,
which
are
the
brief
sketches
of
best
practices,
basically
that
easy
to
glance
through
and
for
people
to
quickly
get
information
about
them
again.
These
are
work
in
progress.
B
Even
the
ones
that
have
been
published
in
some
sense
are
working
progress
in
the
sense
that
they're
not
written
in
stone.
We
are
actively
seeking
feedback
from
the
community,
not
just
in
terms
of
modification
of
these
documents,
but
if
people
tell
us
that
some
specific
documents
will
be
very
helpful,
please
get
in
touch
with
us
at
this
project
and
we
will
be
incorporated.
B
A
A
Crime
was
one
one
comment
or
one
one
statement
question
that
came
up
was
just
an
addition
to
that
list
was
a
statement
that
I've
been
very
impressed
by
the
software
engineering
process
employed
by
the
Trevino's
project
and
truly
no
San
Diego.
If
people
don't
know
that
it's
tr
I
ll
I
own
OS,
we
will
include
that
elsewhere,
but
the.
B
Person
who
is
going
to
talk
about
the
testing
and
documentation
is
actually
a
representative
of
is
a
member
of
the
truly
knows
project.
So
I,
you
will
get
more
examples
from
truly
knows
in
back
duck.
I'm
sure
see
we
have
collected
people
from
various
different
projects,
so
every
one
of
these
presentations
will
be
somewhat
tilted
in
terms
of
its
examples
towards
the
project
from
which
a
particular
person
came.
It's
only
natural
because
that's
when
that's
the
one
you
know
the
best.
B
So
when
you
are
looking
at
software
engineering
practices
and
you're
trying
to
adopt
them
for
yourself
up
for
yourself
here
is
a
general
set
of
guidelines,
best
prints,
best
practices
or
whatever
one
might
want
to
call
them.
There
is
no
all
or
none
you
don't
have
to
either
do
the
practice
or
not
at
all,
just
use
as
much
of
it
as
is
useful
for
you
and
ignore
the
rest.
B
So
the
focus
in
adopting
a
practices
practice
should
be
absolutely
on
improving
improving
productivity
rather
than
any
purity
of
process,
and
it
I
mean
we
burn,
runs
against
people
who
emphasize
the
purity.
Much
more
cannot
accept
that
a
practice
can
be.
You
know
that
you're
not
following
a
practice
as
it
is
designed.
There
is
no
such
thing
in.
You
should
absolutely
adopt
it
for
yourself,
but
at
the
same
time,
I
would
like
to
caution
that
there
is
people.
B
The
first
knee-jerk
reaction
is
to
be
dismissive
of
things
that
you
that
are
getting
introduced
to
you
that
are
likely
to
place,
require
some
mental
energy
from
you.
So
there
is
a
danger
in
being
too
dismissive
too
soon,
because
it
may
be
that
if
you
examine
that
practice
without
a
bias
and
with
open
mind
that
you
might
actually
find
that
practice
to
be
very,
very
useful
and
I
knew
I
had
an
example
in
mind
at
the
moment.
I
can't
remember
it.
If
I
do,
I
will
give.
A
It
because
it's
completely
lined
up
with
that
another
question
in
scientific
research.
We
want
to
get
some
quick
results
to
make
sure
that
we're
on
the
right
track.
If
we
spend
an
enormous
amount
of
I'm
up
front
on
software
processes,
programming
can
be
a
distraction
from
science
and
we
may
not
know
what
we're
what
that
we
are
on
the
wrong
track
until
a
lot
of
time
has
already
been
spent
on
programming
so
shouldn't
there
be
a
compromise
between
the
software
process
and
testing
in
the
beginning
phase
of
co-development.
So.
B
B
There
is
a
fine
balance
between
getting
a
buy-in
from
the
team
and
imposing
a
process
on
them,
and
this
again
we've
had
experience
in
flash
that's
many
times.
People
from
hardcore
engineering
and
science
will
come
in
and
they
see
the
amount
of
process
we
have
in
place
and
at
first
rebel
and
say
why
are
we
may
be
made
to
waste
this
time?
But
fortunately
that's?
There
is
a
team
and
there
are
others
like
them.
So
there
what
they
find
is
slowly.
B
Scientists
do
and
comparable
resources
and
autonomy,
at
least
a
resources
comparable
to
what
is
needed
by
the
developers
are
given
to
them
and
their
contributions
are
recognized
and
there
is
a
careful
balance
between
long
term
and
short
term
objectives.
In
all
of
this,
it
is
extremely
helpful
to
have
people
that
either
have
multidisciplinary
training
or
are
willing
to
learn
about
the
other
discipline,
because
they
become
the
ambassadors
for
interdisciplinary
communication
between
teams.
So,
and
that
completes
this
section,
any
other
questions
that
are
pending
before
I
go
into
Community
Development.
Oh
no,.
B
Okay,
so
the
last
section
of
my
talk
is
on
Community
Development.
Why
community
codes
community
codes
are
useful
because
scientists,
if
there
is
a
community
code
available,
scientists,
can
focus
on
developing
their
algorithmic
needs
instead
of
getting
bogged
down
by
the
infrastructure
development,
and
this
goes
back
to
one
of
the
earliest
questions
that
had
come
about
why
a
loan
scientist
can't
do
make
use
of
LCF
loan
site
scientists
absolutely
make
use
of
is
the
leadership
class
facilities.
B
There
was
an
example
at
the
flash
Center
who
had
taken
the
flash
board,
which
was
designed
for
type
1a,
supernovae
added
a
whole
bunch
of
physics
that
was
necessary
for
him
to
do
his
research
and
gone
on
to
run
simulations
and
get
good
results.
But
the
point
is
that
he
still
relied
upon
Flash
and
all
of
the
capabilities
that
were
already
built
into
flash.
If
he
had
to
do
the
same
thing
from
scratch.
There
is
no
way
that
he
would
have
been
able
to
successfully
do
his
research.
B
So
in
these
communities,
where
there
are
community
codes,
what
happens
is
graduate
students
do
not
start
developing
codes
from
scratch,
which
means
that
they
can
get
become
invested
in
the
research
very
early
on
in
the
cycle.
They
just
look
at
a
Bell
Bell
public
code
and
figure
out
which
ones
meet
their
needs.
The
most
they
look
at
the
effort,
effort
of
customization
and
based
on
all
of
these
criterion.
They
select
a
public
code
and
build
upon
it
as
they
need
now.
B
So
it
is
important
that
they
still
need
to
understand
the
components
that
are
developed
by
other
studies
that
they're
using
they
should
know
their
limitations
and
the
range
of
validity,
and
but
in
particular
they
don't
have
to
worry
about
details
like
infrastructure,
solvers
IO
that
are
too
well
understood
to
have
any
research
component,
but
are
time-consuming
to
implement
right.
So
they
get
disadvantaged.
Students
are
not
the
only
ones
that
get
advantage
from
this.
The
researchers
can
build
upon
work
of
others
and
get
further
faster.
Instead
of
reinventing
the
wheel.
B
It
gets
stressed
in
different
ways,
and
so
code
just
becomes
better
and
more
robust,
as
time
goes
on.
Also
open
science
results,
open
source
science
results
in
more
reproducible
results,
and
that
is
generally
good
for
the
credibility
of
science
through
computation,
so
I'm,
taking
a
closer
look
at
the
astrophysics
community.
For
two
reasons,
one
is
that
it
is
one
of
the
communities
that
was
an
earliest
adopter
of
the
community
code
model
and
second
of
all,
I
have
been
particularly
close
to
this
community
myself.
B
So
this
is
a
natural
community
for
me
to
take
take
as
an
example,
so
this
community
had
an
early
culture
of
releasing
research
software
and
they
started
doing
it
as
early
as
the
80s,
where
the
n-body
codes
for
gravitational
interactions
first
became
public
and
then
zoo,
studi
added
hydrogen
Amyx,
which
then
later
became
also
zoos,
3d
and
smooth
particle
hydrodynamics,
quartzite,
hydra
and
gadgets
are
other
quotes
that
have
since
come
into
being
over
time.
These
public
codes
became
more
and
more
sophisticated.
B
For
example,
mr
appeared
in
flash
in
the
early
2000s
and
then
later
shot
capturing
image
de
radiation,
hydro,
etcetera
also
started
to
appear,
and
here
is
the
reason
why
astrophysics
community
was
quicker
to
adopt
community
code
model.
It's
because
if
you
look
at
the
figures
that
I'm
showing
every
one
of
them
has
many
modules
of
physics,
some
modules
of
physics
are
common
among
all
of
these
codes,
but
many
more
are
different.
So
what
happens
exceeds
is
many
different
components
in
the
code.
They
need
need
mesh
methods
for
gas
dynamics,
gravitational
potential
radiation.
B
They
need
particle
methods
for
tracing
or
massive
particles,
foreign
body
interactions.
They
need
the
zero
dimension
or
point
wise
calculation
for
all
kinds
of
equations
of
state
in
source
terms.
They
also
because
they
have
multiple
scales.
They
also
need
adaptive.
Mesh
refinement
for
data
and
computation
compression,
and
so
development
of
such
complex
codes
is
beyond
the
resources
of
individuals
or
even
small
groups,
and
community
codes
are
the
only
solution.
What
about
other
communities?
If
you
examine
communities
that
have
a
community
code
model?
B
What
is
almost
always
true
in
them
is
they
need
multi
physics
and
and/or
multi
scale
they
all
started
have
have
had
a
visionary
that
saw
the
benefit
of
software
use
and
released
the
code
and
caused
this
movement
to
start
and
in
all
of
these
communities.
Their
benefits
they
reap
is
that
the
sophistication
in
modeling
advances
much
more
rapidly
than
the
communities
that
keep
their
codes
close.
B
B
So,
let's
examine
what
takes
what
does
it
take
to
build
a
community
code?
Community
code,
just
popularizing
a
code
doesn't
build
a
community
because
they'll
just
be
users
and
neither
does
customizability,
because
different
users
want
different
capabilities.
So
what
is
important
is
enabling
contributions
from
users
and
providing
support
for
them,
including
policy
provisions
that
can
balance
IP
there
with
open
source
needs
and
relaxed
distribution
policies,
because
the
bottom
line
is
that
the
more
in
Tolosa
VT
is
there
is
in
a
code.
B
There
is
a
greater
success
in
building
community,
an
investment
in
robust
and
extensible
infrastructure
and
a
strong
culture
of
user
support
is
absolutely
necessary
and
now
I
go
on
to
three
examples:
flash
Enzo
and
whitey.
They
have
three
different
development
models
and
community
building
models.
So
flash
was
Flash's
developed
under
sustained
funding
from
the
ASC
Alliance
program,
funded
by
the
NNSA
Dewey.
One
of
the
mandated
outcomes
from
that
was
a
publicly
released
code.
B
B
The
second
takeaway
is,
so
let
me
go
back
to
that
slide.
So
what
worked?
What?
What
worked
here
was
that
in
the
first
iteration
it
was
a
code
that
was
just
smashed
together
and
there
were
only
a
very
few
people
that
could
work
with
that
code
effectively
and
get
science
done
so.
Fortunately,
the
center
had
resources
to
start
doing
one.
We
were
cleaning
up
of
the
code
and
turning
it
into
a
much
better
structured
code,
and
that
exercise
was
undertaken
in
the
process.
B
The
code
development
developers
had
internalized
some
of
the
science
needs,
and
so
they
were
more
effective
in
building
it,
and
the
third
iteration
is
the
time
when
both
parties
had
come
to
understand
the
need
of
the
other
side
much
better,
and
there
was
much
higher
degree
of
collaboration
in
building
the
code.
So
the
answer
is
no
one
way
of
doing
development
is
a
clear
answer
for
any
code.
However,
even
for
doing
a
quick
scientific
development,
giving
some
thought
to
future
is
always
a
good
idea,
though
one
should
always
be
prepared.
B
A
B
B
So
at
the
moment
flash
has
flash
collects
a
database
of
people
that
had
papers
that
are
reported
by
the
users
of
code
themselves.
In
that
database
there
are
1,100
publications
now
I'm,
not
saying
that
that's
the
exact
number.
It
may
be
an
overstated
number,
but
it
gives
a
rough
idea
of
the
kind
of
impact.
Flash's
development
has
had
very
little
modifications
to
be
too
basic.
Basic
infrastructure
was
needed
to
accommodate
those
capabilities
and
any
new
capability
that
is
added
proves
to
be
useful
to
other
communities.
B
Also,
for
example,
radiation
was
added
for
high
energy
density
physics,
but
it's
equally
frequently
used
by
people
in
astrophysics
and
cosmology.
Flash
most
force
mostly
follows
a
cathedral
model,
so
this
is
from
this
book
by
Eric
I.
Don't
remember
the
last
name
at
the
moment
on
two
models
of
development
Cathedral,
where
there
is
a
strong,
gatekeeping
and
bazaar
where
everyone
can
contribute
anything
and
Flash
is
operating
with
the
cathedral
model.
Where
the
center
does
the
gatekeeping
community
building
took
several
years.
B
It
started
out
by
getting
better
interactions
between
the
center
scientists
and
the
developers
the
alumni
of
the
center
of
the
culture
encode
with
them,
and
in
addition
to
that,
many
on-site,
tutorials
and
tutorials
at
scientific
conferences,
helped
the
as
code
became
more
robust
and
mature
more.
It
also
became
more
inclusive
and
started.
Taking
contributions
from
external
contributors
and
customizability
also
helps
the
greatest
impact,
though,
was
in
relative
ease
in
getting
started.
Customized
ability
and
reliability
ENSO
transition
from
closed
to
open
source,
so
ENSO
is
a
cosmology
code.
B
It
started
as
a
close
quote
from
1996
to
2003.
Its
first
public
release
was
in
March
2004.
Now
it
has
moved
very
close
to
the
bizarre
model
where
the
developers
are
scattered
among
many
institutions,
and
they
write
their
science
proposals
and
in
that
they
make
provision
for
development
resources
and
combined
together
than
they're
able
to
keep
it
going.
It's
an
entirely
distributed
development
model
with
small
number
of
develops
developers
per
inches
per
institution.
We
they
use
code,
forks
and
pull
request
to
move
features.
B
Almost
all
discussion
happens
on
our
pipe
public
mailing
list
and
on
Google
Docs.
It
has
its
own
challenges
in
that,
because
there
is
no
leader
it's
hard
to
make
major
code
revisions.
The
developers
are
part-timers,
so
they're
distractions
and
significant
work
is
required
to
build
consensus
and
keep
community
together.
B
In
contrast
to
both
of
these
is
YT,
which
is
a
project
that
started
out
by
being
open
right
from
the
beginning,
and
it
sought
to
build
a
community
of
builders,
and
this
was
the
spearheaded
by
Matt
Turk,
and
if
you
get
a
chance
to,
he
gives
a
really
great
talk
about
sociology
of
code
development
I
think
it's
available
somewhere
on
the
web.
Also,
you
should
absolutely
see
it.
It's
a
fabulous
talk,
but
so
whitey
is
the
third
part
of
this
community,
where
it's
the
newest.
B
So
it
is
also
true
that
the
the
codes,
the
three
of
them,
reflect
different
time
frames
in
the
development
flash
was
early,
2000s,
late,
1990s,
early
2000s
and
so
became
public
in
2003
and
by
the
time
it
became,
went
into
a
bizarre
mode.
Many
much
more
of
distributed
version
control
was
available
and
whitey
is
a
is
a
much
newer
project
and
so
right
from
the
beginning,
and
it
adopted
many
more
software
processes
that
the
other
codes
had
not
done.
B
And
that
brings
me
back
to
the
summary
that
brings
me
to
the
consumer
of
community
codes
they're
their
most
open
source
with
governance
structure
in
place.
They
invest
in
trust,
building
among
themes.
They
are
committed
to
transparent
communications.
They
have
strong
commitment
to
user
support.
They
are
they
either
have
an
interdisciplinary
team
or
a
group
of
people
that
are
comfortable
with
both
science
and
code
development.
They
give
attention,
pay
attention
to
software
engineering
and
documentation
and
understand
the
benefit
of
sharing,
as
opposed
to
being
secretive
about
their
codes.
B
They
pay
attention
to
contributing
policies
where
they
balance
the
country
and
distribution
needs.
They
have
maintainable
code
requirements.
What
they
ask
for
from
users
is
source
code,
bill,
scripts,
test,
documentation,
abdomen
done
user
support
and
add
on
slight
components,
not
included
with
distribution,
but
work
with
the
code.
So
in
conclusion,
there
are
many
reasons
why
software
engineering
practices
are
good
and
should
be
encouraged,
because
science
and
engineering
community
by
simulation
needs
more
scrutiny
into
its
methods
and
software
if
it
has
to
become
the
gain.
The
same
kind
of
credibility
that
experiment
and
theory
shion's
do.
B
There
is
no
need
to
keep
reinventing
the
wheels
so-
and
this
is
especially
true
a
bookkeeping.
So
it's
much
better
to
reuse,
infrastructural
components.
The
days
of
heroic
programming
are
well
I.
Shouldn't
say
entirely
passed,
but
the
that
is
productive
with
very
few
and
far
between
projects
and
these
kind.
The
software
engineering
practices
are
indispensable
for
extreme
scale
computing.
B
It
is
extremely
important
to
recognize
that
science
through
computing
is
only
as
good
as
the
software
that
produces
it
and,
as
last
thought,
I
want
to
leave
you
with
this
example
that
comes
thread
straight
from
flash
in
2005,
a
blue
gene/l
platform
was
made
available
to
us
to
run
a
large
simulation
at
a
very
short
notice.
We
needed
particles
trace
of
particles,
capabilities
which
did
not
exist
at
that
level
in
the
core
at
that
time.
So
we
did
a
very
quick
and
dirty
development.
B
When
the
simulation
started
running.
There
were
very
many
in-flight
correction
of
defects
and
we
caught
quite
a
few,
but
we
did
not
catch
all
of
them.
One
of
them
was
simplification
of
the
data
structure.
Where
tags
that
identify
individual
particles
were
encoded
in
the
form
of
single
precision
floating
point
numbers,
we
hadn't
paid
attention
to
the
fact
that
they
were
enough
particles
and
that
when
they
got
converted
to
integers
they
did
roundoff.
So
he
had
duplicated
particle
tags.
B
When
we
started
to
do
scientific
analysis,
we
had
to
develop
tools
that
could
differentiate
between
two
tags,
that
had
particles
and
it
took
things
like
tracing
their
history
to
determine
which
particles
should
get
which
tag
and
for
that
entire
project.
It
took
the
flash,
even
though
flash
already
had
a
software
process
in
place
and
it
was
tested
regularly.
This
was
one
instance
when
we
could
not
follow
the
full
process,
so
we
got
ready
for
the
run
in
one
month.
The
run
itself
took
a
week
to
week
and
a
half
with
many
on
fly-in
flights
Corrections.
B
A
B
B
So
at
least
one
test
that
could
be
included
in
the
test,
but
even
before
the
contribution,
there
was
a
pre-negotiated
time
when
the
contributed
code
would
not
be
included
in
public
release.
So
but
at
the
end
of
that
pre-negotiated
time,
the
code
would
be
included
in
the
public
release,
because
otherwise
flashy
flash
developers
had
no
interest
no
interest
in
actually
helping
maintain
that
code.
So
in
this
pre-negotiated
time
the
code
was
maintained
by
the
flash
Center,
it
was
left
to
the
use
by
the
contributor
and
anybody
else
that
that
contributor
gave
access
to.
A
Of
it
very
related,
but
I'll
ask
in
case
it
inspires
anything
different
was.
What
do
you
recommend
is
a
balance
between
making
code
open
source
versus
maintaining
some
sort
of
I
pre
protect
protection
on
the
code
I
mean
this
seems
very
consistent
with
what
you're
discussing
about
that
the
window,
but
right.
B
So,
while
look
before
the
questions
come
through,
I
want
to
say
that
the
response
that
this
webinar
series
has
generated
so
far
has
been
overwhelming,
and
we
are
so
pleased
and
pleased
if
you
think
that
you
can
give
any
feedback
that
will
improve
this
further,
which
will
be
particularly
helpful
to
you.
Please
feel
free
to
write
to
me
or
to
anybody
else
in
this
team,
and
secondly,
this
is
not
the
end
of
interaction.
B
A
A
You
also
might
see
that
in
the
chat,
the
actual
chat,
not
QA,
but
we
can
also
reach
out
and-
and
let
you
know
when
those
are
posted
and
and
again
following
on
the
phone
she
says
and
what
is
online,
we
really
welcome
feedback
we'd
really
in
you
know.
We
see
this
as
a
way
to
help
the
larger
community
and
so
understanding.