►
Description
Reproducibility is considered one of the cornerstones of science: to count as scientific, an observation or prediction has to be verifiable by others. However, over the last 20 years, researchers in many disciplines of science found that a large number of published scientific studies is not reproducible. This has been called the ""reproducibility crisis"".
Konrad will start by illustrating this phenomenon using a few examples, and explain the double role that computing has played in the crisis. Then Konrad will provide a deeper analysis of computational reproducibility, one of the three categories relevant for science. Technically, it is the same problem as reproducible builds, and that means that functional package management as implemented in Nix and Guix is an important part of a solution. However, both Nix and Guix lack some tooling and documentation to make them useful to and acceptable by a wider community of researchers.
Special thanks to the NLnet Foundation, the European Commission, the NixOS Foundation, and Tweag for making this event a reality!
The continued discussion for the lecture series is happening over here:
https://matrix.to/#/#son2022-lectures:matrix.org
More information about the Summer of Nix can be found on the website:
https://summer.nixos.org
A
A
I
will
go
through
the
usual
again
I'll.
Keep
it
short,
because
there's
some
people
that
I
know
of
that
are
even
more
excited
than
I
am
so
thank
you
to
the
announcement
Foundation,
the
European
commission
and
excess
foundation
and
tweak
for
making
these
lecture
series
happen
and
making
them
be
the
way
that
they
are
one
interesting
side.
Note
that
I
am
excited
about
as
well
about
infrastructure
is
donkey's
instance
should
be
faster
today.
If
not,
please
let
me
know,
but
let's
hope
everything
goes
well
on
that
end.
B
B
B
I've
used
both
I've
used
Nyx
in
the
past,
when
I
buy
main
platform
was
Macos
a
while
ago,
I
switched
to
Linux
and
now
I'm
more
in
the
in
the
Geeks
world,
for
the
simple
reason
that
I
have
a
scheme
history
and
that's
more,
that's
nicer
for
me,
but
I
I.
Think
for
me.
Both
of
these
are
very
nice
systems
and
they're,
pretty
much
equivalent
for
the
purposes
of
reproducible,
Science
and
actually,
as
long
as
both
continue
and
still
good
ideas
from
each
other
I
think
everybody
is
going
to
gain.
B
So
what
I
will
talk
about?
It
will
be
will
apply
to
both,
but
my
personal
experience
is
more
up
to
date
with
geeks.
Okay,
so
far,
now
back
to
science,
so
reproducibility
is
an
old
Topic
in
science.
It
goes
back
to
the
good
old
days.
Institutionalized
science
has
started
in
the
17th
century.
The
first
scientific
institution
in
the
world
was
the
Royal
Society
of
London
for
improving
natural
knowledge,
which
was
founded
in
17th
century
and
in
1663
to
be
precise
and
on
its
logo,
and
it's
it
shows
it
has.
B
The
the
the
its
motto
was
always
nulios
in
verba,
which
means
don't
take
anybody's
word
for
it
and
the
idea
there
was
that
if
you
want
to
contribute
to
doing
science,
you
can't
just
claim
something
you
have
to
show
evidence
and
you
have
to
open
it
up
for
inspection.
It's
not
not
enough
to
say
I
am
an
honest
person.
Trust
me.
B
B
There
is
a
more
subtle
issue
with
I've
tried
to
formulate
it
diplomatically
that
personal
biases
are
often
in
conflict
with
a
search
for
truth.
So
this
is
things
like
falling
in
love,
with
your
own
Theory,
having
a
personal
interest
from
some
Theory
gaining
compared
to
another
and
so
on,
and
so
on,
and
the
scientific
method
I
mean
from
engineering
point
of
view
could
be
considered
a
narrow
correction
protocol.
So
you
have
all
these
people
doing.
B
Research,
publishing
research
and
all
this
is
may
be
wrong
for
various
reasons,
and
then
you
have
something
on
top
of
that
called
the
scientific
method,
which
is
a
social
process
in
which
the
individual
errors
and
biases
are
eliminated
by
continuous
peer
verification.
So
everybody's
looking
over
everybody
else's
shoulder
and
the
idea
is
that
in
in
this
way
we
will
figure
out
what's
wrong
and
the
right
things
will
will
survive.
B
So
I
can't
go
around
and
say
I've
seen
a
flying
saucer
tonight.
So
therefore
flying
sources
exist.
What
I
have
to
say
is
while
every
Monday
morning,
if
you
stand
at
that
place
on
the
Eiffel
Tower
and
look
in
that
direction,
you'll
see
a
flying
sources.
So
then
everybody
else
can
can
join
in
and
then
check
and
have
an
opinion
on
that
and
we
could
start
a
discussion.
B
There
are
many
subtle
details
which
I'll
skip
over
so
of
course,
if
you're,
an
astronomer
and
you're
watching
supernovas,
they
don't
happen
twice.
So
not
everything
is
reproducible,
but
that's.
The
general
guideline
is
that
as
far
as
possible,
you
should
aim
for
things
to
be
reproducible,
so
that
was
the
idea
300
years
ago,
now,
three
centuries
later,
what
is
going
to
happen?
We
are
in
the
reproducibility
crisis
or
as
Wikipedia
calls
it
the
replication
crisis.
So
let's
still
do
the
same.
For
now
so
I'll
come
into
go
to
the
jargon
details
later
on.
B
So
what
is
what
happened?
That's
an
ongoing
mythological
crisis
that
sounds
serious
and-
and
it
actually
is
in
some
way,
I'll
give
you
a
couple
of
examples
to
illustrate
what
happens,
but
what
happens
and
then
some
of
them
you'll
see
that
Computing
actually
plays
a
role,
and
that's
why
I'm
talking
here
today,
I'll
I'll
start
with
the
most
recent
case
in
the
in
this
crisis
that
has
been
has
made
a
lot
of
noise
last
year.
B
That
was
a
big
study
to
check
the
replicability
of
cancer
biology.
So
the
these
these
people
got
a
lot
of
money.
A
lot
of
people
were
to
work
on
this
to
to
redo
experiments,
crucial
experiments
for
understanding
how
cancer
works
and
how
it
can
be
treated
and
I
found
roughly
that
half
of
them
could
not
be
repeated
successfully
so
that
either
they
could
not
be
repeated
because
something
was
missing
or
they
could
be
repeated,
but
the
result
was
not
the
same.
B
B
It's
also
closer
to
my
own
work.
This
topic
is
NMR
chemical
shifts.
Nmr
is
a
technique
for
investigating
the
structure
of
molecules
using
something
called
chemical
shifts,
which
is
a
phenomenal
natural
phenomenon,
and
this
is
a
standard
technique.
The
many
Labs
have
the
equipment
and
Gathering
data
and
interpret
the
data.
B
A
couple
of
years
ago,
on
some
team
published
in
a
journal
called
nature
methods,
a
protocol
finalizing
data
with
Python
scripts
to
do
the
work,
and
the
idea
was
to
make
people's
life
easier,
so
they
could
just
run
the
scripts
and
didn't
have
to
to
worry
about
the
technical
leaders
themselves,
so
many
people
jumped
on
that
and
did
it
and
then
two
years
ago
or
three
years
ago
now
some
people
found
a
mistake
in
this.
Not
a
mistake,
it's
actually
it's
more
subtle.
B
B
We
have
no
idea.
So
this
is
the
case
which
is
close
to
the
topic
of
today.
There
is
another
topic
where
Computing
plays
a
role
in
a
slightly
less
obvious
way.
That's
a
nice
example!
It's
you
can
look
it
up.
It's
a
free
article
from
physics
today,
a
very
nice
summary
of
over
a
seven
year
dispute
heavy
dispute
of
people
shouting
at
each
other
about
supercooled
water.
Well,
that's
the
topic
doesn't
really
matter
both
groups
opposing
each
other.
B
Here
we're
doing
computer
simulation
studies,
each
of
them
had
written
their
own
software
and
there
were
a
principal
calculating
the
same
thing.
Also
they
thought,
but
they
got
contradicting
results.
The
question
was:
how
many
faces
does
super
good
water?
Have?
Is
it
two
or
three
and
one
set
two,
the
other
set
three
and
they
basically
shouted
at
each
other
said
you're,
you're,
stupid
idiots
and
so
on,
and
so
on.
B
The
issue
was
solved
and
it
one
side
at
some
point
published
their
code.
The
other
side
never
did
and
at
some
point
they
made
a
strategical
mistake
by
promising
to
pop
to
to
to
to
send
a
code
on
request,
and
they
probably
didn't
mean
to
do
that,
but
that
was
too
late
and
the
others
had
jumped
on
this
occasion
said:
oh
yeah
I
will
request
it
formally.
B
So
now
we
can
look
at
it,
and
that
was
the
occasion
when
somebody
could
actually
compare
the
two
pieces
of
software
which
are
quite
complex
and
figured
out
that
they
would
actually
not
do
the
same
and
that
one
side
had
made
a
methodological
Choice
which
was
supposed
to
be
a
shortcut
for
optimization
and
which
introduced
wrong
results.
Now
the
journalists
got
it
wrong
in
the
title
it
says
it
was
a
coding
error.
There
was
no
coding
error,
it
was
a
methodological
Choice
which
the
people
considered
a
technical
detail.
B
It
existed
only
in
the
code
and
was
never
mentioned
in
any
publication.
So
before
the
code
got
out,
nobody
knew
what
was
actually
happening,
and
that
is
actually
a
very,
very
frequent
situation,
even
if,
if
it
results
rarely
in
a
seven
year
dispute
this
one
also
is
a
bit
scary.
Just
I
mean
just
the
title
of
the
paper
should
be
enough
to
say
where
the
problem
is
so
there's
some
there's.
B
This
group
figured
out
that
the
the
analysis
of
a
certain
type
of
experimental
data
actually
got
very
different
results
depending
on
on
Plenty
of
plenty
of
software
version
numbers.
And
of
course
you
don't
want
a
scientific
result
to
depend
on
the
software
version
number
something
is
wrong
there,
and
this
is
again.
This
is
a
very
frequent
situation,
even
though
people
rarely
talk
about
in
the
tablet,
publish
it.
So,
let's
take
a
step
back.
B
So
you
don't
have
the
same
patients
as
your
your
colleagues
across
the
Atlantic,
so
you
necessarily
change
the
sample
and
the
questions
does
this
still
give
you
similar
results
sufficiently
close
to
be
credible
and
the
third
type
is
computation,
rapid
visibility
and
the
other
issues?
Can
you
read
your
computation
and
get
identical
results?
B
So,
as
you
will
guess,
my
topic
is
mainly
the
last
one
and
if
you
look
at
the
ongoing
reproducibility
crisis
and
all
the
examples
that
are
being
discussed
and
studied
and
experimental
reproducibly
just
doesn't
happen,
it's
not
in
the
list.
It
never
happens.
It
doesn't
mean
that
it's
not
no
longer
an
issue,
but
it's
something
that
people
have
been
aware
for
300
years.
They
know
that
sometimes
reproducibility
is
not
there.
It's
not
easy
to
achieve
for
good
reasons
and
and
that
talk
about
it
when
they
handle
it.
B
The
two
others
actually
recent
and
that's
why
they
caused
the
crisis.
Basically,
statistical
reproducibility
is
recent,
because
actually
statistics
and
science
is
recent,
because
it
could
only
be
done
with
computers.
Nobody
does
statistical
inference
techniques
by
hand,
it's
just
not
not
doable
and
computational.
Reproducibility
is
obvious
that
it
depends
on
computers
and
it's
in
a
sense,
the
most
surprising
one
because
well,
computation
is
supposed
to
be
deterministic
procedures.
B
Finally,
it's
a
comment
on
jargon.
So
there's
reproducibility
and
replicability
I
think
the
dominant
use
of
the
two
terms
is
the
one
I'm
going
to
explain
here
today.
Unfortunately,
there's
people
who
use
them
exactly
the
other
way
around,
so
they
exchange
the
meaning
of
the
two
terms.
So
that's
that's
unfortunate.
That's
life
so
reproducibility
as
as
a
technical
level,
it
is
a
verification
in
engineering,
as
opposed
to
validation.
So
the
questions
posed
are:
has
the
procedure
be
executed
correctly
as
described?
B
It
is
pretty
straightforward
and
it
gives
a
simply
answer.
Is
this
yes
or
no,
whereas
replicability
is
is
a
scientific
question
that
is,
is
the
procedure
that
has
been
described
the
the
right
thing
to
do
to
answer
the
scientific
question.
It
is
difficult
because
it
cons,
it
requires
scientific
judgment
and
the
answer
is
typically,
if
maybe
conditional.
B
So
it's
it's
not
the
same
thing
and
in
a
specific
context
of
scientific
Computing,
reproducibility
tests
involve
running
the
same
software
on
the
same
parameters
and
the
same
data
and
check
if
you
get
identical
results,
so
essentially
you're
doing
deterministic
operations
and
we
check
if
we
have
really.
If
we
really
everything
is
the
same,
whereas
replicability
studies
change
something
they
use
different
software,
usually
the
same
parameters
and
same
data,
sometimes
not
depending
on
the
situation
and
check
if
the
result
is
equivalent.
B
So
reproducibility
is
same
software
same
data,
same
parameters,
same
everything,
and
why
do
you
do
such
checks?
As
I
said?
If
it
comes
out
wrong,
if
you
don't
agree,
then
then
you
have
computed
something
else,
so
reproducibility
checks.
If
people
really
know
for
sure
what
has
been
computed,
and
that
gives
the
ability
to
verify
the
computation
there's
no
point
in
verifying
something.
B
I
showed
the
slide
of
a
paper
about
version
numbers
and
how
they
cause
trouble
now,
obviously,
scientific
methods
should
not
be
sensitive
to
such
details.
Unfortunately
they
are
so
what
you
need
to
do
is
to
do
tests
you,
you
do
robustness
tests
and
you,
you
vary
version
numbers.
You
check
different
versions,
different
combinations
of
versions
and
this
check,
if
you
still
get
similar
results-
and
that
is
actually
is
not
reproducibility,
this
is
replicability.
This
is
checking
for
the
scientific
appropriateness
of
the
procedure.
B
In
particular,
it's
robustness
so
same
software
really
means
same
for
software
stack
and
I.
Think
in
this
audience
people
understand
this
probably
better
than
my
usual
audience,
which
is
my
scientist,
colleagues,
who
don't
quite
see
the
points,
and
so
I'll
go
over
this
a
bit
more
quickly
here,
but
just
to
destroy
my
point.
What
is
a
computation?
Was
the
question
asked
and
answered
by
Alan
Turing
in
his
famous
paper
in
1937?
B
So
a
computation
basically
means
you
have
a
you:
have
the
computer
memory
with
lots
of
zeros
and
ones,
and
you
push
a
button.
The
computer
does
something
and
it
improves
a
new
state
of
output,
state
of
computer
memory,
which
is
another
series
of
zeros
and
ones.
So
that's
a
computation
from
a
computer
point
of
view.
There's
nothing
else
to
it.
It's
the
turing
machine.
B
Now
the
human
point
of
view
interprets
these
bits
in
a
certain
way,
and
this
is
where
the
the
trouble
comes
in.
So
we
typically
distinguish
between
data
programs
and
the
computational
environments
and,
for
example,
if
you
compile
a
piece
of
software
and
the
the
data
is
the
source
code,
the
program
is
the
compiler
and
the
environment.
Environment
is
the
operating
system
and
whatever
it
takes
to
get
the
compiler
running
and
the
output
is
the
compile
binary,
you're,
probably
all
familiar
with
that
I'm.
B
This
is
what
I,
what
my
center
of
Interest
the
programs
are
used
by,
that
is
they've,
typically
been
written
by
maybe
myself
or
by
my
colleagues,
people
I
know
more
or
less
people,
I
could
contact
and
so
on.
So
this
is
kind
of
in
my
sphere
of
interest,
but
not
necessarily
my
focus
of
Interest
and
the
environment
is
well
yeah,
I,
don't
care
about
it!
It's
just
stuff,
I
mean
how
it
works,
and
it's
just
it's
always
been
there.
It
will
always
be
there
and
it's
going
to
be
fine.
B
Now
we
can
imagine
where
the
travel
comes
from
right,
yeah,
you're,
right,
it's
from
the
environment,
because,
unfortunately,
if
you
look
at
the
semantics
of
computation,
the
stack
is
the
importance
in
the
stack
is
quite
the
opposite,
so
data
is
just
zeros
and
ones
for
the
computer,
there's
no
there's
no
meaning
attached
to
it.
It
is
the
program
that
reads
the
data
that
actually
decides
how
to
interpret
the
bits.
So
whether
a
sequence
of
bits
is
a
character
string
or
a
floating
Point.
B
B
So
the
environment
is
actually
what
what
what
has
is
the
last
word
on
on
what
everything
means
and
and
that's
what
scientists
consider
irrelevant
or
uninteresting
and
like
to
forget
about
there's
one
more
complication
on
this,
and
that
is
that
computation
is
as
if
dynamical
systems,
point
of
view.
It
is
chaotic,
so
here
on
the
left,
I've
changed
a
single
bit,
so
the
there's
a
red
zero
somewhere
there,
but
this
can
change
almost
everything
in
the
output.
B
So
there's
no
no
limit
to
that.
You
can.
You
cannot
know
if
you
change
a
single
bit
that
can
have
no
change
in
the
output
that
can
have
a
small
change
in
the
output
and
can
have
a
huge
change
in
the
output.
There's
no
way
to
predict
this
in
general,
there's
no
limit
on
that.
That's
what
it
is
called
chaotic.
B
B
This
is
universally
accepted.
Today
it
is
precisely
defined.
It
is
deterministic
operations.
You
can
redo
them
any
number
of
times.
You
will
get
the
same
result,
so
there
should
not
be
any
reproducibility
issue
in
reality.
There's
plenty
of
reproducibility
issues,
and
why
does
it
happen?
It
happens
because
you
can't
write
software
in
terms
of
the
IEEE
operations.
They
Define
a
set
of
operations,
like
addition
and
so
on,
with
grounding
modes,
and
so
no
programming
language
gives
you
access
to
that.
B
So
it's
actually
a
compiler
writer
who
decides
how
a
formula
in
C
or
Fortran
or
whatever
is
translated
into
IEEE
754
operations
and
actually
means
that
if
you
take
the
reproducibility
and
point
of
view
of
one
of
needing
to
run
the
same
software
and
the
compiler
is
an
integral
part
of
your
program,
you
can't
Factor
it
out.
You
can't
say
here:
I
have
a
fortune
program.
B
I,
don't
care
which
compiler
I
use
you
can't
you
have
to
care,
because
the
compiler
makes
a
decision
for
you,
so
the
environment
matters
basically,
and
if
you
look
at
all
the
details
and
what
I've
shown
the
pictures
I've
shown
before
you
actually
come
to
the
conclusion
that
reproducible
builds,
which
are
important.
The
important
topic
in
in
the
next
Universe
actually
the
same
problem
as
reproducible
science.
B
If
you
look
at
from
a
distance,
it's
fundamentally
the
same
problem,
it's
capturing
the
complete
soft
Quest
Tech
of
a
computation
in
order
to
be
able
to
redo
it
and
get
the
same
output
from
the
same
inputs
and
and
nicks
and
Geeks
have
solved
it
in
principle.
There's
no
question
about
it.
You
know
they've
also
solved
it
in
practice
for
software
builds,
and
the
open
issue
from
my
point
of
view,
is
that
they
did
not
yet
solve
it
in
practice
for
scientific
computing.
B
B
One
issue
is
that,
let's
put
it
this
way,
why
can't
you
just
do
a
data
analysis
as
a
package
there's
in
principle
there's
nothing
opposing
a
package
producing
a
PDF
file,
which
is
a
scientific
paper
with
figures
in
there
that
were
computed
from
the
data
and
so
yeah,
there's
nothing
in
in
Nix
or
Geeks.
That
says
that
a
package
has
to
produce
an
executable
binary.
It's
it's
just
a
file
that's
produced
by
by
computational,
so
you
could
actually
do
a
simulation
or
a
data
analysis
in
the
form
of
a
package.
B
One
problem
is
that
this
would
mean
everything
has
to
be
stored
in
in
the
next
store,
and
that
is
designed
to
be
a
cache
for
program
binaries
and
actually
certainly
not
good.
For
let's
say
big
data.
You
don't
want
10
gigabyte
files
in
your
store
for
various
reasons,
there's
also
no
support
for
incrementally
updated
data
sets.
That's
a
very
frequent
situation,
scientific
computation
that
you
have
several
steps:
different,
different
scripts,
different
programs,
reading
a
data
file
structured
composite
data
file,
something
like
hdf
says:
five.
B
If
you
know
about
this,
and
each
of
them
picks
data
out
of
it
and
puts
new
data
into
that,
so
the
file
changes
over
time,
but
in
in
a
controlled
incremental
way
and
finally,
there's
no
support
for
sharing
and
Publishing
data.
You
you
have
to
you
can
share
the
the
recipe
for
redoing
the
work.
But
if
you
do
a
three-week
simulation,
you
want
to
share
the
result
and
and
not
just
the
recipe,
their
ways.
This
could
be
handled
and
I
said.
Maybe
some
of
you
will
do
something
about
this?
B
B
You
store
a
hash
of
the
file
and
in
a
locator
and
that's
what
you
put
in
your
repository
and
and
then
you,
you
have
a
procedure
outside
that
that
get
lets.
You
look
up
and
download
the
data
from
the
hash.
A
related
idea
would
be
to
to
use
ipfs,
meaning
content.
That's
addressable
storage
system
as
a
backend
and
Only
Store,
the
the
content
identifiers
in
maybe
in
the
next
store
or
whatever
else
it
will
be.
B
B
Another
issue
that
is
what
I
call
not
near
the
situated
software.
It's
a
term
that
has
been
indented
I,
don't
know
about
a
while
ago,
roughly
situated
software
is
software
written
for
a
specific
situation
or
use
case,
as
opposed
to
generic
software,
which
has
a
wide
range
of
use,
use
cases
and
can
be
used
by
many
people
in
their
various
situations,
and
so
so
situated
software
has
a
very
limited,
reuse
and,
and
a
big
part
of
scientific
software
is
in
this
category.
B
This
software
that
that
produces
the
figure
of
my
latest
paper
is
absolutely
not
adaptable
to
anything
else.
It's
a
huge
script
that
basically
says
load
this
data
data
set,
do
the
following
transformation
and
produce
a
figure
while
one
curve
is
labeled
in
blue
and
the
other
thread
is
written
in
green
there's.
This
is
really
not
you,
don't
want
such
a
program
in
the
next
package
channel.
There's
no
point
to
that.
B
And
finally,
there
is
a
lack
of
tools
currently
for
a
reproducible
research
workflow,
which
takes
into
account
the
different
phases
of
a
research
project
which
typically
starts
from
interactive
exploration.
Let's
say
you
run
a
small
simulation,
look
at
the
results,
run
the
bigger
one
or
take
a
small
data,
set,
develop
a
procedure
for
analyzing
it
and
then
scale
up
the
data
set
at
some
point
to
switch
from
a
notebook
like
jupyter
notebook
to
a
workflow
as
in
make
or
snake
make,
and
then
freezing
this
into
packages
for
distribution
for
building
on
it.
B
So
there
are
there
are.
There
are
many
steps
in
this
process
that
you
want
ideally
to
be
handled
by
by
tool
support
in
a
way
that
makes
every
the
system
at
every
point,
reproducible
and
at
the
time
there
is
limited
tool,
support
and
all
the
tools
we
have
do
not
ensure
a
disability
and
to
not
interact
with
reproducibility
friendly
tools
like
Nyx.
B
So
that's
my
conclusion.
So
we
have
with
Nicks
and
Geeks.
We
have
excellent
infrastructure
for
scientific
Computing.
In
theory,
it's
actually
already
useful
and
used
by
a
small
minority
of
scientists.
Most
scientists
do
not
appreciate
them
yet
enough
to
adopt
them.
One
reason
is
that
the
tools
are
a
bit
too
complicated.
B
The
learning
curve
is
a
bit
steep,
but
also
they
don't
really
see
the
advantage,
because
the
the
last
mile
is
missing
and
they
basically
see
what's
missing
before
to
see
that
how
much
of
the
foundation
is
already
there.
So
there's
a
bit
of
Education
to
be
done,
but
it's
it's
also
important
to
to
cover
the
scenarios
which
are
not
yet
covered
and
and
actually
dominate
the
world
in
this
way
and
I
hope
I
hope
you
will
all
contribute
in
this
exercise.
B
A
B
A
I
will
also
pull
in
a
colleague
of
mine,
Julianne
that
came
out
weird
I'm,
sorry,
so
hello,
Julia
and
yeah.
So
there
is
already
one
question
coming
in
over.
Oh
wait,
there's
multiple
questions,
but
there's
one
on
matrix
by
a
kanji
and
they
are
asking
what
is
the
percentage
of
Publications
that,
in
your
definition,
are
reproducible
at
the
moment.
B
That's
it
depends
a
lot.
It
varies
a
lot
from
from
one
discipline
to
the
other,
and
it
obviously
depends
on
on
time
so
15
year
old
Publications
are
almost
zero
reproducibility
in
my
domain,
which
is
computational
biophysics
protein
simulations
I
would
say
it's
0.5
percent
at
best,
which
is
reproducible
today,
and
the
reason
is
that
this
is
a
field
dominated
by
long
simulations
running
on
supercomputers
and
there's
absolutely
no
tool
support
for
for
doing
parallel,
computation
on
on
weird
Platforms
in
a
revolution
way.
B
Now
we
check
reproducibility
by
by
reviewing
so
the
reviewers
are
supposed
to
check.
Yes,
I
can
reproduce
these
results,
and
we've
already
figured
out
that
two
years
later,
that
may
no
longer
be
true,
so
yeah.
This
would
be
a
nice
opportunity
and
we
have
discussed
this
for
introducing
tools
like
like
nics
or
Geeks
and
or
or
Docker
containers
or
whatever
else,
and
for
now
our
stance
is
that
we
we
in
kind
of
encourage
people
to
use
such
tools
if
they
want
to.
B
We
don't
want
to
make
it
a
requirement,
because
for
many
people,
the
the
technical
over
it
is
still
too
much.
It's
still
not
user-friendly
enough,
and
if
we
make
this
a
requirement,
we'll
simply
get
fewer
submissions.
So
that's
not
quite
the
point,
but
we
do
want
to
move
towards
such
a
way
to
adjust
the
situation
and,
as
I
said,
we
really
need
better
tool
support
to
get
to
the
point
where,
where
the
average
scientist
would
be
willing
to
do
the
the
necessary
steps
to
make
it
happen,.
A
C
Yeah
sure
I
thought
on
that
topic.
You
know
NYX
has
kind
of
amended
to
help
who
suffer
tooling
and
doing
reproducibility
on
the
software
side.
What
what
can
we
do
to
kind
of
get
these
tools
inside
his
hands
in
a
way?
That's
more
usable,
more
user
friendly
like
what
are
the
top
things
you
would
tackle.
B
This
is
some
python
package
which
uses
some
exotic
build
system
or
it's
on
its
own
sub
package
manager,
all
these
kind
of
things
or
it
has
25
dependencies
which
are
Hardware
package
and
or
it
does
it,
doesn't
build
even
this
python
packages
nowadays,
which
you
can't
even
build
from
Source
easily.
So,
basically,
you
have
python
packages
that
send
use
conda.
Otherwise
we
can't
help
you
so
I.
Don't
really
know
how
to
tackle
this
best.
I
think
the
the
the
the
I
think,
the
main.
B
Maybe
the
best
Vector
of
attack
would
be
to
convince
the
developers
of
such
packages
to
do
the
work
and
to
make
things
more
friendly
for
packaging
with
Nyx.
Ideally
do
it
themselves,
but
at
the
very
least
not
make
it
more
difficult.
So
I
think
that
it's
probably
easier
to
communicate
between
the
Nix
community
and
and
python
developer
community
that
should
be
easier
to
communicate
and
with
individual
scientists.
B
A
A
B
Yes,
definitely
because
the
nice
thing
about
both
of
these
tools,
they
they
both
allow,
let's
say
variations
on
on
environments
in
very,
very
different
ways.
So
that's
really
one
where
the
tool
with
the
two
will
differ
quite
radically
in
their
approach
to
it,
but
both
allowed
to
do
it,
and
this
is
important.
Just
the
very
possibility
of
of
having
two
versions
of
the
same
package
on
the
same
computer
at
the
same
time
is
so
important
for
this.
B
So
but
again,
there's
there's
no
good
tool
support
as
far
as
to
know
for
actually
doing
experiments.
To
give
you
an
idea
would
be
nice.
There's
a
I.
Don't
have
a
point
right
now,
but
there's
a
there's.
A
software
package
called
version
climber.
You
can
look
it
up
by
Google
on
another
name
which
is
specifically
for
for
python.
B
Environments
is
for
visual
pipe
environments
and
it
basically
takes
a
list
of
version
numbers
for
all
the
dependencies
and
it
it
Loops
over
them,
and
it
starts
all
the
environments
and
runs
the
program
and
sees,
if
runs
a
script
and
sees
that
the
output
is
the
same.
This
this
at
the
level
of
Nixa
Geeks.
Would
be
a
lot
more
useful
because
quite
often
you
have
dependencies
which
are
not
in
Python
at
some
point
and
then
yeah.
B
That's
the
old
problem
with
python
package
managers
as
soon
as
you
have
the
C
dependency
they
break
down
and-
and
that's
all
it's
always
the
case
so
and
they're
not
really
sufficient
for
real
life
exploration,
but
I
think
there's
a
that's
a
space
to
explore
there
in
for
automatically
testing
robustness.
Much
like
continuous
integration.
You
would
do
a
scanning
for
version
numbers.
A
B
No,
no
I,
don't
think
so.
It's
it's
really.
It's
the
starting
point
of
of
every
new
scientific
study
is
situated
software.
It's
what
comes
with
a
paper.
It's
only
then
when
a
method
is
published
in
in
a
specialized
script
becomes
in
more
widely
interesting
taken
up
by
other
people.
It's
only
then
that
people
get
together
and
and
say:
okay,
let's,
let's
Factor
the
common
method
out
into
a
reusable
package.
So
this
will
never
go
away.
We'll
have
we'll
always
have
at
the
leaf
of
the
tree
of
of
software
dependencies.
B
We
always
have
situated
software.
I
think
this
is
not
the
default.
Is
it's
it's
the
way
things
work?
It's
it's
it's!
How
new
new
methods
come
into
being
so
I
I,
don't
think
the
I've
heard
people
say
we
should
get
away
with
it
in
every
every
small
script
should
be
reproducible
and
the
old
software
authors
should
ensure
maintenance
of
their
of
their
scripts
for
their
papers.
I
think
this
is
completely
unrealistic
and
and
I
don't
think
we
should
aim
for
this
now.
A
That
makes
sense
modules
asked
another
question:
there's
probably
also
an
educational
effort
necessary
to
get
someone
up
to
speed
with
tools
like
NYX.
Do
you
have
any
idea
how
scientists
and
Nexus
can
connect
to
help
each
other
out?
Are
there
interesting
conferences
or
online
spaces
to
join
as
well.
B
So
they
are
aware
they're
looking
for
a
solution
to
a
problem,
because
the
majority
of
scientists
at
this
time
is
basically
waiting
for
the
rest
of
the
world
to
provide
them
with
a
solution
in
the
meantime,
go
on
doing
business
as
usual.
So
most
people-
don't
don't
consider
themselves
concerned
right
now,
but
there's
a
small
set
of
of
people
who
are
interested
in
this
and
I
think
that
they
are
the
the
the
key
audience
for
for
for
any
educational
measures.
B
Unfortunately,
there's
much
in
terms
of
of
conference
it's
these
are
small
meetings.
I
know,
for
example,
there's
a
field
that
is
very
very
well
interested
in
this
is,
is
neuroscience
I
know
that
they
have
at
one
of
their
big
conferences.
They
have
a
a
special
track
on
representability
each
year,
so
that
would
be
a
a
good
point
to
attack.
For
example,
you
can
look
it
up
of
the
resize
Journal
there.
They
we
do
a
special
number
every
year
for
from
this
reproducibility
trick.
So
you
see
who's
running
this
who's.
B
B
So
open
science
is,
it
goes
Way
Beyond
computation,
of
course,
is
is
a
ongoing
movement
that
says
that
science
is
there
for
public
benefit
and
should
not
be
hidden
behind
paywalls
and
and
and
strange
conventions
and
Conference,
and
so
on
so
whatever's
about
especially
public
science.
Producers
should
be
accessible
to
the
public
and
everybody
should
be
able
to
to
access
the
material
which
was
just
articles
but
also
code
and
data,
and
everybody
should
be
able
to
look
into
this
head
on
this
and
so
on
and
so
on.
B
So
it's
kind
of
the
generalization
of
the
open
source
and
free
software
idea
to
science,
with
some
adaptations
and
science
funders
have
jumped
on
this
and
they've
said
yeah.
Actually
it's
true
man.
We
are
paying
money,
and-
and
we
want
this
to
be,
would
it
be
effective
to
deposit
here
in
France,
for
example,
open
science
is
now
a
legal
obligation
since
December
3
2021
there
was
a
law
passed
that
says
all
French
public
science
must
be
open.
B
Some
I've
seen
it
in
my
mailbox
the
number
of
requests
for
seminars
and
reproducible
research,
and
it
says,
multiplied
by
effect
of
tensions,
so
yeah
politics
sets
The
Wider
goal
and
now
that
you,
your
job
or
you're,
together
with
people
like
me,
who
are
kind
of
in
the
middle
way,
is
to
to
make
the
link
and
say
yeah.
Nyx
is
good
for
open
science,
but
I
mean
you
can't
expect
politicians
to
make
that
link.
C
B
B
B
B
At
some
point,
it
started
out
being
extremely
stable,
very,
very
concerned
about
backwards
compatibility
and
at
some
point
it
exploded
to
the
point
that
at
some
point
almost
everybody
was
new
coming
in
new
and
everybody
else
in
idea
to
do
things
better
and
said
backwards,
compatibility
who
cares?
We?
We
are
discovering
a
new
world
so
who
cares
about
all
these?
B
These
old
farts,
with
their
old
with
that
10
year
old
code,
I,
mean
let's
do
something
new
and
that
started
a
complete
change
of
attitude
with
where,
where
you
suggesting
projects
which
are
maintained
at
every
given
time
by
five
PhD
students,
but
these
first
PhD
students
change
very
frequently
and
all
the
new
ones
coming
in,
say
hey.
We
have
a
good
idea
to
make
things
better.
B
Oh
yeah,
it
will
break
something,
doesn't
matter,
it's
not
our
problem,
and
this
has
been
gotten
a
lot
of
worse
when
python
got
the
mainstream
language
for
machine
learning,
because
in
machine
learning,
I
mean
a
method:
that's
half
a
year
old
is
oscillator.
Who
cares
for
a
script
at
CEO?
Nobody
wants
to
run
it
again.
B
In
my
field,
I
mean
I,
regularly
run
scripts
that
are
seven
years
old,
so
10
years
old,
because
I'm
in
research
projects
which
are
between
me
as
a
theoretician
and
experimentalists
who
do
experiments
I
mean
that
takes
a
lot
of
time.
So
a
10-year
research
project
is
normal,
so
I
have
a
very
different
time
frame
of
work
and
that
most
people
are
somewhere
in
between
these
two
extremes.
So
yeah,
maybe
so,
there's
two
issues.
B
One
is
that
python
packages
break
compatibility
too
often
I
think
that's
hard
to
change,
because
it's
a
social
issue,
but
that's
another
problem
is
the
python
package.
Managers
are
just
awful
building.
Python
packages
has
become
so
difficult
that
hardly
anyone
does
it
anymore.
Everybody
depends
on
some
pre-built
binaries,
which
are
very
fragile
and
I.
Think
they're.
The
python
Community
is
aware
of
that,
except
that,
if
you
tell
them,
we
have
a
solution,
we
have.
We
have
a
new
package
manager
for
you
that
they'll
just
say:
no,
no,
not
not
another
package
manager.
B
We
have
enough
of
those.
So
that
might
be
a
constructive
approach
and
then,
of
course
you
can
also
say,
let's
jump
ship
and
do
something
else
in
Python,
but
yeah
I
think
this
will
not
solve
the
problem
for
any
Community.
This
will
maybe
maybe
be
a
good
idea
long
term,
but
then
what
what
are
you
going
to
replace
it
with
I
mean
there's
no,
no
decent
candidate
at
the
view
for
now
so
not
easy.
A
Yes,
thank
you
we'll
go
over
to
matthias's
last
question
over
on
Matrix,
unless
he
types
more,
of
course,
speculating.
Should
a
scientific
study
of
the
Future
run
on
a
distribution
of
sorry
should
a
scientific
study
of
the
Future
run
on
a
distribution
of
environments
which
different
versions
producing
an
outcome
distribution
rather
than
bit.
Reproducibility
did
I
read
that
correctly.
As
my
next
question.
B
Yeah
that
that's
again
a
good
point,
I
mean
I
I.
Think
that's,
there's
two
things
first
thing
is
I.
Think
bit
forbid.
Reproducibility
is
is
a
worthy
goal
in
itself,
as
a
first
step,
the
main
reason
being
it
can
be
fully
automated
a
NYX.
Does
it
and
Geeks?
Does
it
you?
You
can
run
things
again
and
again
and
again
check
if
they're
a
bit
probably
the
same,
and
that's
a
useful
thing
to
have
it's
the
ground
layer,
I,
would
say
and
I
already
having
a
hard
time
convincing.
B
My
scientist,
colleagues,
that
this
is
a
useful
thing
to
have,
because
what
they
say
is
what
we
really
care
about
is
the
next
level
and
that's
what
yes
refers
to
to
have
things
which
we
can
check
to
be
robust.
Until
until
all
these
technical
details
we
don't
care
about
so,
yes,
I,
think
doing
such
robustness.
B
Checks
is
useful,
but
it's
it's
not
automatable
I
mean
what
He
suggests
is
and
what
I
also
mentioned
recently
is
you
can
have
tools
that
do
the
work
for
you
that
run
it
for
on
all
the
combinations
of
version
numbers
for
various
dependencies,
but
there's
no
automatic
tool
that
can
decide
for
you
if
the
result
is
close
enough,
because
what
close
enough
means
is
a
scientific
judgment,
and
you
can't
say
it's
a
floating
Point
agreement
to
three
decimals.
It
may
be
good
enough
for
some
applications
for
others.
B
So
in
the
end,
it's
always
it
takes
an
expert
to
judge
if
it's
good
enough
and
I
don't
think
there
are
many
experts
willing
to
look
at
5,
000
numbers
and
judge
if
they
are
close
enough
to
each
other,
so
I
think
robustness.
Checks
at
this
time
are
useful,
as
as
part
of
software
development
as
part
of
regression
tests.
I
would
really
see
it
more
like
a
regression
test.
B
So
it
is
the
developers
of
some
package
that
would
do
a
regression
test
like
this
run
it
on
their
software
regularly
and
then,
if
something
comes
out
differently,
it's
one
of
their
experts.
That
will
check
if,
if
their
regression
test
is
still
satisfied,
yes
or
no,
but
as
a
tool
for
the
general
public
I
think
this
will
not
find
many
friends
it's
just
it's
too
heavy.
A
B
Well,
I
think
the
last
part.
The
answer
is
clearly
no
right,
I
mean
if
you're
happy
with
failure,
then
no
no
layoffs
when
you
need
an
error
correction
layer,
but
you
could
Implement
that
in
software
presumably,
but
that's
not
what
Nix
does.
The
other
question
is
no
I
think
that
there
are
studies
about
this.
This
becomes
an
issue
if
you
go
to
to
to
large
scale
Computing.
It
is
simulations
like,
like
climate
simulations
that
run
for
a
couple
of
weeks
on
on
a
thousand
processes.
B
Then
you
get
to
the
point
that
error
rates
from
my
Cosmic
race
is
the
most
sexy
sounding,
but
it's
actually
a
hardware.
Hardware
fatigue
is
actually
more
important
in
practice,
but
such
errors
are
are
very
rare
and
become
relevant
only
for
for
very
large
computers
and
very
long-running
computations
and
people
in
the
Computing
centers
that
do
care
about
them.
They
do
monitor
them.
B
They
are
handled
using
redundancy,
so
they
run
the
same
computation
on
several
nodes
and
compare
and
check
for
them.
So
there
are
error
Corrections
in
place
in
these
centers
and
that's
a
different
issue.
I
think
this
is
not
something
that
Nix
could
or
should
address,
I
think
it's.
So
it's
a
specialist
problem
handled
by
Specialists
and
they
know
how
to
handle
it.
A
B
A
B
There
aren't
really
any
conferences
on
reproducibility.
These
are
small
workshops,
but
those
people
who
don't
need
you
need
to
convince
in
a
way
so
yeah.
What
would
be
a
good
good
place?
I
said
it's:
it's
increasingly
General
Science
conferences
on
specific
domains
that
that
have
an
afternoon
workshops
tutorials.
This
is
deep
space.
How
would
you
find
them?
That's
a
good
point.
This
is
yeah.
B
It's
no
easy
answer
that
I've
actually
participated
in
a
couple
of
them
on
the
I've
done
trainings
myself
more
using
Geeks
than
leaks,
but
this
is
all
a
small
scale.
People
I
that
invited
me
because,
because
we
had
been
in
contact
before
so
this
is
not
very
organized.
It's
small
scale,
Networks,
no
I,
don't
have
any
easy
answer
for
that.
So
sorry,.
B
B
Okay,
yeah
yeah
I
mean
yeah
yeah
people
say
interested
people
like
me,
and
my
colleagues
yeah
that
would
be
an
option.
I
think
the
the
general
public
scientists
would
not
be.
It
would
say
what
what
should
I
care
about
it,
but
yeah
for
me
personally,
I
put
certainly
something
of
Interest,
yeah
and
and
I
could
list
a
couple
of
other
people
to
invite.
But
that's
a
small,
a
handful
of
Specialists,
basically
yeah.
A
B
Well
for
Advocate:
this
is,
if
the
journals
have
these
kind
of
badges,
then
they
are
convinced
or
that
the
job
is
done.
You
can
still
convince
them
that
you
have
got
some
technical
solution
for
them,
but
they
already
that's
good
sem
made
quite
a
bit
of
Noise
with
that,
because
it's
it's
a
huge
institution
and-
and
that
was
a
big
step
forward
for
for
them
to
to
introduce
these
badges.
B
There
are
a
few
other
journals
in
various
domains
that
do
that.
That
still
I
think
it's
a
minority
almost
everywhere,
almost
all
scientific
Fields,
but
it's
it's
increasing.
There's
also
there's
an
interesting
variant
idea.
That's
there
is
an
institution
in
France
which
is
actually
a
officially
a
support
lab
for
research,
and
it's
financed
and
run
by
the
French
research
organization,
whose
goal
is
to
verify
validate
certify
the
reproducibility
of
submitted
computations.
B
B
That's
a
I
mean
I
have
a
list
of
handful
of
journals
that
that
do
have
reproducibility
requirements
or
suggestions.
They
are
always
a
minority,
but
the
number
is
increasing,
so
the
tenancy
is
right
and
yes
contacting
them.
I
don't
think
contacting
them
is
directly
with
the
technical
position
is
a
good
idea,
because
the
these
are
the
Zone.
Editors
are
basically
convinced
that
practices
must
improve,
but
they
usually
don't
understand
the
technicalities.
So
if
you
send
them
and
say,
hey,
hey
next
is
great,
they
say
yeah.
Maybe
Nix
is
great.
B
C
A
They
ask
another
question
over
and
on
cast
so
it's
off
Oregon
regarding
support
for
research
workflow.
There
are
parallels
between
development
and
deployment
of
software
different
needs
for
deployment
of
software
than
for
distribution
and
deployment.
We
have
Dev
shells
an
example
next
developed
for
this.
How
close
does
this
get
to
the
needs
for
research?
How
is
research
flow
different
than
software
development.
B
I
think
the
difference
is
that
the
focus
on
software
development
is
the
software
awareness
in
research.
It
is
on
on
the
computed
results
and
as
I
try
to
explain
in
my
my
lecture
there's
some
there's
a
limited
similarity
in
that
both
are
the
result
of
the
computation
which
you
want
to
make
reproducible,
but
the
nature
of
the
resulting
artifact
is
very
different,
so
there's
no
equal
deployments,
you
can
deploy
software,
you
don't
deploy
scientific
results,
they
are.
B
They
are
meant
for
human
conception,
they're
packaged
in
in
things
like
PDF
files
and
movies
and
images,
and
this
kind
of
stuff,
maybe
websites.
It's
so
that
there
is
not
really
deployment
equivalent.
The
closest
equivalent
to
deployment
would
be
publication
that
is
putting
it
on
the
public
facing
web
server.
That's
probably
the
closest
we
can
get
to
deployment.
That's
why
I
said
you
don't
want
these
things
in
the
next
store.
That's
not
useful!
You
want
them
on
the
web
server.
So
the
final
part
is
a
bit
different.
A
B
Yeah
I
think
the
in
the
history
of
Geeks
is
probably
closer
to
science.
I
mean
the
the
the
the
the
the
main
founder
of
the
project
works
for
a
French
research
lab
on
the
the
second
initial
maintainer
is
a
bioinformation
and
German
research
lab,
so
Geeks
was
always
basically
driven
by
scientific
Computing.
There's
the
Geeks
HPC
project,
which
I
don't
know
who
finds
it,
but
it's
specifically
for
reproducible,
parallel
Computing,
which
is
very
important
for
scientific
Computing.
B
So
I
think
that
yeah
I
think
Geeks
has
a
stronger
foot
in
the
scientific
computation
business
simply
by
its
Community,
which
has
which
is
that's
simply
more
scientists
than
there
and
as
a
consequence,
there
are
more
scientific
software
packages
present
in
the
collection,
I
guess,
but
then
again
the
answers
probably
depend
on
on
the
field
you're
in
so
what
it's
I
think
it's
a
historical
as
to
the
accident
at
some
point.
I,
don't
think
there
is
a
fundamental
technical
difference
that
makes
Geeks
more
appropriate
for
scientific
computation.
A
B
No,
of
course
not
I
mean
the
reproducible
conference
is
one
where
every
presentation
could
be
redone,
silently
and
by
everybody
at
home
later
on,
but
the
next
year
they
wouldn't
talk
about
the
same
thing.
Of
course
they
cost
them
two
different
presentations,
but
they
would
all
be
archived
in
a
way
that,
at
any
point
you
could
say
well.
Five
years
ago,
I
saw
this
interesting
conference.
B
A
A
B
B
A
Thanks
to
everybody
who
made
this
summer
for
next
public
collection
series
2022
possible
the
annelnut
foundation,
European,
commission,
the
Nexus
foundation
and
tweak
are
the
special
thanks
thanks
to
speaker
of
the
actual
Conrad
and
as
they
have
announced,
nexcon
2022
I
hope
to
see
you
there
as
well.
Don't
worry,
it
won't
be
the
same
as
the
last
next
con.
It's
not
reproducible
in
that
sense,
as
well
as
Conrad
just
said,
he
will
still
be
in
The
Matrix.
A
So
if
you
want
to
ask
some
more
questions
or
just
have
a
discussion,
not
necessarily
with
an
economy
but
just
a
community
in
a
general,
you
can
head
over
there
and
ask
it
as
well
I
hope
to
see
you
in
the
next
lecture,
which
will
be
about
NYX
and
ipfs
if
I'm
not
mistaken,
bye,
Name,
Escapes
Me,
oh
boy,
I
should
memorize
these
things,
but
yeah
I
will
see
you
in
the
next
lecture
next
week
same
time
same
places
and
I
do
hope.
Doncaster
instance
survived
today,
so
yeah.
Thank
you.