►
From YouTube: A data-driven look at practices behind exemplar open source projects - GitHub Universe 2019
Description
Presented by:
Stephen Magill, CEO at Muse Dev
Gene Kim, Author, Researcher & Founder at IT Revolution
In this session, you'll hear about the year-long collaboration between Gene Kim (researcher and co-author of “The Phoenix Project,” “DevOps Handbook,” “Accelerate”), Dr. Stephen Magill, (expert in software security and program analysis), and Sonatype (maintainers of the Maven Central Repository). They examined 10,000 open source Java components that were published to Maven Central and hosted on GitHub. They combined team and project performance metrics from GitHub, popularity data from Maven Central, and vulnerability and dependency data from Sonatype to examine what properties are shared by exemplary open source teams. They discuss these findings, including the differences we see between exemplary small teams and large teams, the fact that popularity does not predict security, and how remarkably difficult it is to keep dependencies patched while being “almost” up-to-date. They also highlight the organizational and technology practices they observe among exemplar open source teams, which release new versions 2x more frequently and remediate security vulnerabilities 3x more quickly, all while delivering a level of value that makes them standouts in terms of popularity and adoption.
About GitHub Universe:
GitHub Universe is a two-day conference dedicated to the creativity and curiosity of the largest software community in the world. Sessions cover topics from team culture to open source software across industries and technologies.
For more information on GitHub Universe, check the website:
https://githubuniverse.com
A
Afternoon,
thank
you
all
right,
so
my
name
is
Stephen
McGill
I've
been
doing
academic
research
and
software
analysis,
security
and
programming
languages
for
more
than
15
years.
First,
as
part
of
my
PhD
work
at
Carnegie
Mellon
and
then
at
other
universities
and
Industry
research
labs
over
the
last
few
years,
I've
been
getting
more
and
more
interested
in
the
practice
of
software,
so
how
open
source
development
practices,
how
those
work,
how
enterprises
approach
software
development
and
really
how
to
best
contribute
to
these
communities
by
improving
tools
and
practices?
Thank.
B
You,
dr.
McGill,
my
name
is
gene
Kim
I've
been
studying,
high-performing
technology
organizations
since
1999.
That
was
a
journey
that
started
back
when
I
was
the
technical,
founder
and
CTO,
but
a
company
called
tripwire
in
the
information
security
space,
one
of
the
most
fun
things
I've
gotta
had
the
opportunity
to
work
on
is
the
state
of
DevOps
report,
and
that
was
with
dr.
B
Nicole,
fors,
grin
and
just
humble,
which
resulted
in
the
accelerate
book
and
so
I
think
that
was
so
fun
because
a
largest
cross
population
study,
where
we
really
got
to
study,
30,000
respondents
and
understand
what
are
the
best
way
to
measure
technology
performance
and
what
are
the
specific
behaviors
that
lead
to
high
performance.
My
areas,
passion
these
days,
is
really
studying
how
large
complex
organizations
are
adopting
DevOps
principles
and
patterns,
and
so
I
run
the
DevOps
Enterprise
limit
and
I
have
a
book
coming
out
in
a
week
and
a
half
called
the
Unicorn
project.
B
So
the
may
be
just
a
set
the
stage
for
this
presentation.
We
all
know
that
open
source
software
is
everywhere.
An
at
Friedman
yesterday
said:
99%
of
new
software
projects
include
open
source.
That's
great,
and
so
the
question
then
becomes
you
know
how
do
the
teams
that
work
on
these
components
in
terms
of
their
update,
behavior
security
and
testing?
Suddenly
all
of
their
behaviors
and
practices
affect
us
right.
So
yesterday
that
Friedman
also
said
you
are
now
inviting
thousands
of
developers
into
your
code
and
when
Steve
and.
A
B
Right
so
that
that
will
now
impact,
you
know
the
quality
of
the
software
that
you
ship
so
will
they
help
or
hurt,
and
so
really
the
question
becomes
what
practices
correspond
to
good
security
outcomes
and
therefore
good
security
for
your
the
things
that
you're
working
on
so
just
in
terms
of
a
brief
background,
I
had
mentioned
the
state
of
DevOps
report.
This
is
what's
been
so
great
about.
B
We
found
that
all
those
metrics
go
up
and
down
together
right
and
those
predicted
not
just
this
thing,
that
we
call
IT
performance
but
also
organizational
performance
and
there's
all
these
behaviors,
that
you
know
make
those
metrics
go
up
and
down,
and
so
for
the
longest
time
I
wanted
to
understand
like
how
do
we
take
those
learnings
and
apply
them
to
you
know
how
software
projects
I've
worked
on
and
so
I'm
the
next
slide.
Oh
yes,
so
the
specific!
Oh!
Can
we
go
back?
One
slide,
please,
so
you
know
specific
questions
would
be
now.
B
What
are
the
structures
and
practices
that
you
in
the
source
code
repose
that
lead
to
exemplary
outcomes
that
might
map
to
the
metrics
that
we
studied
in
the
DevOps
state
of
DeVos
report,
and
will
we
find
those
same
behaviors
in
enterprise
projects
as
well?
So,
a
couple
of
years
ago,
the
folks
at
Senate
type
approached
me
and
I
came
up
with
this
amazing
idea.
They
invited
they
said
that
might
be
an
opportunity
to
actually
study
the
maven
central
eco
system.
B
For
those
of
you
who
don't
know
maven
is
to
Java
as
rubygems
is
to
Ruby
pipe
I
as
a
Python
and
NPM
is
to
JavaScript,
and
potentially
you
know
in
the
cards
was
the
ability
to
actually
look
at
the
over
decades
of
what
was
being
downloaded,
maven
central
and
specifically,
what
are
the
what
is
happening
inside
of
the
project,
including
updates,
and
so
this
was
just
such
a
delightful
thing
for
me
because
as
a
I
could
go
back
one
slide.
Oh.
B
A
I'm,
more
of
a
Haskell
programmer,
which
is
sort
of
down
on
the
far
right
of
this
slide,
but
you
know
Scala
as
a
typed.
Functional
language
in
the
Java
ecosystem
is
also
close
to
my
heart
and
so
like
getting
to
look
at
this.
This
broad
ecosystem
that
really
it's
not
even
about
a
single
language.
It's
about.
You
know
this
collection
of
languages
and
tools
and
frameworks
and,
in
particular,
a
ton
of
open
source
that
we
all
benefit
from
absolutely.
B
So
I,
you
know
the
words
I
would
use.
This
is
gratitude
and
enthusiasm
and
so
on.
The
next
slide
is
the
is:
what
really
came
out
of
it,
so
we
spent
over
a
year
working
on
the
state
of
the
software
supply
chain,
and
so
dr.
McGill
was
the
lead
investigator
and
we
were
supported
by
an
amazing
team
at
sona
type,
Bruce
Mayhew,
gauzy,
Mahmood,
Kevin
Witten
Dirk
weeks,
Matt,
Howard
and
I
forgot
to
have
mentioned
Brian
Fox,
with
the
Invo
of
The
Adventures
of
maven
that
the
longest-lived
repository
across
all
of
the
language
ecosystem.
A
A
So,
and
we
came
into
the
work
with
a
number
of
hypotheses
that
we
wanted
to
investigate
the
first
one
was
whether
projects
that
release
frequently
have
better
outcome,
so
really
the
open
source
version
of
this
effect
that
we
see
on
the
enterprise
side.
When
we
look
at
the
effect
that
DevOps
has
had
on
enterprise
outcomes
and
practices,
we
also
wanted
to
look
at
whether
projects
that
update
more
frequently
are
generally
more
secure
and
whether
projects
with
fewer
dependencies
stay
more
up-to-date
right.
A
So
there's
a
number
of
intuitive
correlations
that
we
expected
to
find
in
the
data,
and
so
we
wanted
to
walk
through
and
see
whether
we
actually
saw
a
statistical
support
for
these
hypotheses
and
then
finally,
hypothesis
number
four
which
we'll
get
to
at
the
end
is
the
effect
of
popularity.
Whether
popularity
is
a
good
proxy
for
some
of
these
quality
outcomes
that
we
all
care
about,
but
before
I
go
further
into
the
hypotheses
and
the
evidence
that
we
found
for
against
these
various
hypotheses.
A
I
want
to
talk
about
the
data
set
and
what
data
set
we
pulled
together.
What's
involved
and
the
sort
of
attributes
that
we
focused
on
so
first
of
all,
as
Jean
mentioned,
we
started
with
the
maven
central
data
maven
central
as
of
a
couple
of
days
ago,
I
updated.
These
statistics
on
the
slide
contains
300
and
over
two
three
hundred
ten
thousand
Java
opponents.
A
So
we
can
look
not
just
at
the
maven
central
data
but
at
the
github
manifestation
of
that
and
the
metadata
and
the
contributor
profiles
and
everything
that's
associated
with
those
open
source
projects
and
their
development
on
github
and
then,
of
course,
we've
heard
a
lot
about
security,
both
at
this
morning's
keynote
and
then
in
other
talks
throughout
the
conference.
And
that's
first,
that's
a
central
concern
here
as
well
right,
so
statistics
show
that
almost
9%
of
these
components
are
vulnerable,
and
so
you
know
what
what
is
it
about?
A
The
vulnerable
components,
the
non
vulnerable
governments,
which
components
are
best
at
responding
to
security
incidents
and
remediating
vulnerabilities,
and
so
we
wanted
to
look
at
that,
not
just
at
the
individual
project
level,
but
taking
into
account
the
dependency
structure
right.
So
here
is
the
dependency
graph
for
a
particular
job
of
component,
and
you
can
see
you
know
it's
quite
deep
right.
A
So
this
is
this
gets
back
to
the
to
the
comment
from
this
morning
that
you
know
you
have
to
consider
your
dependencies
and
their
dependencies
in
the
dependencies
of
the
dependencies,
and
it
goes
on
on
right.
So
what
happens
when
one
of
these
updates
right?
It
kicks
off
a
whole
chain
of
things
right,
the
the
the
projects
that
use
that
dependency
now
have
to
update
their
version
and
that
update
has
to
propagate,
and
so
we
looked
a
lot
of
that
propagation,
behavior
and
so
to
do
this.
A
We
started
as
I
said
from
the
maven
central
data,
but
then
we
we
did
a
number.
We
applied
a
number
of
filters
to
focus
down
on
a
set
of
data
where
we
really
had
all
the
attributes
and
and
associated
metrics
that
we
cared
about
so
that
we
could
really
do
an
in-depth
analysis,
and
so
the
first
thing
we
did
is
we
looked
at
just
recent
projects.
So
we
looked
over
the
last
five
years
because
development
practices
tools,
development,
culture,
changes
over
time
and
we
wanted
what
we
found
to
be
relevant
to
developers.
A
Today,
then
we
looked
at
whether
the
component
was
connected
into
this
open
source
supply-chain
right,
there's
a
lot
of
components
that
are
just
isolated
projects.
Maybe
it's
a
personal
project.
Maybe
it
never
took
off,
but
you
know
if
no
one
in
the
open-source
community
is
using
it
and
it's
not
using
any
open-source
libraries
itself,
then
it's
really
not
part
of
this
supply
chain.
So
we
filtered
out
based
on
that
we
filtered
on
our
ability
to
recognize
version.
So
maven
central,
the
maven
tool
has
a
particular
standard
for
versioning.
A
It's
the
it's.
The
includes
all
of
the
projects
that
are
widely
used
that
are
well
connected
in
this
open
source
dependency
graph,
all
right.
So
the
attributes
we
looked
at
are
popularity
and
we'll
talk
in
more
detail
about
each
of
these
in
the
next
couple
of
slides,
but
I'll
just
go
through
them.
Popularity
release
frequency,
then
a
number
of
metrics
that
that
we
get
by
connecting
to
that
github
data
that
I
talked
about
right.
A
So
some
of
these
projects
have
associated
github
repos,
and
so,
if
we
look
at
the
github
activity,
we
can
see
a
development
activity.
We
can
see
the
size
of
the
development
team,
whether
CI
is
being
used
by
pulling
the
repo
and
looking
for
CI
markers
and
then
there's
a
few
attributes
foundation
support
and
then
some
security
and
update
oriented
attributes
that
require
their
own
sort
of
slide
deck
to
go
into
and
we
will
go
into
them.
But
first
I
want
to
focus
on
just
how
we
collected
these
popularity
and
activity
metrics.
So
Jean,
perfect.
B
Yeah,
so
the
first
set
of
attributes,
as
opposed
to
the
outcomes,
are
I'll,
go
through
each
one.
So
the
first
one
is
like
popularity
right,
and
so
this
is
a
measure
of.
How
often
is
this
component
being
used?
How
often
is
being
downloaded
so
the
specific
measurements
are,
how
many
times
is
being
downloaded
from
maven
central,
how
Oh
number
of
stars
and
four
so
the
libraries
io
data
was
phenomenal,
so
we
actually
used
that
dataset,
which
actually
provides
a
number
of
Forks
and
stars
for
various
github
repos.
B
So
shout
out
to
the
great
people
behind
that
project,
and
then
we
actually
have
to
say
CI,
oh
and
then
that
we
were
able
to
get
access
from
the
Nexus
IQ
server.
So
these
are
the
specific
projects
inside
of
someone's
like
customers,
enterprise
projects,
so
we
can
actually
see
what
components
they
were
using
anonymized.
Of
course,
the
second
set
of
data
that
we
gathered
was
around
commit
activity,
and
so
here
we
used
the
phenomenal
tools
from
the
chaos
project
included,
percival,
and
so
one
of
the
things
that
we
gathered
was
the
commits
per
month.
B
So
for
every
we
use
percival
to
use,
go
to
github,
which
I'd,
which
actually
downloading
target
repo,
and
then
you
know
crack
that
open,
and
so
we
could
actually
see
how
many
commits
were
being
made
four
months
and
then
we
actually
measured
how
many
unique
developers
were
there.
So
this
is
kind
of
proxy,
for
you
know
how
many
active
developers
are
there
on
that
specific
project,
and
then
we
also
looked
for
okay
and
so
on.
The
way
we
did
is
we
use
percival.
A
That's
actually
the
last
you'll
hear
about
CI
in
this
talk,
because
we
didn't
find
a
connection
between
use
of
CI
and
some
of
the
other
outcomes
we
were
looking
for,
which
was
really
surprising
and
really
you
know,
I
think
we'll
get
to
this
I'll
come
later,
but
like
it's
not
just
about
the
tools,
you
use
it's
about
the
practices
and
the
culture
around
those
tools,
and
so
you
know
that
was
one
indication
of
that.
It.
B
A
Alright,
so
we
also
looked
at
a
number
of
other
project
level,
metrics,
so
the
first.
The
first
is
and
is
relevant
to
here,
especially
as
we
heard
in
the
keynote
this
morning
about
yesterday
about
support
for
open
source
projects
and
what
github
is
doing
there.
What
sort
of
support
does
the
project
have?
Is
it
foundation
supported?
Is
it
commercially
supported,
or
is
it
just
an
independent
project?
B
Is
okay,
so
hypothesis
number
one
was
that
projects
that
released
more
frequently
will
have
better
outcomes,
so
this
is,
of
course,
based
on
this
experience
coming
out
of
the
state
of
develop
art
where
we
found
that
deployment
frequency
went
hand-in-hand
with
a
component
lead
time
and
in
general
right,
the
more
you
release
the
better
all
the
other
outcome
measures
are.
So
this
is
a.
We
wanted
to
look
at
all
right.
Do
projects
that
output
more
releases
more
frequently,
you
know,
would
they
have
better
outcomes,
confirmed
yeah.
A
This
was
validated,
excuse
me,
yeah,
so,
on
the
whole
projects
that
release
more
frequently
are
more
popular.
We
took
popularity
as
the
main
proxy
for
a
positive,
open
source
outcome.
Right
people
contribute
to
open
source
because
they
want
their
code
to
have
impact.
They
want
to
provide
capabilities
to
the
rest
of
the
world,
and
so
popularity
is
really
a
great
measure
of
how
well
they're
accomplishing
that,
and
so
here
are
the
statistics
behind
this
right,
so
the
projects
that
released
most
frequently
so
the
top
20%
of
projects
our
five
times
more
popular
on
average.
A
B
That's
interesting
by
the
way,
just
to
back
up
an
analysis.
So
you
know
so
we
split
the
population
into
those
that
had
the
top
20%
of
release
frequency
right
and
then
compared
all
the
measures
that
Steven
talked
about
and
if
you
could
go
back,
one
slide
write
that
so
the
big
surprise
here
was
about
again
that
development
team
size
all
right.
The
question
is:
do
larger
teams
lead
to
more
frequent
leases
or
does
like
popularity
and
active
projects,
attract
more
developers
right
and,
of
course,
with
the
sort
of
analysis?
A
Yeah
so
then
I
want
to
say
more
about
these
last
two
metrics
that
I
mentioned
as
more
complicated
right
so
security
and
update,
speed,
and
so
here
I
want
to
walk
through
sort
of
a
progression
of
updates
in
a
typical
supply
open-source
supply
chain.
So
here
we
have
just
some
packages,
labeled
a
b
c
d
or
a
b
c,
and
you
can
view
time
as
marching
from
left
to
right
right,
and
so
we
have
several
versions
of
each
package
displayed
here.
Dependencies.
A
The
solid
lines
represent
dependencies,
so
here
package
c
is
dependent
on
a
and
B
and
we
can
see
several
releases
of
a
right.
It
starts
at
version,
2.2
moves
on
to
version
2.3
and
then
releases
version
2.4
that
new
version
of
dependency,
a
has
eventually
picked
up
when
C
releases
its
next
version.
So
at
this
point
right-
and
we
also
have
vulnerability
information
depicted
here
right.
A
So
these
these
red
points
are
points
at
which
we're
assuming
a
vulnerability
is
discovered
and
reported
in
this
case
to
component
B
and
so
there's
some
period
of
time
where
B
is
vulnerable
and
because
C
depends
on
B
there's
some
period
of
time
where
C
is
vulnerable
and
so
and
that
vulnerable
time
is
interesting.
But
what's
really
more
relevant
and
actionable.
Is
this
remediation
time
right?
A
How
long
does
it
take
from
when
B
patches
that
vulnerability
to
the
point
in
time
when
C
adopts
that
patched
version
of
B,
and
so
we
call
that
remediation
time
TTR
and
we
calculate
that
for
each
security
relevant
update?
We
also
for
each
update
whether
security,
relevant
or
not
capture
the
update
time.
So
how
long
again
does
it
take
see
to
adopt
this
updated
version
of
B,
whereas
B
is
the
only
vulnerable
component?
In
this
example,
there
are
other
components
that
update
right.
A
So
there's
an
update
time
for
a,
and
so
we
can
record
that
update
metric
for
every
update
to
every
dependency,
whether
security,
relevant
or
not.
Then
we
call
that
TTU
so
you'll
see
that
in
the
next
slides
as
well,
and
then
this
is
this
notion
of
stale
dependencies
that
I
mentioned
earlier.
We're
see
here
is
releasing
a
new
version,
there's
a
more
up-to-date
version
of
a
available
at
the
time
of
that
release.
A
But
C
doesn't
adopt
that
me,
so
that
really
gives
us
three
key
dependency
oriented
metrics
time
to
remediate
time
to
update
and
stale
dependencies,
and
we
get
these
on
a
per
update
basis,
though
we
can
also
roll
them
up
to
a
per
project
basis.
By
looking
at
the
median
of
those
collections
of
metrics
yeah.
B
So
this
is
so
cool,
because
now
we
have
the
time
to
mediate
security,
vulnerabilities
and
the
time
to
update
any
dependencies
across
the
entire
population
of
the
14,000
projects
across
all
of
their
versions.
So
the
graph
here
shows
on
the
x-axis
is
in
days
to
meet
remediate
from
vulnerabilities
and
the
Y
is
what
percent
of
the
population.
So
this
is
a
cumulative.
B
Right
so
like
at
the
very
right
you
know,
ideally,
a
hundred
percent
will
have
updated
or
radiative
vulnerability.
So
here's
kind
of
how
to
read
this
graph.
The
median
is
a
hundred
eighty
days.
The
mean
is
three
on
twenty-six
days.
So
just
maybe
take
a
pause
there.
It
takes
a
half
year
for
the
average
component
to
remediate
published
security
vulnerability
and
by
the
time
you
get
to
three
and
a
half
years,
95%
have
updated.
So
we
actually
cut
off
the
graph
because
I
just
did
you
know
it
because
on
forever
a
long.
B
A
log
normal
distribution,
so
that's
the
kind
of
sobering
finding
I
think
this
is
consistent
with
some
of
the
statistics
we've
seen
that
we
were
able
to
validate.
You
know
in
this
analysis,
so
how
one
other
observation
here
is
and
so
on
the
next
slide
right,
so
that
it's
actually
this
lower
left
quadrant
that
we
want
to
study.
These
are
the
potential
exemplars
who,
for
our
reason,
write
mediates
security
patches
quickly,
right,
so
that's
kind
of
the
population
we
want
to
study.
B
So
the
other
thing
to
note
is
that
the
time
to
mediate,
metrics
are
actually
pretty
sparse,
that
not
every
component
actually
has
a
CBE
published
against
it.
So
overlaid
onto
this
on,
the
green
is
the
time
to
update.
So
if
a
component
has
updated,
you
know
we
have.
You
know
they're
gonna
be
in
this
population
which
is
much
bigger
set
and
again,
you
know
just
to
read
this:
that
median
time
for
projects
to
update
is
a
hundred
thirty
days.
B
A
Yeah
and
interesting,
so
you
can
see
these
distributions
match
very
closely
at
the
at
the
population
level,
but
we
wondered
at
the
project
level.
Is
there
also
this
correlation
right,
yeah,
and
so
here
we
have
time
to
update
and
time
to
remediate
so
security
relevant
and
non-security
relevant
updates
graphed,
where
each
point
these
are
each
point,
corresponds
to
a
single
project.
So
where
does
it
live
in
this
space
of
time
to
update
for
this
time
to
remediate,
and
so
the
line
that's
depicted
there?
Those
are
really.
A
Those
are
projects
that
lie
on
that
line
are
projects
that
take
treat
security
relevant
and
on
security,
relevant
updates
equally
right.
Equally,
they
respond
in
the
same
way
to
both,
and
so
the
in
going
back
to
the
previous
slide
real
quick.
So
we
did
find
a
correlation
here,
so
the
correlation
coefficient
is
0.6.
A
It's
maybe
not
super
obvious
from
the
from
the
graph,
but
but
there
is
some
signal
there
in
terms
of
those
two
tracking
together
and
you
can
look
at
it
a
couple
of
other
ways
to
you
can
say
you
know
what,
if
you,
if
we
loosen
it
up
a
little
bit
and
we
say:
okay,
they
don't
have
to
match
exactly.
You
know
for
what
percentage
of
the
population
do
we
have
update
behavior
and
remediation
behavior?
That's
that's
similar
within
20
percent
of
each
other.
A
B
Maybe
just
pause
there
if
we
go
back
that
slide,
so
I
think
the
graph
depiction
is
actually
maybe
a
little
bit
misrepresentative,
because
the
dots
are
so
big,
but
it
just
to
reiterate
what
Stephen
said
is
that
most
of
them
fall
within
that
cone
right
that
MTTR,
an
MTT
you
do
generally
kind
of
are
within
that
cone.
Yeah.
A
And
if
you
think
about
you
know
what
would
it
mean
to
not
be
in
that
cone
to
not
have
similar
update
and
remediation
behavior,
that
would
be
being
say
better
than
average
at
remediating
security
vulnerabilities,
but
worse
than
average,
for
updates
in
general
right.
So
these
a
project
like
that
would
be
paying
a
lot
of
attention
to
security,
relevant
updates,
but
basically
ignoring
other
updates,
and
we
only
see
15
percent
of
the
population
falling
within
that
category.
So
it
really
is
rare
to
have
exemplary
security
performance
without
just
generally
good
update
performance.
B
B
Yeah,
so
so
the
conclusion
I
field
interpretation
is
that
projects
that
update
depending
more
frequently
are
generally
more
secure.
In
fact,
I
think
on
the
next
slide,
there's
actually
some
more
guidance
on
this.
This
actually
came
from
Jeremy
long,
so
he's
the
author
of
the
famous
Oh
wasp
dependency
check
project,
and
he
and
one
of
the
things
that
he
said
at
a
security
conference
that
just
really
struck
me
was
it.
Was
this
the
best
way
to
stay,
secure,
meaning
and
apply
security
patches
is
just
to
stay
up
to
date.
B
Right,
in
other
words,
security
patching
is
a
subset
of
dependency,
update,
behavior,
and
he
gave
a
really
interesting
case.
It
was
a
prime
phases
CD
that
came
out
in
2017,
so
this
is
a
vulnerability
and
some
component,
but
what's
interesting
is
that,
even
though
the
vulnerability
was
published
in
2017,
the
fix
was
actually
pushed
out
in
2016.
So
if
you
had
stayed
up
to
date,
you
would
not
have
been
vulnerable
to
the
vulnerability.
In
fact,
the
publishing
of
the
vulnerability
is
actually
what
then
drove
you
know.
B
The
exploit
is
long
billy
and
now
suddenly
have
crypto
mining
happening.
You
know
on
your
servers
so
again,
I
think
this
is
a
great
evidence.
Point
that
says
you
know.
The
best
thing
that
you
can
do
is
integrate
updating
dependencies
into
your
daily
work.
There
should
be
a
part
of
your
regular
development
activity.
Another.
A
B
Okay,
so
hypothesis,
one
was
about
our
beliefs,
frequency
that
lead
to
better
outcomes.
Yes,
I
possible
number
two
was:
is
there
a
link
between
MTTR
and
MT
tu?
Right,
yes,
hypothesis.
Number
three
is
pretty
obvious
right
prod.
If
it's
difficult
to
update
dependencies,
it's
pretty
obvious
right
that
projects
with
fewer
dependencies
will
be
able
to
stay
up
to
date
more
easily
more
frequently
whatever
right.
It's
something.
B
A
B
A
All
right,
and
so
projects
with
more
dependencies
also
tend
to
have
larger
development
teams,
and
if
you
just
look
at
larger
development
teams
by
themselves
that
correlates
with
with
faster
MTT,
you
better
update
hygiene,
better,
you
know,
release
frequency
and
so
forth
and
so
yeah.
So,
as
Jean
said,
this
is
a
plot
of
those
two
against
each
other
and
you
can
see
sort
of
as
dependencies
March
up.
You
know
you
have
to
buy
more
pizza,
but.
B
B
B
Interesting
thumbs
up,
oh
cool,
and
so
specifically,
if
you
go
back
one
slide,
maybe
just
land
the
point:
larger
developing
teams
have
50%
faster
mt,
tu
and
release
2.6
times
more
frequently.
So
that's
really
interesting
and
in
some
ways
curious,
so
now,
they're
doing
a
lot
more
questions
is
it
that
is
it
that,
as
you
grow
development
size,
if
you
grow
the
team
size,
do
they
all
just
bring
in
their
own
components
or
is
it
the
other
way
around?
B
A
All
right,
the
last
type
othe
assists
that
we
want
to
talk
about,
has
to
do
with
popularity
and
in
particular
whether
more
popular
projects
will
be
better
about
staying
up-to-date.
So
you
know
if
we
take
update,
behavior,
update
hygiene
and
and
thus
remediation
behavior
as
important
security
and
and
project
health
metrics
are
popular
projects.
A
You
know
exemplary
with
respect
to
those
attributes,
and
this
is
important
because
I
think
a
lot
of
people
like
choose
primarily
based
on
popularity
right
I
need
a
logging
library,
I'm
gonna
go
see
what
the
most
popular
logging
library
is
right,
but
actually
we
did
not
find
evidence
for
this
and
and
one
one
bit
of
counter
evidence
is
present
in
this
chart.
So
this
is
a
graph
where
the
x-axis
is
released.
Frequency
so
at
the
far
left,
is
products
that
release
very
frequently.
The
far
right
is
very
slow.
A
Release
cycles
and
popularity
is
on
the
y-axis,
and
this
is
a
log
scale.
So
you
can
see
it
goes
up
and
it's
a
number
of
daily
downloads
on
maven
central
and
it
goes
up
into
the
millions
right
and
so
what
I've?
What
we've
drawn
here
with
this
box
is
a
slice
of
this
population
that
is
quite
popular
with
respect
to
maven
central
downloads.
I
mean
the
the
bottom.
A
There
is
10th
you're,
getting
at
least
10
maven
central
downloads
per
day,
if
you're
in
that
box,
but
there's
a
lot
of
poor
performing
projects
from
an
update
hygiene
perspective
in
there.
So
I'm,
not
I'll,
say
more
about
the
different
colors
in
a
minute,
but
red
is
bad
okay.
So
the
fact
that
there
are
so
many
red
dots
you
know
in
this
in
this
popular
box
shows
that
there
certainly
are
popular
projects
that
are
not
good
with
respect
to
update
hygiene.
A
So
that's
the
first
point
here:
there's
plenty
of
components
with
4mt
to
you
that
are
popular,
but
you
know
maybe
those
are
just
out
wires.
There's
always
some
outliers
right
and
but
you
can
look
at
it
a
couple
of
other
ways
and
see
that
actually,
if
you
look
for
a
correlation
between
popularity
and
MTT,
you
know
maybe
there's
some
weak
signal
there,
but
no
nothing
statistically
significant
in
terms
of
correlation
there,
and
even
if
you
try
and
slice
up
the
data-
and
you
say
well,
let's
just
focus
on
like
the
top
10%
by
popularity
right.
B
So
this
is
like
in
one
of
those
categories
of
fun,
yeah
now,
findings
that
you
just
really
try
to
find
right.
You're
gonna
do
whatever
it
takes
to
like
get
that
signal
come
out,
and
we
spent
weeks
on
this
and
no
matter
how
we
cut
the
data
and
no
matter
how
we,
you
know
applied
the
you
know,
clustering
analysis.
You
know
we
could
get
to
show
up
and
I
think.
B
The
reason
why
it
was
so
troubling
to
me
is
that
whenever
I'm
working
on
a
project
and
I
have
to
pick
a
new
something
right,
logging,
Frank
or
whatever,
like
you
know,
here's
my
heuristic
I
go
to
github
and
I.
Look
for
how
many
stars
they
have
right
now.
If
it's,
you
know,
thousands
of
people
are
using.
It
I,
just
assume,
that's
a
good
project
to
use,
and
so
apparently
this
is
actually
not
a
very
good
heuristic
at
all.
A
A
A
Alright,
so
then
we
we
wanted
to
look
more
at
sort
of
the
the
high
performing
projects,
the
low
performing
projects
and
really
see
like
what
what
factors
bind
them
together.
What
behaviors
do
they
have
in
common?
Really
like
what
behavioral
clusters
can
we
identify
in
this
data,
and
so
one
interesting
thing,
so
we
took
the
exemplars
from
an
update
performance
perspective,
so
the
top
20%
according
to
update
behaviors
and
looked
at
them
more
closely
and
one
thing
I
find
really
interesting
here.
Is
we
found
roughly
equal
clusters
where
one
has
a
small
development
team?
A
A
So
you
don't
need
a
huge
team
to
stay
up-to-date
with
dependencies
right.
It
just
takes
focus
and
attention
to
that
being
a
key
development
concern.
We
also
certainly
saw
large
team,
exemplars
and-
and
really
this
is,
if
you
look
at
the
open
source
foundation,
supported
projects,
the
Apache
foundation,
Linux
Foundation,
those
sorts
of
things
that
really
the
you
know,
the
the
bedrock
of
the
open
source
community.
These
large
projects
that
are
that
are
an
important
part
of
so
much
infrastructure.
A
B
And
by
the
way,
this
is
like
one
of
the
sort
of
like
kind
of
really
surprising
things
for
me
was
this:
when
you
do
this
kind
of
clustering,
that
the
thing
that
pops
out
is
that
you
have
this
in
my
mind,
I
sort
of
kind
of
called
it,
the
Oh,
the
open
source
industrial
complex.
These
large
projects
that
have
on
you
know
small
army
of
developers
working
on
it,
there's
measured
by
10
plus
unique
developers
on
it's
clearly
part
of
their
day
job
to
work
on
this.
B
You
know
the
fact
that
you
know
those
exists,
you
know
confirmed
what
was
so
interesting
is
that
being
example,
I
was
not
in
a
domain
of
these
commercially
supported
or
foundationally
supported
projects.
So,
in
hindsight
you
know,
of
course
you
know
being
exemple
or
doesn't
depend
on
the
size
of
team
right.
It's
about
the
values
of
the
team
right,
it's
about
the
practical
practice
of
what
do
they
hold
important
and
for
the
same
reason,
that's
why?
I,
probably
the
presence
of
a
CI
configuration
file
is
not
a
marker
performance.
A
As
further
evidence
of
the
importance
of
values
and
philosophy,
we
have
this
cluster
in
the
bottom
middle
here
features
first
right
and
the
the
this
was
a
group
of
projects
that
actually
releases
updates,
frequently
right
so
they're
they're,
pushing
out
new
code,
they're
tagging
versions
and
and
and
cutting
new
releases
all
the
time.
So
they
have
the
bandwidth
to
stay
very
up-to-date,
but
they're
not
right.
A
Dependencies
and
sort
of
what's
happening
there
and
just
focusing
on
features
and
changes
to
the
it
to
their
code
itself.
Right
and
so
that's
an
example
of
you
know
it's
not
just
about
the
practices.
It's.
What
you
hold
is
important
as
you're
as
you're
doing
the
development
practice
right,
and
so
then
there's
there's
the
laggards
cluster
that
I
mentioned.
Those
are
the
red
ones.
So
you
know
I
said
red
is
bad.
A
That's
the
bottom
20%
with
respect
to
update
behavior
and
then
there's
this
interesting,
cautious
cluster
that
actually
manages
to
stay
within
a
couple
of
versions
of
the
latest
but
they're.
You
know,
for
whatever
reason,
not
willing
to
sort
of
be
on
that
bleeding
edge.
They
wanna
the
community
to
vet
it
a
little
bit
before
they
move
forward,
but
they
do
pay
attention
to
updates
to
their
dependencies
and
and
eventually
adopt
them
right.
B
And
this
is
to
me
that
says
that
there's
some
disciplined
approach
there's
some
sort
of
target
ban.
They
want
to
be
in
terms
of
being
up-to-date,
maybe
that
the
latest-
but
you
know
clothes,
the
latest
yeah.
So
you
know
so
one
of
you
do
these
clusters
right
often
you're
kind
of
detecting
differences.
You
know
in
attributes
of
groups
and
then
usually
what
you'd
like
to
do
then
is
really
understand
the
psychographics
of.
Why
do
they
act
that
way
so
clusters?
B
B
You
know,
thanks
to
the
amazing
team
of
snow
type,
is
we're
able
to
put
out
a
survey
where
we,
the
goal,
was
to
actually
try
to
uncover
some
of
these
psychographics
or
at
least
philosopher's
philosophic
decision
of
these
groups,
and
so
the
outcome,
measures
that
we
asked
was
to
what,
how
paint
on
a
scale
of
one
to
seven,
how
painful
is
updating
dependencies
and
how
painful
on
a
scale
one
to
seven,
how
painful
is
updating
a
vulnerable
components?
And
so,
let's
focus
on
that.
So
this
is
cluster
that
they
said
there
were.
B
So
this
group,
when
compared
to
the
others,
they
were
ten
times
more
likely
to
schedule
updating
dependencies
as
a
part
of
their
daily
work,
in
other
words,
they're
making
updating
dependencies
a
part
of
what
they
do.
You
know
whether
it's
every
day
or
every
week,
therefore
six
times
more
likely
to
strive
to
use
the
latest
version,
or
you
know
n,
whatever
right
I,
don't
want
to
be
on
latest
I
wanna
be
on
n
minus
one
or
something
right
whatever
that
band
is,
there
are
that's
one
of
their
desires.
B
There's
some
there
11
times
more
likely
to
have
some
sort
of
process
before
adding
a
new
dependency
to
their
project.
In
other
words,
instead
of
adding
things
willy-nilly
or
on
a
whim,
they
go
through
some
process,
evaluate,
approve,
standardize,
etc.
11
times
more
likely,
they're
ten
times
more
likely
to
have
a
process
to
remove
problematic
dependencies
and
they're
twelve
times
or
unless
they
have
some
sort
of
automated
tool
to
manage
track.
The
compliance
towards
those
policies,
yeah.
A
And
so
those
survey
result
more
on
those
survey
results
and
more
detail
on
everything
that
we
discussed
is
in
the
actual
report.
So
you
know,
go
download
it
if
you'd
like
to
receive
a
copy
of
the
report.
Actually,
my
email
address
is
there
at
the
bottom,
if
you
just
send
an
email
right
now,
my
out
of
office
autoresponders
on
and
will
reply
with
a
link
directly
to
the
report,
so
you
know
easy
way
to
go,
get
the
report
and
dig
down
deeper
and
see
all
the
other
things
that
we
found
and.
B
To
summarize,
I
mean
I
think
this
is
such
a
rewarding
and
interesting
project,
because
I
think
this
is
the
first
time
that
we've
actually
start
see.
You
know
the
behaviors
that
we
suspect
existed
and
it
really
illuminates
some
surprising
insights.
So
again,
thanks
for
the
sonoran
type
team
for
giving
us
access
to
the
data
and
assisting
this
project
that
does
agree
that
they
did
so.
A
We
had
so
much
fun
with
year,
one
of
this
we're
already
talking
about
year.
Two
and
you
know
the
additional
analysis.
Additional
research
we'd
like
to
fold
into
the
second
version
of
this
study,
and
and
so
some
of
the
things
that
we'd
like
to
focus
on.
Are
this
idea
of
breaking
changes
right
like
art?
A
Are
there
we
just
looked
at
updates
in
general,
but
are
some
updates
easier
to
adopt
than
others,
and
could
that
underlie
some
of
the
behavior
that
we
see
looking
at
transitive
dependencies
so
deeper
into
the
dependency
tree
and
identifying
some
of
the
leading
indicators
right?
Can
we
get
beyond
correlation
and
start
talking
about
causal
factors
that
underlie
some
of
these
behaviors?
Now.
B
B
You
wait
long
enough,
you
will
take
a
breaking
change
in
Java,
and
so
this
is
frightening.
I
think
it
resonates
with
our
common
experience
on
the
next
slide,
which
is
that
you
know
the
reason
why
we
don't
patch
don't
update
our
components.
Is
that
every
time
we
do
everything
breaks
right
and
so
how
one
of
those
our
goals
is
one's
a
help,
but
we're
looking
for
is
this:
is
there
theater
on
depend
a
bot?
You
know
that
says
what
happens
when
that
pull
request
is
made?
B
What
percentage
of
them
are
quickly
and
easily
integrated
and
which
ones
are
never
integrated?
Because
you
know
everything
blows
up,
you
know
when
you
do.
We
also
think
that
you
know
even
the
notion
of
breaking
changes.
There
might
be
something
fundamentally
wrong
with
that
right,
in
other
words
in
the
closure
community,
they've
gotten
almost
12
years,
without
a
breaking
change
in
the
standard
library,
right
and
so
I
think
that's
part
of
the
functional
programming
orientation.
So
is
there
a
marker
where
we
can
find
libraries
are
really
good
at
not
breaking
on
updates?
B
They
add
that's
okay,
but
you
can't
subtract
right
and
we're
also
looking
at
mechanisms
to
find
pull
requests
lead
times
and
issue
resolution
times,
and
thank
you
to
the
code
gov
team
who
presented
yesterday
I'll
be
finding
you
I
love
the
fact
that
you've
done
this.
We
were
hoping
to
use
that
next
year.
Study
as
well.
B
Second,
one
is
like
it's
not
enough
to
clean
up
your
house.
You
got
to
help
clean
up
other
people's
houses
for
all
the
dependencies
you
take
for
you
to
update
their
dependencies
right.
Put
in
pull
requests.
I
think
is
enormous,
ly
helpful,
because
we
know
that
you
know
there's
a
lot
of
work
to
do
that.
Anything
you
can
do
to
help,
helps
yeah.