►
From YouTube: Technical Lag: Measuring the outdatedness, vulnerability and bugginess of software | Tom & Ahmed
Description
Technical Lag: Measuring the outdatedness, vulnerability and bugginess of software
This talk presents a generic and empirically validated measurement framework based on the technical lag concept. The framework can be used to assess and reduce the outdatedness, vulnerability and bugginess of software deployments, software projects, software containers and reusable software libraries. We argue that such a metric is very relevant for assessing the health of software (eco)systems, and should become part of the CHAOSS metrics portfolio and tooling. The concept of technical lag aims to quantify to which extent a deployed collection of components is outdated with respect to an ideal deployment. How to interpret this ""ideal"" and the ""outdatedness"" w.r.t. this ""ideal"" is highly context-specific. Depending on the needs and goals of a specific project or a maintainer, the focus may be on functionality, security, stability or even other factors. The ""components"" under consideration could be individual software packages, third-party libraries, component dependencies, or software containers bundling collections of components. To cover this high variability, we have come up with a generic framework for technical lag. We have operationalised and empirically validated this framework in different contexts: to assess the technical lag incurred by outdated dependencies in reusable package repositories such as npm; and to quantify the outdatedness of Debian-based Docker containers in terms of missing updates, vulnerabilities and bugs. We also report on the qualitative evaluation of the usefulness of measuring technical lag, through online surveys conducted with open source software developers. Finally, we present Conpan, an open source tool that we have developed for analysing technical lag in Docker containers. Useful references about technical lag are available on https://secoassist.github.io/results.html
Slides: https://chaoss.github.io/website/CHAOSScon/2020EU/slides/techlag.pdf
-----
Tom Mens
Professor in Software Engineering - University of Mons, Belgium
@tom_mens
Ahmed Zerouali
Postdoc in Software Engineering - University of Mons, Belgium
@a_zerou
-----
About CHAOSScon
Learn about open source project health metrics and tools used by open source projects, communities, and engineering teams to track and analyze their community work. This conference will provide a venue for discussing open source project health, CHAOSS updates, use cases, and hands-on workshops for developers, community managers, project managers, and anyone interested in measuring open source project health. We will also share insights from the CHAOSS working groups on Diversity and Inclusion, Evolution, Risk, Value, and Common Metrics.
https://chaoss.community/chaosscon-2020-eu/
A
So
here
today,
I
will
talk
about
some
research.
It
has
been
going
on
I
start
to
do
the
collaboration
with
azuz
Gonzalez,
who
is
unfortunately
not
not
here
today,
also
from
bitter
joy
from
diversity
in
Madrid.
It's
also
a
part
of
the
PhD
research
of
Hamid,
who
is
over
there
hidden
in
the
back
with
digital
Jia
guys,
and
it's
part
of
the
sake
of
Asus
project
that
we
are
currently
involved
in
basically
elizabeth's
try
to
provide
support
for
assisting
software
communities
in
software
ecosystems,
so
not
individual
software
projects,
but
larger
collections
of
projects.
A
Basically,
what
we
wanted
to
talk
about
today
is
this
notion
of
technical
life.
Basically,
the
problem
is
so
you
are
either
a
software
deployer
or
a
software
developer
and
you're
facing
you
have
components
that
might
be
out
of
date.
You
depend
on
other
components
that
might
be
out
of
date,
and
then
you
have
the
you
have
to
find
the
right
balance.
Should
I
update
my
components?
Should
I
stay
out
of
dates?
Is
it
good
or
bad?
The
problem
is,
if
you're
out
of
date,
you
might
have
worked,
you
mighta
vulnerabilities.
A
A
Well,
if
you
are
depending
on
like
external
libraries
or
components,
what
would
be
the
most
appropriate
version
you
would
like
to
depend
on
because
would
you
like
to
have
the
latest
version,
the
most
secure
version,
the
most
stable
version?
There
is
many
different
factors
that
may
play
a
role
so
based
on
a
short
survey
with
17
online
respondents.
Actually,
we
see
that
the
results
are
quite
varied.
Some
people
prefer
to
have
the
most
stable
versions
from
the
latest
versions
from
the
most
secure
version.
So
it's
difficult
to
say
this
is
what
you
want
to
trigger.
A
So,
if
you
to
fight,
we
want
to
define
a
metric
that
that
can
measure
what
is
the
best
ideal
version
you
would
like
to
depend
on.
You
should
be
a
generic
measure
that
can
take
into
account
all
of
these
different
cases,
so
we
have
multiple
dimensions
that
play
a
role
so
in
order
to
define
this
ya
know.
A
A
A
Is
this
one,
the
increasing
difference
between
deployed
software
packages
and
the
ideal
packages
that
are
available
somewhere
upstream
now
that
you
see
here,
basically
there's
two
notions,
difference
and
ideal,
it's
quite
vague.
What
does
this
mean
difference?
How
do
you
measure
difference
and
what
does
it
mean
ideal?
What
is
the
ideal
upstream
package?
As
we
have
seen
in
the
survey?
There
is
no
single
notion
of
ideal
and
there
is
no
single
way
to
measure
difference.
A
So
ideal
could
be
the
most
stable,
the
most
bug-free,
the
most
secure,
the
most
functionality,
the
most
recent,
while
difference
will
be
difference
measured
in
time
in
terms
of
versions
in
terms
of
amount
of
perks,
amount
of
vulnerabilities.
Any
other
thing
you
can
think
of
so
last
year
in
Foss,
then
we
also
did
some
interviews
to
find
out,
we
just
admit,
went
to
first
then
he
explained
this
notion
of
technical
AK
informally.
A
To
find
out
would
it
be
useful
to
have
such
a
notion
that
can
measure
this
notion
of
technical
lag
and
basically
most
of
the
respondent
says?
Yes,
it's
good
to
have
such
a
technical
like
notion,
but
it's
important
to
take
into
account
all
of
these
different
dimensions,
and
in
addition
to
this,
it
would
also
be
useful
to
be
able
to
measure
the
effort
required
to
update
to
the
most
ideal
fashion.
So
this
was
basically
the
reason
why
we
said:
okay,
let's
try
to
come
to
such
a
generic
framework
for
measuring,
technically
I.
A
A
In
that
case,
you
can
say
that
your
collection
of
some
type
of
policy
is
outdated
and
you
have
a
technical
lag,
which
is,
let's
suppose
that,
for
this
particular
version
there
is
a
new
version
here,
which
is
which
use
which
do
you
consider
to
be
the
ideal
version.
In
that
case,
you
can
say:
I
have
a
lag
of
this,
which
can
be
either
measured
in
time
or
measured
in
number
of
versions
or
measured
as
number
of
works.
A
Of
other
pivots,
so
what
we
did
was
now
we
try
to
formalize
this.
So
basically,
here
it's
I
try
to
limit
the
amount
of
mathematics,
so
it's
very
limited.
Basically,
we
define
just
a
technical
like
framework
where
the
most
basic
things
are.
You
need
a
function
ideal
that
says,
if
you
have
your
currently
installed
components,
what
is
the
most
ideal,
one
that
you
can
find
upstream
and
a
delta
function
with
switch
says
that
if
you
have
two
components,
your
current
installed
one
and
the
upstream
one?
A
What
is
the
lag
that
you
can
compute
as
a
difference
between
those?
If
you
have
these
two
functions,
you
can
define
your
notion
of
technical,
like
technical
AK
is
basically
simply
the
difference
between
your
currently
installed
version
and
the
ideal
one
and
the
aggregated
lag
over.
Your
entire
collection
of
component
problems
is
any
kind
of
function
that
allows
you
to
aggregate
this
technical
act
over
all
of
your
components.
Currently
installs
deployed
developed
I
have
a
tendency
to
talk
a
little
bit
too
quickly.
So
let
me
brief
a
bit
not
so
now.
A
This
is
the
technical
extreme
work.
So
now
I
will
show
you
a
couple
of
different
instantiations
of
this
technical
act
framework,
for
example.
Suppose
you
would
like
to
measure
the
time
based
technical
AK.
In
that
case,
suppose
I
have
a
package
that
deploys
and
the
current
version
of
these
packages
that's
being
deployed
is
one
not
one
of
zero,
and
that
is
you
can
see
that
already
a
couple
of
different
versions
later
on
available
the
most
recent
version.
Suppose
we
want
to
select
the
most
recent
extreme
release.
A
A
Suppose
I
want
to
measure
the
version
based
technical
like
in
that
case,
you
are
simply
looking
at.
If
this
is
your
currently
installed
version,
which
is
the
most
recent
version
with
versioning,
sometimes
it
can
be
that
the
most
recent
release
is
not
the
most
recent
version,
so
here,
for
example,
this
is
the
most
recent,
not
this
one.
So
in
that
case
your
technical
neck
will
be
your
behind
with
one
major
release
and
then
your
microphones
patch
release.
A
So
the
technical
leg
will
be
one
major
and
one
patch
version
or
another
one
for
the
ulnar
ability
based
technical,
like
you're,
currently
deployed
component.
Is
this
one
you
want
to
have
the
most
least
vulnerable
ones,
you're
going
to
look?
Is
there
another
later
version
that
has
less
vulnerabilities?
This
one
is
better,
so
you
can
take
this
as
the
ideal
version
the
measurement
would
be.
You
can
reduce
your
vulnerability
with
one
by
operating
to
a
newer,
make
it
the
next
release.
A
This
is
called
it.
A
vulnerability,
lack
or
security
like
ba
black
are
basically
the
same.
You
look
at
the
number
of
bugs,
and
then
you
can
see
my
currently
deployed
package
has
one
bug.
There's
another
one
here:
three
bugs
known
drugs
I've
got
the
unknown
bugs
we
don't
know
about
it.
In
that
case,
you
can
say:
okay
I
will
not
upgrade
to
newer
one,
because
it
has
more
bugs
so
either
I.
A
A
Okay,
so,
based
on
this,
we
did
like
a
quantitative
empirical
study
on
NPM
packages,
so
we
took
call
and
PM
packages
and
for
each
package
we
studied
the
amount
of
technical
equi
had
since
there
are
so
many
different
notions
of
technical,
like
we
only
looked
at
time,
lag
and
version
like
all
of
the
results.
If
you
want
to
really
know
about
them
can
be
found
in
this
paper.
Basically,
let
me
just
take
a
simple
example.
If
you
take
NPM
packages
for
each
package,
you
have
dependencies.
So
this
one
has
three
runtime
dependencies
15
development
dependencies.
A
We
only
focus
on
the
runtime
dependencies.
The
meta
data
is
available.
There
is
some
animation
here,
so
here
you
can
see,
for
example,
in
the
meta
data,
how
you
depend
on
each
other
package.
We
use
dependency
constraints
to
see.
Basically,
you
have
a
kind
of
flexible
mechanism.
Since
you
can
depend
on
other
versions.
You
have
a
range
of
dependencies
using
the
carrots
which
basically
means
I,
accept
all
patches
or
all
minor
releases,
but
I
don't
accept
the
major
releases
by
doing
so.
A
For
this
particular
example,
if
I
take
the
transitive
dependency
graph,
I
have
something
like
this,
and
you
can
see
that
this
package,
this
particular
version
of
this
package,
suppose
that
this
is
the
one
you
have
installed.
Then
it
is
lagging
the
hives,
because
using
this
constraints,
its
behind
its
it
can
still
accept
this
version.
But
all
the
versions
with
major
version
3
cannot
be
installed.
So
you
are
lagging
behind
with
3
versions,
with
the
distance
between
this
state
and
this
state.
A
Here
is
suppose:
was
it's
6
months,
6
MOBA
hind,
so
there,
depending
on
how
you
measure
your
technical
ACK,
you
could
only
take
the
direct
dependencies
into
account
in
that
case
is
quite
acceptable
if
you
take
the
indirect
dependencies
into
account
as
well.
You're
like
we'll
be
much
higher
so
based
on
this
and
using
the
measurement
framework
suppose
that
these
missed
versions
are
fixing
strong,
bugs
or
vulnerabilities.
If
you
don't
update,
you
might
have
vulnerabilities
in
your
components.
A
If
you
know
that
they
are
fixed,
you
will
not
have
the
case,
so
you
can
actually
use
these
metrics
to
measure.
How
far
am
I
behind?
If
I
only
take
the
diary
dependencies
we'll
be
four
days
in
terms
of
time
lag,
if
I
will
take
the
indirect
ones
into
account,
it
will
be
198
days
if
I
add
negate
all
of
this
together.
A
The
technical
lack
of
my
component
in
terms
of
Steinbach
will
be
the
maximum
of
both,
which
is
198
days,
assuming
we'll
take
the
maximum
which
you
can
take
whatever
other
aggregation
function
you
would
like.
So
so
we
think
that
this
is
a
really
generic
and
useful
metric
to
use.
If
you
want
to
be
aware
of
how
up-to-date
you
are,
there
is
already
like
some
tools
are
starting
to
implement
this.
For
example,
there
is
the
David
dependency
management
tool
which
is
available
for
JavaScript
I.
A
A
Package,
this
all
of
its
dependencies-
and
you
can
see
using
these
constraints,
which
of
the
dependencies
are
out
of
date
or
up
to
date
up
to
date,
is
green
out
of
date
is
red,
and
you
can
also
more
why,
because
of
the
use
of
your
dependency
constraints,
so
it
is
already
like
a
kind
of
instantiation
of
the
technical
egg.
We
didn't
implement
it
ourselves,
it's
also
quite
recent.
Well,
it
would
be
nice
to
have
these
types
of
things
in
I
need
to
and
in
a
generic
way.
A
A
So,
basically,
a
container
is
a
set
of
collection,
of
components
that
are
installed
deployed
and
then
again
the
problem
is,
you
are
your
topper
containers,
your
image
could
be
out
of
dates,
and
because
of
this
you
can
have
security
problems
and
all
of
the
other
things
I
mentioned
before
and
again
you
can
measure
the
technical
act
of
your
daughter
containers
by
looking
at
all
possible
package
releases
within
your
local
container
and
looking
at
if
there
is
already
a
new
upstream
version.
That
is
more.
Recent
in
this
case
is
for
Debian
package
releases.
A
We
did
a
similar
study
where
we
looked
at
docker
containers
that
are
actually
new
the
NPM
packages,
and
then
you
can
replace
this
by
NPN
package
releases,
some
similar
notion
and
then,
of
course,
you
can
again
start
measuring
the
technical
ACK.
The
main
purpose
of
this
is
to
just
show
it
doesn't
really
matter
if
you
have
a
collection
of
components
that
are
either
depending
on
others
or
installed
together
as
soon
as
as
long
as
you
can
define
some
notion
of
upstream
versus
downstream,
you
can
implement
this
notion
of
technical
act.
A
So
here
again,
there
is
another
example
of
to
support
that
you
can
provide
in
this
case,
I
took
the
example
of
the
sneak
security
management
system.
It's
it's
not
a
service
commercial,
but
there.
It
also
has
support
for
docker
containers,
for
example.
Here
you
can
see
even
in
a
particular
docker
container,
this
image
it
has
a
certain
number
of
vulnerabilities,
and
basically
this
tool
also
supports
how
you
could
fix
these
vulnerabilities
and
it's
basically
the
same
notions
of
this
security
like
at
the
container
level,
ideally
to
be
now
have
some
kind
of
open
source
crew.
A
That
also
does
take
this
into
account.
But
to
do
this
we
needs
basically
open
source
database
of
vulnerabilities,
which
is
difficult
to
get
so
to
summarize,
and
personally
I'm
convinced,
I
hope,
partially
convinced
you
as
well.
That
is
useful
to
have
such
a
notion
of
technical
axe
using
different
ways
of
measuring
the
lag
time
version
security,
vulnerabilities,
operationalizing
it
in
different
ways
for
containers
for
dependency
management
systems.
Maybe
there
are
some
other
things
I
didn't
think
about,
and
to
propose
this
as
part
of
the
case
metrics
and
tooling,
that's
currently
existing.
A
There
are
still
some
open
research
challenges.
What
we
didn't
do
and
it's
quite
difficult.
How
can
we
actually
measure
how
much
effort
will
it
take
to
upgrade
from
one
particular
version
to
another,
from
one
collection
of
components
to
a
newer
one?
This
is
very
difficult.
It
requires
static
and
dynamic
code
analysis.
A
How
can
we
combine
actually
multiple
dimensions?
Suppose
I
want
to
have
the
most
secure
and
also
the
most
bird
free,
but
still
the
most
up-to-date
one
I
cannot
I
have
to
balance
this
different
mentions
and
also,
if
I
upgrades
I
might
have
breaking
changes.
How
can
I
avoid
breaking
changes
but
still
use
the
most
up-to-date
version?
A
All
of
these
are
really
difficult
challenges
to
do
so,
based
on
this
well
to
start
implementing
metrics
for
Kaos
I
think
we
have
to
start
with
simple,
simple
things
like
dependency
metrics,
you
can
just
start
measuring
how
many
dependencies
we
have
of
all
of
these.
How
many
are
outdated
of
all
of
these?
A
So
this
is
the
last
slide,
so
just
some
commercial
break
as
last
year.
I'm
not
involved
this
year,
but
there
is
also
the
XE
workshop
on
software
hell
that
will
be
organizing
sale
in
South
Korea.
If
you
happen
to
be
in
the
neighborhoods,
there
is
still
the
opportunity
until
next
week,
Friday
to
provide
either
small
papers
or
talk
proposals
there,
but
the
deadline
is
next
week:
Friday!
That's
it.