►
From YouTube: Securing Critical Projects WG Bi Weekly (April 7, 2022)
Description
No description was provided for this meeting.
If this is YOUR meeting, an easy way to fix this is to add a description to your video, wherever mtngs.io found it (probably YouTube).
A
C
E
Thanks
yeah
amir
is
out
today
so
I'll
be
shepherding
this
meeting
as
usual.
We'll
start
at
about
five.
After,
if
you
have
any
topics,
go
ahead
and
drop
them
into
the
meeting
agenda,
that's
linked
to
the
calendar!
Invite,
if
you
don't
have
any
of
that,
just
ask
in
chat
or
ask
here.
D
So
I
I
want
to
re-add
to
the
agenda
and
I'm
going
to
put
jacques
on
the
spot,
and
I'm
sorry
about
that,
but
I
know
this,
you
know
it's
the
whole
hot.
You
know
I'd
like
to
continue
our
discussion
also
about
how
do
we
read
you
know:
how
do
we
determine
and
update
the
replace
the
critical
projects
list,
combining
the
data
with
human
information?
I
know
you
care
about
this
and
I
do
I
care
about
it.
A
Briefly,
holy
people
talk
a
lot
about
a
lot
of
things.
There
are
several
literatures
and
they're
enormous,
so
I
barely
skim
the
surface
got
it.
A
D
A
He's
his
name
been
added
by
I.
D
F
E
Yeah
thanks
everyone
for
joining
and
thanks
for
adding
adding
to
the
agenda
as
usual.
We'll
start
the
meeting
with
new
and
attendee
introductions.
G
Hi,
I'm
marina,
I'm
a
phd
student
at
nyu,
I'm,
I
think
I
know
some
of
you
from
other
working
groups
and
such
but
new
to
this
particular
one.
I'm
here
kind
of
as
so
I've
been
participating
in
the
securing
software
repositories
working
group,
the
new
working
group
there
and
one
of
the
things
we've
discussed
there.
That's
kind
of
relevant
to
me
as
an
academic
is
the
possibility
of
creating
kind
of
a
repository
of
data
about
software
repositories
and
projects
on
them,
and
so
I
thought
I'd
come
here
and
kind
of
exchange.
G
Oh
sorry,
I
I
don't.
I
guess
I
didn't
explain
that
properly
sorry,
so
I'm
I've
been
working
with
maintainers
of
software
repositories.
So
thank
you
for
like
ipi
npm,
whoever
ruby
gems
I
see
jack
here
and
then
and
there
we
were
kind
of
trying
to
figure
out
if
there's
interest,
both
from
the
repository
maintainer
side
and
from
kind
of
the
open,
ssf
linux
foundation
side
in
having
a
central
place
to
store
some
data
about
these
repositories.
G
The
packages
on
them
the
usage
patterns
for
the
purposes
of
doing
both
research
projects
and
kind
of
things
like
the
stuff.
I
understand
this
working
group
is
doing
in
f
in
identifying
critical
things
common
issues.
You
know
anything
like
that.
That
could
be
useful.
E
So
when
you
ask,
if
you
want
to
know,
if
there's
interest
in
having
a
central
place,
are
you
interested
in
making
that
a
like
a
work
stream
of
this
working
group.
G
That's
part
of
the
question
is
where
this,
where
this
would
best
live,
is
like
maybe
in
this
working
group,
maybe
that
other
working
group,
but
just
want
to
talk
to
everybody
before
we
make
those
kind
of
decisions.
So.
G
E
All
right
any
other
new
attendees.
I
I'm
a
repeating
offender.
I've
been
here
before,
and
I've
been
around
openness
for
quite
a
few
months
now,
but
I
don't
regularly
attend
this
call.
So
I
have
actually
I'm
part
of
the
open
technology
group
at
ibm,
an
open
source
and
stereo
specialist
and
more
recently
I
started
playing
with
the
criticality
score
tool
and
I
actually
submitted
a
couple
of
full
requests
and
I've
had
some
chats
on
the
slack
channel
with
caleb,
and
so
I
was
available.
I
saw
the
call
on
the
agenda
I
figured.
I
would
join
see
what's
up.
J
J
I
have
led
our
work
on
the
stakeholder
specific
vulnerability,
categorization
stuff
criticality
seems
like
it
is
related
to
some
of
the
stuff
that
we
would
expect
to
use
to
prioritize
vols
sort
of
generally,
so
I'm
interested
in
it.
From
that
perspective,
I'm
also
on
things
like
the
cvss
sig
and
the
epss.
They
get
first
for
other
vole
categorization
stuff.
So
I
can
happy
to
talk
about
stuff,
that's
going
on
there
if
it's
relevant.
If
people
need
that
context,
thanks.
E
Welcome
jonathan
glad
to
have
you
anybody
else.
B
B
My
name
is
darcy
clark,
I'm
the
em
for
the
npm
cli
team,
as
well
as
the
github
cli
team,
so
unique
scope
there
and
have
been
for
about
three
years
working
on
npm
and
can
be
a
gateway,
hopefully
for
for
change
and
innovation
in
the
ecosystem
there,
but
also
have
been
working
closely
with
the
openjs
foundation,
helped
champion
the
package,
vulnerability
club
space
that
we
kept
off
last
year,
which
tangentially,
I
think
aligns
with
what's
happening
in
the
open
ssf
space,
so
want
consolidate.
Efforts
also,
ideally
tie
some
of
the
work.
B
That's
being
done
here
back
into
the
mpm
open
source
project,
so
yeah
happy
to
be
here.
I
would
love
to
contribute
in
any
way
possible
and
get
my
team
involved
and
yeah
extend
on
all
branch
when,
when
there's
one
that
can
be
extended,
so
yeah.
E
All
right,
I
guess
that's
about
it
so
move
on
to
the
agenda
and
something
that
we
had
talked
about
in
the
previous
meeting
that
we
wanted
to
do
today
was
discuss
the
charter
and,
if
you
click
the
link
and
it'll,
bring
you
to
issue
52
and
we're
very
thankful
to
see
rob
for
giving
us
a
really
good
example
here
on
the
second
link
on
that
issue.
The
charter
for
the
security
tooling
working
group.
E
And
anyways
so
onto
the
discussion
you
know,
does
anybody
has
anybody
looked
at
it?
Does
anybody
have
any
comments?
Has
anybody
been
part
of
this
process
in
another
working
group?
The
floor
is
open.
Oh
hey,
jacques.
You
have
your
hand
up.
A
Yeah
I've
been
I've.
I
read
through
it
as
part
of
setting
up
the
securing
software
replays
group,
it's
fairly
boilerplatey.
There
were
two
notes.
I
think
that
came
out
of
it
for
me
or
three
one
is
that
it
sets
up
some
guardrails
for
if
things
go
wrong,
so
it
has
a
it
has
a
formal
process
which
basically
would
show
up
once
a
million
years
if
things
are
going
off
the
rails,
so
I
think
that's
its
real
purpose
or
value.
A
So
at
some
point
you
have
to
sort
of
bootstrap
that
list,
and
the
third
is
that
this
is
just
a
very
small
one,
that
I
have
a
bug
open
on
the
exam,
the
template,
repo,
where
it
comes
from
there's
a
fragment
of
text
in
part
2f,
which
doesn't
seem
to
fit
it
talks
about
sending
voting
members
to
the
attack.
It's
just
like
that's
not
a
thing
that
you
can
do
like
the
tag
is
selected.
A
There's
no
there's
no
sort
of
delegating
upwards,
so
those
would
be
my
notes.
D
I,
if
I
may
jump
in,
I
think
the
key
is
you
know
some
sort
of
statement
of
what
is
this
group
officially
is:
what
is
the
scope
of
this
group?
I
think
that's
really
what
they're
looking
for
the
the
boilerplatey
thing
I
mean
there's
a
tsc
that
doesn't
exist.
I
think
that
just
needs
to
be
removed.
D
I
think
the
tac
would
like
to
have
somebody
from
the
working
group
showing
up
you
know
and
that
you
know
that
could
be
attack.
Member
or
somebody
else,
but
just
you
know,
I
think
the
concern
right
now
is
the
tac
doesn't
necessarily
always
hear
what
the
working
groups
are
doing
and
vice
versa.
So
I
think
there's
a
goal
for
some
communication.
A
D
Yeah-
and
I
think
that's
that's
reasonable,
because
you
know
this
is
the
first
year
we
actually
have
funding
now,
as
opposed
to
the
you
know
the
previous
system.
I
will
tell
you
that,
although
things
aren't
set
in
stone,
I've
been
interacting
a
little
bit
with
the
folks
are
trying
to
nail
that
down.
I
think
one
of
the
bigger
changes
that
I
I
would
like
to
see-
and
I
think
it's
it's
going
to
happen-
is
the
tac
would
like
the
working
groups.
D
You
know
kick
up,
prod
kick
kick
off
projects,
but
talk
to
the
tac.
You
know
basically
preliminary
you're
in
but
tell
the
attack.
So
if
there's
an
issue,
because
otherwise
it's
quite
possible
for
the
attack
to
never
hear
about
a
new
project,
and
that's
just
not
you
know
that
does
not
help
coordination,
because
right.
E
D
Do
that
I've
tried
to
do
that
every
one
I
have
put
in
pull
requests
when
you
create
a
new
project,
but
that
doesn't
mean
I've
always
been
successful
at
it.
So
absolutely
that's.
That
is
something
else
they
want
to
do
is
every
time
you
click
off
a
new
project,
hey
tac,
that's,
okay,
probably
it's
fine,
carry,
you
know,
carry
on,
add
it
and
then
and
then
add
it
to
the
the
readme
page,
so
that
yeah.
E
D
No,
no
because
I
I
they
want
to
be
notified
and
regardless,
if
it's
a
new
project,
hey
notify
as
far.
D
Sure
you
know
make
make
a
github
issue
on
on
the
attack
list.
That
would
be
fine.
Okay,
I
would
imagine
I
guess
I.
F
D
It
would
it
would,
it
would
not
be
a
bad
idea,
but
I
I
guess
technically
the
attack
hasn't.
I
think,
right
now,
just
the
folk
what
I
would
focus
in
on.
What
do
you
think
this
working
group
supposed
to
do
the
scope
so
kill
all
the
stuff
about
the
tsc,
because
we
don't
have
one?
I
think
the
key
is
just.
What
does
this
group
think
it's
doing?
We
have
it
some
text
in
the
readme.
A
E
Should
we
do
should
we
duplicate
the
mission
and
scope
between
the
charter
and
the
readme?
I.
A
D
Yeah
keep
keeping
it
dry
does
make
sense.
I
mean
if
from
the
charity
you
link
to
it.
I
guess
the
only
problem
would
be.
You
know
you
change
the
the
read
me
without,
but
yeah
I
mean
I.
I
think
the
point,
though,
is
that,
right
now
it's
not
always
clear
what
the
working
groups
are
think
they're
doing.
D
E
Okay,
so
yeah,
I
propose
that
we
update
the
we
take
the
charter,
as
is,
except
for
updating
number
one
to
be
jacques,
recommended
super
terse
and
and
pointing
to
our
readme,
where
we
have,
where
we'll
attend,
to
have
a
more
descriptive
mission
scope
and
then
on
the
issue
that
you
raise
shock
about.
The
kind
of
the
boilerplate
looks
wrong.
I
think
we'll
just
wait
until
we
get
that
resolved
upstream
and
then
pull
it
down.
A
E
I'll
put
that
onto
the
issue
that
would
have
been
a
better
idea.
E
Oh
great,
I
copied
the
zoom
link
into
the
issue.
Anyways
I'll
fix
that
okay,
so
yeah
as
far
as
getting
this
merged.
If
we
make
those
changes,
leave
the
pr
open
should
just
leave
the
pr
open
until
next
week
next
meeting
and
then
we'll
have
a
last
call
for
comments.
D
A
Don't
need
a,
I
see
that
mostly
as.
A
As
part
of
the
mechanism
of
guard
rails,
that's
why?
I
think
the
only
thing
that's
really
essential
is
to
define
your
maintainers,
because
in
it's
been
a
week
or
two
since
I
read
it.
So
it's
a
little
big
and
fuzzy
in
my
memory,
but
it
mostly
bootstraps
off
who
are
the
maintainers.
D
A
D
I
Yeah
and
there
were
discussion
exactly
on
that
point
david.
I
agree
with
you,
but
I
don't
think
there's
been
any
formal
resolution
and
I
suspect
the
answer
is
going
to
be.
Oh,
we
have
a
task
force
working
on.
You
know
the
guardians
documents
and
that
will
be
the
answer,
but
so
it
means
it's
not
going
to
happen
for
quite
a
while
all.
D
Right,
maybe
the
real
issue
to
start
with
is
at
least
make
it
clear
what
the
groups
working
groups
charter
scope
if
the
they
think
it
is.
What's
the
mission
what's
the
scope
and
if
it's
just
a
copy
and
paste
for
the
readme
or
even
linked
to
the
readme,
I
think
that's.
That
would
be
the
most
important
part
right
now.
A
Well,
this
this
puts
me
in
an
uncomfortable
position
because
well
in
terms
of
the
security
software
repo
group,
in
the
sense
that
we've
been
aiming
to
adopt
the
charter
at
our
next
meeting
and
then
the
following
week,
bring
it
to
tac
for
a
blessing
so
that
we'd
be
we'd,
be
a
grown-up
big
boy
group,
that's
a
terrible
analogy.
Just
grown
up
and
insofar
as
you
know,
I
saw
the
trc
mechanisms.
I
figured
okay
that
that
looks
pretty
much
like
something
that
springs
to
life.
A
D
A
Yeah
really,
the
really
the
big,
the
big
thing
that
that
makes
the
whole
engine
turn
over
is
maintainers
who
who
are
defined
as
maintainers,
because
otherwise
you
know
the
boundaries
of
a
working
group
are
very
porous.
You
know
like
at
the
moment.
Shopify
is
not
even
a
member
of
the
open
ssf,
and
yet
here
I
am.
D
Okay,
I'm
sorry.
D
D
F
Sure
I
suspect
this
is
one
of
the
things
that
the
the
process
docs
working
group-
that
that
jury
pulled
together.
That's
meeting
tomorrow
at
11am
eastern
and
has
had
a
few
a
couple
of
working
meetings
we'll
get
around
to
is
kind
of
formalizing.
The
the
working
group
remember
the
working
groups
charter
for
the
working
groups,
kind
of
structure.
F
I
think
it's
been
very
ad
hoc
because
that's
how
very
much
how
open
ssf
has
has
started
and
grown,
and
we
didn't
want
to
come
in
and
say
you
know,
here's
the
one
true
path,
but
I
think
they're
working
on
the
one
true
path.
F
So
I
think,
for
now,
simpler
is
better
with
the
charter.
Saying
here's
the
it.
If
you
have
a
defined
membership,
that's
I
think
that's
better
than
than
not
than
a
defined
lead
for
the
working
group
is
better
than
not
and-
and
I
think
simpler
for
now
is
probably
better,
and
I
would
expect
the
charters
probably
will
get
kind
of
templated
and
standardized
across
the
working
groups
over
time
that
helpful.
E
Yeah,
it
makes
me
feel
like
we
should
just
wait.
I
I
was
going
to
suggest
you
do
exactly
that
and
I
know
it's
been
confusing,
because
the
attack
on
one
hand
said
hey
all
the
working
groups
need
to
get
a
charter
and
then,
if
you,
if
you
had
asked
okay,
what
does
it
take,
they
would
say:
oh
wait.
We
don't
really
know
yet
yeah
at
least
that's
what
they
should
have
said,
because
that's
ability-
and
this
is
what's
coming
up
here
now
it's
becoming
apparent
yeah
you're,
probably
I
agree
just
wait
a
bit.
I
I
And
by
the
way
I
I'm
involved
in
the
group
that
brian
was
just
talking
about
so
we
had
a
meeting
yesterday
and
we're
going
to
meet
again
tomorrow,
but
I
can
tell
you
based
on
what
you
know,
there's
very
good
discussion
going
on.
But
it's
not
we're
not
going
to
answer
tomorrow.
So
sure
it'll
take
a
bit
of
time.
E
Okay,
so
I'll
update
the
issue,
I
mean
we
have
the
notes
here
but
I'll.
Let
you
the
issue
that
we're
just
kind
of
waiting
but
yeah.
It
sounds
good.
Takeaways,
I
think,
are
that
it
sounds
like
we
need
to
agree
on
who
the
maintainers
are
or
the
the
leaders.
E
I
mean
we
have
the
the
co-chairs
and
then
who,
if
there
is
any
process
if
to
be
a
member
and
then
the
goal,
which
is,
I
think,
a
separate
discussion,
and
you
know,
but
we'll
just
essentially
be
referencing
our
goals
that
we
have
on
the
readme
and
then
that
can
be
updated
again
with
a
group
discussion.
B
E
All
right,
I
guess
we'll
move
on
so
yeah
I'll,
preface
the
the
next
bullet
with
a
little
bit
of
history
for
those
that
aren't
aren't
in
the
group.
E
E
Are
we
going
to
be
doing
a
similar
process
with
more
and
better
data
and
more
and
better
people,
not
better
people
but
more
people,
or
are
we
going
to
be
doing
some
other
kind
of
you
know
integrating
more
input
that
involves
reaching
out
outside
the
people
that
show
up
to
this
meeting
and
we've
had
a
lot
of
good
ideas
thrown
around
for
processes
or,
for
you
know,
ways
that
we
can
incorporate
that,
and
so
I
don't
know
that
we
have
any
kind
of
decisions
or-
or
you
know,
exact,
like
ideas
on
what
anything
concrete
or
anybody.
E
Nobody
has
raised
their
hand
on
saying
we're,
gonna
go
ahead
and
move
forward
with
this
procedure,
but
we
but
jacques
had
a
lot
of
good
things,
good,
good
input
and
david's
asking
jacques
for
an
update.
I
don't
know.
A
So
also
also
summarizing
history,
the
two
things
that
I
covered
in
my
presentation
back
in
honestly,
the
end
of
february
or
early
march.
It's
out
there
if
you
look
for
ranking
software
projects
in
these
minutes,
the
two
things
that
I
covered
was
voting
a
voting
system
or
a
direct
elicitation
of
probabilities,
and
my
sort
of
recommendation
you
could
say
at
the
time
was
to
focus
on
another
elicitation
over
over
voting
mechanisms
for
a
number
of
reasons.
A
A
A
I've
also
got
in
the
back
of
my
head
to
take
some
time
and
smash
a
prototype
just
to
get
something
to
do
anything.
It
doesn't
matter.
If
it's,
you
know
the
the
formally
proved
approach
to
things
that
somebody
wrote
in
a
paper,
it's
more
just
a
prototype
of
the
user
interface
of
what
it
would
look
like,
but
I
did
the
most
important
step,
the
most
important
step
of
any
open
source
project,
which
is
to
come
up
to
it
with
a
name
that
collides
with
a
bunch
of
other
open
source
projects.
A
D
Thought
that
was
a
requirement,
but
we'll
let
you
get
away
this
time.
A
D
All
right,
okay,
so
you
said
two
things:
one
was
voting
systems
versus
elicitation
of
probabilities
and
you
think
the
elicitation
of
probabilities
makes
more
sense,
given
the
literature
that
you
found.
What
was
the
second
part,
though,.
A
The
second,
the
second
part,
was
that
somehow
at
some
point
I
want
to
find
the
time
to
build
a
prototype
of
what
it
would
look
like
for
the
expert.
A
So
the
point
is
that
they're
meant
to
be
shown:
here's
a
project,
here's
some
information
about
the
project
that
you
have.
You
have
asked
to
be
shown
because
we
can.
We
can
show
dozens
of
data
points
between
scorecard
metrics
chaos
metrics,
you
know,
whatever
else
we
we
come
up
with
over
and
above
links
to
the
project's
own
home
pages,
source
repositories,
etc.
A
A
So
my
thinking
is
that
they
should
be
allowed
to
configure
which
things
they
look
at,
but
anyway,
the
idea
is
that
they'd
have
an
interface.
It
would
show
them
some
data
points.
They
would
then
say
I
think
it's
this
likely
to
go
bad
and
if
it
goes
bad,
it's
likely
to
be
this
bad.
A
Yeah,
that's
that's
an
open
question,
whether
there's
a
lot
of
problems
still
to
be
solved.
This
this
is
one
of
the
downsides
of
the
elicitation
approach
is
that
there
are
different
problems
from
the
voting
approach,
not
necessarily
better
or
worse.
So
one
of
the
big
problems
is
deciding
whether
a
probability
has
been
realized.
A
If
somebody
says
it
was
like,
there
are
scoring
systems
for
events
like
something
has
or
has
not
happened.
So
if,
for
example,
we
define
the
probability
that's
being
elicited
as
the
probability
of
a
cve
of
a
severity
x
within
this
this
next
five
years,
that's
relatively
concrete
and
measurable.
A
Much
harder
to
measure
is
the
impact
you
know.
What
is
your
estimate
of
dollars
of
impact,
because
anything
that
comes
up
as
the
the
validation
will
itself
be
an
estimate.
A
D
A
It's
not
everything
this.
This
is
why
we
want
expert
visitation
right
like
if,
if,
if
there
was
a
direct
line
between
something
like
you
know,
downloads
from
a
package
repository-
and
that
was
you
know,
correlated
0.95
with
with
worldwide
impact,
that
would
be
lovely.
But
we
we
know,
for
example,
from
the
classic
example
of
something
that
gets
downloaded
once
or
a
dozen
times
into.
A
ci
pipeline
then
shows
up
in
a
million
iot
devices
that,
unfortunately,
that
relationship
is
not
one-to-one.
A
A
There
is
a
possibility
in
the
longer
run,
that
those
expert
elicitations
can
form
the
feed
stock
for
a
machine
learning
model
that
can
suss
out
relationships
that
we
haven't
noticed
between
the
data
points
that
are
exposed
and
the
predictions,
and
that
will
hopefully
give
us
a
bootstrap
for
the
tens
of
thousands
of
projects.
That
might
not
get
an
opinion
given
about
them
by
experts.
A
A
Now
that
that's
actually,
incidentally,
why
I
was
thinking
about
the
impact
of
the
site,
could
you
gamify
it
because
there
are
ways
of
scoring
predictions
based
on
outcomes
like
once
an
outcome
is
realized.
You
can
then
back
calculate
the
score.
You
know
how
good
somebody
was
predicting
something,
but
that
would
still
require
us
to
say
the
impact
was
x.
J
Yeah
the
there
was
a
attempt.
I
don't
know
if
you
saw
this
in
the
literature
five
or
six
years
ago,
when
making
a
market
like
a
futures
market
for
security
advice,
but
they
tried
to
just
pay
people
for
getting
the
right
answer
and
if
I
recall
correctly,
it
failed
to
elicit
expert
opinions
about
what
they
thought
was
going
to
happen,
even
though
they
were
literally
proposing
to
pay
people
for
their
information.
A
Well,
that's
depressing
because
I
mean
that
that
was
one
of
the
things
actually
that
I
I
didn't
talk
about
at
the
presentation,
except
in
the
questions
where
sort
of
like
would
a
would
a
production
market
or
a
futures
market
work.
The
difficulty
with
futures
markets
is
that
prediction
markets
to
effectively
elicit
opinions
and
to
derive
a
probability.
A
You
need
enough
people
betting
enough
money
like
it
has
to
be
sufficiently
deep
liquidity,
otherwise
it's
very
lumpy
and
it
just
falls
back
to
being
individual
experts.
I
have
some
experience
with
this
because
I've
I've
previously
received
options
in
in
a
company
that
was
very
thinly
traded
and
you
know
it
was.
It
was
always
a
sort
of
an
exciting
day
when
things
bounced
around
or
when
it
took
several
days
to
sell
your
shares
and
so
on
and
the
same
problem
I
think,
would
arise
with
expert
elicitation.
A
D
I
I
I
do
think
that
there's
hope
for
identifying
lib
nebraska
in
the
sense
that,
with
some
data
support,
I
mean
you're,
absolutely
right
they
just
you
know.
If
you
talk
to
just
random
people
looking
at
their
direct
dependencies,
you
won't
notice
but
stuff,
like
the
analysis
done
by
harvard
and
some
of
these
other
things.
I
think
they
at
least
do
a
decent
job
of
helping
people
identify.
D
You
know
you
no
longer
have
to
get
people
to
figure
out
what
they
might
want
to
talk.
Think
about
and
analyze
here.
Look
at
this
now
you
can
use
your
human
judgment
now
that
you
don't
have
to
try
to
guess
the
world
at
random.
A
Yeah
and
that's
that's
the
thing
like
you
want.
One
of
the
things
I've
also
been
considering,
including
is,
is
whether
I
want
people
to
self
identify
their
level
of
knowledge
of
a
project
on
a
scale
like
they
could
say.
I've
never
heard
of
this
in
my
entire
life,
I'm
just
going
on
the
data
you
gave
me
up
to
I'm
one
of
the
creators
or
maintainers
of
this
project,
but
I
I
have
two
questions.
You
know
sort
of
things
with
that
is
one.
A
Does
it
add
that
much
value
compared
to
the
cost
of
another
thing
that
has
to
be
elicited
every
every
single
data
point
that
gets
elicited
adds
that
much
more
to
the
overhead
of
of
getting
elicitations,
because
this
is
going
to
be
tremendously
boring
and
head
head
thinking,
exhausting
work
where
you're
sort
of
staring
something
or
or
it's
going
to
be.
People
are
just
going
to
click
through
quickly.
Just
pick
things
at
random,
there's
no
way
around
it
with
with
elicitation
literature.
It
mostly
focuses
on
high
stakes.
A
Small
number
of
predictions
like
about
the
largest
I've
seen
is
100
variables
elicited
over
several
days,
whereas
you
know
with
a
couple
of
like
two
dozen
experts,
whereas
we're
sort
of
like
trying
to
talk
about
solicitations
on
tens
of
thousands,
ideally
from
thousands
of
experts
and
then
hopefully,
building
a
large
enough
data
set
that
we
can
start
to
come
up
with
guesstimates
of
the
things
that
have
not
been
estimated
for
which
there's
no
elicitation
available.
D
D
Can
anybody
suggest
somebody
we
could
contact
that
might
have
a
you
know,
know
of
a
different
way
to
turn
over
that
rock
and
the
academic
world.
So.
J
I
don't
have
an
easy
answer,
but
the
if
the
purpose
is
sort
of
labeled
data
elicitation,
which
is
analogous
to
other
machine
learning
problems.
J
D
A
Very
possibly,
that's
that's
why
I
was
interested
in
whether
it
was
gamifiable
whether
you
could
use
one
of
these
scorings
well.
A
You
know
setting
like
whether
you
would
out
of
the
box,
give
more
weight
to
someone
who
had
higher
awareness,
although
in
theory
such
an
expert
should
give
narrower
bands
because
they
have
high
confidence
in
their
prediction.
On
the
other
hand,
research
shows
that
experts
who
walking
off
the
street
are
hilariously
overconfident,
tend
to
be
calibrated
very
poorly
and
will
give
you
very
narrow
bands,
no
matter
what
you
do,
even
though
the
data,
if
they
refer
to
it,
show
them
that
it
should
be
much
wider.
A
I
sort
of
talked
about
this
in
the
presentation
where
there's
this
process
called
calibration,
where
you
give
a
series
of
questions
to
which
the
answer
will
be
presented
afterwards,
and
they
go
through
that
elicitation
process
for
this
series
of
questions
and
the
feedback
is
meant
to
show
them
that
they
are
being
hilariously
overconfident
to
widen
their
confidence
bands
to
pay
attention
to
whether
they're
being
too
pessimistic
or
too
optimistic,
and
that
that
does
show
a
measurable
improvement
in
performance.
A
The
thing
about
this,
though,
is
that
in
the
studies
I've
seen
a
depressing
amount
of
the
time.
What
lines
are
happening
is
that
out
of
a
group
of
experts,
one
or
two
experts
are
the
only
ones
to
listen
to,
and
everyone
else
gets
discarded,
which
you
know
seems
very
wasteful
to
me.
You
know
like
if
we
did
that
then
we'd
be
throwing
out
potentially
thousands
of
of
solicitation
results,
and
we
just
have
vast
gaps
in
the
data
you
know
like
I
would.
I
would
rather
have
bad.
A
Well,
I
mean
let
me
qualify
that
I
would
rather
have
an
opinion
that
could
then
form
the
anchor
for
further
investigation
than
to
have
no
opinion.
No
opinion
gives
me
no
signal
as
to
whether
I
should
care
whether
this
is
live
nebraska.
If
somebody
comes
in
off
the
street
gives
an
amazingly
high
score
that
shoots
it
to
the
top
that
attracts
attention
which
draws
more
experts,
more
experts
can,
then
you
know,
give
it
a
more
fulfilled
and
detailed
estimate
from
multiple
elicitations.
D
I
I
think
the
spot,
the
the
relative
paucity
of
data
is
a
significant
issue.
Maybe
we're
going
about
this
the
wrong
way.
Maybe
it
would
be
better
to
have
some
sort
of
automated
algorithm
to
do
an
estimate
and
then
use
humans
to
say
to
propose.
A
H
D
Yeah
I
mean
that
debian
does
have
popcorn,
although
I
don't
think
that's
enabled
by
a
lot
of
folks.
So
you
get
that
I
believe
that's
opt-in
data,
the
popularity
contest.
D
And,
of
course,
that
only
covers
you
know
system
packages
of
a
particular
distro.
I
have
used
this
as
a
data
source
for
other
things,
so
it
I
mean
it
actually.
It
helps.
I
mean
at
least
you
do
have
some
real
data
on
use,
but
you
don't
have
that
for
ecosystem.
Only
for
a
system.
A
H
H
So,
like
you
know,
debian
gradle,
maven
palm,
like
you
know,
maven
central,
you
know
all
those
artifact
servers
that
can
be
a
good
starting
point,
which
is
not
necessarily
always
public
data,
but
we,
you
know
you
ask
kindly
enough
and
say
you
know
this
is
the
purpose
and
you
can
get
that
as
a
starting
point.
Instead
of
data.
A
Yes,
which,
which
is
actually
a
really
wonderful
segue,
considering
we
have
12
minutes,
left
to
marina's
question
about
a
data
warehouse
for
for
repository
systems.
E
Yeah,
I
agree:
let's,
let's
move
on,
but
before
we
do
any
ideas
on
like.
What's
the
next
step
here
like?
What's
what
do
we
need
more
discussion
about,
and
you
know
potentially
doing
something
like
this.
A
I'm
still
you
know
like
doing
reading
based
on
you
know
where
I
can
squeeze
it
in
just
to
see
if
there's
sort
of
like
some
breakthrough
hidden
in
somebody's
book
or
more
to
the
point
like
it's
been
infuriatingly
difficult
for
somebody
to
just
give
me
a
formula
like
there's
a
lot
of
partial
formulas
floating
around,
and
I
am
not
a
data
scientist
to
start
with.
So
a
lot
of
these
things
are
based
in
bayesian
analysis
or
even
more
exotic
stuff.
A
So
that's
the
thing,
but
the
the
next
step
for
me
is
basically
going
to
be
in
so
far
as
I
can
find
time-
and
I
know
we're
all
busy
is
to
prototype
the
user
interface
for
experts
and
such
things
as
this
we
could.
We
could
even
elicit
opinions
without
combining
them
yet,
but
it
would
definitely
be
worth
like
having
something
in
front
of
people
like
there's,
there's,
no,
no
better
design
than
working
software.
D
Well,
I
I
have
to
admit
it
depends
on
how
complex
I've
had
more
little
success
of
here's,
your
prototype,
it's
a
piece
of
paper.
We
could
write
this
code,
but
you
answer:
let's
try
this
three
times
and
see
if
it's
actually
doable
because
no
point
in
writing
the
code.
If
you
couldn't
get
somebody
to
do
it
even
after
you
wrote
it.
E
Sounds
great
yeah,
let's,
let's
go
ahead
and
move
on
to
marina
floor
is
yours.
G
All
right
hello,
so
I
think
I
mentioned
a
little
bit.
The
idea
here
is
just
a
proposal
for
having
a
data
warehouse
for
software
repository
data.
I
think
the
overall
deal
is
fairly
straightforward.
Kind
of
the
reason
I'm
here
right
is
that
I
think
one
of
the
key
benefits
of
this
is
some
of
the
stuff
that
you
were
talking
about.
G
Just
now
is
identifying
data
about
how
projects
are
used,
the
projects
that
are
stored
on
these
different
repositories,
I
think,
there's
also
some
other
benefits,
specifically
kind
of
how
I
came
to
this
project,
had
a
lot
to
do
with
being
able
to
test
performance
of
security
solutions
on
real
data
to
make
sure
that
they
actually
would
work
at
the
scale
of
these
different
repositories
and
then,
of
course,
the
the
challenging
part
here
is
figuring
out
the
engineering
of
the
system.
G
You
know
where
stuff
is
going
to
be
stored,
who
wants
to
run
this
whole
thing
and
what
data
is
available
today
versus
and
that
can
just
be
like
uploaded
to
the
system
versus
what
data
we
would
have
to
work
with
repository
maintainers
to
obtain
some
of
the
larger
software
repositories.
Folks,
like
specifically
rubygems
and
pipi,
already
have
a
fair
amount
of
this
data
available
publicly,
but
it's
just
all
in
different
formats
in
different
places,
so
this
would
help
centralize
that
and
then
for
other
folks.
D
I'm
trying
to
carry
out
to
have
a
single
db
to
query
so
I
I
tried
to
add
notes
to
the
for
this
meeting
to
because
I've
actually
looked
into
this
myself.
So
there
is
an
existing
system,
it's
called
libraries,
dot,
io
and
tide
lift
is,
I
would
say,
well
maintain
I
guess
from
a
technical
sense.
It
lives
at
a
tide.
Lift
I
don't
want
to
I'm
actually
not
trying
to
cast
shade
on
tide.
D
Lift
I've
contributed
patches
to
libraries.io
in
the
past
back
when
is
it
ida?
I
used
it
harvard
used
it
in
their
analysis
to
use
dependency
analysis,
so
there
so
clearly
there's
some
value
to
it,
because
there's
something
that's
there,
but
this
is
my
opinion.
D
Maybe
it's
changed
recently,
but
at
least
when
I've
interacted
with
it
with
harvard's
interactive,
it's
not
really
well
maintained,
and
it
actually
makes
sense
because
I
think
tide
lift
originally
was
thinking
that
this
is
going
to
be
critical
to
their
business,
and
I
don't
think
it
really
is
again.
D
I'm
not
I'm
not
somebody
from
if
somebody
here
was
here
from
tide,
lift
I
I
could
get
the
store
if
we
could
get
the
store
straight
from
the
horse's
mouth,
so
I
I'm
not
going
to
try
to
speak
for
them,
but
I,
I
suspect
so
how's
this.
I
suspect
they
thought
this
was
going
to
be
central
and
it's
turned
out
not
to
be,
and
if
that's
not
true,
then
I'd
love
to
hear
what
the
story
is.
So
I
think
it
would
be
valuable
to
have
something
like
this.
D
So
I
think
that
this
is
this
does
have
legs.
What
might
be
helpful
would
be
talking
with
the
title
of
folks,
because
I
don't
know
what
they
found,
what
they
feel
about
libraries.io,
at
least
historically,
it
was
open
source,
as
far
as
I
know
it
still
is,
but
unless
they've
had
a
recent
change
of
heart,
I
don't
think
that
it's
likely
to
go
anywhere,
so
they
might
be
willing
to.
D
You
know,
transition
some
of
that
stuff.
Well,
we
could
use
their
code,
but
you
know,
I
think,
having
conversations
so
that
it's
all
in
one
place
would
be
really
helpful.
G
Yeah,
it's
a
great
idea,
and
I
do
think
that
the
scope,
at
least
of
the
full
proposal
is
maybe
a
little
bit
broader.
But
I
think
the
stuff
that
they've
done
is
a
great
starting
point.
Just
getting
aggregate
data
about
these
different
packages
and
these
different
things.
G
D
It's
the
dependency
network,
a
depends
on
b
depends
on
c
depends
on
d,
so
that's
been
their
focus
of
getting
that
kind
of
data,
because
you're
you're
right,
you
can
get
that
data,
for
example
from
ruby,
gems
and
so
on.
But
now
you
have
to
process
everybody's
different
formats,
so
loading
it
all
in
one
place,
first
of
all
beating
on
those
poor
repositories.
Only
once
because
I
hear
all
the
repos
of
other
things
to
do
and
then
having
a
common
format
is
really
helpful.
A
I
I
had
a
question
about
that.
Actually,
maybe
this
is
diving
into
the
weeds,
but
is
is
you're
thinking
that
you
would
have
a
processing
pipeline
on
the
left
side
that
goes
out
to
whatever
is
available
and
then
turns
it
into
into
shape.
Or
did
you
have
in
mind
that
you
have
an
api?
That
repositories
would
then
report
to.
G
Yeah,
I
think,
there's
kind
of
a
short
term
and
a
long-term
answer
right.
The
short-term
answer
is
just
collect
the
data
and
in
whatever
form
it
is
and
and
put
it
somewhere.
I
think
in
the
longer
term
it
would
be
nice.
G
I
think
that
to
do
the
processing
on
the
linux
foundation
side,
so
I
think
we,
the
goal
here,
is
not
to
give
more
work
to
repository
maintainers
and
I
think
they
have
plenty
of
other
things
to
do,
but
if
they
can
just
give
whatever
format
of
data
they
have,
and
we
can
make
an
automated
processing
step,
because
I
think
most
of
the
data
is
somewhat
similar.
It's
in
some
kind
of
sql-like
or
java
json-like
format
that
we
can,
then
you
know,
put
into
a
standard
format.
H
G
Yeah
this
is
download
accounts
and
and
and
more
basically,
so
it's
all
information
about.
I
think
the
the
most
full
data
set
currently
available.
I
think
rubygems
has
the
download
logs
for
complete
download
logs
with
anonymized
ips
for
all
of
the
different
data
that
that,
basically,
the
request
that
they
get,
they
then
make
public
in
an
anonymized
way.
G
Pipevi
has
something
similar
with
just
no
ip
address
information,
so
it's
you
know
like
it's
not
just
download
counts
but
download
frequencies
like
where
the
downloads
are
coming
from
repeat
people,
that
kind
of
stuff
which
might
be
interesting
for
various
types
of
analysis.
The.
H
Other
one
to
look
at
is
github
has
their
dependency
graph
data.
That
is,
if
you
go
to
like
dependency
insights
on
any
repository.
That's
using
like
this.
It
doesn't
work
with
gradle,
but
it
works
like
maven
stuff
like
that
and
that'll
that
that
information
is
also,
I
don't
know.
If
it's
bulk
available,
you
can
talk.
People
like
github
to
get
that
information
potentially.
D
So
I
need
not
just
the
dependencies
for
a
package
but
the
dependencies
dependencies
of
all
packages,
and
you
can
get
that
with
varying
levels
of
effort
and
sometimes
it's
a
whole
lot
of
effort.
E
Speak
yeah,
so
facilitator
question,
you
know
what
would
you
be
looking
to
get
out
of
the
open
ssf
or
this
working
group?
You
know
resources,
discussion,
just
a
home.
G
Yeah,
I
think,
I
think,
a
combination
of
things
I
think
either
resources.
I
think
resources
would
definitely
be
helpful.
I
think
it
might
make
sense
for
the
project
to
like
officially
live
in
the
software
repositories
working
group,
because
just
because
the
maintainers
of
these
repositories
are
there,
and
so
that
way
you
can
use
that
communication.
But
I
do
I
just
like
some
clients,
just
kind
of
collaboration,
maybe
resources,
and
also
just
you
know
letting
you
know
this
data
might
be
available.
E
E
You
know,
vote
or
you
know,
consensus,
we'll
just
put
it
on
the
agenda
for
the
full
two
weeks
and
then
doing
a
like,
not
a
vote,
but
you
know
a
discussion
in
in
the
next
meeting
or
whichever
meeting
you'd
like
that's,
not
right
at
the
end.
E
Does
that
sound
good
to
everyone?
One.
H
D
Yeah
marina,
can
you
shoot
me
a
quick
email,
because
maybe
I
can
introduce
you
some
other
folks
I,
but
I
can
at
least
confirm
that
harvard
used
this
kind
of
data.
We
use
this
kind
of
data,
so
there
are
users
of
this
kind
of
data
which
I
think
was
at
least
at
least
your
initial
question.
D
Okay-
and
maybe
I
can
connect
you
some
I'll
off
research
and
harvard
folks
to
continue
that
conversation.
E
Thanks,
marina
thanks
everyone
else
for
joining
great
to
see
you
all
and
have
a
good
two
weeks
and
see
you
see
you
next
time.