►
From YouTube: Security Tooling WG Meeting (July 13, 2021)
Description
No description was provided for this meeting.
If this is YOUR meeting, an easy way to fix this is to add a description to your video, wherever mtngs.io found it (probably YouTube).
A
And
there
the
recording
goes
so,
does
anybody
have
any
opens?
They
would
like
to
bring.
A
B
C
Honestly,
it's
a
pain
in
my
backside
and
it's
just
really
irritating.
We
have
just
gone
through
a
funding
round
of
github
to
get
a
whole
load
of
additional
investment
and
everything
we're
doing
on
the
supply
chain
security
side.
We
were
plus
50
heads,
so
we
had
plans
we're
hiring
for
those
plans
with
bottlenecks,
I'm
hiring
and
then
because
of
the
executive
order
I've
suddenly
got
like
about.
You
know
dozens
and
dozens
of
microsoft.
People
that
are
suddenly
like
this
is
a
really
exciting
sales
opportunity.
C
C
So
there
is
no
upside
to
all
of
your
energy
and
that's
a
very
frustrating
experience
for
them
and
a
very
frustrating
or
time
consuming
experience
for
me
constantly
telling
people
that,
just
because
there
is
an
exec
order
does
not
mean
there
is
suddenly
loads,
more
product
truth
and
the
solution
is
not
suddenly
fixed.
So
it's
frustrating
I'd
love
to
hear
other
people's
experiences.
C
There
there's
definitely
going
to
be
a
lot
of
anxiety
and
anxiety,
driven
sales
feels,
like
you
know,
it's
it's
certainly
going
up,
but
when
we
come
to
talk
about
whether
we're
actually
making
a
difference
to
the
security
of
the
world
software,
I'm
not
seeing
much
goodness
there
just
yet.
A
Yeah,
I
I
for
better
or
worse
agree.
A
I
mean
honestly,
I
think
people
are
just
running
around
right
now,
just
going.
We
have
to
go
be
compliant
with
this,
but
we
have
no
idea
what
it
is
and
because
of
that,
they're
trying
to
all
do
things
running
in
different
directions
and
they
don't
know
what's
going
on.
A
So
fun
george.
D
You
hear
me
sorry
about
right
now,
so
sorry
to
everybody
there.
I
think
there
was
a
misunderstanding
on
when
we
would
present.
We
were
ready
to
go
last
week
two
weeks
ago,
but
I
have
my
slides
today.
I
see
daniel
silverstone
has
actually
seen
a
version
of
this
deck
in
another
group,
so
apologies
for
the
the
do-over
to
him,
but
I
think
for
everybody
else,
it'll
be
new.
So
let
me
just
share
my
screen.
D
D
No
worries,
okay,.
D
And
you,
let
me
know
if
you
can
see
the
powerpoint.
A
D
Of
course-
and
I
I'm
consent
to
that-
that's
fine!
So
just
don't.
A
D
So
good
afternoon,
everyone
today,
I'm
going
to
summarize
the
secure
code,
reuse
survey
that
iqt
labs
recently
conducted.
Today's
talk
has
three
components:
I'll
start
with
a
quick
background
on
our
approach,
then
give
you
a
demographic
baseline.
You
know
the
highly
educated
python
reliant
cohort
that
took
the
survey.
D
Then
I'm
gonna
highlight
some
key
results
that
my
colleague,
john
speed,
asked
me
to.
You
know,
hit
my
prepared
remarks
and
then
we'll
take
questions.
So
just
a
brief
item
on
our
motivation
methodology.
You
know
the.
Why
and
the
how
of
this
survey.
D
We
think
ken
thompson's
1984
reflections
on
trusting
trust
is
a
really
good
starting
point
for
research
on
code
reuse.
Today
you
know,
unfortunately,
in
the
years
since
you
know
there
hasn't
been
a
lot
of
sort
of
digital
ethnography
research.
You
know
there
in
this
area.
D
There
are
annual
developer
surveys
that
ask
salary
questions,
questions
about
you
know
what
time
do
you
usually
start
working
as
a
developer,
but
they
don't
really
dig
into
security,
and
so
inkytel
labs
thought
there
was
an
opportunity
here
to
start
to
ask
questions
about
to
what
extent
developers
trust
the
the
packages
they
install
and
you
know,
rather
than
taking
the
trust
as
a
given.
Looking
at
you
know
what
contexts
that
trust
manifests
in.
So
we
developed
an
18
item
online
questionnaire.
D
We
had
early
review
by
some
academic
partners-
frank
naugle
at
harvard
business
school
was,
you
know,
extremely
helpful.
I
know
he
participates
in
this
forum
and
ultimately,
we
collected
a
convenience
sample
of
150
data
scientists,
software
developers,
web
developers,
we
recruited
them
through
personal
and
professional
networks
and
I'll
talk
about
some
of
the
limitations
there.
In
just
a
moment
and
crucially,
we
wrote
the
questions
on
this
survey
to
enable
cross-comparison
with
the
redmonk
tyobi
stack
overflow
data.
D
You
know,
particularly
with
programming,
language,
usage,
etc,
and
then
I
just
want
to
thank
this
group
here.
Actually,
david
wheeler,
you
know
was
one
of
the
folks
who
helped
us
disseminate
the
survey.
D
So
what
did
we
find?
Well
just
to
start
everybody
off.
As
I
mentioned
this,
this
was
a
very
python
heavy
population.
D
D
Let
me
before
I
move
on,
we
also
asked
them
which
package
managers
they
use,
and
you
know
pi,
pai
and
pip-
was
also
by
far
the
most
popular
followed
by
npm,
maven,
nougat
and
and
others
like
that.
D
D
The
second
part
was
coding
overall,
and
so
you
know
the
the
salmon
orange
here
is
that
total
inclusive
of
school
and
this
data
was
consistent
with
the
stack
overflow
results.
D
D
I
want
to
spend
just
a
minute
on
the
education
level
results
95
of
our
sample
in
the
you
know,
salmon
orange
here
had
a
bachelor's
degree
or
higher.
If
you
look
at
the
stack
overflow
equivalent,
their
figure
was
75
percent,
and
so
I
think
part
of
the
difference
that
we
see
here
you
know
has
to
do
with
how
we
recruited
our
survey
participants,
you
know,
iq2
labs
is
extremely
privileged
to
you,
know,
know
people
with
high
levels
of
educational
attainment.
D
That
is
not
necessarily
representative
of
you
know
the
world
and
the
developer
world
writ
large.
So
if
we
were
going
to
do
another
iteration
of
this
survey,
we
would
probably
try
to
target
survey
takers
who
who
learn
to
code.
You
know
in
non-bachelors
non-higher
education
settings,
but
this
this
does
give
us
confidence.
You
know
that
the
survey
takers
were
reading
the
questions,
we're
we're
thinking
about
their
data
science
and
software
development
processes.
D
This
is
the
final
demographic
baseline
item
and
then
I'll
get
into
what
these
people
actually
reported.
You'll
see
the
majority
of
our
our
survey
takers,
this
75
percent.
You
know
again
in
the
orange
salmon,
not
the
blue,
worked
in
an
organization
with
more
than
50
full-time
employees.
D
D
So
if
you,
if
you
read
this
slide
and
the
previous
slide
together,
you
know
the
the
takeaway
is
inkytel
labs
was
able
to
reach
high
educational
attainment,
software
developers
and
data
scientists
working
at
reasonably
large,
presumably
resourced
organizations,
and
so
one
of
the
first
you
know
key
results
that
we
obtained
had
to
do
with
support
resources
that
these
organizations
are
providing.
D
You'll.
Note
that
you
know
23
of
our
survey
takers
out
of
139
who
answered
this
question,
had
no
organization
organizational
support
resources,
no
training,
no
policies
governing
package
installs,
that's
obviously
concerning
we're
in
the
process
of
analyzing
this,
and
we
have
a
blog
post
coming
out
that
that
talks
about
survey
takers,
who
had
two
or
more
of
these
organizational
support
resources-
and
you
know,
you'll-
find
that
very
few
of
our
survey
takers
had
two
or
more
of
these,
and
so
we
read
this
result.
D
D
Then,
when
we
asked
what
criteria
survey
takers
use
to
determine
if
a
software
package
is
safe
to
install
you'll
note
that
the
two
most
popular
criteria
are
are
fairly
superficial
ones,
package
seems
popular
and
trusted
and
in
the
original
phrasing
of
this
question
that
had
to
do
with
github
stars
maintenance
seems
recent.
We
defined,
as
you
know,
any
update
in
the
past
six
months.
D
Those
you
know,
as
this
group
well
knows,
are
not
guarantors
of
a
package
being
safe
to
install
security.
Advisory
cves
was
important
to
only
about
one
one-third
of
our
population.
You
know
test
coverage.
Badges
is
another
one
that
you
know
it
was
potentially
informative,
potentially
misleading,
so
I
think
there's
more
work
that
needs
to
be
done.
D
Crucially,
you
know
about
half
the
survey
taking
population
said
that
they
wanted
to
see
the
source
code
wanted
the
source
code
available
for
direct
inspection.
That
gives
us
some
comfort,
but
you
know
to
what
extent
survey
takers
and
and
developers
out
in
the
wild
actually
do
that.
I'm
I
I
have
my
doubts.
D
We
asked:
how
often
do
you
use
these
information
sources
to
evaluate
packages
and
we
used
a
sort
of
five-point
likert
scale
going
from
never
to
always
and
you'll
note
that
package
scans
there
at
the
bottom
has
the
greatest
red
area
and
that's
showing
that
you
know
the
majority
of
our
survey
takers
who
answered
this
particular
question:
do
not
run
package
scans.
You
know
and
are
willing
to
admit
that
do
not
look
at
source
code,
and
you
know
you
see
this
green
area
here.
D
The
vast
majority
of
survey
takers
do
look
at
the
read
me,
but
that's
that's
cold
comfort
for
those
who
worry
about
security
and
and
sort
of
package,
vetting
and
package
due
diligence.
So
two
more
likert
type
responses.
In
this
case.
The
scale
goes
from
strongly
disagree
to
strongly
agree.
We
asked
some
questions
on.
You
know
asking
the
respondents
to
agree
or
disagree
with
I
statements,
so
in
the
first
instance,
I
trust
the
code
I
reuse,
you'll
see
the
majority
of
survey
takers
unequivocally
agree.
D
I
believe
package
registries
are
responsible
for
keeping
code
safe.
That's
the
second
bar
here
also
very
high
levels
of
agreement,
we're
also
digging
into
a
related
question
that
asked
whether
respondents
view
this
as
a
shared
responsibility,
I.e.
I
believe
developers
are
responsible
and
I
believe
package
registries
are
responsible
because
there
seems
to
be.
You
know,
sort
of
diffusion
of
responsibility
here
and
then
again
the
two
more
eye-opening
results,
a
fair
number
of
people
willing
to
admit.
I
do
not
engage
in
pre-install
code
vetting.
D
I
think
we
need
to
be
more
like
these
11
here
who
strongly
agree
and
are
committed
to
doing
it
final
set
of
results,
and
again
this
should
tell
a
fairly
consistent
story.
I
wish
I
knew
more
about
the
vulnerabilities
associated
with
reuse.
These
are
people
with
phds.
These
are
people
who
work
at
large
organizations.
They
wish
they
knew
more.
D
D
I
think
that's
something
we
can
evaluate
in
a
in
a
future
survey
or
follow-up,
and
then
I'll
just
talk
about
this
final
item.
I'm
comfortable
evaluating
a
new
library
for
security
risks.
Plainly
more
than
one-third
of
our
survey
takers
are
not
comfortable
evaluating
libraries
and
yet
they're
using
them.
So
sorry,
if
I'm
repeating
myself,
but
I
think
there's
more
work
needs
to
be
done
here
to
educate
and
to
potentially
look
at
the
usability
of
package
install
tools.
D
E
I
want
to
ask
lots
and
lots
of
questions,
so
thank
you
so
much
for
sharing
this.
The
number
of
folks
with
degrees
is,
I
think,
far
more
skewed
than
the
general
population.
I
think
you
me
mentioned
that
earlier.
Have
you
tried
to
pull
out
just
the
folks
without
degrees
to
see
if
they
were
substantially
different,
because
I
I
don't
think
your
your
story
is
going
to
be
any
better
if
you
got
more
people
without
degrees,
taking
yourself.
D
That's
the
sort
of
interesting
results
that
sort
of
educational
attainment
as
a
as
a
proxy
for
sophistication,
and
it's
not
the
only
way
you
can
be
a
sophisticated
developer
data
scientist
sure
it
doesn't
seem
to
have
an
impact.
D
I
mean
I
I
I'd
be
low
to
generalize
from
the
you
know,
ninth
grade
or
the
to
twelfth
population,
but
I
don't
think
we'll
find
a
difference
and
you
know
I
don't
think
we
have
a
large
enough
data
set
to
say
anything
about
the
statistical
power
of
that
difference,
but
that
could
certainly
be
a
follow-up.
I.
E
I
I
I
don't
actually
want
to
imply
that
I
have
no
intention
of
implying
that
not
having
a
degree
means
somehow
you're
stupid
or
anything.
That's
not
what
I'm
saying
it's
that
somebody
could
say:
hey
your
sample!
Isn't
representative!
D
E
E
Don't
think
we've
got
a
large
enough
sample
to
at
least
get
an
estimate
that
it's
similar
in
nature.
D
D
Similar
in
nature,
but
what
I'm
saying
is
it
would
be
interesting
if
we
could
find
some
actual
difference.
You
know
organizational
size
does
seem
to
matter.
In
this
context,
people
who
reported
more
resources
in
place
tended
to
be
at
larger
organizations.
I
mean
not
a
surprise,
but
I
don't
think
any
any
grand
policy
flows
from
that.
We
should
all
work
at
large
companies.
D
To
david's
question,
you
know
what
one
other
area
that
we
are
going
to
be
talking
about
in
this
forthcoming
blog
post,
that's
coming
out
in
a
week
or
two
that
stood
out
to
us
was
years
of
coding
experience
and
for
a
couple
of
these
questions.
D
If
I
can
generalize
people
who've
been
coding
professionally
for
over
a
decade
report
sort
of
greater
skepticism
when
they
encounter
a
new
package
for
the
first
time,
so
the
sort
of
grizzled
veteran
effect
does
show
up
in
the
data,
and
that
has
nothing
to
do
with
education
or
organization
size.
I
I
can
explain
burned
at
least
once
that's
bitten
twice,.
E
C
Is
there
any
kind
of
discontinuity
there
I'm
just
curious
about
whether
or
not
that's
folks
that
started
their
careers
when
package
management
wasn't
really
a
thing
or
not.
If
you
look
at
like
the
length
of
time
that
communities
like
npm
have
actually
existed,
it's
just
shy
of
a
decade,
so
I
would
be
curious
to
see
if
there's
a
discontinuity
in
your
data
there
or
whether
it
is
you're
becoming
more
grizzled
year
by
year.
D
I
love
that
yes,
so
sort
of
controlling
for
survivorship
bias
on
package
manager,
existence,
yeah.
C
It's
one
of
the
big
things
that
we
have.
We
did
a
study
that
looked
at
how
long
vulnerabilities
existed
before
they
were
discovered
and
we
found
that
they
existed
for
a
longer
time
in
communities
that,
frankly,
just
had
package
managers
for
longer,
and
that
was
the
main
determining
thing.
So
you
could
look
at
it
and
say
like
oh,
you
know,
npm
is
doing
better
than
movie
gems
or
something
like
that.
But
in
fact
it
was
just
the
movie
gems
had
existed
for
like
three
years
longer,
and
that
was
the
only
thing
affecting
the
average.
D
Good
point-
and
this
is
something
john
speed
and
I
talk
about
a
fair
amount.
You
know
I
we
think
if
we
can
get
a
little
prescriptive
for
just
a
moment.
There
are
lessons
from
the
older
provenance
package,
managers
and
language-based
ecosystems
for
the
newcomers
and
rather
than
sort
of
devolving
into
the
language
wars
that
are,
you
know,
really
productive.
D
E
Just
ffyi
one
of
the
things
that
I
have
proposed
as
some
work
to
be
done,
and
not
particularly
for
this
group,
but
just
something
that's
that
should
be
done-
is
to
create
some
sort
of
spec.
Basically,
for
what
would
be
the
ideal
package
manager
from
a
security
perspective,
what
were
all
the
things
that
would
do?
E
E
You
know,
if
you
develop
pip
you're,
probably
not
talking
very
much
to
mpm,
or
vice
versa,
or
the
linux
distro
package
managers
they're
just
each
of
them
recreates
some
of
the
same
ideas.
It
seems.
E
E
My
theory
was
a
spec
would
be
the
kind
of
the
obvious
way
to
do
that,
but
some
way
to
break
down
the
gardens
so
that
the
good
ideas
of
each
can
be
shared
because
I'm
sure
that
and
some
of
them-
and
I
think
there
are
some
good
ideas
that
none
of
them
do
today
but
could
be
done.
If
only
there
was
a
way
to
kind
of
sit
down
and
figure
out.
What
would
be
the
good
things
that
could
be
then.
D
I
agree-
and
I
I
didn't
have
a
slide
on
this,
but
you
know
this
is
definitely
something
to
follow
up
on.
We
did
ask
some
questions
about
usability
and
you
know:
do
you
prefer
to
use
command
line
interface
versus
gui
again
in
keeping
with
this
experience?
D
Finding
there
does
seem
to
be
a
generational
difference
that
shouldn't
surprise
anyone
people
have
been
developing
longer
are
committed
to
their
to
their
bash,
but
you
know,
I
think,
there's
some
lessons
there
also
for
how
the
package
registries
deliver
blobs
of
code-
and
you
know
how
information
surfaces
in
that
context
and
different
package
registries
have
tried
different
approaches
where
your
your
terminal
squawks
back
at
you
and
throws
a
security
warning.
D
A
Cool
I'm
curious.
F
A
You
talked
a
little
bit
about
you
know
different
ecosystems.
Did
you
see
significant
differences
between
you
know
you?
You
have
a
lot
of
folks
who
are
doing
python
development.
What
about
you
know
the
results
that
you
saw
with
them
versus
folks,
whose
primary
language
in
daily
languages
is
c,
for
example,
much
more
low.
D
Level
there
didn't
seem
to
be
much
difference,
one
other
area
where
we
did
find
difference
that
nothing
to
do
with
languages
or
ecosystems.
D
D
We're
we
just
spoke
with
our
marketing
team
this
morning.
The
target
is
the
29th
of
this
month
so
and
we'll
definitely
email
this
group
and
david.
If
we
can
ask
for
another
retweet
you
you
have
a
following,
I
I
I
know
how
to.
D
Excellent
and
just
the
final
item,
if
I
may
we're
going
to
make
the
data
and
the
questionnaire
itself
available,
because
we
know
there
are
other
groups
interested
in
running
their
own
surveys
and
digging
into
this
kind
of
stuff.
That
is
something
john
speed
and
I
are
definitely
happy
to
follow
up
on
and
you
know,
gray
sounds
like
your
group,
you
know,
has
done
similar
research.
C
Yes,
we've
not
done
much
questionnaire
based
stuff,
and
it's
not
my
team.
It's
under
a
woman
called
nicole
forsgren,
who
leads
our
kind
of
research
arm,
but
yeah
I'll
certainly
circulate
it
within
github
and
make
sure
that
folks
have
eyes
on
it
here
appreciate
it.
A
And
greg,
can
you
share
the
the
research
that
you
were
talking
about.
C
Yes,
the
security
octoverse
report,
I'll
dig
it
out
and
share
it.
Thank
you
kindly.
E
E
D
Yeah
and
you
know,
maybe
we
can
have
a
generation-
that's
been
coding
for
over
a
decade.
You
know
who
doesn't
have
to
learn
the
hard
way.
I
don't
know,
maybe
that
that's
too
optimistic.
E
E
Defaults
well,
sensible,
defaults,
but
also
more
information.
I
I
I
you.
You
said
earlier:
hey
there.
What
was
the
word
you
used
naive
or
something
about
hey?
Is
it
popular?
I
don't
actually
think
that's
terribly
naive.
In
fact
it's
I
mean.
If
you
look
at
how
people
buy
cars,
you
know
spend
thousands
of
dollars.
They
definitely
look
at
things
like
you
know.
Does
anybody
else
seem
to
like
this
car?
It's
not
a
crazy
way
to
to
make
a
decision,
but
I
think
the
problem
is
their
star
for
information.
D
I
agree
it's
it's
efficient,
but
perhaps
a
contextual.
E
G
A
A
David,
I
know
you
were
you
came
in
late,
I
didn't
know
if
you
had
any
opens.
E
I
I
didn't
I
I
in
fact
I
I
ran
away
and
took
a
vacation
for
a
week,
so
I'm
trying
to
get
back
in
the
saddle
here.
Let's
see
here,
I
did
mention
that
this
group
last
time,
flawfinder
now
outputs
serif
work
with
some
other
folks
to
try
to
get
a
an
easy
to
add,
get
a
marketplace
plug-in
for
that.
I
think
we're
close.
E
A
A
E
It
was
a
good
vacation.
What
am
I
doing
back
here.
A
Excellent
matt,
I
wanted
to
ask
you
if
you
could
expand
on
on
some
of
the
the
things
that
you
brought
up
a
couple
meetings
ago
regarding
the
secure
criteria
from
the
securing
critical
projects
working
group
and
how
you'd
like
to
see
that
adopted
here.
H
That
feedback
that
we
actually
start
creating
lists
using
taking
the
products
that
we've
evaluated
and
been
presented
here
are
not
even
about
we
haven't
even
evaluated
they've,
been
presented
and
then
taking
the
securing
critical
project's
advice
on
ins
and
on
how
to
evaluate
them
and
start
putting
those
scores
up
and
looking
at
you
know,
seeing
if
we
can
recommend
those
things
in
terms
of
what
tools
you
can
bring
to
bear
to
solve
problems
and
I'd
love
to
see
which
ones
are
especially
out
in
the
not
under
some
strong
governance,
a
foundation
or
otherwise
be
brought
under
the
ossf
like
flat,
finder,
bring
it
under
the
ossf
and,
let's,
let's,
let's,
let's
use
it
as
a
model
product.
H
Let's,
let's
create
a
pr
governance
process.
Let's,
let's
create,
let's
sign
it,
let's
put
it.
Let's
put
s-bombs
in
place,
let's
show
people
how
they
should
do
it
and
work
with
the
experience
projects
group
and
you
know
practice
what
we
preach
create
the
exemplar.
You
know
that's
what
I'd
love
to
see.
H
So
I've
been
going
through,
I've
been
I've,
been
taking
my
spare
time,
because
I
don't
I
don't
work
in
the
cso
office.
I
don't
work
as
here,
I'm
just
an
open
source
guy.
I've
been
asked
to
look
at
this
stuff
right
and
it
seems
so.
I've
been
pouring
through
every
all
the
all
the
tools
are
in
the
oasplus
and
finding
half
of
them
are
are
not
really
open.
H
A
H
Oh
yeah,
I'd
love
to.
I
gave
up
on
the
on
the
open
os
list
about
two
thirds
three,
I'm
working
through
a
couple
of
other
lists,
but
I
I
will
say
that
I'm
hoping
I
get
to
move
into
into
the
cso
office
and
start
doing
this
more
full-time,
but
yeah.
E
I'm
sorry
to
say
I
do
have
also
a
list
of
open
source
static
analysis
tools,
though
I'm
sure
that
many
of
my
links
are
are
are
dead
for
the
same
reasons.
H
One
thing
that
someone
could
really
help
me
with
because
that
you
know
it
seems
like
the
center
of
gravity
for
solving
these
problems
is,
is,
is
the
s
bomb
and
new
guidance
was
released
on
just
the
12th
just
three
just
yesterday
actually-
and
I
read
through
that
last
night-
so
I
mean
I'd
love
to
figure
out
I'd
love
to
figure.
I'm
not
that's.
H
What
I'm
saying
more
of
my
time
on
recently
is
like
trying
to
analyze
formats
and
figure
out
what
use
cases
they
solve
or
intend
to
solve,
and
what
can
be
done
with
them.
That's
really
where
I'm
spending
the
bulk
of
my
time.
H
Well,
it's
that's
what
I'm
trying
to
figure
out
I'm
trying
to
figure
out
what
the
intent
of
these
groups
are
so
cycling.
Cyclone
dx
is
what
I'm
looking
starting
to
look
at.
Only
last
few
days
like
in
detail,
but
they've
got
a
really
good
set
of
use.
Cases
thought
out
that
are
they're
based
upon
a
different
attack,
vectors
and
and
seeing
the
value
of
the
s
bomb
go
well
beyond.
H
So,
if
you
it
goes
back
to
governance,
it
goes
back
to
trusting
the
package
managers.
What
governance
do
they
have
in
place?
So
there
are
a
few
choke
points.
You
know
if
you
go
to
pi
pi
and
you
go
to
npm
and
you
go
to
maven
and
you
can
you
basically
analyze
their
governance
practices
and
and
speak
to
you
know
a
smaller
audience.
H
You
don't
have
to
do
this
in
math,
you
don't
have
to
educate
in
mass,
that's
there's
economies
of
scale
that
can
be
gained
by
going
after
a
few
people
at
these
package
manager
places
and
getting
them
to
create
governance,
practices
for
and
and
adding
some
of
our
the
tools
we
recommend
into
their
processes,
and
I'm
really
encouraged
by
seeing
you
know
the
what
you
know:
the
the
inclusion
and
of
github
people
here
and
and
that
they're
stacking
up,
because
you
know,
I
think,
creating
opinionated
workflows.
H
That's
where
I
said
this
on
in
this
meeting.
Maybe
another
meeting
is
is
that's
why
we
need
to
step
in.
You
know,
create
some
opinionated
workflows.
They
don't
have
to
be.
You
know
they
don't
have
to
be
guarantees,
or
you
know
disclaimers.
You
know
they
put
aside,
but
at
least
they
give
people
a
guidance
and
and
teach
them
how
to
do
things.
You
don't
teach
them
necessarily
by
putting
on
classes
or
doing
tutorials
or
taking
quizzes
you
do
it
by
saying
here's
a
good,
workflow
and
you're.
The
study
just
presented
showed
that.
H
F
Yep
I
just
wanted
to
like
a
sperm
itself,
won't
solve
all
the
supply
chain.
Security
problems
right
like
it
can
help
some
small
part,
and
I
think
there
are
multiple
use
cases,
as
matt
mentioned
like
you,
can
find
for
a
cyclone
dx
and
spdx.
What
are
their
use
cases.
E
Yeah
I
mean
the
the
the
s-bombs
can
help
in
the
in
the
situation,
where
there's
a
known
vulnerability
in
some
component.
Now
that
does
imply
that
there
are
some
other
tools
being
used
to
find
the
vulnerabilities.
No,
no.
H
No,
no,
I
didn't
know
david,
that's
that's
the
original
spx
use
case.
That's
use
case
101.,
but
if
you're
looking
at
where
cyclone
dx
is
going
and
what
you
can
do
with
it,
you
can
actually
create
s
bombs
of
s
bombs
and
you
can
add
tagging
to
it.
That's
what
I
see
our
team's
doing.
They
want
to
add
their
own
tagging,
name
value
pairs
and
it
their
list
for
use
cases
and
cyclone
dx
is
over
a
dozen
things
long.
Their
review
vision
seems
to
be
much
more
than
just
license.
That's
that's
101!
That's
the.
E
H
Don't
see
them,
I
don't
see
them,
articulate
those
use
cases
in
the
either
in
their
canonical
documentation,
use
cases,
examples
or
any
discussion
materials.
They
don't
enumerate
through
them
at
all.
I'm
not
so
sure
about
that.
But
it's
been
awesome.
I
went
through
I
went.
I
ran
every
sbx
tool.
I
went
through
every
example.
They
do
not
make
an
attempt
to
enumerate
them
or
create
comprehensive
examples
of
how
you
how
you
do
them.
They
don't
they're
more.
H
You
they're
more
focused
on
language
coverage
and
can
I
do
transforms
in
my
current
license
list
and
that's
it
they're
still
gravitating
towards
the
license
and
that's
and
that's
that's
the
central
use
case.
That's
the
overall
use
case.
I
see
tools,
java
tools,
python
tools.
Go
I've
been
through
them
all.
I've
used
them
all.
I
ran
them
all.
I
built
them
all.
H
H
I
think
the
pr
url
thing
that
that's
being
used
in
cyclone
dx
is:
you
have
to
have
an
identification
system
based
upon
your
eyes,
your
scheme,
that's
flexible,
because
you're
looking
at
ci
cd
processes
and
things
change,
agile
computing
methodology
and
git
ops.
For
that
matter,
your
your
your.
Your
identifiers
changed
on
a
on
a
daily
nightly
hourly
or
pull
request
basis.
E
Yeah,
both
both
spdx
and
and
cyclin
dx
have
url
schemes
to
identify
software
because
of
that
very
problem.
Swit
assumes
that
you
can
quick
a
hash
and
that
identifies
it.
It's
really
intended
for
a
proprietary
software
license
compliance
schemes
where
I
have
microsoft
word
installed
on
my
computer,
and
it
has
this.
H
H
C
When
I
look
at
it,
it's
probably
the
data,
the
s
form
stuff
is
essentially
enumerating
what
your
dependencies
are
that's
great,
but
you
need
other
methods
to
apply
on
top
of
it
and
for
a
long
time,
all
the
databases
of
advisories
were
locked
up
in
proprietary
setups.
It's
still
the
case
right.
You
know
like
the
sneak
database
or
the
sona
type
one.
You
can't
use
them
because
the
license
on
them.
You
know
you
need
to
purchase
to
use
it
on
on
most.
H
Software
yeah,
yeah,
they're
reviewed
the
same
way.
It's
an
all
or
nothing
thing.
You
know
until
the
entire
system
is
locked
down,
you
can't
try
and
s.
Doms
can't
be
used
for
a
security
base
of
trust
at
all.
Absolutely
not,
but
what
you
can
do
is
if
you
bring
the
stop
before
you
bring
the
stuff
in
house,
and
you
do
your
own
scanning,
you
can
apply
your
own
risk
assessment.
H
You
can
actually
create
a
new
s-bomb
of
your
own
and
incorporate
the
old
s-bomb
into
it
and
actually
add
your
own
name
value
tags
and
actually
show
proof
that
you've
done
your
own
scanning
and
actually
record
the
tools
you've
used
to
scan
those
things
with.
So
you
can
add
your
own.
You
can
actually
build
your
own
trust
and
if
you
can
use
reuse
the
same
document
representation
format
that
can
be
parsed,
you
can
go
a
long
way.
H
If
you
can,
I
mean,
if
you,
I
think,
most
people,
google
and
everything
I've
heard
that
they
they're
building
they
plan
to
build
a
house
what
I've
heard
at
other
work
groups
here.
Is
you
know
this?
What
we're
seeing
indicators
of
salsa
and
sig
store
is
that
you
know
people
are
building
their
own
artifactories
where
they're
rebuilding
the
software
house
after
they
bring
it
in
through
an
open
source
clearance
process.
That's
what
you
know
ibm
does
we've
always
had
an
open
source
clearance
process.
A
How
do
I
put
this?
Is
it
centralized
in
a
way
that
it's
it's
a
clearinghouse
for
for
all
open
source,
that's
used
or
or
do
business
units?
Do
they
have
a
way
of
doing
stuff
on
their
own,
or
I'm
just
curious?
How
you're
handling
that.
H
Well,
I
think,
we're
I
think,
we're
subject
to
the
to
the
to
the
greater
changes
that
have
happened
over
the
last
20
years,
which
is
you
know,
every
product
needs
to
go
through
the
oss
clearance
process.
They
should
they
need
to.
But
typically
you
know
we
we
find
more
routinely.
You
know
the
average
developer
going
to
a
package
manager
or
going
someplace
and
just
sucking
it
in
right
and
it's
getting
harder
because
then
there's
those
aren't
often
found
because
often
they're
they're
whitewashed
through
containers
they're
brought
in
you,
know,
opaquely.
H
So
I
mean
so
I
mean
and
and
I'll
I'll
admit
I
mean
it's,
you
know
our
opposite.
Clearance
process
is
based
upon
source
code
and
you
know
we're
trying
to
improve
it
to
beef
it
up
to
be
more
container
container
centric.
You
know
to
move
towards
a
zero
trust
model.
So
once
we
create
this,
you
know
we're
actually
building
things
at
the
oci
container
level.
H
E
In
fact,
dan
lawrence
posted
a
blog
post
a
couple
months
ago
about
the
challenges
of
trying
to
go
backwards.
Here's
a
container
what's
in
it
and
it
was
just
it
it's
a
very,
very
long
post
and
the
brief
summary
is:
you
have
to
make
a
very
large
number
of
guesses
and
assumptions
and
hope
you
maybe
were
right
and
even
then
it's
not
clear
and
it's
just
incredibly
difficult,
even
for
relatively
such
simple
situations
to
figure
out.
What's
in
the
you
know,
I
got
a
container
what's
in
there.
Lord
only
knows
is
the
answer.
H
Yeah
absolutely
absolutely
difficult,
but
you
know
I
have
to
say
that
you
know
going
back
to
the
foundation
thing.
One
thing
I
wanted
to
contribute
is
talking
about
package
managers.
I'm
surprised
no
one
mentioned
java
because
of
java
you
basically
were
or
even
c
languages,
objective
c
or
whatever
you
were
given
a
set
of
libraries,
and
that
was
it.
That
was
your.
That
was
your
world.
You
didn't
have
a
package
manager,
but
in
fact
you
were
given
a
static
set
of
libraries
or
packages
that
you
drew
from.
E
That
hasn't
been
true
for
a
long
time
in
the
java
world,
and
you
know,
and
even
for
the
cnc
plus
plus
world,
if
you
work
with
like
a
linux,
distro
you're
bringing
you're
typically
using
it's
distro
package
manager
to
bring
in
stuff
so
package
managers
are
widely
used.
H
No,
I
no.
I
understand
that
david,
I'm
not
saying,
but
I'm
saying
that
you
you
could
you
could
easily
the
world
was
such
that
you
could
easily
go
through
even
a
binary
form.
I've
seen
a
lot
of
companies
create
both
direct.
You
know
once
you
have
the
bi,
the
java
binary,
you
can
actually
scan
and
see
where
things
are
brought
in.
You
had
a
centralized
place,
at
least
to
do
scanning
of
where
these
things
were
brought
in.
H
Yeah
and
and
99
I
mean
most,
it
was-
it
would
be
quite
obvious-
easy
to
scan
for
both
a
buyer
level
and
at
source
code
level
to
see
where,
where
these
things
were
going
from
from
the
package
naming
conventions
and
whatnot
and
so
yeah.
H
From
what's
great,
I'm
guessing
it's
old
school,
maybe
cool!
You
know
in
terms
of
governance
processes,
yeah
glad
you
laughed,
but
I
mean
apache
foundation
I
hated
it.
I
mean
I
was
forced
to
bring
a
project
apache
foundation,
I'm
like
why
can't
we
be
cool,
go
to
cncf?
Well,
politics
aside,
we
brought
it
to
apache
and
now
I'm
grateful
for
it,
because
you
know,
because
my
team
actually
put
you
know,
got
all
the
repo
security
set
up.
We
got.
H
We
actually
created
all
our
written
processes
and
created
tooling
to
back
those
automate
them
as
much
as
possible
and
apache
gave
us
great
prescriptive
guidance
in
terms
of
how
to
how
to
release
software
and
and
how
to
have.
You
know
multiple.
You
know
multiple
eyes
on
code
reviews,
open
processes,
open
governance
and
you
know
we
were
left
to
the
details
of
it
all.
H
But
there
was
very
prescriptive
and
we
you
know
so,
and
they
and
they
all
their
source
code,
is
sign
and
they
keep
increasing
the
strength
of
the
the
keys
that
are
used
to
sign
it.
So
and
I'm
I
think,
that's
what
needs
to
be
that
education
needs
to
be
brought
out.
I
think
newer,
more
lightweight
organizations
should
be
able
to
build
it.
H
In
I
mean
that's
what
I'm
saying:
github
is
fabulous
and
you
know,
and
package
management
groups
could
be
trained
to
do
this
as
a
matter
of
course,
when
people
submit
new
packages
or
you
know
or
code
changes,
they
should
be
running.
You
know
scans
themselves,
you
know,
so
there
are
choke
points
in
these.
In
these
ecosystems
you
get
great
economies
of
scale.
By
going
after
very
few
package
management.
A
Systems
thanks
man,
I
I
appreciate
you
sharing
your
thoughts.
I
I
know
you
put
them
all
down
on
the
the
the
notes
a
couple
weeks
ago,
but
I
wanted
to
talk
about
them
a
little
bit,
so
I
can
get
some
clarity.
I
appreciate
it.
H
E
Kate
stewart
one
I'd
quickly
point
you
to
she's
done
a
lot
of
work
with
spdx
and
I
was
also
involved
in
the
ntia
efforts.
You
know
what
I
mean.
I
will
just
put
k
stewart.
E
Okay,
so
that
would
be
somebody
I
think
that
she'd
be
somebody
definitely
worth
talking
to,
and
you
know
now.
As
far
as
your
larger
question
about
the
tooling
and
the
package
managers
choke
points
thereof,
I
mean
making
a
list
of
here's.
Some
of
the
actively
maintained
tools
is,
I
think,
a
worthy
endeavor,
particularly
as
you
noted
a
lot
of
the
existing
lists,
I
guess
are
not
are
not
being
maintained.
I
know
that
I've
posted
a
list
on
my
personal
website,
but
I
have
not
tried
to
keep
that
up
to
date.
H
See
I
suggested
somebody
last
thursday
they
get
with
you,
I
might
have
been,
it
might
have
been
right.
No,
it
was
dan
dan.
I
I
that
he,
I
told
him
about
your
list.
You
didn't
know
about
it.
I
said
I
told
him
to
get
together
with
you
and
put
that
on
github
repo,
so
everyone
and
let's,
let's
practice
what
we
preach.
Let's
put
it
in
the
github
repo.
E
E
My
point
exactly
you
know,
you
know
I
I
would
want
a
single.
It's
a
group
maintain
not
hey.
You
know
it
was
easy
for
me
to
start
and
post
on
my
own
website,
because
I
was
interested
in
it,
but
but
yeah
is
that
something
that
this
group
would
be
interested
in
and
trying
to
be.
We
have
a
very
good.
H
Reason
this
work
group
and
two
other
group
rates
were
creating
their
own
lists
and
I
went
to
them
and
said:
don't
create
your
own
list
talk
to
david,
so
other
groups
are
other
and
that's
why
I
just
got
mad
at
dan
because
he
was
creating
sas
and
gas
tools.
He
started
going
to
that
realm.
They
started
with
best
practices.
They
started
going,
adding
sas
and
dastal.
He
said
well
we're
already
doing
that
at
this
work
group.
So
don't
duplicate
effort.
Let's
all
work
against
the
same
repo.
E
E
I
mean
we,
we
can
do
it.
However,
we
want,
I
mean
my
personal
druthers
would
be
posting
it
on
on
git.
On
github
I
mean
we've
already
got
a
github
location,
anybody
can
go
edit
and
prove
modify,
and
then
it's
no
longer.
You
know.
F
E
Mine
or
just
anybody
else's
just
put
in
markdown
on
a
simple
file,
there's
nothing.
We
don't
need
to
be
fancy.
A
simple
markdown
file
on
get
on
github
would
be
easy
to
maintain
and
that's
probably
more
important
than
anything
else.
Somebody
feel
free
to.
H
Disagree
yeah.
I
think
that
I
think
that's
great,
I
mean
if
you,
if
we
can
get
even
a
mark
down,
that's
that's
simply
parsed
easily
and
if
we
can
get
work
with
the
screen,
critical
projects,
work,
group
and
stuff
like
that
and
get
them
just
actually
start
using
our
own
tools
against
them
and
creating
reports
that
are
again
checked
in
the
same
repo
that'd.
Be
awesome.
Here
are
the
tools
that
are
on
the
list.
We
ran
our
own
tools
against
it,
and
here
are
the
results
for
these
tools.
That'd
be
awesome.
I
I.
H
H
I
mean
proprietary
doesn't
mean
no
good,
you
know
I
can
you
know,
I
mean
I
work
for
ibm,
you
know
we,
we
have
money,
we
can
pay
people,
we
do
pay
people,
I
won't
say
who
we
pay,
but
we
pay.
Multiple
people
depending
on
the
industry,
but
open
source
is,
is
where
you
know.
That's
that's
where
we're
open
ssf,
so
we
should
focus
on
the
open.
You
know.
H
I'm
really
pissed
at
the
wasp
group
to
include
open,
and
this
is
an
ibm
tool.
Ibm
app
scan,
got
scrolled
to
html
scan,
I'm
showing
them
not
a
hypocrite.
I
don't
want
them
evaluated
either,
because
because
they're
open
is
great
but
they're
open
as
an
sdk
and
stuff
to
call
their
back
in
service,
that's
not
open
to
me.
That's
an
open,
you
know,
ap,
that's
an
api
yeah.
C
Yeah
one
of
the
things
that
we've
pushed
for
as
github
and
we're
a
vendor,
we
sell
a
security
tool
called
coke
up.
The
wines
that
we've
pushed
for
is
better
benchmarking
of
those
tools,
because,
whilst
in
an
ideal
world,
you
know,
I
think
everybody
would
get
security
for
free,
even
avoid
a
large
company.
There's
a
lot
of
energy
in
private
enterprise
trying
to
build
out
these
tools
and
at
the
moment
a
lot
of
them
are
sold
basically
on
they're
sold,
based
on
like
some
benchmark
suite
that
they've
totally
gained,
and
it's.
C
It
doesn't
matter
how
good
your
product
is,
and
one
of
the
things
that
I
think
that
I
see
the
role
of
this
team,
as
is
providing
those
open
ways
to
benchmark
those
tools
and
to
be
able
to
say
actually
we
stand
behind
this.
C
You
know
this
test
and
if
you
pass
it
you're,
probably
quite
good,
and
until
we
created
the
cbe
benchmarking
initiative,
where
we're
like,
which
of
the
things
that
actually
caused
real
vulnerabilities
in
open
source,
can
a
tool
find
and
I'd
like
to
see
this
team
continuing
to
engage
with
kind
of
those
proprietary
tools
in
that
way
where
it's
like?
You
know
what
we
can
be
here
to
provide
a
test
for
them.
C
H
I'm
gonna.
I
want
to
capture
and
replay
what
you
said
the
last
few
minutes
to
make
my
day.
It's
made
my
day.
It's
my
week,
it's
made
my
month.
I'd
love
to
hear
that,
and
I
think
it
goes
a
long
way
to
what
was
presented
to
show
when
you
start
the
demographics
security
through
security
doesn't
work.
Okay
and
the
only
way
you're
going
to
educate
is
by
opening
up
the
processes.
H
E
Curated,
okay,
security
list
of
oss
tools,
a
quick
note-
and
I
realized
we're
out
of
time.
Nist-
has
actually
done
some
of
these
tool
benchmarks.
There's
a
group
called
summate.
I
know
those
folks
there's
a
link
there.
They
have
some
test,
suites
called
juliet
and
they've
actually
done
some
annual.
You
know
here,
let's
show
up
in
the
tool
and
evaluations.
E
E
I
see
a
groan
if
you
don't
know
what
they
are
consider
yourself
blessed,
but
you
will
eventually
receive
the
curse,
and
this
is
where
a
dewitt
clause
is
basically
forbids
the
publication
of
benchmarks
unless
the
toolmaker
likes
the
benchmarks,
which
is
very
very
good
for
ensuring
that
the
good
tools
can't
be
shown
better
than
the
bad
tools.
That's
what
this
this
legal
threat
is,
why
greg
you're
gray
you're
not
seeing
well?
Why
can't
I
differentiate
myself
because
it's
the
people
who
could
do
the
evaluations
are
scared
off.
A
Yeah,
no,
absolutely
no
doubt,
and
you
know,
we've
done
a
number
of
internal
analyses
using
the
juliet
benchmark,
but
because
of
the
debate
clause,
we
can't
share
anything.
E
That's
right,
that's
right
and
I
will,
by
the
way
I'll
add
I
can.
I
can
share
this
much.
The
us
government
internally
evaluates
tools
and
shares
them
among
the
government.
It
often
can't
even
share
that,
among
its
never
mind
the
public
because
of
dewitt
clauses,
so
there
are
people
who
know
about
the
capabilities
of
different
tools,
but
nobody's
been
allowed
to
say
publicly
anything
about
them.
E
A
Well,
folks,
we're
we're
past
the
top
of
the
hour.
I
I
appreciate
the
great
discussion.
Thank
you
again,
george,
for
coming
in.
Thank
you
again
matt
for
for
taking
the
time
to
share
your
thoughts.
It
was
a
good
discussion.
I
appreciate
everyone.