►
From YouTube: Protecting Ourselves from CNCFgate. Software Supply Cha... Andres Vega, Emily Fox & Jonathan Meadows
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon North America 2021 in Los Angeles, CA from October 12-15. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Protecting Ourselves from CNCFgate. Software Supply Chain Security at CNCF - Practices, and Tools - Andres Vega & Emily Fox, CNCF SIG-Security & Jonathan Meadows, Cyber Security at Citi
As the complexity of our software systems grows – and they ingest more and more dependencies to deliver their functionality – the supply chain becomes more complex, and thus more difficult to secure. The industry is forming a consensus around a baseline set of properties for a secure software supply chain, yet these are not enough to protect against some of the high-profile attacks we have seen in recent years. In some cases they may not even have made detection easier. The industry needs to do better, we need to do better. An attacker who compromises a software supply chain can greatly increase the blast radius of their attack to all eventual users of the system. In some cases the exploits are overlooked or unintended bugs; some others have been known to be more deliberate and insidious (most recently, SunBurst/Solarigate).
This presentation shares the experience of the CNCF SIG-Security Supply Chain Working Group with particular attention to intricacies and sharp edges of the practice of creating and maintaining a tightly-secured software supply chain.
A
A
Nobody
really
knows
what
to
do,
because
they'd
like
to
have
that
single
product
to
solve
all
of
their
problems.
So
realistically,
what
is
supply
chain
security
about?
Why?
Why
are
we
talking
about
this
today?
Supply
chain
security
is
about
defense
and
depth,
and
it's
not
only
about
the
layers
of
different
security
components
and
configurations
and
checks
and
verifications
that
you're
doing
on
your
images
and
on
your
signatures
and
on
your
hashes.
It's
also
about
the
things
that
you
are
doing,
layered.
A
Well,
we
wanted
to
create
some
form
of
documentation
or
instruction
for
the
community
to
be
able
to
understand
more
about
why
supply
chain
security
is
so
important
in
cloud
native
ecosystems,
as
a
as
I
talked
about
previously
everybody's
going
to
the
cloud
and
cloud
native
is
becoming
more
and
more
prevalent,
but
we
can't
just
provide
the
community
with
a
list
of
tools
and
say
here
go
check
this
out.
We
want
to
do
more
than
that.
A
We
wanted
to
provide
them
with
recommendations
specific
to
their
risk
tolerance,
whether
or
not
they
have
low
risk
acceptability,
meaning
that
their
environment
is
so
important
or
whether
or
not
they
have
a
software
assurance
requirement.
If
they're
just
writing
something
to
play
with
it
or
create
a
pilot
or
just
learn
a
new
capability,
then
they
don't
have
the
same
level
of
assurance
requirements
as
government
or
industry
is
going
to
have
or
even
healthcare.
A
Why
we're
talking
about
the
things
that
we
are,
we
want
to
provide
them
with
reasonable,
sound
explanations
from
the
community
who
have
been
doing
supply
chain
security
for
as
long
as
as
they
care
to
admit,
or
have
recently
gotten
into
the
habit
of
instituting
these
tools
and
practices
and
have
valuable
lessons
learned
about
why
we
should
be
doing
things
the
way
that
we're
recommending
and,
of
course
we
want
to
provide
the
list
of
tools
that
can
help
along
this
path.
Like
I
said
before,
it's
not
just
one.
B
Thanks
emily,
so
working
with
the
supply
chain,
working
group
we've
written
a
document
detailing
recommendations
for
how
to
secure
your
supply
chain.
I
think
it's
really
important
as
a
document
to
show
how
we
can
leverage
these
cloud
native
technologies
to
start
addressing
some
of
the
issues
that
we
see
at
the
moment.
So
if
this
is
of
interest,
I
definitely
recommend
coming
along
to
the
sig
security
or
the
supply
chain
working
groups
and
contributing
now.
B
Now,
whilst
writing
the
document,
it
became
clear.
We
also
needed
to
look
at
two
different
levels
of
security,
one
for
moderate
use
cases,
but
also
higher
level
use
cases
such
as
where
we're
looking
at
a
critical
build
component
or
working
with
a
regulated
industry,
for
example.
So
we
have
two
different
levels
of
recommendations
within
the
paper
now
in
terms
of
recommendations
themselves.
B
We
really
needed
to
start
with
the
basics,
so
you'll
start
to
see
the
starting
with
the
source
code
itself,
and
this
is
really
to
ensure
that
we
focus
on
who
is
contributing
code
to
the
source
code
repository
and
the
way
of
doing
this
is
pretty
basic.
We
just
look
to
ensure
that
we
have
appropriate
authentication
and
authorization
on
that
repository
and
all
of
the
mainstream
vendors
provide
core
capabilities
in
the
area.
We
just
need
to
ensure
we
use
them
and
configure
that
appropriately,
but
to
take
it
a
step
further.
B
We
also
looked
at
his
working
group
and
have
suggested
using
mfa
for
access
to
those
repositories
as
well
and
additionally
for
higher
levels
of
security.
We
re
recommend
ensuring
that
we
sign
those
code
commits
that
we
put
into
the
repository
again
using
mfa,
and
this
will
mean
that
any
time
that
developer
credentials
are
compromised,
an
attacker
can't
contribute
code
without
the
mfa
device
providing
that
extra
level
of
security
there.
B
So
we
move
on
from
the
source
code,
which
is
pretty
basic,
but
then
go
on
to
the
dependencies,
which
is
a
really
important
area
for
the
supply
chain
and,
if
you
think
about
it,
it's
similar
to
manufacturing
any
other
product
where
you
have
raw
materials
entering
the
manufacturing
process
and
they
check
to
make
sure
that
they're
of
the
right
quality
and
you
understand
the
provenance
of
those
raw
materials
before
using
them.
And
it's
just
the
same
for
software
as
well.
We
need
to
do
the
same.
B
Now,
there's
a
ton
of
work
going
on
at
the
moment
to
evangelize
s
bombs
and
to
assist
with
supply
chain
security,
and
much
of
that
work
has
been
led
by
alan
friedman
from
the
ntia
they're
doing
a
lot
of
good
work
to
evangelize
the
usage
throughout
the
industry
and
have
a
number
of
proof
of
concepts
running,
particularly
with
the
energy
sector
at
the
moment,
but
also
in
addition
to
the
s
bonds.
We
need
to
ensure
we
have.
B
We
trust
where
the
s
bomb
is
coming
from
and
then
it
actually
is
associated
with
the
software,
we're
bringing
in
that's
a
little
bit
of
a
gap
we
have
at
the
moment
or
perceived
gap,
but
chris
blask
and
his
colleagues
are
looking
on
a
d-bomb
initiative,
digital
bill
of
materials,
which
is
looking
to
create
a
secure
channel
that
provides
access
to
those
s.
Bonds.
B
So
in
addition
to
understanding
those
dependencies,
it's
also
really
important
to
understand
the
operational
health
of
that
dependency,
as
well
as
the
contents,
and
this
really
relates
to
the
fact
that
cves
are
a
trailing
indicator
of
security.
It
shows
that
someone's
identified
a
vulnerability,
but
it.
C
B
In
and
of
itself,
a
guide
to
the
security
posture
of
that
dependency
and
to
mitigate
this
issue,
one
of
the
recommendations
we're
looking
at
is
really
having
a
view
of
the
operational
security
of
the
dependency
as
well
as
a
cve.
And
to
do
this,
we
could
potentially
look
at
a
number
of
committers
or
the
number
of
committers
to
that
project,
or
perhaps
how
long
ago
those
contributions
were
made
and
really
assessing
that
information
before
you
ingest
that
data
into
the
into
your
software
pipeline.
B
If
possible,
we
also
need
to
ensure
that
the
libraries
really
came
from
where
we
think
they're
they
come
from,
and
that's
really
requires
us
to
validate
the
signatures
on
these
dependencies
or
signatures
on
metadata.
That's
provided
as
part
of
that
build
press
process
when
they
were
built
using
functions
such
as
notary
or
in
toto
that
we'll
be
describing
a
bit
later
on
and
finally,
for
truly
high
security
dependencies.
B
Now,
building
from
source
is
not
particularly
scalable,
that's
going
to
provide
or
require
a
lot
of
additional
focus
to
do
that,
but
it's
very
important
to
do
so,
if
you're
really
going
for
that
high
level
of
maturity
and
high
level
of
security.
So
this
brings
us
to
the
pipeline
itself
and
essentially
the
way
we
look
at.
It
is
a
factory
that
takes
all
of
the
source
code
and
all
the
dependent
components
brings
them
all
together
and
builds
the
end
product.
B
The
devsecops
reference
reference
paper
that
is
definitely
recommended
viewing,
but
that
software
factory,
what
it
really
discard
describes,
is
the
ability
to
create
multiple
immutable
and
potentially
ephemeral
pipelines,
and
each
pipeline
is
tasked
with
building
a
specific
type
of
artifact
that
can
be
anything
from
a
single
product
or
a
libra
or
anything.
In
between
now.
The
factory
itself
is
built
up
by
joining
together.
B
Now
the
other
point
to
think
about
there
is
those
images
and
ensuring
those
images
are
hardened.
So
you
have
to
build
out
a
pipeline
just
to
create
those
image
images
and
ensure
that
you
have
a
high
level
of
security
for
those
images
themselves,
so
put
them
through
security
tests,
harden
them,
remove
extraneous
software
and
such
and
really
treat
them
with
a
strong
security
effect
ethic.
From
the
start.
B
Now
that
details
how
the
pipeline
itself
is
constructed,
but
we
need
to
also
take
it
a
step
further
and
ensure
that
the
pipeline
wasn't
tampered
with
during
the
running
of
that
build
process,
and
that's
where
we're
ensuring
or
recommending
we
have
a
really
high
level
of
automation
within
the
pipeline.
We
shouldn't
have
human
intervention
within
it.
It
really
does
need
to
be
immutable,
and
that
removes
any
issues
we
may
have
with
people
maintaining
that
pipeline
and
causing
accidental
security
issues.
B
But
we
also
need
to
look
at
the
pipeline
in
how
we're
operating
it
and
that's
where
the
recommendations
we're
focusing
on
around
we're,
making
sure
that
there's
no
network
access
in
or
out
of
that
pipeline,
except
from
to
pull
in
the
dependencies
and
publish
that
end
artifact
and
also
we're
looking
at
how
we
can
monitor
the
build
process
or
the
pipeline
itself.
Now
not
every
stage
within
that
build
is
created.
Equally
there's
some
parts
of
that
stage.
B
An
additional
area
we're
really
looking
into
around
the
the
build
itself
is
for
high
level
or
high
security
applications
around
reproducible
builds,
and
this
is
an
approach
that
focuses
on
ensuring
that
the
build
itself
when
it's
produced
is
done
in
a
highly
reproducible
manner,
and
this
is
to
allow
us
to
do
a
bit
by
bit
comparison.
If
we
replicate
that
build
somewhere
else
to
ensure
that
the
builds
were
exactly
the
same,
and
by
using
that
approach,
we
can
have
multiple
different
software
factories
creating
the
same
software
artifact.
B
But
it's
not
easy
builds
are
often
by
by
then
just
the
way
that
they're
set
up
non-deterministic
due
to
timestamps
being
added
to
output
files
or
differences
in
the
ordering
of
dependencies
within
the
file
system.
The
debian
team
have
made
a
lot
of
progress
in
this
area.
They're
really
sort
of
leading
the
charge
there
and
they've
got
some
good
tooling
around
something
called
the
diffiscope.
B
That
shows
you
the
differences
between
different
different
files
and
tries
to
highlight
the
issues
that's
making
that
build
non-reproducible
at
that
point.
So
that's
an
area
we're
looking
at
focusing
on
from
a
recommendation
standpoint.
I
have
a
lot
of
work
at
the
moment
on
the
reproduciblebuilds.org
site
that
we're
looking
at
pointing
to
now
when
we
have
the
build
complete.
We
need
to
sign
that
final
build
for
deployment,
and
this
is
where
we
have
a
number
of
cncf
projects
already
available
to
help
us
items
such
as
in
toto
and
notary.
B
Now
these
tools
allow
us
in
toto,
for
example,
to
sign
the
metadata
from
the
build
process
which
details
the
exact
inputs
and
outputs
from
each
stage
and
then
change
that
metadata
together.
So
we
have
real
software
provenance
throughout
that
build
pipeline
and
an
understanding
of
what
happened
as
he
built
that
product
and
as
you
deploy
that
that
artifact
or
even
at
runtime,
you
then
have
the
ability
to
validate
the
provenance
of
that
artifact
before
you
make
those
decisions
to
deploy
it
properly.
B
Now,
although
we
haven't
really
looked
in
detail
at
the
architecture,
yet
that
supports
these
recommendations.
That
is
really
one
of
the
next
steps
we're
looking
at.
We
have
started
to
look
at
how
we
can
generate
the
key
material
for
in
toto
and
notary,
so
that
we
have
a
really
small
attack
surface
and
a
small
window
for
compromise.
B
This
is
where
we're
starting
to
look
at
technologies
such
as
spiffy
inspired
ways
of
generating
the
appropriate
key
material
so
that
we
can
use
them
to
sign
those
in
total
signatures.
The
very
very
small
time
to
live,
potentially
leveraging
the
s
vids
to
sign
the
in
total
metadata
at
that
point
and
then,
finally,
using
notary
to
sign
that
end
product
before
we
distribute
the
artifact
externally.
B
B
C
Thank
you,
john.
Between
the
five
main
areas
you've
covered.
There
are
existing
gaps
identified
that
still
require
problem
solving,
while
the
software
factory
analogy
is
a
helpful
model.
To
reason
about,
it
is
important
to
recognize
supply
chains
can
be
complex
or
simple,
but
do
extend
beyond
the
software
factory
and
often
multiple
factories.
C
C
C
Third
validation
of
signed,
artifacts,
backed
by
transparency
log
technologies.
A
solution
that
can
potentially
help
here
is
a
new
product
from
the
linux
foundation
that
provides
transparency.
Logs
six
store
can
act
as
a
source
of
provenance
and
integrity.
While
six
store
is
a
public
service,
anyone
can
stand
up
and
host
the
transparency,
lock,
project
known.