►
From YouTube: Securing Cloud Native Software Supply Chain: The Road Ahead - Luke Hinds (Red Hat) OSCG 2022 Spain
Description
Securing Cloud Native Software Supply Chain: The Road Ahead
Speaker: Luke Hinds (Red Hat)
OpenShift Commons Gathering Kubecon EU
May 17, 2022 Live from Kubecon EU in Valencia, Spain
Full Agenda here: https://commons.openshift.org/gatherings/OpenShift_Commons_Gathering_at_Kubecon_Europe_2022.html
Learn more at: https://commons.openshift.org
A
Okay,
welcome
everybody
and
great
to
be
in
a
around
real
people.
Again,
you
know,
I
can't
remember
the
last
time
I
went
to
a
conference
and
I've
met
a
lot
of
folks
that
I've
worked
with
for
one
or
two
years
and
it's
the
first
time
we've
met
in
person-
and
I
think
that's
probably
common
for
quite
a
few
of
you
there
as
well,
so
yeah
we're
going
to
be
delving
into
software
supply
chain.
A
Now,
a
software
supply
chain
and
the
security
of
it
there's
many
many
components
as
many
areas,
so
I'm
having
to
limit
the
coverage
a
bit
okay.
So
this
I'm
not
saying
this
is
the
entirety
of
what
constitutes
a
secure
supply
chain.
It's
the
areas
that
I've
taken
an
interest
in
and
have
been
working
on
in
the
open
source
communities.
A
A
So
all
of
the
vulnerabilities
that
are
found
and
reported
in
kubernetes
come
to
myself
and
and
several
other
people
in
that
team,
and
we
then
triage
those
those
vulnerabilities
there's.
Some
of
you
might
have
heard
of
the
open,
ssf
the
open
source
security
foundation.
So
I'm
on
the
technical
advisory
council
there
and
last
of
all,
I
helped
start
a
project
founded
a
project
called
sigstor,
which
has
started
to
get
a
fair
amount
of
recognition
recently.
A
A
A
Okay,
and
essentially,
if
we
look
at
we're
kind
of
shifting
from
the
the
left
to
the
right
here,
we'll
typically
have
a
developer.
They'll
be
working
on
some
code.
They'll
commit
that
code
they'll
push
a
pull
request
to
maybe
something
like
github
or
gitlab,
some
sort
of
version
control
system
and
then
other
humans
will
review
that
code.
Okay,
so
they'll
make
assumptions
on
the
developer's
identity,
they'll.
Think.
Well,
that's
bob
or
jane.
I
know
them.
I
can
see
they've
made
a
pull
request,
they'll
review
that
pull
request.
A
They'll
approve
it.
At
the
same
time,
machines
start
to
get
involved
and
run
integration
tests
and
unit
tests
and
and
various
sort
of
automated
actions
start
to
happen.
There
might
be
staging
environments,
you
know
various
sort
of,
like
I
say,
integration
tests
that
kick
off
and
then
from
there
some
sort
of
artifact
might
be
generated.
So
a
container
image,
a
tarball,
a
package
of
some
sort,
some
sort
of
artifact,
is,
is
generated
and
then
perhaps
pushed
out
to
other
systems.
A
It
starts
to
go
into,
for
example,
an
oci
registry
would
be
a
typical
example,
and
then
we
have
sort
of
this
middle
juncture
where
we
have
a
package.
Okay,
as
I
said,
an
artifact,
and
it
could
in
fact
be
a
library,
okay,
so
somebody
else
then
pulls
that
code
into
their
own
ci
flow,
okay
and
then,
eventually,
you
have
your
end
users
and
again
we're
looking
at
a
mix
of
people
and
machines
here.
A
So
we
obviously
have
humans
that
are
going
to
use
software,
but
lots
of
different
vertical
industries
now
use
open
source
software.
So
we
have
energy,
telco,
military
public
sectors
and
so
forth.
Government
systems
so
there's
a
very
wide
pool
of
users
and
machines
that
interact
with
this
supply
chain.
Okay,
now
not
news
to
you,
but
one
of
the
problems
with
these
supply
chains
is
they're
very
susceptible
to
attacks,
okay
and
there's
a
lot
of
attacks.
A
So
I'm
not
really
going
to
go
into
the
the
details
that
the
meter
of
what
these
attacks
consist
of,
but
they
are
throughout
the
chain
so
humans,
their
identity
can
be
compromised.
The
individual
could
be
compromised,
they
might
have
a
pair
of
cryptographic,
keys
that
are
stolen,
somehow
or
their
identity
credentials
are
stolen.
Somebody
maybe
gets
hold
of
their
single
sign-on
access,
which
allows
them
to
then
log
into
their
github
account
or
their
gitlab
account
and
from
there
on
so
so
human
humans
can
be
compromised
and
are
compromised.
A
A
You
can
go
and
look
at
the
yaml
and
look
up
machine
ip
addresses
in
there
or
any
sort
of
details
that
would
allow
them
to
sort
of
perform.
An
attack.
Essentially
typo
squading
is
something
that
happens
very
regular
within
especially
open
source
packages.
Okay,
so
somebody
will
change
a
single
letter
to
make
it
look
like
you're,
a
common
library.
Somebody
makes
a
typo
when
they're
listing
their
dependency
matrix
and
then
they're
compromised.
A
Another
one
which
we've
started
to
see
recently
and
we
can
all
relate
to
is
just
burned
out
developers
you're,
maintaining
a
project,
it's
often
a
thankless
task,
and
then
sometimes
people
get
to
the
edge.
Where
they
just
think
and
they'll
they'll
act
out
and
they'll
do
something
to
sort
of
to
make
their
package
a
lot
of
the
time
it's
relatively
harmless
stuff
like
it
will
perhaps
spam
out
a
message
on
a
terminal
to
stand
it
out
or
but
it
does
happen
and
we're
starting
to
see
that
happen
more
and
more.
A
We
have
these
open
source
developers
who
have
got
projects
that
are
critical
and
widely
consumed
by
other
projects.
You
know
and
they
get
burned
out
and
stressed.
So
that's
something
that
can
happen
quite
a
lot.
Vulnerability
has
not
been
patched
your
system's
not
been
updated,
and
then
we
get
into
things
like
replay
attacks,
okay,
which
package
managers
could
be
quite
prone
to
so
there's
a
lot
more
here
and
if
you've
been
following
the
news.
You'll
know
that
there's
more
and
more
attacks
that
are
coming
out
every
week.
A
So,
just
a
kind
of
a
snippet
of
the
increase
that
we're
seeing
a
few
people
have
started
to
use
this
graphic
recently
as
a
650
percent
increase.
I
do
remember
the
report
from
the
previous
year,
where
it
was,
I
can't
remember
the
actual
amount,
but
there's
been
a
seismic
rise
there,
and
this
is
centered
on
open
source
projects.
A
So
we've
got
a
clear
problem
here,
a
lot
of
the
time
security
people
like
to
be
a
bit
of
a
scaremonger
and
make
things
seem
worse
than
they
are,
but
there
is
a
problem
here
so
just
to
kind
of
further.
My
point
we're.
Actually
you
know
this
is
a
topic
that's
being
discussed
in
the
white
house.
Just
I
think
it
was.
Last
week
there
was
a
meeting
at
the
white
house
where
there's
this
executive
order,
around
software
supply
chains
and
zero
trust
and
so
forth.
A
So
one
of
the
things
that
I
and
several
others
started
to
do
is
we
looked
at
what
are
the
gaps?
Okay,
what
are
the
what's
the
low
hanging
fruit
that
we
can
try
to
to
fix
and
resolve
in
a
software
supply
chain?
Okay
and
one
of
them,
that
was
very
clear,
was
a
a
significant
lack
of
sign-in
adoption.
Okay,
so
essentially
we're
talking
about
somebody
will
have
some
cryptographic,
keys
and
they'll
sign
a
given
object.
Okay,
it
could
be
some
code,
an
artifact,
a
commit,
a
container
image:
okay,
very,
very
low
adoption.
A
Okay,
there's
no
credible,
trustworthy
path
of
like
a
breadcrumb
of
provenance.
So
when
you're
getting
something,
do
you
really
know
who
you're
getting
it
from?
Is
it
who
you
expect
it
to
be?
Okay
and
then
the
other
one
as
well?
Is
that
quite
a
lot
of
the
time
when
you
do
have
something
that
explains
to
you
some
sort
of
provenance?
The
thing
that
created
that
is
actually
software
itself,
so
you're
trusting
software
to
tell
you
what
software
you
can
trust?
Okay,
so
so
these
are
some
gaps.
A
A
So
the
first
one
I'm
going
to
look
at
is
digital
signatures.
Okay,
so
what
does
software
signing
get
us?
Okay?
Well,
the
first
one
is:
you
can
verify
the
integrity
of
content.
So
if
anybody
changes
a
single
bit
in
this
artifact
or
code
or
whatever
it
is
that
you're
ingesting
single
bit
changes
it
change.
It
completely
breaks
the
cryptographic
structure,
which
will
then
invalidate
the
signature.
A
A
I
can
assume
that,
as
I
say
this
a
particular
individual
machine
was
the
one
that
performed
that
action
and
then
I
can
grant
them
some
sort
of
access
or
or
some
sort
of
autonomous
flow
that
can
that
can
manifest
from
that
and
if
it
contains
a
time
stamp,
which
is
something
we're
starting
to
do
a
lot
more
commonly
now
you
can
be.
You
can
have
guarantees
around
when
that
event
happened
when
that
signing
event
happened,
and
this
can
have.
This
can
really
help
around
replay
attacks
forwarded
attacks.
A
Where
one
system
says
this
is
you
go
to
update
and
one
system
says
this:
is
the
latest
version
you
think
great
I'm
covered
now?
I've
got
the
latest
version,
but
there
is
actually
another
more
recent
version,
but
you've
been
spoofed
into
believing
you're
getting
the
latest
version
that
contains
a
load
of
nasty
cves.
So
that's
kind
of
a
common
sort
of
replay
attack.
A
Okay,
so
we
did
some
due
diligence
around
the
open
source
ecosystem
to
look
at
who
was
signing
okay
and
a
very
mixed
picture,
so
the
linux
kernel
they
use
something
called
tofu
trust
on
first
use.
This
is
pretty
similar
to
ssh.
You
know
when
you
ssh
to
a
new
box,
it
sends
a
fingerprint
and
then
you
set
you
you
put
yes
and
then
once
you
do
that
the
key
is
imported
into
your
ssh
known,
hosts
file,
okay
and
then
from
there
on.
You
trust
that
machine.
A
That's
that's
trust
on
first
use,
so
a
lot
of
the
time
people
will
store
a
public
key
somewhere
or
they'll
circulate
a
public
key.
You
import
that
key,
and
then
you
trust
it
from
there
on
okay,
there's
no
latter
junction,
where
you
reconfirm
that
trust
to
make
sure
that
that
individual
is
a
valid
and
trustworthy
entity.
Okay,
again
is
a
kind
of
a
very
typical
picture.
We
started
to
notice
a
lot
of
keys
that
were
stored
on
github
repositories,
websites,
all
mediums
that
can
be
hacked
and
have
been
hacked.
You
know
like
a
wordpress
site.
A
A
Interestingly,
kubernetes
were
also
one
that
had
no
signing
of
any
of
their
release,
artifacts
and
so
forth,
but
I've
tagged
that
one
to
green,
because
just
last
week
the
sig
release
team
rigged
in
six
store,
so
they're
actually
a
green
now
so
so
I
got
to
update
that
one
and
a
lot
of
these
communities
were
actually
working
with
them.
We'll
look
at
that
a
bit
later
to
to
integrate
this
sig
store
system
same
again
for
package
managers.
So
I'm
not
talking
about
rpm.
A
Here,
we've
been
doing
this
for
quite
a
while
in
red
hat,
we've
got
a
very
robust,
established,
sign-in
system,
okay,
but
a
lot
of
the
open
source
package
managers.
Again
some
of
them
have
systems,
but
the
adoption
is
incredibly
poor.
It's
typically
two
to
three
percent.
Again,
it's
it's
the
trust
models.
Are
you
know
a
questionable?
Some
have
nothing
at
all.
I
do
quite
a
lot
of
work
in
rust,
I'm
quite
a
fan
of
rust,
but
with
crates
you
pull
in
everything
completely
untrusted.
A
So
there's
a
lot
of
rust
has
a
lot
of
nice
things
around
memory.
Safety
is
very
strict
ownership
model
within
the
compiler,
so
it's
very
good
language
for
security
and
not
suffering
from
general
sort
of
memory
attacks.
That
can
happen
but,
as
I
say,
all
of
the
modules
are
pulled
in
completely
untrusted
and
sure
enough.
There
was
a
hostile
package
found
just
last
week
on
craits
okay,
so
why
is
nobody
really
signing?
A
We
know
that
we've
got
this
technology,
that's
been
around
for
quite
some
while
okay
and
it
provides
all
of
those
guarantees
to
you,
but
the
adoption's
quite
poor.
So
why
is
that?
So?
Generally
most
people
find
the
managing
of
private
keys
to
be
a
challenge.
Okay-
and
there
is
some
expense
in
this.
So
if
you
want
to
do
it
right,
you
need
to
buy
something
like
a
ubi
key.
You
need
to
set
it
up.
A
Okay,
now,
there's
some
folks
in
the
audience
that
will
be
very
comfortable
about
doing
that,
but
you're
the
minority,
a
majority
of
people
just
don't
know
how
to
do
that.
They
don't
know
how
to
leverage
it
into
their
their
everyday
development,
environment.
Okay,
a
handling
key
rotation
is
is
difficult.
Knowing
what
to
do
when
there's
a
key
compromise,
if
you
ask
most
people
what
would
happen
if
your
gpg
was
compromised,
what
action
would
you
take?
They
wouldn't
know,
you
know,
they'd
have
to
start
searching
google
and
panic.
A
Okay,
as
I
said,
key
compromise
is
scary
and
the
existing
of
identities
to
keys.
So
really
what
we
have
is
something
called
a
weber,
trust,
okay
and
I'm
talking
specifically
about
gpg
here.
So
you
would
generally,
you
would
meet
in
people.
You'd
have
these
key
sign-in
parties
where
you'd
meet
in
person,
okay
and
you
would
all
sign
each
other's
keys
and
you'd
show
each
other
your
driving
license.
A
So
a
lack
of
adoption
is
largely
due
to
the
tooling
okay,
so
people
find
it
cumbersome
to
use
they're
concerned
about
how
do
I
react
when
there's
a
key
compromise?
How
do
I
securely
store
my
keys?
Do
I
keep
them
on
my
laptop?
What
happens
if
my
laptop's
stolen?
What
happens
if
somebody
hacks
my
laptop,
you
know
how
do
I
trust
other
people's
keys?
A
How
do
I
use
them
in
a
on
somebody
else's
system
if
I
want
to
run
a
github
action
or
or
some
sort
of
automated
flow
and
the
general
consensuses
are
on
among
the
people
that
are
a
lot
smarter
than
me
that
it's
broken
and
it's
time
to
move
on,
the
isrg
started
to
look
up
working
on,
let's
encrypt
generally
on
the
internet,
the
proliferation
or
the
count
of
websites
that
were
running
https.
Was
it
very,
very
low?
Okay,
I
mean
amazon.com
and
bank,
you
know
various
banks,
they
used
to
run
on
http,
okay.
A
This
is
going
back
quite
a
quite
a
while
ago.
Okay
and
most
of
this,
the
adoption
obviously
improved
within
the
enterprise
sector,
but
the
the
you
know
your
every
day,
user
that
had
a
website
or
was
running
some
sort
of
system,
very,
very
low
rate
of
adoption.
Okay-
and
most
of
that
was
because
it
was
a
cumbersome
process.
A
A
A
So
now,
when
you
go
to
a
site,
if
it's
not
running
tls,
it
kind
of
feels
e,
it
feels
a
bit
dirty
like
I'm.
Am
I
going
to
get
a
virus
here?
It's
just
you
know,
and
the
browser
gives
you
lots
of
angry
warnings
about
stay
away.
Okay,
so
our
idea
was
what,
if
we
could
do
the
same
for
software,
so
when
you
pull
a
container
image
or
you
import
a
library
or
any
particular
action,
the
general
social
construct
is
that
if
it's
not
signed,
I
don't
want
to
use
it.
A
A
So
it's
a
group
of
projects,
okay,
they're,
all
open
source
projects.
It
was
founded
by
red
hat.
Okay,
so
there's
a
lot
of
media
attention
around
sig
store
at
the
moment,
but
it
was
a
project
that
was
grown
in
red
hat
okay
and
it's
a
collection
of
projects
that
come
together
to
provide
a
service.
Okay,
an
infrastructure
for
software,
signing
okay
and
we'll
go
into
what
that
consists
of
shortly
now,
with
sigstor
there's
two
models
which
I'll
cover
again
a
little
bit
later,
but
it's
important
to
point
it
out.
A
One
of
them
is
what
we
call
a
public
good
service,
okay,
so
we're
looking
to
launch
we're
actually
in
soft
launch
of
a
public
service.
Okay.
So
this
is
free
to
use,
there's
tooling,
to
make
it
easy
to
sign
things
and
to
verify
things
and
again
it's
modeled
on
this
successful.
Let's
encrypt
rollout,
that's
happened.
A
So
if
we
jump
into
the
different
projects,
we
have
full
co,
which
is
rca
okay.
This
is
what
provides
the
software
signing
certificates
I'll
cover
this
a
little
bit
more
lately
but
later,
but
we
leveraged
something
called
open:
identity
connect,
okay,
which
is
a
key
part
of
full
cio.
We
have
recall,
which
is
the
transparency
log.
That
was
one
of
the
first
projects
that
we
worked
on
and
this
is
you
can
kind
of
think
of
it
like
a
ledger.
It's
it's
an
immutable
store.
Okay,
so
anything
that's
recorded
in
there.
A
It
cannot
be
tampered
okay
and
then
it's
publicly
auditable.
So
this
transparency
log
is
open
for
anybody
toward
it,
cryptographically
audit.
So
if
any
of
you
are
curious
on
the
protocol,
it's
something
called
a
merkle
tree,
okay,
which
is
pretty
similar
to
what
git
uses
and
other
things
like
os
tree
and
so
forth.
A
We
then
have
various
clients.
Okay,
one
of
them
that's
become
very
popular
is
cosign,
which
is
a
container
signing
tool.
Okay
and
there's
clients
for
maven,
ruby,
gems,
python.
I've
been
working
on
a
sig
store,
rust
library
and
and
many
more
there's
many
more
projects
coming
online.
So
you
have
your
infrastructure,
your
service,
and
then
you
have
your
client
implementations,
which
sign
and
verify
using
that
that
service,
so
I'll,
try
not
to
be
too
hand
wavy
here,
but
one
of
the
key
technologies
that
we
have
is
something
that
we're
called
kila
sign-in.
A
A
What
happens
then,
is
you'll
connect
to
our
system
and
we'll
see
this
happening
a
bit
later,
with
a
demo
you'll
connect
to
our
system.
It
will
then
send
you
back
an
open
identity,
connect
screen.
Okay,
so
then
you
can
log
in
with
github.
You
could
log
in
with
your
red
hat.
Google
email
address
okay,
and
we
then
have
an
exchange
there's
an
id
token.
We
do
a
challenge
and
then
what
happens
is
you're.
A
Then,
given
a
code
signing
certificate,
which
your
client
can
then
use
to
sign,
an
artifact
okay
and
then
all
of
that
is
stored
into
something
called
a
transparency
log,
so
that
whole
event
of
this
individual
or
this
machine
was
in
possession
of
this
key
pair
at
this
particular
time,
because
it's
time
stamped
and
they
signed
a
particular
artifact
by
a
digest.
Okay
and
then
the
the
real
kind
of
bit
to
to
keep
in
mind
here.
If
you've
not
followed.
What
I've
just
outlined
is
that,
once
this
process
completes,
you
can
discard
the
keys?
A
Wow
wow,
okay,
that's
throwing
me
off
a
bit.
Okay,
so
probably
have
to
drop
the
demo,
okay,
so
yeah.
So
this
is
a
a
view
of
of
what
we
capture.
Okay,
so
the
top
one
is
you
can
see
that
there's
a
github
action,
that's
run,
okay,
and
this
is
actually
stuck
in
an
x509
certificate,
which
is
also
recorded
in
the
transparency
log,
or
you
see
another
example
as
well
where
we
have
a
sam
of
my
email
address.
A
So
I
sign
something
using
my
email
address,
so
I'm
going
to
have
to
skip
the
demo.
I
was
going
to
show
you
commit
signing,
so
we've
just
actually
got
this
new
project
that
released
just
a
few
days
ago,
where
you
can
now
sign
commits
with
sigstor
okay.
So
we
don't
just
do
keyless
as
well.
We
do
lots
of
integration
with
kms
systems,
so
aws
as
azure
gcp
volt.
A
You
can
use
local
key
management.
If
you
wanted
to,
we
wrote
we
support
pkcs11,
it's
a
crypto
standard,
so
you
could
use
a
ubq
or
a
hsm.
So
this
is
more
leveraged
by
your
enterprise
type
customers,
okay
and
community.
Adoption
has
been
rife
I'll
skip
over
this
quite
quickly,
but,
like
I
said,
kubernetes
have
just
come
online
and
there's
lots
of
other
communities
that
are
bootstrapping
and
planning
to
go
into
production
with
six
store.
A
A
You
can
sign
kubernetes
objects,
so
yaml
files
with
sigstor,
okay
and
then
an
admission
controller
will
validate
that
sign-in
before
that
is
allowed
to
to
execute
okay,
podman,
the
podman
team,
so
daniel
walsh
and
folks
they're
integrating
the
container
signing
parts
into
podman
okay.
So
this
is
a
using
sig
store
and
part
of
the
old
red
hat,
simple
sign-in
protocol,
okay
and
then
you'll
be
able
to
verify
signatures,
and
they
also
have
a
policy
implementation
that
they're
working
on
quay
has
shipped
with
support
for
sigstor.
A
A
You
can
also
generate
something
called
an
s-bom,
a
software
bill
of
materials
which
will
list
the
actual
various
actions
that
were
part
of
that
task
or
pipeline,
and
it
will
sign
it
and
it
will
store
it
into
the
transparency
log.
So,
all
of
the
time
all
of
these
signature
events
are
stored
into
the
transparency
log.
A
This
is
experimental,
but
somebody
in
my
team's
recently
been
working
on
a
kogi
plug-in,
so
we're
looking
at
sticking
rpm
signatures
into
into
the
recoil
transparency
log
and
last
of
all,
just
coming
back
to
what
we
said
earlier.
We
have
two
sort
of
models.
One
is
the
public
service,
so
you
can
think
of
that,
like
let's
encrypt,
but
the
other
model
is
to
deploy
this
behind
a
firewall.
A
So
if
you're
a
customer
of
red
hat-
and
you
want
to
run
for
some
reason-
your
own
internal
sig
store-
you
can
do
that.
You
know
it's.
It's
all
cloud
native
technology,
it's
relatively
easy
to
to
set
up
and
to
run
and
or
you
could
run
a
sort
of
a
hybrid
model
where
you
utilize
the
public
upstream
service
for
when
you're
consuming
software
artifacts
from
upstream
or
you
could
have
your
own
sort
of
internal,
only
code
and
artifacts
that
you're
generating.
A
So
that
brings
me
to
the
end
happy
to
take
questions.
Anybody
wants
to
come
and
grab
me
afterwards.
Do
do
so,
and
I
can
even
show
you
the
demo
and
last
of
all,
there
is
a
sig
store.
Booth!
That's
going
to
be
in
the
marketplace,
so
you
can
also
come
there
and
find
some
more
information
about
the
community
and
very
very
last
one.
Sorry,
if
you
want
to
integrate
and
hack
on
this
come
and
see
me,
you
know
we
really
are
encouraging
people
to
to
start
to
innovate.
On
top
of
this
platform,
okay,.