►
From YouTube: Make the Secure Kubernetes Supply Chain Work for You - Adolfo García Veytia, Chainguard
Description
Don’t miss out! Join us at our upcoming hybrid event: KubeCon + CloudNativeCon North America 2022 from October 24-28 in Detroit (and online!). Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Make the Secure Kubernetes Supply Chain Work for You - Adolfo García Veytia, Chainguard
Starting in Kubernetes 1.22, SIG Release started building new security features into Kubernetes releases to make the project a better citizen in the software supply chain. The push to secure the release process has produced tools and processes that have improved the way other projects in the ecosystem are released. At the same time, we have made sure that Kubernetes plays well in the wider chain: verifying what we get from upstream and making sure consumers of our artifacts can trust what they get from us. This talk will give an overview of lessons learned and tools we have created that you can reuse in your own projects to secure your releases. It will center around three key moments and technologies: The initial effort involved producing SBOMs to describe sources and artifacts along with their dependencies. Then, we'll understand the provenance attestations that make the release process SLSA compliant. Finally, we'll see how digital signatures are implemented in the project.
A
Okay,
welcome
everyone.
Thank
you
for
coming
to
my
talk,
how
this
is
like
a
really
exciting
and
also
stressful
time
for
me,
because
it's
like
a
huge
auditorium
and
shout
out
to
the
people
in
the
cheap
seats.
Thank
you
so
much
for
attending
my
name
is
adolfo
garcia.
I
am
a
staff
software
engineer
with
changard
and
we
are
a
company,
it's
a
new
startup
that
we
are
working
on
the
supply
chain,
security
space
and
my
I
am
also
one
of
the
tech
leads
with
kubernetes
sig
release.
A
A
A
A
So
remember
that
time
when
you
wanted
to
be
like
that
intergalactic
hero
and
just
travel
all
the
way
out
to
the
outer
rim
with
your
pet
robot
and
you
finally
convince
your
uncle
to
go,
get
one
great
model
that
you
wanted
only
to
arrive
to
your
house
and
have
it
malfunction
and
explode
in
your
face
and
now,
of
course,
a
guy
that
sold
it
to.
You
is
laughing
at
you
and
making
you
cry
and
you're
all
desperate.
Well,
yeah!
That's
a
big
problem.
What
was
the
problem
right
there?
A
A
Just
to
ensure
that
you
at
least
were
aware
of
what
you
were
getting
right,
so
story
number
two
career
goals,
so
it
finally
happened
right
that
job
that
you
really
wanted
has
become
a
reality
and
you
are
among
the
organizations
that
you
always
wanted
and
you're
not
just
gonna,
be
there.
Actually,
you
are
not
that
guy.
You
are
that
one
but
you're
gonna
grow,
you're,
gonna
go
places
and
you've.
A
Since
you
were
a
dutch
kid,
you
were
always
follow
those
role
models
that
you
wanted
to
become,
and
your
organization
always
opens
up
new
opportunities
for
people
to
grow,
so
you're,
actually
pretty
fine.
So,
finally,
that
project
that
you
really
wanted
to
work
on
happens
it
would
it
shows
at
your
door
or
well,
maybe
just
in
ingested
by
the
tractor
beam
into
the
space
station.
A
Your
team
is
already
working
management,
upstairs
is
doing
like
all
of
the
important
stuff
and
you
are
in
charge
of
the
most
critical
piece
of
the
project,
the
ssu
and
you
of
course
understand.
What's
your
what's
on
on
your
hands
right,
so
once
you
understand
your
part
in
the
organization,
you
get
like
a
straight
vision.
A
Visit
and
of
course,
you'll
see,
your
team
is
fooled
by
those
guys
which
convinced
to
convince
them
to
to
get
those
people
in
and
everything
breaks
loose
inside,
and
not
only
that,
but
they
do
so
by
bad,
acting,
which
also
well
it's
those
two
guys
that
always
get
the
girls
and
you're
the
only
which
makes
it
even
worse.
So
your
boss
is
now
really
angry
at
you
and
you
simply
all
of
the
project
went
down
because
of
you.
So
what
happened
there?
A
A
So
what's
the
solution
to
that
sign
your
things
whenever
you
are
accepting
or
producing
things
for
others,
you
might
as
well
sign
them
and
verify
them.
If
you
ingest
some
dependency,
you
have
to
make
sure
that
it's
coming
from
where
it's
supposed
to
be.
There
are
now
tools,
sticks
or
salsa.
That
will
help
you
build
that
signing
architecture
right
and
also
provide
you
with
the
proper
tools
to
make
that
easy.
A
Final
story.
Yeah
bad
carbonite
results.
So,
since
forever,
you
wanted
to
make
a
really
nice
picture
for
your
living
room,
comprised
of
your
worst
enemy.
Hang
it
there
in
carbonite,
and
so
you
finally
raise
the
money
rent
the
facility
invite
all
of
your
friends
to
watch,
set
the
thing
going,
and
then
you
get
this
guy.
A
So
for
those
who
are
not
star
wars
fans,
this
is
not
what's
supposed
to
happen.
This
is
george
lucas.
Instead
of
han
solo
in
the
carbonite,
and
then
you
are
all
confused
that
you
don't
understand
what's
happening,
and
so
how
did
that
guy
get
in
there?
So
someone
put
it
in
there
did
I.
What
did
I
make
a
mistake?
A
A
So
why
couldn't
he
tell
what
went
on
down
there
because
there
was
no
provenance
information,
so
you
need
to
get
like
a
record,
a
formal
record
that
you
can
query
and
understand
what's
happening
inside
of
your
build
process,
for
that
we
have
like
also
salsa
can
help,
because
it
helps
you
structure
where
and
it
makes
you
understand
where
you're
doing
things
right,
where
you're
doing
things
wrong
and
if
you're
missing
one
of
the
pieces.
Also,
the
intolerable
project
makes
you
like
tie
all
of
those
information
about
your
build
process
together.
A
So
the
idea
is,
if
you
generate
a
problem
and
satisfaction
effect,
each
of
the
steps
in
your
build
process.
You
get
like
a
nice
record
of
what
went
in
one
without
and
what
were
the
parameters
that
you
used
to.
You
used
to
run
that
that
step
and
you
sign
everything
and,
at
the
end,
you
can
easily
detect
where
each,
where
any
of
those
things
when
went
wrong.
A
A
This
is
not
only
for
kubernetes
itself,
but
also
for
some
of
the
tanir
kubernetes
project,
so
you
would
have
things
like
cluster
api
kops.
All
of
those
projects
are
in
theory.
Well,
at
least
they
have
the
what
I'm
going
to
show
next
available
to
them
to
to
to
release
their
their
artifacts.
So
a
project
in
kubernetes
usually
gets
its
own
project
and
staging
pocket,
and
I'm
gonna
not
gonna
go
into
details,
but
the
idea
is
that
they
build
their
artifacts
stage
them
in
their
pocket
and
in
their
registries
and
from
there
we
promote
them.
A
So
this
is
like
a
high
level
overview
in
and
I'm
gonna
go
a
little
bit
deeper
into
how
kubernetes
gets
released,
the
actual
kubernetes
images
and
binaries.
So
we
start
with
a
build.
We
build
the
things
we
build
the
source
code
with
at
a
certain
build
point
and
we
generate
the
bin,
the
binaries
and
the
images,
and
then
we
push
those
to
staging
like
we
showed
on
the
on
the
previous
one.
But
we
what
we
did
to
secure
make
this
a
little
bit
more
hardened
and
secure.
A
Was
we
pushed
more
steps
into
between
building
and
pushing
so
the
first
one?
We
built
an
s1.
This
kubernetes
got
its
first
test
bomb
like
what,
like
two
years,
one
year
and
a
half
ago
detailing
everything
that
you
needed
to
know.
So
this
satisfies
the
problem
of.
If
we
ship
a
kubernetes
with
a
bad
motivator
inside,
we
want
to
know
right
away
as
soon
as
possible.
So
even
at
some
point
you
can
even
imagine
stopping
the
build
if
we
detect
one
of
those
components
to
be
bad
or
compromised.
A
The
second
one
we
attest
to
the
provenance
of
the
artifacts
that
we
are
generating
there.
The
first
we
record
the
source
code,
so
we
clone
kubernetes
kubernetes.
We
establish
the
build
point
that
which
we
are
going
to
be
building
and
all
of
that
goes
into
the
other
station
and
we
record
the
parameters
for
that
for
that
bill
and
which
apply
to
all
of
the
subjects
that
we
are
attesting
to.
A
A
No,
so
the
problem
is
this:
we
are
generating
the
provenance
at
the
stations
inside
of
the
of
the
staging
process.
You
should
not
do
that
if
you
generate
a
prominence
attestation
from
inside
your
build
process,
anyone
that
compromises
that
is
gonna
can
falsify
potentially
falsify
the
artifact.
So
you
may
get
an
attestation
that
says:
okay,
I
built
this
from
kubernetes
kubernetes,
whatever
build
point
well
in
reality,
it
could
be
from
pulling
from
another
repository
somewhere
a
clone
of
a
hacked
kubernetes
tree.
A
So
this
is,
we
did
this
like
in
a
very
early
days
of
stats
at
0.1.
So
this
is
one
thing
that
we
have
to
change.
I
just
wanted
to
explain
and
everybody
let
everybody
know
why
this
is
structured.
The
way
it
is
right
now
we're
it's
going
to
be
changing
soon.
So
after
we
have
everything
built
and
signed,
we
push
everything
to
the
staging
repositories
and
we
sign
those
artifacts
and
then,
after
that,
we,
the
next
step
is
that
we
check
the
the
we.
A
We
start
the
promotion
process,
so
kubernetes
itself
doesn't
yet
use
the
file
promoter.
We
just
use
the
image
promoter
so
that
we
have
two
processes,
one
for
promoting
images
and
state
and
publishing
them
to
the
production
registries
and
another
process
to
promote
files.
We
haven't
yet
gotten
to
finish
the
file
part,
but
it's
it's
getting
there.
A
A
So
we
check
those
signatures
to
ensure
that
we
are
dealing
with
the
real
thing
that
we're
expecting.
The
idea
is:
don't
let
the
artifact
promoter
trust
the
staging
just
treat
it
as
a
process
somewhere
else.
Zero
trust,
I
think
that's
would
be
the
the
catchphrase
here
so
after
that
we
copy
all
of
the
images
to
their
production
registries
and
once
they're
there.
A
We
take
the
signatures
from
the
original
staging
and
transfer
those
and
append
them
to
the
to
the
images
in
one
of
the
registries
and
then
the
production
registry,
and
then,
finally,
we
signed
them
again
and
why
do
we
send
them
again?
The
idea
is
once
we
we
promote
these
images.
They
are
going
to
end
up
in
the
production
registry
and
they
will
get.
They
will
have
two
signatures
attached
to
them.
A
A
Well,
the
reason
is
this:
if
you
think
about
it,
the
images
are
already
published,
but
at
this
point
and
the
problem
is
we
cannot
so
this
this
process
here
is
is
kicked
off
by
a
via
a
process
outs,
an
external
process
that
we
cannot
trust.
So
we
check
the
artifacts
that
it,
the
the
invocation
is
commanding
us
to
run.
A
So
we
check
okay,
if
I'm
going
to
release
this
list
of
active
hacks
and
we
have
the
complete
list
because
it's
in
the
s
bomb,
we
check
the
provenance
and
the
prominence
tells
us
okay,
that
list
of
artifacts
you're
about
to
release
actually
comes
from
the
staging.
So
that's
the
first
one.
The
second
is:
we
check
the
image
signatures.
A
Images
at
this
point
are
already
published,
so
even
if
we
haven't
yet
sent
those
emails,
you
may
be
familiar
with
kubernetes
120.
Whatever
is
released,
you
can
pull
the
images
at
that
point
so
before
we
call
the
release
complete,
we
check
those
same
dentures
because
at
in
release
we're
going
to
be
publishing
a
set
of
artifacts
that
also
reference
those
images,
so
we
publish
release,
notes,
documentation,
we
publish
the
s-bombs,
so
we
need
to
ensure
that
those
digest
of
the
images
we're
publishing
are
in
fact
what
they're
supposed
to
be.
A
A
There
are
some
patterns
emerging
now
which
are
useful
and
happy
to
chat
about
them
with
anyone,
and
we
finally
copy
those
files
to
the
production
bucket
at
this
stage
relevant
to
this
talk,
we
push
to
the
production
pocket
the
at
the
stations
and
the
s
bombs.
So
we
get
like
at
that
point.
We
can
have
a
complete
release
where
we
have
all
of
the
artifacts
all
of
the
dependencies
described
fully
in
the
husband
we
have.
A
A
We
are
generating
at
the
station
for
this
process,
one
for
this
process
and,
if
you
think
about
it,
are
the
image
promoter
is
missing:
it's
it's
at
the
stations
and
that's
a
project
that
will
be
happening
soon.
I
hope
by
the
next
cubecon
we
can
show
off
like
a
complete
picture
of
sas
at
the
stations
from
end
to
end
we're
almost
there.
We
have
all
the
tooling
libraries
everything
already
developed,
so
they're
going
to
be
there
so
well
yeah.
It's
it's
a
lot
to
consider.
A
Just
gonna
leave
this
a
little
bit
so
that
you
can
sink
and
let
it
digest
it.
But
if
we
are
gonna
have
like
the
main
takeaways
for
this
this
process,
I'd
say
would
be
like
this.
So
three
things
to
remember
so:
first
one
observe
and
document
your
bills
with
salsa
provenance
at
the
station.
So
you
want
to
be
sure
that
you
can
go
back
and
check
what
happened.
Did
what
what
was
the
actual
source
code
that
I
pulled?
A
What
were
the
invocation
of
the
commands
that
I
did
and
the
more
granular
that
you
can
split
those
at
the
stations
and
the
more
detail
you
can
get
into
them?
The
better
at
some
point-
and
this
is
something
we've
been
seeing
at
chengar
working
and
talking
with
customers
builds,
are
gonna
have
any
so
your
workloads.
A
A
The
next
one
ensure
that
your
artifacts
are
authentic
and
immutable
by
signing
and
verifying
them,
and
it
goes
both
ways
you
as
a
player
in
the
chain
and
you
as
a
consumer
of
others.
So
you
need
to
to
make
sure
that
those
artifacts
cannot
be
changed
from
the
moment.
Someone
released
them
to
the
moment
that
you're
consuming
them
and
you,
if
you're
a
nice
player
in
the
whole
supply
chain,
you
should
be
ensuring
the
same
thing
for
others
and
finally,
document
all
of
your
components
by
consuming
and
generating
s
bonds.
For
all
of
your
releases.
A
It's
I
mean
I
hate
to
buy
it
on
the
same
thing
all
the
time,
but
the
examples
sometimes
are
useful
log4a
how
many
people
are
using
loc4j
for
examples?
Well,
if
you
have
s
bombed
the
whole
world,
it
would
be
like
fairly
easy
to
detect,
which
workloads
have
have
a
vulnerable
look
for
j
vulnerability
in
them
and,
finally,
it's
I
would
like
to
just
leave
you
with
thank
you
and
just
making.
A
A
The
small
tool
can
generate
part
of
the
provenance
attestations
from
you
for
for
from
your
binaries,
and
so
as
we
move
forward
most
most
most
of
those
artifacts
are
gonna,
be
most
of
those
tools
are
gonna,
be
available
to
you
and
the
final
one
is.
What
was
my
final
point?
Well,
okay,
I
think
that's
it
and
and
yeah,
and
if
I'm
happy
to
open
it
up
for
questions,
if
you
have
one
and
otherwise
thank
you.
B
So,
first
of
all,
thank
you.
It
was
an
excellent
presentation.
I
really
enjoyed
it
a
lot
thanks
a
lot,
and
my
question
is
how
to
which
format
do
you
use
to
generate
the
s
bomb?
I
saw
the
spdx
logo,
but
I
I
was
under
the
impression
that
that
was
for
licenses
more
than
actual
software
versions,
and
how
do
you
feed
it
to
I
mean
scanners
or
anything
that
can
consume
that
format.
A
Kubernetes
is
a
project
of
the
cncf
which
in
turn
is
part
of
the
linux
foundation
and
aspdx
is
the
s1
standard
from
the
linux
foundation.
So
we
use
that
there
have
been
some
calls
to
request
from
people
to
to
for
us
to
us
to
use.
Also
cyclone
dx
and
I
mean
we're
open
to
it.
But,
as
you
can
see,
we
still
have.
I
mean
secretly
sounds
like
a
big
deal,
but
in
reality
it's
like
it's
a
really
thin
team
as
we're
always
looking
for
contributors.
A
If
you
want
to
join
on
any
of
these,
please
that
was
my
family
message.
Please
come
and
help,
and
so
we're
like
really
stretched.
In
so
I
mean
I
would
love
to
to
to
to
work
on
a
cyclone
dx
version
of
the
s1,
but
we
simply
don't
have
the
volunteers
all
the
time.
So
if
anyone
wants
to
work
on
that
happy
to
have
them
over,
I
would
certainly
love
to
have
our
tool
produce
both
standards
but
yeah.
A
Currently,
that's
one
and
the
second
is
we
produce
our
first
test
bomb
and
then
we've
been
trying
to
feed
it
into
some
of
the
of
the
security
tools
and
we've
been
doing
making
improvements
to
the
output
as
time
has
been
going
down.
So,
for
example,
we
we
saw
that
some
of
the
the
the
security
tools
key
on
on
the
p
url
of
the
packages.
So
we
added
them
and
then
there's
another
project
well,
another
pr
going
to
to
add
the
cpes
to
it
and
well
that's
the
current
status.
No
okay.
D
A
So
one
there
are
mainly
two
patterns
that
I
have
seen
to
deal
with
a
testing
to
the
build
process.
One
is
having
a
binary
calling
another
binary
or
kicking
off
a
process
to
ensure
that
so
in
observing
it,
so
you
have
a
binary
that
calls
the
process.
Then
the
process
runs.
Has
some
outputs
and
the
same
binary
sees
what
it
outputs
and
records
that
and
it
so
it
knows
what
it
pulls,
what
the
way
it
run
it
and
what
came
out
so
that's
one
and
the
other
is
calling
out
some
sort
of
a
web
hook.
A
E
A
Question
so
I
don't
know
if
you
saw
the
six-star
logo
on
the
talk,
so
we
used
six-story
for
signing
six
stories,
awesome
project
that
you
can
go
and
visit
our
booth
and
down
in
the
in
the
expo
hall
and
sixther
allows
you
to
sign
things
with
the
female
keys.
We
call
this
process
keyless,
it's
not
really
keyless.
We
just
use
ephemeral
certificates
and
keys
that
you,
where
the
way
it
works
is
you
generate
a
certificate?
A
It's
it
lasts
like
five
minutes
or
so
to
sign
your
things
and
then
that
certificate
expires
and
the
keys
are
now
useless
because
the
certificate
is
expired
and
then
you
can
those
well
you
can.
You
can
always
verify
the
the
the
the
images,
because
the
certificate
and
its
public
key
are
stored
in
a
public
transparency
law.
So
there's
no
key
management,
and
that's
that's
all,
and
also
the
way
you
you
you
add.
A
E
Okay,
so,
and
probably
I
didn't
get
the
the
last
part
because
and
if
you
are
creating
ephemeral
keys,
you
need
to
somehow
sign
those
as
well.
In
order
to
be
and
trusted
right,
I
mean
we
are.
You
are
creating
a
framework
in
order
to
sign
that
artifacts,
but
those
affirmatives
that
you
have
been
using
for
those
process
should
be
verified
as
well
that
it's
coming
from
the
right
source,
yeah.
A
Yes,
yeah,
so
all
of
the
all
of
the
signatures
and
and
the
operations
that
happen
with
the
six
or
tooling
gets
stored
in
the
in
the
transparency
log.
So
when
you
receive
an
image,
you
get
the
the
signature
so
that
the
signature
has
the
metadata
necessary
to
go
back
to
the
to
the
transparency
log
and
see
who
who
gave
the
identity
to
the
center
and
who
and
if
it
well
the
the
certificate
to
verify
the
the
artifacts.
F
F
I'm
really
excited
to
see
this
work
because
it
aligns
with
a
lot
of
the
discussions
that
we've
had
within
the
cncf
security
tag.
So
my
question
for
you
is:
can
you
describe
what
kind
of
constraints
or
considerations
you've
put
in
place
for
the
actual
build
environment
itself
and
what
those
images
are
and
what
they
contain?
Whether
or
not
you
have
additional
packages
built
into
them
and
what
those
signs
look
like.
A
So
the
the
the
kubernetes
release
consists
of
about
50
images
for
all
architectures
and
all
of
the
kubernetes
components.
So
there's
one
for
the
api
one
for
the
proxy
one,
all
of
those,
and
we
also
produced
the
the
binaries
for
for
cube,
ctl,
cube
avm.
I
think
it
gets
released
with
the
the
binaries.
F
What
are
in
your
build?
What
are
you
using
in
your
build
host,
those
build
environments
themselves?
What
constraints
do
you
have
on
them,
such
as,
obviously
you
don't
want
anybody
ssh
into
those
boxes,
while
you're
doing
the
build.
So
what
what
do
you
have
installed?
What
do
you
not
have
purposefully
installed?
Okay,.
A
Yeah,
so
the
way
it
works
is
this
whole
world.
All
kubernetes
has
a
set
of
images
that
are
the
build
environment
for
kubernetes
and
for
other
projects.
This
image
is
called
gates
cloud
builder,
if
I
don't
remember
correctly,
and
it's
an
image
that
we
in
sig
release
produce
with
all
of
the
tools
that
we
wanted
want
them
to
to
that.
We
need
during
the
build
process.
This
image
is
also
built
and
signed
by
us,
and
this.
This
is
the
environment.
That
image
creates
the
environment
to
run
the
wheels.
A
The
way
the
image
promoter
works
is
that
you
stage
the
images
or
files,
and
then
you
open
a
pr,
adding
those
files
to
a
manifest,
and
then
the
community
can
go
and
authorize
the
promotion
of
those
images.
So
someone
opens
a
pr
and
it's
the
the
actual
owners
of
the
code
that
get
the
authority
to
approve
those
and
after
it
gets
approved,
a
post,
submit
job
triggers
a
promoter
and
and
then
they
it
well
kicks
off
the
job.
To
promote,
I
don't
know
if
that
cannot.
D
You
are
also
building
the
build-up.
Is
that
potentially,
like
a
kind
of
vector
like
in
our
own
company?
We
are
worried
about
the
same
thing
where,
if
the
ci
team,
who
run
the
runners,
if
they
are
compromised
and
the
the
promotion
tool,
is
then
compromised,
would
it
be
better
to
have
the
build
be
done
by
a
separate
team?
Because
then
you
can't
you'd
have
to
get
collusion,
then
right
to
get
the
compromisation
yeah.
A
Absolutely
so
the
question
is:
if
you're
building
the
builder
within
the
same
process,
can
you
trust
it?
Well,
I
would
say:
never
trust,
that's
the
first
one,
but
the
second
is
yeah.
There
is
a
that
is
one
that
may
be
concerning.
We
build
it
in
a
separate
process,
so
it
runs
in
a
different
step.
So
it's
kind
of
isolated,
but
it
would
be
better
to
ensure
that
we
are
using
a
previous
build.
A
I
mean
just
to
ensure
that,
because,
if
you
are
always
pulling
the
latest
version
of
the
code,
you
don't
know
what
got
merged
right,
but
if
you
know
that
you
have
a
known
good
build,
you
can
simply
use
that
one.
That's
a
point
for
improvement.
I
can't
help
it
if
you
want
to
merge
that
it's
I'm
happy
to
to
to
guide
you
and
but
yeah
yeah.
Ideally,
the
builder
should
be
already
built
and
trusted
before
you.
You
can
get.