►
From YouTube: Who's Verifying Your Signatures? Approaching Private Container Image Signing - Ethan Lowman, Datadog
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2023 in Amsterdam, The Netherlands from April 17-21. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Who's Verifying Your Signatures? Approaching Private Container Image Signing - Ethan Lowman, Datadog
By some estimates, the rate of software supply chain security attacks has more than doubled in recent years, leading to renewed demand for software integrity defenses, especially for popular open source projects. In response to this demand, healthy competition has emerged between signing technologies like Sigstore and Notary v2 to set a new standard for secure delivery of open source container images. But how do these technologies fare when applied to private container image signing? While building integrity controls for their internal Kubernetes software supply chain, Datadog's security team has found that signing and verifying images internally is subtly different than in an open source setting. This talk will compare the unique challenges of signing container images internally versus in open source, and discuss how the leading open source signing frameworks meet those challenges at scale.
A
Hello,
it's
great
to
be
joining
Sig
storecon
2022.
My
name
is
Ethan
lohmann
and
the
title
of
this
talk
is
who's
verifying
your
signatures
approaching
private
container
image.
Signing
in
this
talk,
I'll
discuss
what
we've
learned
at
datadog
as
we've
begun
signing
our
internal
container
images
datadog
in
case
you're
not
familiar
is
an
observability
and
security
platform
collecting
tens
of
trillions
of
events
per
day
from
millions
of
hosts
to
provide
this
service.
We
run
a
lot
of
container
images
internally.
A
Today,
I'm
here
representing
datadog
I'm,
a
senior
software
engineer
on
a
team
called
software
integrity
and
Trust,
where
our
mission
is
to
secure
our
internal
and
customer
facing
software
Supply
chains
to
support
some
of
this
work
while
contributing
back
to
open
source
I'm.
Also
one
of
the
maintainers
of
the
go
tough
Library,
which
is
a
go
implementation
of
the
update
framework.
A
A
A
To
begin,
why
should
you
sign
and
verify
container
images
internally,
just
like
any
open
source
setting
verifying
an
image?
Signature
allows
you
to
have
a
high
confidence
that
the
image
comes
from
a
trusted
Source
such
as
your
Ci
or
a
build
environment,
and
that
it
has
not
been
modified
before
it
runs
here.
I
have
a
very
basic
model
for
an
internal
software
supply
chain,
targeting
a
kubernetes
runtime
first
developers
write
code
and
push
that
code
along
with
its
dependencies
to
our
source
code
management
platform
like
GitHub.
A
A
These
workloads
are
managed
by
the
kubernetes
control
plane
and
are
eventually
scheduled
as
pods
and
kubernetes
nodes,
where
the
container
images
we've
built
are
finally
pulled
and
run
internal
software.
Supply
chains
like
this
one
are
composed
of
many
parts
and
your
trusts
in
each
of
these
parts
may
vary.
A
For
example,
some
parts
may
be
more
exposed
to
human
operator
access
which
can
be
a
source
of
weakness
in
the
supply
chain.
If
an
employee
laptop
is
compromised,
citing
and
verifying
images
in
your
internal
software
supply
chain
allows
you
to
reduce
your
trust
in
each
of
these
software
delivery
components.
So
your
production,
infrastructure
and
data
is
not
automatically
at
risk.
If
one
component
of
your
delivery
pipeline
is
compromised,
for
example,
if
you're
signing
container
images
you're
substantially
protected
against
compromise
of
the
container
registry.
A
This
signature
then
represents
an
attestation
of
provenance
authenticating
that
the
image
came
from
CI,
then
every
time
the
image
is
used
or
referenced
later
on
in
the
software
supply
chain,
verifying
the
image
signature
allows
you
to
reduce
trust
in
each
of
the
components
in
between
the
general
idea.
Is
that
the
closer
to
run
time
you
verify
image
signatures
the
fewer
components
of
your
software
supply
chain?
You
need
to
fully
trust.
A
Now,
once
you've
decided
to
sign
and
verify
images
to
protect
your
build
and
release
pipelines,
what
tools
are
out
there
to
start
signing
six
stores.
Cosine
tool
is,
of
course,
one
option,
but
I
think
it's
useful
to
understand
the
context
around
Sig
store,
so
you
can
make
an
informed
choice
about
how
to
sign
images
for
your
particular
situation.
A
A
A
A
A
This
means
that
it's
not
easy
to
copy
an
image
and
its
signature
between
Registries
and
how
the
signature
still
be
valid
in
the
new
registry,
because
using
Docker
content
trust
requires
the
notary
service.
In
addition
to
the
registry
itself
and
notary
wasn't
implemented
by
most
Cloud
registries,
you're
also
severely
restricted
with
what
registries
you
can
use
to
host
your
signed
images
for
our
internal
image
signing
use
case
at
datadog.
This
is
the
main
reason
why
notary
V1
wouldn't
make
sense
for
us
in
order
to
deploy
to
multiple
independent
data
centers
each
with
many
kubernetes
clusters.
A
We
may
build
inside
an
image
in
just
one
registry,
but
to
support
verifying
that
image
in
every
data
center.
It
runs.
We
need
to
be
able
to
verify
the
image
from
any
registry
it's
copied
to
so
for
datadog
ease
of
signature.
Replication
is
one
of
the
top
considerations
for
how
we
sign
images
and
notary
V1
does
not
have
good
support
for
this.
A
A
It
aims
to
tackle
more
than
just
container
image.
Signing
graphase
is
an
artifact
metadata
API
that
represents
structured
events
that
occur
throughout
the
software
development
life
cycle.
These
events
could
be
things
like
the
result
of
a
vulnerability
scan
for
a
container
image
or
at
a
station
that
a
build
artifact
came
from
a
trusted
system.
A
A
A
If
you're
releasing
images
publicly
and
your
customers
will
be
verifying
your
signatures
on
their
own
infrastructure.
Graphase
is
probably
not
the
right
choice.
It
doesn't
provide
any
mechanism
for
public
key
Discovery
and,
as
far
as
I'm
aware,
the
API
is
not
meant
to
be
exposed
outside
of
internal
infrastructure.
A
A
The
notary,
V2
project
is
an
evolution
of
notary
V1,
which
began
in
2019,
aiming
to
fix
the
issues
discovered
in
the
docker
content.
Trust
rollout,
notary
V2,
supports
signing
images
using
x509
based
pki.
The
signature
metadata
is
stored
in
the
registry.
Alongside
the
signed
images
to
link
signature
artifacts
with
the
images
they
sign,
notary
V2
uses
a
new
part
of
the
oci
spec
called
refers.
A
However,
because
the
referrer's
API
is
so
new,
it
lacks
widespread
registry
support.
Notary
V2
is
still
in
the
alpha
stages
and
it's
looking
promising,
but
it's
not
production
ready.
Yet
so,
what's
standing
in
the
way
of
notary
V2
in
production,
echoing
the
theme
of
this
whole
talk
that
really
depends
on
the
verification
context.
A
However,
there's
not
an
off-the-shelf
solution,
I'm
aware
of
to
verify
notary
V2
signatures
within
an
environment
like
kubernetes,
so
this
is
something
that
still
needs
to
be
implemented
before
we
get
to
talking
about
cosine
and
the
rest
of
sixstore.
I
want
to
quickly
mention
how
the
update
framework
fits
into
this
picture.
A
A
Tough
snapshot
feature
which
is
meant
to
make
sure
a
client
has
complete
and
up-to-date
set
of
trusted
files
and
no
more
doesn't
easily
apply
its
image.
Registries
I
won't
go
into
too
much
detail
here,
but
we
found
that
snapshotting
brought
a
lot
of
complexity
that
didn't
deliver
concrete
value.
We
can
map
to
our
internal
registry
threat
model.
A
A
A
Cosine
Loosely
couples
the
image
and
its
signature
using
a
predictable
tag,
naming
pattern
which
gives
it
broad
registry
support.
Signature
sets
itself
apart
from
the
competition
and
open
source
image
signing
by
including
an
easy
to
use
certificate.
Authority
called
falsio
that
issues
short-lived
certificates
to
signers,
based
on
their
oidc
identity.
A
This
means
that
without
any
of
your
own
key
management,
you
can
use
a
short-lived
certificate
that
is
cryptographically
tied
to
a
CI
job
on
the
verifier
side.
This
also
lends
itself
really
well
to
public
key
Discovery,
since,
if
you
trust
six
Source
routes,
then
you
just
have
to
verify
an
email
address.
Instead
of
keeping
track
of
public
keys
for
each
software
publisher,
these
components
will
be
covered
in
more
detail
throughout
the
conference
today,
but
together,
full
Co
and
recore
make
cosine
a
really
ergonomic
choice
for
signing
and
verifying
open
source
images.
A
However,
it's
a
little
more
nuanced.
If
you're
assigning
for
internal
verification
for
internal
signature,
verification
organizations
may
prefer
to
have
a
private
read
of
trust
rather
than
using
six
stores.
Additionally,
using
record
does
by
Design,
make
certain
information
about
your
usage
public,
such
as
how
many
artifacts
you're
signing.
A
This
may
not
be
a
problem
for
you,
but
if
it
is,
then
you'll
need
to
implement
your
own
key
management
for
cosine.
That
means
you'll
need
to
think
about
key
lifetimes
and
rotation.
Hyal
resign
images
already
running
on
your
infrastructure
to
keep
them
valid
and
how
you'll
secure
securely
distribute
public
keys
to
all
of
your
verification
clients.
A
So,
as
always,
there
are
trade-offs
and
if
you
do
decide
to
use
cosine
how
you
use,
it
will
largely
depend
on
who
or
what
services
you
expect
to
be
doing.
The
verification
and
it's
crucially
important
that
you
think
about
how
verification
will
work
and
that
you
make
it
easy,
because
an
image
signature,
that's
not
actually
verified,
is
of
little
value.
A
There
are
some
features
in
the
works
to
help
accomplish
this
like
something
called
six
door:
Trust
delegations
for
internal
image,
signing
which
has
been
our
primary
focus.
Our
approach
looks
very
different
when
we
began
this
project
earlier
this
year
we
decided
that
running,
folsio
and
recore
internally
was
hard
to
motivate
our
internal
service.
Identity
system
isn't
compatible
with
full
co-supported
identity
providers,
so
using
the
full
coca
internally
wouldn't
make
sense.
It's
easier
for
us
to
manage
trusted
Keys
directly,
additionally
running
a
transparent
log
like
recore
internally
didn't
make
sense
for
us
in
our
situation.
A
A
We
also
decided
that
running
a
timestamp
server
like
recore
internally,
didn't
make
sense
for
us.
This
is
because
the
design
of
timestamp
servers
makes
the
most
sense
when
they're
run
by
a
trusted.
Third
party,
truly
isolated
from
the
environment,
where
the
siding
happens
for
the
basic
image
signing
component.
We
also
opted
against
using
cosine
instead
we're
using
the
custom
signing
service,
which
I'll
explain
soon.
A
A
This
also
makes
integration
with
new
repos
and
updates
easier,
since
updating
image,
signing
clients
across
many
repos
would
be
a
headache
if
the
client
changed,
often
by
making
the
client
as
simple
as
possible.
Deferring
most
of
the
work
to
the
signing
service,
we
considerably
reduce
how
often
we'll
need
to
update
signing
clients
across
many
repos
to
implement
an
assigning
service.
We
could
have
potentially
used
cosine,
but
at
the
time
of
implementation,
cosine
was
primarily
a
CLI
tool,
not
meant
to
be
used
as
a
library.
A
The
second
requirement
for
image
signing,
which
is
a
little
more
subtle,
is
that
we
wanted
to
avoid
creating
new
image
tags
in
the
same
repository
as
the
signed
image.
This
is
because,
at
the
time,
our
internal
registry
retention
policies
were
based
on
the
number
of
tag
digests
for
each
repository.
So
if
we
sign
every
tag
that
would
double
the
rate
of
tech
churn,
but
it's
potentially
causing
images
to
be
prematurely
expired.
A
A
A
Instead,
we
plan
to
implement
runtime
image
verification
at
the
node
level
within
container
D,
which
is
the
container
runtime.
We
use
we're
currently
working
with
container
D
maintainers
to
merge
these
new
node
level
image
verification
features
Upstream,
it's
worth
noting
that
we're
only
able
to
customize
container
D
in
this
way,
because
we
manage
our
own
kubernetes
clusters.
A
A
A
A
First,
it
talks
to
the
registry
to
resolve
a
digest
for
that
image.
Container
D
uses
the
image
verifier
plugin
system,
we're
contributing
Upstream
to
ask
our
custom
verifier
plugin,
whether
the
image
digest
is
okay
to
run
our
plugin
pulls
the
replicated
signature
metadata
from
the
registry
and
verifies
the
image
signature
in
payload.
A
If
the
signature
is
valid
and
the
sign
digest
matches
the
one
in
the
request,
the
plugin
tells
containerd
to
finish
pulling
and
running
the
image
we
manage
key
rotation
through
both
built-in
key
versioning
ballot.
Public
keys
are
baked
into
the
machine
image
and
refreshed
from
a
secure
service
in
each
Data
Center.