►
From YouTube: Sigstore Or: How We Learned to Stop Trusting Registries and Love Sig... Wojciech Kocjan & Tyson Kamp
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2023 in Amsterdam, The Netherlands from April 17-21. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Sigstore Or: How We Learned to Stop Trusting Registries and Love Signatures - Wojciech Kocjan & Tyson Kamp, InfluxData
Presentation talks about how InfluxData added signing of container images to its SaaS offering that uses around 100 different container images, is deployed on dozens of Kubernetes clusters in all major clouds. It shows the process from the perspective of DevOps and security teams.
It starts off by answering the important questions - “why are we doing it?” and “what would we get when this is done?”.
Session covers the roadmap InfluxData has taken to move from not signing any images, having partial checks in place to all critical workloads requiring signed images.
The SaaS offering consists of over 50 microservices, whose images are built multiple times a day via CD/CD.
It also uses open-source images by other teams inside the company as well as images provided by other companies.
The session provides details as to how each group differs and gets signed.
Presentation gives technical details on some aspects of the implementation - i.e. adding secure signing of container images in multiple CI/CD systems, key management.
It shows plans for reacting to security issues with images - from regular key rotation to getting all image signatures updated and invalidating older public keys.
A
So
we're
the
only
ones
between
you
and
lunch
so
we're
trying
to
make
this
quick
so
right,
so
the
talk
is
sister
and
I
really
love
to
stop
trusting
registration
trust
Resident
of
signatures.
The
title
was
mostly
based
on
Dr
Strange
Love.
For
instance,
I
was
like
this
a
bit
off.
We
also
try
to
use
Dali
for
generating
all
the
images
just
to
make
it
a
bit
more
intriguing,
interesting.
We
hope
you
like
it
right.
So,
let's
start
by
introducing
ourselves.
B
A
Right
and
I'm
a
platform
engineer
at
inflex
data,
so
I
was
working
with
Tyson
more
on
the
technical
aspect
of
this.
This
is
my
GitHub
and
Linkedin
profile
as
well.
There's
multiple
veggie
questions,
I
guess:
I,
don't
know
if
we
all
have
hair,
but
it's
the
link
is
you
should
be
able
to
find
us
me
right.
So,
let's
quickly
introduce
the
inboxdate
in
fixed
DB.
So
we
come
from
a
company.
B
B
Thanks,
okay,
so
some
other
stuff,
so
so
high
level.
What
were
we
trying
to
do
so
number
one?
The
first
bullet
point
is
the
one
I
just
said
number
two
one
thing:
I
I'm
a
Dev
in
my
bones
and
so
I
when
I
hear
c
key
rotation.
It
just
kind
of
makes
me
cringe
so
I
want
it
to
be
really
quick
and
really
trivial
and
as
easy
as
possible
and
you'll
see
how
we
did
that
later.
B
We
have
many
many
clusters
also,
so
this
had
to
scale
and
be
really
simple
and
the
last
one
is
kind
of
a
repeat
of
the
second
bullet
point.
Let's
give
that
one,
okay,
so
non-goals,
so
I
did
my
best
to
solicit
as
many
different
groups
around
influx
as
I
could
because
I've
only
been
there
for
a
year
for
one
but
number
two
I
want
as
many
opinions
as
I
can
I,
certainly
can't
think
of
every
everything,
and
so,
when
you
start
talking
to
people
about
security,
you
probably
all
know
this
the.
B
What
if
questions
start
happening,
really
quick
and
what?
If
what?
If,
what?
If
and
so
while
getting
the
information,
I
was
also
reminding
people,
we
don't
trust
the
registry.
That's
the
only
problem
that
we're
trying
to
solve
here
right
now
that
we'll
deal
with
the
rest
later,
so
that
was
number
one
and
number
two.
We
were
getting
pretty
jazzed
about
all
the
stuff
that
Sig
store
has
to
offer.
So
we
had
to
sort
of
contain
ourselves
and
do
what
I
just
said
just
keep
it
to
register
and
Trust
still
me,
okay,
so
why
don't?
B
We
should
put
my
name
on
it.
So
when
are
we
done
when
every
single
container
that
we
have
is
being
signed
and
being
verified
in
the
Clusters?
That's
when
we're
done
and
from
my
experience
from
other
domains
outside
of
cyber
and
outside
of
engineering,
I'm,
a
big
believer
in
training
and
being
able
to
perform
under
stress,
and
so
our
SRE
team
and
people
that
might
have
to
handle
the
breach
when
it
happens,
not
if
it
happens,
I
need
to
know
that
it's
easy
for
them.
B
It's
really
really
easy
and
it's
quick
and
that
they're
trained
in
doing
it.
So
that's
another
definition
of
done.
This
can't
be
something
that
voycheck
and
I
think
up
and
it
looks
neat
and
we
have
some
diagrams
and
sock
it
away
somewhere,
and
then
we
move
on
and
18
months
later
we're
gone.
Somebody
has
to
be
able
to
handle
this
really
quickly
and
really
easily
when
there's
a
breach,
so
that
was
a
that
was
a
big
criteria
and
number
three.
Our
Dev
teams
need
to
know
how
to
how
to
onboard
really
easily.
A
A
bit
more
details
about
implex,
DB
Cloud,
just
enough
so
I,
understand
what
the
problem
was
that
we
were
trying
to
solve
so
it
as
I
mentioned,
it's
kubernetes
based
it's
partially
stateful.
Well,
I
guess
any
database
has
to
be
stable
at
some
places,
so
most
of
the
workloads
are
stateless
microservices.
A
So,
for
example,
when
we
get
an
API
call,
there's
a
lot
of
micro
services
that
pass
it
out,
call
it
call
it
out
and
then
talk
to
the
to
the
storage
there,
and
this
is
where
we
get
to
the
to
the
interesting
bit
of
inflex
data.
So
we
have
a
storage
here,
that's
basically
using
PVCs
or
volumes
at
the
kubernetes
level,
with
the
data
being
also
persisted
at
the
cloud
native
objects,
also
like
first
three
gold
star,
depending
on
on
the
cloud
it's
running
on,
we're
also
using
managed
databases
for
the
metadata.
A
So
anything
like
the
users
organizations
we
try
to
put
it
in
SQL,
some
of
the
things
that
are
in
different
places
as
well,
but
most
of
it
is
in
SQL
and
kafka's
will
keep
better
for
the
rider
headlock.
So
when
someone
starts
writing
data,
it
could
start
there
first
and
then
gets
right
into
the
proper
storage
there
and
the
thing
that
we're
really
happy
with
is
that
with
because
this
has
been
built
as
Cloud
native
from
day
one.
It's
it's
a
fully
automated
pipeline,
we're
using
infrastructures,
Code
github's
property
ICT
in
place.
A
I
think
this
is
also
similar
to
previous
talks.
We
have
the
concept
of
phases
so
anything
that
we
want
to
introduce.
First
gets
run
in
in
the
staging
or
testing
environments
that
don't
really
have
any
external
customers
using
it.
Once
all
the
post
deployment
tests
pass
and
everything
looks
properly,
then
that's
gets
deployed
into
production,
so
there's
that
integrated
in
our
pipelines
as
well
and
again,
I
think
we're
on
all
three
major
clouds
and
in
multiple
regions.
There's
a
lot
of
a
lot
of
deployments
happening
a
lot
of
things
changing
all
the
time.
A
It's
it's
actively
developed.
So
we
have
multiple
well
I,
guess
a
large
number
of
deployments
per
day
have
no
idea
what
the
numbers
are.
It's
it's
happening
all
the
time.
Basically,
so
we
have
multiple
sources
of
images
and
they're
treated
differently,
so
we
have
like.
We
have
CI,
that's
building
the
code
that
we've
written
specifically
for
NCLEX
DB
cloud
and
that's
a
lot
of
code
and
I
guess.
A
A
lot
of
the
images
we
run
is
is
that
so
with
that
with
those
images,
It's
relatively
simple
because
they
get
built,
they
get
signed
or
we'll
get
to
the
details
on
how
they're
assigned
later,
but
that's
just
the
subset
of
images
we
use.
In
fact,
data
also
provides
some
open
source
called
open
source
code
and
open
source
based
containers.
A
That's
that's
actually
running
some
of
the
logic
in
cluster
same
with,
for
example,
Telegraph.
We
we
try
to
follow
the
major
releases
of
telegraph,
but
it
doesn't
mean
we
we
update
them
the
same
day,
so
we
basically
sign
them
periodically,
meaning
that
we
have
a
list
of
images
and
those
are
specific
shots
that
we
that
we
decided
we're
going
to
trust
and
they
get
they
get
signed
periodically
and
maybe
I'll
get
to
the
details
of
this.
So
this
is.
A
This
is
a
diagram
of
how
it
looks
like
so
we
have
a
cluster
kubernetes
cluster
that
has
all
of
the
logic
in
it.
So
we
have
bold
same
as
Ethan
from
mentioning
data
they're,
also
using
Vault
and
key
versions
involved
with
the
transit
key,
and
we
use
that
we
use
that
to
generate
the
signatures.
Then
we
build
up
just
a
service
on
top
of
cosines
code,
so
I
we
had
to
extract
some
of
it
away
from
the
CLI
into
our
library.
A
So
in
most
CI
systems
it's
possible
to
change
the
CI
Logic
on
a
branch,
and
we
wanted
to
avoid
a
common
Pitfall
that
that
happens
in
CI.
That,
if
you
have
the
secrets
for
signing
the
images
and
if
someone
really
wants
to
do
something
nasty
and
they
have
access
to
git
repository,
they
would
be
able
to
to
change
the
CI
logic
in.
A
So
for
both
of
these
cases,
the
the
signatures
land
in
the
in
the
image
Registries,
one
thing
I
didn't
mention
previously
for
our
images.
We
just
store
signatures
alongside
the
images
in
the
same
in
the
same
registry
for
everything
else,
so
like
Telegraph,
Argo,
exec
and
all
the
other
images
that
we
don't
control.
Specifically
for
this
project.
We
have
a
dedicated
external
signatures
register,
which
is
something
cosine
and
policycontroller
support.
A
So
we
just
basically
sign
it
and
put
just
the
signatures
in
another
location,
but
regardless,
if
we
use
tax,
shot,
digest
or
anything
else,
depending
on
the
tool
and
whether
the
tool
provides
ability
to
to
force
the
shots
to
be
in
use,
we
signed
a
very
specific
shots
of
those
images
and
we
keep
those
signatures
in
our
in
our
own
in
our
own
registry.
So
we
so
we
manage
that
and
every
single
cluster
that
runs
our
product.
What
happens?
A
A
So
we
keep
on
rotating
the
keys
as
a
station
and
make
it
make
it
simple
and
transparent
for
everybody.
That's
that's
using
the
system.
We
have
a
thin
client
in
the
on
the
CI
side
that
could
be
integrated.
That
would
be
calling
our
image
signing
service
and
then
in
each
of
the
Clusters.
We
have
systems
that
make
sure
that
all
the
Clusters
are
up
to
date
set
of
public
keys
and
because
of
that,
we
didn't
have
to
set
up
an
ECA
CA
certificates
and
because
most
of
the
images
we
use
are
just
internal.
A
We
didn't
want
to
specifically
use
all
of
the
like
all
of
the
solutions
that
we
include
transparent,
log
and
and
keyless
signing.
We
wanted
to
just
choose
a
frequent
key
rotation
and
just
using
key
based
signatures
with
with
Vault,
providing
that
and
with
all
their
nice
upside
is
as
well
that
the
private
key
Never
released
fault.
It's
just
it's
just
used
for
signing,
so
it's
it's
relatively
secure
as
for
adding
signatures.
A
So
what?
When
Tyson
and
I
were
task
with
this,
we
didn't
really
know
a
lot
about
six
thread
at
this
point
in
time.
So
we
and
we
started
about
looking
at
the
at
the
available
tooling.
The
obvious
choice
was
cosine,
because
that
that
seemed
like
a
like
something
that
was
well
established
and
standard.
A
We
decided
not
to
go
with
Fulton
recoil
just
because
we
wanted
to
have
a
simple
solution
based
on
frequent
key
rotation,
and
we
also
didn't
want
to
make
some
of
the
metadata
about
our
images
public
as
they're
just
private,
and
we
don't
provide
those
images
to
customers.
We
we
run
our
own
product
in
our
own
cluster,
so
it
just
seemed
easier.
We
started
getting
involved
in
in
sixth
round.
We
hope
to
get
more
involved.
My
goal
is
to
at
least
contribute
something
back
as
soon
as
possible,
maybe
even
in
the
next
weeks.
A
So
we
started
small
by
just
deploying
this
alongside
experimenting
manually,
then
we
switched
to
Vault
just
in
Dev
mode,
but
it
was
enough
to
play
with
key
rotation
play
with
all
the
aspects
of
the
of
the
system
that
we
wanted
to
build,
and
then
we
moved
into
scaling
out.
So
we
we
used
volt
that
we
already
used
for
a
lot
of
other
things
in
our
infrastructure
and
just
switch
to
the
proper
Productions
that
have
evolved.
A
We
spent
some
time
thinking
about
the
transplant
reasons.
As
I
mentioned,
we
decided
to
build
a
dedicated
image
signing
service.
One
spring
is
it's
using
call
science
logic
internally,
it's
just
it's
just
exposed
as
an
endpoint
that
we
can
call
from
the
CI.
So
one
reason
is
again
because
we
can
do
additional
validation
of
whether
we're
actually
signing
the
image
that
that
we
want
to
be
running
in
production.
A
The
other
thing
was
that,
because
of
that,
CI
CI
systems
do
not
have
access
to
the
to
the
private
keys.
They
just
have
access
to
keys,
to
talk
to
the
image
science
service,
so
this
this
makes
it.
This
makes
it
better
that
that
key
rotation
doesn't
mean
that
we
have
to
rotate
it
in
all
the
sea
ice
because
we
don't
have
to
do.
A
We
just
take
those
those
keys
used
to
talking
to
the
image
sign
service,
but
I
guess
Tyson
also
mentioned
more
about
the
plans
for
for
how
we
would
handle
that
in
case
of
a
security
incident
and
again
we
we
also
have
some
additional
Logic
for
checking
whether
the
image
should
be
signed
inside
image
sign
service.
So
the
CI
for
one
type
of
images,
calls
and
says
I
want
to
sign
different
types
of
images.
A
Image
sign
servers
may
just
say:
I'm
not
going
to
sign
it,
and
this
provides
additional
layer
of
security
in
terms
of
policy
controller.
We
we
had
to
do
some
tweaks
and
do
some
settings
such
changes,
and
we
also
opened
some
some
issues
and,
like
I,
said
we're
hoping
to
contribute
back
to
policy
control.
B
Sure,
okay,
so
getting
like
I
said,
key
rotation
is
really
important,
so
important
and
so
easy
I
wanted
it
that
Nobody
Does
It
cron.
Does
it
it's
it's
handled
by
config
variables
and
it
doesn't
even
have
to
happen
and
it's
happening
all
the
time
we
basically
have
dials
to
say
how
quickly
do
you
want
to
deprecate
the
Keys?
How
often
do
you
want
to
create
new
ones,
so
you
can
create
them
twice
a
day.
B
You
can
create
them
once
every
two
weeks
you
can
say,
throw
them
away
after
a
month
throw
them
away
after
a
week,
it's
just
in
git
Ops.
It's
that
easy!
That's
how
easy
I
want
my
life
to
be
integration
with
the
Automation
and
processes
that
was
some
heavy
lifting
to
me.
Voice
check
mostly
handled
that,
but
that
just
I'll
just
say
make
make
sure
you
set
some
time
aside
for
that.
That
was
difficult
and
again
just
to
highlight.
B
I
tried
to
work
as
cross-functionally
as
possible,
so
everybody's
cool
with
this,
especially
the
people
that
carry
the
pagers
you
know
they
have
to
I,
don't
want
to
see
any
concern
on
their
face
when
I'm
rolling
this
out,
because
I'm
really
usable
security
is
really
important
to
me,
like
people
find
workarounds
really
fast.
If
they,
if
it
seems
like
another
hoop,
they
have
to
jump
through
something
that
you
did
so
so
working
with
other
teams.
It's
the
dev
teams,
but
really
the
SRE
team
and
the
people
Downstream.
B
You
watch
Argo
CD
for
everything
to
deploy,
check
out
the
file
change
the
flag
back
check
it
in
that's
it.
So
that's
what
I
wanted
I
wanted.
This
super
easy
like
solution
for
the
for
the
SRE
team
and
that's
it
I
can
explain
it
to
our
CFO
and
I
can
tell
them
why
we're
doing
it
and
they
love
to
hear
that
kind
of
stuff.
So
that
was
that
was
a
big
concern
of
mine.
So
that's
about
it!
B
B
Using
what
furkey
rotation
just
more
than
we
needed
to
do
for
our
scenario?
That's
it
it's
just
more
stuff.
You
know
what
I
mean,
so
any
introduction
of
the
complexity
has
to
have
a
really
huge
upside,
and
so
we
we
could
we
may
you
know
we
may
do
that,
but
right
now
this
is
involved
in
setting
up
almost
nothing
else.
It
was
really.
B
It's
got
to
be
really
easy
and
it
can't
leave
a
big
footprint
because
I've
seen
a
lot
of
I've
seen
a
lot
of
people
come
and
go
and
build
systems
and
they're
gone
and
then
and
then
we're
trying
to
figure
out
how
those
systems
work
and
there's
a
lot
of
smart
people
and
a
lot
of
stuff
being
done
at
influxes,
so
we're
really
sensitive
to
like
exactly.
Why
do
you
want
to
do
this?
So
we
have
a
lot
of
those
tug
of
wars.
B
A
And
just
to
share
some
other
interesting
issue,
we're
running
into
sometimes
so
rotating
the
case.
But
the
validity
of
the
key
is
relatively
long,
because
sometimes
we
see
Engineers
having
to
roll
back
specific
services
or
previous
version
because,
for
example,
of
incidents,
so
we
it's
happening
I,
wouldn't
say
on
a
daily
basis,
but
we've
seen
issues
where,
for
example,
someone
introduced
a
refactoring
of
a
piece
of
code
and
then
for
a
specific
customer
for
a
specific
cluster.
A
That's
used
in
a
in
a
non-standard
way
that
that
generates
some
issues
so
we're
seeing
people
there
is
a
there
is
a
relatively
easy
mechanism
to
what
we
call
pin
an
image
which
means,
instead
of
just
rolling
just
the
latest
one
in
a
specific
location
for
a
specific
microservice
people.
Do
that
and
okay
I
suspect,
with
David's
also
possible,
but
like
that
was
one
of
the
things
we
didn't
want
to
make
more
difficult.
A
So
there
are
procedures
in
place
to
to
do
that
if
you
need
to
use,
make
an
exception
use
code,
that's
on
a
branch
because
you
have
a
fixed
for
an
incident
as
well.
There
are
mechanisms
to
do
that.
To
do
that,
that
is
not
as
simple
but
rolling
back
to
an
image
from
two
days
ago
is
something
you
want
to
do.
Rolling
back
to
an
image
from
six
months
ago
is
something
you
want
to
prevent,
because
that
that
one
may
have
vulnerabilities
in
there.
B
Yeah,
that's
kind
of
bookended
with
the
replay
attack
scenario
where
we
want
to
keep
the
window
of
public
keys
that
are
accepted
as
small
as
possible.
That's
the
replay
attack
says
well.
Why
don't
you
do
it
for
an
hour?
You
know
like
a
two-hour
key
is,
is
deprecated
and
we
haven't
really
figured
out
where
the
com,
the
middle
ground,
is
right.
Now,
yeah
cool,
that's
it
all
right.
Let's
eat.