►
From YouTube: Using Kubernetes Secrets in GitOps Workflows Securely - Seth Vargo & Alex Tcherniakhovski, Google
Description
Don’t miss out! Join us at our upcoming events: EnvoyCon Virtual on October 15 and KubeCon + CloudNativeCon North America 2020 Virtual from November 17-20. Learn more at https://kubecon.io. The conferences feature presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Using Kubernetes Secrets in GitOps Workflows Securely - Seth Vargo & Alex Tcherniakhovski, Google
Adopting GitOps for Kubernetes provides a single source of truth for cluster configuration and enables facilities like peer reviews and automated rollbacks. While many organizations store their Kubernetes configurations in git, Kubernetes Secrets are often managed via bespoke implementations outside of source control. Storing plaintext secrets in source, even in private repositories, is a horrible idea, but can we meet somewhere in the middle? In this session, attendees will learn how to securely store and manage Kubernetes Secrets in source control using Javascript Object Signing and Encryption (JOSE) and a Key Management Service (KMS). After this talk, attendees will be able to securely store and manage their Kubernetes Secrets in source the same way they manage their existing Kubernetes configurations.
https://sched.co/ZeiP
A
A
We
have
a
problem
and
that
problem
is
best
summarized
in
three
simple
words:
coupe
cuddle
create
secret,
that's
right
alex,
and
I
are
talking
about
kubernetes
secrets
again,
but
this
time
we're
not
talking
about
their
security
properties,
but
rather
focusing
on
the
secrets,
management
workflow.
Specifically,
how
do
you
get
your
secrets
in
and
out
of
kubernetes?
A
A
More
so
was
the
secret
even
tested
which
application
version
or
versions?
Is
it
known
to
work
with?
How
do
we
roll
back
in
the
event
of
failure
and
ultimately,
what
is
the
source
of
truth
for
this
secret
and
that's
really
what
we
want
to
zoom
in
on
today
in
this
talk
alex-
and
I
are
going
to
attempt
to
answer
all
of
those
questions
by
focusing
on
just
one,
what
is
the
source
of
truth
for
your
kubernetes
secrets?
A
So
this
begs
the
question
that
if
our
code
is
already
in
git
and
our
kubernetes
manifests
are
already
in
git.
Well,
why
not
put
our
secrets
in
git
and
now
I
know
what
you
might
be
thinking
to
yourself:
you're,
probably
thinking
that's
crazy,
you've,
given
so
many
talks
before
seth.
Why
would
you
ever
tell
us
to
put
plain
text
secrets
in
a
git
repository?
This
is
just
a
disaster
waiting
to
happen
and
you're
right,
but
nobody
said
anything
about
plain
text
secrets
and
that's
really
the
core
of
our
talk
here
today.
A
We're
going
to
be
focusing
on
workflows
that
enable
you
to
use
get
ops
without
exposing
the
plain
text
secrets
inside
your
repositories
and
then
before
we
jump
into
the
meat
of
it.
I
just
want
to
address
one
issue.
You
might
be
thinking
we
don't
use
git.
This
talk
is
totally
useless
for
me
and
that's
simply
not
the
case
while
alex-
and
I
are
using
git-
is
the
example
here.
These
concepts
are
not
specific
to
git,
they
would
work
with
svn
or
mercurial
or
perforce
or
any
other
source
control
technology
that
you
have
today.
A
The
core
of
our
design
is
rooted
in
asymmetric
cryptography,
so
let's
have
a
quick
overview
of
how
asymmetric
crypto
works
before
diving
in
even
more
with
asymmetric
cryptography.
We
generate
an
asymmetric
key
and
that
asymmetric
key
has
two
parts:
a
public
part
and
a
private
part
data
that
is
encrypted
with
the
public
key
can
only
be
decrypted
with
the
private
key.
A
We
chose
the
json
web
encryption
format
or
jwe,
but
other
message
standards
might
work
as
well.
We
picked
jwe
because
there's
an
ecosystem
of
tools,
including
the
popular
go
jose
library
for
kubernetes
that
revolve
around
jwe.
There's,
no
reason
you
couldn't
use
a
different
algorithm
or
standard,
but
that's
what
we're
focusing
on
here
today.
So
just
like
earlier,
we
said
you
know
if
you're
using
svn
replace
git
with
svn
when
we
say
it.
The
same
thing
is
true
with
jwe.
A
You
might
be
asking
yourself
why
even
use
jwe
at
all,
why
not
just
use
a
key
management
service
directly
to
encrypt
and
decrypt
this
data?
Well,
there's
two
main
reasons:
the
first
and
a
little
bit
more
practical
is
that
kubernetes
secrets
can
be
up
to
one
megabyte
in
size,
but
most
key
management
systems
restrict
data
to
64
kilobytes
in
size.
So,
by
talking
directly
to
the
kms,
we
would
be
severely
limiting
the
size
of
kubernetes
secrets
objects
on
the
security
front.
A
Moreover,
giving
the
secret
directly
to
a
key
management
system
may
be
undesirable
or
not
available
in
certain
situations,
depending
on
your
jurisdiction
or
organizational
policies.
So,
depending
on
your
trust
model,
it
may
not
be
acceptable
to
give
the
secret
directly
to
a
key
management
system
by
using
the
envelope
the
jwe
envelope.
We
are
giving
an
encryption
key
to
the
kms,
not
the
secret
itself.
A
A
Our
next
persona
is
the
secret
admin.
The
secret
admin
is
responsible
for
managing
sensitive
data
like
passwords
and
api
keys,
they're
also
unlikely
to
be
familiar
with
kubernetes
or
the
kubernetes
api,
and
our
third
persona
is
the
cluster
admin.
The
cluster
admin
is
responsible
for
deploying
managing
and
configuring
the
kubernetes
cluster
and
any
workloads
that
run
on
it.
Hopefully
they
are
familiar
with
kubernetes
tools,
since
that's
really
what
they
do
every
day.
A
A
A
A
A
Next,
our
secret
admin
uses
that
public
key
in
the
git
repository
to
create
a
jwe
envelope
with
the
secret
contents.
This
envelope
becomes
the
value
of
the
kubernetes
secret
file
or
the
value
that
goes
into
the
kubernetes,
manifest
again
just
like.
Before,
depending
on
your
setup,
you
can
add
more
customization
points
like
automation,
ci,
cd
and
approvals,
and
then
finally,
our
cluster
administrator
pulls
the
kubernetes
secret
file
or
the
manifest
that
contains
the
kubernetes
secret
with
the
encrypted
secret
from
the
repository
and
pushes
it
to
the
kubernetes
api
server.
A
A
Now
you
might
be
thinking
to
yourself,
seth,
didn't
you
and
alex
give
an
entire
talk
about
storing
secrets
in
plain
text
and
ncd,
and
to
that
I
would
say
yes
and
there'll
be
a
link
in
the
show
notes.
At
the
end,
we
don't
advocate
for
storing
secrets
in
plain
text
and
ncd.
You
should
use
application
layer
encryption
with
a
kms
plugin,
but
that's
a
totally
different
topic
and
a
talk
all
on
its
own,
so
alex.
Why
don't
you
show
us
a
little
bit
of
what
about
what
this
might
look
like
in
real
life.
B
B
We
believe
that
the
solution
we
are
proposing
could
be
adapted
to
a
wide
range
of
environments
in
both
cloud
and
on-prem,
for
this
walkthrough
we'll
be
using
google
cloud
platform
for
the
simple
reason
that
this
is
the
environment.
We
are
most
familiar
with
concretely
we'll
be
using
google
cloud
kms,
but
any
kms
that
supports
asymmetric
encryption
would
suffice.
B
B
B
Key
admins
must
choose
from
the
list
of
asymmetric
encryption
protocols
supported
by
the
jwe
standard
for
the
purposes
of
our
walkthrough.
We
picked
rsa
oap
256
suite,
since
it
is
supported
by
both
google
cloud
kms
and
gwe
standard.
However,
protocols
from
the
elliptic
curve
family
would
work
as
well.
B
B
B
B
B
So
this
is
how
our
secret
looks
like
in
the
repository.
This
is
a
standard
kubernetes
secret.
If
we
were
to
submit
the
secret
to
cube
api
server,
cube
apesor
would
happily
accept
it.
However,
in
this
current
form,
it
would
not
be
much
use
to
our
applications,
since
they
need
the
credential
itself
and
not
some
encrypted
envelope.
B
B
B
B
B
A
So
the
first
question
says
I
feel
like
sometimes
there's
a
chicken
and
egg
scenario
where
I'm
not
sure
if
an
encryption
key
for
a
kubernetes
secret
should
be
kept
in
a
vault
like
hashicorp
vault
for
this
kubernetes
cluster
and
at
the
same
time
I
want
to
use
the
key
manager
involved
for
my
application
level
secrets
as
well.
Should
I
reuse
the
same
key
management
system
or
should
I
have
a
whole
separate
key
management
system
dedicated
to
a
particular
namespace
alex?
What
are
you?
What
are
your
thoughts
on
separation
of
keys.
B
Well,
I
think
it's
a
very
actually
it's
a
very
interesting
question.
I
think.
Ultimately,
ultimately,
this
is
comes
down
to
this
first
secret
problem,
the
term
that
seth
coined
at
the
end
of
the
day.
No
matter
what
you
do,
you
will
end
up
with
some
sort
of
initial
secret
initial
key
that,
on
top
of
which
you
bootstrap
the
rest
of
your
keys
and
encryption
decryption.
B
A
Thanks
alex
so
the
next
question
is:
what
do
you
think
about
the
the
sealed
secrets
project?
I'm
assuming
that's
the
bitnami
I'll,
add
my
take
and
then
alex.
If
you
want
to
add
more,
I
think
the
key
difference
is
in
our
architecture.
We
really
wanted
to
assume
that
the
person
who
was
generating
these
passwords
or
certificates
in
these
third-party
systems
didn't
have
experience
with
kubernetes
and
the
same
would
be
true
of
the
the
key
administrator
right.
So
when
we
were
going
through
the
personas,
we
talked
pretty
heavily
about.
A
You
know
this
person
might
not
use
kubernetes.
They
might
not
be
familiar
with
the
kubernetes
api,
whereas
the
sealed
secrets
project
basically
automates
the
those
first
two
personas
that
we
discussed
into
kubernetes
itself.
So
it
does
require
that
the
person
creating
the
secrets
also
has
familiarity
with
kubernetes,
and
we
just
didn't
want
to
push
that
requirement
onto
people,
but
largely
they
use
like
the
same.
You
know
asymmetric
cryptography,
it's
really
just
about.
How
is
your
organization
set
up
and
and
where
do
your
expertise
lie.
B
Yeah,
absolutely,
I
would
just
only
add,
I
think
I
think
there
is
a
slight
advantage
of
what
seth
and
I
are
proposing,
is
because
bitnami
relies
on
creating
a
custom
object
for
storing
incorporated
information,
which
is
absolutely
nothing
wrong
with
that,
and
there
are
pros
of
doing
that.
But
what
we
demonstrated
is
just
basically
using
plain
kubernetes
secrets,
so
your
existing
application
can
be
completely
unaware
of
all
this
additional
machinery
that
was
introduced.
B
A
A
So
next
question
is:
how
do
you
ensure
that
the
secret
admin
encrypts
with
the
correct
public
key,
for
example?
How
do
you
ensure
that
this
isn't
an
attacker
that
has
replaced
the
public
key
with
their
own
so
that
they
can
easily
decrypt
the
secret
that
the
secret
admin
unknowingly
encrypted
with
the
attacker's
key
alex?
Do
you
wanna.
B
Right
right,
so
so
this
is
the
so
the
yeah.
This
is.
This
is
a
good
point.
I
I
think
what
needs
to
happen
here
is
that
basically,
the
secret
secret
administrator,
the
sorry,
the
secret
administrator
and
the
key
administrator,
we
need
to
do
another
sort
of
handshake
and
basically
the
secret.
The
key
administrator
would
need
to
ensure
that
the
key
administrator
is
aware
of
the
hash
of
the
public
key
that
they
need
to
use.
So
so
yeah,
that's
a
very,
very
good
point.
There
are
also
some
you
know.
B
Obviously
the
facilities
available
like
in
in
in
system
like
github,
where
you
can
you
can
apply
access
control
list
so
that
attackers
you
know,
can
just
simply
replace
the
public
key
without
compromising
github
security
system,
but
absolutely
right.
It's
it's
basically
like
with
using
any
public
key
cryptography.
Everything
comes
down
to
basically
trust
that
you
can
place
in
the
public
key,
so
yeah
there
isn't.
Maybe
we
should
have
made
it
more
clear
in
the
presentation
that
there's
a
little
bit
more
inter
interaction
between
the
key
admin
and
secret
admin.
A
Yeah
and
then
there's
probably
some
level
of
like
established
trust
there
right,
like
they're
communicating
over
a
secure
work
channel
or
they
know
one
another.
This
is
this
is
part
of
a
larger
classification
of
attack
called
inband,
key
negotiation,
and
it
applies
to
public
and
you
know,
symmetric
cryptography,
asymmetric
and
symmetric
cryptography.
B
B
B
A
Would
any
I
think,
question
number
six
and
question
number
seven
are
the
same.
It
says
I
understand
how
you
encrypt
and
store
passwords
inside
git
and
secret,
but
I
don't
understand
how
to
customize
by
example
like
using
a
helm
chart
to
be
able
to
use
it.
So
it's
I
think
it's
really
like
how
do
you
get
started.
B
A
B
Right
so
is
as
long
as
actually
I'm
not
too
familiar
with
helm,
unfortunately,
but
I'm
going
to
assume
that
somehow
helm
can
consume
a
kubernetes
secret
and
if
and
if
it,
if
it
is
true,
then
then
you're
already
done.
Our
solution
does
not
require
you
to
customize
your
existing
applications.
If
you
are
using
a
seeker
today,
you
you
can
continue.
Your
applications
do
not
need
to
change
in
any
way
shape
or
form.
B
So
I
I
think
the
the
challenge
here
is
around
basically
the
ensuring
that
the
timing
is
right
between
your
change,
your
your
password
or
rotate
your
secret
and
basically,
the
secret
being
developed
delivered
at
the
right
time
to
the
applications,
so
that
you,
basically
you
basically
don't
have
the
upper.
You
don't
have
the
situation
where
you
rotated
your
secret
on
the
system,
but
the
application
is
still
using
the
old
all
secret.
So
the
the
idea-
and
this
requires
a
little
bit
more
like.
B
Unfortunately,
we
won't
be
able
to
cover
this
in
detail,
but
we
believe
that
the
correct
approach
here
is
to
basically
provide
sort
of
a
backup
secret.
So
basically
because
it's
very
difficult
to
time
everything
to
the
point
where
application
just
starts
using
the
right
secret
at
the
right
time,
we're
basically
we're
looking
at
the
ideas
where
we
would
send
sort
of
the
old
secret
and
the
new
secret
through
you
know
your
git
workflow
and
the
application
will
basically
retry.
B
If,
if
you
know
like,
if
the
old
secret
fails,
then
we
just
basically
try
the
new
one.
So
so
that's
sort
of
our
current
thinking
about
this,
but
it
is
a.
It
is
not
a
simple
problem
and
there's
also
issues
around
like
how
would
the
secrets
secret
admin
basically
initiate
the
change
so
which
you
know
very
much
depends
on
the
kind
of
the
system
they're
using
etc,
but
but
the
idea
of
overlapping
secrets.
I
think
that's
kind
of
what
we
would
recommend
today.
A
B
B
So
I
think
the
decision
you
would
have
to
make
for
yourself
is
if
you
are
comfortable
with,
if
you
are
comfortable
with
externalizing
secrets
manager,
like
basically
not
storing
secrets
and
get
like.
Basically,
you
have
only
a
pointer
to
the
to
the
secret
and
basically
essentially,
if
you
are
interested
in
using
gcp
secrets,
I
think
the
right
approach
is
to
use
the
csi
csi
driver
which
into
like
there
is
a
maybe
we
should
add
a
link
to
the
to
the
slides.
B
A
Yeah,
I
would,
I
would
just
chime
in
like
there
are
some
architectural
differences
between
google,
secret
manager
and
kubernetes
secrets,
the
biggest
of
which
is
the
data
format
and
the
versioning.
So
by
default,
kubernetes
secrets
are
not
versioned,
whereas
secret
manager,
secrets
are
kubernetes,
secrets
are
key
value
pairs,
whereas
google,
secret
manager
secrets
are
just
a
slice
of
bytes
like
arbitrary
characters.
So
there
are
some
architectural
differences
but
to
alex's
point
about
like
externalization.
B
I
I
will
need
to
look
it
up.
It's
it's
probably
yeah.
I
don't
know
I'm
not
familiar
with
mozilla
subs.
We
did
explore
multiple
formats,
we
look
at
vrs,
encryption,
serialization
standards
and
essentially
the
reason
we
chose
gwe
is
because
gwe
is
extensively
used
in
kubernetes
within
kubernetes
itself,
like,
for
example,
we
use
it
for
encrypting
tokens
and
serializing
tokens.
So
it's
already
linked
a
lot
of
kubernetes
developers
already
familiar
with
this
framework
and
it
has
nice
gold
language
support.
So
that
was
the
primary
reason.
B
A
A
Another
way
I
could
interpret
this
question
is
why
not
use
vault's
transit
back-end,
which
is
the
key
management
system
built
into
vault,
like
alex
said
in
his
person
like
we're
using
cloud
kms
as
an
example,
but
you
can
use
any
key
management
system
that
supports
asymmetric
cryptography.
So,
like
there's,
no
reason
you
couldn't
use
vaults,
transit
back-end.
To
also
do
this.
Like
signing
operations,
we
just
chose
to
use
cloud
kms,
because
it's
what
we're
familiar
with.
B
B
So
obviously
you
need
some
sort
of
token
or
some
other
mechanism
for
for
webhook
to
authenticate
to
call
or
any
other
kms
system
for
that
matter,
and
that
may
represent
a
problem
like
if,
for
example,
if
you
dropped
a
secret
on
the
disk,
an
attacker
managed
to
compromise
the
disk
on
the
control
plane
of
kubernetes
control
plane.
We
were
presumably
running
that
web
hook.
Then
then
it
is
a
problem.
B
So
so
I
think
it
is
important
to
look
at
the
the
authentication
mechanism
between
the
cl
between
the
web
hook
and
cloud
kms,
and
I
alluded
in
my
presentation.
I
alluded
to
the
fact
that
we
are
recommending
to
use
underlying
security
mechanisms
provided
by
your,
for
example,
cloud
provider
or
your
virtualization
provider.
B
Like,
for
example,
I
showed
that
we
could
use
a
service
account
which
is
exposed
by
the
hypervisor
in
gke
or
sorry,
rather,
google,
google
compute
engine,
in
other
words,
that
token
that
allows
the
web
hook
to
authenticate
to
cloud
kms
is
only
available
in
memory,
and
it
is
only
available
when
the
the
vm
is
running,
in
other
words
like
attacker
would
actually
need
to
compromise
gain
root
access
on
the
on
the
control
plane
which,
by
that
time,
basically,
the
game
is
over
anyway.
So
in
other
words,
the
it's.
A
So
I
think,
we're
out
of
time
I'll
defer
to
the
moderators.
If
we
have
time
for
one
more
or
not,
they
said
sure,
okay,
so
in
the
scenario
described,
this
is
number
17..
The
cereal
describer.
The
secret
is
available
unencrypted
on
the
cluster,
which
is
a
weaker
position
compared
to
the
sidecar
in
it
container
approach
to
decrypting
at
runtime.
Can
you
contrast
this
over
runtime
decryption,
which
is
preferred
so
the
the
point
we're
making
is
that
in
this
world
the
secret
is
stored
in
plain
text
in
etsy
d.
A
So
what
we
would
recommend
is
that
you
go
watch
the
talk
that
alex
and
I
have
linked
on
the
slide
on
the
screen,
where
there's
actually
a
solution
by
which
we
can
do
application
layer
encryption.
So
the
secrets
are
never
in
plain
text
in
std,
and
in
that
case
the
only
difference
is
where
like
who
owns
the
secret.
Does
the
application
own
the
secret,
in
which
case
you
want
to
do,
run
time
decryption
or
does
kubernetes
own
the
secret?
A
A
The
kubernetes
api
server
owns
the
secret
and
you're,
relying
on
like
our
back
and
the
hackles
to
make
sure
that
only
the
right,
pods
and
services
have
access
to
it.
Whereas
if
you
do
run
time,
decryption
you're,
basically
saying
that
the
application
or
the
pod
owns
the
secret
and
you're
relying
on
something
like
workload,
identity
or
pod,
identity
to
authenticate
and
provide
that
first
secret
to
access
the
rest
of
the
secrets
alex.
Do
you
have
anything
to
add
or
no.
B
Absolutely
yeah,
no,
you
answered
it
perfectly.
Well
perfectly
correctly,
I
think
architecturally.
If
you
think
about
the.
I
believe
the
approach
suggested
in
the
question
is
to
to
delay
decryption
up
until
the
the
the
l.
The
last
point
possible
basically
keep
the
secret
encryption
until
the
last
basically
millisecond
before
it
actually
is
needed.
B
So
we
definitely
think
that's
a
that's
a
good
approach,
the
pro
the
and,
if
you
are
willing
to
change
your
applications
and
basically
introduce
a
site
basically
take
the
web
hook
like
the
go
into
the
github,
take
the
web
hook
and
basically
create
a
sidecar
out
of
this
like
using
the
same
logic.
If
you're
willing
to
make
that
change
to
the
application,
then
that
would
have
better
security
property.
I
think
because
of
the
delayed,
because
you
you're
basically
delaying
the
encryption
until
the
last
the
last
possible
point.
A
But
if
you
imagine
that
you're
at
a
larger
or
you
know,
or
talk
directly
to
the
secret
manager,
if
you're
at
a
large
organization
where
you
know
your
job
is
to
just
write
code,
and
you
don't
even
know
that
it's
being
deployed
on
kubernetes
right,
you're,
just
writing
some
ruby
code
or
some
go
code,
and
eventually
it
gets
deployed
on
kubernetes.
It's
a
lot
to
ask
that
software
developer
to
understand
that
complexity,
and
you
know,
write
all
of
the
the
yaml
and
configuration.