►
From YouTube: Turtles All the Way Down: Managing Kubernetes Secrets - Maya Kaczorowski & Alexandr Tcherniakhovski
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Turtles All the Way Down: Securely Managing Kubernetes Secrets With Secrets - Maya Kaczorowski & Alexandr Tcherniakhovski, Google
Secrets are the cornerstones of Kubernetes' security model; they are used both by Kubernetes itself (e.g., service accounts) and by users (e.g., API keys). In this talk, we will discuss users’ options for protecting secrets in Kubernetes. We’ll start with an overview of how secrets are protected and mounted by default in Kubernetes. Then, we’ll cover improvements that have been made in recent releases, including secrets encryption (1.7), and KMS plugins (1.10 Alpha), and how these work with external providers like cloud KMS plugins and HashiCorp Vault. We’ll discuss the tradeoffs of these options based on your requirements. Lastly, we’ll demo how to use a KMS plugin with Kubernetes, and discuss planned improvements to the secrets system in Kubernetes. You’ll leave with an understanding of your secret management options, and an idea of which one is best for your particular needs.
To learn more click here: https://sched.co/FuKP
A
In
the
talk
today
will
cover
how
crimini
secrets
are
protected.
Secret
management
features
that
have
recently
been
added
to
communities,
namely
kms
plugin,
and
what
to
take
into
account
when
picking
the
solution
that
works
best
for
you.
So
first,
let's
talk
about
protecting
secrets.
So
what
is
a
secret?
A
secret
can
be
a
lot
of
things
right.
It
could
be
anything
that
an
application
needs
at
build
or
at
runtime.
Some
common
examples
include
credentials:
API
keys
things
like
your
stripe,
API
key
username,
password,
combos,
etc.
What's
not
a
secret
is
your
entire
application.
B
B
A
B
Okay,
we're
going
to
start
with
the
motivating
was
a
motivating
demo,
so
what
I
would
like
would
like
us
to
imagine
that
we
are
attackers
and
we
are
interested
in
penetrating
a
financial
system,
let's,
let's
say,
store
some
great
card
data,
for
example,
something
that
would
be
a
value
to
us.
So
we've
been
trying
to
be
required
to
attack
this
financial
system
directly,
but
we
haven't
really
been
getting
anywhere.
It
seems
to
be
well
patched
and
protected,
so
we
started
to
look
around.
B
Are
there
any
other
systems
that
we
can
pivot
on
to
attack
this
financial
system
and
through
reconnaissance,
we
found
out
that,
in
fact,
there
is
a
Cuban
in
this
cluster
that
communicates
with
this
financial
system,
and
we
assume
that
this
particular
cluster
must
have
some
credentials
in
this
configuration
to
a
knack
to
this
financial
system.
So
we
expanded
our
scope
of
attack
and
we
are
now
looking
at
the
this
debate,
this
kubernetes
cluster.
B
However,
we
do
not
give
up
good
attackers,
don't
give
up
right,
so
we
expand
our
attacks
attack
surface
and
we
do
more
reconnaissance,
and
we
find
out
that,
interestingly
enough,
this
particular
kubernetes
cluster
is
being
backed
up
daily
to
remote
file
server.
Interesting
from
our
perspective,
we
are
not
so
much
interested
in
this
particular
cluster.
We
don't
want
to
control
it
directly.
Rather,
we
are
interested
in
this
configuration
somewhere
on
that
cluster.
There
is
a
password
that
will
get
us
access
to
that
financial
system.
B
So
really,
if
we
can
get
an
offline
copy
of
the
master,
that's
really
all
we
need.
So
we
started
to
reckon
essence
against
the
backup
server
and
we
got
now
we
got
lucky.
This
backup
server
has
an
attached
owner
ability
and
now
we're
able
to
control
the
backup
server.
So
now
what
I'm
going
to
do
is
I'm
going
to
show
you
a
sort
of
the
experience
of
an
attacker
that
was
able
to
was
able
to
get
a
copy
offline
copy
of
Cuban,
a
deus
master
right.
B
So
imagine
please,
remember
I'm
still
an
attacker,
and
finally,
this
is
sort
of
days
or
months
of
work
and
they're.
Finally,
paying
off
I
have
an
offline
version
of
the
master.
I
did
some
googling
around,
because
I
don't
really
that
much
about
@cd,
but
I
heard
that
Q
Burnett
is
uses
that
city
for
its
configuration
store
and
a
little
of
googling
around
I
found
that
HCD
stores
its
data
in
the
file
called
DB
in
the
directory
called
member
snap.
B
So
at
this
point,
I'm
kind
of
curious
like
what
should
I
look
for
well,
how
about
I
just
do
a
grab
for
secret
I
mean
that
sounds,
seems
obvious,
but
see
what
happens
and
in
fact,
inside
this
database
there
is
a
secret.
You
can
see
that
EDD
is
actually
helping
me
here
quite
a
bit
because
s,
it
is
a
key
value,
store
pair
key
value
pair
store
and
it
nicely
it
provides
me
very
good
hints
as
to
where
to
find
where
to
look
for
secrets.
B
So,
basically
all
secrets
are
stored
on
the
registry
secret
default
namespace
and
here
I
find
an
interesting
fact.
They
have
in
fact
DB
secret
0
1.
Well,
let
me
use
that,
and
it
enhance
my
my
my
search
and
here's
the
password
in
clear-text,
okay,
I'm,
tired
of
being
an
attacker
I
want
to
switch
roles
now
and
I
want
to
become
a
security
conscious,
savvy,
kubernetes,
administrator
and
I
won't
ask
a
question:
how
could
I
have
prevented
this
attack
from
happening?
And
probably
encryption
comes
to
mind
here?
B
We
probably
should
have
encrypted
that
CD
so
before
we
moved
it
to
the
backup
backup
server.
So
you
in
the
right
talk
by
the
time
of
the
talk
is
finish.
You
will
know
all
about
how
to
do
something
like
that
and
my
will
walk
mine
I
will
walk
you
through
this
process,
but
for
now
I
just
wanted
to
show
you.
B
Attackers
experience
if
that
particular
server
was
enabled
for
for
encryption
at
rest,
so
this
is
for
this
demo.
I
created
basically
similar
version
exactly
the
same
configuration
of
a
cluster,
but
this
particular
cluster
uses
the
encryption
at
rest
feature
and,
as
you
can
see,
yes,
the
secret
is
still
there,
but
it's
not
really
usable
to
the
attacker.
What
this
means
is
that
now
I
have
to
as
an
attacker
now
have
to
find
probably
another
way
of
attacking
that
system.
My
vacuum
awesome.
A
So
the
demo
that
Alex
showed
us
showed
one
way
of
you
know
doing
an
offline
attack
against
a
system,
but
it's
not
the
only
way
right.
You
couldn't
also
imagine,
while
I
try
to
drag
this
back
to
the
other
screen,
which
is
not
gonna
work.
Oh,
that
might
work.
You
could
also
imagine
other
attacks
that
occurred
for
an
offline
server
things
like
I
snapshotting,
a
VM,
reminding
the
database
on
a
remote
server,
even
somebody
walking
into
your
data
center
and
walking
away
with
a
hard
disk
right.
A
So
stealing
it
API
credentials
through
mismanagement
is
not
that
crazy
of
an
idea
right.
We
see
that
happen
in
the
real
world,
we
see
people
post
credentials
on
github,
we
see
CEOs,
have
access
to
private
keys,
we
see
secrets
and
buckets
on
in
public
buckets
and
it's
the
first
thing.
I'm
gonna
look
like
it
look
for
if
I'm
an
attacker,
I'm
gonna,
look
for
credentials
in
your
environment
and
can
I
find
a
way
to
use
those
credentials
to
escalate
my
privileges.
A
So
if
you're
gonna
protect
your
credentials
in
your
secrets,
there's
a
couple
of
desirable
properties.
You
can
go
for.
You
know
go
through
these
really
quickly,
identity,
to
enforce
access
management
and
use
tight
authorization
scopes.
You
need
this
to
maintain
the
privilege,
the
principle
of
least
privilege
and
audit,
who
is
accessing
your
secrets
in
terms
of
auditing
itself,
verifying
the
proper
use
of
those
secrets,
ideally
it
at
a
per
secret
level.
This
lets
you
trace
back
what
happened,
who
accessed
a
secret
or
when
it
was
changed.
A
Encryption
at
rest
secrets
should
only
be
in
plain
text
in
memory
but
encrypted
on
disk
rotation
to
refresh
secrets
things
like
changing
a
password
in
isolation
to
create
a
separation
of
duties
from
where
secrets
are
used
versus
where
they're
managed.
So
today,
we'll
focus
specifically
on
the
encryption
piece
of
it.
B
Okay,
when
it
comes
to
encryption,
we
have
several
options
available
to
us.
Technologists
such
as
seven
cooking
drives
full
disk
encryption.
Encrypted
file
systems
have
become
ubiquitous
in
their
recent
in
recent
years,
so
the
choices
that
we
have
to
make
is
at
which
level
do
we
enable
encryption?
Do
we
just
use?
Do
we
just
let
Hardware
do
the
encryption?
Do
we
let
operating
system
do
the
encryption
or
we
do
encryption
ourselves
like
what
you
will
see
in
this
in
the
store.
Kubernetes
community
decided
that
we
should
take
encryption
into
kubernetes
at
the
application
level.
B
So
so
what
is
application
level
encryption
application
level?
Encryption,
as
the
name
implies,
is
the
encryption
scheme
where
the
encryption
protection,
confidentiality
protection
of
data
is,
is
in
the
hands
of
there.
That's
all.
This
is
in
contrast
to
full
disk
encryption
or
hardware-based
encryption,
we're
operating
they
where
application
may
not
even
be
aware
that
is
data
being
encrypted.
However,
application
level.
Encryption
has
a
couple
of
interesting
characteristics
which
will
be
of
an
interest
to
us
here,
so
it
is
applied
at
the
earliest
possible
point,
so
think
about
it.
You
submitted
a
create
new
secret
request.
B
A
cube,
API
server
cube
a
pacer.
It
takes
this
data
and
immediately
protects
it.
In
other
words,
from
that
point,
on
your
data
travels
through
the
rest
of
the
system
in
the
encrypted
form.
Second,
we
protect
at
the
most
granule
level.
We
even
if
you
take
probably
the
biggest
class
that
I've
ever
seen,
maybe
have
hundreds
or
maybe
thousands
of
Secrets
you're,
still
talking
about
a
couple
of
kilobytes
of
data
to
be
encrypted.
Why
is
this
important?
B
Well,
when
you
have
that
little
data
to
encrypt,
you
can
be
encrypted
on
a
regular
basis
without
any
significant
performance,
overhead
and,
last
but
not
least,
application
level.
Encryption
stays
with
the
data
as
it
moves
around
from
server
to
server.
Let's
just
step
back
a
little
bit
and
think
about
that
example.
The
original
example
that
I
showed
where
the
data
was
copied
from
one
system
to
the
next.
Let's
pretend
for
a
second
that
the
original
cluster
was
using
full
disk
encryption.
By
the
time
the
data
is
copied
using,
for
example,
copy
command.
B
The
Fuji
stream
pressure
would
have
already
decrypted
the
data
and,
by
the
time
it
lands
on
a
different
system.
It's
in
plaintext.
This
is
not
the
case
with
application
level
encryption.
Having
said
that,
I
don't
want
to
paint
the
picture
that
somehow
full
disk
encryption
is
worse
or
application
level.
Encryption
is
better
they're
complementing
each
other.
There
are
some
very
cool
things
that
you
can
do
with
full
disk
encryption
encryption
like,
for
example,
secure
boot,
and
there
are
certain
things
that
you
can
do
only
with
application
of
encryption.
B
A
So,
let's
talk
a
bit
about
rotation,
so
once
you've
encrypted
your
secrets,
now
you
have
these
keys
to
manage
and
that's
what
used
to
encrypt
it.
You're
your
protector
data,
but
the
keys
are
just
as
important,
if
not
more
important
than
that,
then
the
data
that
they
protect
because
they
give
that
access.
If
your
key
management
practices
aren't
good,
it
doesn't
really
matter
how
good
your
algorithm
is
or
how
many
bits
you're
using
because
someone
you
just
like
get
access
to
the
keys.
So
how
should
you
think
about
key
management?
A
Well,
a
very
simple
first
thing
to
do
would
just
be
generate
a
local
encryption
key
on
your
laptop
and
use
that
right
and
that's
that's
a
great
first
step
and
without
even
getting
into
how
to
protect
that
key
once
you've
generated
it
if
you're
using
it
for
a
long
time
like
for
many
years,
it's
not
a
great
way
to
protect
your
data
right.
You
might
have
changed
system
ins
during
that
time.
A
What's
considered
a
world
bet
best
algorithm
might
have
changed
during
that
time,
but
also
that
key
that
you
just
tried
and
rated
on
your
laptop
one
day
is
now
protecting
a
lot
of
data
and
potentially
a
lot
of
sensitive
data,
and
that's
not
ideal.
So
instead,
a
better
model
might
not
be
a
use
one
key,
but
use
multiple
keys
or
multiple
key
versions
that
you
change
over
time.
A
You
can
refer
to
them
as
a
single
key
for
naming
purposes,
but
the
point
is
these:
have
different
bits
that
change
changing
keys
frequently
is
meant
to
limit
the
amount
of
data
compromised
if
a
single
key
is
compromised
potential
access
to
the
key
and
corresponding
data,
it's
kind
of
like
changing
a
password
regularly
and
the
potential
time
for
a
crypt
in
the
crypt,
analytic
or
brute-force
attack
on
that
data.
Encryption
keys
can
be
rotated
in
two
ways,
either
through
a
regular
rotation
or
a
regular
rotation.
A
Regular
rotation
is
an
ongoing
schedule,
for
example,
every
90
days
to
change
your
keys
that
might
be
required
for
an
internal
business
compliance
reason.
For
example,
this
provides
some
security
benefit
without
significant
complexity.
By
limiting
the
amount
of
data
encrypted
with
a
single
key,
irregular
rotation
is
an
ad
hoc
rotation
that
you
would
do.
For
example,
after
a
suspected
incident
as
an
additional
stopgap,
you
might
also
have
to
re
encrypt
the
data
that
was
affected.
A
When
we're
describing
a
particular
key
rotation,
we
call
the
crypto
period
at
the
time
during
which
the
particular
key
version
is
used
to
encrypt
data.
So
that's
why
so?
It
could
be
like
90
days
one
day.
Whatever
is
your
requirement
in
your
organization
now
envelope
encryption?
So
once
you
start
encrypting
and
managing
large
amounts
of
data,
rotating
Keys
and
Riaan
crypting,
a
bunch
of
data
becomes
really
annoying
and
really
computationally
intensive.
So
you
typically
end
up
using
a
central
key
cryptographic,
a
key
management
system
with
multiple
layers
of
keys
for
the
encrypted
data.
A
This
is
something
known
as
envelope
encryption,
which
is
the
process
of
encrypting
one
key
with
another
key,
usually
a
key
that
encrypts.
The
data
you'll
need
three
pieces
here.
First,
your
data,
for
example
your
secret
next,
a
data
encryption
key
or
a
dek,
which
is
used
to
encrypt
your
secret.
This
is
typically
stored
with
the
data
that
it
protects
and,
third,
a
key
encryption
key
or
a
ke
ke,
which
envelopes
or
encrypts
the
dek
and
that's
typically
stored
somewhere
centrally
and
like,
for
example,
a
key
management
system
to
use
envelope
encryption.
A
A
A
A
Now,
as
I
mentioned
earlier,
the
de
K
is
typically
stored
with
a
data,
so
that'll
be
stored
in
the
data
store
and
the
ke
KS
are
typically
stored
centrally
so
they'll
be
stored
in
the
canvas,
and-
let's
assume
in
this
example
that
the
kms
has
a
rotation
schedule
that
rotates
keys
monthly
okay,
so
we
are
in
oh,
no
I,
can't
click
forward.
I.
A
Lied
we
are
in
November,
so
first
we're
gonna
create
essentially
eight
the
first
instance
of
a
key,
a
ke
ke.
The
first
ke
ke
comes
up
great,
it's
ready
to
be
used
for
encryption,
and
it's
also
important
to
note
that
again,
as
I
mentioned,
these
are
in
separate
places
so
that
the
people
who
manage
this
ke
ke
are
different
than
the
people
who
manages
C.
A
Ok,
you
can
have
a
separation
of
duties
here
now,
let's
encrypt
our
for
a
secret
secret
one,
so
we're
going
to
generate
a
dk4
secret
one
and
use
that
to
encrypt
it
and
then
the
de
ke
is
going
to
encrypt
it
with
the
ke
ke
version,
one
that
we
have
now
note
that
we're
versioning
the
ke
case,
because
these
are
gonna
change
over
time
as
different
versions
of
the
same
key,
whereas
the
decays
we're
not
going
to
rotate
we're.
Just
gonna
use
brand
new
decays
every
time,
so
we're
just
gonna
number
them
sequentially.
A
Okay,
we're
still
November.
Now
we
create
secret
secret
was
generated,
it
gets
its
own
DK,
DK
and
DK.
2
is
also
encrypted
with
KK
version
1
because
that's
still
the
current
version
in
November
so
far
so
good
now
we're
in
December.
We
generates
another
KK
and
the
next
version
of
the
KK.
We
don't
delete
or
remove
KK
P
1.
We
don't
re-encrypt
these
secrets.
We
just
have
a
new
version
for
any
new
data
that
we're
going
to
encrypt
now.
Unfortunately,
in
November
nobody
writes
a
secret,
so
we
move
on
to
January
same
thing.
A
January,
we
generate
a
new
key
KK
KK
V
3.
This
time
when
we
write
a
secret,
they
write
secret
v,
3,
sir
secret
3.
We
encrypt
it
with
its
own
DK,
DK
3
and
that's
encrypted
with
ke
ke
v
3.
That
makes
sense
now
suppose
that
we're
going
to
rewrite
one
of
those
secrets
so
we're
gonna
change
secret
to
an
update.
What's
written
there.
First,
we
have
the
decrypt
secret,
so
we
have
to
go,
get
DK,
2
and
DK.
2
is
encrypted
with
KK
1.
So
we
have
to
go,
get
that
and
use
it.
A
Like
I
said
this
is
still
valid.
It
hasn't
been
removed.
Now
we're
gonna.
When
we
write
secret,
we're
gonna
write
it
with
a
new
DK
again
we're
not
reading.
Sorry
we're,
not
rotating
decays
we're
generating
a
new
te
case,
and
this
DK
is
gonna
encrypted
with
the
current
valid
KK,
the
most
recent
version,
which
is
KK
v3,
and
so
that's
how
using
envelope,
encryption
and
and
key
rotation.
You
give
this
constant
freshness
in
your
environment
and
encryption
of
these.
These
things
without
actually
a
lot
of
management.
A
B
Ok,
Thank
You
Maya,
so
I
explained
us
in
great
details
about
versioning
of
keys
and
envelope
encryption,
but
I
thought
we
will
make
it
a
little
bit
more
concrete
with
a
practical
example.
So
in
this
demo,
we'll
look
at
using
Walt
another
great
open
source
project
for
impor
secrets
management,
and
we
certainly
encourage
you
to
to
look
at
more
deep
because
I'm
certainly
not
going
to
have
time
to
do
a
justice
in
this
presentation.
B
But
concretely
volt
provides
us
with
an
ability
to
play
a
role
of
external
key
management
service,
so
think
of
a
vault
in
transit
mode.
By
the
way
world
has
many
different
facets:
you
can
you
can
use
it
as
an
external
key
secret
management
system,
but
we
will
just
simply
use
it
as
a
key
management
service.
Think
of
it
in
terms
of
you,
have
a
service
I
have
a
I
would
like
something
to
be
encrypted:
I
just
send
it
to
vault
and
it
returns
back
ciphertext.
In
other
words,
it
never
stores
my
data.
B
It
simply
takes
something
encrypts
it
and
returns
it
back,
but
we
also
talked
about,
and
hopefully
I
will
be
able
to
illustrate
the
idea
of
thinking
in
terms
of
keys
as
a
container
of
version
of
keys.
So
when
you
create
a
key,
it's
not
a
single
thing.
It's
basically
something
get
gets
versioned
over
time,
so
hopefully
I'll
be
able
to
illustrate
that
and
just
sort
of
this
to
set
the
dam
up
both
is
a
restful
api.
You
can
think
of
it
as
restful
api
with
certain
endpoints.
B
B
That's
you're,
doing
great.
Okay,
just
make
sure
I'm
on
the
right
here.
Okay,
what
we're
going
to
start
with
was
the
idea
of
keys.
So
the
first
thing
you
will
do
when
you're
working
with
vault
in
transit
mode,
you
will
create
a
key
and
just
to
save
some
time.
I
already
created
this
key
for
us
and
I
was
using
it
to
build
the
rest
of
our
demos,
but
what?
If
I
I
wanted
to
sort
of
request?
B
So
the
name
of
the
key
is
key
WS
and
what
I'm
doing
in
this
I'm
actually
calling
this
REST
API
to
get
the
metadata
on
the
ski
in
the
interesting
part.
If
we
scroll
up
the
metadata
contains
nine
different
key
versions
so
think
about
in
terms
of
a
container
so
key
that
I
created,
it's
called
KWS,
but
it's
not
actually
the
like.
When
we
think
of
keys,
we
think
of
this
long
string
of
random
bytes,
but
it's
in
fact
it's
a
container
that
contains
many
array
of
random
bytes.
A
B
Thank
you.
So
the
latest
version
is
version
10
and
those
by
the
way
are
not
the
actual
secrets.
This
is
just
AI
ideas
like
those
long
numbers
and
X
2,
1
&,
2,
3,
&
4.
Those
are
just
ideas
that
identify
uniquely
the
key.
So
what
I
want
to
do
is
sort
of
show.
A
very
simple
use
of
vault
in
this
example,
I'm
simply
I'm,
creating
a
new
secret,
hello,
secret,
1
and
I'm,
sending
this
plaintext
to
the
ink
transit
encrypt
endpoint
and
what
I
get
back.
B
B
Okay,
so
that's
work
at
what
has
expected,
but
what
I
wanted
to
show
you
here
is
make
sure
that
this
is
the
encrypted
payload,
but
it
has
a
metadata
attached
to
it.
So
it's
and
the
current
version
of
the
key
is
the
attend.
So
you
notice
that
v10
is
attached
to
the
data
itself.
So
if
I
wanted
to
decrypt
this
payload
I
needed
to
not
only
supply
the
ciphertext,
but
I
also
need
to
tell
what
what
version
of
the
key
was
used.
Okay,
so
let
me
show
you
something
else
so
this.
B
Basically,
we
are
pretending
that
30
day
test,
let's
say
I,
don't
want
to
wait.
30
days,
I'm
gonna
rotate
the
key
right
now,
so
just
another
API
call
and
we
rotated
the
key.
So
what
that
means
is
what
is
the
current
version?
Now?
That's
11!
It
should
be
11.
So
if,
if
at
this
point,
I
wanted
to
encrypt
something
else,
let's
do
that.
B
Version
11
is
attached
in
the
payload,
so
this
is
this
is
the
really.
The
crux
of
this
presentation
is
to
explaining
that,
as
you
rotate,
the
key
new
keys
are
being
used
in
protected
data,
and
the
point
being
here
is
that
version
10
service
for
30
days
and
it
encrypted
some
data,
then
30
days
next
30
days
come
along
and
we
want
to
use
another
key.
So
if
v10
is
ever
compromised,
only
the
data
that
was
encrypted
in
the
previous
30
days
is
compromised.
Be
b11
is
still
is
still
something
that
we
can
trust.
A
Okay,
so
that
was
a
lot
of
buildup
you're
like
what
does
this
have
to
do
with
kubernetes?
Aha,
this
is
this
is
because
you
need
to
understand
all
this
stuff
it'll
help
to
understand,
what's
going
on
in
kubernetes,
so
by
default
in
Crewe
bananas
as
Alex
demoed.
At
the
very
beginning
of
this
talk,
secrets
are
actually
stored
in
net
CD
and
have
no
additional
protection
right,
they're,
not
further
encrypted
they're
encrypted
the
same
way
as
the
rest
rest
of
a
tedious.
A
A
pod
can
access
all
secrets
for
the
file
system
as
environment
variable
or
via
crannies
API
call.
Now,
starting
in
caranas
1.7
Rene's
offers
application
layer
secrets
encryption
using
something
called
an
encryption
config
using
this
there's
a
differential
protection
of
foetus
secrets
with
a
key,
that's
specified
in
a
local
@
CD
secret
can
fit
so
the
key
is
to
store
the
net
CD.
Now
we
actually
don't
recommend
this
option
because
it
doesn't
actually
improve
your
security
right.
It's
the
same
threat
model
that
you
had
before.
A
If
somebody
can
break
in
and
see
your
secrets,
they
can
break
in
and
get
this
key
and
get
to
your
secrets
and
it
Cronus
1.10
criminales
introduced
envelope
encryption
or
these
secrets
using
something
called
a
kms
provider
using
that
once
again,
you
have
a
DK,
that's
stored
locally.
That
protects
those
secrets,
but
this
DK
is
encrypted
with
a
kek
that's
stored
somewhere
else.
So
let's
see
how
this
works.
B
Ok,
thank
you.
So,
let's
take
a
look
at
the
secrets
of
it.
Now
we're
talking
actually
how
application
encryption
is
implemented
in
kubernetes
version,
1
point
10.
So
what
are
the
players
here
like?
What
are
the
main
components
involved?
So
you
need
a
version
of
kubernetes
at
at
least
one
point
and
because
that's
that's
where
we
implemented
encryption
with
EMS
with
the
certain
kms
provider.
B
Here,
the
kms,
plugin
and
I'll
explain
what
Camus
plugin
is
in
a
second
and
you
need
kms
external
system,
for
the
purposes
of
this
demo
were
using
volt
in
transit
mode,
but
many
other
came
as
providers
for
work.
As
a
matter
of
fact,
at
this
point
in
time
there
are
EMS
providers,
Cuban
atheists
will
work
with
Azure
Google
cloud
AWS
key
management
servers
services.
Rather
so
what
is?
Why
do
we
need
kms
plugin?
Why
couldn't
Cube
API
server
directly
talk
to
kms?
B
Interestingly,
now
that's
actually
how
things
started,
but
later
we
discover
that
it
was
actually
not
going
to
scale,
because
in
that
way,
if
we
were
implemented
directly,
basically,
if
we
implemented
the
communication
protocol
become
cube,
API
server
in
all
the
kms
implementations,
it
would
not
scale.
It
would
be
very
difficult
to
maintain.
So
the
decision
was
made
to
abstract
to
abstract
you
BPI
server
from
the
implementation
details
of
the
areas,
EMS
providers.
So,
for
example,
there
is
a
kms
plugin
for
a
zero.
There
is
a
kms
blogger
plug-in
for
AWS.
B
B
So
ok,
thanks
so
user
requested
to
create
a
new
secret
and
what
happens
at
this
point.
The
first
thing
that
the
cube
a
piercer
will
do
is
it
will
generate
using
pseudo.
Random
generator
generate
a
data
encryption
key
once
the
data
encryption
key
is
generated,
this
deck
will
be
passed
to
JMS
plugin
and
tim
has
plugin.
All
it
really
does
is
just
forwards
it
to
the
KMS.
However,
I'm
kind
of
skipping
through
a
lot
of
details,
obviously
camus
plugin,
you
start
and
akitoki
MS,
but
for
now,
let's
just
keep
it
at
the
sky
level.
B
Assuming
everything
went
well,
KMS
returns,
wrapped
data
encryption
key
and
KMS
plugins
simply
forwards
it
up
to
the
cube
API
server
now
cube.
Api
server
actually
has
everything
it
needs.
Remember
that
envelope
consists
of
two
parts:
the
secret
itself
encrypted
with
a
deck
and
the
deck
encrypt.
It
was
CAG.
We
put
together
those
two
things
and
we
create
an
envelope
and
the
Senate
to
hcd.
Hcd
is
not
aware
of
the
fact
that
data
is
encrypted.
It
just
sees
a
normal
write
request
from
its
perspective.
Nothing
really
interesting.
What's
happening
there.
B
A
B
B
So
really
there
are
a
couple
of
encryption
files
configuration
files
that
we
need
to
set
up,
so
I
already
started,
kms
plug-in
volt
and
it's
already
running
and,
as
you
can
see,
when
I'm
starting
came
as
plug-in,
I
need
to
supply
the
remote
address
of
the
vault,
which
we
need
to
know
how
to
talk
to
to
the
kms
itself,
and
we
need
to
supply
a
talk
in
so
that
the
chemists
wagon
can
authenticate
itself
to
the
kms.
I
will
have
more
to
say
in
the
next
slide.
B
About
this
token,
also,
please
note
that
this
is
a
demo.
In
reality
you
will
never.
You
should
never
have
a
local
address
so
that
they
came
as
plug-in
talks
to
this
2-day
to
the
kms
on
the
same
server,
because
that
kind
of
defeats,
the
purpose
you
want
to
make
sure
that
the
keys
are
actually
on
the
remote
system
protected
by
key
management
server.
B
There's
another
configuration
file.
This
one
is
actually
supplied
to
the
this.
One
is
supplied
to
the
cube
API
server,
and
here
we
have
some.
Let
me
let
me
highlight
a
couple
of
points
here.
So
the
first
one
is:
you
need
to
supply
what
type
of
resources
you
would
like
to
be
encrypted.
Typically,
those
are
secrets,
but
you're
not
limited
to
secrets.
You
can
encrypt
any
type
of
kubernetes
resource.
If
you
feel
like
it,
you
can
encrypt,
for
example,
pods
or
anything
like
any
configuration
data
can
be
encrypted
inside
inside
at
CD.
B
Then
we
come
down
to
the
to
the
configuration
of
they
came
as
plug-in
itself.
It's
actually
remarkably
simple
because
remember
all
the
details
of
how
cube
API
server
talks
to
the
kms
is
implemented
inside
the
kms
plugin.
So
all
that
came
as
bi-cubic
a
server
needs
to
know
is
how
to
connect
to
the
JMS
plug-in,
and
this
is
accomplished
via
a
UNIX
domain
socket,
and
this
is
the
file
paths
which
we
need
to
ensure
that
both
kms
plugin
and
the
cube
API
server
talking
to
the
same
socket,
you
can
also
use.
B
Okay
and
the
final
piece
of
this
puzzle
is
when
we
start
in
few
API
server.
We
need
to
supply
another
flag,
experimental
encryption
provider,
config
we're
actually
removing
the
word
experimental
in
hopefully
113
so,
but
really
is
just
the
only
thing
that
QB
pacer,
when
used
to
know,
is
how
to
find
that
encryption.
Config
we
just
described,
which
was
just
described
here.
So
this
is
it
so.
Let
me
start
up
cube
API
server
now
in
in
the
encryption
mode,
with
encryption
mode
enabled.
B
Okay,
let's
fire
up
qvp
ice,
alright!
So
it's
now
the
setup
for
encryption.
But
how
would
we
test
this?
So
why
don't
we
I'll
just
give
it
a
couple
of
seconds
to
start
up
and
what
I'm
going
to
do
here
is
I'm
going
to
create
another
secret,
this
time,
I'm
creating
a
new
secret
code,
you
can
shine
a
demo
and
once
it
is
created,
I
want
to
kind
of
play.
The
role
of
an
attacker
and
I
want
to
look
at
the
raw
LCD
because
I'm
on
the
same
machine.
B
And
there
it
is,
it
is
protected,
so
an
attack,
an
offline
attack,
would
have
failed
at
this
point,
and
this
is
unfortunately
I
know
this
I
had
to
go
fairly
fast
through
this
demo.
There
is
a
couple
of
interesting
details,
but
I
want
to
make
sure
that
I'm
being
not
run
out
of
time.
If
you
have
any
more
questions
regarding
this,
please
see
us
up
at
the
talk.
B
The
only
track
alright
yeah
okay,
so
we
saw
this
talkin
in
the
configuration
just
a
minute
ago.
Just
a
couple
of
seconds
ago.
We
saw
that
in
order
for
kmetz
plug-in
to
to
talk
to
the
vault.
What
has
to
be
what
the
the
game
is?
Playin,
somehow
needed
to
authenticate
to
the
remote
EMS
provider,
which
is
which
is
what
in
this
case.
However,
if
we
think
about
it
carefully,
what
just
happened?
B
We
had
to
supply
a
secret
and
place
it
on
the
file
system
inside
the
same
box,
which
is
master.
So
if
an
attacker
was
able
to
get
complete
file
system,
what
would
have
happened
in
this
case?
They
could
potentially
reverse
the
whole
process.
They
can
take
the
talkin.
They
can
call
into
a
vault
decrypt,
the
decrypted
DAC,
and
they
can
unwrap
everything
and
basically
get
back
to
the
secrets.
So
this
seems
like
a
kind
of
a
circular
relation
a
circle
of
dependency,
but
there
are
some
ways
to
get
around
in
a
nutshell.
B
What
we
need
to
look
at
I
recommend
using
technology
like
trusted,
protected
modules
TPMS
to
to
seal
the
secret
to
the
hardware
on
the
server
that
way.
If
an
attacker
gets
an
offline
version,
they
would
not
be
able
to
read
this
talkin.
Please
see
me
after
the
talk.
If
this
wasn't
clear
and
unfortunate,
we
are
running
out
of
time.
So
up
best
of
the
matter,
yeah.
A
The
last
last
option
we
haven't
talked
about
is
not
using
criminal
secrets
at
all,
so
you
don't
have
to
use
secrets
and
kubernetes
if
you're
already
using
an
external
secret
store
to
store
secrets.
That
might
be
a
great
option
for
you.
One
popular
example
is
hash
record
vault
Alex,
just
demoed
the
first
way
of
using
vault,
which
is
by
using
the
kms
plugin,
so
that
actually
keeps
your
secrets
of
record
in
kubernetes
and
uses
vault
as
the
root
of
trust.
A
The
alternative
is
to
have
communities
authenticate
directly
to
hash
a
court
fault
using
the
kubernetes
off
back
in
for
vault
that
uses
vault
as
your
root
of
trust
for
for
community
secrets.
Typically,
I
hear
users
pick
the
second
option:
that's
because,
if
someone's
talking
about
using
vault
in
Korean,
together,
they're,
probably
already
using
vault
and
already
have
an
external
secret
store,
so
that
makes
that
make
sense
in
that
case.
So
to
recap
all
the
different
options
we
went
through
today,
extremely
quickly
by
default,
your
secrets
are
not
encrypted.
A
A
Encryption
rotate
keys
regularly
to
limit
the
amount
of
that--of
that
can
be
compromised
when
encrypting
a
lot
of
data
and
Volokh
encryption
is
a
common
scheme.
This
lets
you
encrypt
data
by
using
a
local
data,
encryption
key
and
a
remote
key
encryption
key
to
encrypt
that
data
and
in
kerbin
Eddy's
we'd
recommend
you
either
use
that
option
the
kms
plugin
or
if
you
already
have
one
use
your
external
secrets
provider,
and
that
was
a
very
quick
recap
and
I
think
we're
out
of
time.