►
From YouTube: Protecting Sensitive Code with Encrypted Container Images on... Brandon Lum & Harshal Patil
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Protecting Sensitive Code with Encrypted Container Images on Kubernetes - Brandon Lum & Harshal Patil, IBM
Many enterprises are driven by trade secrets in their code - whether it is a proprietary AI model, or a secret high frequency trading strategy. It is of utmost importance that critical algorithms, proprietary code, or other content that is highly sensitive have minimum exposure unencrypted. In this talk, we will show the end-to-end process of how users can create an encrypted container during the build process, to running encrypted container images on a Kubernetes cluster with the proposed ImageDecryptSecrets. We will show how the Encrypted Images OCI spec allows fine-grained encryption through leveraging layering of container images. Finally, we will talk about how Image Encryption will integrate into the container ecosystem, and talk about several possibilities for innovation in the container DevSecOps pipeline.
https://sched.co/NrpO
A
Getting
hacked
and
a
plenty
of
data
was
compromised
and
that
if
this
kind
of
situation
arises
in
another
history,
which
is
popular
regularly,
what
could
happen
is
an
attacker
can
change
the
images
that
you
trust
well
down.
Where
that
you
download-
and
you
may
end
up
downloading
the
images
that
you're
that
are
actually
compromised
so
and
this
this
does
not
mean
that
it
only
happens
in
the
case
of
docker
hub.
You
could
have
one
registry,
and
this
could
happen
in
anyone's
case
so,
but
you
you
might
argue
that
there
are
ways
to
prevent
that.
A
If
anyone
gets
let's
say
an
access
to
your
registry,
you
could
have
a
docker
node
tree
or
a
port.
It
is
set
up
in
a
way
that
even
if
your
server
gets
compromised,
your
sign
images
will
not
match,
and
you
still
can't
protect
yourself
in
such
scenario,
however,
you,
if
you
registry,
holds
images
which
have
a
sensitive
content
in
it.
It
does
not
prevent
you
prevent
it
from
the
core
getting
compromised
in
that
sense,
let's
see
how
we
can
envision
our
envision,
a
scenario
where
we
can
mitigate
such
attacks.
A
So
all
what
we
are
looking
for
here
is
we
have
we.
We
need
to
be
able
to
create
container
images
where
we
should
be
able
to
during
the
build
process.
We
should
be
able
to
encrypt
them
and
then
upload
it
to
the
registry
and
then
decrypt
them,
while
you
run
them
using
sa
kubernetes.
That
way,
the
advantage
of
that
process
is,
you
do
not
depend
on
registry
being
highly
privileged,
so
as
I
was
talking.
So
if
you
start,
if
you
about
image
encrypting
your
docker
images,
you
do
not
need
to
have
extremely
high
privilege
registry.
A
That
means
you
do
not
have
to
trust
your
administrator
either
malicious
or
are
the
intentional
compromises
of
the
registry
as
well
as
it
brings
in
image
confidentially.
So
you
you
can
have
extremely
confidential
algorithms.
Like
say
you
might
have
a
tree
intention
for
model,
for
example,
which
you
do
not
want
to
be
exposed
to
the
outside
world.
A
You
can
argue
that
you
know
you
can
keep
the
registry
password
protected,
but
even
in
that
case,
the
administrator,
all
the
or
the
where
the
registries
hosted.
Storing
the
data
has
access
to
the
contents
of
the
store
so
and,
and
once
you
bring
in
encryption
it,
it
allows
you
to
do
a
lot
of
interesting
stuff
like,
for
example,
you
can
sign
the
continuing
images
with
target
systems
public
key
in
such
a
way
that
you
can
enforce
that.
The
content
images
that
I
am
building
will
run
only
in
specific
Geographics
geographies.
A
For
example,
you
can
say
image.
A
particular
image
will
only
run
inside
the
European
unity,
European
Union's
Java
borders.
So
let's
see
how
we
can
envision
this
now.
The
I
would
like
to
tell
you
that
the
work
has
been
happening,
all
the
way
from
OCI,
spec
and
kubernetes
and
anything
in
between
and
we
are.
We
are
talking
to
all
these
communities
and
the
final
output
that
you
may
say
when
everything
falls
in
line
may
or
may
not
actually
order
lap
with
what
I
am
saying
right
now,
but
it
will
be
very
similar
to
this.
A
So,
let's
see
how
we
will
be
able
to
encrypt
the
container
image,
so
you
have
a
darker
build
command.
Let's
imagine
and
you
should
be
able
to
pass
a
public
key,
so
I,
let's
say
I
want
to
be
able
to
create
a
container
image
in
such
a
way
that
only
Brandon
should
be
able
to
run
it
so
I
take
his
public
key
during
that
Aqil
build
process.
I
should
be
able
to
take
his
public
key
encrypted,
and
then
this
this
encrypted
key
I
will
push
it
to
the
registry.
A
Once
inside
the
registry,
it
would
gets
interesting
how
you
gonna,
use
it
or
consume
it
within
the
context
of
kubernetes.
So
we
all
know
kubernetes
has
this
concept
of
secrets
right,
cubrir
is
secret.
Is
it's
a
way
to
store
sensitive
data
within
the
within
the
confines
of
Cooperman
kubernetes
so
and
there's
a
very
particular
type
of
secret
called
image
pool
secret?
A
This
secret
is
used
for
fetching
the
images
that
are
behind
the
username
password
in
the
registry,
so
we
thought
that
we
have
a
very
similar
use
case
here
where,
instead
of
pulling
the
image,
we
want
to
decrypt
the
image
on
the
host
right.
So
we
started,
modeling
are
the
secret
that
holds
the
the
private
keys
required
to
decrypt
the
images
and
we
are
calling
them
image
decrypt
secret,
for
this
work.
We
already
have
submitted
KP
in
his
under
discussion,
as
I
mentioned,
we
already
talking
to
OCS
spec.
A
There
is
a
PR
in
continuity
to
support
it,
because
you
need
to
have
this
support
all
the
way
throughout
the
stack.
So
once
you
have
this
secret,
what
the
flow
will
look
like
I
know,
if
there's
a
request
by
the
user
to
create
a
encrypted
part.
The
image
secret
image,
decrypt
secret,
will
basically
provide
the
keys
required
and
the
part
will
create
it
with
the
keys
that
are
required
and
those
keys
will
be
pushed
down
to
the
kubernetes
worker
node
and
then
the
worker
node.
A
A
Hey
yeah,
this
is
stage
so
let
me
just
say
clear
and
then
let
me
go
to
container
D.
So,
although
I
talked
about
docker
will
to
begin
with,
we
started
playing
with
container
D
directly
because
that
was
easier
to
deal
with
and
Lockard
build
end
of
the
day,
we'll
end
up
working
something
similar.
So
let
me
see,
as
you
can
see,
you
only
have
one
image
right
now:
nginx
and
then
on
the
on
this
system.
I
have
a
sample
GP,
GK,
keeper
and
I'm
gonna,
use
that
to
encrypt
the
image
first.
A
Then
twist
it
and
you
have
this
encrypted
image.
How
so
just
to
understand
things
better?
We
added
a
small
command
in
continuity,
which
helps
you
to
see
the
images
so
LS
zoom
out
a
little
bit
yeah.
So
you
see,
that
is
the
image
here
which
says:
encryption
and
recipients
none.
So
this
is
what
you'll
see
for
a
regular
the
container
image
right
now
that
everyone
is
used
to.
A
But
if
you
try
to
use
the
layer
info
against
the
encrypted
image,
you
have
a
GPG
and
the
target
system
target
recipients
address,
so
it
basically
means
that
this
image
can
only
be
decrypted
with
this
target
key.
That
is
showing.
Let
me
just
clear
this.
Let
me
just
push
the
team,
we
lost
it,
you
have
it.
Let
me
just
push
the
image
to
the
local
register.
B
A
Now
I'm
gonna
change
it,
and
this
is
the
standard
cumin
in
secret.
It's
a
it's.
It's
a
default
open
that
we
have
here
and
then
now
what
happens
the
the
image
that
we
just
encrypted.
If
we
try
to
deploy
it,
let's
say
you:
you
are
running
this
register,
you
upload
it
to
the
public
registry
I
encrypted
image
and
someone
tries
to
download
and
run
it.
So
let's
see
how
it
reacts
to
that.
So
I
do
not
have
the
private
key
at
this
point
and
I'm
trying
to
run
an
encrypted
image
just.
A
For
a
second
yeah,
so
I
got
an
error.
Let's
see
what
the
error
is,
as
you
can
see,
you
are
not
authorized
to
run
this
image
and
the
missing
private
key
is
needed
for
decryption,
so
think
of
it.
This
way
that
if
you
had
an
encrypted
image,
you
know
in
a
public
repository
by
password
or
not.
If
someone
tries
to
run
that
image
without
having
the
private
keys
or
without
having
the
target
system
that
is
encrypted
for
you
will
you
are
supposed
to
get
yeah.
A
A
Okay,
remember
that
key
that
keeper
that
I
had
I
had
already
extracted
the
private
key
required
from
this
GEB
keeper
and
I
have
added
converted
into
base64
and
I'll
create
a
secret.
So
this
is
a
secret
that
we
are
introducing
the
kubernetes,
so
I'm
saying
create
a
new
secret
type
of
image,
decrypt
with
this
private
key.
So
when
you
get
a
secret
there
you
go
now.
You
have
a
key
secret.
A
Let's,
let's
go
deeper
than
that
yeah,
so
it
holds
the
depletion
decryption
key
that
the
base64
key
that
we
had
and
you
get
parts
let's
see.
So
we
will
modify
our
body
ml
slightly.
So
instead
of
having
this,
when
you
want
to
run
this
engine
engine
s
encrypted,
image
will
have
to
provide
the
pre
key
that
that
kubernetes
should
pass
down
to
the
worker.
Node.
So
I
will
say
image:
D,
clip
secrets,
name
key
secret.
The
key
secret
is
the
one
which
we
just
created.
A
Get
part
and
it
works
so
the
earlier
when
I
try
to
run
this
image,
we
are
without
without
the
secret.
You
remember,
you've
got
an
error
saying
you
do
not
have
you're
not
authorized
to
run
this
image,
but
now
I
am
able
to
because
I
associated
this
part
with
the
right
secret
and
that
secret
was
passed
to
connect
you
blade
and
cubelet
pass
to
CRI
and
see
how
I
eventually
gave
it
to
continuity
and
that
continuity
was
able
to
extract
the
not
only
extra
entire
image
but
able
to
decrypt
as
well.
A
Let's
delete
this,
however,
those
who
know
those
of
us
know
how
to
use
the
secrets
little
in
an
elegant
way.
We
know
that
we
can
use
service
accounts
in
kubernetes
to
pass
secrets,
so
we
plan
to
support
service
accounts
for
image,
decrypt
secrets
as
well.
So
let's
see
how
we
can
achieve
that.
So
we
will
first
patch
a
service
account
with
the
secret
that
we
already
created
and
will
say,
describe
the
service
account.
And
if
you
see
this
is
a
new
new
thing
that
we're
trying
add
here
so
now.
A
A
A
And
it's
working
right
so
I
think
that
that's
that's
completes
my
demo.
Okay
and
I
jump
back
to
the
so,
if
you
want
to,
if
you
want
to
thank
Mandan,
so
if
you
want
to
know
more
understand
more
about
the
work
going
on,
please
visit
these
links.
We
have
a
like
a
very
good
and
detailed
discussion
going
on
all
topics
and
we
hope
to
get
this
mod
ready
soon
and
with
that
I
will
dive
deeper
in
that
and
I'll
hand
it
over
to
Brandon.
C
Once
you
have
encryption,
so
I'm
gonna
go
through
a
deep
dive.
I'm
gonna
start
with
a
little
bit
of
crypto
background,
so
just
bear
with
me
a
little
bit
so
we're
gonna
go
through
a
short
primary
encryption,
so
so
we're
using
two
kinds
of
encryption:
we're
using
symmetric,
encryption
and
asymmetric
encryption.
So
for
those
that
may
not
be
as
familiar
of
encryption,
what
symmetric
encryption
is!
Is
you
have
a
piece
of
plaintext?
So
a
message
you
wanna
encrypt
and
you
have
a
key,
which
is
a
symmetric
key
now
in
symmetric
encryption.
C
C
C
So
we
also
introduced
asymmetric
encryption,
so
asymmetric
encryption
has
pretty
much
the
same
setup
as
then
still
have
been
one
key.
You
have
a
key
pass,
so
you
have
the
public
and
private
key
pair,
and
so,
in
this
case,
to
encrypt
the
message
you
will
use
the
public
key
and
encrypt.
The
message
will
be
decrypted
by
the
private
key.
So
the
great
thing
about
asymmetric
encryption
is
the
public
key,
as
the
name
suggests
can
be
public.
So
this
makes
being
able
to
authorize
and
create
the
image
to
someone
ought
to
be
able
to
share
keys.
C
C
So
what
we're
doing
is
we're
taking
symmetric
encryption
and
asymmetric
encryption
and
getting
both
the
best
of
both
worlds,
so
this
will
may
be
familiar.
We
have
our
favorite
people
in
crypto,
Bob
and
Alice,
and
so
in
this
case,
Bob
wants
to
send
an
encrypted
image
status.
Now
we
haven't
forgotten
about
the
duplication,
we're
getting
that
soon.
D
C
Right
so
Bob
does
a
bill
and
what
it
does
is
it
generates
the
symmetric
key,
and
this
symmetric
key
is
used
to
encrypt
the
image.
So
now
we
have
an
encrypted
image
and
what
we
want
to
do
is
we
want
to
say
that
this
is
image
I
want
to
encrypt
for
Alice
right,
so
we
somehow
have
to
get
that
yellow
symmetric
key
over
to
others
and
the
way
we
do
this
is
to
take
editors
public
key
and
create
what
we
call
a
brad
key.
C
C
So,
on
Alice's
side
to
be
able
to
decrypt,
the
image
is
pretty
much
the
opposite
of
what
we
saw
earlier
right.
She
takes
the
rap
key
that
only
she
can
open
and
decrypt
it
with
her
own
private
key,
which
only
she
has
access
to
and
from
that
she
gets
the
symmetric
key,
as
well
as
the
parameters
to
perform
the
decryption
and
she
successfully
decrypts
image.
C
So
how
does
this
relate
to
what
the
images
actually
look
like?
So
if
we
dive
deep
into
what
changes
we
are
looking
at
in
the
OCI
spec,
so
to
start
off
with.
What
we
have
here
is
what
the
oh
she
is
back,
looks
image
right
looks
like
today
right
so
the
image
back
at
least
in
here
you
see
the
manifest
is
really
some
metadata
about
the
image
itself
and
each
layer
is
basically
a
blob
of
a
collection
of
house,
and
in
this
case
the
files
are
just
tada
right.
C
So
the
changes
here
to
the
spec
are
pretty
straightforward.
One
is
we've
added
a
new
media
type,
so
we've
added
the
Plus
ENC
media
type,
standing
from
abbreviated
for
encryption
and
what
this
means
it
tells
the
runtime
that
this
image
is
encrypted
and
it's
gonna
try
and
equipped
it
and
we've
also
added
additional
field,
and
the
annotations
here
called
the
keys
and
what
these
keys
are
the
rap
keys
that
we
saw
earlier
so
the
keys
here
could
be
a
rap
key
for
edits,
I
repeat
for
Bob
and
so
on.
C
So
one
interesting
thing
so
now
we're
back
to
this
application
right.
So
the
the
general
thought
of
this
is
that
by
doing
encryption
on
the
layers,
we
are
able
to
take
advantage
of
a
lot
of
container
like
design
behaviors
right.
So
generally,
if
you
have
a
container
your
container
made
a
couple
layers,
so
you
have
like
the
operating
system.
C
All
right,
so
this
is
what
that
is
currently
ongoing.
I
think
it's
how
I
mentioned
before
this
is
something
that
we
started
off
in
the
OCI
spec
and
it's
made
its
way
and
through
the
entire
stack
between
OCI
and
I'll,
contain
at
the
cupola
and
so
on.
So
one
thing
is
which,
by
still
trying
to
get
this,
we
have
a
PR
open
for
the
OSI
I
suspect,
and
you
know,
if
you
have
any
views-
or
you
know
if
you
like
this
feature,
do
definitely
chime
in
on
that.
C
We
have
ongoing
work
like
the
KP
that
we
just
showed
we're
also
looking
to
integrate
this
to
basically
everything
within
the
sec,
so
the
build
tools
and
cRIO
support
as
well
and
finally,
one
thing
that
we
are
looking
at
also
in
the
long
term
is
when
we're
talking
about
this
encryption
stuff.
When
we
one
features
like
your
execution,
geofencing
execution,
we
want
to
really
be
able
to
tie
this
to
crypto
cryptographic
keys
that
we
have
higher
surance
on.
So
this
would
be.
D
C
So
the
key
never
actually
leaves
the
key
management
service.
So
in
that
sense,
that
is
still
a
bit
of
work
there
do
you
have
a
very
tightly
integrated
solution,
but
right
now,
the
way
that
we
have
it
is
that
we
still
rely
on
the
external
process
for
that
we
do
have.
So
there
are
two
models
of
this
right.
One.
One
of
the
models
that
we
just
saw
is
that
you're
taking
a
secret
and
you're
putting
it
in
the
kubernetes
secrets
right.
So
you
will
manage
that
secret.
The
same
way.
You
manage
your
image,
poo
secret.
C
There
is
another
model
that
we
have
looked
at
and
we
have
some
prototypes
on,
which
is
really
my
key
management
and
authorization
is
tied
to
the
host.
So
in
this
case,
what
we're
doing
is
we
have
a
prototype
that
actually
injects
this
in
the
runtime,
so
there's
validation
within
a
key
management
service
to
the
node,
so
it
does
attestation
the
node
to
ensure
the
it's
authorized
to
obtain
a
key,
and
this
you
know,
ties
back
to
rule
of
trust
like
TPM
or
something,
and
for
that
we
did
a
verdict.
A
Yeah
and
to
add
into
his
you,
you
can
use
kms
services
with
existing
kubernetes
tickets,
and
this
is
another
type
of
secret.
So
if
you
can
tie
up
your
kubernetes
to
your
key
service
provider,
all
your
keys
get
instead
of
they
staying
in
a
master,
they
they
get
pushed
to
the
your
choice
of
key
key
kms.
So
similarly,
this
is
not
any
different.
This
is
just
another
type
of
secret,
so
this
will
transparently
get
get
attached
to
the
kms.
A
B
C
C
This
is
something
that
we
haven't
measured
in
the
wall,
but
theoretically
it
shouldn't
take
up
that
much
resource,
so
the
way
that
we've
with
the
implementation
that
we
have
uses
a
streaming
type
cipher.
So
in
terms
of
the
encryption
process,
it's
part
of
the
pipeline.
So
although
they
can
be
like
it's,
the
latency
is
very
minimal,
but
the
band
which
is
still
gonna
be
about
the
same
yeah.
E
C
E
A
Need
to
do
this
only
once
when
the
first
time
image
comes
in,
because
the
continuity
will
extract
it
all
in
a
runtime
will
extract
it
and
keep
it
on
a
host.
So
it's
not
something
which
gonna
get
executed
get
Connecticut.
Every
time
you
run
the
container.
This
is
only
first
time
the
images
has
to
be
pulled
on
the
host,
and
second
thing
is
the
way
we
have
implemented.
Is
we
next
time
there
is
a
request
to
request,
comes
to
deploy
our
encrypted
image?
We
do
not
download
again
a
week.
C
If
you
so,
there
are
two
concerns.
One
is
like
the
latency
concern
which
I
don't
think
we're
that
worried
about,
because
the
pool
latency
is
still
a
lot
more
than
decryption
latency
totally.
The
other
resource
concern
you
may
be
worried
about
is
maybe
you
know
how
much
actual
computation
this
and
that's
why
I
like
in
our
design.
We,
a
bulk
of
the
encryption,
is
symmetric
encryption
which
isn't
really
that
it's
pretty
efficient
yeah.
Thank.
E
F
C
C
So
there
are
two
main
ways
that
we
kind
of
recommend,
one
of
which
is
you
could
actually.
So
you
know
when
you
saw
the
rap
Keys
earlier,
you
know
I
create
the
right
key
forever
secretary
for
our
show.
So
what
you
can
do
in
the
creation
of
the
devil
pipeline
is
that
you
can
say
that
if
I
trust,
a
certain
service,
for
example,
I
have
a
vulnerability
scanning
service
right,
so
I
say
this
vulnerability.
Scanning
service
has
a
particular
keypad,
so
I
can
say
that
I'm
gonna
encrypt
a
rap
key
for
this
service.
C
So
if
I
trust
the
service,
a
service,
suppose
that
image
should
be
able
to
decrypt
it.
The
other
way
to
do
it
is
we're
trying
to
see
whether
people
are
setting
to
move
the
power
of
the
T
scans
before
I
to
go
to
the
registry,
which
there
advantages
and
disadvantages
like
it's.
True,
though
yeah,
but
no
matter
both
ways,
we
have
models
that
what
would
be
able
to
do
solve
the
issue.
G
C
So
what
we
do
in
this
is
that,
generally,
when
you
encrypt
the
image
you
when
you
create
the
right
keys,
you're
gonna
create
one
for
yourself,
so
if
you're
gonna
be
able
to,
if
you
want
to
add
a
recipient,
let's
say
you
want
to
add
a
new
recipient.
You
don't
have
to
re-encrypt
the
image.
Basically,
what
you're
gonna
say
is
I
want
to
add
this
recipient.
So
what
it's
going
to
do
is
it's
going
to
decrypt
your
own,
wrap
key
and
then
rewrap
a
right
key
for
the
new
person.