►
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Deep Dive: Container Identity WG - Greg Castle & Michael Danese, Google
Over the past year the Container Identity working group has been working on a number of initiatives relating to identity in Kubernetes. These include providing a mechanism to issue scoped JWTs that can be externally validated which improves the security of identity integrations using Kubernetes service accounts, such as Hashicorp Vault. We’ve also made significant progress in providing a new mechanism to issue and mount service account identities inside the cluster that addresses a number of security and scalability issues with existing service accounts. Finally we’ve also enabled new identity integrations by exposing OIDC functionality from the Kubernetes cluster. We’ll discuss these changes, how they can be used today, and where we are headed next.
To learn more: https://sched.co/GreS
A
Hi
everyone
thanks
for
coming,
my
name
is
Greg
and
one
of
the
container
identity
working
group
leads
and
from
Google
and
with
me
today.
I
have
Mike
Denis
his
cigars
chair
and
is
also
been
leading
their
identity,
effort
kubernetes
and
has
written
most
of
the
code
that
we're
talking
about
today.
So
first
thing
we
need
to
kind
of
get
out
of.
The
way
is
that
the
container
identity
working
group
actually
doesn't
exist
anymore.
This
presentation
is
a
lie.
A
Don't
leave
please
so
we
turned
it
down
in
October
of
this
year
and
we
basically
just
folded
it
back
into
sigilyph
so,
and
that
was
just
because
we
didn't
need
need
the
extra
meeting.
We're
definitely
not
done
with
identity
work,
and
this
is
essentially
working
as
intended
for
working
groups.
So
we
stood
up.
A
So
we're
gonna
review
some
identity,
use
cases
and
current
solutions.
I'm
gonna,
do
a
deep
dive
on
the
new
community
service,
account
tokens
and
talk
about
future
work.
So
I've
broken
these
use
cases
into
kind
of
three
buckets.
These
are
not
all.
I
did
comprehensive
identity
use
cases,
but
it's
a
sampling
of
some
of
the
stuff
that
comes
up
regularly
and
the
first
one
is
user
identity,
and
essentially
people
just
want
to
have
one
place
that
uses
a
managed.
A
So
it
hooks
in
with
your
kind
of
payment
systems
and
and
payroll
and
HR
and
whatever
else-
and
you
just
want
that
all
in
one
place,
then
you
also
have
workload
ID
identity,
which
is
identity
for
applications
running
inside
your
cluster.
You
might
use
the
same
system
as
you
use
for
users.
You
might
use
a
different
system.
A
You
might
want
to
have
identity
if
they're
talking
between
services
inside
a
cluster
or
between
different
clusters,
and
you
might
have
a
identity
for
a
workload.
That's
different
for
an
identity
for
a
sidecar,
so
you're
monitoring
container
might
have
a
different
identity
than
what
your
main
application
has
and
then
outside
into
the
cluster.
A
really
common
one
is
your
CI.
Cd
system
needs
to
talk
into
the
cluster
and
there's
kind
of
many
other
categories
of
those
as
well
too.
A
So
that's
kind
of
like
a
rough
overview
of
use
cases
we're
mostly
focusing
on
the
second
one,
the
second
block
there,
the
workload
identity
for
this
talk,
but
I
kind
of
over
the
past
few
days,
I've
had
quite
a
few
conversations
about
user
identity
and
I
realize
a
lot
of
people,
especially
those
who
are
just
starting
out
with
kind
of
integrating
kubernetes
into
their
environment,
is
still
figuring
out
this
piece,
so
I
thought
I'd.
Just
briefly
cover
your
options
there
and
the
solutions.
A
A
A
What
happens
here
is
you
writing
Authenticator
that
understands
whatever
your
custom
identity
system
is,
and
then
you
write
a
plug-in
for
Q
control
that
when
it
needs
an
identity,
it'll
it'll
exec
out
shell
out
effectively
and
get
back
a
credential
to
use
that
your
Authenticator
understands,
and
this
is
actually
how
gke
works.
Although
we
kind
of
wrote
our
wolf
provider
for
Q
control
before
this
extension
mechanism
existed,
but
that's
effectively
how
Google
identity
gets
integrated,
also
available
x.509.
A
So
it's
password
static,
tokens
and
I
wanted
to
call
out
a
nanny
pattern
that
I've
liked
occasionally
hit
when
talking
to
cluster
operators
and
people
new
to
kubernetes
is
that
people
just
creating
a
community
service
account
exporting
the
key
and
then
sharing
that
amongst
multiple
users.
It's
a
really
bad
idea,
if
you
find
yourself
tempted
to
do
that,
it's
basically
the
same
as
like
writing
down
your
password
and
handing
it
out
to
people.
So
don't
do
that.
A
Do
one
of
the
things
that's
in
in
the
top
section
of
the
bullets
here,
so
switching
over
to
workload,
identity,
I
should
say
too.
We've
probably
got
about
15
20
minutes
of
content
here,
so
should
be
a
fair
bit
of
time
for
questions.
If
people
have
questions
so
what
are
the
goals
of
a
workload
identity
system?
We
want
to
get
identity
to
applications
with
very
little
developer
effort.
We
want
to
make
sure
that
we
can
handle
multiple
external
systems.
We
want
to
minimize
how
bad
a
compromise
or
a
leak
of
a
credential
can
be
leaking.
A
One
credentials
should
give
you
broad
access
to
the
whole
system.
Those
credentials
should
have
limited
lifetime
or
to
rotate
and
ideally
be
non
exploitable
if
we're
possible.
So
if
you
have
access
to
something
like
a
trusted
platform
module,
then
you
should
be
able
to
like
hold
your
key
somewhere,
where
it's
really
hard
for
an
attacker
to
get
at
it.
A
A
bunch
of
external
efforts,
I'm
not
going
to
cover
in
any
sort
of
detail,
but
I
kind
of
here
for
reference
in
case
you're
new
to
this
space
spiffy
inspires
x.509
identity.
History
is
also
x.509
identity.
There
is
a
vault
integration
for
kubernetes
that
uses
tokens,
and
that
will
actually
be
the
security
of
that.
A
What
we
had
to
do
was
creating
a
new
service
account,
move
everything
on
to
the
new
service
account
and
then
delete
secrets
for
the
old
service
account
and
that
sort
of
stuff
is
way
more
painful
than
it
should
be
because
of
kind
of
the
design
of
this
system.
The
permissions
are
too
permissive.
They
shouldn't
be
users,
don't
all
users
don't
need
broad
access
to
the
actual
file
on
disk
there's
no
audience
binding,
which
Michael
cover
in
his
section
so
I'll
hand
it
over
to
Mike
to
talk
about
so.
B
I'm
gonna
go
over
the
new
provisioning
flow
for
kubernetes
service
account
tokens.
Essentially
what
happens
when
a
cubelet
is
directed
to
start
a
pod
running
as
a
service
account
before
it
starts
that
pod
it'll
make
a
request
to
the
API
server.
For
a
token,
the
API
server
issues
that
token
and
the
couplet
will
then
expose
that
token
to
the
application
running
inside
the
pod
via
a
volume.
B
That
is
basically
the
flow
from
2017.
However,
one
change
is
that,
because
these
tokens
now
expire,
people
is
also
responsible
for
periodically
refreshing
than
these
tokens
and
keeping
them
up
to
date.
So
what
does
that
mean
for
your
application?
So,
with
these
new
tokens
there
audience
bound,
meaning
you
can
ask
for
a
separate
token
to
communicate
with
vault,
and
you
don't
have
to
use
your
unbounded
token
that
you
had
in
2017.
They
have
limited
time
to
live.
Rotation
is
easy.
B
You
can
just
provision
tokens
for
30
minutes
or
an
hour
and
wait
wait
for
that
hour
to
expire,
and
you
don't
have
to
worry
about
rotation,
but
in
cases
where
you
want
to
do
a
fast
revocation
of
all
tokens,
you
can
redeploy
your
application
or
deployment
and
that
that
will
trigger
a
rolling
update,
and
these
tokens
are
invalid
once
the
pods
that
they
were
provision,
two
are
no
longer
running
the
secret.
These
tokens
are
no
longer
stored
in
secrets,
so
in
2017
we
use
the
secret
mechanism
to
distribute
these
tokens.
B
We
have
some
very
powerful
controllers
with
broad
read
access
which
includes
all
secrets
in
the
cluster,
so
that
essentially
gives
some
controllers,
like
the
ingress
controller.
In
order
to
read
TLS
secrets
could
also
read
the
service
account
credentials
for
every
every
application
in
the
cluster.
That's
just
because
the
way
that
the
kubernetes
api
exists
doesn't
really
lend
itself
to
secret
management,
people
that
also
handles
refresh
in
this
scenario,
kubernetes
client
libraries
that
we
maintain
as
part
of
core
will
begin
to
start
rereading
these
tokens.
B
So
this
is
already
done
for
client
go
and
the
Python
and
the
job
of
official
clients
will
follow
shortly.
So
there
are
some
potentially
incompatible
changes
to
this
new
token
volume.
There's
a
new
feature
introduced
in
113
called
bound
service,
account
token
volume
it's
off
by
default,
it's
alpha
currently
and
what
it
does.
Is
it
swaps
out
the
old
secret
volume
for
a
projected
volume
that
uses
the
new
token
projection?
B
So,
while
this
will
probably
be
transparent
for
many
applications,
certain
applications
doing
a
little
different
things
might
might
notice
a
hiccup,
namely
you
have
to
reread
the
token
periodically
because
it
will
change
out
from
underneath
you
if
you're
using
up-to-date
client
go.
This
might
be
handled
for
you.
However,
if
you're
doing
something
else,
are
you
using
old
client
libraries?
You
need
to
be
aware
of
this.
B
We
lock
down
the
file
permission
from
64
4
to
6
0
0.
So
if
you
were
running
an
application
as
a
user
that
is
non
root,
you
might
you
might
notice
this
and
run
into
this
and
get
file
permission
errors.
The
best
way
to
work
around
this
is
just
to
use.
The
FS
group
feature
in
the
pod
security
context
and
add
a
supplemental
group
to
your
application
so
that
it
still
has
permission
to
read
that
so
with
using
using
that
FS
group
feature
that
file
permission
will
change
to
6
4
0,
which
is
strictly
better.
B
It
only
allows
your
individual
application
running
as
an
unprivileged
user
to
access
that
credential
and,
in
the
case
of
a
container
breakout
other
applications
on
the
node
will
not
be
able
to
read
that
we
used
to
allow.
We
still
allow,
as
secrets
to
be
injected
into
environment
variables,
because
a
downwards
API
you
can
use
the
dollar
sign,
syntax
to
reference
a
secret.
B
We
have
disallowed
that
behavior
in
this
new
iteration
of
token
volumes,
just
because
those
environment
variables
often
leak
in
the
logs
in
unexpected
ways
and
for
for
for
a
number
of
their.
There
are
many
blogs
on
the
internet.
That
say:
don't
use,
don't
use
environment
variables
to
pass
credentials
to
applications.
B
Plot
security
policies
might
need
to
change.
If
you're
making
use
of
this
feature.
Instead
of
permitting
a
secret
volume,
you
not
have
to
permit
a
projected
volume
other
than
that
the
service
account
volume,
the
structure
and
the
location
of
it
on
inside
the
pod
is
unchanged,
so
it
so
has
that
a
CA
cert
still
has
the
tuggin
file,
and
it
also
has
the
the
namespace
file.
So
the
file
structure
is
preserved
in
these
new
token
volumes.
B
B
B
Please
give
us
feedback.
Please
turn
it
on.
If
you're
on
a
security
team,
please
communicate
to
people
that
are
using
service
account
tokens
and
make
them
aware
of
these
changes
if
you're
using.
If
you
would
like
to
use
service
account
tokens
to
authenticate
to
things
other
than
the
API
server.
For
example,
vault.
Please
try
out
the
new
tokens
and
see
if
it
works.
For
you
see
what
problems
it
solves.
B
So,
what's
next
for
us
now
that
we
have
these
new
tokens,
we
actually
encode
per
pod
information,
including
the
pod,
UID
and
pod
name.
So
we
can
actually
start
to
build
node,
scoped
authorization
systems.
So,
right
now
we
have
a
node
authorizer,
which
is
primarily
used
to
to
scope
the
permissions
of
node
identities.
So
we
can,
we
create
a
graph
and
we
we
do
authorization
based
on
pods
that
are
retina
running
on
the
notes.
B
So
what
was
not
covered
by
token?
This
token
work
is
any
form
of
server
authentication.
We
would
like
to
implement
some
sort
of
easy
server
authentication
for
kubernetes
components
that
are
part
of
core.
We
suggest
that
for
user
workload
applications
that
you
use
some
higher-level
system
like
spiffy
or
Sto.
B
However,
since
we
have
this
bootstrapping
problem,
we
need
to
make
sure
that
we
have
a
low
dependency
solution
for
server
authentication
and
we
want
to
authenticate
to
the
same
servers
as
I
mentioned
before
no
problem
detector
cubelet,
so
we're
exploring
mechanisms
for
auto,
approving
or
Auto
provisioning
serving
certificates
for
for
these
workloads.
And
then
the
final
step
is
are
the
final
piece
here
is
non
exploitable
credentials,
so
we're
working,
we're
thinking
about
ways
to
make
make
harden
these
tokens
and
potentially
allow
them
to
be
bound
to
keys
and
allow
those
keys
to
live
inside
TPMS.
B
C
B
This
is
not
written
down
anywhere
and
it
actually
just
came
out
of
some
discussions
that
we've
had
this
week
with
Clayton.
We
were
considering
namespace
level
opt-out,
so
maybe
an
harden
service
account
level
opt-out,
so
add
an
annotation
to
a
service
account
that
says,
do
legacy
token
provisioning
and
for
service
accounts
that
don't
have
that
annotation.
We
we
don't
create
that
secret.
B
Containing
the
token,
the
old-style
token
we
mountain,
the
new
protected
volume
in
situations
like
if
you
had
created
a
service
account
exported
that
key
and
used
it
in
your
CI
system.
You
would
add
that
annotation
and
you
would
that
would
preserve
the
behavior
and
maybe
at
a
certain
point
in
migration.
E
B
So,
revocation
works.
It
is
actually
only
supported
when
using
the
token
Review
API
so
in
any
aggregated
API.
Servers
that
do
delegated
authentication
is
to
do
a
token
review.
This
is
how
people,
it
does
token
authentication.
Basically,
we
encode
the
pod
name
and
pod
UID
in
the
tokens
that
we
provisioned
pods.
So
as
part
of
validating
that
token,
we
do
the
normal
ie,
C
job
validation,
and
then
we
make
sure
that
that
pod
is
still
running
so
revocation
is
just
to
delete
the
pod
and.
F
B
To
the
both
the
app
application
developer,
the
pod
spec
author
can
set
an
expiration
in
in
the
pod
spec,
and
a
cluster
operator
can
set
a
cluster
wide
soft
max
duration.
So
if,
in
an
author,
a
pot
spec
author
says
I
wanted
available
for
two
years,
the
cluster
operator
can
say:
sorry,
you
only
get
it
for
four
hours
and
will
just
pick
them
in.
B
Cubelet
runs
something
called
a
token
manager,
so
it
makes
sure
that
it
has
a
token
available
when
the
pod
starts
up
and
it
that
token
manager
knows
the
expiration
or
the
TTL
of
of
the
token,
and
it
has
some
threshold
where
it
will
begin
to
start
to
request
a
new
token
for
the
API
server
from
the
API
server.
That
threshold
is
either
50%
of
the
token
lifetime
all
the
way
up
to
one
day.
So
you
know
it's
not
configurable.
It's
either
50%
of
the
token
lifetime
or
up
to
one
day.
B
And
how
it
replaces
that
token
on
disk
it
uses
the
same
libraries
that
we
built
for
other
protected
volume
types.
So
it
will.
The
the
secret
volume
in
secrets
is
a
sim
link
to
a
directory,
so
we
rebuild
the
directory,
and
then
we
reread
sim
link
so
that
that
token
file
should
change
atomically
on
disk.
F
B
B
B
We
would
like
these
tools
to
enable
bootstrapping
of
more
feature
full
identity
systems
on
top
of
kubernetes,
so
history
already
supports
meshes
that
span
multiple
clusters
and
the
second
part
of
that
question
is
these
tokens
already
carry
an
indicator
of
the
origin
cluster,
so
the
issuer
is
actually
it's.
We
we
run
a
single
issue
or
per
cluster,
so
the
issuer
field
and
that
jot
the
issuer
URL
indicates
the
origin
cluster
of
that
token.
So
you
can,
you
can
start.
You
can
start
to
reason
about
origin
cluster
in
that
way,.
B
B
Lock
down
a
service
account
permission
per
cluster,
then
you
could,
alternatively,
just
share
the
issuer
across
many
clusters.
However,
sometimes
people
want
to
segregate
the
kubernetes
control
manager
and
cluster.
A
is
only
allowed
to
read
secrets
in
cluster
a
and
that
prevents
cluster
to
cluster
attack,
vector
escalation,
vector.
B
So
I
think
there
are
two
models
for
some
of
these
helper
components
like
a
scene,
I
plug-in
or
a
flex
volume.
Plugin
I
think
it's
reasonable
to
plumb
in
a
service
gun
token,
to
potentially
a
flex
volume.
If
the
Flex
volume
needs
to
use
the
authority
of
that
application
to
maybe
mount
a
remote
volume.
However,
an
alternative
approach
to
that
is
to
run
a
identity
for
the
CNI
component
itself.
In
that
situation,
it
isn't
using
the
application
identity
when
it
sets
up
the
network
for
that
component.
B
I'm,
not
sure
if
these
things
usually
require
Damon
said
I
believe
calcio
does
require
a
process
running
on
all
nodes
in
order
to
work
correctly.
So
I
would
just
give
that
Calico
node
agent,
a
its
own
service
kind,
identity
and
use
that
to
to
do
any
authentication
rather
than
using
the
applications,
identity.