►
From YouTube: Caller ID in Kubernetes - Michael Danese, Google
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Caller ID in Kubernetes - Michael Danese, Google
Kubernetes lets every Pod have a Service Account identity, but previously, that identity has only really been usable to authenticate to the Kubernete API server. The newly-released pod proof of identity can be used to securely establish trusted communication between workloads running in a Kubernetes cluster, to external services, or even to bootstrap the identity of higher level services like Istio. This talk will delve into the design of workload identity in Kubernetes and how you can use it to build more secure systems.
https://sched.co/MPdl
A
All
right,
it's
4:45,
I,
think
it's
time
to
get
started.
I'm,
Mike,
Denice,
I'm
gonna
give
a
talk
about
caller,
ID
and
kubernetes.
Today,
a
little
bit
more
about
me,
I'm,
a
software
engineer
working
on
gke
the
past
four
years.
Currently
the
TL
of
gke
identity
and
I'm
also
active
in
the
kubernetes
special
interest
group
for
authentication
and
authorization.
So
the
first
topic
I
wanted
to
touch
on
is
user
data.
It
plays
a
crucial
role
in
many
amazing
applications
that
provide
tremendous
value
value
to
customers.
A
However,
when
a
user
hands
over
data
to
a
to
a
company,
they
do
so
with
an
expectation
of
privacy
and
expectations
on
how
that
data
will
be
accessed
and
as
application
developers.
It
is
our
responsibility
to
uphold
those
expectations
which
brings
us
to
the
problem
that
I
want
to
cover
in
this
talk.
So
how
can
we
create
an
environment
that
maintains
a
sufficiently
high
level
of
assurance
on
user
data?
So
what
is
sufficient
here?
What
is
assurance?
A
A
Running
on
kubernetes
will
add
some
requirements
along
the
way
in
the
spirit
of
better
protecting
user
data
and
we'll
see
how
those
requirements
can
be
met
with
solutions
built
into
and
on
top
of
kubernetes,
but
before
we
get
into
our
game
of
telephone
I
wanted
to
cover
just
a
couple
concepts
that
I'm
going
to
use
refer
to
throughout
the
the
first
distinction
that
I
want
to
make
between
coarse-grained
authorization
and
fine-grained
authorization.
So
coarse-grained
authorization
is
a
broad
privilege,
independent
of
the
object
targeted
by
the
request.
A
A
Noah's
Ark
or
if
that
was
a
book
cool.
The
second
distinction
that
I
want
to
make
is
around
credentials.
I
want
to
I'm
going
to
talk
about
channel
bound
credentials
which
are
bound
to
the
channel
that
they
received
on
TLS
is
an
example
of
a
channel
bound
credential,
where
it's
bound,
where
the
channel
is
the
TCP
connection
that
the
TLS
session
is
established
on
and
then
on
the
flip
side
there
are
forwardable
credentials
which
are
credentials
are
not
bound
to
the
channel
in
which
they
arrived
on.
So
an
example
of
this
is
a
bearer
token.
A
You
can
send
the
Baird
token
down
a
TCP
connection,
close
that
connection
and
open
another
one,
send
the
Barrett
up
and
down
it
again.
We
call
these
forwardable
credentials
because
the
person
that
you
send
it
to
can
turn
around
and
forward
it
to
another
party.
If
that
other
party
accepts
that
same
bearer
token,
and
the
last
thing
that
I
want
to
discuss
before
we
hop
into
our
game,
is
direct
authentication
versus
delegated
authentication.
A
So
direct
authentication
happens
when
a
server
authenticates,
the
approximate
or
adjacent
requester,
or
the
peer
delegated
authentication
happens
when,
when
a
server
authenticates
a
requester
and
not
by
the
direct
direct
requester,
but
by
the
of
the
request,
so
an
example
of
a
delegated
authentication
would
occur
if
a
back-end
authenticates
a
end-user
that
made
a
request
that
traversed
through
a
friend,
so
that's
a
delegated
authentication.
So
now,
with
that
background,
we
went
to
that
Bret
rather
quick,
but
hopefully,
as
we
go
through
our
scenario,
some
of
these
concepts
will
be
clear
with
examples.
A
So,
let's
hop
into
our
game
of
telephone
I
want
to
introduce
back-end.
It
sits
inside
a
secure
production
network.
It
has
a
lot
of
interesting,
sensitive
data,
but
it's
not
very
interesting
when
it's
all
by
itself.
So
if
we
zoom
out
a
little
bit
we'll
see
that
it
lives
alongside
a
couple
of
other
services,
we
have
a
service
front-end,
which
is
responsible
for
serving
the
front-end
of
my
application,
and
we
have
a
widget
service
which
is
responsible
for
serving
widgets
and
so
far
the
front-end
and
the
widget
service
can
communicate
freely
with
the
backend.
A
We
write
this
all
down
in
the
design
dock
and
submit
this
design
back
to
our
security
team
for
review,
and
they
come
back
to
us
and
say:
hey
widget
service
really
has
no
reason
to
be
able
to
talk
to
back-end.
Back-End
holds
a
lot
of
sensitive
data
and
widget
doesn't
need
access
to
any
of
it
in
order
to
perform
its
function
so
enter
the
first
requirement
of
this
talk.
A
We
think
that
a
possible
Avenue
for
implementing
this
feature
request
is
to
implement
some
form
of
caller
ID
within
our
service
stack.
So,
let's
embark
down
that
journey
as
it
turns
out.
Kubernetes
has
a
built
in
concept
of
a
service
account
of
a
service
identity.
Kubernetes
service
accounts
are
a
resource
type,
they
live
in
a
namespace
and
each
namespace
has
a
default
service
account.
I'll
put
all
pods
are
already
running
as
a
service
account
they
integrate
with
access
control.
A
A
So
the
the
tokens
are
the
built
in
credential
credentials
provided
to
services
to
service
accounts.
They
are
exposed
to
pods
via
keyboard
volume.
They
are
flexibly
verifiable.
They
can
be
verified
either
centrally
by
communicating
to
the
token
Review
API
or
by
distributing
a
verification
key,
and
you
can
do
a
standard
jot
verification
against
these
tokens.
They
are
revocable
via
the
API
to
an
extent
and
then
the
last
three
bullets
are
true
of
the
new-style
service
count
against
our
beta
in
112
kubernetes.
A
They
have
limited
TTL
with
automatic
rotation
and
they
support
audience
scoping
and
they're
never
stored
in
NCD.
So
they
have
a
lot
of
good
good
security
properties
that
make
them
seem
like
they
might
be
useful
for
our
use
case,
there's
quite
a
complicated
flow
that
goes
into
making
these
tokens
available.
That
starts
on
pod
creation.
The
cubelet
will
request
a
token
from
the
api
server
and
it
will
continue
to
request
tokens
to
the
from
the
api
server
throughout
the
lifetime
of
the
pod
and
we're
going
to
make
sure
that
those
are
refreshed.
A
This
is
a
pretty
complex
diagram,
I'm
not
going
to
step
through
it,
because
most
of
this
is
handled
by
the
infrastructure
and
what
you
really
have
to
care
about.
Is
your
front
end
and
your
back
end.
So
your
front
end
reads
a
token
from
disk
presents
it
to
your
back-end
and
back-end
authenticates
that
token
and
possibly
returns
a
response
based
on
the
policy
enforcement
policy.
A
However,
there
is
a
major
downside
to
this
approach,
so
tokens
are
an
example
of
a
forwardable
credential,
so
they
may
be
replayed.
What
this
means
is
that
you
actually
have
to
be
very
careful
with
who
you
send
these
tokens
to.
The
other
thing
that
we
haven't
solved
is:
is
server
authentication
when
front-end
dials
back-end
it
wants
to
be
fairly
certain
that
it's
talking
to
back-end
and
not
something
else
and
I.
Just
we
would
like
to.
We
would
rather
not
have
to
trust
trust
the
network
for
security
properties
like
this.
A
A
So
if
we
move
forward
a
little
bit,
we
can
also
provision
mutual
TLS
certificates
for
service
accounts.
These
are
awesome
because
they
provide
both
server
authentication
and
they
are
channel
bound,
meaning
we
don't
have
to
worry
about.
We
don't
have
to
worry
about
these
credentials.
Being
replayed.
Kubernetes
does
have
support
for
a
certificate
provisioning
via
the
certificates.
Api,
it's
very
flexible,
but
it
requires
I
would
say
substantial
integration
to
get
working
for
your
use
case.
A
So
there's
not
an
easy
there's,
not
an
easy
way
to
get
to
this,
where
we're
only
worrying
about
the
same
thing,
we're
only
worrying
about
the
things
that
we
care
about.
However,
if
we
do
introduce
Sto,
it
actually
provides
a
lot
of
a
heavy
lifting
for
you.
So
sto
has
a
component
called
citadel,
which
is
the
CA
component.
It
provides
an
API
to
exchange
a
community
service,
account
token
for
an
M
TLS
certificate.
A
So
this
sounds
pretty
good
for
now.
Let's
apply
it.
Let's
apply.
Let's
apply
this
to
our
game
of
telephone,
so
armed
with
mutual
TLS
to
secure
services
service
communication,
we
can
implement
some
enforcement
policy
in
the
backend
that
says
the
front-end
identity
is
allowed
to
call
back
in,
but
the
widget
service
is
not,
and
this
looks
pretty
good.
A
We
can
move
on,
however,
the
we
were
still
a
little
bit
too
narrowly
focused
we're
still
focused
only
on
her
internal
network
and
as
we
zoom
out,
we
noticed
that
not
everything
is
in
containment.
Tucked
inside
our
internal
network.
We
actually
have
a
gateway,
which
is
the
conduit
between
users
and
our
services.
This
is
awesome
because
it
means
that
the
front-end
and
the
widget
service
in
the
backend
can
provide
value
to
customers
that
sort
of
a
broad
network.
However,
it
does
create
some
new
challenges.
A
The
security
team
zoomed
out
a
little
bit
too,
and
they
gave
us
a
new
feature
requirement.
This
new
future
requirement
is
slightly
different
than
the
first
one.
They
want
to
model
our
authorization
policy.
It's
such
that
only
user,
a
can
can
access
user
a's
data,
so
this
is
an
example
of
a
fine-grained
authorization
policy,
but
this
presents
a
little
bit
of
an
issue
because
user
a
talks
to
the
gateway
where
a
user
a
is
authenticated
and
the
attributes
that
come
out
of
that
authentication
actually
have
to
are
the
input
to
the
policy
of
the
backend.
A
A
However,
sto
will
is
currently
working
on
a
solution
for
you.
Sto
is
developing
this
concept
of
a
request.
Contact
context:
token:
it's
all
designed
implementation
is
ongoing.
It
is
pretty
alpha,
so
it's
not
something
that
you
can
use
when
you
leave,
but
I'd
like
to
describe
some
of
the
theory
behind
it
to
you
today.
A
The
user
sends
potentially
a
barrier
token
or
some
other
form
of
end
user
authentication
to
the
gateway
gateway
sends
that
their
token
and
some
request
context
to
the
token
service
token
service
return
returns
an
RC
token.
A
gateway
passes
that
RC
token
in
a
side
channel
down
the
down,
the
service
call
stack
in
an
out-of-band
channel,
an
RC
token
verification
key
is
pushed
to
all
applications
running
in
the
cluster
so
that
they
can
verify
those
n
company,
RC
tokens
and
Trust
these
trusts
that
those
assertions
were
made
by
the
Gateway
in
the
token
service.
A
So
if
we
hop
back
to
our
game
of
telephone,
we
now
have
a
user
authenticates
to
the
gateway.
The
Gateway
creates
an
RC
token.
Those
RC
tokens
are
forwarded
upstream
and
back-end
is
able
to
validate
that
RC
token
and
use
those
attributes
to
feed
its
policy.
So
now
we
can
implement
the
policy
that
user
a
is,
has
permission
to
access
user
a's
data.
A
This
is
pretty
good.
This
feels
like
this
feels
like
we're.
Our
security
team
wants
to
be
I
want
to
dig
into
that
final
authorization
policy
that
we
had
in
the
last
slide
and
rephrase
it
a
little
bit.
So
a
rephrasing
of
that
is
with
proof
of
recent
interaction
with
user
a
the
front-end
can
access
user
ace
data,
so
we've
we've
created
an
explicit
requirement
that
the
access
to
user
ace
data
is
always
on
behalf
of
user.
A
A
in
this
specific
scenario
we're
actually
requiring
that
the
access
take
taken
on
behalf
of
the
user
must
be
in
response
to
a
recent
action
taken
by
the
user
and
we're
also
requiring
that
the
access
err,
the
direct
access.
Our
proves
that
this
interaction
occurred.
So
we've
basically
remodeled
our
access
management
story
around
the
idea
that
the
user
owns
their
own
data,
but,
furthermore,
using
those
coarse-grained
authorization
policies.
A
So
the
future
creep
never
ends.
You
may
want
to
restrict
service
calls
I
that
originated
in
your
proud
BBC.
You
may
want
to
restrict
service
requests
based
on
a
code
identity
of
the
service.
We
talked.
We
didn't
talk
about
this
persona
at
all,
but
there
is
also
an
administrative
persona
may
be.
The
administrative
persona
does
not
need
proof
of
user
interaction.
To
read
to
access
to
surveys,
data
may
be
just
a
justification
in
the
form
of
a
support
ticket
or
a
bug.
Id
is
sufficient
and
you
also
might
want
to
require
that
requests.
A
Another
side
effect
of
of
this
approach
is
now,
if
we
record
that
who,
what
when
and
why
all
these
accesses
occurred,
we
can,
with
with
the
improvements
that
we
made
of
our
authors
in
our
authorization
system.
The
context
of
why
these
accesses
are
occurred
are
actually
much
more
meaningful
and
significant
and
actionable
it's
incredibly
useful
when
you
need
to
do
forensics
to
figure
out
why
something
happened
when
things
go
south.
A
A
A
The
the
reshift
of
this
authorization
model
to
to
include
both
fine-grained
authorization
that
takes
into
account
the
user
and
also
coarse-grained
authorization
to
control
the
flow
of
user
authority
through
our
system
actually
has
a
actually
moves.
The
responsibility
of
data
protection
on
to
individual
sres
service
owners
administrators,
as
as
these
teams,
as
these
teams
develop
code
or
debug
problems
in
production,
they
make
decisions
that
are
conscious
of
the.
A
Did
we
solve
it
with
authentication?
The
bad
news
is
probably
not
it's
a
pretty
complex
problem.
It
requires
a
holistic
approach
and
sustained
diligence
and
you're
never
going
to
have
a
perfect
solution.
However,
we
did
laid
the
foundations
for
this
holistic
approach
by
allowing
granular
authorization
which
enables
lease
privileged
access
and
a
complete
audit
history,
which
builds
in
an
environment
of
accountability
within
the
within
organization.
A
So
in
closing,
I
wanted
to
talk
about
what
I
look
at
when
I
evaluate
solutions
in
this
space,
I
like
to
see
solutions
that
are
easy
to
adopt
hard
to
use
and
correctly
and
noisy
to
circumvent,
applied
consistently
across
the
infrastructure.
This
usually
means
that
it
is
a
core
service
provided
by
the
infrastructure
built
on
open
standards
and
easy
to
develop
an
annex
and
extend.
A
So
the
other
thing
that
I
wanted
to
touch
on
is
what
belongs
in
kubernetes
core
and
probably
not
a
lot
of
this
right
now.
We
want
to
build
extension
points
that
allow
experimentation
outside
of
core.
We
also
do
want
to
adopt
features
that
improve
and
harden
our
core
infrastructure.
We
have
to
actually
move
very
cautiously
because
anything
that
we
incorporate
into
core
kubernetes
will
be
stuck
with
for
a
long
time,
including
any
mistakes.
So
that
brings
us
to
the
call
to
action.
What
can
you
do
when
you
leave
this
room?
A
Continue
thinking
about
this
problem
start
from
the
basics.
This
isn't
going
to
happen
overnight.
If
you
don't
have
service
identity,
it's
a
great
great
spot
to
start.
If
you
decide
that
you
need
want
to
embark
down
this
path,
please
give
us
plenty
of
feedback
of
what
works
well
and
what
doesn't
work
and
what
we
can
do
better.
A
I
also
want
to
point
out
that
there
are
a
few
communities
that
are
doing
great
work
in
the
space
Sagat,
which
I
am
involved
in
spiffing
inspires,
are
building
great
identity
solutions
and
the
sto
security
working
group
developed.
Many
of
the
features
that
I
discussed
today,
built
on
top
of
kubernetes
and
they're
working
hard
to
build
a
platform
that
integrates
very
smoothly
with
kubernetes
and
that's
it
and
I
think
we
have
some
time
for
questions.
A
Think
that
it's
going
to
be
a
requirement
to
to
support
these
in
some
capacity,
especially
not
not,
every
application
is
going
to
be
greenfield.
We
need
to
have
support
for
migrating,
so
we
talked
about
a
gateway.
Kerberos
is
often
used
for
administrator
identity.
That
gateway
should
probably
be
able
to
authenticate
using
Kerberos
and
support
ad
for
Kerberos
to
services.
A
First
Kerberos
for
service
identity,
I
think
that's
a
less
of
a
clear
space
for
me,
but
I
think
that
it
is
important
we're
just
looking
for
we're
looking
for
users
and
use
cases
that
need
that
and
people
that
are
motivated
to
start
to
sketch
out
what
a
solution
would
look
like
we're
happy
to
build
in
the
extension
points
that
are
needed
for
that
and
explore
it.
We
just
don't
have
the
motivation
yet.
C
It's
a
nice
idea
to
build
things
up
well,
layer,
well,
am
have
you
compiled
a
sort
of
matrix
of
that
feature,
creep,
use
cases
lists
which
build
up
layer
by
layer
to
show
what
solutions
can
be
applied
at
each
layer,
so
you
could
from
that,
derive
your
security
architecture.
That
fits
your
particular
needs
and
it
would
probably
help
a
lot
of
people
to
start
and
see
which
layers
combine,
which
are
orthogonal
to
each
other
and
stuff
like
that.
Right.
A
Yeah
I
I
think
that
there
were
naively
I
can
probably
categorize
three
without
thinking
about
it.
There
was
the
layer
where
we
could
only
authenticate
to
the
kubernetes
control
plan
to
get
certificates
because
we
don't.
We
could
only
trust
the
kubernetes
control
plane
and
there
was
when
we
established
service
to
service
identity,
and
then
there
was
the
layer
where
we
started
to
plumb.
A
D
A
Maybe
I
didn't
call
that
out
the
new
proof
of
identity
was
the
service
count
token.
So
in
112
there's
a
two
new
features
called
the
token
projection
which
most
of
the
security
properties
that
were
improved
were
actually
properties
of
only
the
new
service
grant
again.
So
those
are
automatic
rotation
time
that
time,
bounding
and
also
audience
counting
and.
A
E
A
A
Yes,
so
the
Gateway
is
responsible
for
verifying
that
the
token
is
a
valid
user,
cookie,
the
Gateway,
the
system
of
the
token
service
in
the
Gateway.
So
the
token
service
is
actually
the
Gateway,
for
example,
sends
headers
and
source
IP
to
the
Gateway
and
the
Gateway
does
sorry.
The
Gateway
sends
headers
and
source
IP
and
some
other
requests
ambient
request,
attributes
to
the
token
service.
F
A
So
it
could
be,
you
could
use
OAuth
coming
into
the
Gateway.
However,
the
the
RC
token
is
not
a
barrier
token.
It's
is
less
powerful
because
it
because
it
can
only
be
presented
by
specific
peers.
So
that
was
the
idea
that
you
control
the
flow
of
the
RC
token
through
the
system,
so
they're,
not
actually
usable
generally,
like
a
bearer
token,
is
so
a
bearer
token
is
defined
as
something
that
any
client
can
use
without
proof.
Cryptographic
proof
does
that
make
sense.