►
From YouTube: Identity Bootstrapping in Multi-tenant Multi-cluster Kubernetes - Manish Mehta & Derek Suzuki
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Identity Bootstrapping in Multi-tenant Multi-cluster Kubernetes - Manish Mehta, Volterra & Derek Suzuki, The Voleon Group
With the increasing popularity of Kubernetes, providing managed K8s has been a great way to convert enthusiasts into adopters. However, current solutions mainly focus on providing isolated clusters and adopters are responsible for making workload identities work across clusters. If multi-tenancy is added to the mix, the challenges of bootstrapping identities that work across clusters - but within tenancy - are even greater. In this presentation, Manish Mehta will share challenges of securely bootstrapping identities in such a setup (especially when the individual clusters could be running in untrusted environments), the tradeoffs, and possible solutions. Manish will also introduce planned open-source components of a solution used by Volterra Edge Services for identity bootstrapping and other security services.
https://sched.co/UaZW
A
Alright,
you
know
about
30
seconds
left,
but,
let's
simply
start
with
the
introduction.
So
thanks,
first
of
all,
I
think
you
are
the
only
ones
who
braved
through
the
last
three
days.
See
you
see,
most
of
them
are
gone,
will
not
make
this
very
difficult
so
that
you
can
safely
leave
home.
My
name
is
Manish.
Mehra
I
am
chief
security
architect
at
Volterra.
We
do
some
cloud
cloud
cloud
computing
and
distributed
cloud
as
a
platform
edge
computing
platform,
stuff.
B
C
A
You
get
started,
I'm
gonna
kick
off
and
then
Derek
will
explain
whatever
we
discuss
here,
how
he,
how
this
kind
of
design
it
helps
him
solve
the
problems
that
the
volume
group
is
trying
to
solve.
So
very
quick
introduction,
not
a
sales
pitch.
So
we
are
a
SAS
provider.
We
aspire
to
enable
customers
to
build,
deploy,
secure
and
operate,
distributed,
applications
and
data
across
multiple
cloud
providers
and
edge
locations
all
right,
so
it
could
be
either
or
both
all
right.
A
So
I'm
not
going
to
go
into
a
lot
of
detail
earlier,
because
I
think
in
the
past
lot
of
blogs
they
have
been
written.
A
lot
of
you
know
folks
have
talked
about
how
why
multi
cluster
and
stuff
so
TL
DR
is,
if
you
are
not
running
multiple
clusters,
kubernetes
clusters
today,
it's
just
matter
of
when
it's
not
if
you
will
get
there
sooner
or
later.
So
whatever
we
going
to
talk
about
today
is,
although
came
to
light
in
our
design.
A
When
we
were
doing
the
multi
cluster
stuff,
it
doesn't
necessarily
mean
that
it's
not
applicable
to
signal
passes.
Most
of
them
are,
and
once
you
start
right
in
a
single
cluster,
you
can
easily
expand
into
multiple
clusters,
and
today
we
are
going
to
show
some
challenges
that
we
ran
into.
Of
course,
there
are
many,
and
we
are
going
to
today
focus
only
on
ISIL
when
you
have
multi-tenancy
in
our
case,
tenancy
actually
is
at
two
levels.
A
We
have
multiple
customers
and
within
every
customer
we
have
multiple
tenants,
so
let's
say
different,
be
use
or
something
so
how
the
how
the
isolation
works
and,
more
importantly,
how
the
security
works,
which
is
kind
of
some
buzzwords.
You
have
probably
heard
our
last
three
three
days
and
we'll
go
through
them
as
well.
A
So,
first
of
Amin,
as
we
stated,
the
goal
was
to
actually
be
able
to
deploy
a
secure
operate,
distributed
applications,
so
we
didn't
necessarily
have
to
use
kubernetes,
but
we
chose
kubernetes
because
of
the
following
reasons,
and
it's
pretty
extensible,
and
one
of
the
examples
that
you
will
see
today
for
identity
is
due
to
the
fact
that
it
is
actually
extensible
so
motivation
why
I
actually
wanted
to
have
this
identity
related
talk
to
begin
with
right.
So
the
reason
was
two
years
back.
A
I
was
on
this
stage
this
stage,
meaning
coop
con
in
Austin,
not
here,
but
I
was
talking
about
at
that
time.
I
used
to
work
for
Netflix
and
I
started
talking
about
authorization,
how
we
handle
authorization.
However,
what
happened
between
then
and
now
multiple
people
started
talking
about
hey
you're,
using
OPA
I
think
this
was
the
first
presentation
OPA
had
at
least
in
pucon.
If
I
remember
correctly,
that
started
using
for
authorization.
The
the
thing
was.
A
lot
of
people
were
very
interested,
it's
a
great
product,
but
they
did
not
know
how
to
get
there.
A
So
they
started
talking
about
odd
Z,
which
is
what
their
goal.
However,
they
didn't
have
a
solution
for
auth
end.
First,
so
I'm
like
you
need
to
figure
out
how
your
attendees
have
going
to
happen.
Then
you
can
go
to
all
Z.
Then
they
say:
ok,
fine,
so
or
ten
mins,
MT,
LS,
right
I'm
like
yes,
what
does
empty?
Let's
need
certificates
Oh.
My
problem
shifted
right,
so
the
problem
shifted
to
identity.
A
A
B
A
A
However,
this
is
dictionary
definition,
just
like
lawyers
like
to
put
stuff
in
their
dictionary
definitions,
don't
work
in
code
of
law,
so
this
definition
what
won't
work
in
code
of
crypto
mix,
so
you
have
to
have
a
crypto
geek
definition
with
all
the
crypto
stuff
in
interlaced
into
it.
What
this
is
trying
to
say
is
basically,
if
your
identity
is
not
solid,
it
is
not
coming
from
a
channel
from
a
protocol
that
is
secure
enough.
You
and
you
understand
all
the
way
down
to
where
the
keys
are
generated
and
stuff.
The
key.
A
The
identity
itself
is
weak.
If
identity
itself
is
weak,
your
foundation
of
security
is
weak
and
then
there
is
really
no
point
in
talking
about
OAuth,
nm,
TLS
or
Zee
on
top
of
it
all
right.
So,
let's
start
with.
Why
do
we
need
cryptographic,
identity?
So,
most
most
times
you
will
hear
MPLS
thrown
around
all
the
time.
Secure
communication,
em
TLS
short.
It's
just
one
reason
why
you
want
secure
identity.
That
means
you
want
cryptography
identity
that
is
useful
when
you
do
a
mutual
TLS.
However,
what
we
forget
is.
A
We
also
need
mutual
identity,
especially
in
case
cases
like
ours,
because
we
are
providing
a
platform
for
our
customers,
and
we
don't
make
any
judgment
about
what
kind
of
workload
they
will
be
running
on
that
platform.
They
may
be
using
their
identity
to
actually
a
test.
Something
say
something
about:
yes,
this
is
coming
from
me,
so
it
is
not
just
about
communication.
There
is
something
about
proving
eligibility
vindication,
its
audit
from
the
verifiers
perspective.
A
If
something
happen,
I
actually
have
to
keep
a
record
and
know
precisely
exactly
which
piece
of
code
make
that
call,
and
that
is
not.
Identity
cannot
be
lose
there
and
we'll
go
there
into
the
granularity
of
it.
So
what
makes
a
solid
identity?
So,
in
the
course
of
this,
you
know
designing
our
solution.
I
wanted
to
basically
bubble
this
up
and
basically
try
to
categorize
them
in
main
categories.
A
So
the
main
categories
I
came
up
with
these
are
three
things
which
are
granularity
of
the
identity,
how
granular
it
is
and
we'll
get
into
the
actual
examples
of
it.
That
means
everybody
has
heard
of
this
privilege.
You
know
principle
of
least
privilege
that
if
the
resource
side
of
people
don't
give
resources
more
than
what
somebody
needs,
however,
nobody
talks
about
the
granularity
on
the
identity
side.
That
means
give
only
give
the
access
only
to
people
who
actually
need
so
there's
a
difference
between
that
and
we'll
get
there.
A
Second
is
how
how
to
deliver
secure
your
identity
securely
and
mint
it
and
deliver
it
securely.
So,
for
example,
if
I
give,
if
I
told
you
you
know,
your
password
was
set
by
an
administrator,
and
he
knows
your
password,
you
cannot
change
it.
That's
not
securely
delivering
it,
because
somebody
knows
your
password
we'll
get
into
that,
and
last
one
is
rich
and
usable
identity,
which
means
whatever
identity
you
are
getting,
as
in,
in
particular
case.
In
this
particular
case,
we'll
be
talking
mostly
about
x.509
certificates.
A
What
does
that
expire
and
certificate
contain
and
what
what
makes
it
usable
or
not
usable
or
if
in
order
to
use
it,
what
kind
of
changes
you
will
have
to
make?
So
all
those
kinds
of
things
have
to
be
also
considered
all
right.
So,
let's
start
with
the
granular
one.
So
very
first
thing:
first,
when
we
started
at
least
I
started
reading
about
kubernetes,
this
is
the
first
thing
I
went
through
in
the
documentation.
A
Pods
are
the
smallest
deployable
units
of
compute
in
kubernetes
all
right,
so
if
they
are
the
smallest
deployable
unit
of
compute,
why
would
you
not
want
to
know
for
forensics
purposes,
auditing
purposes
of
what
not
exactly
what
pod
performed
a
particular
given
action?
If
you
see
most
of
the
solutions
talked
about
nowadays,
their
identities
are
either
based
on
namespace
when
I
and
we'll
get
into
namespace
its
service
account
its
service,
but
you
can
imagine
more.
All
three
of
them
basically
can
have
multiple
pods
that
kill
that
will
satisfy
this
condition.
A
Multiple
parts
can
be
in
one
namespace.
Multiple
parts
can
be
in
one
service
account.
Multiple
parts
can
be
attached
to
a
service,
so
the
identity
is
not
any
of
this
identity
is
actually
part.
So,
let's
start
with
them.
A
given
part
can
have
a
service
account.
A
given
part
can
be
attached
to
a
service.
A
given
part
can
be
in
a
namespace,
but
those
are
attributes
of
that
part.
It's
those
are
the
attributes
of
the
identity
of
that
part
right.
The
reason
I
say
this
is
identity.
A
Boundary
is
trust
boundary
some
years
back,
some
log
version
and
I
think
it
was
said
little
bit
differently,
like
identity
is
the
new
perimeter.
Some
some
of
you
may
have
heard
that
the
point
is
that
whatever
your
identity
boundary
is,
is
what
you
trust.
That
means
anything
outside
of
that
boundary
cannot
take
your
identity
and
do
something
without
your
knowledge
right,
so
because,
once
they
do,
that
means
you
are
trusting
them
to
act
on
behalf
of
you,
which
is
not
the
point.
A
A
If
a
pod
is
running,
somebody
else
is
minting
the
private
key
and
give
handing
off
to
you
not
a
great
solution
right
or
the
private
key
is
generating
generated
somewhere
outside
and
handed
off
to
you
right
or
is
put
in
a
location
which
is
actually
accessible
by
more
than
that
pod,
not
ideal,
because
somebody
can
actually
take
your
private
kid.
It's
called
private
for
a
reason
right,
so
you
have
to
read
think
about
all
these
things.
The
system
that
is
venting
the
private
key.
Is
it
outside
inside
of
the
the
system
that
is
maintained.
A
If
that
was
compromised,
what
is
the
blast
radius
right
if
the
CA
is
actually
sitting
right
on
the
cluster
right?
There
are
some
solutions
where
the
CA
is
actually
sitting
on
each
cluster.
So
if
I'm
running
thousand
clusters
today,
if
one
of
them
compromises
right
in
that
CA
can
be
used
to
mean
certificates
for
any
other
clusters.
A
That's
not
a
great
design
right,
so
yeah.
We
have
to
consider
all
those
things
and
then
we
also
have
to
make
sure
that
the
delivery
mechanism
of
your
identity
is
not
limiting
the
use
of
the
identity.
When
I
said
I,
don't
want
to
make
any
judgment
about
what
kind
of
workload
is
being
run
on
that
part.
What
it
means
is
I
cannot
necessarily
say
it's
an
HTTP
server,
or
even
it's
a
TCP
based
server.
It
could
be
running
something
that
needs
identity
for
IPSec.
A
So
as
a
platform,
at
least
at
the
point
of
a
platform,
those
kind
of
assumptions
are
out
of
questions,
so
some
people
actually
discussing
things
of
like
does
it
mean
that
your
thing
cannot
be
used
for
service
mesh?
Yes,
it
can
be
used.
What
I'm
saying
is
we
are
not.
We
are
not
tying
our
identity
to
any
particular
use.
So
in
this
particular
case,
if
your
delivery
mechanism
is
is
tied
to
some
particular
way
of
using,
for
example,
service
mesh,
what
is
being
talked
about?
It
is
tying
you
to
that.
A
Tcp
based
service,
HTTP
based
service,
but
guess
what
a
month
from
now
you
are
going
to
have
database
or
something
that
doesn't
use.
Http
protocol
sometimes
doesn't
even
use
TCP
protocol.
What
then
right
plus,
of
course
it
is
service,
it
may
not
even
be
a
service,
it
can
be
a
create
as
job
which
basically
comes
up.
Every
24
hours
and
dies
in
24
hours
it
comes
up
runs
for
10
minutes
makes
bunch
of
calls;
they
all
have
to
be
authenticated.
A
That
means
this
guy
needs
identity,
to
talk
to
someone
else,
but
the
guy
dies
is
not
a
service,
so
that
is
why
your
identity
is
not
tied
to
necessarily
the
service.
It
is.
The
pot.
Next
is
rich
usable
and
extensible.
So,
as
I
mentioned
in
this
case,
we
will
talk
about
x.509
certificate.
So
what
is
in
the
certificates?
I
document?
What
is
contained
in
the
certificate
is
very
important,
so
of
course,
in
regular
certificates.
You
are
very
used
to
seeing
DNS
names
right
in
case
of
kubernetes.
A
It
will
be
service
name
somehow
translated,
root,
DNS
names.
The
the
question
is
what
makes
a
good
set
of
identity
attributes
that
you
want
to
put
into
the
certificate.
The
answer
is
anything
that
you
want,
because,
ultimately,
if
it
is
your
identity
certificate,
you
want
to
be
able
to
control
what
goes
into
it,
so
that
you
can
prove
to
somebody
that
I'm,
a
mayor,
I
am
X,
X
years
old
or
I
have
black
hair
or
whatever.
A
So
what
attribute
I
want
to
prove
to
somebody
else,
I
get
to
decide
what
is
contained
in
my
identity
document,
so
we
get
to
how
you
can
control
those
things
and
also
I,
don't
want
to
put
it
in
a
way
that
I
have
to
make
everybody
else
change
their
libraries
so
that
how
to
parse
my
certificate
now
right,
because
it
looks
so
different
from
anything
that
is
standard
in
RFC's.
So
those
are
another
other
set
of
things.
A
Do
you
may
want
to
consider-
and
last
thing
I
would
say,
is
if
let's
say
a
pod
is
running
in
kubernetes
and
you
have
given
this
identity,
can
you
write
something
very
simple
kind
of
translation
service
where
the
identity,
although
it
is
coming
from
within
your
cluster
or
Multi
cluster
environment?
And
now
you
want
to
give
this
guy
I
am
credentials
from
AWS?
How
easy
is
it
to
first
take
the
natural
identity
and
then
convert
it
into
I
am
or
Google
or
wherever?
You
may
be
running
right
all
right,
so
this
is
a
busy
slide.
A
This
is
basically
what
Volterra
sort
of
shows
on
their
glossy
for
this
presentations
purpose.
I'll
try
to
simplify
this
slide.
The
point
is
that
there
are
lot
of
kubernetes.
Cluster
is
running
in
lot
of
different
environments,
and
these
are
also,
of
course,
online.
So
if
I
simplify
this
design,
you
can
the
diagram
you
can
imagine.
There
is
a
big
task,
back-end
that
has
all
the
controls,
control
panels
or
or
control
systems
from
Volterra
that
are
running
that
are
able
to
control
and
deploy
kubernetes
clusters
across
different
environments.
A
It
could
be
public
cloud,
any
public
cloud
provider,
it
could
be
your
own
private
cloud,
it
could
be
data
center,
edge,
location,
industry,
factory,
floors,
whatnot
right
all
at
once.
You
run
multiple
kubernetes
clusters
and
SAS
provider
basically
is
able
to
control,
configure
all
of
them
individually,
based
on
just
labels
right.
A
So
let's,
let's
see
how
this
works
and
if
you
take
example
of
one
we
will
be
able
to.
Basically,
just
extrapolate
and
generalize,
so
what
we
did
here
is
we'll
take
example
of
one
and
every
customer
in
on
Volterra
platform
gets
a
completely
separate
CA,
so
they
are
not
intermingle,
so
there
will
never
be
a
case
where
one
workload
from
customer
one
is
somehow
able
to
talk
to
customer
two.
A
So
here
what
we
do
is,
let's
say
there
is
a
kubernetes
api
server
and
we
first
install
something
called
voucher
which
basically
serves
two
purposes:
it's
a
web
hook,
which
is
a
mutating
web
book
hooked
into
the
pod
creation
process,
and
then
there
is
something
called
watcher.
The
watcher
is
basically
watching
for
some
objects
on
kay
kubernetes
api
server.
Now
this
is
the
initial
setup.
This
is
running
in
completely
different
namespace,
which
is
privileged
and
it's
considered
voltage
loss
platform.
So
customers
don't
have
any
of
this
access.
A
Let's
say
customer
schedule
support
and
the
apart
is
basically
consistent
of
one
consists
of
one
application
in
one
sidecar.
This
pod
creation
request
will
be
intercepted
by
the
web
book
and
it
will
automatically
inject
a
sidecar
called
wingman,
and
our
goal
is
to
basically
was
to
actually
open
source,
voucher
and
wingman.
As
part
of
this
talk,
it's
got
engineering-wise
it's
got
delayed.
The
point
is
that
wingman,
as
you
can
see,
is
actually
inside
the
pod.
It
is
not
sitting
outside
the
pod.
A
As
soon
as
that
happens,
wingmen
goes
to
identity
Authority,
which
is
the
backend
for
us,
as
or
wilderness
ass,
which
is
nothing
but
that
an
N,
specific
or
customer
specific
CA
and
gets
an
identity
back
from
their
some
details
are
skip.
Therefore,
in
the
interest
of
time,
but
if
you're
interested
you
can
come
back
and
talk
to
me,
the
the
point
is
that
that
part
now
got
this
certificate
and
the
there
are
two
main
aspects
of
this
certificate
to
be
remembered.
They
a
it
is
global
identity.
A
That
means
this
spot
certificate
now
can
be
used
across
all
clusters.
We
just
tested
three
thousand
clusters
last
month,
so
that
can
that
can
basically
have
a
huge
usefulness,
because
the
every
pod
can
talk
to
any
other
part
in
those
3,000
clusters.
As
long
as
your
policy
allows
the
point
if
the
identity
is
valued
across
and
is
you
can
see
what
is
inside
the
thing
inside
the
certificate
inside
the
certificate?
You
have
lots
of
attributes
and
those
are
all
configurable,
and
this
is
what
I
mean
by.
A
If
it
is
my
identity,
I
get
to
choose
exactly
how
the
world
sees
me
or
how
I
want
to
present
myself
to
the
world.
So
these
are
all
configurable
items
in
Volterra's
identity
and
what
happens
is-
and
this
is
also
important-
you
can
have
a
part
that
gets
basically
attached
to
another
service
after
the
pod
was
launched.
A
That
means
the
identity
also
changes
the
one
more
attribute
to
identity
god,
I
added,
so
in
this
case
the
pod,
the
wingman,
as
soon
as
it
is
launched
and
gets
the
first
set
of
certificates
immediately
starts
watching
voucher,
which
will
notify
it
if
it,
if
it
and
anything
changed
about
that
part
and
if
anything
change,
it
will
go
back
to
identity
or
Authority
get
a
fresh
certificate.
So
all
the
rotation,
all
those
things
are
already
taken
care
of.
The
point
is,
though,
the
certificate
was
generated.
Private
key
was
generated
and
never
leaves
that
pod
area.
A
Okay,
so
your
your
trust,
boundary
is
still
within
the
pod.
If
I,
if
I,
have
that
identity
once
with
me,
we
can
now
use
it
for
something
like
secrets,
management
right,
which
can
be
completely
third
party
solution.
In
this
case,
we
have
welder
secrets
management,
which
is
a
slightly
different
way
of
thinking
about
secrets
management,
because
we,
even
though
we
we
are
able
to
decrypt
the
secret
for
the
customers,
we
never
get
to
see
them
and
direct.
We'll
talk
a
little
bit
about
that.
A
But
if
you
see
trail
of
bits,
security
assessment,
you
will
see
that
almost
all
of
the
high
security
problems
in
Kas
actually
came
from
within
within
the
cluster.
We
just
lock
that
down.
So
if
you
are
running
in
a
namespace,
you
only
good
get
to
become
or
talk
within
named
namespace
or
other
namespaces
of
your
own
within
that
customer
account,
but
you
never
get
to
talk
to
the
Cape
kubernetes
api
server
or
any
other
infrastructure.
A
So
in
summary,
once
you
are
done
with
this,
you
can
achieve
something
like
this
without
doing
any
security
related
stuff.
This
is
all
out-of-the-box
right.
So
the
point
is
that
if
you
have
multiple
of
these
clusters,
a
lot
of
people
are
trying
to
figure
out
how
to
make
these
guys
work
together.
So
this
is
what
identity
gives
you
direct
will
now
talk
about
a
little
bit
more
about
the
secrets,
management
path
as
well.
Thanks.
B
So
a
little
background,
we
started
our
container
journey
about
four
years
ago.
We
were
starting
to
deploy
Apache
mezzos
as
a
platform
for
running
distributed,
job
graphs,
and
we
weren't
even
really
thinking
about
containers
back
then.
But
after
we
had
a
few
apps
running
on
the
same
infrastructure,
it
became
obvious
that
containers
it
would
be
a
great
way
to
maintain
consistent
execution
environments
and
and
reproducible
testing,
and
so
within
a
few
months
we
converted
all
our
deployment
pipelines
our
infrastructure
to
run
these
jobs
on
containers,
and
that
was
really
successful.
B
We've
we've
been
doing
that
ever
since
then,
over
time
things
expanded.
We
rolled
out
more
clusters.
We
implemented
in
business
for
DCOs,
started
doing
some
work
with
services,
so
that
led
us
to
the
mesas
kubernetes
engine,
as
well
as
some
standalone
kubernetes
implementations,
and
these
clusters
would
be
launched
in
an
office
in
a
data
center
in
the
cloud.
But
in
any
case
each
one
was
an
island.
It
was
managed
separately
from
the
others.
B
But
well
the
infrastructure
was
getting
more
complex,
so
was
the
development
organization
and
we
had
a
lot
more
use
cases
where
one
development
team
would
not
have
any
reason
to
access
the
credentials
or
data
being
used
by
another.
So
we
started
really
looking
to
our
options
for
like
serious
secrets,
management
and
getting
started.
There
was
some
obvious
places
to
look
so
standard,
kubernetes
secrets
available
out
of
the
box.
They
have
their
limitations,
but
you
know
they
were
easy
enough
for
us
to
get
started
with.
B
Dcos
Enterprise
has
its
own
embed,
its
secrets
fault,
and
we
were
able
to
use
that
to
inject
secrets
into
jobs,
maybe
a
more
robust
solution,
but
it
has
its
own
limitations.
The
access
rules
aren't
as
flexible
as
we
would
like,
and
it's
also
limited
in
scope
to
a
single
cluster
when
we
have
many
clusters
at
that
point,
so
we
started
looking
into
ways
to
manage
secrets
globally
and
there's.
B
One
thing
we
started
to
do
was
to
set
up
a
Shakur
vault
as
a
way
of
cermagic
secrets
outside
of
any
particular
cluster,
and
you
know
we
might
have
a
single
source
of
truth
for
a
particular
secret,
because
it
was
pretty
common
to
have
say
like
a
database
password
that
was
needed
in
three
different
clusters.
It
was
pretty
annoying
to
have
to
keep
it
updated
in
three
different
places.
We'd,
probably
get
it
wrong
at
some
point,
so
how
to
tie
everything
together.
B
B
You
have
to
go
through
to
get
the
secret
into
a
particular
container
or
application,
and
then
we
tried
coming
at
this
from
the
other
direction,
which
was
to
try
some
of
these
third-party
container
security
platforms
like
aqua
and
twistlock,
and
you
know
these
are
for
every
comprehensive
platforms
with
a
lot
of
capabilities
beyond
secrets
management.
But
you
know
I
had
a
smaller
team
at
the
time
and
we
didn't
have
the
resources
to
look
into
all
the
other
stuff.
B
B
They
also
had
very
flexible
rules,
engines
to
determine
which
applications
would
get
each
secret
and
also
good
is
that
management
of
these
security
policies
was
decoupled
from
management
of
the
clusters
and
the
applications.
So
it
was
a
little
easier
to
figure
out
how
we
would
separate
responsibilities
on
the
technical
teams.
On
the
downside.
You
know
there
were
pretty
new
products
at
the
time
and
to
be
fair.
This
was
like
over
a
year
ago,
and
we
did
this.
There
were.
B
There
was
just
a
lot
of
added
complexity
involved
and
we
were
already
making
the
environment
more
complex,
and
so
we
decided
we
probably
didn't
want
to
take
that
on.
At
that
time,
at
least
me,
and
it
would
probably
be
more
worthwhile
once
we
were
prepared
to
evaluate
all
the
other
functionality
that
they
had
to
offer.
So
we
we
backed
off
for
the
time
and
you
know
we're
prepared
to
revisit
them
at
a
later
date.
B
So
for
the
moment
we
have
solutions
that
are
essentially
good
enough
for
our
current
use
cases.
Now
we
can
iterate
on
them
and
make
them
incrementally
better,
but
we
also
wanted
to
take
a
step
back
and
think
about
what
what
is
our
future
architecture
going
to
look
like,
and
what
capabilities
do
we
want?
B
We
want
to
be
able
to
store
those
configurations
in
one
place.
We
want
to
store
secrets
in
one
place,
but
if
it's
in
a
vendors
infrastructure
or
software,
we
of
course
need
it
to
be
encrypted
when,
when
we
hand
it
over
to
them
and
then
zooming
way
in,
we
want
to
have
really
fine-grained
control
over
which
which
app,
which
part
or
container
can
access
which
secrets-
and
you
know,
apply
these
access
policies
in
a
very
selective
fashion.
B
So
when
I
first
started
talking
to
Volterra,
they
were
you're
mostly
talking
about
the
service
best
functionality
at
the
time,
and
that
was
of
great
interest
to
us.
But
it's
on
a
longer
time
horizon
for
us,
but
when
they
start
explaining
their
plans
for
a
global
identity
for
security
policy
management
and
secrets
management
that
piqued
my
interest
because
it
seemed
like
they
were
really
considering
the
sort
of
specific
concerns
and
requirements
that
we'd
already
raised
internally.
So
their
web
console
is
that
that
single
control
plane
to
manage
their
functionality
across
potentially
thousands
of
clusters.
B
If
they
will
store
the
secrets
for
us.
But
you
know
they
never
touched
them.
Unencrypted
their
use
of
identity
as
a
selector
for
security
policy
potentially
gives
us
a
lot
of
flexibility
over
how
we
want
to
apply
those
policies.
And
then
there
a
technique
for
getting
these
secrets
into
the
pods
also
seems
to
work
pretty
well
from
my
standpoint.
A
As
I
mentioned,
this
was
not
supposed
to
be
a
sales
pitch,
but
I
would
say.
One
of
the
things
I
want
to
stress
is
the
identity
piece,
because
that
has
nothing
to
do
with
even
kubernetes.
In
some
ways.
This
is
a
philosophy
where
the
smallest
piece
of
code
that
you
run,
and
you
want
to
identify
separately,
is
how
you
are
going
to
put
identity
together
and
then,
whatever
mechanism
you
end
up,
choosing
it
needs
to
basically
go
all
the
way
down
to
that
smallest
base
level
in
case
of
kubernetes.
A
A
Handing
off
to
you
good
enough
for
you,
then
that's
fine
from
cryptography
perspective,
really
not
rich
usable
and
extensible
very
important,
because
unless
your
identity
document
is
rich
or
it's
not
extensible,
you
are
not
going
to
be
able
to
write
very
good
authorization
policies,
authorization
policies,
effectiveness
and
granularity
directly
depends
on
granularity
of
fire.
Your
identity
and
resource,
so
this
was
the
first
part
and
takeaway
is
it's
it's
a
fundamental
problem.
A
Identity
is
a
fundamental
problem
and
then
only
when
you
solve
this
and
completely
grabbed
it,
you
can
move
on
move
on
and
talk
about,
em,
TLS
and
Anzhi,
and
all
those
things
later,
abstract
away,
identity
from
provisioning
from
Kate
s
and
when
I
say
caters.
Actually,
it's
not
just
Kate
as
anything
they
are.
You
run
your
identity
or,
if
you
keep
it
one
level
above
your
infrastructure,
that
is
actually
running
the
code
that
will
help
you
go
across
infrastructures
as
well.
A
We
can
do
the
exact
same
thing
on
some
platforms,
which
are
not
even
care
tests
and
still
be
able
to
make
it
as
work
lost
out
to
non
Kate
s
workloads
under
the
same
global
identity.
It's
just
that
the
abstraction
layer
is
one
level
above.
We
will
just
figure
out
how
to
do
that
in
the
other
way
of
doing
it.
A
Multi
cluster
deployments
turned
cumbersome
into
unmanageable.
I
mean
I'm
pretty
sure
that
if
your
administrator
of
kubernetes
today,
you
have
been
tired
of
writing
service
accounts
and
cluster
role,
binding
and
role
binding,
and
you
know
trying
to
make
sure
that
this
secret
injected
into
this
namespace
is
not
available
to
other
people
in
the
namespace,
and
for
that
you
have
to
basically
be
very
diligent
about
loading.
Every
launching
every
application
in
a
separate
service
account
and
manage
those
things
find.
A
Maybe
you
can
do
that
in
one
cluster,
two
clusters:
five
clusters,
once
you
start
talking
about
tens
and
plus
it's
unmanageable.
So
if
you
have
this
engineering
right,
then
you
know
it
doesn't
matter
how
many
clusters,
you
add,
there's
no
Federation,
nothing
required
because
you
you're
you're,
basically
just
adding
to
your
infrastructure
without
actually
changing
any
of
the
underlying
mechanisms.
A
So
if
your
secrets
were
sitting
in
your
cater
secret
in
some
namespace-
and
it
was
popped,
it
will
basically
go
read
all
of
them
and
then
laterally
move
somewhere
else
start
attacking
other
pieces
of
infrastructure.
So
we
are
here
for
maybe
took
a
couple
of
more
minutes
for
questions
and
then
we
may
have
to
take
them
questions
outside,
but
yeah.
Thank
you
for
your
attention.
A
Good
question,
so
the
question
was:
if
I
have
an
identity
in
the
pod,
and
now
I
want
to
map
it
to
some
identity,
which
is
say
underneath
it's
running
on
AWS
and
now
I
want
to
give
a
very
specific
role
to
this
part
and
so
that
it
can
go
off
and
maybe
download
something
from
s3.
The
thing
is,
the
person
who's
in
charge
of
the
cluster
can
always
define
that
this
pod
will
have
a
label
of
AWS
IM
role
right
and
then,
as
soon
as
that,
pods
identity
contains
that
label.
A
It
can
go
off
and
have
another
agent
just
like
we
have
voucher.
You
can
basically
launch
something
that
is
allowed
to
go
to
sts
service
in
case
of
AWS
and
exchange
it
and
give
it
a
particular
role
right.
It's
the
it's
the
enabler.
It's
not,
it
doesn't
come
for
free,
but
it's
the
enabler
if
you
do
the
exact
same
example
in
case
of,
if
your
certificate
only
had
let's
say
a
service
account
if
it
was
only
service
account.
A
A
Spiffy
was
slightly
differently.
Spiffy
technically
does
not
dictate
that
you
need
to
have
a
sidecar
like
wingman.
That's
number
one
number
two
spiffy
says
actually
I
shouldn't
say
spiffy,
it's
more
like
spire,
because
PV
is
just
you
know
how
you
name
your
thing
and
I
mean
nothing
precludes
us
from
using
spiffy
way
of
naming
something
here.
Right,
spire
is
more
sort
of
appropriate,
so
the
way
spire
works.
Is
it
basically
allows
you
to
go
to
something
called
workload
API
and
pull
out
your
sort
of
private
your
credentials?
A
The
point
I
was
mentioning
here
is
that
that
piece
spire
sits
outside
of
your
pod.
It
generates
private
key
for
you,
it's
different
way
of
thinking
about
it
and
there
are
other.
Basically,
you
know
aspects
of
security.
You
may
want
to
consider
when
you
consider
the
the
platform
as
a
whole,
because
in
our
case
the
as
I
mentioned,
we
are
in
edge
computing
business.
Some
of
the
edge
boxes
will
be
sitting
on
some
unmanned
area
where
it
is
quite
possible
that
somebody
will
compromise
one
of
them.