►
Description
Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Building Flux's Multi-Tenant API with K8s User Impersonation - Leigh Capili, VMware
Kubernetes is hard to operate in a multi-tenant manner. As organizations add API's and privileged controllers to their clusters, it becomes infeasible to build clusters that teams can share with each other safely. This is a design issue with the way projects extend Kubernetes. While policy engines like Gatekeeper and Kyverno enable cluster owners to patch over insecure API surfaces to protect tenants, there are patterns that produce API's resistant to cross-tenant issues. It's possible to extend Kubernetes without relying on admission-based policy engines to restrict API boundaries and controller implementations. This session will teach you how to enable multiple organizations and teams to work safely together across namespaces and clusters. Flux will be used as an example on how to use RBAC, impersonation and kubeConfig secrets, but the techniques shown can be used to improve projects across the ecosystem!
A
So
yeah
also
was
really
involved
with
the
flux
project.
So
I'm
a
git
ops
experimenter,
just
like
all
of
you
friends
and
it's
really
important
for
us
as
practitioners
to
get
together
and
to
build
good
tools
so
that
we
can
have
the
experience
with
git
ops
that
we
deserve,
because
I
think
that
we
can
all
agree
that
right
now
our
platforms
and
our
tooling.
A
A
So
when
we're
building
a
kubernetes
controller,
we
typically
have
like
our
own
registered
api
types
right,
and
so
I've
got
like
my
kind
of
thing
and
then
maybe
I'm
working
with
some
other
kinds
of
things
right.
Whatever
my
domain
of
programming
that
I'm
trying
to
reconcile
and
automate
and
then
those
kinds
of
things,
sometimes
they
have
child
objects
right.
So
maybe
my
first
kind.
It
needs
to
read
and
write
to
config
maps,
and
then
it
also
needs
to
read
a
bunch
of
credentials
from
a
secret
right.
A
That's
pretty
common,
like
lots
of
things,
need
secret
or
secrets
so
that
they
can
get
credentials
to
access
external
apis
and
they
can
control
things
and
then,
like
my
second
kind,
oh
that
also
like
mutates,
ingresses
or
whatever.
And
so
I
have
my
use
cases
in
mind
and
I
make
my
api
and
I
write
my
controller
code
and
then
that
controller
needs
a
service
account
and
that
service
account
needs
a
role
binding
to
a
role
that
lists
a
bunch
of
these
kinds
of
things
right.
A
And
so
you
can
see
it's
sort
of
a
non-trivial
problem.
But
at
least
it's
like
well
understood
because
when
we
come
to
the
process
of
writing
our
controller,
we
know
what
we're
working
on,
but
in
the
get
ups
controller
landscape
right,
say.
I
have
like
a
couple
of
types
of
things
like
web
hooks
and
like
sources
that
represent
git,
repos
or
oci
registries
or
whatever,
and
then
I
have
my
other
resources
that
are
actually
designed
to
declare
how
I
want
to
mutate.
A
The
cluster
right,
like
my
applier
type
resources
right
with
with
flux,
we
have
the
customization,
we
have
the
helm
release
in
other
projects.
We
might
have
an
app
or
an
application
depending
on
which
project
you
have,
and
so
we
still
have
that
same
thing
where
it's
like.
Okay,
where
our
controllers,
they
need
some
objects.
They
gotta
go
get
some
secrets.
Maybe
they
gotta
go
talk.
These
objects
together.
Right,
like
I,
can't
just
apply
nothing.
I
have
to
apply
some
sort
of
source
like
a
git
repository.
A
It
needs
something
more
granular
per
object
right.
Every
time
that
I
want
to
declare
that
I
want
to
apply
something
over
and
over
again,
I
can
have
multiple
of
these
things
and
I
want
to
granularly
define
which
one
we
or
for
a
particular
apply
object.
What
are
back
permissions?
It
has,
and
so
this
idea
of
the
applier
identity.
A
You
can
just
say
hey
when
I
do
this.
I
wanted
to
use
this
particular
service
account
and
usually
the
way
that
works
is
the
apply.
Controller,
has
r
back
access
on
its
controller
service
account
to
fetch
some
secrets
and
the
cool
thing
about
service
accounts
is
they're,
this
very
peculiar
mechanism,
which
will
break
apart
a
little
bit,
but
basically
the
api
server
in
kubernetes
when
it
sees
a
service
account.
It's
like.
A
There's
a
you
know:
we
have
these
pieces
right.
Service
accounts,
make
these
essay
token
secrets,
and
then
we
allow
our
apply
controller
when
it's
reconciling
the
apply
object
to
go
and
get
that
secret
and
change
the
way
it
talks
to
kubernetes,
and
then
it
produces
most
likely
a
deployment
right.
It's
reconciling
some
folder
from
a
git
repository
and
there's
like
deployments
and
config
maps
and
all
sorts
of
stuff
in
there
right,
and
we
want
to
run
some
pods
because
we're
using
kubernetes
and
we
want
some
compute
well.
Service
accounts
are
designed
for
pods.
A
It's
a
very
particular
use
case,
actually
we're
sort
of
abusing
a
kubernetes
mechanism
here
when
we
want
to
use
essay
token
secrets
inside
of
a
control
loop
temporarily,
but
they're
meant
for
pods
and
and
every
pod
has
that
spec
service
account
name.
And
so
you
can
actually,
if
you
happen
to
be
deploying
resources
to
the
same
namespace,
that
your
apply
customer
resources
in
those
pods
can
be
defined
to
actually.
A
They
can
accept
that
service
account
identity
right.
They
can
take
it
on,
but
this
service
account
identity
is
likely
purpose
built
for
reconciling
your
application.
It's
designed
to
modify
and
deploy
workloads
to
the
cluster
and
to
run
your
custom
code
or
somebody
else's,
and
so
say
you,
you
know,
take
an
app
update
of
somebody's.
You
know
vendored
application
or
something
from
a
third
party,
and
they
decide
to
specify
the
service
account
name
of
the
thing
that
reconciles
itself
well
now
that
pod
has
the
ability
to
deploy
workloads
to
your
cluster
right.
A
So
one
solution
is
to
actually
just
always
apply.
If
you're
using
service
accounts,
you
can
just
go
to
a
different
name
space
right
now.
There's
no
service
account
that's
adjacent
to
the
pod
in
the
same
name
space,
and
then
you
don't
have
that
path
for
privileged
escalation.
It's
sort
of
a
privilege
escalation,
because
your
pods
are
not
really
meant
to
be
reconciling
themselves
right.
A
So
I
want
to
talk
a
little
bit
about
user
impersonation.
This
is
a
mechanism
that
we
became
really
curious
about
with
the
flex
project
and
we
started
writing
up
a
proposal.
It's
proposal
582
on
the
repository.
If
you
want
to
go
and
look
at
it
and
service
accounts
they're,
basically
just
normal
users,
every
user,
you
know
just
has
a
string
that
represents
it
inside
of
the
kubernetes
api
right.
There
are
some
users
that
are
maybe
tied
to
gmail
accounts.
A
There
are
some
users
that
might
be
tied
to
some
saml
flow
or
some
social
off
flow
to
github,
and
they
each
have
a
string
with
a
domain
name
or
some
sort
of
delimiter
and
namespace.
That
tells
it
where
it
came
from
and
so
service
accounts,
they're
just
username
strings.
It's
actually
possible
to
impersonate
a
service
account
directly,
and
I
can
demonstrate
that
right
now
here
all
right.
So
like
I'm
myself,
I'm
authenticating
with
a
certificate
to
a
kubernetes
style
cluster
and
I
can
get
pods,
but
I
can't
get
pods
as
some
other
user.
A
Like,
oh,
okay.
Well,
you
know
this
api
server
response
here.
It's
saying
forbidden
some
user
can't
list
the
resource
pods
right,
so
I
have
changed
my
identity
and
that's
because,
as
a
cluster
administrator,
you
have
the
permission
to
impersonate
anybody
and
oftentimes.
Our
git
ops
controllers
do
have
these
very
close
to
super
user
level
privileges.
So
this
this
capability
is
already
here
and
if
I
also
tack
on
a
group
system
masters,
then
suddenly
I
can
get
privilege
again
and
I
remembered
how
I
mentioned
that
you
could.
A
Impersonate
a
service
account
well
here
is
me
impersonating,
the
pod
garbage
collector,
but
you
can
see
that
not
every
service
account
has
that
permission
right,
the
one
from
the
default
name
space
that
one
can't
do
that
right,
and
so,
if
you've
never
played
with
this
impersonation
feature
before
even
just
as
a
cluster
administrator,
I
would
really
encourage
you.
Try
it
out
super
cool.
A
But
what
can
we
do
with
that
right?
So
if
a
service
account
string
is
kind
of
just
this
like
system
colon
service
account,
then
you
have
namespace
and
then
the
name
of
the
service
account.
Can
we
copy
that
like
what,
if
we
gave
our
controllers
the
permission
to
impersonate
users
and
then
every
time
the
controller
impersonated
a
user
when
it
was
operating
in
an
object
from
a
particular
namespace?
A
This
is
not
something
that's
just
unique
to
gitops
or
to
flux.
You
can
also
impersonate
groups.
Every
service
account,
if
you
didn't
know,
actually
has
a
group
there's
a
couple
that
gets
hacked
on
there.
So
there's
system
service
accounts,
which
is
every
service
account
across
the
entire
cluster
ever
and
then
there's
the
namespace
group
as
well.
A
So
if
you
wanted
to
target
a
particular
number
of
service
accounts
from
some
namespace
or
say
that
you
as
a
cluster
administrator,
wanted
to
role,
bind
to
all
of
the
service
accounts
from
a
team's
namespace,
you
can
actually
do
that
with
a
group.
Super
cool
right.
This
is
worth
mentioning,
but
with
role-based
access
controlling
kubernetes.
A
But
you
know
we'll
like
say
with
flux
right.
We
we
put
these
groups
on
there
as
well,
and
so
we've
been
talking
a
lot
about
the
in
cluster
identities,
right,
focusing
on
service
accounts
and
usernames
that
are
happening
inside
the
cluster.
But
we
do
have
the
use
case,
with
git
ops
controllers
to
actually
control
what
we
want,
our
declarations
to
be
from
one
kubernetes
cluster
as
a
management,
cluster
style
of
of
a
topology.
A
A
A
Well,
these
things
need
to
hook
up
right
like
they
need
to.
They
need
to
link
together
and
the
apply
needs
to
be
able
to
to
fetch
some
http
resources.
You
know
about
the
source
object
that
got
unpacked
from
some
registry.
Well,
if
these
are
in
the
same
name,
space
just
like
how
pods
can
mount
secrets
and
use
service
accounts
from
the
same
name,
space,
that's
kind
of
not
a
big
deal
right,
like
you
put
the
source
and
the
applier
together,
but
not
everyone
wants
to
structure
the
way
that
they
work
like
that
right.
A
Some
people
want
to
manage
their
apps
from
one
name
space,
and
they
want
to
have
like
another
team
that
manages
the
helm
repository
in
another
name
space,
and
so
how
do
you
get
these
things
to
be
able
to
be
used
in
an
intentional
relationship
where
every
applier
in
the
cluster
can't
access
every
source
in
the
cluster?
Right
now
you
have
this
policy
problem.
How
can
I
get
some
tenancy
going
and
one
approach
to
that
is
to
add
this
access
control
list
and
we're
starting
to
see
this
in
some
service
and
gateway
style
apis
as
well?
A
It's
the
trend,
that's
happening
in
the
industry.
It's
a
very
quick
thing
to
reach
for
and
the
way
that
it
works
is
say,
your
source
controller
and
your
apply
controller.
You
have
some
trusted
code
sections
in
there
and
when
the
apply
controller
fetches
a
source
object,
it
checks
to
see
is
then
is
the
object
that
I'm
operating
on
in
a
namespace
or
in
some
category
or
whatever,
that
is
allowed
in
the
access
control
list
of
the
source.
Object
that
I
just
fetched
and
you
trust
the
apply
controller.
A
The
right
thing
make
sure
that
the
apply
controller
is
always
impersonating
an
identity,
and
then
you
can
roll
bind
to
that
identity
so
that,
if
the
apply
controller
doesn't
have
access
to
that
source
object,
it
can't
even
fetch
it.
It
can't
get
the
information
about
the
http
server
and
the
credentials
that
it
needs
for
for
getting
the
source
code,
and
this
allows
you
to
have
a
multi-tenant
environment
inside
of
a
cluster
right.
A
The
ability
to
create
the
role
bindings
for
you,
so
this
is
a
way
to
get
that
convenience
style
api
without
putting
the
ons
on
the
apply
controller
implementers
without
giving
them
the
trust
to
implement
the
actual
access
control
right.
So
the
control
stays
within
source,
controller's
generation
of
the
role
binding
and
it's
implemented
by
the
kubernetes
api
server
instead,
and
I
think
that
this
is
a
really
good
design.
A
The
complexity
trade-off
allows
you
to
do
things
like
take
that
code
that
you
have
inside
of
the
apply
controller
and
distribute
it
as
a
command
line
to
a
library
so
that
now,
since
you
don't
have
to
trust
your
client,
you
can
actually
give
that
and
run
it
on
any
computer
that
has
access
to
kubernetes.
So
that
way,
you're
not
like
just
trusting
people
to
follow
the
rules.
You
actually
get
to
ask
kubernetes
to
enforce
it
using
the
api.
That's
designed
to
do
it,
and
but
yeah
there's
there's
a
lot
going
on
in
this
proposal.
A
I
really
want
to
share
these
ideas
with
all
of
us
implementers.
I
want
us
users
to
have
an
expectation
and
to
have
faith
that
when
we
are
using
our
tools
that
we
can
trust
them
to
provide
a
secure
basis
for
a
platform,
that's
enjoyable
to
use
and
is
extensible
right.
If
I
have
the
idea
to
add
something
to
the
command
line
tool,
I
want
to
be
able
to
do
it
and
not
revoke
the
the
trust
of
other
platform,
admins
everywhere,
so
check
out
the
pull
request.