►
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon North America 2021 in Los Angeles, CA from October 12-15. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
I Can RBAC, and So Can You! - Marc Boorshtein, Tremolo Security, Inc.
This session takes the sharp edges off of Role Based Access Controls in Kubernetes and demystifies how to design and debug policies in both single tenant and multi-tenant clusters. This session will start with a quick overview of how Kubernetes integrates identity, how that identity is applied to authorization, and finally different strategies for automating authorizations in a cluster. This session will cover policy creation, debugging, aggregate role design, and automation. We'll also touch on the impacts of external systems that control your clusters, such as GitOps controllers. After this session attendees will have a clearer direction on how to approach authorizations in their clusters,
A
So
first,
I'm
going
to
give
a
real
brief
overview
of
who
I
am
and
why
I
can
talk
to
you
about
our
back.
We'll
then
get
into
what
our
back
canon
can't
do.
We'll
talk
about
some
of
the
sharper
edges
around
our
back
where
authentication
comes
into
play
and
why
that's
important
and
we'll
start
talking
about
some
of
the
business
roles
of
our
pack,
especially
multi-tenancy.
A
So
who
am
I
my
name
is
mark
washington,
like
I
said,
I'm,
the
cto
of
tremolo
security.
I've
got
20
plus
years
of
experience
and
identity
management.
I've
been
working
with
kubernetes
since
about
2016,
with
several
contributions
to
the
project.
You
can
often
find
me
inside
of
the
slack
channels
talking
about
authentication
authorization
and
finally,
I
just
released
the
co-authored
a
book
called
kubernetes
and
docker
and
enterprise
guide.
A
A
A
One
thing,
that's
really
important
that
you
cannot
do
is
you
cannot
write
a
policy
that
says
I
want
to
do
everything
except
x.
I
want
to
give
access
to
everything,
except
these
four
secrets
in
the
namespace
or
I
want
to
give
cluster
admin
access
to
everything,
except
for
these
three
namespaces
or
the
kube
system.
Namespace,
those
types
of
policies
are
really
difficult
to
implement.
Our
back
won't
implement
them
for
you.
There
are
different
ways
that
you
can
do
that.
But
quite
frankly,
if
that's
your
model,
you
probably
need
to
reevaluate
what
you're
doing
alright.
A
So
what
makes
our
back
so
hard
once
we
start
getting
into
the
syntax
of
the
the
objects
themselves?
You'll
see
that
they're
pretty
straightforward.
There
isn't
a
lot
there
there,
but
there
are
a
lot
of
things
that
give
it
really
sharp
edges
and
the
way
I
have
this
table
broken
up
is
I
wanted
to
show
that,
while
some
of
these
things
make
it
harder
to
get
started
in
the
long
term,
as
you
approach
maintenance-
and
you
know,
what's
often
called
day-
two
maintenance-
it
actually
makes
it
a
lot
easier.
A
A
If
you
create
an
rbac
policy
and-
and
this
is
true
of
almost
anything
in
kubernetes-
that
references
other
objects,
if
those
objects
don't
exist,
kubernetes
isn't
going
to
tell
you
so
that
can
make
debugging
a
little
bit
difficult.
You
got
to
be
really
particular
in
in
how
you
set
things:
users
and
groups.
They
don't
exist.
There
are
a
couple
of
corner
cases,
server
count,
service,
accounts,
exist
and
service
accounts
have
some
static
groups,
but
in
general
you
can't
just
define
an
object,
called
a
group
and
add
members
to
it
doesn't
exist.
A
A
You
can
combine
cluster
roles
and
role
bindings,
which
you
know
would
think,
make
things
easier
and
does
in
the
long
run,
but
can
often
make
it
harder,
because
people
have
a
in
the
short
term,
because
people
have
a
harder
time
conceptualizing
it.
I
know
I
did
when
I
first
got
started
this
one.
I
have
harder
and
harder
authorization
is
a
business
problem.
A
The
apis
and
kubernetes
are
not
self-documenting.
So
when
you
go
to
design
your
policy
object,
what
you're
going
to
find
is
that
you
can't
look
at
a
yaml
and
say:
oh,
that
yaml
tells
me
everything.
I
need
to
know
to
design
a
policy
around
it
and
then
same
as
what
we
talked
about
earlier,
with
not
being
able
to
write
a
policy
that
says
everything
except
there's
very
limited.
Wild
card
supports.
Basically,
you
can
either
spec
specify
and
enumerate
all
of
what
you
want
or
a
wild
card
of
star
for
everything.
A
What
you
can't
say
is
hey.
I
want
a
group
that
applies
to
anything
with
a
name
that
starts
with
generation,
for
whatever
reason
you
can't
do
that
with
our
back,
and
there
are
really
good
reasons
for
that,
one
of
which
is,
let's
say
you
defined
a
you,
deploy
an
operator
that
builds
cloud
objects
and
it
just
happens
to
have
the
same
name
as
that
wildcard
and
you
didn't
realize
it
well
now
you've
authorized
access
to
it,
so
our
back
is
very
explicit.
A
A
So
speaking
of
identity
providers,
let's
talk
about
authentication
here
for
a
minute,
so
these
are
the
most
common
authentication
methods.
First,
one
everybody
runs
into
is
certificate
authentication
your
user
is
stored
in
the
subject.
Groups
can
be
stored
as
ous
in
the
subject
dn
you
should
not
use
it.
Kubernetes
does
not
support
certificate,
revocation,
which
means
once
a
certificate
has
been
minted.
It
will
be
accepted
by
kubernetes
until
either
it
expires
or
the
certificate
authority
gets
changed
to
a
different
root
certificate.
A
So
break
glass
in
case
of
emergency,
but
really
it's
just
not
dynamic
enough
for
day-to-day
use.
Service
account
token
should
never
be
used
from
outside
of
the
cluster
they're.
Only
ever
designed
to
work
in
cluster
token
request
api
kind
of
changed
that
a
little
bit,
but
you
still
need
a
way
to
authenticate
and
get
that
token
so,
and
we
have
a
link
at
the
end
about
how
you
can
do
more
secure
access
from
outside
your
cluster
now
impersonation.
This
is
where
you
have
a
reverse
proxy
between
your
users
and
your
api.
Server.
A
Reverse
proxy
authenticates.
You,
however,
it
needs
to
and
then
injects
headers
into
the
request.
This
is
a
very
powerful
tool,
especially
when
you're
talking
about
cloud
managed
kubernetes,
where
you
don't
want
to
use
cloud,
am
open
id
connect.
This
is
the
best
way
to
go.
To
be
honest,
your
token
stores,
who
you
are
what
you
can
do.
A
Excuse
me
and
can
have
short-lived
tokens,
which
really
adds
to
your
security
profile
and
then
finally,
you
have
your
cloud
iam,
vendors
or
implementations.
Every
cloud
has
its
own
implementation
and
they'll
have
different
ways
of
managing
user
and
group
identifiers
cloud.
Iim
and
impersonation
are
often
tied
to
the
hip,
because
this
gets
back
to
the
business
question
a
lot
of
times.
The
cloud
group
doesn't
want
to
own
who
has
access
to
kubernetes.
They
want
to
just
deploy.
A
A
A
A
The
downside
to
this
approach
is
that
it's
really
poor
on
utilization
management,
not
just
hardware,
utilization
or
vm
utilization,
but
also
in
resource
utilization,
there's
a
big
difference
between
somebody
who's,
an
expert
at
writing,
applications
that
will
run
on
kubernetes
and
somebody
who
knows
how
to
manage
the
nuts
and
bolts
of
kubernetes.
Those
aren't
there's
overlap,
but
they
aren't
the
same
skill
sets
that's
why
you
have
different
tests
right.
A
You
have
a
cka
and
a
ckad,
and
so
having
a
multi-cluster
environment
means
that
you
have
to
have
more
people
that
know
how
to
manage
the
individual
cluster
versus
how
to
write
applications
for
the
cluster.
You
also
need
to
you:
haven't
eliminated
your
boundaries
in
our
back,
you've
just
moved
them,
so
maybe
you're
doing
less.
Work
in
your
kubernetes
are
back,
but
you've
now
moved
the
authorization
layer
into
the
control
plane
pipeline
tenants.
A
So
there
are
a
lot
of
places,
implementations
that
will
say
you
know
what
I
don't
want:
users
interacting
with
the
api
server.
I
just
want
them
checking
in
their
code.
I'm
using
my
kodi
fingers
here.
Code
could
be
you
know,
application
code
or
infrastructure
as
code.
If
we're
talking
about
get
ops.
There
are
some
implications
here,
though,
just
like
with
control,
plane,
multi-tenant
or
cluster
multi-tenancy.
A
Additionally,
as
you
look
at
how
you
design
this
out,
you
have
to
think
about
silos
and
management,
especially
in
enterprise.
The
people
who
write
the
apps
and
who
are
responsible
for
the
apps
and
who
ultimately
their
paycheck,
is
based
on
things
like
uptimes
and
stuff
like
that
are
going
to
want
as
much
control
as
possible,
so
if
you're
between
them
and
their
application,
when
something
goes
wrong,
you're,
ultimately
going
to
get
the
blame
for
it,
and
so
a
lot
of
cloud
teams
that
start
with
this
approach
make
this
an
option,
but
not
a
requirement.
A
And
then
finally
namespace
multi-tenancy-
and
this
is
where
you
get
the
most
richness
with
rbac-
every
application-
gets
its
own
namespace
or
set
of
namespaces.
I
will
talk
a
little
bit
about
that
more.
When
we
get
into
the
toolbox,
authorization
will
be
controlled
by
admin.
So
often
you
know
you
don't
want
you.
Don't
want
system
admins
controlling
that
authorization.
You
want
application
or
team
admins
controlling
it.
You
get
much
higher
density,
not
just
of
your
resources
from
a
hardware
standpoint,
but
also
from
your
people
standpoint.
A
You
can
better
utilize,
your
your
cluster
admins
and
have
application
admins
inside
of
each
team.
Of
course,
the
major
downside
to
this
is
that
if
you
don't
have
proper
controls
in
place,
you
can
have
a
larger
blast
radius.
Somebody
comes
in
there's
a
bug
in
their
application,
leads
to
somebody
owning
the
node
and
all
of
a
sudden.
You've
now
exposed
multiple
applications.
So,
like
I
said
before,
this,
isn't
a
you're
gonna
pick
one
out
of
these
three.
A
A
The
blue
stuff
comes
right
out
of
your
url,
so
your
rule
comes
right
out
of
the
url
definition
for
your
cluster
role.
Your
resource
also
comes
out
of
your
url
couple.
Things
to
note
your
version.
Doesn't
matter
versions
are
not
part
of
our
back.
Also,
if
you
have
any
sub
urls
any
sub
resources
off
of
your
resource,
like
in
this
instance,
approval
pods
have
like
logs
and
an
exec.
A
A
So
once
you
have
your
role,
you
then
have
to
blind
it
to
subjects,
and
so
this
part
again
pretty
straightforward,
really
the
red
part
doesn't
really
matter.
That's
just
your
metadata,
your
role,
ref.
What
role
are
you
going
to
reference?
If
this
is
a
role
binding,
you
can
still
reference
a
cluster
role.
So
this
way
you
can
centralize
your
management
of
roles
without
having
to
recreate
it
in
every
single
name
space
and
then,
finally,
you
list
your
subjects.
A
Let's
talk
a
little
bit
about
subjects.
Three
types
service
accounts,
so
each
service
account
has
to
be
scoped
to
a
specific
name:
space,
a
user
so
for
open
id
connect.
If
you're
not
using
your
email
address,
you
have
to
pre-pen
the
url
of
the
user.
Please
don't
use
email
addresses
as
your
identifier
names
change
for
a
lot
of
different
reasons.
A
Emails
are
tied
to
names,
it's
better
to
use
something
that
will
never
change,
no
matter
what
happens
with
the
person's
name
and
finally,
groups-
and
this
is
really
where
you
want
to
be-
is
let
the
identity
provider
store
groups
and
then
specify
it
by
group.
You'll
have
smaller
rbac
definitions,
they'll
be
easier
to
manage
and
so
much
easier
to
audit
a
directory
or
database
than
it
is
to
try
and
audit
a
kubernetes
deployment.
Kubernetes,
you
can't
say
give
me
all
the
users
that
are
a
member
of
this
rbac
policy.
A
A
All
right,
so
we're
going
to
go
through
a
few
tools
here.
That'll
make
it
easier
for
you
to
manage
our
back.
So
the
first
one
is
this
idea
of
an
aggregate
role.
So
if
you
look
at
the
admin
or
editor
cluster
roles,
you'll
see
these
giant
roles
that
have
access
to
almost
everything
so
like
the
admin
cluster
role
is
designed
to
let
somebody
own
a
name
space
without
affecting
anybody
else
in
the
cluster.
So
you
can,
you
know,
change
our
back.
A
A
A
So
whatever's
repetitive
go
ahead,
automate
it.
This
gives
you
the
ability
to
have
naming
standards,
lots
of
tools
to
do
this
open,
unison.
Our
open
source
project
is
what
I'm
going
to
show
in
the
demo,
but
terraform
continuous
delivery
tools.
Do
it
yourself,
there's
no
end
of
ways
that
you
can
automate
this,
and
so
you
can
see
here.
We've
got
three
namespaces.
Each
name.
Space
gets
a
couple
of
role
bindings,
it
goes
to
groups
and
everything
is
consistent
and
it's
much
easier
to
manage
and
you
as
a
human,
are
not
building
stuff
custom
controllers.
A
So
we
talked
about
the
role
aggregator
another
one
that
I
really
like
is
the
fairwinds
are
back
manager.
So,
instead
of
having
to
have
that
repetitive
role,
binding
in
every
single
namespace
and
having
this
proliferation
of
groups,
you
get
to
have
a
single
object
that
defines
a
label
and
says
a
namespace
with
this
label
gets
this
role
binding,
and
so
now
you
can
have
like
a
team
based
approach
where
different
name,
spaces
get
team
labels
and
then
the
custom
controller
automatically
generates
your
bindings
for
you.
A
A
And
then
finally,
policy
generation
audit
to
rbac
is
a
great
tool
written
by
jordan
liggett.
When
you
get
your
error
messages,
they
don't
give
you
enough
information.
You
need
to
be
able
to
get
something,
that's
machine
readable
to
to
do
it.
So
if
you
look
at
the
event
you
can
see,
this
comes
right
of
the
event
log.
You
can
see
the
verb
that
was
created,
the
request
api,
what
was
actually
requested
and
the
user
and
groups
that
were
requested
to
do
it.
A
So
this
gives
you
enough
information
so
always
enable
your
audit
api.
You
want
to
try
something
you
see
that
fails.
Tell
this
tool
to
go.
Look
at
your
logs,
it'll
generate
a
policy
for
you.
I
use
it
all
the
time.
It's
a
great
system,
all
right!
So
let's
talk
about
the
demo
real
quick
here
we're
going
to
go
with
the
team-based
multi-tenancy
approach.
I
see
this
a
lot
where
each
team
can
have
multiple
namespaces
sysadmins,
don't
want
to
be
in
the
job
of
creating
those
name
spaces
or
adding
access
to
those
teams.
A
A
Somebody
wants
access,
they
get
access
to
the
team.
They
then
automatically
have
access
to
everything
else.
So
how
does
that
get
implemented?
Well,
open
unison
is
going
to
do
the
automation,
so
an
admin
is
going
to
go
in
and
request
to
create
a
team
that
will
create
groups
inside
of
a
database.
It
will
create
an
rbac
definition
for
fairwinds
to
be
able
to
run
off
of
and
then
when
they
create
namespaces,
each
namespace
will
get
a
label
for
that
team
and
then
the
fairwinds
are
back
manager
we'll
go
ahead
and
generate
the
r
back
binding.
A
A
If
we
take
a
quick
look,
we'll
see
that
it's
bound
to
a
group
that
group
exists
inside
of
our
database
now
and
it's
going
to
match
any
name
space
with
the
label
team
demo
3..
So
as
an
admin,
I'm
now
going
to
go
ahead
and
log
in
and
let's
create
a
couple
of
namespaces.
So
local
deployment
we're
going
to
create
it
for
team
demo
3
and
we're
going
to
call
this
u1.
A
A
A
A
So
I'm
going
to
approve
the
request
and
if
I
come
over
here
I
go
to
our
reports.
This
is
where
it
becomes
so
important
to
be
able
to
externalize
your
group
memberships.
You
can
see
here
that
we
can
now
run
reports
against
the
database.
We
don't
have
to
go
against
the
the
api
server
to
see
who
has
access
to
what
the
groups
are
stored
right
in
the
database.
A
There
we
go,
and
so
now,
if
I
go
to
eu
2,
I
now
have
access
I'm
able
to
get
in
and
we'll
see,
pods
and
secrets
and
config
maps,
and
am
I
in
the
right
one?
No,
I'm
not
demo3
or
oops
u2.
There
we
go
so
we
can
see.
There's
the
cube
root,
ca,
cert
secrets
all
there.
Now
here's
the
great
thing
about
the
use
of
teams,
let's
say
for
some
reason:
we
need
to
cut
off
access
immediately
from
this
namespace
from
all
developers.
A
So,
within
a
moment,
what
we're
going
to
find
you
can
see
it
happened
in
real
time
right
there.
I
no
longer
have
access
to
that
team,
because
fairwinds
are
back
manager
removed
the
roll
binding,
because
I
changed
the
downstream
object.
This
works
great
also,
if
you're
doing
r
back
or
I'm
sorry
get
ops
where
you're
checking
this
object
in,
let
let
the
controller
do
the
work
for
you.
A
If
you
want
to
take
a
look
at
open,
unison,
here's
a
link
to
our
website
and,
if
you're
interested
in
the
book
link
to
that
on
amazon.
Finally,
if
you
want
to
roll
your
sleeves
up
and
get
your
hands
dirty
on
our
back
and
authentication
and
pod
security
policies,
we
put
together
a
lab
that
you
can
just
download.
Do
it
yourself,
you
just
need
active
directory.