►
From YouTube: Fine-Grained User Authorization for Kubernetes with OPA and LDAP - Cagri Cetin & Quentin Long
Description
Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Fine-Grained User Authorization for Kubernetes with OPA and LDAP - Cagri Cetin & Quentin Long, Yelp Inc.
Speakers: Cagri Cetin, Quentin Long
Yelp recently migrated their container-orchestration system from Mesos to Kubernetes. However, existing Kubernetes authorization mechanisms were insufficient to implement least-privilege access control rules. Yelp needed to authorize its users to hundreds of services owned by hundreds of different teams. By leveraging the Open Policy Agent (OPA), Yelp has implemented an authorization system that allows defining fine-grained authorization rules: These can rely on service ownerships, resources’ or actions’ sensitivity levels. This talk covers Yelp’s journey to a fine-grained Kubernetes authorization using OPA and LDAP. It will discuss: - Shortcomings of existing Kubernetes authorization mechanisms - Design details of the new OPA-based system - Strategies for provisioning authorization rules at scale - Migration to the new system with zero downtime - Issues encountered along the way and lessons learned
A
Before
we
start,
I
would
like
to
thank
all
of
the
kubecon
organizers
for
giving
us
the
opportunity
to
share
our
story
with
you
all.
Let's
get
started
quentin
and
I
work
on
the
security
team
at
yap.
Today
we
are
going
to
talk
about
how
we
utilize
the
open
policy
agent
and
active
directory
to
provide
fine-grained
role-based
access
control
to
the
kubernetes
infrastructure
at
the
app
and,
let's
start
with
the
motivation
for
this
work.
A
If
you
are
not
familiar
with
yelp,
it's
a
company
that
connects
people
with
great
local
businesses,
as
of
at
the
end
of
2020
yelpers,
have
contributed
to
224
million
reviews.
On
our
platform,
approximately
31
million
unique
devices
access
yap
every
month
on
average
and
more
than
half
a
million
business
locations
spend
money
on
yelp
ads
every
month
to
promote
their
businesses.
A
And
as
you
can
imagine,
it
takes
a
lot
of
infrastructure
to
support
an
application
of
that
scale.
Today
we
have
more
than
1000
geographically
distributed
software
engineers
across
hundreds
of
different
teams.
We
have
a
containerized
microservice
architecture,
managed
by
an
in-house
build
open
source
platform
as
a
service
framework
called
pasta
under
the
hood.
A
Pasta
uses
kubernetes
as
the
container
orchestration
framework
to
manage
thousands
of
workloads,
and
we
have
a
dozen
of
kubernetes
clusters
that
we
run
on
ec2
instances
with
several
custom
name
spaces
where
we
run
our
web
microservices
and
patch
ups
and
other
types
of
statement.
Workloads
like
cassandra
clusters,
kafka,
filling
spark
and
other
many
various
use
cases.
A
A
A
Finally,
we
needed
to
maintain
a
formal
paper
trail
for
all
changes
to
the
authorization
policies
and
all
group
memberships
for
this
project.
We
prioritize
human
users
since
there's
a
lot
of
them
with
lots
of
use
cases
and
thankfully
we
only
had
a
handful
of
service
users
and
they
each
had
their
own
client
certificates
and
nonsense.
The
role
bindings
and
our
back
configurations,
therefore,
the
starting
with
the
human
users
made
a
lot
of
sense
for
us.
A
B
You
charlie,
at
a
high
level,
we
have
human
users
authenticate
using
their
octa
credentials.
Kubernetes
supports
this.
Basically
out
of
the
box,
a
human
user
runs
a
command
like
cube,
ctl
get
pods.
We
provide
a
wrapper
script
that
first
prompts
the
user
for
their
octa
authentication,
which
consists
of
their
active
directory
credentials
and
a
second
factor
like
yubikey
or
an
authenticator
app
upon
success.
They
receive
a
jwt
token
from
octa
valid
for
one
hour,
which
is
then
sent
to
the
cluster
via
cubectl's
dash
token
parameter.
B
B
The
benefits
of
using
octa
for
authentication
are
pretty
straightforward.
Root
access
is
no
longer
needed
to
interact
with
the
kubernetes
clusters.
Each
action
can
be
tied
back
to
a
specific
user
rather
than
a
generic
administrative
user
and
credentials
obtained
from
octa
are
temporary
and
behind
the
second
factor.
This
mitigates
the
risk
of
credential
theft
and
replay
attacks.
B
Now,
let's
talk
about
authorization,
here's
an
overview
of
the
architecture.
First
of
all,
on
the
top
you've
got
the
user,
who
interacts
with
kubernetes
using
cube.
Ctl
kubernetes
sends
a
request
to
open
policy
agent,
which
makes
the
authorization
decision
and
on
the
bottom,
we've
got
the
data
that
feeds
into
that
decision.
We
collect
data
from
git
and
from
active
directory,
and
then
we
put
that
data
into
an
s3
bucket
for
open
policy
agent
to
read.
B
B
Next,
let's
talk
about
the
policy.
The
opa
policy
is
the
logic
that
oppa
uses
to
take
all
of
the
data
provided
and
make
a
decision.
It
is
written
in
rego,
which
is
a
language
based
on
data
log.
I
won't
cover
the
details
of
the
rego
policy,
but
I'll
cover
specific
examples
of
what
the
input
looks
like.
B
B
B
B
B
B
B
Now
we'll
go
over
just
a
couple
of
end-to-end
examples
of
how
this
can
be
used.
First,
we'll
cover
a
basic
example
in
the
top
one:
a
user
is
trying
to
list
pods
in
the
default
namespace.
The
list
verb
matches
the
specified
verb
in
the
capability,
the
namespace
and
the
resource.
Don't
have
any
restrictions
in
the
capability
and
the
user
is
a
member
of
the
opa
kubernetes
dev
unprivileged
group.
So
this
request
is
allowed
in
the
bottom
example.
B
The
verb
doesn't
match
and
the
request
is
denied
in
these
next
two
examples:
we
are
using
a
team
based
capability
which
depends
on
the
request
matching
with
our
service
metadata.
The
subject:
access
review
only
contains
the
name
of
the
pod,
and
so
the
first
step
is
actually
for
oppa
to
request
the
pod
metadata
from
kubernetes.
B
This
pod
metadata,
then
contains
the
name
of
the
service
and
that
service
can
be
matched
against
our
service
metadata.
In
this
case,
the
service
is
owned
by
intersect
team
and
the
user
making.
The
request
is
also
a
member
of
that
team,
so
the
request
is
allowed
in
the
bottom
example.
The
service
name
is
now
pe
service,
and
so
the
own,
the
owner,
no
longer
matches
with
the
user's
team,
and
the
request
is
denied
note
that
we
could
still
add
a
static
value
into
the
capability.
B
This
has,
of
course,
the
basic
information
like
whether
or
not
the
result
was
allowed
or
denied,
and
also
the
various
input
in
the
subject,
access
review
that
kubernetes
sent
to
oppa
what
was
really
neat
about
it
is
that
it
shows
you
which
groups
would
have
been
allowed
versus
which
groups
the
user
had.
This
makes
it
very
easy
for
us
to
debug
requests
and
to
figure
out
what
capabilities
we
should
add
to
someone
who
needs
to
do
something.
B
A
Our
major
challenge
was
switching
to
a
new
authorization
system
under
heavy
usage
to
avoid
any
system
or
user
disruption.
We
came
up
with
the
following
approach.
First,
we
ensure
that
all
of
the
changes
are
rollback
safe.
Then
we
configured
our
infrastructure
to
support
a
dry
run
mode.
Then
we
rolled
out
the
dryer
mode
incrementally
cluster
by
cluster.
A
A
A
A
A
A
A
The
next
problem
that
I
would
like
to
talk
about
is
how
do
we
associate
the
team
with
non-port
resources
with
metadata,
so
we
previously
talked
about
how
we
use
part
metadata
to
associate
a
request
with
our
own
method
service
metadata
to
make
team-based
access,
but
some
non-part
resources
require
special
treatment,
for
example,
secrets
that
are
bound
to
the
services
that
they
do
not
have
team
membership
associated
with
them
to
enforce
least
privileged
team-based
access
to
the
secrets.
We
came
up
with
a
special
case
handling
in
the
regula
policy.
A
A
This
approach
made
the
regular
policy
long
and
overly
complicated,
and
this
brings
us
to
our
next
problem,
which
that
our
regular
policy
become
overly
complicated
and
hard
to
test
and
debug
so
to
to
make
sure
that
we
have
enough
coverage.
We
wrote
extensive
test
cases,
unit
test
coverage
like
in
fact,
we
had
unit
test
coverage
more
than
two
thousand
lines
of
code,
so
this
was
overall
problems
and
challenges
that
we
encountered
encountered
throughout
this
pro
project.
A
So,
let's
go
through
some
of
these
scenarios.
First,
one
is
that
what
if
we
push
a
bad
policy
that
has
a
catastrophic
side
effects
like
blocking
access
from
old
users,
we
do
enforce
a
strict
code
review
process,
but
nobody
is
perfect.
After
all,
like
some
bad
thing
might
slip
through
in
case
this
happens,
we
have
automated
checks
in
our
ci
cd
pipeline
that
will
prevent
uploading.
A
The
next
case
that,
even
with
the
automated
tests,
it
is
still
possible
that
a
bad
pulse
might
still
true
this
or
any
other
kind
of
issue
with
the
open
policy
agent
service
could
cause
it
not
to
respond
and
block
people's
authorized
into
the
kubernetes
clusters
to
mitigate
that
risk.
We
distribute
administration
keys
to
kubernetes
hots
and
check
cubes
created,
cube,
ctl
admin
reports
that
only
the
system
administrators
can
use.
A
In
our
architecture,
we
rely
on
many
different
systems
and
we
implemented
our
architecture
in
a
fault
tolerant
way,
for
example,
if
for
any
reason,
github
or
active
directory
does
not
respond,
we
have
the
last
frozen
state
of
the
world
in
the
s3.
Bucket
and
system
will
continue
to
work
and
if
any
reason
s3
stops
responding,
we
still
have
the
last
frozen
state
of
the
world
cache
in
the
oppa
at
the
kubernetes
clusters
and
users
still
continue
to
work,
continue
to
access
kubernetes
clusters.
A
A
First,
shortcoming
is
that,
unfortunately,
not
every
resource
has
meaningful
metadata
in
our
infrastructure,
so
for
that
reason
we
cannot
make
a
team-based
access
decision
on
every
resources,
although
these
cases
are
rare
and
but
still
in
the
future,
we
want
to
give
ownership-based
access
to
all
resources
to
be
like
to
to
go
all
the
way
with
the
least
privilege
to
enforce
all
resources
to
have
meaningful
metadata.
We
are
planning
to
use
the
admission
controller.
A
A
The
next
one
is
that
the
octa
authentication
has
one
hour
ttl
and,
to
be
honest,
we
do
not
consider.
This
is
a
shortcoming
and,
in
fact,
from
a
security
engineering
perspective.
This
is
actually
good
for
our
security
posture,
but
we
receive
some
complaints
from
the
users
that
they
have
to
enter
their
password
every
hour.
A
A
So,
to
conclude,
this
talk
with
some
of
our
learnings.
I
would
like
to
first
say
that
if
you
are
making
a
fundamental
shift
with
how
you
interact
with
a
platform
such
as
exposing
more
levels
to
people
or
expanding,
the
surface
area
do
not
just
bluntly
carry
over
the
security
model.
It's
important
to
re-evaluate
the
security
model
at
the
major
system.
Changes
like
this.
A
In
fact,
during
our
project
we
had
a
very
smooth
reviewing
process
to
implement
such
a
large
project
and
since
its
inception,
the
server
the
system
never
malfunctioned
to
prevent
people
to
access
next
is
that
it
is
totally
okay.
The
first
view
the
tools
tools
that
can
support
the
least
privilege
access
without
actually
doing
the
least
privilege
rules,
because
doing
both
can
be
really
challenging,
and
after
some
time
you
can
actually
do
the
least
privileged
setup
as
a
follow-up.
A
At
the
end
of
this
project,
we
obtain
the
ability
to
give
least
privilege
access
to
kubernetes
clusters
based
on
the
many
parameters
need
and
ownership,
and
as
a
result,
we
now
have
a
stronger
security
posture
by
enforcing
lease
privilege,
access
for
hundreds
of
teams
to
thousands
of
different
kubernetes
resources,
and
this
concludes
our
talk,
and
I
would
like
to
thank
all
of
you
for
joining
us
today
and
we'll
be
happy
to
answer
any
questions.
Thank.