►
Description
Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
AuthN and AuthZ at Cruise: Crawl, Walk, Run - James Barclay & Roman Porter, Cruise
Authentication and authorization are crucial to achieving a zero trust security model. But how do you apply them to your organization in a simple and digestible way? Do you prescribe each engineering team what to do and give them the tools to do it? Do you enforce it globally? Our approach to handling authentication and authorization within our Kubernetes clusters at Cruise has matured over the years, and we’d like to take you through the history, challenges, and (what we think are) unique solutions to these problems at our scale.
A
B
B
And
so
I'll
just
start
at
the
beginning.
You
know
this
is
more
of,
like
you
know,
handling
often
an
aussie
at
cruz,
to
be
honest
during
the
early
days,
was
sort
of
like
the
wild
west,
and
this
was
really
more.
You
know.
Our
talk
is
called
crawl,
walk
run.
I'll,
say
this
is
more
of
like
the
embryonic
stage,
but
it's
worth
mentioning
nonetheless.
B
B
And
the
crawl
in
the
crawl
phase,
the
first
thing
we
did
was
we
just
built
a
saml,
authenticating,
proxy
sidecar
and
just
sort
of
hoped
that
people
would
use
it
in.
In
this
case,
the
service
owner
would
add
the
sidecar
to
their
pod
and
then
every
http
request
that
hits
the
the
service
will
eventually,
or
you
know,
would
if
we
would
enforce
authentication
via
a
saml
flow
before
proxying.
The
request.
B
But
in
this
case
we
have.
You
know
these
different
types
of
clients
that
I
mentioned
earlier.
They're
making
requests
from
outside
of
our
you
know
cluster,
and
they
come
in
and
pass
through
this
nginx
ingress
gateway
and
eventually
we'll
end
up
hitting
the
the
service.
In
the
case
of
you
know
this
early
stage
where
we're
using
a
sample
authenticating
proxy,
you
know
once
the
request
would
reach
the
service.
B
This,
the
authenticating
proxy
would
check
to
see
if
the
user
is
authenticated
and
if
not
just
bounce
them
over
to
the
identity
provider,
to
complete
authentication
and
then
once
the
user
is
logged
in
the
sample.
Sidecar
would
set
an
http
header
that
contained
the
subject,
which
was
a
user
or
application,
and
then
it
would
sort
of
be
up
to
the
service
owner
to
figure
out
what
to
do
with
that
in
terms
of
authorization.
B
This
also
required
a
fair
deal
of
setup
for
our
users.
First
of
all,
they
needed
to
like
set
up
the
sidecar.
They
also
needed
to
talk
to
it
to
get
like
a
saml
integration
set
up
with
our
identity
provider
or
to
have
groups
created,
for
example,
and
another
problem
here
is
that
saml
as
a
protocol
for
the
most
part,
isn't
really
friendly
to
non-browser
clients.
So
when
people
needed
to
provide
like
api
access,
for
example,
they
just
passed
like
a
bypass
header
to
circumvent
authen,
entirely,
which
you
know
obviously
isn't
good.
B
B
So
now
I'll
talk
about
the
next
stage,
which
I'll
say
is
the
walk
phase,
and
you
know,
since
I
met
you
know.
I
mentioned
that
you
know
we
realized.
There
were
a
lot
of
problems
in
the
the
crawl
stage.
You
know
to
solve
them,
and
you
know
we
sort
of
set
out
to
to
do
a
couple
of
things
and,
starting
you
know,
starting
by
choosing
an
authentication
and
authorization
standard.
B
We
also
built
a
custom
entitlement
based
authorization
broker
that
we'll
call
c
z.
We
also
set
out
to
build
self-service
tools
to
promote
use
of
this
standard.
One
of
those
such
tools
is
something
called
identity,
orchestration
platform
or
iop,
and
iop
has
a
lot
of
functionality.
But
one
thing
in
particular
that's
worth
noting
here
is
that,
rather
than
users
having
to
create
a
saml
or
oidc
integration
directly
with
an
identity
provider
or
like
askit
to
do
it,
we
built
a
web
a
web
app
that
allowed
them
to
do.
B
B
It's
a
lot
going
on,
but-
and
you
know
you
can
also
tell
that
you
know
roman
made
it
in
my
case
because
he
you
know,
he
worked
for
microsoft
for
a
long
time
and
he
put
the
dll
icon
on
here
so,
but
nonetheless
moving
on
so
yeah
we'll
go
ahead
and
break
it
down.
So
so
we'll
start
with
ciaz,
which
I
mentioned
earlier,
and
I
also
said
that
it,
you
know,
has
to
mint
tokens,
but
you
know
how
does
it
go?
B
B
Once
the
you
know
see,
autistic
system
is
able
to
like
talk
to
the
idp
in
order
to
figure
out
like
what
users
and
groups
are
are
present.
It's
then
able
you
know
or
like
service
owners
at
that
point,
are
able
to
create
applications
in
this
system
and
that
allows
this
authorization
broker
to
you
know,
assign
or
allow
service
owners,
rather
the
ability
to
assign
various
permissions
I
mentioned
entitlements
earlier.
It
is
not
you
know
they
can
assign
permissions
to
various
users,
groups
or
even
other
applications.
B
So
with
this
information,
autism
is
able
to
create
these
tokens,
which
are,
you
know,
contain
claims
that
can
be
verified
and
the
claim
you
know.
Speaking
of
verification,
you
know
the
the
tokens
themselves
are
cryptographically
signed
like
I
mentioned,
but
how
do
you
go
about
verifying
them?
So
in
this
case
we
gave
service
owners
two
options.
Basically,
in
one
case
they're,
you
know
they
could
use
a
library
like
the
opinionated,
often
and
nazi
library
that
I
mentioned
earlier
to
validate
tokens
as
well
as
to
handle
the
oidc
authentication
with
our
identity
provider.
B
How
do
you
get
a
ciat
z
token,
then,
and
in
this
case
we
don't
actually
get
a
c
out
z.
Like
a
jwt
token.
What
happens
in
that
case
is
the
after
the
user
authenticates
the
we
do
like
a
real
time.
Api
call
with
clc
to
see
what
actions
or
you
know,
or
operations
they're
entitled
to
perform,
and
now
I'm
going
to
pass
it
off
to
roman
to
actually
talk
about
some
of
the
problems
with
this
approach
and
then
how
we,
you
know
how
we
built
the
next
stage
of
it
to
address
them.
A
All
right
so
james
talked
about
the
walk
stage,
which
you
know
was
definitely
an
improvement
over
the
crawl
stage.
No
doubt
about
that,
but
it's
lots
of
problems
that
we
didn't
like
so
much,
especially
in
security.
So
you
know
the
first
most
obvious
problem
is
there's
like
no
enforcement
of
any
of
the
stuff
right
like
yeah
we're
providing
some
libraries
and
some
side
cars
and
people
can
use
them,
but
they
don't
have
to.
We
recommend
that
they
do,
but
you
know
coming
from
a
kind
of
a
security
side
right.
A
We
definitely
don't
want
to
be
in
a
position
where
this
is
an
optional
thing.
So
you
know
if
you're
going
to
choose
the
the
library
that
frankly,
a
lot
of
folks
have
you
know
you
need
to
change
your
code
right.
You
need
to
pull
in
this
library.
You
need
to
change
a
bunch
of
code.
You
need
to
call
it.
You
need
to
put
in
the
right
path.
A
You
need
to
also
you
know,
as
this
library
is
changing
right,
like
as
we
fix
it
up,
find
bugs
fix
them.
You
have
to
go
and
update
it
right,
so
there's
a
whole
cycle.
There-
and
you
know
james
mentioned-
that
your
permissions
or
entitlements
are
actually
encoded
into
this
jwt
right,
which
is
great,
but
you
know,
as
these
applications
evolve,
they
become
bigger
and,
as
they
become
bigger,
a
user
has
more
permissions
to
various
aspects
of
you
know
this
application.
A
Need
to
be
shoehorned
into
the
token
so
now
the
token's,
growing
and
growing
in
these
tokens
jwts
are
sent
as
headers
right
and
then
that
means
the
header
size
is
growing
and
as
that
grows
more
and
more
right,
then
you'll
hit
the
dreaded.
I
think
it's
http
431
or
something
like
that
for
24.29
for
29..
Sorry-
and
you
know-
that's
really
hard
to
debug
right
because,
depending
on
the
state
of
your
headers
at
any
given
time,
this
will
fail
or
succeed.
So
it's
just
not
great.
A
You
know
for
us
in
security
right,
that's
great,
that
we're
providing
all
these
libraries
and
side
cars
for
folks.
But
you
know,
maintaining
them
is
really
hard.
You
know,
at
the
end
of
the
day,
we're
kind
of
a
security
team,
we're
not
just
like
a
development
team,
but
we
have
to
not
only
support
these
libraries
but
support
them
in
like
four
or
five
different
languages
right
with
their
own,
like
weird
gotchas,
without
really
a
common
core,
and
you
know
as
these
languages
languages
change.
A
You
know
the
libraries
that
they
require
change
and
we
have
to
like
figure
all
that
out
and
then
you
know
build
them
for
every
different,
like
every
language
has
its
own
package
manager.
So
it's
pretty
difficult,
but
even
then
you
know
when
we
build
it.
Let's
say
we
figured
all
that
out
and
we
kind
of
did
for
a
while
to
get
folks
to
use.
The
latest
version
is
very
difficult,
as
you
know
you
might
imagine,
so
you
know
we'll
what
what
ends
up
happening
to
us.
A
A
We
the
only
way
we
could
do
that
is
then
we
have
to
go
to
every
service
and
say:
hey.
You
need
a
bunch
of
code
changes.
You
need
a
bunch
of
code
changes
right,
that's
again,
not
scalable.
So
to
move
on
to
the
run
phase.
Well,
I
know
we
initially
said
run
phase,
but
it's
more
of
a
speed,
walking
phase
for
us
a
little
presumptuous
to
say
that
we're
at
the
run
phase,
but
there's
a
bunch
of
things
that
we
kind
of
want
to
improve
here
right.
A
A
You
know
it
has
authentication,
it
has
authorization
right
just
from
the
get
go
and
then
for
some
reason
you
want
to
do
something
custom,
then
maybe
you
don't
want
our
authentication
authorization,
that's
cool
too,
but
then
you
have
to
opt
out
of
that
right,
whereas
you
know
kind
of
in
this
state,
you
have
to
specifically
add
authentication
authorization
to
your
application.
So
the
next
thing
is,
you
know
we
want
to
you
notice.
We
built
a
lot
of
libraries
and
site
cars
and
stuff.
You
know,
which
is
great.
A
You
know,
gave
us
something
to
do
for
a
while.
But
ideally
you
know
we
don't
really
want
to
support
all
that
stuff
going
forward.
As
I
mentioned
all
the
problems
around
it.
So
we
want
to,
you
know,
participate
in
this
amazing,
open
source
ecosystem,
and
you
know
both
use
the
fruits
of
other
people's
labor.
A
Other
people
can
review
it
and
you
know,
verify
it
and
make
sure
it's
okay
without
having
to
dive
into
the
application
itself,
and
you
know
obviously
right
as
if
that
wasn't
tough
enough.
We
want
to
do
all
this
without
going
to
every
service
and
saying
you
got
to
change
a
bunch
of
code
right
because
you
know,
frankly,
anybody
got
time
for
that.
A
So,
let's
start
here
right,
so
the
first
move
that
we
wanted
to
make
is
from
using
the
nginx
ingress
to
using
an
ister
ingress.
Now,
why
would
we
want
to
do
that?
Well,
the
ester
ingress
yeah,
it
is
actually
powered
by
envoy
and
envoy-
has
a
lot
of
a
lot
of
cool
things
outside
of
this.
But
one
cool
thing
that
we
wanted
to
leverage
was
its
extensibility
mechanisms.
It's
got
a
bunch
of
those.
One
of
them
is
called
external
authorization.
A
A
So
that's
very
helpful,
but
in
addition,
if
you
deny
the
request,
you
can
actually
mutate
it
as
well
right.
So
we
leverage
that
by
having
another
service
live
alongside
the
ingress
and
hooking
into
the
ingress,
via
this
external
authorization
mechanism
that
I
described
and
for
every
request,
this
other
service
called
cappy
basically
takes
a
request
and
says:
hey.
Is
this
request
authenticated
and
if
it's
not,
it
denies
the
request,
but
it
actually
denies
it
by
overwriting
the
response
and
sends
the
user
302
to
the
idp.
A
Now
the
user
completes
the
idp
flow
right
logs
in
and
because
we,
when
we
do
that,
302
we
actually
set
a
redirect
url
for
the
idp,
the
user
gets
bounced
all
the
way
back
to
where
they
came
from
to
their
original
request.
This
time
you
know
cappy
looks
at
the
request
and
says:
okay,
you've
already
authenticated
with
the
idp,
I'm
happy
with
that.
I'm
gonna
actually
just
set
a
cookie
just
so
you
don't
have
to
go
through
this
flow
again
and
then
it
lets
the
request
through
right.
A
A
So
requests
been
authenticated,
that's
great,
but
you
know
we
also
want
authorization.
Obviously,
so
how
do
we
do
that?
Well,
we
decided
to
move
to
a
different
site.
Car
you'll
notice,
they're
different
from
the
one
on
the
right,
and
this
is
an
istio
proxy
sidecar
right
because
now
we're
living
in
the
esteo
ecosystem
now
istio
proxy.
A
What's
really
cool
about
the
esto
proxy
sidecar
is
guess
what
it's
also
envoy,
it's
the
same
envoy,
but
this
time
instead
of
living
kind
of
globally
in
the
cluster
it's
living
inside,
of
the
the
context
of
the
service
itself
right
basically,
but
doing
the
same
thing
right.
So
it
takes
every
request
and
proxies
it
through
now,
because
it's
envoy
guess
what
extensibility
mechanism
it
has
external
authorization
right.
So
let's
use
that
again,
but
this
time
slightly
more
appropriately
by
actually
allowing
or
disallowing
the
request
right.
A
So
we
need
something
else
to
hook
into
this
envoy
right
to
allow
or
disallow
the
request
we
had
capi
globally
right.
So
what
are
we
gonna
do
here?
Well,
there's
another
really
cool
project
called
open
policy
agent
right
so
open
policy
agent.
What
it
allows
you
to
do
is
it
allows
you
to
hook
it
up
to
your
sidecar
via
external
authorization
and
then
for
every
request:
open
policy
agent.
A
Basically
lets
you
write
a
policy
in
this
language
called
rego
and
basically
the
policy
is,
if
you
think
about
it,
as
as
implementing
a
function
that
always,
you
know,
returns
true
or
false
right.
So
you
can
have
kind
of
a
big
flow
of
things
that
you
can
do,
but
at
the
end
of
the
day,
you're
returning,
true
or
false
you're,
either
letting
those
requests
through
or
you're,
not,
and
in
our
case
what
we
did
was
we
actually
used
the
same
exact
policy
for
every
service
in
our
cluster.
A
We
were
able
to
do
that,
because
open
policy
agent
also
lets
this
policy
be
augmented
with
additional
data.
You
can
think
of
it
as
like
parameters
to
a
function
right,
so
the
actual
implementation
of
the
rego
policy
really
all
it
does.
Is
it
says,
okay,
like
give
me
the
the
routes
that
give
me
the
the
route
that
the
request
is
trying
to
access
like
the
path
right.
A
Let's
say
the
request
is
trying
to
access
my
application,
slash
admin,
so
it
says
okay,
you're
trying
to
access
slash
admin,
so
it
gets
that
from
the
request,
and
then
it
says:
okay,
this
is
additional
data
that
it
has
specific
to
the
application.
So
it's
the
access
admin
which
permissions
do
you
need?
So
it
looks
at
that
information
and
then
it
looks
at
the
request
again
to
look
at
the
incoming
token,
the
jwt
right
that's
attached
as
a
header,
as
we
said
and
says:
okay,
does
this
token?
A
Have
those
permissions
right
and
that
operation
of
matching
the
routes
and
their
permissions
to
the
token
permissions
is
the
same
across
all
applications
right?
The
only
thing
that's
different
is
that
those
two
lists
right.
The
lists
of
you
know
which
routes
need
which
permissions
and
you
know
the
actual
token
itself
right.
A
So
you
might
be
wondering.
Okay,
that's
cool
and
everything,
but
you
know,
as
I
mentioned
right
like
the
open
policy
agent
actually
needs
to
have
some
data,
that's
specific
to
every
application
like
these
extra
files
right.
So
where
does
that
come
from?
Well,
the
other
cool
thing
that
it
has
is
it
supports
something
called
a
bundle,
api,
so
bundle
api
is,
is
really
just
a
polling
mechanism
using
http
that
it
can
pull
various
cloud
providers,
so
it
already
supports
polling
like
his
three
buckets.
A
A
So
it
knows
if
there's
a
change
and
if
there
is
it
downloads,
it
loads,
the
new
policy,
the
new
brego.
In
our
case,
it's
always
the
same,
just
loads,
the
additional
files
and
it's
good
to
go
now.
How
does
this
bucket
get
populated?
Well,
remember.
James
talked
about
clz,
having
all
this
context
right
having
the
users,
their
groups,
the
various
permissions
that
users
are
allowed
to.
You
know
action
on
a
given
application.
A
Well,
all
we
did
was
we
took
all
the
clz
right.
We
serialize
all
that
data
on
a
regular
cadence
and
we
write
it
to
each
application's
bucket.
Basically,
so
you
know
that's
great
and
all,
but
you
know
it
can
be
possibly
that
simple,
so
it
wasn't
and
it
isn't
so.
You
know,
let's
kind
of
go
step
by
step,
to
see
what
the
challenges
were
so
and
kind
of
how
we
dealt
with
some
of
them,
so
basically
moving
from
nginx
to
envoy.
A
You
know
that
pretty
much
involves,
for
the
most
part,
taking
an
ingress
resources
defined
in
each
service
and
converting
it
to
kind
of
a
matching
virtual
service
resource.
They're
kind
of
analogous,
not
exactly
but
kind
of,
and
istio
has
a
utility
that
lets
you
kind
of
convert
from
one
to
the
other,
but
it
doesn't
always
work.
You
know,
depending
on
the
complexity
of
your
ingress
resource
and
the
routing
rules
that
you
set
up.
So
you
know
that
required
some
hand-holding
and
kind
of
doing
some
things
by
hand.
A
One
thing
that
I
want
to
mention
that
we
did
there,
which
was
a
good
idea
and
I
think
it
helped
folks
is.
We
actually
ran
both
side
by
side
for
every
cluster
and
the
way
we
decided.
You
know
which,
like
when
you
do
a
request
to
a
service
which
ingress
to
use
is
just
simply
by
the
dns.
So
the
dns
of
every
application
was
mapped
to
the
cname
to
the
dns
of
whichever
ingress
they
were
using.
A
So
in
the
beginning
they
were
all
cnamed
to
the
nginx
ingress
and
as
folks
you
know
made
changes
right.
They
could
actually
create
a
new
dns
that
c
name
to
the
envoy
ingress
to
the
steering
grass
and
then
just
test
that
out
on
their
own.
You
know
mucking
around
with
a
virtual
service,
but
while
they're
doing
all
that
nobody,
you
know
the
ingress.
The
nginx
ingress
doesn't
look
in
that
data,
so
it
doesn't
break
anything
so
they
can
iterate,
they
can
make
sure
it
works.
A
And
then,
when
they're
happy
with
it,
they
just
switch
over
the
dns
to
the
environments
and
off
they
go
so
that's
kind
of
how
we
are
handling
the
nginx
envoy,
ingress
migration.
So
with
capi
you
know
we
obviously
want
to
enable
it
globally
on
the
whole
cluster.
But
you
know
we
got
to
start
with
still
an
opt-in
model
make
sure
you
know
things
don't
break
and
the
way
we
did
that
is.
A
We
basically
had
people
add
an
annotation
in
the
virtual
service
and
then
we
have
cappy
basically
scanning
the
cluster
and
saying
hey.
If
I
see
a
service
with
this
annotation,
I'm
gonna
authenticate
traffic
routing
to
it.
If
I
don't
see
that
annotation,
I'm
just
going
to
step
out
of
the
way
I
mean
this
is
kind
of
like
a
solution
for
the
migration.
Now
that
code
is
going
to
get
completely
removed.
You
know
once
everybody's
migrated,
so
you
might
be
wondering
like
a
lot
of
these
applications.
A
You
know
as
services
as
james
said,
you
know,
are
using
these
libraries.
These
legacy
site
cars
legacy.
Libraries-
maybe
doing
things
on
their
own.
So
how
do
we
handle
that?
Well,
the
good
news
is
that's
totally
cool
like
we
can
have
multiple.
We
can
do
authentication
multiple
times
and
that
still
achieves
their
goals.
So,
for
example,
what
can
what
can
happen
is
you'll
get
a
request
into
the
cluster.
A
Cappy
will
do
that
whole
magic
of
authenticating,
the
request,
the
request
will
go
to
the
the
actual
service
and
the
service
will
authenticate
it
again.
Right
sorry
authenticate
it
again:
yeah
security
people
don't
install
their
updates
anyways,
but
the
good
news
is
that
our
idp
already
has
a
session
for
that
user
and
so
really
it'll
just
be
an
additional
redirect.
So
from
a
user
perspective,
it's
not
a
perceivable
difference,
even
though
authentication
is
technically
happening
twice
right
and
that,
like
kind
of
latency
of
that
additional
redirect
is
only
present.
A
While
you
have
this
two
authentication
mechanism,
so
once
that's
removed,
you
know
you
don't
have
to
do
that.
But
the
cool
thing
is,
you
know
we
can.
We
can
still
enable
it
and
say
the
entire
cluster
has
authentication
some
folks
have
it
twice,
but
none
have
it
zero
times
right.
So
that's
the
goal,
and
then
we
can
also
integrate
with
some
of
these
libraries
and
site
cars
that
we
provided
previously,
because
you
know
we
do
own
them.
A
So
we
can,
you
know,
add
some
code
there
to
make
cappy
kind
of
more
intelligently
reason
about
some
of
these
services.
Yes,
it
requires
folks
to
upgrade
to
the
latest
versions-
and
I
mentioned
some
of
the
problems
with
that,
but
that's
kind
of
another
option.
So,
moving
on
to
enable
the
steel
proxy
stu
makes
it
relatively
simple.
You
just
need
to
set
this
annotation
and
the
istio
proxy
will
be
injected
into
the
pod.
You
can
set
the
sanitation
at
you
know
the
name
space
level,
the
pod
level,
a
bunch
of
different
levels.
A
Once
you
have
that
you
can
actually
start
working
with
the
opa
agent,
the
open
policy
agent
and
that's
again
opt-in.
So
you
set
the
soapy
agent
enabled
annotation,
and
then
you
get
this
additional
sidecar.
Now
we're
up
to
two.
You
can
see
how
many
sidecars
you
can
shove
into
every
pod
and,
as
I
mentioned,
you
know,
there's
a
single,
regular
implementation
of
the
policy
in
every
for
every
service
and
really
that's
doing
what
we
like
to
call
stateless
authorization.
A
So
basically
it's
just
looking
at
the
request,
just
looking
at
the
token
just
looking
at
the
local
data
that
it
has
to
decide
if
that
request
should
be
authorized
or
not.
Now,
if
a
service
needs
to
do
finer,
grain
authorization,
so,
for
example,
it's
maybe
it
has
a
matching
database
that
it
needs
to
look
up
a
user
in
and
kind
of
augment
this.
This
authorization
data
it
can
still
do
that
right.
So
the
idea
is
that
you
know
the
ob
agent.
A
Does
one
authorization
step
right
and
then
it
can
be
authorized
further
by
the
application
itself,
if
they
so
choose
to
do
that
right
so
again,
you're
doing
kind
of
authorization
twice,
but
one
is
coarse.
Grain
one
is
fine
grain
and
then,
lastly,
our
clz
broker
right
needs
some
new
ui
so
that
you
can
go
and
define
some
of
these
policies.
You
can
define.
A
You
know
different
routes,
as
I
mentioned
like
you,
have
an
admin
route,
and
these
are
the
permissions
that
are
required
to
access
it,
and
you
know
obviously
folks
don't
just
want
to
use
the
ui
they
want
to
use
the
ci.
You
know
the
current
ci
pipeline
to
deploy
some
of
the
stuff,
so
we
also
have
that
as
well
for
them
and
then
lastly,
you
know,
as
I
mentioned
we
want
to
in
the
background
serialize
all
this
information,
including
the
group
memberships,
and
push
all
that
to
those
buckets.
A
A
That's
not
something
we've
been
able
to
say
before
before
it
was
kind
of
like:
maybe
they
do,
maybe
they
don't
authorization
policy
now
does
not
live
in
the
depths
of
every
service.
It
actually
lives
in
our
broker.
Right
that
can
be
reviewed,
can
be
exported
our
clc
broker.
I
mean
it
could
be
audited
centrally
right,
we're
no
longer
using
you
know,
custom
tools
that
we've
built
and
have
to
support
right,
we're
using
open
source
tooling,
and
the
best
part
of
this
is
actually
you
know,
we're
not
proxying
the
traffic
ourselves
right.
A
We're
using
envoy,
which
is
you
know,
kind
of
a
battle
tested
piece
of
software
to
do
all
the
proxying
and
traffic
management
and
all
the
cool
stuff
that
it
does
we're
just
kind
of
hooking
into
it
to
provide
to
augment
some
of
its
capabilities
a
little
bit
right,
but
we're
not
doing
it
ourselves
and
best
of
all.
You
know
now
we
kind
of
control
both
the
minting
part
of
this
equation,
right,
the
creation
of
these
tokens
and
also
the
validation
right
and
all
the
way
through
the
entire
channel.
A
A
So
that's
pretty
cool,
but
there's
more
right,
there's
more
that's
the
image
I
should
have
used.
So
there's
more
that
we
can
do
here.
We
because
we
have
these
osteocyte
cars.
We
can
actually
do
transparent
mtls
between
services
right.
Why
can
we
do
that?
All
of
a
sudden
well
remember
that
those
istio
proxy
site
cards
are
doing
both
ingress
and
egress
filtering
for
each
service.
A
It
actually
funnels
through
those
side,
cars
right,
so
it
goes
egresses
out
of
its
internal
sidecar
and
ingresses
into
the
destination
services
sidecar,
and
that's
even
cooler,
because
now
you
can
actually
have
mtls
between
the
ingress
itself
and
the
service
right,
so
that
channel
is
usually
not
secured
right
so
like,
for
example,
if
the
ingress
were
to
add
any
headers
right
as
part
of
that
request.
Technically
speaking,
you
can't
trust
those
headers
right
because
that
channel
is
not
secured.
But
if
you
have
mtls
between
those
two
endpoints,
those
headers
can
now
be
trusted.
A
Also,
mtls
you
know,
brings
you.
Certificates
certificates
usually
bring
some
form
of
identity
that
can
be
used
here,
and
you
know
we
have
that
now
and
so
now,
instead
of
using
something
like
api
keys
for
services
to
authenticate
and
authorize
themselves,
to
like,
say
the
clc
broker
or
other
other
systems,
they
actually
they
can
use
their
own
service
identity
that
is
actually
rooted
in
the
the
cloud
provider's
identity
right
in
terms
of
the
rego.
A
You
know,
we've
had
I'll
give
you
one
example
of
a
request
that
we
had
so
you
know
somebody
wanted
to
authorize
based
on
the
method
of
the
of
the
http
request.
So,
as
I
mentioned,
you
know
you
can
only
right
now
authorize
based
on
the
route
right.
So
choose
admin
right.
Sorry,
but
if
you
wanted
to
say
hey,
you
can
access
this
route,
but
only
as
a
get
in
this
other
route.
A
Only
if
it's
a
post,
you
know
that's
not
possible
in
our
the
regular
that
everybody
uses
today
right
but
it'd,
be
really
cool,
it'd,
be
trivial
to
add
this
functionality
right.
So
we
want
to
move
to
a
world
where
this
rego
is
actually
it
can
be
modified
by
services.
Service
owners
themselves
to
fit
their
needs,
but
we
don't
really
want
to
just
unleash
rego
on
them,
as
is
it's
very
complicated.
A
So
what
we're
thinking
is
we
want
to
provide
some
libraries
as
well
as
kind
of
like
a
ci
pipeline
with
some
linting
validation
and
various.
You
know
approvals
and
various
other
things
so
that
when
they
make
a
change
you
know
we
can
have
some
confidence
that
it
is
doing
what
they're
intending
that
it
would
do.
A
B
A
Yeah,
so
the
question
was:
how
do
you
do
service
to
service
authentication
and
in
authorization?
I
guess
and
do
we
use
jwts
or
mtls
so
right
now
we're
still
using
jwts
because
we're
not
fully
in
the
istio
model
of
having
mtls
available.
Now
we
do
when
we
do
have
mtls
available.
What
we're
thinking
is.
We
would
use
mtls
again
use
the
same
coarse
grain,
fine-grain
method
right
so
use
coarse-grained
authorization
to
say,
like
hey.
A
A
Yeah,
that's
a
really
good
point.
You
know
right
now,
we're
just
simply
scale
it
and
and
use
kubernetes
kind
of
to
handle
that
you
know.
If
it
blows
up,
we
you
know
have
more
instances
of
instances
of
it
available
and
we
also
have
multiple
clusters.
So
we
have
cappy
kind
of
one
per
cluster,
so
we
don't
have
one
like
globally
right,
so
one
is
kind
of
matched
up
with
an
istio
ingress
and
you
have
you
know
one
is
here
ingress
per
cluster,
so
I
think
we
just
have
maybe
more
clusters.
A
B
Yeah
so
yeah,
I
think
it
it.
It
did
like
predate
both
roman
and
I
at
cruz,
but
I
think
like
it
was.
It
was
originally
something
that
was
built
by
like
a
product
team
for
things
to
do
with,
with
like
our
our
product
and
the
car
and
and
sort
of
like
we
inherited
it,
because
it
was
responsible
for
like
dealing
with
you
know,
making
authorization
decisions.
B
I
think
it
also
like,
provided
you
know,
quite
a
bit
of
flexibility
in
terms
of
giving
these
like
service
owners
that
are
like
registering
their
applications
with
this
ciat
system.
The
ability
to
like
define,
like
I
mentioned
entitlements
so
basically
like
being
able
to
express
like
what
what
operations
are
able
to
be
performed
by
different
human
and
non-human
subjects
in
like
a
really
flexible
way.
I
don't
really
know
of
anything
like
like.
There
are
a
handful
of
other
like.
A
B
Guess
maybe
my
follow-up
question
would
be
like
what
like
what
alternatives
are
there
like?
We
did
also
like
at
a
later
point
kind
of
investigate
using
like
oauth
for
authorization,
but
it
really
isn't
the
same
type
of
you
know
it
didn't
really
work
very
well
for
us
in
terms
of
yeah.
You
know
it's
like
handling
like
a
delegated
authorization,
but.
A
A
That's
existed
in
the
past
with
basically
all
our
other
solutions
as
well,
and
so
you
know,
sometimes
people
will
have
like
test
accounts
that
they'll
use
in
a
different
dev
environment
as
opposed
to
using
the
production
environment
and
then
have
a
cluster.
That's
specific
for
for
that
use
case.
If
they
need
complete
end
to
end
right,
there's
nothing
like
if
they're
already
in
the
cluster
right,
then
they
can
do
that
without
ever
interacting
with
cappy.