►
From YouTube: How Yelp Moved Security From the App to the Mesh with Envoy and OPA - Daniel Popescu & Ben Plotnick
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
How Yelp Moved Security From the App to the Mesh with Envoy and OPA - Daniel Popescu, Yelp & Ben Plotnick, Cruise
From its inception, Yelp's service infrastructure has treated security as a fundamental component. For many years, developers carried the burden of building security features directly into their services. By using standard cloud native building blocks, the service infrastructure now provides security features by default; this enables hundreds of developers to focus on shipping features for more than 100M monthly active Yelp users. This talk will cover Yelp’s journey from a legacy service proxy to a modern, secure service mesh based on Envoy and Open Policy Agent. It will discuss -Authn and Authz mechanisms using mTLS and JWT with Envoy and OPA -Migration from using an in-house policy decision engine to standardized open source tools (OPA) -Transpiling legacy policy data to rego and other best practices for policy maintenance -Strategies for quickly and safely rolling out policy changes
https://sched.co/UaZT
A
All
right
so
hi
everyone,
my
name,
is
Rita
I'm.
The
truck
host
for
this
session,
so
I
just
want
to
say
before
the
session
starts,
that
if
you
have
any
feedback
comments,
concerns
questions
please
feel
free
to
reach
out
to
me.
I
have
a
very
physical
shirt
and
also
to
remind
you
to
fill
out
the
surveys
at
the
end
of
the
session.
So
the
next
session
combines
a
few
of
the
most
hot
buzz
words.
A
B
Thank
you
now
before
we
begin
we'd
like
to
extend
a
huge
thank
you
to
the
cloud
native
computing,
Foundation
and
Kubb
Khan
for
the
opportunity
for
us
to
share
our
story
and
our
experiences
with
all
of
you,
who
are
we
I'm,
Daniel
Papa
skew.
This
is
the
most
professional
picture
of
myself.
I
have
on
file.
I
am
an
infrastructure
security
engineer
at
Yelp.
B
In
my
free
time,
I
enjoy
yoga
flow
arts
and
for
some
reason,
reading
AWS
I
am
policy
Docs
and
one
of
those
not
like
the
others,
and
this
has
been
he's
a
senior
platform
engineer
at
cruise
automation
and
at
the
time
that
we
submitted
this
talk,
we
were
both
working
at
Yelp
together
now.
The
intention
of
this
talk
is
not
to
tell
you
how
to
make
your
environment
more
secure.
There
is
no
one-size-fits-all
security
solution
that
will
work
for
any
one
person
or
organization.
B
B
Now,
what's
Yelp
in
case
anyone
is
unfamiliar
with
Yelp.
Let
me
fill
you
in
Yelp
is
a
company
that
connects
people
with
great
local
business.
Yelp
has
been
providing
this
priceless
service
to
business
owners
and
consumers
for
the
past
15
years,
15
years,
Wow
15
years
is
a
really
really
long
time.
A
lot
can
change
in
15
years.
Right,
Yelp
certainly
has
so
has
the
Internet
as
a
whole.
B
Think
about
it
back
then
things
were
so
simple
single
codebase
single
database,
a
couple
of
front-end
web
servers,
handful
of
developers
to
build,
deploy
and
maintain
the
website
reviews
and
all
other
user-generated
content
are
publicly
visible
on
the
site
for
all
visitors
to
see,
there's
not
really
a
whole
lot
of
interesting
stuff
here
from
a
security
perspective,
but
if
we
fast
forward
to
the
present
day,
2019
things
get
a
little
more
interesting.
The
internet
has
changed
dramatically
with
the
introduction
of
social
media,
personalization
background,
location,
tracking,
etc,
etc.
B
It's
now
the
norm
for
users
to
exchange
their
personal
data
in
exchange
for
awesome
features
to
provide
happiness
and
convenience,
such
as
private
messages
to
business
owners,
GPS,
location-based,
check-ins,
credit
cards
for
purchasing
food,
the
list
kind
of
goes
on
and
on
and
on
and
Yelp
takes
the
security
and
privacy
of
our
end
users
very
seriously.
As
such,
we
have
to
ensure
that
we
handle
this
sensitive
data
appropriately
right.
So
how
would
we
do
that?
B
Well,
a
common
solution
to
this
problem
is
the
walled
garden
approach.
This
intuitively
makes
sense,
especially
when
there's
only
one
special
service
that
needs
to
be
secured.
Let's
say
we
have
a
monolith,
application
called
Yelp
main
and
it
has
a
bunch
of
functionality
such
as
handling
reviews
and
search
requests
and
stuff
like
that,
but
we
also
want
to
start
handling
credit
cards.
B
Well,
we
could
put
that
in
the
monolith,
but
maybe
that's
not
necessarily
appropriate.
So
this
is
something
that
sounds
really
sensitive.
We
want
to
secure
it
somehow,
so
we
can
extract
the
code
that
handles
the
payments
into
its
own
separate
service
and
then
introduce
a
bunch
of
bespoke
infrastructure.
Components
makes
this
service
difficult
to
access,
firewalls,
bastions,
V,
pcs,
special
access
deployment,
considerations,
etc,
etc.
This
is
not
exactly
an
elegant
or
lightweight
solution,
but
it
kind
of
meets
the
requirement
of
securing
the
sensitive
data
such
as
this
credit
card
information.
B
So
then,
as
additional
scenarios
arise
that
involve
sensitive
data,
it
becomes
apparent
that
this
solution
won't
just
like
work
for
all
the
other
sensitive
services
that
come
up
the
sensitivity
and
access
control
requirements
are
likely
to
be
unique
for
each
of
these
different
services.
As
such,
we
have
to
implement
similar
same
same
but
different,
independent
solutions
for
each
one
and
recall
the
solution
was
not
exactly
lightweight
or
straightforward
or
cost
effective.
So
now
we
have
a
problem
given
that
we
have
an
increasing
number
of
sensitive
services.
How
do
we
generalize
our
security
features?
C
Yeah,
if
we
were
starting
Yelp
today
yeah,
maybe
we
would
go
with
SEO.
But
again
this
is
kind
of
what
Yelp
looks
like
we
have
our
own
platform
as
a
service
called
pasta.
It's
got
kubernetes
Amazo,
suite
of
smart
stack.
We
have
multi-tenancy
I
mean
it's,
it's
really
really
complicated
and
we
have
a
lot
of
developers
that
are
constantly
working
on
it
so
like
how
do
we
make
this
more
secure
without
disrupting
developers
like
but
another
way?
How
do
we
incremental
e
build
security
into
this
system?
C
C
Yeah
authentication
refers
to
who
you
are,
and
authorization
refers
to
what
you're
permitted
to
do.
So
we
can't
determine
what
you
can
do
unless
we
know
who
you
are.
So
that's
why
we're
going
to
start
with
authentication
as
I
said,
it's
who
you
are
and
kind
of
prove
it.
Some
examples
are
like
username,
password,
API
keys,
x.509.
C
Certificate
to
come
up
with
our
own
kind
of
specific
requirements,
so
these
three
are
kind
of
generally.
What
we
came
up
with
authentication
has
to
handle
the
identity
resolution
of
computers
in
humans
which
I'll
get
into
in
a
second.
The
identity
should
be
unforgeable
and
we
really
want
to
avoid
impacting
developer
velocity.
It's
it's
very
common,
for
you
know
security
to
come
in
and
say,
like
you
can't
do
this,
but
that
puts
a
huge
damper
on
developer
productivity,
so
yeah
huge
disclaimer,
this
you're
you're
gonna,
do
this.
C
C
Okay,
let's
build
some
context.
I
showed
this
diagram
earlier.
It
may
seem
complicated,
but
we
can
simplify
it
with
some
block
diagrams.
Basically,
we
have
a
bunch
of
containers
running
on
a
bunch
of
hosts.
We
have
a
single
what
we
called
egress
envoy
used
to
be
HT
approach,
a
chia
proxies
swapped
out
for
Anglais
check
out
my
own
boycott
talk
from
last
year,
for
if
you
wanna
know
how
that
runs
locally,
one
per
host
and
services
use
that
to
proxy
their
connections
directly
to
a
service.
C
We
don't
support
sidecar
containers
for
historical
reasons,
mais
us,
which
makes
things
more
complicated,
I
lost
my
mic.
Oh
there
we
go,
let's
see
what
it
takes
to
add
authentication
in
a
way
that
meets
our
requirements.
Ok,
let's
start
with
services,
authentication
authenticating.
No,
we
don't
think
that
we
came
up
with
an
optimal
solution
from
the
start
and
we
definitely
didn't,
and
nor
do
we
think
that
we
have
an
optimal
solution.
Now,
it's
an
iterative
process,
so
here's
kind
of
our
iteration.
C
So
the
first
thing
that
we
can
do
is
nothing
that's
kind
of
the
easiest
thing
to
do.
You
see
our
simplified
view
of
a
service
mesh
here.
Maybe
you
do
some
network
segmentation
create
some
walled
gardens,
but
inevitably
somebody
comes
in
and
they
you
know,
make
a
request
and
maybe
get
some
of
your
data,
and
maybe
that's
fine.
If
you
don't
have
any
sensitive
data,
then
by
all
means
let
your
data
flow
free.
C
That's
that's
one
option.
Unfortunately,
that's
not
a
viable
option
for
us,
so
the
next
thing
you
might
do
is
come
up
with
an
ad-hoc
solution
for
applications.
One
solution
we
had
was
to
pass
a
client
identifier
in
the
client
libraries.
We
we
had
a
common
set
of
client
libraries,
so
this
was
a
relatively
easy
solution.
The
server
can
look
at
the
client
identifier
and
say:
ok,
I
know
who
you
are
you're.
What
I
call
it
service
a
here
in
some
header,
but
not
only
is
this
easy
to
forge.
C
It
was
inconsistently
implemented
across
different
languages
and
frameworks.
We
did
have
common
client
libraries,
but
we
also
had
different
client
libraries
for
different
languages.
We
could
sell
solve
the
forge
ability
problem
by
doing
something
like
signing.
That
would
require
extensive
client
and
server
library.
Support
that
doesn't
really
solve
our
satisfier
and
developer
velocity
requirement.
C
Okay,
let's
now
not
trust
the
application
to
do
anything,
we'll
pull
it
out
of
the
application
and
put
it
into
the
proxy.
Here
we
have
like
this
workload:
identity
map,
it
basically
maps
the
IP
of
the
workload
to
some
identity,
and
we
could
put
in
some
checks
in
our
proxy
to
make
sure
that
only
the
proxy
can
inject
this
workload.
Workload
identity.
This
actually
kind
of,
looks
pretty
familiar.
This
looks
a
lot
like
spiffy
spire.
If
you're
familiar
with
it,
we
kind
of
backed
into
that
but
yeah.
C
This
is
what
we
came
up
with
again
to
follow
a
theme.
We
have
a
slight
problem.
If
we
look
at
our
threat
model,
it's
still
spoof
able
and
the
server
needs
to
validate
it.
So
so
we
we
don't
satisfy
our
unforgeable
'ti
requirement
and
we
still
need
developers
to
do
something.
That's
fine
one
problem
at
a
time:
okay,
let's
stick
another
envoy
on
the
other
side,
this
is
starting
to
look
like
a
service
mesh
buzzword
check.
C
This
is
now
what
we
call
full
mesh
and
it's
able
to
handle
authentication
on
both
sides
without
the
application
being
involved
at
all,
that's
great,
but
you
can
probably
guess
what's
coming
next
devil.
Emoji
guy
can
just
spoof
the
identity
of
one
service,
but
the
last
step
is
really
really
easy.
We
put
locks
on
our
diagram
and
things
are
secure.
C
Yeah.
We
can,
we
can
add
em
TLS
to
to
make
sure
that
we're
only
trusting
this
identity
if
it
comes
from
an
envoy
that
we
trust
the
envoy
proxies
authenticate
against
each
other
and
glossed
over
time
details,
but
we're
multi-tenant.
So
this
is
really
important
that
we
have
that
we
have
this
on
board,
that
we
can
trust
and
makes
the
it
puts
the
identity
in
for
us.
C
C
C
C
C
One
interesting
thing
here
is
that
we
wrapped
curl
so
you'll
notice
like
down
there.
We
have
yelp
curl,
which
is
our
version,
and
this
is
nice
actually,
because
this
abstracts
away
all
of
the
details
of
like
getting
a
token
and
passing
it
in
the
headers
and
we'll
see
in
a
second
that
that
that's
kind
of
important
one
problem
with
this
solution
is
again.
It
requires
the
service
owners
to
do
work
and
that
doesn't
satisfy
our
developer
velocity
requirement.
C
So
you
should
have
guessed
well,
I
was
going
to
say
next,
we
pulled
it
out
of
the
application
and
stuck
it
in
envoy
envoy
has
native
support
for
for
JSON
web
tokens.
We
hooked
into
our
existing
SSO,
which
was
octa
so
op.
What
octa
does?
Is
it
distributes
short-lived
tokens?
That
envoy
can
then
verify
envoy
yeah
natively
supports
this.
C
It
grabs
the
jwk
from
octa
and
can
verify
the
the
validity
of
the
token
and
service
a
can
not
worry
about
this
at
all,
and
they
can
spend
its
request
budget
on
things
that
it
actually
needs
to
do
like
business
logic
and
again.
Developers
didn't
even
know
that
we
did
this
because
we
swapped
out
the
implementation
in
this
yelled
curl
abstraction.
So
this
was
really
nice
in
terms
of
a
migration
path.
C
Okay,
now
we
have
identity
for
humans.
The
identity
is
unforgeable,
it
has
a
short
TTL,
so
if
it
gets
stolen,
it
can
only
be
used
for
a
short
period
of
time
and
because
of
the
wrapper
on
the
client
side
and
envoy
on
the
server
side
developers
don't
need
to
do
anything
and,
overall,
what
do
we
have?
We
have
authentication
that's
out
of
the
application
code,
so
it
will
work
with
any
language
or
framework.
We
have
em
TLS
and
we
have
an
identity.
Okay.
B
Thank
You
Ben
for
covering
the
nuances
and
complexities
of
authentication.
So
now
that
we
know
who
you
are
with
a
strong
degree
of
confidence,
let's
move
on
to
the
next
piece
of
the
puzzle,
which
is:
are
you
actually
allowed
to
do
whatever
it
is
you're
trying
to
do?
Let's
look
at
the
requirement.
The
primary
goal
obviously
is
to
prevent
unauthorized
access
to
our
services.
We
want
to
abide
by
the
principle
of
least
privilege
for
sensitive
services.
We
don't
want
to
forget
about
new
endpoints
and
have
them
be
accidentally
exposed.
B
The
security
features
that
we're
going
to
build
should
be
available
for
all
services
in
our
service
mesh.
We
can't
impact
developer
to
productivity.
This
is
super
super
important,
whatever
we
build
won't
be
successful
if
it
ends
up
being
a
burden
on
our
engineering
team,
and
we
have
to
make
sure
that
the
policies
are
simple
and
easy
to
understand
and
easy
to
update.
B
So,
let's
take
a
look
at
the
evolution.
Much
like
authentication,
we
didn't
come
upon
the
optimal
solution
from
the
get-go,
nor
do
we
think
we
have
an
optimal
solution
now.
It's
super
super
iterative.
So
let's
talk
about
it.
Where
should
I
start?
Let's
do
nothing
first
yeah,
so
we
could
do
nothing.
Let's
see
what
that
looks
like
service
a
wants
data
from
service
B,
that's
fine!
Definitely,
moji
person
wants
data
from
service
B.
Also
fine
devil.
Emoji
person
wants
to
bypass
the
infrastructure
entirely
and
go
straight
to
service
B.
Also
fine.
B
B
B
So
yeah,
that's
pretty
hard
fail.
Let's
try
something
else.
Well,
the
next
logical
progression
is
to
put
authorization
into
the
application,
but
we
can't
exactly
federate
this
out
to
all
the
other
engineering
teams.
Not
everyone
is
a
security
fixture,
not
oh,
and
even
if
they
were,
we
don't
really
want
to
each
team
to
reinvent
the
wheel
and
we
don't
worse.
We
don't
want
to
leave
room
for
user
error
in
these
re
implementations.
B
So
let's
come
up
with
our
own
framework
for
authorization,
we'll
design
our
own
policy
language.
This
policy
language
will
be
super
simple
and
it
will
cover
authorization,
semantics
for
HTTP
requests,
we'll
provide
a
Python
library
that
services
can
use
to
evaluate
authorization,
decisions
based
on
the
incoming
requests
and
the
policy
data
authorized
request
should
be
allowed
on.
Authorize
requests
will
be
rejected.
B
It
sounds
like
a
slam
dunk,
but
let's
talk
about
that
custom
policy.
Language
super
super
simple
for
HTTP
requests
right,
so
that's
kind
of
how
it
started,
and
then
someone
came
to
us
and
said:
oh,
you
guys
are
handling
authorization
for
things.
Can
this
handle
authorization
for
like
our
get
a
light
servers
to
like
yeah?
Okay,
that
makes
sense
fine,
we're
doing
this
sure.
Another
team
then
came
around
and
said,
asked
us
about
Cassandra
clusters.
Like
hey,
we
have
Cassandra
clusters,
we
want.
We
want
authorization
semantics
for
that
too,
like
okay,
I
mean
I.
B
Guess
we're
well
we're
at
it
sure
fine
yeah.
Let's
do
that
and
then
someone
was
like
yo
dawg.
I
heard
you
like
policies.
We
have
a
bunch
of
these
AWS
IAM
policies
that
we
keep
copy
pasting
all
over
the
place
like.
Could
we
maybe
use
your
new
authorization
framework
as
like
a
solution
to
that
problem
and
we're
like
wait?
Maybe
this
could
be
the
end-all,
be-all
policy,
language
and
like
baby,
we
basically
let's
go
creep,
get
the
best
of
us
and
we
kind
of
like
nerd
sniped
ourselves,
basically
and
like
in
the
end.
B
It's
so
ironic
because
the
the
policy
language
ended
up
being
too
complex
for
the
primary
use
case
that
we
set
out
to
solve
and
feedback
from
all
the
users
of
this
policy.
Language
was
that
it
was
super
confusing
and,
like
the
best
part
about
this
story,
is
that
we
never
actually
ended
up
flushing
out
any
of
those
use
cases
so
like
it
didn't
actually
work
very
well
for
for
anything.
B
B
However,
since
service
developers
are
on
the
hook
to
make
code
changes
and
incorporate
this
library
into
their
application
code,
we
leave
room
for
user
error
where
they
could
make
mistakes
such
as
misinterpreting
or
ignoring
the
policy
decisions,
and
I
mean
we
knew
this
at
the
time,
but
not
every
service
is
written
in
python,
and
so
therefore,
these
authorization
features
are
not
going
to
be
available
for
every
service
in
our
service
match.
We
kind
of
justified
this
at
the
time
saying:
that's
fine,
we're
mostly
Python
shop.
We
can.
B
B
So,
with
this
approach,
developers
are
required
to
make
code
changes
to
their
services
and
they
would
actually
have
to
be
on
the
hook
to
make
ongoing
updates
so
like.
If
there's
a
new
feature
in
this
authorization
library
they'd
have
to
update
their
dependencies.
If
there
was
a
security
patch
they'd
have
to
update
this
dependency.
Has
anyone
here
ever
tried
to
get
a
bunch
of
other
teams
to
update,
even
just
like
a
minor
patch
version
of
one
of
their
dependencies?
Yeah?
It's
it's
no
fun!
B
It's
just
like
a
cat
herding,
yak
shaving,
whichever
analogy
like
kind
of
exercise
and
then
of
course,
these
developers
had
to
ramp
up
on
the
complex
policy
language
semantics.
So,
if
I'm
feeling
generous
I'm
gonna
give
us
a
b-minus
on
this
whole
approach
and
I
think
we
can
do
better.
Where
do
you
think
we're
gonna
go
next?
B
Well?
So
what
if
we
put
an
envoy
on
the
other
side
on
voice,
supports
deferring
authorization
decisions
to
an
external
entity
via
the
X
table,
z,
plug
in
the
caller
and
identity
request?
Details
are
passed
along
to
this
external
entity
and
if
the
call
should
be
authorized,
the
requests
will
be
routed
to
the
target
service,
otherwise
unauthorized
callers
would
be
rejected
and
never
reached
the
target
service.
B
Additionally,
my
favorite
part.
We
can
now
update
the
network
configuration
so
that
requests
can
only
route
through
envoy
and
we
can
no
longer
go
directly
to
services
themselves.
So
now
we
have
a
very
important
decision
to
make.
Do
we
extend
our
homegrown
policy
language
solution
to
fit
into
this
environment,
or
do
we
leverage
some
other
open
source
technologies
that
might
already
do
a
lot
of
the
heavy
lifting
for
us.
B
Since
we
already
have
a
homegrown
policy
framework,
it
might
sound
really
tempting
to
just
use
it
here.
We
could
move
the
Python
library
logic
into
its
own
standalone
service
and
implement
on
boys
ext
off
ze
interface.
Besides,
if
it
ain't
broke,
don't
fix
it
and
we
invested
a
bunch
of
time
and
energy
and
cost
into
this.
This
custom
solution
of
ours,
sunk,
cost
fallacy
and
all
that,
but
it
was
broken.
We
knew
that
there
were
usability
and
scalability
issues
with
our
policy
language.
B
Additionally,
we
knew
that
there
were
other
open-source
general-purpose
policy
engines
and
it
seemed
worthwhile
to
shop
around
and
see
what
else
was
out
there,
and
in
doing
so
we
came
upon
the
open
policy
agent
or
OPA
spoiler
alert
in
case
you
didn't
see
the
title
of
this
talk.
We
ended
up
using
OPA
and
why
did
we
end
up
using
OPA
hope?
Is
a
general-purpose
high,
the
general
purpose
policy
engine?
It
has
a
high-level
declarative
language
called
reg.
B
Oh
I'm,
not
gonna,
go
into
any
details
about
rigor,
I
think
there's
been
many
many
talks
that
have
explained
a
lot
of
the
policy
semantics
for
OPA,
but
additionally,
there's
tons
of
documentation
and
details
and
examples
about
the
policy,
language,
semantics
and
different
types
of
applications,
and
what
I
thought
was
really
neat
about.
It
is
that
it
OPA
provides
the
ability
to
unit
test
your
policies,
so
this
is
really
great
for
functional
verification
of
your
authorization
policies
or
reproducing
bugs
that
happen
in
production.
B
Stuff
like
that
and,
most
importantly,
unlike
our
prior
solution,
OPA
is
open-source.
So
there
are
forums
for
discussing
policy
design
with
other
folks
who
are
solving
the
same
problems
that
was
really
powerful
for
us,
but
there
were
also
arguments
against
using
policy
the
open
policy
agent.
Not
only
did
we
write
a
bunch
of
code
for
that
for
that
policy
engine
that
we've
built.
We
also
have
a
bunch
of
policies
that
would
mean
to
be
migrated
over
to
the
new
OPA
to
work
with
OPA
and
now
well,
opus
policy.
Language
is
really
powerful.
B
There's
kind
of
a
steep
learning
curve
associated
with
the
Rago
language
semantics
and
recall
that
a
big
complaint
from
users
of
our
custom
solution
was
that
it
was
a
bit
too
complex,
so
long
story
shorter.
We
went
with
OPA,
but
we
had
to
make
sure
that
we
could
address
these
concerns
and
how
did
we
do
that?
We
did
that
by
building
a
service
that
we
call
the
OPA
policy
manager.
B
So
one
big
problem.
We
know
that
Reger
has
a
steep
learning
curve,
so
we
built
an
abstraction
layer
to
express
basic
HTTP
semantics
and
we
stuck
to
the
scope
and
created
a
basic
service
called
the
OPA
policy
manager.
The
OPA
policy
manager
takes
these
basic
high-level
HTTP
rules
from
the
extraction
layer
and
it
transpiled
them
down
to
a
format.
That's
optimized
for
fast
look.
Ups
in
OPA
with
this
approach.
Most
engineers
don't
actually
have
to
learn.
B
B
B
Nobody
wants
to
do
that,
so
we
realized
that
we
could
have
the
OPA
policy
manager,
also
trans
pile,
these
legacy
policies
down
to
the
same
data
structures,
because
our
new
policies,
and
that
way
we
could
kind
of
replace
the
wings
in-flight
behind
the
scenes
and
the
end
users
would
be
none
the
wiser
they
could
continue
to
make
updates.
In
the
in
the
confusing
custom
policy,
language,
semantics
Wow,
we
worked
on
replacing
the
back
end
and
expose
and
bringing
up
the
new,
the
new
HTTP
rule
in
abstraction.
B
B
B
So
how
did
we
do
this
time?
I
think
we
actually
met
all
of
our
requirements.
We
prevented
our
throws
access
to
services,
we're
doing
lease
privilege
we
deny
by
default.
This
is
available
for
all
services
in
our
surface
mesh,
regardless
of
what
language
those
services
are
written
in
developers,
productivity
is
not
impacted
and
the
policies
are
easy
to
use
and
easy
to
understand.
Developers
are
stoked
so
putting
this
all
together.
B
The
authentication
layer
gives
us
a
strong
guarantee
of
the
caller
identity
and
the
authorization
layer
powered
by
OPA
decides
whether
the
caller
is
allowed
to
hit
a
certain
service
endpoint.
Now
every
service
gets
this
essentially
for
free.
This
makes
Yelp
Services
secure
by
default,
which
is
awesome,
I
think
that's
a
buzzword
too
buzz
phrase.
So
where
are
we
now?
Opa
is
deployed
to
all
hosts
in
our
service?
Mesh
sensitive
services
are
protected
by
authorization,
rules
that
are
simple
to
express
and
understand,
and
we
have
ample
monitoring
and
alerting
solutions
on
this
data.
B
So
we
aggregate
decision
logs
from
each
OPA
instance
stream
that
data
into
Splunk,
which
enables
us
to
monitor
and
alert
so
that
we
can
detect
malicious
activity
or
miss
configurations
and
here's
a
screenshot
of
one
of
those
dashboards.
This
is
the
authorization
results
dashboard.
It
allows
us
to
pivot
and
filter
on
various
dimensions
so
that
we
can
see
the
distribution
of
authorization.
Results.
We'd
probably
want
to
keep
a
close
eye
on
this
when
we
roll
out
a
new
service
or
a
new
versions
of
our
policies,
or
do
like
major
refactoring
to
the
OPA
policy
manager.
B
So
future
ideas-
that's
where
we
are
today,
so
for
future
ideas
in
our
service
mesh.
We
think
it'd
be
really
really
cool
to
do
some
kind
of
phased
policy.
Rollouts
where,
like
the
when
policy
updates,
are
made,
they
first
rollout
to
a
small
percentage
of
canary
open
instances,
and
then
we
could
have.
You
know
the
service
mash
dude
to
smart,
1%
load-balancing
to
those
new
open
instances,
and
then
we
can,
you
know,
evaluate
what
was
the
impact
of
those
policy
changes.
B
This
would
be
especially
useful
for
major
Rico
or
policy
refactoring.
A
lot
of
the
pieces
are
kind
of
there.
Opa
supports
like
a
configurable,
prefix
and
path
for
where
to
load.
Its
bundle.
File
from
envoy
can
pretty
much
easily
support
it
to
I
think
with
some
minor
updates
or
to
control
playing
really
a
hard
part.
B
It
would
be
the
automatic
promotion
logic
like
how
are
you
sure
that
the
increase
in
denies
was
good
or
bad
kind
of
depends
on
and
who
you're
asking,
and
you
can
actually
kind
of
have
to
have
the
context
and
look
at
the
logs
to
know.
What's
going
on
to
know,
if
you
know,
5%
increase
in
denies
is
a
good
thing
or
a
bad
thing,
so
for
authentication
in
the
future,
we
would
like
to
improve
our
identity
attestation
and
do
something
more
sophisticated
than
that
IP
based
identity
map.
B
B
Those
were
the
future
ideas
for
within
the
context
of
our
service
mesh.
We
have
other
future
ideas
for
additional
use
cases
for
using
OPA.
So
we
use
all
these
different
technologies
which
I'm
sure
a
lot
of
you
do
to
kubernetes
docker
SSH.
You
know
terraform
Kafka,
all
these
all
these
different
technologies
that
we
use
on
a
day
to
day
basis.
B
B
We
could
not
have
done
any
of
this
without
incremental
changes,
incremental
changes
worth
the
key
to
making
progress
towards
the
end
goal.
We
couldn't
introduce
a
bunch
of
huge
changes
and
force
developers
to
react
to
the
major
infrastructure.
Ellipse,
however,
do
note
and
be
careful
that,
with
incremental
changes,
sometimes
they're
associated
with
migrations,
and
so
to
the
extent
that
you
can
try
to
automate
those
migrations
so
that
your
end
users
don't
have
to
incremental
fast
iterative
deployment.
B
But
if
you're,
if
your
engineers
have
to
migrate
things
of
every
month
or
so
they're,
just
gonna
not
listen
to
you
or
take
any
new
versions
of
your
stuff.
They're
gonna
stay
on
that
classic
policy.
Language
forever,
so
really
important
build
your
use
case
on
top
of
a
platform
rather
than
building
a
platform
for
your
use
case.
Remember
that
we
had
a
very
simple
use
case
and
we
prematurely
try
to
generalize
it
ultimately
building
something
that
didn't
really
fit
for
any
of
the
use
cases
to
that
effect.
B
D
B
C
We
have
a
that's
kind
of
the
hard
part.
Is
the
PKI
back-end
that
to
distribute
the
certificates,
but
our
control
plane
can
distribute
these
certificates
we're
not
currently
using
SD
s
the
secret
discovery
service
in
in
Envoy,
but
that
is
one
kind
of
pretty
straightforward
way
to
distribute
those
certificates.
E
B
A
F
A
C
Didn't
load
test
it,
but
it
the
OPA,
runs
right
alongside
on
voice.
So
it's
not
it's
not
making
an
external
network
request.
It's
it's,
making
a
request
locally
to
the
instance
of
OPA.
That's
running
yeah
didn't
check
the
the
QPS
I
didn't
really
detect.
Any
latency
degradation
is
at
least
on
our
kind
of
an
order
of
magnitude
related
see.
Yeah.
B
G
A
question
the
OPA
policy
manager,
the
policy
manager,
it's
basically
migrating
the
policies
you
know
sort
of
like
those
people
who
doesn't
know
reco.
You
know
it's
doing
this,
so
how
do
you
validate
like
once
it's
doing
this
migration?
If
it's
successful
or
it's
done
successfully,
does
it
any
validation
process
to
it?
Secondly,
what
is
an
OPA
instance
if
you
can
define
that
and
also,
lastly,
is
this
open
policy
manager
that
you
guys
are
not
it's
going
to
be
open
sourced
or
something.
B
I'll
go
backwards,
it's
not
gonna
be
open
source.
Most
likely
I
mean
it's
it's
it's
like
a
handful
of
lines
of
Python
that
just
do
like
some
basic
translation
from
one
yeah,
Moe
syntax.
That
made
sense
for
us
into
something
that
makes
sense
for
our
regular
policy.
So
we
don't
have
plans
to
open
source.
It
I'm
happy
to
share
like
what
the
code
looks
like
it's.
It's
really
straightforward.
B
What
is
an
open
instance,
so
you
can
install
so
open
written
and
go
so
opens
just
a
single
binary
and
OPA
has
a
server
mode,
so
you
can
just
run
the
OPA
binary
Jeff
server
and
it
runs
as
a
as
a
server
and
it
has
a
config
file
that
can
pull.
You
know
it's
config,
it's
bundles
and
everything.
So
that
is
an
open
instance.
So
there's
a
there's
a
separate
instance
of
OPA
running
on
each
host
in
our
service
mesh.
C
B
C
We
currently
don't
I,
believe
I
think
we,
we
usually
just
rolled
them
out
to
dev
the
dev
environment
before
we
roll
them
out
to
prod,
but
we
did
have
a
design
for
for
running
unit
tests
against
it,
mainly
for
regression
tests.
We
actually
do
have
regression
tests
in
the
control
plane
testing,
but
that
does
not
include
real
policies,
but
that
was
definitely
something
that
was
very
attractive
about.
Apa
is
the
unit
testing
framework
yeah.
B
And
that
that
dashboard,
that
shows
the
distribution
of
denies
or
the
distribution
of
authorization
results.
That
was
definitely
instrumental
in
like
invalidating
some
of
some
of
the
semantics
of
the
legacy
policy
stuff
to
the
new
policy
stuff
and
making
sure
that
we
got
similar
results
across
like
different
environments,
one
of
which
was
using
the
old
stuff
and
one
that
was
using
the
new
stuff.