►
Description
Don't miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe in Amsterdam, The Netherlands from 18 - 21 April, 2023. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Solving Multi-Service Without a Service Mesh - Evan Anderson, VMware
Kubernetes is famously a “platform for building platforms”. In this talk, we will un-pack the primitives Kubernetes provides for enabling microservices to securely communicate with each other without relying on a service mesh. Together, we’ll explore how technologies like NetworkPolicy, token projection, API gateways, cert-manager, and language runtimes play poorly or nicely together. We’ll cover authentication options, encryption, rate limiting, multi-tenant infrastructur eservices, and the interplay between L4 and L7 features with an eye on compliance as well developer ease of use. Drawing on his experience as Knative Security Working Group lead and background solving application runtime challenges on Kubernetes, Evan will teach participants about how to build without a service mesh, as well as a deeper understanding of the value that service meshes provide.
A
Okay,
so
multi-service
without
a
mesh,
so
I'm
sure
that
some
of
you
are
here
because
you
really
don't
like
service
meshes.
This
talk
is
not
actually
going
to
tear
down
service
meshes,
but
hopefully
we'll
get
a
little
deeper
understanding
why
people
are
using
service
meshes
and
how
you
can
solve
it
without
a
service
mesh
and
then
you
can
decide,
do
I
need
a
service
mesh.
Is
that
the
right
tool
for
this
job
or
you
know,
can
I
take
two
or
three
components
and
get
the
pieces
that
I
want?
A
A
So
you
know,
hey
look
I've
I've
built,
you
know
multiple
microservices
wired
together.
This
is
awesome.
You
know
it's
a
nice,
simple
architecture,
diagram
I've
got
a
front-end
service.
I've
got
a
couple
back-end
services
to
each
have
their
databases
they're
nicely
factored.
They
talk
to
each
other
through
apis
and
I'm,
not
ready
for
service
mesh.
Yet
can
I
just
slap
some
kubernetes
on
it,
make
it
work.
A
A
We've
got
a
DB
operator.
Everything
else
is
just
out
of
the
box
kubernetes.
It's
awesome.
We're
done.
Okay,
great,
go
home,
Oh,
wait
we're
at
a
security
talk.
So
this
is
the
part
where
I
start
to
make
you
sad
and
we
will
keep
getting
sadder
over
the
course
of
the
talk
and
at
the
end
we
will
go
to
another
talk
and
repeat
the
process.
I,
don't
know
why
you
are
here
so
start
with
it's
not
too
bad.
Our
users
are
connecting
to
us
over
the
Internet.
A
A
We
can
just
go
and
connect
things
assuming
that
I'm
using
Ingress
or
Gateway
API,
if
I'm,
using
Kong's,
apis
or
traffic's
apis
or
Contours
apis
or
istio's
apis,
then
I
need
to
do
some
manual
wiring
together,
it's
not
great,
but
do
we
really
expect
cert
manager
to
expect
to
support
15
different
ingresses,
no
Gateway
API?
Maybe
we'll
give
you
those
capabilities
later
in
the
year.
I
don't
know
so
it's
okay,
not
too
bad
great.
So
we've
encrypted
the
traffic
over
the
internet.
Now
we're
secure
we're
safe.
A
A
A
A
A
We
may
also
want
to
actually
be
able
to
restrict
access,
so
in
this
case,
oops
that
went
too
far.
We
want
to
be
able
the
front
end
should
be
able
to
do
orders
and
addresses
to
the
back
ends,
but
maybe
orders
needs
to
be
able
to
read
addresses,
but
the
order
server
should
never
be
updating
someone's
address.
A
A
They
can
also
go
to
the
API
server
and
do
anything
you
could
do
do
not
send
those
tokens
to
other
people.
They
let
them
they.
Let
lets
the
other
person
impersonate
you
fortunately
I'm
going
to
say
this
is
a
year
year
and
a
half
or
so
ago.
The
token
account
the
service
account
projection.
Api
showed
up
which
lets
you
say:
hey
I
want
a
oauth
token
for
my
service
account,
identity
and
I
wanted
to
have
a
different
audience.
I,
don't
want
it
to
be
believed
by
the
API
server.
A
A
That's
been
a
standard,
Enterprise
solution
for
I,
don't
know
20
years
or
so
you
know
you
get
a
c,
you
get
a
self-signed
cert
and
you
use
it
to
sign
your
search
within
your
DNS
names
that
aren't
public
to
the
internet
and
remember
all
these
names
are
service.cluster.local.
If
you
want
to
go,
let's
encrypt
and
get
a
cert
for
service.cluster.local
and
they
give
it
to
you,
that's
going
to
be
real,
exciting
you
get
to
publish
your
blog
post
about
a
vulnerability
you
found
in
let's
encrypt
and
it
shows
up
in
the
certificate,
transparency
log.
A
So
everyone
can
see
it
happened.
It's
not
going
to
happen,
so
you
need
a
self-sized
cert,
which
is
great.
Your
other
option
is
that
you
could
use
spiffy
and
you
could
roll
out
spiffy
and
teach
all
your
systems
to
use,
spiffy
and
interact
and
and
manage
that
and
so
forth.
It
looks
like
you
can
pick
your
language
any
language
you
want
as
long
as
it's
go,
Java
or
rust.
A
So
if
you
were
using
typescript
now
you
get
to
learn
rust
and
how
to
interact
with
it
or
maybe
go
or
Ruby
or
Java,
but
I'm
guessing
rust
is
going
to
be.
The
best
of
the
three
in
any
case
might
not
be
what
you
were
thinking
of
when
you
were
like.
Oh
I'll
write
a
typescript
front
end
so
that
I
can
write
typescript
and
then
I
can
write
typescript
in
the
browser.
A
And
in
any
case,
we've
now
said:
oh
identity
problems
are
hard,
I
know
will
simplify
to
a
key
distribution
problem.
We'll
say:
oh
now,
we
just
need
to
get
this
CA
cert
to
all
the
different
places
and
mounting
the
certificate
and
okay
I'll
serve.
The
certificate
is
not
too
bad.
Getting
a
CA
insert
into
an
arbitrary
Docker
container
is
not
great.
A
It's
not
too
bad,
you
say
like
SSL
serger,
all
my
certs
are
here
and
it
will
just
pick
things
up
and
load
them
open,
SSS
too
bad.
You
said
SSL
sector,
then
you
need
to
take
all
the
search
and
you
need
to
Hash
their
names
to
the
special
hashing
format
that
openssl
wants,
and
so
you
get
a
directory
full
of
names
that
are
seven
characters:
dot
zero
in
a
Sim
link
to
your
search
yeah.
You
could
do
this
someplace
else.
You
could
do
it
in
a
knit
container.
A
This
needs
to
happen
before
your
binary
starts,
so
you
can't
just
take
a
container
using
openssl
and
Magic
assert
into
it
without
a
bunch
of
extra
work
and
in
Java
there's
a
different
tool
that
you
need
to
load
things
in
that's
Java
specific.
There
may
be
other
TLS
libraries
that
have
different
ways
of
doing
this.
I
couldn't
find
any
well
widely
used
ones,
but
here
is
my
private
CA.
Please
trust
it
has
been
a
problem
for
20
years
and
we
apparently
still
don't
have
a
standard
interface.
A
If
you
pick
a
service
mesh,
the
service
mesh
controls
all
this,
and
so
you
have
one
implementation
and
it's
whatever
your
service
mesh
is
implemented
in.
But
now
you
can
see
why
you
might
want
to
do
this.
Cloud
native,
build
packs
and
service
bindings
have
some
stuff
that
you
can
stick
in,
but
again
you're,
starting
to
build
a
big
layer
to
replace
that
service
mesh.
A
Maybe
service
mesh
is
a
good
choice,
but
it
can
be
hard
to
roll
out
a
service
mesh
and
have
part
on
and
part
off
and
figure
out
how
to
ramp
all
this
stuff
up.
It's
getting
easier,
especially
if
everything
is
in
kubernetes,
but
it's
still
hard.
Sidecar
injection
has
a
lot
of
problems.
I
know
istio
ambient
Ambient
mode
is
the
new
fix
for
that
it
looks
a
lot
like
a
cni.
A
Fortunately,
feedback
may
be
slightly
better,
but
it's
still
potentially
a
lot
and
the
resource
requirements
can
be
high.
I've
heard
people
complaining
about
how
much
CPU
they're
spending
on
Envoy
proxies
instead
of
on
like
their
application
stuff,
it's
got
a
lot
of
functionality,
which
is
great.
If
you
need
that
functionality
and
if
you
don't
there's
a
whole
bunch
of
complicated
things
that
you
need
to
figure
out,
do
I
need
this
or
not.
Is
it?
Is
it
important?
A
And
you
know
some
organizations
are
going
to
take
one
step
at
a
time,
they're
going
to
they're
going
to
figure
out
how
to
get
on
kubernetes
and
then
they're
going
to
figure
out.
You
know,
do
I
need
a
service
mesh
and
if
so,
what
does
that
look
like
some
organizations
are
happy
to
jump
in
with
both
feet
to
all
new
technologies.
A
Another
reason
might
be
multi-tenancy.
So
let's
say
I
want
a
shared
Reddit
service
and
I
want
to
have
you
know?
Okay,
this
namespace
asks
for
100
Megs
of
red
as
cash.
This
namespace
asked
for
500
and
then
I'm
going
to
have
one
pool
of
workers
behind
it
that
service
these
things
and
so
I'm
going
to
have
a
router
that
comes
in,
and
you
want
that
router
to
present
a
service
in
each
one
of
those
namespaces
and
service
mesh
really
wants
to
know
what
your
underlying
topology
of
pods
are
and
you're
like.
No,
no.
A
A
For
both
of
these
cases,
you
can
tunnel
things
through
with
TLS
Sni,
but
if
you're
running
one
of
these
multi-tenant
Services
it
breaks
stuff
like
Network
policy,
because
Network
policy
says
it's
all
going
to
one
port.
It's
all
one
L4
path,
token
projection,
if
you're
a
client,
if
you're
a
multi-tenant
client,
if
you
go
out
and
call
different
things,
token
projection
wants
to
protect
one
service
account.
That's
the
service
account
your
pod
runs
as
and
there's
no
way
to
say,
I'm
authorized
to
use
this
other
account.
A
A
Let's
talk
a
little
bit
about
how
we
can
undo
some
of
the
damage
that
we
did
by
wanting
to
go
back
and
share
resources
across
the
cluster
for
efficiency's
sake.
You
know
full
disclosure
here
I
work
on
the
k-native
project,
K
native
effectively.
Does
this
for
your
pods,
because
sometimes
we
scale
you
down
to
zero.
Sometimes
we
scale
you
up
real
big.
A
We
have
a
central
shared
piece,
because
if
there's
something,
if
you
had
to
run
something
in
each,
you
know
for
each
deployment
where
you
said
hey,
you
don't
have
to
run
your
pod,
but
you
have
to
run
this
other
thing,
and
sometimes
you
run
your
pod
too.
Everyone
would
be
like
that.
Doesn't
save
resources,
but
if
we
have
one
Central
component
and
you
have
10
namespaces,
which
you
know
scale
up
and
down
to
zero,
that
Central
component
can
give
you
big
savings.
A
So
Network
policy
I
was
just
saying
these
are
all
the
same
TCP
port.
Well,
unless
you
allocate
a
whole
bunch
of
different
ports,
one
for
each
service
on
your.
You
know
on
your
router
instance,
and
then
you
can
see
here.
You
know
this
looks
like
you
know.
This
name
red
is
Foo,
is
on
this
port
and
then,
if
I
create,
you
know
a
bar
instance,
I'll
name
it
four
bar
and
it'll,
be
cluster
IP
and
the
Target
Port
will
be
maybe
one
higher
or
something
like
that.
A
A
You
know
I
can
use
TLS
Sni
in
combination
that
this
or
not,
but
it's
nice
to
have
defense
in
depth
options
and
when
you
route
everything
to
a
single
endpoint
inside
the
cluster,
you
can't
use
the
destination
IP
address,
because
Sni
has
stopped
that,
and
you
can't
use
the
sender
because
you
don't
know
who
that
sender
was
it
could
be.
You
know
it
might
be
a
little
Fair
which
note
it
is,
but
a
pod
can
appear
and
disappear
at
any
time.
So
you
don't
really
want
to
say.
A
Oh
I
looked,
and
at
this
moment
it
was
this-
maybe
you're
looking
at
Old
sale,
cash
or
something
like
that
so
destination
Port
can
let
you
recover
this.
So
token,
projection
doesn't
work,
but
if
it's
important
for
you
to
be
calling
out
I,
don't
have
a
great
example
of
this.
Yet
but
I've
been
thinking
about
it
because
I
I
fear
that
I
will
in
the
future.
A
You
could
actually
be
your
own,
open,
ID
provider
and
say:
hey
I'm,
going
to
you
know,
be
this
client
or
that
client
or
something
like
that
there
might
be
a
kubernetes
API
to
do.
I
can
act
as
something
else.
I
know
that
AWS
has
that,
but
I
haven't
seen
it
I
haven't
seen
any
caps
suggesting
that
impersonate.
Another
service
account
is
on
the
kubernetes
to-do
list.
So
this
was
the
best.
A
If
you
wind
your
head
way
back
to
the
beginning,
we
talked
about
why
you
might
want
an
API
Gateway
and
you
might
want
to
do
rate
limiting
on
a
per
API
key
basis,
and
that
might
be
something
where
you'd
rather
keep
that
on
a
central
API
Gateway,
even
if
you're
using
a
service
mesh.
So
you
might
have
traffic
comes
in
service
mesh
to
API
Gateway,
to
service
mesh
to
your
application,
backend,
and
that
might
be
a
right
choice.
A
And
now,
if
any
of
you
were
thinking
about,
oh
wait,
isn't
that
a
lot
of
envoy
CPU
yeah?
We
talked
about
that
earlier
so
and
I'm
going
to
call
back
as
well
to
Zach's
Five
Points
in
the
keynote
yesterday,
pretty
much
every
service
that
has
user
data
and
some
sort
of
authorization
model
is
in
a
way
multi-tenant.
A
You
know,
if
you
think
about
it
like
something
like
Facebook
or
you
know
your
shopping
cart
on.
You
know,
Amazon
like
that,
is
a
service
and
my
shopping.
Cart.
Isn't
your
shopping?
Cart?
Isn't
your
shopping
cart?
So
that's
really
a
multi-tenant
service.
It's
not
an
API
controlling!
You
know
compute
things,
but
you
still
need
a
lot
of
that
same
authorization
logic,
and
so
you
actually
will
need
to
get
your
application
involved
in
the
security.
Even
if
you
have
a
service
mesh,
you
may
be
able
to
make
some
course
Grade
Rules.
A
That
X
can't
talk
to.
Why
add
a
little
bit
more
of
an
intermediate
level
than
Network
policy
is
so
rather
than
saying
you
know
in
this
diagram.
Orders
can
talk
to
you
know,
orders
can't
talk
to
addresses
or
orders
can
talk
to
addresses.
You
can
say:
orders
can
only
talk
to
addresses
with
its
own
identity.
A
A
And
yeah,
you
could
do
a
lot
of
that
logic
in
mesh
whoops.
You
could
do
a
lot
of
that
logic
in
the
mesh,
but
imagine
writing
istio
policies
like
this
for
every
single
user
your
mesh
would
die.
You
would
hate
your
life
and
it
would
be
much
easier
to
build
this
into
your
application.
So
maybe
your
mesh
can
can
answer
some
of
those
lower
level
questions
about
who's
talking
to
me,
but
you
need
some
higher
level
questions
about.
What's
the
application
supposed
to
do
that,
you
actually
don't
want
to
pull
out
into
your
service?
A
A
A
A
You
know
hey
these,
you
know
is
this
allowed
to
come
in
and
you
know
who
you
know
what
server
on
the
network
sent
it
to
me
and
there's
a
second
level
of
what
user
made
the
request,
and
so
you
actually
need
to
build
both
of
these
layers,
and
then
you
can
start
to
make
more
complicated
decisions
as
well
like
hey,
you
know,
is
a
user
coming
in
from
the
orders
system
able
to
go,
and
you
know,
update
their.
You
know,
call
an
API
to
update
their
address
and
you're
like
well.
A
No,
it
doesn't
make
sense
for
that
to
come
from
the
order
system.
That
should
only
come
from
the
front
end
and
maybe
in
the
future,
it's
like
it
could
come
from
the
front
end
or
it
could
come
through
customer
support,
and
so
maybe
the
customer
support
portal
becomes
another.
You
know
top
level
Network
policy
source
that
you
can
consider
and
you're
like.
A
A
So
we've
got
these
awesome
service
meshes,
but
we
can't
get
a
CA
cert
into
a
random
container,
because
that
would
require
talking
to
Java
people
and
to
go
people
and
to
C,
plus
plus
people
into
python
people
and
getting
all
these
libraries
to
agree
on
a
place.
Even
if
it's
not
the
place
they
used
to
do
it.
A
Authentication
and
identity
is
hard.
Should
you
use
spiffy?
Should
you
use
cluster
API
token
projection?
Should
you
use
your
Cloud
provider's,
managed
identity,
service
I
can't
answer
those
things
for
you,
but
there's
trade-offs.
None
of
them
is
great
or
perfect,
but
we've
been
working
on
it
for
40
years,
so
I'm
sure
the
answer
is
just
around
the
corner.
A
Authorization
is
hard
figuring
out
the
right
place
to
make
an
authorization
decision
is
hard
and
we
can
build
tools
to
help
you
authorize
things,
but
you
need
to
figure
out
which
are
the
right
tools
and
the
right
places
for
it,
and
sometimes
the
answer
is
going
to
be.
You
write
your
own
code.
You
write
your
own
policy
engine
and
yeah.
A
A
That
makes
me
as
someone
who
believes
in
defense
Adept
a
little
bit
nervous
if
it's
and
all
your
trust
is
in
Acts.
There
was
a
talk
yesterday
that
was
talking
about
okay,
so
you
get.
You
know
you
get
encrypted
communication
between
the
mesh,
but
then
the
mesh
to
your
application
is
clear
text
and
that's
over
localhost,
so
you
can
only
Snoop
it
if
you
get
like
rude
on
localhost,
which
you
know
at
which
point
I
can
also
just
rip
it
out
of
your
application's
memory.
A
Like
you
know,
devmem
is
a
thing,
but
it
also
means
that
my
application
doesn't
understand
whether
the
other
end
is
encrypted
or
not,
and
so
I
go
make
a
connection
and
I
think.
Oh,
this
is
great
and
then
I
end
up
on
a
cluster
where
there
isn't
a
service
mesh
and
there's
no
feedback.
That
says:
hey
wait.
You
tried
to
make
this
connection
and
you
wanted
it
to
be
encrypted
and
it's
not
I
have
no
way
of
hinting
that
to
my
environment.
A
A
A
I
was
talking
to
someone
over
at
keyless
I
think
earlier
and
they're
like
we
have
a
great
library
for
that
stuff
and
I'm
like
great.
We
have
another
library
for
SSL
like
it's
good,
to
have
some
diversity
there.
You
don't
want
back
when
heart
bleed
happened,
I,
remember
it
affected,
basically
everyone,
because
there
was
one
SSL
library
and
everyone
used
openssl,
and
so
if
there
was
a
bug
it
hit
everybody
and
so
the
fact
there's
boring
SSL
now
and
a
couple
other
implementations
I
think
is
good.
A
A
It's
not
like
people
are
going
to
wake
up
tomorrow
and
be
like
hey
I
think
we
should
make
kubernetes
release
more
often
and
with
features
that
are
less
thought
through
and
stable
that
that
pressure
is
never
is
never
going
to
win
at
this
point.
So
the
best
we
can
do
is
start
today
and
look
at.
How
can
we
do
better.
A
A
I
also
am
a
big
believer
that
everyone
here
is
trying
their
hardest
and
we're
all
doing
the
thing
that
looks
best
from
our
own
positions.
So
this
is
really
not
a
oh
X
is
bad
or
Y
is
bad
other
than
to
point
out
that
like,
why
have
we
not
figured
out
how
to
do
custom
Cas
in
20
years?
But
you
know
tomorrow
could
be
better.