►
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2023 in Amsterdam, The Netherlands from April 17-21. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
How Many Proxies Do You Need? - Liz Rice & Thomas Graf, Isovalent
To provide app-level network functionality like L7 load balancing and TLS termination, Service Meshes use a proxy component that terminates L7 connections on behalf of applications. Traditionally the proxy has been co-located in each application pod as a sidecar container, but Cilium Service Mesh changed this with the innovation of sidecarless service mesh. Istio have also now adopted a sidecarless approach that was recently announced as Istio Ambient Mesh.
But “sidecarless” doesn’t mean “proxyless”! It’s a question of where you deploy the proxies, and how you create the relationship between apps and proxies. In this talk we’ll explore the pro’s and con’s of different models, and explain where eBPF makes a difference (and where it doesn’t) in not only network performance, but also to provide observability and security capabilities.
B
I'm
Liz
rice,
I'm,
Chief,
open
source
officer
at
ISO
valence,
you
may,
if
you've
been
to
kubecons
in
the
past
a
long
long
time
ago,
I
was
a
chair
of
keepcon
and
I
used
to
be
chair
of
the
technical
oversight
committee,
which
gave
me
this
amazing
privilege
of
seeing
a
lot
across
the
whole
Cloud
native
landscape
and
when
I
chose
to
come
and
work
with
ISO
villain.
The
capabilities
of
psyllium
and
ebpf
had
really
kind
of
stuck
out
to
me.
So
I'm
very
excited
to
be
working
in
this
technology.
B
So
we're
going
to
talk
a
little
bit
about
how
many
proxies
you
need
to
run
a
service
mesh
I.
Think
in
this
room.
We
don't
really
need
to
tell
you
what
the
service
mesh
is
there
for,
and
we
probably
don't
need
to
tell
you
the
history
of
how
service
mesh
allows
us
to
abstract
functionality
that
is
common
across
multiple
applications.
We
no
longer
need
to
write
that
directly
into
an
application
or
in
a
library.
B
We
want
to
abstract
that
out
into
sort
of
independent
service,
mesh
functionality
that
sits
behind
the
application
or
in
front
depending
on
your
perspective,
and
for
a
long
time
the
model
was
to
use
sidecars.
So
you
would
have
a
proxy
running
in
a
sidecar
container
outside
of
the
application,
providing
all
these
networking
and
resiliency
and
security
and
observability
capabilities
that
we
want
to
get
from
a
service
mesh.
B
And
if
we
have
the
proxy
running
inside
the
network,
namespace
for
the
Pod
we've
got
to
kind
of
connect
through
that
Network
stack
twice
before
we
even
get
as
far
as
the
host
Network.
So
there's
a
lot
of
path.
A
lot
of
processing
for
packets
to
go
through
before
they've
even
left.
The
pod
people
have
also
found
configuring
sidecars
pretty
complex
and
I
I
love.
This
quote
I
mean
it's
it's
a
couple
of
years
old
now,
three
years
old
now
from
Kelsey.
But
you
know,
complexity
is
a
real
issue
with
service
mesh.
B
So
when
I
joined
i7
I,
remember
Thomas
telling
me
that
selling
was
already
kind
of
80
percent
service
mesh.
We
were
already
providing
a
lot
of
the
functionality
that
we
expect
from
a
service
mesh
in
psyllium
that
is
traditionally
known
as
a
cni.
So
could
we
use
this
as
a
basis
to
remove
sidecars
and
provide
a
sidecar
free
service,
mesh
and
long
story
short?
We
did
round
this
time
last
year.
Just
about
a
year
ago
we
announced
the
psyllium
service
mesh
beta.
Does
anybody
hear?
Did
anybody
here?
Try
the
service
smash
beta?
B
We
have
a
couple
of
hands
up
great
excellent.
This
was
a
very
kind
of
common
comment
that
we
got
from
people
who
signed
up
for
for
that
beta
that
they
really
liked
Envoy.
But
they'd
found
the
overhead,
the
resource
usage,
the
complexity
of
configuring,
sidecars,
the
complexity
of
getting
making
sure
that
all
their
apps
were
instrumented,
the
additional
Network
overhead,
the
latency
involved.
B
There's
a
really
great
podcast
about
that
ambient
mesh.
The
kubernetes
podcast
totally
recommend
if
you
are
interested
in
ambient
mesh,
taking
a
listen
to
that,
because
it's
really
really
interesting
to
hear
how
those
Engineers,
Ethan
and
Justin
thought
about
that
model
and
one
one
of
the
the
quotes
from
that
was.
You
know
that
they'd
thought
for
a
long
time
that
sidecars
were
inefficient
and
that
users
have
a
tendency
to
allocate
resources
for
the
worst
case.
So
you
can
end
up
in
this
worst
case
scenario
of
resource
allocation.
B
We
can
share
resources
when
we're
using
ebpf
much
more
easily,
partly
because
we're
not
constrained
by
this
idea
of
having
isolated
pods
when
you
have
a
sidecar
in
a
pod
by
design
that
sidecar
is
isolated
from
all
the
other
side
cars.
It's
in
a
pod,
deliberately
isolated
from
everything
else.
When
we
push
as
much
as
we
can
of
the
service
mesh
functionality
into
the
kernel,
it
is
shared,
and
that
can
be
a
really
good
thing.
The
kernel
is
there
to
be
shared
amongst
all
the
applications
that
run
on
it.
B
A
Colonel
has
always
been
delegating
functionality
from
kernel
space,
user
space
and
we'll
see
a
couple
of
examples
in
the
case
of
Salem
services
in
general,
there
are
aspects
that
are
not
great
matches
to
be
solved
in
the
kernel,
such
as
TLS
termination,
HTTP
authorization
or
complex
HTTP,
parsing
and,
of
course,
identity
management,
TLS
control,
handshakes
and
so
on.
We'll
get
into
the
into
the
details
here.
A
This
means
that
sodium
service
mesh
has
this
advantage
that
we
can
be
ebpf
native
when
possible
and
benefit
from
the
performance
gains
of
evpf
and
then
fall
back
to
an
Envoy
proxy
when
ebpf
is
not
the
ideal
instrument
to
implement
other
examples.
I
mentioned
that
this
is
actually
have
been
been
the
case
for
for
kernel
development
for
ages.
The
kernel
is
always
delegated
some
functionality
to
user
space.
A
Very
good
example
is
IDs,
sorry,
color
running
in
user
space
and
doing
deep
packet
inspection
using
iptables
nfq
to
to
punch
packets
user
space
or
even
just
kernel
module
loading
when
your
laptop
loads,
the
Wi-Fi
driver
it.
Actually,
the
kernel
module
is
getting
loaded
through
a
so-called
user
space
mode
helper.
A
A
B
B
So
if
we
think
back
to
the
sort
of
original
sidecar
model,
we
had
potentially
a
what
could
be
seen
as
a
security
advantage
that
the
proxy
shared
all
the
same
isolation
that
the
application
pod
did.
So
it
naturally
has
access
to
the
applications.
Secrets
it's
isolated
in
the
same
way
as
the
application
is
so
from
a
security
model
perspective.
B
You
don't
need
to
think
about
it
anymore,
because
it's
the
same
as
the
application
and
psyllium
has
supported
this
model
for
sidecars
for
service
mesh
for
a
long
time
now,
it's
it
has
been
possible
and
people
are
running
istio
on
top
of
psyllium
as
the
networking
layer
that
you
know,
that's
a
completely
supported
model,
but
if
we
want
to
move
away
from
that,
we
want
to
not
have
a
proxy
for
every
one
of
our
application
pods.
If
we
put
the
proxy
on
the
Node.
What
does
that
mean?
B
You
don't
need
to
have
additional
proxies
Elsewhere
on
the
network,
and
you
can
you
can
have
this
very.
You
have
a
proxy
right
next
to
you,
which
I
think
is
a
good
thing.
This
is
the
kind
of
default
model
for
psyllium
service
mesh
today,
where
we
would
have
one
proxy
per
node,
but
we
are
working
on
the
idea
of
having
multi-tenancy
by
having
multiple
proxies
running
on
the
Node,
so
that
can
give
that
isolation
that
people
are
potentially
looking
for.
B
And
this
is,
you
know,
I,
would
say
a
relatively
new
idea.
It's
something
that
we're
interested
in
and
we'd
like
to
understand
whether
psyllium
users
would
like
to
see
this
model,
because
it
would
certainly
be
possible
to
implement
that
Z
tunnel
approach
in
the
psyllium
layer,
but
I
think,
most
importantly,
there
isn't
a
single
solution.
B
For
you
know
you
must
have
proxies
this
number,
this
ratio
of
proxies
to
applications,
or
this
particular
location
of
proxy
relative
to
your
applications
for
some
required
for
some
deployments
for
performance
reasons,
you
will
choose
to
have
a
proxy
on
every
node
that
will
almost
certainly
give
the
highest
performance
and
probably
reduce
complexity.
In
other
environments,
you
may
choose
to
emphasize
more
multi-tenant
isolation.
A
What
about
encryption?
So
we
talked
about
number
of
proxies
and
we
figured
it's
actually
interesting
to
look
at
the
most
common
use.
Cases
of
why
you
want
to
use
or
why
you
use
a
service
mesh
and
encryption
and
observability
are
probably
among
the
top
asks
that
you
have.
So,
let's
look
at
encryption.
First
of
all,
if
we
look
at
encryption,
there
is
two
broad
models
to
achieve
encryption.
We
can
achieve
it
at
the
network
level
using
functionality
such
as
ipsec
or
wireguards,
where
we
essentially
encrypt
into
higher
Network
nodes
to
nodes.
A
This
can
be
fine
for
many
use
cases
and
has
been
the
status
quo
and
not
working
for
a
long
while,
but
it
does
rely
or
it
just.
It
does
require
that
you
trust
your
node.
So
if
the
node
gets
compromised,
the
key
on
that
now
gets
compromised
and
then
the
network
encryption
gets
compromised
until
the
key
gets
rotated,
so
it
uses
node
identity.
That
can
be.
That
can
be
great
if
that
is
your
security
assumption.
What
if
we
want
deeper
security,
better
security
and
actually
take
it
one
level
further.
A
Obviously,
mtls
comes
into
play.
The
downside
of
mtls
as
implemented
with
sidecars,
is
that
first
of
all,
it
is
shared
with
the
layer
7
proxy,
which
means
that
if
the
layer,
7
proxy,
gets
compromised,
all
certificates
and
secrets
in
that
proxy
get
compromised
as
well,
and
the
second
big
downside
of
using
side
calls
for
mtls.
It
is
that
it's
pretty
much
limited
to
TCP
and
in
particular,
Enterprise
use
cases
will
require
multicast,
UDP
scdp
and
a
variety
of
other
protocols
that
need
to
be
supported.
A
So
we
looked
at
this
and
and
and
asked
ourselves,
can
we
support
an
mtls
model,
a
mutual
authentication
model,
while
benefiting
from
the
network
level
to
support
any
network
protocol,
and
this
is
what
we
came
up
with
for
all
the
details
on
how
exactly
this
works.
I've
linked
the
blog
post
that
goes
into
the
details,
we'll
also
see
the
pull
request
with
the
initial
work
later
on.
A
Essentially,
this
splits
the
authentications
piece
from
the
payload
from
the
data
path,
and
this
means
that
the
authentication
happens
through
a
regular
mtls
handshake
where
we
can
integrate
with
spiffy
serve
manager
and
so
on,
and
to
a
regular
Mutual
authentication
using
certificates
that
are
per
service
and
then,
instead
of
TR,
instead
of
carrying
the
actual
payload
and
network
data.
The
data
in
that
mtls
connection.
We
carry
that
through
the
network
directly,
while
using
the
keys
to
secrets
that
have
been
negotiated.
A
Why
the
mtls
handshake
using
ipsecond
wireguard
and
we
kind
of
gain
the
benefits
of
both
models
where
we
get
a
strong,
Mutual
authentication
handshake
with
mtls.
While
we
can
support
any
network
protocol
without
introducing
any
sort
of
proxies,
so
we
can
establish
encryption
and
mutual
authentication
network-wide
without
requiring
proxies
while
supporting
any
protocol.
A
Let's
look
at
how
this
actually
looks
like
because,
let's
say
you're
not
using
this
and
you're
going
you're
you're
you're
going
down
this
road
and
want
to
use
this
we're
integrating
this
into
cylinder.
Our
policy,
which
is
an
Integra
which
is
an
extension
of
kubernetes
network
policy
and
all
Southern
users,
have
to
do
essentially
opt
into
this
and
say
for
these
connections.
Let's
say
front-end
talking
to
backend,
you
need
to
authenticate
and
as
as
soon
as
that,
intent
has
been
defined,
authentication
is
required
before
the
connection
can
be
established.
A
The
actual
Service
specific
certificates
or,
in
this
case,
provided
by
spiffy,
serve
manager.
Integration
is
also
on
the
roadmap,
but
currently
the
pull
request
and
the
issue
that
I've
listed
here
on
the
slide
is
essentially
enabling
the
spiff
integration
that
allows
not
only
to
generate
certificates
for
each
service
and
thus
give
or
make
X
509
certificates
available
to
psyllium.
It
also
allows
us
to
actually
tie
Network
policy
to
spiffy
IDs,
which
means
instead
of
selecting
policy
by
via
kubernetes
labels.
We
can
actually
use
a
strong,
spiffy
IDs.
B
It's
a
kubernetes
Ingress
with
of
psyllium
type
and
as
a
result
of
that
Ingress
being
created,
we
should
have
actually
let
me
see
if
I
can
find
you
know.
I'll
show
this
first,
this
Ingress
causes
a
psyllium
Envoy
configuration
to
be
created,
which
is
that
one
and
hopefully
I'm
trying
to
do
this
with
one
finger.
That's
why
I'm
so
that
Envoy
configuration
was
automatically
created
from
the
Ingress
configuration
and
what
I'm
looking
for
is
the
bit
that
says
here
we
go.
B
B
So
let's
just
do
a
little
bit
of
telescope
shopping
to
generate
a
bit
of
traffic
we
can
buy.
These
I've
bought
a
lot
of
telescopes.
Today,
it's
quite
quite
good,
come
on.
Let's
place
order,
I,
don't
know
why
that
didn't
doesn't
really
matter,
I'm,
just
trying
to
generate
some
traffic
and
when
I
generate
some
traffic.
B
You
can
see
here
at
the
right
hand,
side.
Those
are
my
requests.
My
most
recent
requests-
and
here
is
me
just
checking
that
the
demo
Works
a
few
minutes
ago-
and
we
get
this
really
nice
layer,
seven
metrics
generated
by
psyllium
through
ebpf.
A
lot
of
these.
These
metrics
are
being
gathered
by
using
ebpf
and
they're
generated
in
Prometheus
formats,
so
they're
very
easy
to
pull
into
grafana
dashboards
like
this
getting
things
like
the
duration,
the
the
latency
rather
very
nicely
graphed
out.
We
also
get
a
service
map.
B
Now,
if
you're
familiar
with
psyllium
and
Hubble,
you
will
probably
have
seen
the
Hubble
service
map
this.
This
is
something
that
you
know
we've
had
in
Hubble
for
a
while,
and
you
can
see
individual
Network
flows
as
well
as
the
way
those
flows
go
to
build
up
an
overall
service
map.
But
we
can
now
also
see
that
service
map
inside
the
grafana
dashboard
I
hope
there
we
go
yeah.
B
A
A
The
last
point,
I
think
we've
looked
at
observability,
we
looked
at
encryption,
mtls,
Observer
and
the
last
part
is
kind
of
traffic
management.
So
how
is
it
possible
to
Define
and
use
traffic
Management
in
solemn
service
mesh,
and
we
want
to
quickly
show
like
what
is
the
state
of
the
individual
configuration
options
that
you
have
Ingress,
Gateway,
API
and
kubernetes
services,
as
well
as
the
envoy
config?
The
rest
of
the
stack
I
think
is
hopefully
pretty
obvious.
Obviously,
kubernetes
is
the
control
plane
itself.
You
don't
really
need
anything
beyond
that.
A
Kubernetes
Secrets
is
the
current
identity
management
integration
we
have
and
on
the
roadmap
is
spiffy
and
serve
manager,
and
then
the
data
plane
ebpf
and
on
y.
So
let's
look
at
these
individual
layer,
7
traffic
management
options,
Ingress
Services,
Gateway,
API
and
Envoy
config,
Ingress
and
Envoy
config
are
already
fully
supported
in
112.
A
services
with
annotations
and
Gateway
API
are
coming
in
113..
The
code
is
actually
complete,
it's
just
not
merged,
but
it's
available
in
Dev
branches
for
testing.
If
you
are
interested
I've
linked
the
pull
request
with
the
branch
or
links
in
there,
Ingress
very
simple,
it's
simply
fully
conformant.
So
it's
a
basic
example
of
path
based
or
path,
prefix
based
routing
using
Ingress
and.
A
So
this
is
the
simple,
the
simplest
option:
right:
okay,
Ingress
services
with
annotations
and
now
Gateway
API,
right,
initially
or
originally
called
Ingress,
V2,
probably
the
way
forward
for
most
of
you
how
to
define
traffic
Management
in
the
future.
We
have
the
code
complete
to
support
the
latest
Gateway
API
spec.
This
is
a
typical
example
using
the
Gateway
and
HTTP
route
resource,
also
using
path
path,
prefix
and
then
the
last
option,
and
this
is
really
kind
of
a
low
level
building
block.
But
it's
also
available
to
you
as
users.
A
You
can
actually
Define
and
configure
raw
Envoy
configuration
in
a
custom
resource.
This
is
how
the
Ingress
controller
and
the
Gateway
implementation
actually
map
or
use
sodium
service
mesh,
so
they
essentially
translate
into
slim
service
mesh.
But
if
you
want
to
use
this
lower
level,
API
you're
absolutely
free
to
do
so.
This
way,
you
can
essentially
use
the
full
Envoy
feature
set
in
your
own
service
mesh.
Definitely,
as
you
can
even
see
in
the
example,
definitely
a
lot
harder
to
use
and
but
with
a
lot
more
capabilities,
and
that
is
it
from
our
side.
C
C
A
Just
correct
so
mtls
is
used
for
the
handshake.
The
emptylus
handshake
generates
the
secrets,
the
keys
and
they
are
handed
down
to
wireguard
and
ipsec,
where
the
symmetric
encryption
happens
in
the
kernel.
So
hmac
and
encryption
is
happening
in
the
kernel,
but
the
authentication
hand,
authentication
handshake
is
done
in
user
space
using
mtls.
D
A
So
hopefully
I
got
the
question
right,
so
we
are
using
in
this
in
the
example
shown
either
kubernetes
Secrets
or
certificates
created
by
spiffy,
so
that
those
are
service,
specific
certificates
for
the
mtls
model
and
in
our
mtls
model
the
data
gets
encrypted
using
ipsecond
wireguard.
So
it's
there's
no
additional
TLS
based
encryption.
There
is
no
double
double
encryption.
Hopefully
I
got
your
question
correct.
Otherwise,
please
clarify.
F
A
B
That
was
I,
guess
a
nod
towards
the
our
understanding
of
how
ambient
mesh
is
doing
proxies
that
they're,
not
specifically
located
on
any
given
node
I,
mean
we've
kind
of
shown
it
floating
in
space.
I
think
it
would
be
on
a
node
somewhere,
but
the
point
was
more:
it's
not
specifically
associated
with
the
the
node
where
an
application
is
running.
It.
B
A
I
mean
I
think
for
both
models,
where
it's
on
the
Node
or
on
the
network.
Any
granularity
could
work,
I,
think
it's
more
just
location
versus
I.
Think
the
benefit
of
the
network
model
is
that
you
can
share
that
proxy
for
the
same
tenants
that
are
not
on
the
same
notes,
whereas
if
you're
on
that
note,
you
share
it
only
with
tenants.
On
that
same
note,
that's
the
clear
difference.
A
I
think
from
a
selling
service
merge
perspective,
we're
interested
to
support
both
models.
So
if
you
want
us
to
also
support
like
the
proxy
on
the
network
model
and
support
waypoints
file,
a
GitHub
issue
or
just
talk
to
us
on
slack
and
we're
happy
to
add
it
like
it's
actually
very
compatible
with
how
we
think
about
service
mesh
as
well.
E
Okay,
fantastic
talk
it
with
the
like
sidecar
for
every
pod.
You
kind
of
those
individual
Envoy
instances
are
only
going
to
have
their
tunnels
to
whatever
that
application
happens
to
be
talking
to
right,
whereas
in
you
know
the
on
the
Node
one,
this
was
I
know.
This
came
up
in
the
istio
example
as
well.
It's
like
on
the
Node
you've
now
got
like
a
single
Envoy
instance.
That's
going
to
have
to
do
all
the
wire
guard
or
ipsec
tunnels
from
every
application
on
that
node
to
every
application
that
that's
talking
to
like.
E
F
F
F
A
Yeah,
so
Envoy
has
several
mechanisms.
First
of
all,
you
can
Define
the
number
of
listeners
like
how
many
threats
only
can
use,
and
then
we
also
limit
the
resource
per
listener.
So
when
we
have
a
shared
Envoy
instance,
we
actually
configure
a
separate
Lister
for
every
Ingress
and
for
every
egress,
a
use
case
or
a
part
that
actually
needs,
and
we
limit
the
number
of
HTTP
requests
or
layer,
4
requests
or
layer,
4
connections
and
scale
by
the
number
of
threads,
and
obviously
it's
also
subject
to
the
CPU
memory
limits
that
you
configure
all.