►
Description
Don’t miss out! Join us at our upcoming events: EnvoyCon Virtual on October 15 and KubeCon + CloudNativeCon North America 2020 Virtual from November 17-20. Learn more at https://kubecon.io. The conferences feature presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Multi Cluster and Multi Mesh Patterns - Christian Posta, Solo.io
The more services connected by a mesh, the more value in consistent observability, security, and routing can be achieved. In today's complex heterogeneous environments, deploying everything into a single cluster and single mesh can be impossible. In this talk we look at some of the options and challenges for adopting a service mesh across multiple clusters and potentially across multiple meshes. We look at the challenges such as federating identity, single pane of glass for observability, developing policies et. al, as well as ways to solve these challenges. The patterns discussed here are not mesh specific and will use Istio, Linkerd, and App Mesh as examples.
https://sched.co/abc9
A
In
this
session
for
the
service
meshcon
track
we're
going
to
talk
about
multi-cluster,
kubernetes
and
service
mesh
patterns,
so
my
name
is
christian
posta,
I'm
the
field
cto
at
solo.I,
o
at
solo.
We
build
api
infrastructure
for
connecting
services
across
cloud-based
platforms
like
kubernetes,
using
things
like
envoy
proxy
to
build
api
gateways,
service,
mesh
and
thinking
about
it
through
the
lens
of
a
multi,
multi-cluster
and
hybrid
world.
How
do
we
solve
some
of
the
difficult
operator
challenges
and
we'll
talk
about
some
of
those
challenges
in
this
talk?
A
I've
been
involved
in
the
open
source
community
for
quite
a
long
time.
More
recently,
I
would
say
the
last
few
years
been
focusing
on
service
mesh
technologies,
kubernetes
and
istio,
and
some
of
those
related
ecosystem
projects,
I've
written
some
books
on
this
topic.
I'm
currently
writing
the
istio
in
action
book
for
manning.
Please
reach
out
to
me
on
twitter.
I
blog
about
these
topics
quite
frequently,
and
I
have
been
doing
so
over
the
last
five
years.
So
just
go.
Take
a
look
reach
out
if
you
have
any
questions
or
thoughts.
A
A
A
We
originally
saw
this
this
idea
of
deploying
into
large
clusters
and
making
the
cluster
multi-tenant
and
highly
available
and
so
forth.
More
recently,
I
would
say
the
last
few
years
we've
seen
a
trend
go
more
toward
many
having
many
smaller
clusters,
where
you
might
have
clusters
dedicated
for
specific
data
locality
reasons
for
specific
networking
reasons.
Maybe
you
have
clusters
at
the
edge
in
a
dmz
and
different
ones,
in
your
core
network
for
isolation,
performance
reasons
and
compliance
and
so
forth.
A
We
we
see
a
lot
of
different
motivation
to
going
toward
a
multi-cluster
setup,
whether
that's
on
premises
and
you're,
running
multiple
clusters
of
kubernetes
or
in
a
public
cloud
like
a
eks
or
a
gke
or
both
right.
So
there
might
be
when
we
run
into
customers
and
work
with
customers
to
solve
solutions
for
both
on-prem
and
in
a
public
cloud.
A
And
so
that
brings
us
to
this
need
at
times
to
share
data
between
the
clusters
to
give
the
illusion
or
appearance
of
a
sort
of
loose
federation,
so
that
services
in
these
different
clusters
can
talk
with
services
in
the
other
other
clusters
and
do
that
in
a
regulated
way.
If,
if
needed,
but
also
in
a
more
uniform
way.
So
we're
not
dealing
with
the
challenges
of
you,
know
configuring
and
managing
applications
across
these
different
clusters
completely
independently
and
having
to
bespoke,
try
to
figure
out
how
to
how
to
put
the
pieces
together.
A
A
So
we
can
have
multiple
clusters,
but
we
can
have
them
all
on
the
same
network
and
have
them
all
routable
the
different
applications
and
service
services
and
pods
within,
let's
say,
kubernetes
routable
and
talking
directly
to
each
other,
and
this
is
not
a
use
case
that
we
run
into
very
often,
but
if
you
can
achieve
it,
it
is
simpler
right
and
so
trying
to
build
these
microservice
based
platforms
or
or
platforms
to
support,
microservices
and
so
forth.
You
know
simple
is
good.
A
Another
option
is
if
they're
on
different
networks
is
to
expose
the
services
running
in
a
remote
cluster
to
anyone
running
outside
of
that
that
cluster,
and
so
maybe
in
a
in
a
public
cloud,
you
may
expose
these
as
cloud
load
balancers,
maybe
internally,
if
you
don't
have
load
balance,
you
might
use
something
like
node
port
and
pin
them
to
specific
ports
in
your
cluster,
and
this
can
have
a
very
interesting
ramifications
right.
You
can
have
if
you
have
a
lot
of
services
and
you're
trying
to
expose
and
allow
communication
between
the
the
clusters.
A
A
A
This
is
a
fairly
common
pattern,
because
ingress
is
what's
used
to
allow
traffic
into
a
particular
cluster
in
the
first
place,
and
so,
if
the,
if
a
cluster
is
trying
to
talk
or
application
in
cluster,
one
is
trying
to
talk
to
services
running
in
cluster,
two
they're
gonna
have
to
go
through
the
ingress.
Now
the
trick
is
to
get
cluster
one
to
understand
that
when
it
talks
to
an
application,
does
it
have
to
know
exactly
that?
It's
talking
to
another
cluster.
A
A
A
Now,
when
you
have
multiple
multi-cluster
setup-
and
you
have
let's
say
the
same
either
this
replicas
exact
same
clusters,
you
know
multiple
of
those
or
you
just
have
the
same
service
running
in
multiple
clusters.
Maybe
they're
not
replicas,
but
if,
if
traffic's
coming
into
a
particular
service-
and
it
starts
to
fail
or
one
of
its
dependencies
starts
to
fail,
you
want
to
load
a
failover
to
a
different
working
service
that
might
end
up
being
in
a
different
cluster.
A
You
got
a
design
for
well
understood,
and
you
know
well
thought
fault
domains
so
that
you
don't
just
accidentally
start
creating
a
system
that
if
one
part
of
the
system
goes
down,
it
starts
to
slowly
take
everything
else
down
with
it
right.
So
you
want
these
isolated
fault
planes
and
you
don't
want
faults
to
jump,
jump
the
the
different
boundaries
that
you
set
up.
A
So
when
we
talk
about
services,
communicating
with
each
other
and
some
of
the
challenges
around
that
you
know
for
there's
various
ways
of
solving
some
of
the
network
or
application.
Networking
challenges
that
crop
up
in
that
in
that
area,
things
like
timeouts
and
retries
load,
balancing
client-side
load,
balancing,
telemetry
collection,
service
discovery
so
forth.
A
This
is
an
area
where
a
service
mesh
can
help.
Now,
whether
or
not
you
need
a
service
mesh
for
your
use
case,
that's
you
know.
Let's,
let's
put
that
aside,
but
a
service
mesh
can
help
in
this
area
and
what
we're
seeing
especially
around
multi-cluster
support,
is
these
different
implementations
continuing
to
mature
so,
for
example,
linker
d,
which
is
heavily
kubernetes
based
just
released.
You
know
a
little
while
ago
the
their
multi-cluster
support
in
lingerie
2.8
istio
has
various
flavors
of
multi-cluster
support,
console
service
mesh
from
hashicorp.
A
You
know
there
they
have
multi-cluster,
support
and,
and
so
forth,
so
definitely
encourage
you
to
take
a
look
at
each
individual
implementation,
and
you
know
we
can
map
it
back
to
some
of
the
patterns
that
we
talk
about.
I
will
use
some
concrete
examples
in
the
in
the
rest
of
the
presentation,
but
I
encourage
you
to
check
out
the
implementation
details
for
the
mesh
that
you
choose.
A
So
one
of
the
building
blocks
of
quite
a
few
service
meshes
is
a
technology
called
envoy.
Now
envoy,
isn't
the
only
way
to
build
a
service
mesh,
and
we
see
that
linkerd
folks
successfully
built
their
own
proxy,
but
then
you
know
the
the
istios,
the
app
matches,
the
consoles
and
so
forth.
They're
they're,
built
on
unboying,
we
see
more
kuma
and
so
forth.
A
So
all
the
network
traffic
going
into
the
application
or
out
of
the
application,
goes
through
onboard
proxy
and
envoy
proxy
has
a
lot
of
networking
capabilities
at
the
l7
layer
that
that
can
do
very
sophisticated
things
like
client-side
load,
balancing
priority
failover
circuit,
breaking
tls
origination
and
termination
statistics,
collection
and
so
forth.
Right
so
envoy
is
a
very
powerful
component
and
plays
a
large
role
in
in
the
solution
that
we're
going
to
be
and
the
solutions
and
the
patterns
that
we're
going
to
be
talking
about
so
envoy.
A
Some
of
the
features
that
envoy
brings
the
table
things
like
request,
hedging
things
like
retries
and
retry
budgets,
load,
balancing
and
locality,
aware
load,
balancing
or
zone
aware
routing.
So
we
can
route
to
different
zones
based
on
failure,
parameters,
things
like
aggregated
cluster,
it
and
point
some
of
these.
A
It's
not
exactly
the
same,
but
it's
called
locality
aware
and
in
the
zone
aware.
Example,
we
saw
envoy
having
the
heuristics
on
its
in
the
proxy
itself
to
make
the
determination
of
when
to
fail
over
locality,
aware
load,
balancing
is
more
of
a
weighted
load,
balancing
that's
driven
by
the
control
plane.
So
in
this
case
the
service
mesh
control
plane.
So
the
service
mesh
control
plane
might
know
where
the
different
endpoints
are
potentially
in
different
clusters
and
and
configure
the
envoy
proxy
running
with
the
application.
A
With
that
knowledge
of
that
that
locality,
awareness
and
the
weights
that
need
to
be
assigned
for
the
different
localities,
so
you
again
can
prefer
communication
from
service
in
this
case
account
to
you
know
whatever
other
dependent
service,
it
has
to
be
local.
So
in
the
same,
let's
say:
availability
zone
or
same
subset,
and
then,
if
that
starts
to
go,
you
know
those
endpoints
become
unhealthy.
Then
we
can
slowly
fail
over
to
the
next
availability
zone
or
region
and
so
forth.
A
But
this
locality
awareness
is
driven
by
again
the
control
plane,
not
envoy,
making
the
decision
itself.
Another
another
approach
that
envoy
allows
for.
Is
configuring
a
a
priority
fallback?
So
if
we're
talking
again
locally
services
communicating
locally
and
they
start
to
become
unhealthy
envoy
can
route
instead
of
to
the
local
unhealthy
services
can
route
to
a
different.
You
know
different
fallback
services
that
may
be
running
in
a
different
cluster
and
may
have
different
load,
balancing
parameters
and
so
forth.
A
A
A
So
now
you
have
you
have
services
running
in
different
clusters,
but,
like
we
said
in
the
beginning,
if
you
have
a
flat
network
and
they're
directly
addressable,
the
services
can
just
reach
out
and
talk
with
each
other.
The
control
plane
can
build
a
locality
awareness
into
the
into
the
the
different
proxies
so
that,
for
example,
when
account
is
calling
the
user
service,
it
will
prefer
to
call
the
user
service
in
its
same
locality,
but
then
fail
over
to
other
ones.
A
A
But
if
user
starts
to
become
unhealthy,
that
it
needs
to
fail
over
to
the
second
cluster
through
that
cluster's
gateway
and
the
control
plane
can
be
smart
enough
to
create
the
illusion
or
the
mirroring
configuration
necessary
so
that,
when
account
talks
to
user,
it
actually
talks
directly
to
that
other
gateway.
Not
to
you
know
the
user,
that's
running
in
the
second
in
the
second
or
remote
cluster,
because
those
separate
networks
here
right.
So
we
do.
We
do
need
to
make
that
hot
through
the
gateway.
A
A
It
all
you
know,
gets
affected
and-
and
you
you
have
this
increase
in
burden
on
the
operator
to
automate
and
label
the
networks
and
some
of
these
endpoints
correctly.
So
you
know
that
for
this
one
control
plane
where
the
different
endpoints
are
the
last
pattern
we'll
look
at
is
having
separate
control
planes
and
one
per
cluster
right.
So
we
kind
of
solve
the
failure
domain
problem
here.
So
if
the
control
plane
cluster
one
goes
down,
it
doesn't
affect
the
operability
of
cluster
two.
It
has
its
own
control
plane,
but
the
traffic
flow
is
similar.
A
A
We
are
able
to
have
control
and
be
very
opinionated
about
what
services
should
be
mirrored
across
the
network
and
it
it
allows
the
traffic
to
flow
through
the
network
and
across
to
these
different
clusters
through
the
ingress
points,
as
we
expect
right.
So
we
have
a
nice
and
uniform
controlled
way
of
traffic
flowing
through
the
network.
A
A
So
there's
some
service
discovery
orchestration
that
needs
to
happen
there.
We
need
the
capability
to
define
traffic
rules
for
these
different
services,
with
the
multi-cluster
use
case
in
mind
all
right,
so
this
is
going
to
involve
more
automation
and
shuffling
configurations
around
to
get
the
illusion
of
you
know
a
single
federated
name
namespace,
I
guess
for
these
different
services.
A
So
you
know
this.
This
ends
up
being
shifted
from
the
from
the
the
mesh
all
right
to
the
operator.
Now
operator
is
responsible
for
all
these
things,
and
that
can
be
a
lot
that
can
be
a
lot
to
set
up.
More
importantly,
there
can
be
a
lot
to
maintain
going
forward,
because
you
know
the
these
topologies
changes
the
services
change
they
move.
A
What
that
would
look
like
is
service
mesh
hub
running
in
its
management
plane
able
to
do
the
configuration
shuffling
the
service
mirroring
the
traffic
control,
regardless
of
you
know
how
many
clusters
that
you
have-
and
you
can
be
very
multi-cluster
aware
in
the
configuration
that
you
use
to
drive
this
behavior,
so
it's
probably
best.
I
have
a
few
minutes
left
here
to
to
show
you
a
quick
demo
of
what
that
looks
like.
A
A
A
So
at
the
moment,
the
discovery
is
able
to
find
clusters
based
on
istio
clusters
based
on
linker
d
and
clusters
based
on
console
service
mesh,
as
well
as
so.
It's
based
on
app
mesh
from
aws
and
the
architecture
for
surface
machine
is
fairly
plug
or
very
plugable,
so
that
we
can
plug
in
additional
ones
like
like
akuma
or
whatever
pretty
easily
so
we
have
cluster
one
registered
cluster
two
seems
to
be
okay
now
registered.
A
Now,
let's
see,
let's
see
what
service
mesh
hub
has
found.
So,
firstly,
we
see
the
clusters
that
we
registered
and
knows
about
those.
So
that's
a
good
step.
The
second
thing
is:
let's
ask
service
mesh
hub
what
meshes
are
running
in
these
different
clusters?
We
see
we
actually
have
two
istio
clusters,
one
running
in
cluster,
one
one
running
in
cluster
two.
A
Now,
let's
see
how
easy
it
is
to
start
to
build
the
federation
automation
around
these
two
different
service
meshes
that
have
been
deployed
so
in
service
mesh
hub.
We
have
a
resource
called
this
virtual
mesh
and
the
virtual
mesh
is
a
way
of
defining
the
different
federation
mode,
the
identity
sharing
or
the
the
identity
federation
mode.
A
So
now,
if
we
apply
that
service
mesh
hub
will
then
start
to
do
some
of
the
automation
around
building
the
federation.
For
these
meshes
and
let's
take
a
look
at
the
virtual
mesh
and
cross
our
fingers,
we
see
that
the
federation
has
been
accepted.
All
the
config
looks
good.
The
certificates
for
unifying
the
identity
looks
good
and
the
information
around
access
control,
which
we
haven't
specified
anything
yet
looks
good.
A
So
we'll
give
it
a
second
using
this
federated
mesh.
We
can
do
things
like
in
this
case,
we'll
look
at
what
it
what
it
created
in
terms
of
service
discovery,
so
the
service
mirroring
part.
The
configuration
in
istio
uses
an
object,
called
the
service
entry,
and
we
see
that
in
cluster
number
one
we
have
entries
for
the
services
that
are
running
in
cluster
number
two
and
vice
versa,
all
right
so
now
the
services
in
cluster
two
can
call
the
services
running
in
cluster.
A
One
servicemenship
can
also
do
a
global
globally
unique
namespace
for
the
different
services
so
that
you
don't
have
to
have
have
them
cluster
aware,
but
in
this
demo
we're
we're
showing
how
you
know
that
automation
can
be
can
be
put
in
place
now.
The
last
thing
we're
going
to
look
at
is
the
traffic
routing
api
that
we
can
enable
for
service
through
the
for
the
clusters
right.
It's
basically
a
way
of
orchestrating
traffic
policies
and
traffic
rules
in
a
multi-cluster
aware
way.
A
You
know
using
this,
this
abstraction,
this
federation,
that
we
built
on
top
of
it.
So
if
we
apply
this
resource,
what
this
is
going
to
allow
us
to
do
is
call
the
so
this
is
using
the
book
info
demo
in
istio
we're
going
to
be
able
to
call
between
cluster
one
and
cluster
two.
Do
I
have
the
port
forwarding
thing?
No,
but
all
right,
I
guess
I
won't
show
the
port
forwarding
thing,
but
what
we
can
see
is
automatically
the
different
resources
under
the
cover
for,
in
this
case,
istio
were
automatically
created
all
right.
A
The
last
thing
I
guess
I'll
show
is
that
we
can
pull
back
any
other
any
of
the
rules
that
apply,
even
if
there's
multiple
rules
that
apply
to
a
particular
service
fairly.
Simply
so
that
is,
you
know
how?
How
do
we?
How
do
we
build
this
type
of
system?
Following
some
well-known
patterns,
the
service
meshes
typically
implement
those,
but
then
you
know
nothing's
free.
So
you
get
this
operational
challenge
that
and
burden
that
you
have
to
solve
for,
and
that
is
where
the
service
mesh
hub
project
helps.
A
So
I
thank
you
for
joining
this
talk
and
I
hope
you
enjoyed
the
previous
talks
in
this
track
for
service
mesh
con.
I
think
there's
a
couple
left
and
then
we
get
to
the
closing
remarks
from
nick
jackson,
but
go
check
out.
You
know,
link
or
d
go
check
out,
istio
and
console,
and
and
some
of
these
other
ones,
kuma
just
joined
the
cncf,
go
check
out,
kuma
and
and
come
take
a
look
at
us
and
what
we're
doing
at
solo.