►
From YouTube: Multi(Control Plane/Network/Mesh)??: A Practical MultiCluster... Nicholas Nellis & Vikas Choudhary
Description
Multi(Control Plane/Network/Mesh)??: A Practical MultiCluster Deployment - Nicholas Nellis & Vikas Choudhary, Tetrate
While Working with several traditional customers spanning defense, finance, etc., we found that the service mesh multicluster models that exist today are completely unusable from an enterprise point of view. They are designed with the network administrator in mind, focusing on how to connect two clusters, and not on how developers across teams like to consume the services exposed by other teams. The multicluster models that app teams want, turned out to be dramatically simpler than the ones out there today. This talk discusses our experiences working with these teams, our learnings from how they built out an API-centric multicluster model and what we as a community of (mostly) infrastructure developers should do to better support the application teams
A
Hello,
everyone-
this
is
vikas
and
nick
from
tetrad
at
data
we
work
at
stu
and
other
service
smash
technologies
around
stu.
In
this
talk,
we
are
going
to
discuss
and
share
our
experiences
and
learnings
that
we
gained,
while
working
with
some
of
our
large
enterprise
customers,
trying
to
help
them
in
magifying
their
multi-cluster
installations.
A
B
Thank
you,
yeah.
Congratulations.
Everybody
you've
been
you've
made
it
to
multi-cluster
you've
successfully
scaled
your
application
from
a
single
cluster
to
multiple
but
you're.
Looking
at
back
and
your
slas
haven't
improved,
you
broke
a
bunch
of
security
rules
in
doing
so
so
that
you
could
get
the
networking
to
connect
properly
and
then
you're
realizing
that
management
has
got
a
lot
more
difficult.
B
It
seems
like
the
problems
that
you're
facing
are
bigger
than
the
ones
you
were
doing
earlier.
So
trading
off
a
better
sla
for
more
problems,
didn't
really
seem
advantageous
to
you
and
so
we're
going
to
walk
through
some
of
these
problems
that
a
lot
of
our
customers
have
early
on
when
they're
adopting
the
multi-cluster
model,
the
first
one
that
we
want
to
talk
about
is
the.
This
is
a
typical
deployment
that
we
that
you
would
see
when
you
go
from
a
one
to
multi-cluster
environment.
B
Typically,
you
have
a
low
balancer
within
the
cluster
engine
x.
Ingress
is
like
the
default
one
that
most
you're
probably
familiar
with,
and
then
we
connect
all
of
those
clusters
and
do
the
routing
with
a
tier
one,
which
we
call
tier
one
low
balance
here,
which
is
typically
your
your
cloud
load
balancer
and
that's
so
you
have
a
single
point
of
entry
and
can
route
between
multiple
clusters,
but
you're
you're
team.
One
you've
you've
been
tasked
to
move
your
microservice
from
one
cluster
into
a
multicultural
environment.
B
Now
you're
you're
gonna
follow
the
coattails
of
team.
Two
who's
already
been
doing
it
for
a
little
while
and
so
you're
going
to
take
your
application
and
stick
it
behind
that
same
default,
low
balancer
that
that
team
won
or
team
2
has
been
using,
but
also
you
start
to
know.
You
have
more
outages
than
you
have
before
you've
bound
yourself
now
to
the
failures
of
t2
team.
B
And
finally,
as
your
multi-cluster
architecture
grows,
you
can
get
some
really
complex
dependency
chains.
You
it's
be
very
difficult
for
you
to
trace
all
the
all
of
the
consumers
that
depend
on
you
and
your
functionality,
and
so
without
service.
Mesh
you'll
find
these
problems
are
very
prevalent,
and
so
then
you'll
be
looking
for.
That
solution,
which
you
know
service
mess,
seems
to
be
the
one
that'll
that'll
solve
these
problems
for
you,
and
so
as
you're
adopting
service
mesh.
B
A
So
yeah
there's
no
denying
that
istio
is
super
powerful.
It
can
it's
you
can
help
in
in
in
in
is
you
can
help
in
managing
these
complexities
of
multi-cluster
installations,
but
but
how
exactly?
I
am
going
to
overlays
to
measure
multi-cluster
installations,
because
there
are
different
teams,
different
different
personas,
different
administrative
boundaries
over
these
clusters.
So
how
how
one
is
going
to
do
that.
A
So
we'll
start
with
taking
a
look
over
what
the
steel
community
documentation
has
been
suggesting.
So
at
the
moment
there
are
in
the
latest
release
which
is
1.7.
There
are
two
multi-cluster
installation
approaches,
the
one
is
replicated
control
planes
and
another
one
is
shared
control
plane
because
of
the
resiliency
requirements.
A
A
The
services,
which
are
the
shared
services
and
are
supposed
to
be
accessed
from
the
remote
clusters,
are
exposed
through
service
entries,
for
example.
Here
the
service
who,
which
is
in
the
cluster
2,
is
exposed
on
the
cluster
1
using
a
service
entry
and
the
host
name
in
the
service
entry
has
this
dot
global
prefix
added
to
the
name
and
namespace
of
the
service
and
the
endpoints
of
the
service
entry
points
to
the
gateways
of
the
remote
clusters
at
runtime.
A
It
looks
something
like
this.
When
a
client
makes
a
request,
it
goes
to
kubernetes
a
dns
server,
kubernetes
kubernetes
dns.
We
have
to
run
with
a
core
dns
plugin,
which
is
shipped
by
all,
so
all
the
dot
global
prefixed
hostname
queries,
kubernetes
dns
pass
these
queries
to
core
dns
and
poor
dns
resolves
these.
A
These
hostname
dns
queries
with
the
virtual
id
of
the
service
entry
and
using
this
virtual
ip
on
the
sidecar,
picks
up
the
listener,
and
there
the
listener
has
the
is
pointing
to
the
remote
gateway
where
the
actual
the
services,
the
back-end
service,
is
running.
And
this
way
the
request
reaches.
There
is
cluster
tools,
gateway
and
sni
host.
It
still
has
the
hostname,
which
has
got
global
prefix,
so
it
won't
make
to
the
actual
backend
service.
A
A
A
We
cannot
simply
if,
if
we
have,
if
we
have
a
local
instance
of
the
service
running
in
cluster,
ideally
it
is
desired
that
the
for
that
the
first
priority
should
be.
The
request
should
be
served
by
the
local
instance,
so
we
should
be
able
to
somehow
the
local
instance
should
be
part
of
the
load
balancer
pool,
but
we
cannot
simply
here
in
this
approach.
We
cannot
simply
put
the
cluster
ip
in
the
service
entry
endpoints
and
the
reason
is
the
same.
This
dot
global
thing
we
can
fix
this.
A
We
can
make
it
correct
by
adding
virtual
service
to
rewrite
this
host
name,
converting
this
door
global
to
canonical
name
of
the
service
in
the
local
local
cluster.
But
this
problem
is
not
the
only
problem
there
is.
There
are
much
bigger
problems
than
this
in
this
approach,
the
so
so
the
host
names
in
the
clients
are
directly
depending
on
the
actual
back-end
implementation
of
the
services.
A
So
the
the
clients
on
the
remote
clusters
here
at
the
cluster
one
should
use
the
host
name,
which
is
the
service
name,
actual
service
name,
dot
service
name,
space
namespace.global,
and
this
is
pretty
bad,
because
if
there
are
n
number
of
clusters
serving
this
service
serving
this
api
serving
this
service,
then
the
owner
of
these
services
cannot
change
the
backend
implementation
of
the
service
at
all,
because
the
clients
will
get
broken
in
that
case.
So.
A
A
So
in
the
1.8
we
have
this
a
new
approach
and,
and
this
replicated
control
plane
is
not
there
anymore
in
the
coming
releases,
but
so
in
this
approach,
for
example,
which
is
coming
here
each
here
also,
we
have
a
stereo
control
plane
running
in
each
of
the
clusters.
And
what
change
here
is
this
that
the
study
of
each
cluster
is
watching
the
api
servers
of
all
the
remote
clusters.
A
So
this
solves
the
locality,
the
aware
routing
problem,
because
because
the
end
points
it
has
because
it
has
all
the
local
endpoints
and
the
remote
endpoint
in
the
same
lb
pool,
but
the
problem
of
the
direct
dependency
on
the
canonical
name
of
the
service
and
namespace
is
still
there.
Still.
There
is
no
abstraction
and
plus
the
scalability
issues
are
there
and
the
security
concerns
are
there
because
these
clusters
are
owned
by
different
teams
and
they
may
not
want
to
expose
all
their
internal
implementation
of
the
services
to
other
teams.
B
Well,
developers
just
want
to
consume
applications
as
sas
products
even
internally.
They
want
to
access
your
api
that
assume
that
it's
and
that
you're
you're
routing
locally
or
you
know
efficiently,
and
so
they
don't
really
necessarily
care
where
your
your
service
is
hosted.
It's
that
they
can
just
reach
it
easily.
B
And
so
with
that
in
mind,
we
came
up
with
a
much
more
simplistic
approach
to
initial
installation
and
that's
by
deploying
istio,
essentially
isolated
per
cluster.
We
deploy
the
control
plane
on
every
cluster.
We
scope
that
control
plane
to
only
know
about
services
within
the
cluster
that
it's
residing,
and
so
we
have
a
locally
scoped
mesh.
B
But
why
so?
Why
do
we
want
to
do
separate
pistol
control
points?
Well,
we
really
want
to
align
the
failure
domains
of
your
application
to
that
cluster,
and
so,
if
you
push
bad
configs
or
you
upgrade
istio
incorrectly
and
it
causes
an
outage,
it
is
now
localized
to
that
cluster
that
you
were
working,
and
so
it
gives
you
a
lot
more
control
over
not
bringing
down
your
entire
environment,
and
it
aligns
that
issue
control
plane
with
the
underlying
cluster
that
it's
running
out.
B
So
networking
and
node
management
stuff
like
that,
and
so
it's
a
lot
a
lot
more
effective
for
our
customers
to
manage
them
these
days
and
ca,
especially
for
outages
and
then
it
also
allows
you
to
do
safer
upgrades.
So,
as
you
upgrade
issue,
you
might
be
able
to
pick
a
cluster
that
has
less
usage
update
that
one.
If
it
goes
successfully,
then
you
can
roll
it
out
to
other
clusters,
and
this
this
would
differ
from
the
shared
control
plane.
That
was
one
of
the
seo
recommendations.
B
But
we
want
you
to
use
them
on
a
product,
focused
architecture
and
so
currently,
you're,
probably
using
the
engine
x,
ingress
gateway
or
you're
if
you're
using
istio
the
defaulting
gas
gateway
and
istio
system.
But
we
want
you
to
kind
of
get
rid
of
those
and
move
to
this
product
focus
database,
and
so
in
istio.
That's
really
easy.
B
So
in
a
in
a
cluster,
this
is
kind
of
what
you
would
imagine,
logically,
what
is
happening
from
using
a
gateway,
the
consumer
name.
Space
on
the
right
here
is
not
related
to
the
payments
api
product,
but
it
does
consume
that
api,
and
so
logically,
you
should
be
consuming
it
at
the
gateway
level
that
allows
you
to
scale
microservices,
add
functionality
behind
it
without
interruption
of
that
service,
and
so
when,
when
we
go
to
a
multi-cluster
environment,
this
becomes
really
easy.
B
We
can
then
just
replicate
that
payments
namespace
into
another
cluster
with
that
gateway
in
as
a
whole
package,
and
then
the
consumer
namespace
now
has
multiple
endpoints
to
reach
you
at,
and
so
we
we
add
those
those
hosts
to
that
consumer
namespace.
So
now
it
has
two
options
so
making
your
your
payments
api
highly
available,
but
we
didn't
have
to
stop
there.
B
We
can
say
we
can
do
some
more
intelligent
routing,
that's
saying
if
your
consumer
is
in
the
same
cluster
as
the
gateway
that
it's
consuming
and
that
gateway
is
closer
than
say,
an
external
gateway
in
another
cluster.
Let's
prefer
routing
locally
first,
and
so,
if
you're
in
the
same
cluster,
we
can
actually
use
the
sidecar
to
act
as
the
gateway,
and
so
in
this
example.
B
This
is
what
we're
doing
we're
using
the
sidecar
to
act
as
the
gateway
to
those
microservices
to
access
them
directly,
but
in
the
case
of
failure
or
outage,
we
can
actually
reroute
requests
over
to
the
cluster
2
in
the
u.s
west
region.
So
you
still
get
that
highly
available.
But
now
we've
added
a
component
of
this
intelligent,
local,
aware
routing.
B
So
why
should
you
embrace
gateways
more
than
you
are
today?
It
gives
you
a
way
to
abstract
your
apis
and
your
microservices
and
with
a
product
spin
on
a
product
focus
is
really
architected
for
growing
those
in
a
multi-cluster
environment.
It's
more
of
a
copy
paste
than
a
one-off
or
trying
to
figure
out
who
owns
the
local.
B
B
So
if
we
put
this
all
together
and
what
we're
doing
at
tetra
is
we're
making
that
gateway,
discovery,
automated
and
so
from
any
number
of
clusters
that
you
you
can
stand
up,
the
gateways
will
automatically
be
discoverable
in
all
other
clusters.
So
that
means
we
don't
need
to
know
about
all
the
other
services
microservices
running
in
other
clusters.
We
just
need
to
know
where
those
gateway
in
ingress
gateways
are.
B
We
are
making
sure
that
when
you
run
to
those
other
gateways
outside
of
your
own
cluster
that
it's
encrypted
and
it's
authorized
and
then
we're
also
implementing
locality-based
routing
so
to
improve
your
cost
savings
and
then
failover
to
failover
effectively
and
then
finally
we're
using
a
newer
technology
seo,
which
is
mesh
dns.
But
it
allows
you
to
eliminate
the
need
for
that
name,
space
routing
that
the
cost
was
talking
about
earlier.
A
Thanks
nate
all
right,
so
so
from
the
service
owner
point
of
view.
Exposing
a
service
to
the
external
user
has
the
same
security
concerns.
If
the.
If
the
intention
is
to
expose
the
service
within
the
mesh
but
to
the
remote
clusters,
which
are
sitting
across
the
public
internet,
but
essentially,
whether
it
so
even
the
even
the
request
is
coming
from
within
the
mesh,
but
it
is
coming
through
the
public
internet.
A
So
in
so
with
this
point,
we
in
our
model,
the
service
owner,
is
supposed
to
just
expose
the
service
to
the
external
world
over
the
chosen
gateway
port
and
the
user
is
supposed
to
configure
the
whatever
security
measures,
authentication
and
authorization
he
or
she
feels
comfortable
with
to
deal
with
the
public
internet
security
concerns,
and
what
we
do
programmatically
is
that
the
the
the
gateway
hosts
the
apis
which
are
exposed
to
the
external
world.
We
auto
automatically
expose
these
apis
within
the
mesh
to
be
consumed
from
the
remote
clusters.
A
We
we
we
discovered
what
the
apis
are
exposed
to
the
external
world
and
we
expose
the
same
apis
over
htmtrs
on
this
reserved
port
for
the
east-west
traffic
and
additionally,
in
addition
to
the
mdls,
we
apply
all
the
authentication
and
authorization
configurations
which
are
which
are
which
the
user
has
configured
for
the
external
traffic.
We
apply
those
on
on
this
one,
five,
four,
four,
three
port
as
well,
so
it
is
like
double
secure.
A
In
addition
to
that
now,
this
is
a
very
important
slide.
We
create
a
local
service
entry
as
well,
and
the
point
to
focus
here
is
that
the
host
name
in
the
the
host
name
in
the
service,
for
example,
here
peer.example.com,
is
same
the
same
as
the
host,
which
is
exposed
to
be
consumed
by
external
world
over
944
report.
What
this
means
is
that
whether
the
client
of
the
service
is
within
this
same
cluster
or
in
some
other
remote
cluster
or
some
external
user,
all
are
consuming
this
service
with
the
same
abstracted,
api
pay.example.com.
A
And
now
this
is
how
it
will
look
if
there
are
more
than
one
clusters
where
this
back-end
service
has
instances
running.
So
this
service
entry
in
the
local
cluster
has
two
endpoints.
The
one
is
the
cluster
ip
of
the
local
service
instance
and
this
another
one,
the
in
the
green
car.
This
is
this
is
the
gateway
ip
address
of
the
remote
cluster,
where
one
more
instance
of
the
service
is
running.
A
So,
first
of
all
on
when
the
client
makes
a
request,
the
client
makes
a
request
to
the
abstracted
api
paid.example.com,
whether
it
is
for
the
local
links.
So
the
client
did
not
bother
about
where
the
service
instance
is
running
locally
or
globally
or
wherever
it
is
so
abstracted.
Api
is
used
by
the
clients.
A
A
Additionally,
the
gateway
will
authorize
the
request
using
the
using
the
extra
authentication
and
authorization
configurations
which
the
user
service
owner
has
configured
for
the
external
external
world.
So
here
it
can
be
any
external
oauth
service
which
will
authorize
the
request
in
addition
to
htmtls,
and
if
everything
passes,
then
the
request
is
gets
routed
to
the
actual
instance.
A
A
A
We
are
handling
multi-tier
architectures
as
well.
We
are
handling
where
the
clusters
are
in.
The
the
clusters
are
in
are
part
of
the
different
vpcs
which
are
not
directly
connected,
but
talking
through
a
shared
vpcs.
So
we
would
really
like
to
explain
all
these
things,
but
because
of
time
constraints,
we
cannot
talk
over
this,
so
we
can
take
these
things
offline.
If
you
are
interested
in
knowing
more
about
this,
and
I
think
that's
pretty
much
all
thank
you.
Thank
you.