►
From YouTube: Here Be Services: Beyond the Cluster Boundary with Multicluster Servi... Laura Lorenz & Stephen Kitt
Description
Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Here Be Services: Beyond the Cluster Boundary with Multicluster Services - Laura Lorenz, Google & Stephen Kitt, Red Hat
For most of Kubernetes history, the edge of the cluster was always a hard boundary that was difficult to traverse. Single clusters can scale up with huge node pools and extensive geographic coverage; but many users prefer to use multiple Kubernetes clusters. This gives them more control, with isolated workloads, manageable latency, and clear data jurisdictions, reducing the effort to provide reliable platforms. However that isolation requires a lot of work to access services from one cluster in another. By using the Multicluster Services API, an OSS standard sponsored by SIG-Multicluster, users with an application spread across multiple clusters can make services available between them using simple Kubernetes object primitives available as CRDs, without needing to install a full service mesh. In this talk, we will explain the concepts and architecture behind the MCS API and walk through demos on Submariner.io and GKE, the two current implementations of the MCS API.
A
And
we're
going
to
talk
to
you
about
the
uncharted
unexplored
world
of
multi-cluster
services.
Let's
start
with
the
map,
shall
we?
This
is
the
linux
globe
famous
for
anachronistically,
representing
an
unknown
region
with
the
words
here
be
dragons
they're
in
the
corner
here
where
it
says
hickson
draconis.
A
A
You
want
to
open
trade
routes,
but
that
means
having
open
ports
and
sending
ships
which
risk
being
attacked
by
pirates
across
those
uncharted
seas.
Even
if
port
access
is
controlled,
which
is
good
because
you
can
then
collect
taxes,
there's
a
lot
of
effort
in
each
port
and
since
the
seas
are
uncharted.
How
do
other
islands
know
about
your
ports.
B
Now
there
are
some
other
ways
that
people
have
been
trying
to
do
this.
We
are
at
the
service
mesh
track
of
this
conference.
After
all,
so
istio
is
an
option.
This
is
an
open
source
service
mesh
that
you
can
deploy
across
your
kubernetes
clusters
and
vms
in
any
provider.
It's
super
powerful,
but
not
everyone
chooses
to
implement
a
whole
service
mesh
or
it
might
have
a
smaller
kubernetes
deployment
for
which
it
feels
too
heavy
other
folks
choose
to
roll
their
own
solution.
B
B
The
whole
idea
sigmulti
cluster
is
going
for
is
to
provide
a
multi-cluster
solution
that
feels
really
natural
to
have
people
already
use
kubernetes
services.
Today,
everything
is
modeled
around
the
existing
service.
Primitive
and
mcs
defines
the
minimal
amount
of
extra
stuff
needed
to
make
them
available
across
the
clusters.
B
The
goal
is
that
client
applications
should
have
barely
any
changes
in
order
to
use
a
multi-cluster
service
instead
of
a
cluster
local
service.
This
api
is
based
around
the
idea
of
a
cluster
set
capital
c
capital
s,
which
is
some
number
of
clusters
that
are
working
together.
The
standard
describes
certain
concepts
that
must
hold
true
for
clusters
participating
in
a
cluster
set
for
them
to
safely
deploy
service
objects
across
multiple
clusters.
B
So,
let's
talk
about
what
those
concepts
are,
shall
we
first
up
is
namespace
sameness.
This
is
the
driving
principle
for
mcs
that
shared
namespaces
across
a
cluster
set
should
have
the
same
contents
everywhere,
especially
services.
This
is
similar
to
the
idea
of
extending
services
to
a
cluster
set
or
extending
namespaces
to
cluster
sets.
B
This
is
like
the
fact
that
if
your
name
space
exists
across
both
of
those
clusters,
you
would
expect
the
same
services
to
be
able
to
grow
in
both
of
those
name
spaces
too.
So
in
real
practical
terms,
this
means
that
services
in
a
given
name
space
must
be
the
same
in
any
cluster.
In
the
cluster
set,
where
they
are
deployed,
they
will
be
considered
functionally
identical
and
the
back
ends
of
any
matching.
A
A
So
let's
talk
about
some
of
those
api
objects,
especially
the
ones
that
are
new
in
the
mcs
api.
A
service
export
is
a
really
simple.
Api
object,
pretty
much
just
name
and
namespace,
but
the
point
of
it
is
to
mark
a
service
for
export,
basically
by
creating
a
service
export
with
the
name
and
namespace
of
a
service.
A
A
So
what
does
the
mcs
controller
do
with
these
service
exports
to
synchronize
the
information?
The
mcs
controller
makes
service
import
objects
transporting
the
metadata
of
the
exported
service
and
to
each
namespace
same
cluster
in
the
cluster
set.
It
includes
information
like
the
type
of
service
and
the
ports
and
protocols
of
the
originating
service.
A
This
information
is
used
to
create
endpoint
slices,
which
is
part
of
the
kubernetes
api.
So
we
are
back
in
core
api
territory,
which
we
left
behind
of
the
service
stage.
These
are
all
resources
that
the
mcs
controller
makes
for
you
as
part
of
its
role
synchronizing
the
information
across
this
cluster
set.
A
So
together,
it's
like
this.
A
service
in
one
cluster
can
be
marked
for
export
by
creating
a
matching
service
exports
in
that
cluster,
the
mcs
controller
will
watch
those
service
exports
and
when
it
sees
one
we'll
create
the
corresponding
service
import
in
all
clusters
in
the
cluster
set
that
have
the
same
name
space.
A
The
mts
controller
will
also
create
the
endpoint
slices
in
each
consuming
cluster
as
well
as
you
can
see.
The
mcs
controller
is
acting
like
a
cargo
ship
transporting
the
information
from
a
service
export
into
new
service
imports
in
all
ports
of
call.
We
hope
you'll
agree.
The
mcs
doesn't
add
much
to
achieve
its
goal
from
kubernetes
native
services
on
the
left
back
to
kubernetes
native
endpoint
slices
on
the
right.
B
B
Multi-Cluster
services
come
with
a
multi-cluster
dns.
This
resolves
a
and
srb
queries
for
the
dot
local
domain,
so
that
consuming
applications
can
determine
what
ip
addresses
to
use
to
connect
to
multi-cluster
services.
Simply
by
changing
your
classic.cluster.local
to
clusterset.local,
there
is
a
detailed
spec
with
all
the
different
dns
forms
required,
but
for
mcs
consumers
there's
two
main
kinds.
We
expect
people
to
interact
with
which
we
put
on
the
slide
here
so
for
cluster
set
ip
services,
there's
one
main
name:
the
service
name,
followed
by
the
name
space
and
then
that.sbc.clusterset.local.
B
This
is
a
variant
on
the
cluster
local
service
name,
continuing
the
practice
of
extending
the
standard
core
behaviors
for
headless
services,
each
pod
hosting
the
service
in
each
cluster.
Exposing
the
service
gets
a
dns
entry
using
its
pod
name
and
a
cluster
identifier
which
we'll
revisit
at
the
end
of
this
presentation.
B
A
A
A
A
The
situation
is
a
bit
more
complex
if
any
of
the
clusters
have
overlapping
siders.
In
this
case,
submariner
sets
up
an
overlay
network
to
end
up
in
a
scenario
close
to
the
simple
case.
Each
pod
and
service
ends
up
with
a
unique
address:
port
combination
in
the
cluster
set.
We
add
ports
to
the
mix
to
reduce
the
ip
address
base
requirements.
A
Now
that
we
have
connectivity,
we
can
start
sharing
services
following
the
model
we
described
earlier
in
submariner.
This
is
handled
by
a
component
called
lighthouse
inside
the
cluster,
where
the
shared
service
is
hosted.
The
lighthouse
agent
watches
for
service
exports.
When
a
service
export
is
created,
it
produces
the
corresponding
service,
import
and
stores
it
in
the
broker.
A
A
The
corresponding
service
can
be
resolved
using
lighthouse's
dns
server,
submariner
configures,
each
cluster's
dns
resolver
to
forward
mcs
queries
in
the
clusterset.local
domain
to
the
lighthouse
dns
resolver.
This
checks
for
service
imports
and
returns
the
appropriate
ip
addresses
those
addresses
are
accessible
directly
thanks
to
the
connectivity
provided
by
submariner,
all
right
we're
going
to
switch
over
to
some
demos.
Now,
I'm
going
to
let
laura
share
her
screen
because
she's
going
to
show
us
how
this
works
on
gte.
First.
B
So
first
let
me
mention
some
of
the
setup,
so
I
have
a
terminal
running
over
here
and
I'm
going
to
run
this
command
to
show
you
that
I
have
these
two
clusters:
gk1
and
gk2
they're
there
gk1,
gkd
and
they're
also
both
registered
in
the
same
cluster
set.
So
how
we
do
that
in
gke
is
by
being
members
of
the
same
gk
hub.
So
that's
our
version
of
a
cluster
registry.
We
can
check
this
out
with
this
g
cloud
command
g
cloud
container
hub
memberships
list,
so
there
they
both
are.
B
This
is
the
output
that
it
looks
like,
but
this
is
that
they're
both
they
both
have
membership
the
same
clusters
yet
and
then
the
last
thing
I
want
to
point
out
here
for
the
setup
is
that
they
both
have
the
same
namespace
cubecon.
So
the
setup
is
two
clusters
they're
in
the
same
cluster
set
and
they
have
the
same
namespace
coupon.
B
So
let's
look
over
here
at
this
yaml
that
I've
got.
This
is
the
gamble
we're
working
with
so
on.
The
left
is
the
service
object.
I
already
deployed
this
in
my
cluster,
along
with
the
deployment
backing
it,
and
the
important
part
here
is
that
we
have
this
namespace
kubecon
and
this
name.
Where
am
I
and
later
in
this
demo,
I'm
going
to
deploy
this
service
export,
and
this
is
the
one
with
this
exact
same
name:
space,
kubecon
and
name.
Where
am
I
so?
B
That's
the
whole
opt-in
process
for
you
as
a
user
which
I'm
going
to
demonstrate.
If
you
want
to
make
sure
that
the
service
gets
exported
across
your
cluster
set,
then
you
create
a
service
export
that
matches
its
name
and
namespace
exactly
so.
Let's
start
here
with
my
two
clusters:
I've
labeled
my
islands
here
gk1,
gk2
and
gk1-
is
one
where
I
have
the
service
deployed
on
it,
but
I
haven't
made
that
service
export.
Yet
so
I
cannot
access
anything
from
2k1
the
service
from
gk1
from
gk2.
B
Nothing
has
been
connected
yet
and
I'm
going
to
prove
that
over
here
I
have
another
pod
running
in
gk2,
which
I
have
a
shell
open
in
this
terminal
window
here
and
I'm
going
to
try
and
curl
the
multi-cluster
dns
name
for
the
service
over
in
gk1,
but
we're
going
to
get
that
response
here.
In
fact,
it
doesn't
even
know
what
host
we're
talking
about,
because
there's
no
multicultural
genus
has
been
set
up
right.
So
there's
no
connectivity
set
up
between
the
two
of
them
yet.
B
But
what
we're
going
to
do
here
back
over
in
this
view
is
we
are
going
to
create
the
surface
export
now,
and
this
is
going
to
kick
off
the
whole
process.
That's
going
to
make
the
service
import
and
point
slices,
and
everything
else
happen
in
gk2,
so
I'll
just
cut
that
out
real
quick.
So
you
can,
you
know
just
double
check
we're
talking
about
this
kubecon.
Where
am
I
one?
B
B
So,
let's
check
out
what
that
service
import
looks
like
so
over
in
gk2.
I've
got
a
command
here
where
I'm
going
to
look
in
gk2
for
service
imports
in
this
name
space.
So
this
can
take
a
second
to
get
set
up
right.
Our
ferry
is
still
chugging
along
across
the
ocean
here,
so
we're
gonna
just
check
a
couple
times
until
it
shows
up
and
if
necessary,
use
some
movie
magic
to
speed
this
up.
B
Okay,
great
there,
it
is
right,
let
me
clear
this
out,
so
we
can
see
it
beautifully
all
right.
So
here
is
what
the
service
import
looks
like
right.
So
we
can
see
there's
some
information
here
from
the
originating
service
so
like
what
ports,
it's
exposing,
what
type
it
is,
etc,
and
then
we
also
have
this
the
spec.ip.
B
B
So
we
can
check
to
make
sure
that
dns
has
gotten
programmed
too
and
just
generally
approve
the
connectivity
between
the
two
of
these
over
in
our
dns
window.
So
actually
I'm
going
to
take
this
vip
with
me
and
first
we're
going
to
curl
this
bit
right.
So
we
get
a
response
back
and
the
response
for
this
application
that
I
have
deployed
in
gk1
conveniently
tells
us
the
cluster
name
that
it's
responding
from
right.
B
So
we
are
successfully
getting
this
response
back
from
gk1,
even
though
again,
this
pod,
that
I'm
curling
from
is
in
gk2
and
then
to
see
that
dns
is
working
as
well.
We
can
curl
this.
Where
am
I.q.svc.clusterset.local
see
it's
the
same
thing
we're
getting
the
backend
from
gk1,
even
though
I'm
making
this
request
from
gpa
too.
So
this
is
the
fun
part
right
everything's
connected
together.
The
dns
is
really
easy
to
switch
out.
B
We
just
put
this
like
set
at
the
end
cluster
set.local
instead
of
cluster,
so
that's
kind
of
the
basic
setup,
but
I'm
also
going
to
export
the
service
from
gk2
as
well,
so
that
we
can
see
a
little
bit
more
going
on.
So
I'm
going
to
go
ahead
and
apply
some
yaml
that
has
my
actual
application
in
here
into
gk2.
B
So
I'm
creating
this
service
and
this
deployment
right
the
same
one
that
I
had
put
in
gk1
and
if
I
just
left
it
as
that,
then
it's
just
a
cluster
local
service,
but
if
I
also
create
a
service
export
that
matches
it
right,
so
this
thing
the
same
one.
With
the
same,
we
can
even
double
check
it,
one
more
time
that
has
this
namespace
cubecon,
and
this
name.
Where
am
I,
which
is
going
to
match
our
service,
that
we
just
made
gk2,
then
this
is
going
to
take
information.
B
The
other
way
right,
so
we're
going
to
be
taking
getting
our
ferry
to
take
some
metadata
from
gk2,
bring
it
over
to
gk1.
Make
service
imports
over
there
make
endpoints
over
there
technically
it
will
actually
make
this
information
available
in
gk2
as
well.
So
we'll
actually
take
a
look
at
that
in
a
second.
B
B
So
I'm
going
to
kind
of
explain
the
details
and
maybe
talk
a
little
slower
too,
but
what
we're
looking
at
over
here
in
the
terminal
is
the
endpoint
slices
inside
gk2
and
you'll,
see
for
this
cube
vending
space
and
you'll
see
that
there's
two
instances
here
this
first
one
here
is
the
cluster
local
service.
I
just
deployed
the
service
in
gk2
right.
So
that's
this
is
the
endpoint
slices
for
that.
B
But
then
this
77
seconds
ago
right,
but
this
one
here-
is
our
multi-cluster
service.
So
this
is
more
of
an
implementation
detail
on
gke,
but
the
way
we
actually
hold
multi-cluster
service
endpoints
is
by
making
a
new
service
object
and
then
that
also
creates
new
endpoints
or
endpoint
slices
in
the
consuming
cluster.
So
that's
what
this
sort
of
prefix
here
gk-mcs.
B
So
the
reason
this
takes
a
second,
oh,
it
already
happened
here
we
go
sorry.
I've
talked
too
slow.
Maybe
so
now
we
have
the
back
ends
for
both
of
them
in
this
endpoint
object
right
I'll
make
it
a
little
bit
more
clear,
clear
this
out
and
take
off
my
watch
just
so.
We
can
see
the
full
current
state
right,
so
we
have
the
cluster
local
service.
That's
always
going
to
exist
here
as
long
as
they
deploy
the
service
in
this
cluster,
and
then
here
is
the
multi-cluster
service
that
now
knows
both
endpoints.
B
So
practically
what
this
means
is,
if
I
go
back
over
here
to
my
dns
world,
if
I
curl
this
dns
name,
I'm
actually
going
to
load
balance
in
front
of
both
gk1
and
gk2,
so
you'll
see
sometimes
I'm
getting
gta2
back.
Sometimes
I'm
getting
gk
one
back,
even
though
I'm
using
the
same
dns
name
right.
We
can
do
this
a
couple
more
times
more
gk1
now
gk2
now
gk1
right.
So
because
these
services
are
considered
the
same
to
us,
they
have
the
same
name
and
name
space
deployed
across
the
cluster.
B
Then
they
can
be
load
balanced
across
just
like
your
cluster
ip
services.
But
in
this
case
we
call
them
cluster
7ip.
So
there
you
have
it.
That's
your
basic
cluster
ip
type
service
in
a
multi-clustered
world
and
I'm
going
to
hand
it
over
to
steven
to
show
us
how
this
works
in
submariner
and
also
with
a
headless
service.
Instead.
A
So
let's
start
by
taking
a
look
at
the
services
in
the
kubecon
namespace
on
cluster2
and
there
we
can
see.
Where
am
I
and
it
is
a
headless
service,
we
can
look
up
its
ip
address
from
inside
the
cluster
and
there
we
have
two
addresses
for
it
and
we
can
contact
one
either
one
to
see
its
output,
and
so,
if
I
repeat
that,
eventually
there
I've
got
the
two
different
ip
addresses,
10.2.96
and
10.2.128.
A
A
A
A
But
you
might
just
have
to
trust
me
on
that.
One
now
remember:
this
is
a
headless
service
and
without
cheating,
by
looking
at
the
cluster
2
pods
in
the
cluster
2
cluster
directly,
we
can
find
the
endpoints
by
looking
up
the
srv
records.
So
this
is
from
cluster
one
again,
I'm
going
to
look
for
the
srv
records
and
we
get
back
the
two
different
host
names
here
for
the
two
pods
and
we
also
get
back
to
port
number.
A
A
A
We
have
two.
This
shows
how
submariner
shares
the
data.
The
entries
in
the
submariner
operator.
Namespace,
are
the
cluster
local
copies
and
the
entry
in
the
submariner
case
broker
namespace
is
the
broker
copy,
so
that's
the
shared
copy,
that's
copied
everywhere,
and
the
imports
are
used
to
produce
endpoint
slices.
A
So
let's
take
a
look
at
those
on
cluster
one
this
time,
so
we
have
just
the
one
endpoint
slice
corresponding
to
the
remote
endpoints
for
the
cluster
sets
and
on
cluster
two
we
get
both
we
get
two
back
one:
that's
the
cluster
set
endpoint
slice
and
the
other
one.
That's
the
cluster
local
and
to
finish
off.
Let's
take
a
look
at
service
availability,
I'll
set
up
where
I'm
on
cluster
one.
A
A
So
if
I'm
a
query
from
cluster
one,
I'm
still
getting
back
and
a
response
from
cluster
two
and
now
I've
got
one
back
from
cluster
one
cluster.
Two
again
and
if
I
drop
the
cluster
two
deployment,
the
endpoints
are
immediately
updated
and
so
I'll
do
a
combined
command
that
deletes
and
queries
the
srv
records
again
just
to
show
you
that
it's
actually
immediate,
that's
it
dropped
and
we're
back
down
to
just
two
entries
for
the
srv
records.
A
B
B
So,
as
we've
alluded
to
in
this
presentation,
there's
parts
of
the
mcs
api
that
each
individual
implementation
is
not
fully
supporting
yet
so,
as
the
standard
becomes
closer
and
closer
to
its
final
form.
Ga
each
implementation
is
working
to
converge
towards
the
specification
so
that
it
can
be
considered
fully
mcs
api
compliant
for
gka.
For
example,
our
implementation
of
multi-cluster
headless
services
doesn't
yet
create
endpoint
slices
in
the
consuming
clusters
and
doesn't
provide
srv
records,
yet
either
for
submariner
cluster
set
ip
services
aren't
allocated
as
stable
virtual
ip.
B
That
clients
could
actually
use
everything
is
handled
through
dns
instead.
But
I
point
this
out
because
this
is
one
of
the
interesting
things
about
working
on
the
mcs
api
right
now.
The
spec
and
the
real
world
implementations
are
developing
together.
So,
as
sig
multi-cluster
debates
points
to
include
in
the
spec,
we
can
try
them
out
in
the
real
world
and
come
back
and
provide
insights.
B
A
A
A
Are
you
ready
to
try
it
out?
We
invite
you
to
install
this
yourself
either
using
openshift
or
gke
as
demoed
today
or
installing
the
crds
and
submariner.io
in
your
own
clusters.
Submariner
has
a
series
of
start
guides
on
how
to
deploy
this
mcs
controller
yourself
using
clusters
running
on
kind,
k3s,
jke
or
rancher.
B
Finally,
sig
multicluster
would
love
for
you
all
to
get
involved
in
the
development
of
the
mcs
api
and
all
the
other
good
multi-cluster
projects
we
have
going
on.
We
meet
bi-weekly
on
tuesdays
over
zoom,
and
there
was
also
a
maintainer
track
talk
published
a
few
days
ago
with
even
more
details
about
all
the
projects
we're
working
on.
So
hopefully
we
can
see
you
there.