►
From YouTube: Experience Report: Running a Distributed System Across Kubernetes Clusters - Alex Robinson
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Experience Report: Running a Distributed System Across Kubernetes Clusters - Alex Robinson, Cockroach Labs
Kubernetes makes it easy to run distributed applications, even those that manage persistent state, within the confines of a single cluster. Running the same applications in a multi-region or multi-cloud fashion across multiple Kubernetes clusters, however, is considerably more difficult due to the networking and service discovery problems involved. In this talk, Alex will walk through his team’s experience over the last six months of running a distributed database across Kubernetes clusters in different regions and their attempts to make the process repeatable on different cloud providers and on-prem environments. He’ll cover common problems they encountered, solutions they’ve tried, how they’re running things today, and the future improvements he’s most excited about from community projects like Istio.
To Learn More: https://sched.co/GrWK
A
A
All
these
great
features
have
been
added
for
managing
disks
for
managing
stateful
applications
for
running
a
cluster
that
spans
multiple
zones
or
for
running
your
kubernetes
cluster.
For
you,
there
are
also
been
improvements
in
packaging,
your
applications,
so
that
other
people
can
install
them
on
their
clusters
with
things
like
home,
charts
and
operators,
but
not
much
progress
has
been
made
in
that
same
time
in
making
it
easy
to
run
applications
that
span
across
multiple
kubernetes
clusters.
A
So
it's
very
easy
to
look
at
the
docs
and
figure
out
how
to
run
something
in
one
cluster,
it's
very
hard
to
look
at
the
kubernetes
Docs
and
figure
out
how
to
run
an
application
that
has
some
nodes
in
one
cluster
over
here
and
some
nodes
in
one
cluster
over
there.
It's
a
much
tougher
problem
that
is
not
really
written
down
anywhere.
It's
not
clear
where
to
get
started.
The
same
is
true.
A
If
you're
trying
to
run
across
multiple
clouds,
you
know
it's
very
tough,
to
figure
out
how
to
run
an
application
that
spans
from
one
cloud
to
another
or
from
one
cloud
to
your
on-prem
environment,
and
so
there
have
been
some
efforts
over
time.
There
was
the
uber
Nettie's
project
a
couple
years
ago
that
didn't
really
seem
to
go
anywhere
and
there
was
the
Federation
v2
project.
That's
still
ongoing
that
we
have
high
hopes,
for
that
should
make
some
of
this
easier.
A
A
A
You
might
also
run
across
multiple
clusters
in
multiple
regions
for
fault
tolerance.
So
if
a
single
region
goes
down,
you
don't
want
your
entire
application
to
go
down.
You
might
also
do
this
for
bureaucratic
reasons.
So,
if
you
were,
you
know,
signed
up
and
got
your
badge
this
morning,
you
probably
notice
the
gdpr
signs.
There
are
laws
springing
up
in
countries
around
the
world
that
require
you
to
store
their
users
data
in
certain
places,
often
in
their
own
countries.
A
It's
tough
to
do
this
if
you're,
storing
all
of
your
data
in
one
place
that
isn't
that
country
and
so
I
would
do
want
to
talk
today
about
this
experience
of
running
an
application.
This
bans
multiple
clusters.
Why
it's
hard?
What
sort
of
things
you
need
to
know
about
kubernetes
in
order
to
get
started,
and
then
what
kinds
of
solutions
there
are
out
there
for
this
problem
and
before
we
move
on
here?
Who
here
is
tried,
writing
an
application
across
multiple
clusters
before
and
who
here
is
doing
it
in
production?
A
Okay,
so
not
very
many
people
in
either
group
I
hope
that
will
be
a
much
higher
number
over
the
next
year
or
two
I
think
things
are
gonna
continue.
Improving
and
I
should
say
that
I'm
not
here
to
prescribe
a
particular
solution.
I
just
want
to
talk
about
things
that
I've
tried
things
that
I've
seen
users
try
things
that
I've
seen
run
in
production
and
what
my
experiences
have
been
with
them.
A
I've
helped
them
run
it
in
different
clouds
from
a
single
zone
to
multiple
regions
and
have
also
had
to
help
people
with
their
own
configurations,
because
they
didn't
want
to
use
one
of
ARDS.
They
made
their
own
ran
into
some
trouble,
so
I
get
to
see
different
ways
of
running
things,
even
beyond
the
ones
that
I
would
necessarily
recommend,
and
we
have
a
number
of
users
using
cockroach
TV
in
kubernetes,
both
in
a
single
cluster
and
spanning
multiple
clusters
today.
So
this
is
something
that
people
are
doing
in
production
in
yes.
A
But
the
initial
question
need
to
ask
is
why
this
is
hard
at
all.
I've
talked
to
many
people
who
have
run
a
system
in
a
single
kubernetes
cluster
and
assume
that
it's
gonna
be
very
easy
to
span
the
multiple
kubernetes
clusters,
because
it
was
easy
to
do
so
in
the
one
cluster
and
Cooper
Nettie's
is
supposed
to
make
it
easy
to
orchestrate
services
right,
but
to
beat
to
make
things
clear:
I'm
using
multi
region
and
multi
cluster,
pretty
much
as
synonyms,
because
you're
never
recommended
to
run
a
single
kubernetes
cluster
that
spans
across
multiple
regions.
A
In
the
early
days
you
weren't
even
supposed
to
do
it
across
multiple
zones
within
the
same
region.
Nowadays,
you
can
do
that
with
what's
called
a
multi
zone
cluster,
but
if
you
want
to
span
across
multiple
regions,
you're
really
going
to
want
to
use
multiple
clusters,
we
run
a
single
cluster
across
regions
at
your
own
risk.
Kubernetes
hasn't
been
designed
to
deal
with
the
latency
differences
between
nodes
and
other
complications,
but
to
get
started
with
trying
to
you
know,
set
up
this
multi
region
application
and
and
learn
how
to
configure
it
in
kubernetes.
A
You
need
to
understand,
like
what
do
these
systems
need
and
as
our
example
today
we're
gonna
be
looking
at
cockroach
DB,
just
cuz,
it's
the
one
that
I'm
most
familiar
with
it's.
What
I've
worked
the
most
with
and
there's
a
very
quick,
landslide
refresher.
It's
an
open
source,
distributed
sequel
database.
It
speaks
the
postcards
wire
protocol.
Your
application
can
talk
to
any
of
the
one
or
more
processes
in
your
cockroach
cluster
and
get
their
sequel
responses
back
internally.
A
It
also
offers
some
geographic
features
such
that
you
can
tell
it
which
locations
you
want
it
to
put
data
in
or
which
locations
you
don't
want
it
to
put
data
in,
but
the
way
that
you
run
it
is
that
you
just
take
the
single
binary
start
it
on
a
number
of
machines,
point
them
at
each
other,
such
that
they
can
find
each
other,
and
then
it
just
acts
as
a
single
homogeneous
cluster
of
nodes.
There
are
no
special
roles
to
worry
about,
so
we
don't
need
to
worry
about
that
in
our
configurations.
A
So,
if
you're
in
a
single
cluster,
you
get
all
these
things
built
into
kubernetes
and
it
works
really.
Well,
it's
been
hardened
over
the
last
couple
years,
but
if
you're
running
across
multiple
communities
clusters,
that's
when
you
start
to
lose
these
things
that
most
stateful
systems
need.
You
lose
the
ability
to
have
a
network
address
that
survives
or
restarts
that's
usable
from
everywhere
and
in
many
clusters,
you'll
lose
that
ability
to
talk
directly
from
every
node
to
every
other
node,
regardless
of
where
they're
running-
and
it
all
comes
down
to
networking.
A
Then
because
these
are
both
networking
problems.
We
need
pod
pod
communication,
even
across
clusters
and
when
applicable
across
things
like
V,
pcs
or
other
private
networks,
and
we
also
need
that
address
that
can
be
used
both
within
and
between
clusters.
You
could
optimize
this
and
say:
you're
gonna
have
a
separate
address
for
communication
within
and
a
separate
address
for
communication
between,
but
to
keep
things
simple,
we'll
just
assume.
We
want
one
address
that
works
in
both
cases.
So.
A
Kubernetes
networking
is
the
problem,
but
why
is
it
the
problem?
Since
the
beginning?
Kubernetes
has
required
a
few
things
in
order
for
an
installation
of
kubernetes
to
be
compliance,
it
has
to
ensure
that
every
pod
has
its
own
unique
IP
address
that
no
other
pod
has
that
IP
address
has
to
look
the
same
to
the
pod
itself,
as
it
does
to
all
pods
that
that
pod
is
talking
to,
and
this
has
to
work
without
any
form
of
network
address
translation.
A
This
might
seem
very
straightforward,
but
at
the
time
kubernetes
was
released.
It
was
pretty
revolutionary
very
much
against
the
current
ways
of
thinking.
It
was
different
from
how
doctor-recommended
doing
networking
it
was
different
from
how
Google
did
networking
internally
with
Borg,
and
so
there
weren't
really
many
ways
of
implementing
this.
Actually
it
was
hard
to
set
up
a
cluster
in
the
early
days,
because
there
was
no
way
of
providing
this
for
the
pods
that
you
were
running.
A
You
know
we
wanted
to
run
a
lot
of
pods
on
each
machine,
but
each
machine
in
a
traditional
network
is
only
given
one
or
two
IP
addresses,
and
so
since
then
a
ton
of
solutions
have
been
built
up
to
satisfy
these
requirements.
If
you
look
at
the
kubernetes
networking,
documentation,
page
you'll
see
more
than
20
different
solutions
listed
and
it's
not
a
complete
list.
There
are
a
lot
of
ways
that
this
gets
done
out
there
in
the
wild
and
because
the
only
requirements
that
were
specified
were
how
a
single
cluster
should
work.
A
The
multi
cluster
case
is
basically
undefined
behavior
and
it's
up
to
implementation
details
of
each
different
networking
solution,
and
so
this
is
kind
of
like
if
you're
writing
a
C
program
as
undefined
behavior
in
it,
and
it
works
on
one
compiler
and
one.
You
know
CPU
architecture,
but
then
you
try
to
compile
it
for
another
architecture,
and
you
find
that
your
program
explodes
that's
kind
of
the
situation
of
trying
to
run
a
multi
cluster
application
across
different
clusters.
A
A
There
are
a
bunch
of
software
solutions
out
there
that
do
things
in
different
ways
as
well,
and
people
have
to
figure
out
how
to
do
this
on
prim
in
a
number
of
different
ways,
often
using
some
of
the
software
solutions
off-the-shelf,
but
maybe
doing
things
in
hardware
or
in
a
custom
way
and
again,
these
all
work
differently.
So
there's
not
much!
A
So,
if
you're
running
on
gke
or
Amazon's
eks,
as
of
earlier
this
year
for
eks,
you
can
talk
directly
from
one
pod
and
one
cluster
to
a
potted
another
cluster
as
long
as
they're
in
the
same
V,
PC
or
you've
done
some
sort
of
pairing.
So
this
is
great.
You
don't
have
to
think
about
it
much
when
you're
in
these
environments,
others
don't
have
a
nun
by
default,
but
make
it
kind
of
easy
to
enable
so
edgers
default
basic
networking
mode
doesn't
support
it.
A
But
if
you
enable
their
advanced
networking
mode,
which
is
just
like
a
checkbox,
when
you
create
your
cluster,
then
you
get
it
cilium,
which
is
one
of
those
open
source
software
solutions
for
providing
kubernetes
networking.
Hence
a
mode
called
a
cluster
mesh
where
you
can
configure
it
to
enable
this,
but
it
won't
do
it
by
default,
but
for
many
other
software
solutions,
it's
pretty
difficult
to
get
this
pod
to
pod
connectivity.
A
So
this
is
Ennis
like
especially
a
problem
if
you're
running
on
Prem
clusters
or
if
you
aren't
in
Google
Amazon-
and
we
don't
have
enough
time
today
to
fully
cover
how
all
this
works.
In
fact,
we
don't
have
much
time
at
all.
So
if
you
want
to
learn
more
about
this
check
out
some
past
coop
contacts
in
particular,
I'd
call
out
the
life
of
a
packet
talk
by
Michael
Rubin
a
couple
years
ago
in
Cuba
on
Europe.
A
If
you
want
to
get
a
refresher
on
how
some
of
these
different
solutions
actually
route
packets
through
a
network-
and
so
you
know
what
can
we
do?
We
know
the
problem
is
that
networking
doesn't
provide
for
direct
communication
between
clusters
sometimes,
and
it
never
actually
gives
us
an
address-
that
we
can
reliably
use
between
clusters
and
I
tried
a
whole
bunch
of
things
out
earlier
this
year
in
the
first
half
of
the
year.
Try
again
doesn't
look,
maybe
10
different
solutions
tried
to
get
them
working,
checked
out
how
reliable
they
were.
A
When
things
failed,
I,
you
know,
did
some
thought
experiments
on
different
ways.
They
could
fail
that
I
didn't
actually
try
out,
and
unfortunately,
there
wasn't
like
a
single
great
solution.
They
all
have
a
bunch
of
problems
and
some
of
them
might
have
improved
over
the
course
of
the
year,
but
for
the
most
part,
things
are
about
the
same,
so
they
either
didn't
work
in
certain
environments
at
all
or
they're
easy
to
break.
A
If
you
are
a
kubernetes
expert
and
you
have
like
a
single
stable
environment
that
you
want
to
get
things
working
in,
your
requirements
will
be
slightly
different
from
mine,
because
I'm
trying
to
let
everyone
to
run
cockroach
across
multiple
clusters,
whereas
you
might
just
want
to
run
something
across
your
own
clusters
in
your
own.
Certain
environment
and
I'm,
also
not
here,
to
prescribe
a
particular
answer
again.
So
take
take
out
of
this.
A
So
in
this
first
solution,
you're,
basically
like
getting
around
all
the
kubernetes
magic
you're,
just
ignoring
it
you're
saying
I,
don't
want
your
scheduling!
I!
Don't
want
your
networking
just
run
me
on
these
machines
and
let
me
use
the
host
machines
network.
So
this
is
very
similar
to
just
running
processes
directly
on
VMs,
without
even
using
kubernetes,
because
you're
picking,
which
processes
are
running
on
which
machines
and
you're
letting
them
use
the
machines
Network.
So
this
works.
A
If
you
don't
have
pod
pod
connectivity
across
clusters,
because
even
in
those
cases,
you'll
usually
be
able
to
talk
from
one
machine
and
one
cluster
to
all
the
machines
and
the
other
cluster
or
you
can
least
set
that
up
pretty
easily.
So
what
you
do
is
you
either
assign
pods
directly
to
nodes
by
creating
a
replication
controller
of
size,
one
and
putting
a
node
constraint
on
it
or
you
use
a
daemon
set,
and
that
way
you
get
cockroach
running
on
all
the
nodes.
A
A
So
this
also
makes
some
people
happy
who
are
otherwise
skeptical
of
running
stateful
workloads
in
kubernetes,
because
it
removes
some
of
the
magic
that
makes
running
stateful
workloads
a
little
scarier
because
you're
not
getting
things
being
rescheduled
underneath
you,
you
can
choose
to
use
the
machines
local
disks
if
you
want,
rather
than
using
network
attached
storage,
because
you
know
the
pods
are
always
going
to
be
running
on
the
same
machine.
And
so
this
is,
this
might
sound
kind
of
silly.
A
Their
IP
addresses
aren't
changing,
they
aren't
being
cycled
in
out
of
the
cluster,
and
this
could
be
a
problem
in
certain
cloud
environments,
so
managed
service
offerings
like
gke
and
eks.
Do
node
upgrades
by
just
removing
the
VMS
and
replacing
them
with
new
ones,
and
that
means
the
IP
addresses
go
away
if
you're
using
local
disks
those
go
away,
and
so
this
is
something
you
might
not
want
to
do
in
the
cloud
unless
you've
taken
special
precautions,
but
if
you're
running
on
Prem.
A
This
is
great
because
you
don't
have
to
you
know
worry
about
connecting
the
networks
of
the
multiple
clusters
and
you
can
be
confident.
Those
machines
are
gonna,
stay
there,
because
they're
physical
machines
in
your
own
data
centers.
This
does
require
a
bit
of
manual
setup
to
get
working
and
it
can
use
up
ports
on
the
host
machines.
So
you
can't
run
like
multiple
cockroach
instances
on
the
same
machine
because
it's
using
the
hosts
ports,
but
this
is
a
surprisingly
good,
very
easy
to
configure
way
of
doing
things
if
you're
on
prim.
A
If
you
are
running
in
the
cloud,
though,
you
might
use
the
external
load
balancer
feature-
and
you
might
laugh
at
this,
but
we
have
a
couple
of
customers
doing
this
in
production
and
it's
been
working
reasonably
well
for
them.
This
is
the
most
universally
feasible
way
because
it
does
not
require
pod
to
pod
connectivity
between
the
clusters,
you're
relying
on
the
external
load
balancers
to
provide
you
networking
from
one
cluster
to
the
next.
A
So
if
you
have
like
this
to
cluster
you're
running
in
two
clusters,
with
two
pods,
each
you'll
end
up
with
these
four
load
balancers
and
each
pod
just
has
to
talk
through
the
correct
load
balancer
when
they
want
to
talk
to
a
certain
pod
and
you'll
notice.
This
does
add,
officially
add
some
extra
hops
to
the
data
path,
but
most
load
balancers
are
reasonably
efficient,
and
so
this
works.
Even
we
don't
have
pada
pada
communication
and
it
solves
the
problem
from
the
past
solution
that
it
continues
working
even
as
pods
move
around
the
cluster.
A
It
lets
you
survive
machine
failures
and
machine
replacements
more
easily
because
the
pods
can
be
rescheduled
but
they're
still
reachable
through
the
same
load
balancer.
And
this
means
you
don't
have
to
configure
anything
else
other
than
those
little
balancers
themselves,
because
the
load
balancers
addresses
never
change.
So
it's
giving
you
that
stable
persistent
network
address
that
we
need.
A
A
This
also
means
that,
whatever
you
want
to
add
new
pods,
you
want
to
scale
up.
You
have
to
first
create
that
load
balancer
for
the
new
pods.
And
if
you
forget
that,
then
your
pod
isn't
gonna
be
doing
very
much.
It's
not
going
to
be
reachable
and
again.
This
can
add
some
extra
hops
on
the
data
path
unless
you
have
a
very
sophisticated
load
balancer.
A
But
this
is
a
very,
very
good
way.
The
customers
that
have
been
using
it
haven't
run
into
you
know.
Any
production
issues
caused
by
this
set
up,
the
networking
has
remained
reliable,
pods
have
been
deleted
and
recreated
on
different
nodes,
and
things
have
continued
working
just
fine.
So
this
is
somewhat
battle-tested.
A
You
could,
if
you
have
pod
de
pod
connectivity
between
clusters,
just
use
the
pot
IPS
directly.
You
aren't
given
a
persistent
name,
so
this
only
works
if
your
piece
of
software
can
handle
having
addresses
change
underneath
them.
So
you
know
you
have
a
pod
running
it
wanted
IP
address.
They
get
stopped,
get
started
up
with
the
same
storage,
but
a
new
IP
address.
It
needs
to
be
able
to
inform
all
the
other
nodes
about
its
new
IP
address,
and
they
have
to
be
ok
with
that.
But
if
your
software
can
do
that,
then
this
works.
A
You
need
to
create
a
one
load,
balancer
up
front
such
that
they
can
find
each
other
in
the
first
place.
So
you
have
one
joint
address
that
you
provide
to
cockroach
that
it
tries
to
connect
to
when
it
starts
up.
If
it
doesn't
know
any
other
members
of
the
cluster,
but
once
they
connect
to
each
other,
they
can
talk
directly
to
each
other
using
those
IP
addresses.
If
your
network
offers
this
to
you
I,
don't
know
of
anybody
using
this
in
practice.
A
But
you
know
it
is
a
pretty
effective
way
of
doing
things.
The
big
con,
though,
is
that
those,
because
those
IP
addresses
are
changing
it's
hard
to
create
TLS
certs
that
are
valid
for
the
nodes,
because
whenever
you
create
a
sign
of
TLS
certificate,
you
have
to
sign
what
addresses
it's
valid
for,
and
those
addresses
are
going
to
be
changing
every
time
that
the
pod
moves
to
a
new
IP
address.
So
you'd
have
to
sign
a
new
certificate
for
that
IP.
A
new
IP
address
every
time.
A
A
pod
moves
and
you'd
have
to
set
up
some
sort
of
automatic
certificate
signer
to
do
this,
and
it's
pretty
questionable
from
a
security
perspective,
because
you
probably
don't
to
be
blindly
signing
certificates
that
have
root
on
your
database
cluster
all
the
time.
So
we
don't
know
if
anyone
that's
actually
doing
this
in
a
secure
way,
but
I've
run
a
number
of
secure
in
secure
clusters
like
this.
A
They
don't
use
TLS
and
it
works
pretty
well
but
say
we
want
to
run
the
secure
node
in
the
same
way
where
the
pods
talk
directly
to
each
other,
taking
advantage
of
that
pod
de
pod
connectivity
when
it's
there,
how
do
we
add
those
persistent
names?
And
you
know,
typically,
when
you
want
to
add
a
persistent
name
to
something
whose
IP
is
changing,
you
use
dns,
that's
the
natural
solution.
A
That's
been
around
for
a
very
long
time
and
the
way
that
DNS
works
in
a
single
kubernetes
cluster
is
that
a
pod
tries
to
talk
to
some
service.
In
this
case,
pod
zero
is
trying
to
talk
to
the
pod.
One
service
cube
DNS,
knows
about
all
the
services
and
end
points
in
the
cluster,
because
it's
been
watching
the
cube
the
kubernetes
api
server
will,
then
you
know,
return
the
IP
address
of
that
service.
If
there
is
one
and
for
stateful
sets
it
actually
return
can
return.
A
You
put
in
arbitrary
plugins
into
how
it
works
and
there's
already
an
existing
plug-in
called
kubernetes
that
was
written
to
do
exactly
what
we
want.
You
can
configure
it
to
the
core
DNS
instance,
to
connect
to
multiple
kubernetes
API
servers
from
different
clusters.
It
can
then
do
dns
the
serve
dns
lookups
from
all
of
those
different
clusters.
However,
it
turns
out
that
actually
doing
this
in
a
managed
communities
offering
is
next
to
impossible,
I
couldn't
figure
out
how
to
do
it,
how
to
replace
cube
DNS
with
core
DNS,
because
the
built-in
add-on
controller
ensures
cube.
A
Dns
is
always
running
and
does
not
like
you
trying
to
change
it,
and
you
could
try
to
reconfigure
the
DNS
settings
on
every
node
in
the
cluster.
You
know
to
talk
to
a
different
DNS.
You
know
server,
but
then
you
have
to
be
constantly
updating
that
and
users
clusters
every
time
the
nodes
we
start
or
are
recreated,
which
might
work
if
you
control
all
the
nodes.
It's
tough
for
me
to
like
help
other
people
with
if
they're
running
in
the
cloud.
A
You
also
would
probably
want
to
give
each
cluster
a
unique
clustered
domain
and
DNS.
So
if
you've
ever
looked
at
the
full,
you
know
DNS
name
of
a
service
and
communities
you'll
see
that
it
ends
and
SVC
dot,
cluster
dot
local.
You
can
actually
customize
that,
but
you
have
to
be
able
to
change
the
flags
being
passed
to
kubernetes
components
on
all
the
nodes
in
order
to
do
so,
which
is
again
very
hard
to
do
and
a
managed
service
offering
because
they
control
all
that
they're
gonna
recreate
the
nodes.
A
Underneath
you
it's
tough
to
do
yourself.
This
might
become
easier
now
that
Cordia
core
DMS
is
the
standard,
but
it
still
doesn't
give
you
a
way
to
modify
the
cluster
domain
and
also
the
default
version
of
core
DNS.
That
shipping
in
these
new
clusters
doesn't
include
that
cout
brunette
I
plug-in
that
you
would
want
to
be
using
so
you'd
still
have
to
try
to
replace
it.
So
this
is
a
bit
of
a
non-starter
in
a
cloud
provider,
but
if
you're
running
your
own
cluster,
this
would
be
a
great
way
to
do
it.
A
Another
variant
of
that
is
that
you
could
try
running
core
DNS
alongside
cubed
us
tell
cube
dienes
to
defer
the
lookups
for
for
certain
suffixes
or
maybe
even
for
all
suffixes
to
core
dns
and
then
you'd
have
to
configure
core
dns
to
watch
all
the
other
communities.
Api
servers,
and
you
know,
then
it
could
serve
the
responses
from
multiple
clusters.
I
haven't
actually
tried
this
on
some
a
little
fuzzy
on
the
details,
but
I
think
it
would
work
out
pretty
well.
If
someone
tries
it.
Let
me
know
afterwards
you.
A
The
only
downside
of
this
is
that
you
do
have
to
copy
the
API
server
credentials
for
all
of
your
clusters
into
all
of
your
other
clusters,
because
core
DNS
isn't
able
to
you
know,
get
the
resources
from
the
other
clusters
unless
it
has
credentials
to
do
so
so
you're,
putting
all
your
all
your
cluster
credentials
into
each
one,
which
means
that
if
someone,
you
know
breaks
into
one
of
your
clusters,
they
could
potentially
get
the
credentials
to
access
all
of
your
other
ones.
If
you're,
you
know
smart
and
only
give
like
read-only
credentials.
A
So
sorry,
but
the
end
and
us
east
1.53
that
local
to
go
to
the
US
East
one
clusters,
DNS
service
and
I've,
put
together
some
scripts
for
people
to
try
this
out
on
GAE,
and
so
the
sub
domain
configuration
is
very
small.
You
just
give
it
a
suffix,
and
then
you
give
it
an
IP
address
that
you
want
all
DNS
lookups
for
that
suffix
to
be
redirected
to.
A
And
so
again,
this
is
how
a
single
cluster
DNS
lookup
works
in
this
solution.
If
the
DNS
lookup
is
for
something
with
one
of
those
other
suffixes,
the
local
cubed
DNS
will
act
as
a
recursive
resolver
and
forward
that
DNS
request
along
to
the
other
cubed
DNS
server,
which
will
then
hopefully
have
the
response
for
you
and
propagate
it
back
to
you.
So
in
this
case,
pod
0
in
the
first
cluster
would
get
the
IP
address
for
pod
1
in
the
second
cluster
and
then
be
able
to
talk
directly
to
it.
A
Assuming
you
still
have
pod
2
pod
connectivity,
so
this
is
really
great,
except
for
one
big
con.
So
it
has
no
overhead
on
the
data
path,
except
for
the
normal
overhead
of
using
docker
versus
host
networking,
it's
very
resilient
to
hosts
being
added
and
removed
or
pods
being
moved
around.
You
don't
need
to
add
any
extra
software
to
your
cluster
and
it's
actually
completely
scriptable.
A
A
You
can
use
a
cloud
load
balancer
for
this
and
then
you're
sort
of
deferring
the
problem
such
that
only
DNS
lookups
have
to
go
through
one
of
these
load
balancers
and
you
don't
need
as
many
of
them,
and
so
you
can
do
that
on
GCE.
That
doesn't
work
on
Amazon,
though,
because
Amazon's
load
balancers,
give
you
at
least
their
standard
elby's.
Give
you
a
and
il
B's.
There's
a
mikvah
load.
Balancers
give
you
like
a
DNS
address
rather
than
an
IP
address,
and
you
can't
give
cube
DNS
a
DNS
name
for
these
stub
resolver
'z.
A
A
They
just
assume
that
it
exists
and
they're
trying
to
solve
the
naming
issue
and
all
the
other
things
that
they
saw
beyond
naming
I
mean
they
do
much
more
than
just
that,
and
the
way
that
it
works
is
that
you
install
sto
in
one
primary
kubernetes
cluster,
all
of
the
sto
control
plan,
components
of
which
there
are
many,
and
then
you
install
the
special
sto
remote
component
in
your
other
kubernetes
clusters.
You
might
notice
already
that
this
means
that,
if
that
primary
cluster
goes
down,
the
remote
clusters
no
longer
have
a
leader
I.
A
Maybe
this
has
changed,
but
last
time
I
looked.
This
was
still
the
case,
and
this
does
look
like
it's
going
to
be
the
best
way
to
do
things
as
it
continues
to
develop.
You
know:
it'll
be
a
well
supported
way
of
doing
this.
There's
not
going
to
be
much
overhead
of
a
data
path
and
it
will
be
very
resilient
to
changes.
As
of
last
time,
I've
tried
this,
though
you
were
still
running
the
entire
control
plane
in
a
single
cluster.
A
The
setup
process
was
very
involved
and
even
after
you
went
through
their
long
docks
page
and
how
to
set
it
up,
they
still
say
this
might
need
more
work
if
you're
using
it
in
production
environments,
because
there
are
some
concerns
around
actually
solving
this
naming
problem
for
the
stl
components
themselves.
So,
just
like
we
had
to
put
in
a
load
balancer
for
DNS
in
the
fast
solution
you
might
have
to
do
that
for
hiss.
A
A
This
isn't
a
great
idea
for
for
cockroach
in
particular,
but
if
you're
running
your
own
stuff,
you
can
always
do
your
own
thing,
so
you
know
set
up
your
own
custom.
Dns
servers
outside
of
kubernetes.
You
know-
maybe
you
could
use
like
console
for
service
discovery,
to
do
this
outside
of
kubernetes
or
set
up
your
own
Auto
certificate
approver,
so
that
you
can
create
a
new
certificate
every
time
a
pod
changes,
its
IP
address.
A
Do
all
sorts
of
different
things
on
your
own
run
your
own
proxy,
to
provide
this
pod
to
pod
connectivity,
but
it's
tough
for
us
to
widely
recommend
any
of
these
in
particular,
and
so
I
haven't
really
explored
this
space
that
much
and
so
to
summarize,
multi
cluster
networking
is
still
very
complicated.
There's
not
like
a
clear
solution
and
because
it's
been
left,
someone
specified
it's
really
tough
to
know
what
you're
gonna
get
out
of
any
given
kubernetes
installation.
A
It's
not
going
to
be
the
same
from
one
installation
to
another,
and
that
much
has
been
done
about
it
until
this
year,
when
a
few,
a
few
services
have
started
paying
attention
to
it,
which
I'm
really
excited
about
so
sto
has
been
working
on
it.
The
cilium
networking
project
that
I
talked
about
has
their
cluster
mesh
mode
that
they
implemented.
There's
the
up
bound
up.
A
Brown
start
ups
cross
plane
project,
which
was
announced
recently
they're
working
on
multi
cluster
and
multi
cloud
use
cases,
and
so
I'm
really
excited
to
see
this
start
to
get
more
attention.
But
it
is
hard
to
recommend
a
single
answer
for
everyone,
so
you
really
need
to
understand
kubernetes
well
and
understand
the
different
options
if
you
want
to
implement
this
yourself
as
of
today.
A
So
thank
you
so
much
for
coming.
If
you
do
want
to
see
more
information
on
this
or
try
it
for
yourself,
we
have
documentation
about
it
on
our
website.
We
have
all
the
scripts
and
config
files
and
our
repo,
if
you
want
to
see
it
or
you're,
always
welcome
to
reach
out
to
me
if
you've
tried
this
or
you've
had
any
questions
and
I
will
be
hanging
around
F
to
talk.
If
you
have
anything
you
want
to
talk
about
today,
do
we
have
any
time
for
questions.