►
From YouTube: Solving Multi-Cluster Network Connectivity With Submariner - Chris Kim & Miguel Angel Ajo
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Solving Multi-Cluster Network Connectivity With Submariner - Chris Kim, Rancher Labs & Miguel Angel Ajo, Red Hat
Today companies face the need to spread workloads across several clusters, leverage the capabilities of specific clouds, create more resilient services, or comply with GDPR by locating and securing specific data on specific geographic locations, but while that is possible today still communication between clusters is not solved in a transparent and secure way. Currently the pods and services in two different kubernetes clusters are isolated from each other, network plugins in kubernetes don't have a common for way creating such connectivity. To connect two services in separate clusters the administrator needs to make the application endpoints public, some network plugins provide partial solutions to this problem and service meshes like Istio solve this and more at the cost of some level complexity and overhead. https://submariner.io solves this problem connecting clusters at IP level
https://sched.co/UadI
A
All
right,
thank
you,
so
we're
gonna
get
started
here.
So
this
is
my
depiction
of
a
kubernetes
operator.
Now
you
might
be
asking
why
he's
frowning
and
I
have
I
have
the
answer
for
you.
So
he's
got
three
kubernetes
clusters
running
in
different
places.
Then
we
can
imagine
that
these
are
different
clouds
and
one
of
them's
in
open,
Jeff
cluster,
because
you
know
some
of
his
devs
when
an
open
shift.
You
know
he's
got
rancher
with
kubernetes
and
he's
got
this
like
weird
skunk
works
came
that
went
off
and
did
their
own
qubit.
A
You
know
cube
ADM
clustered
and
then
he
ended
up
having
to
manage
all
of
these
different
clusters
right.
So
the
problem
is
that,
as
he's
getting
towards,
you
know
his
deployments
and
trying
to
strategize
on
the
way
that
he's
going
to
perform
application
deployments
across
these
clouds
there.
He
doesn't
really
have
that
connectivity
between
his
clusters.
You
know
pods
can't
talk
to
pods.
Unless
you
know
you
find
different
ways
of
transversing
that
traffic,
you
know
using
something
like
an
external
IP
or
load
balancers
I
mean
so
that's
just
not
fun.
A
So
that's
what
submariner
is
really
here
to
do.
It's
a
multi
I
guess
a
multi
cluster
kubernetes
network
connectivity
tool,
and
so
because
of
that
it
makes
our
kubernetes
operator
happy,
and
this
is
I
know
it's
a
pretty
bad.
You
know
graphical
depiction
of
what
this
does,
but
you
know
we
can
actually
talk
about.
You
know
using
text
here.
What
submariner
does
so
submariner
really
is
gonna
allow
your
pods
to
directly
communicate
between
your
kubernetes
clusters.
It's
secure,
so
it's
actually
using
IP.
Well.
A
If
the
current
implementation
today
uses
IPSec
tunnels,
so
it
actually
will
establish
IPSec
tunnels
between
your
kubernetes
clusters.
It's
also
C&I
agnostic,
so
that
is
it's
actually
operating
at
a
layer
independent
of
most
network
providers.
So
you
know
whether
you're
using
something
like
flannel
or
OB
and
kubernetes.
You
actually
can
have
your
pods
talk
to
each
other
and
what
we'll
be
demoing
today
is
actually
an
open
chef
cluster
talking
to
a
kubernetes
or
you
know,
a
rancher
kubernetes
cluster
using
flannels
or
you
know,
flannel
on
our
end
and
then
OpenShift
is
using
I.
Think
it's.
B
A
So
you
know
it's
it's
pretty
neat
stuff.
Now.
How
does
it
work?
I?
Get
this
question
a
lot,
because
you
know
the
idea
of
trying
to
connect
these
over
networks
together.
You
know
there's
a
lot
of
different
sort
of
architectures.
You
could
come
up
with,
but
this
is
the
simplest
one
we
have
so
essentially
we
have
a
central
component
which
we
could
we
can
consider
our
submarine
or
broker
and
the
in
the
current
implementation.
A
Today
it's
actually
a
kubernetes
cluster
that
is
using
CRTs
and
we
have
each
cluster
that's
going
to
be
connected
to
each
other.
I
guess
interacting
with
that
broker
cluster
and
exchanging
information.
Really
it
is
metadata
between
the
data
path
itself
is
going
to
be
using
IPSec
in
each
cluster.
You
actually
have
one
or
many
Gateway
nodes
and
submariner.
Does
leader
election
to
determine
which
gateway
node
is
active
and
will
actually
establish
the
IDS
like
tunnels
between
the
Gateway
nodes,
which
is
why
that
graphic,
there
kind
of
shows.
A
You
know
the
the
overlap
there
and
with
all
of
the
other
worker
nodes
master
at
CD
and
so
forth.
If
they're
running
the
submariner
route,
agent
will
actually
have
the
excellent
devices
configured
on
them,
independent
of
what
you
would
have
for
flannel,
but
that's
actually
how,
within
the
cluster
we
we
send
traffic,
you
know
for
the
remote
I
guess
remote
clusters,
when
you're
trying
to
cross
cross
communicate
so
a
little
bit
of
history
here.
A
So
the
history
of
submariner
I
was
first
conceptualized
in
2017
when
I
was
sort
of
running
openshift
and
had
a
lot
of
questions
come
up
around.
How
do
I
get
you
know,
clusters
you
know
on
one
side
of
the
country
to
talk
to
clusters
on
the
other
side
of
country,
and
that
was
there
was
no
good
answer
at
the
time
I
mean
you
could
do
things
like.
You
know,
layer
for
load,
balancers
or
you
could
try
to
figure
out
how
to
tunnel
your
traffic
there.
A
You
know
ingress
or
your
routes
or
whatever
just
there
was
no
good
solution
for
this.
So
I
kind
of
had
this,
like
this
dumb
idea
of
trying
to
connect
these
clusters
using
IPSec
tunnels,
and
so
the
first
prototypes
were
written
and
actually
late
2018
when
I
was
basically
writing,
bash
scripts,
dropping
them
into
containers
and
then
letting
the
you
know,
letting
it
be
or
letting
strongswan
establish,
IPSec
tunnels
between
them
and
then
I
just
it
was.
It
was
pretty
bit.
You
know
bad.
It
had
hosts
routing
and
stuff
like
that.
A
One
of
the
big
things
that
we
had
problems
with
was
the
fact
that
we
needed
to
have
layer,
two
adjacency
for
the
nodes,
and
that
was
just
I
mean
how
could
you
do
cross
a
Z
if
we're
within
the
same
it
just
didn't
work,
so
we
ended
up.
You
know
basically
integrating
the
excellence
and
now
your
nodes
are
able
to
live
in
other
places
and
still
actually
send
traffic
to
remote
destined
custer's
and
the
other
big
thing
I
kind
of
wanted
to
call
out
here
is
you
know.
A
Submariner
is
a
community
project
right,
so
we're
trying
to
get
you
know,
feedback
from
people
who
want
to
have
use
cases
or
who
want
to
be?
Do
you
know
using
submariner,
for
you
know,
like
basically
few
have
a
use
case,
for
you
know,
we'd
love
to
hear
about
that
and
we'd
love
to
sort
of
you
know
if
you
want
to
sit
down
and
look
at
the
codebase
and
try
to
help
us
out
like
I
mean
we're.
Definitely
welcoming
that.
A
So
we
did
make
a
new
release,
a
summary
to
0:03,
which
has
that
be
excellent
enhancement
and
still
does
the
IPSec
or
you
know,
IPSec
for
tunneling
was
actually
released
yesterday
and
you
know
that's.
We
basically
were
like
oh
we're
IQ
con.
We
might
as
well
release
it.
So
you
know
we
definitely
would
encourage
you
to
check
it
out
after
this.
A
B
B
On
the
cr1,
we
had
not
at
some
point
in
India
in
the
data
path.
So
when
the
packets
arrived,
the
destination
cluster,
the
IP
address
of
the
originating
port
was
lost.
So
we
we
fixed
that
with
the
encapsulation
in
the
clusters
inside
the
clusters,
and
then
we
extended
testing
as
much
as
week,
and
we
are
still
doing
that.
B
So
the
the
testing
that
we
do
is
really
I
mean
we
just
all
the
all
the
possible
cases
that
we
can
think
of
connectivity,
and
we
are
working
now
on
more
complex
things
like
adding
clusters
removing
clusters
trying
to
break
up
things.
There
are
my
colleagues
who
are
working
on
a
nut
stuff
go
tell
other
like
a
lot
of
unit
testing,
and
we
we
still
have
more
to
do
so.
If
you
really
do
want
to
you
in
the
community
and
help
us
out.
B
That
will
be
great,
and
also
we
have
been
working
on
an
operator
to
to
maintain
yours.
Submariner
Diploma
in
the
in
your
cluster
make
sure
that
is
healthy,
that
you
can
upgrade
it
that
you
can
deploy
it
like
really
really
quick.
We
we
are
going
to
show
you
under
them,
I
mean
it's
amazing
and
and
with
the
operator
it
comes
to
what
we
have
called
subsidy
l
and
we'll
just
show
you
so
on
the
roadmap.
We
have
support
for
overlapping,
IP
ciders
like
it's.
B
It's
a
pretty
common
case
that
people
wants
to
connect
already
existing
in
clusters
and
and
it's
very
likely
that
you
will
have
conflicting
IP
address
ranges,
so
we
are
coming
up
with
with
a
solution
for
that.
We
are
also
working
on
service.
Discovery
occurs,
cluster,
a
network
policy
support
because
it's
more
complicated
across
clusters
supporting
other
network
plugins
like
currently
I,
think
it
at
least
it
runs
with
open,
safe
test,
the
end
flannel
with
net
yeah.
B
B
A
So,
as
Miguel
sort
of
alluded
to
earlier,
one
of
the
biggest
problems
we
run
into
a
submariner
is
regarding
overlapping
ciders.
So
if
you
have
to
kubernetes
clusters
and
I'm
just
gonna
because
I'm
most
familiar
with
us,
let's
say
you
know:
I'm
spending
on
my
kubernetes
clusters
and
I
have
my
pot
IPS
on
10
4200,
/
16.
A
What
would
be
cluster
a
and
pods
in
cluster
B?
But
that
is
obviously
still
in
proposal
at
this
at
this
point.
So
if
you
have
any
feedback
or
you
have
any
ideas
on
how
to
like
do-
you
know
I'm
sort
of
global
clusters
with
overlapping
ciders?
We
know
we
definitely,
you
know,
come
check
out
the
proposal
we
have
and
the
git
repository.
B
Okay
and
on
the
roadmap,
we
had
the
service
discovery,
and
so
for
that
we
created
sub
project,
because
we
want
to
keep
the
concerns
as
separate
as
we
can
and
that
project
is
like
house.
So
the
idea
with
like
houses
that
you
can
you
can
resolve
DNS
or
did,
or
we
are
also
looking
for
other
metals,
but
so
your
application
doesn't
really
need
to
know
where
your
service
is
running.
B
Hopefully
at
some
point
we
we
can
make
it
like,
select,
first
local
services.
If
the
service
is
running
on
on
both
and
several
clusters,
say:
okay
I
prefer
to
go
to
the
locker
room
cluster
first
and
if
that
is
not
available,
then
go
to
the
remote
clusters,
and
so
there
are
so
there
is
new.
Ip
is
on
kubernetes.
That
will
let
us
define
that
kind
of
behavior,
and
this
is
how
it
looks
early.
B
Currently,
for
the
time
we
we
are
creating
accordion
s
plugin
and
a
controller
that
will
discover
like
the
remote
services
and
create
locals
here
these
that
this
cárdenas
plugin
can
can
reply
with.
So
a
pod
that
runs
in
this
cluster
will
make
like
DNS
query
and
discover
this
remote
service
living
on
on
the
other
cluster
and
the
other
very
important
topic
that
we
need
to
look
at
is
network
policies.
If
you
are
connecting
your
clusters,
it's
really
important
that
it
that
you
take
care
of
security
and
network
policies
are
for
that.
B
The
problem
with
network
policies
is
that
the
network
plug-in
in
one
cluster
does
not
know
our
the
pots
in
the
remote
cluster,
so
it
needs
to
I
mean
we
need
to
put
some
mechanism
in
place
to
let
the
local
network
plugin
of
that
cluster
to
know
about
the
remote
bow
tie
piece
so,
and
so
this
is
the
normal
case.
No
Koska
are
just
local
traffic
and
not
all
policies
select
pots
based
on
labels
namespaces
and
this
network
policy.
For
this,
this
pot
is
selecting
pots
with
this
label.
In
this
case
it
works.
B
B
What
we
do
with
coast
guard
is
I'm
simplifying
this,
but
we
discover
the
the
remote
pot
and
we
create,
for
this
is
a
proof
of
it
works,
but
we
create
like
additional
metropolises
parallel
to
existing
ones
with
the
remote
by
a
pod
eyepiece
that
much
the
network
policy
when
toiley
that
can
change.
This
is
only
a
proposal
and
something
to
start
with
and.
B
Deployment
options:
currently
you
have,
you
have
several
you:
can
you
can
use
helm,
which
is
what
priests
created
initially
you
can
use.
The
new
Submariner
operator
is
like
a
first
release
that
does
basic
deployment
and
you
can
use
subsidy
L.
That
I
will
show
how
it
looks
because
it's
a
little
bit
complicated
to
otherwise
to
deploy,
and
we
wanted
to
make
it
very
easy.
So
we
went
from
I,
don't
know
eight
pages
document
explaining
how
to
deploy
so
Mariner
to
to
Lyons
and
I
mean
very
happy
about
that
so
using
sub
CTL.
B
This
is
how
it
looks.
You
can
choose
your
broker
cluster.
Do
you
need
the
admin
context
for
that,
and
you
say:
okay
deploy,
deploy,
broker
and
that's
going
to
deploy
the
broker
for
you,
and
it
will
ask
for
anything
that
it
doesn't
cannot
figure
out
and
it
will
produce
this
brokering
for
Submariner
file.
And
then
you
go
to
the
next
cluster
and
you
say:
ok,
joy
and
make
you
comfy.
B
This
is
my
cluster
ID
and
it
will
make
sure
that
it's
installed
and
connected
and
I
will
show
you
how
it
works.
Hopefully
it
will
work.
We
depend
a
little
bit
on
the
Wi-Fi,
which
is
a
little
bit
on
the
edge
that
ok
yeah
I
wanted
to.
So
what
we
are
going
to
demo
is
what
we
had
at
the
start
is
vanilla,
kubernetes,
cluster
renter
cluster.
B
B
Before
I
go
into
the
demo,
I
wanted
to
say
thanks
to
all
my
colleagues
at
that
hat,
which
supported
me
to
make
sure
that
this
was
possible
with
deployment
of
clusters
last
time,
bug
fixes
and
stuff
like
that.
Thank
you.
Three
are
ok,
so
so
the
first
thing
that
we
do
is
go
into
the
rancher
caster
and
we
will
do
the
deploy
broker.
B
A
B
So,
okay,
now
Sumerian
is
running
on
this
on
the
runs
our
cluster
and
we
have
these
brokering
for
file,
which
is
basically
JSON
encapsulated
in
base64
to
make
sure
that
people
don't
break
it.
You
can
break
it
anyway.
So
if
we
look
at
the
other
pots
in
the
rancher
cluster
they
are
running,
that
is,
this
is
a
submariner
engine
and
the
Gateway
and
those
are
the
route
against
that
run
on
every
worker,
not
including
the
Gateway,
and
this
is
how
the
broker
looks
like.
B
We
only
have
one
cluster
Connect
one
minute
ago,
yeah
we
can
look
at
how
a
cluster
looks.
It
has
the
cider,
the
cluster
ID,
the
color
codes
that
it's
something
to
do.
This
simple
way
of
making
sure
deciding
how
cluster
connect
to
each
other's
are.
The
blue
ones
will
connect
over
the
ones
that
at
once
to
the
red
ones,
but
there
are
yeah
abs
to
enhance
that.
B
B
So
this
is
I.
Please
note
I'm
connected
to
the
red
hot
BPM,
because
that
one
that
cluster
doesn't
have
any
public
IP
at
all.
It's
so
every
connection
that
it
does
to
the
internet
this
via
not.
There
is
no
IP
address
pointing
to
that
cluster,
so
we
will
be
doing
IPSec
NAT
traversal
to
make
that
work,
and
this
is
the
vanilla
Co
ordinate
is
cluster.
B
B
Okay,
so
at
this
point
we
can
look
at
the
other
pots.
We
are
having
a
weird
DNS
issue
on
the
hub
cluster.
So
maybe
it's
no!
It's
working
this
time
right.
Otherwise,
if
it
didn't
work
with
the
DNS,
the
kubernetes
magic
will
make
sure
that
it
works
on
Tralee
and
we
can
look
at
the
pot
at
the
manila
kubernetes
cluster.
And
if
we
look
now
the
list
of
clusters
in
the
broker,
we
have
three
okay
and.
A
Yeah,
and
so
one
thing
I
do
want
to
point
out.
So
if
you
remember
earlier,
I
mentioned
the
fact
that
the
broker
is
a
kubernetes
cluster.
That
is
worried
where
we
facilitate
the
exchange
of
that
metadata
for
things
like
endpoints
and
clusters.
So
if
you
actually
looking
at
our
demo
here,
you'll
see
that
you
know
for
the
OpenShift
cluster
as
well
as
the
AWS,
you
know
kind
of
cube,
ATM
cluster.
A
We
are
specifying
the
cube
context
for
those
specifically,
but
with
the
cube
CTL
get
customers,
that's
actually
living
in
that
first,
that
first
I
guess
rancher
kubernetes
cluster,
that
we
spun
up.
So
not
only
is
that
cluster,
a
I
guess
a
component
in
our
sort
of
mesh
of
multi
clustered
multi
multiple
clusters
here,
but
it's
also
actually
storing
the
metadata
for
the
rest
of
the
clusters
as
well.
So
yeah.
B
You
know
in
the
rancher
cluster
I'm
going
to
run
there.
I
have
a
groper.
We
want
to
add
two
two
subsidy
L.
Now
it's
still
not
doing
it,
but
we
can
see
that
if
we
have
two
tunnels
established
to
the
other
clusters,
okay,
yeah
I
will
make
sure
that
we
have
in
UNIX
and
that
should
get
a
list
of
eyepiece.
Maybe
I
need
to
make
the
fault
now
found
is
kind
of
okay.
Can
you
still
see
it?
B
B
B
B
B
B
Ok,
so
if
you
want
to
try
out
subsidies,
you
can
get
it
from
here.
We
just
make
make
their
release
minutes
away.
I'm
very
I'm!
Sorry,
because
we
only
have
the
Linux
binary
version,
but
we
probably
in
the
next
week
we
will
be
adding
like
cross-compilation,
so
you
can
have
the
binary
for
OSX
or
you
know
other
platforms.
A
Yeah
so
as
I
sort
of
mentioned
earlier,
if
you're
like
I,
said
if
you're
interested
in
submariner
as
an
operator,
definitely
join
our
mailing
list
and
ping
us
because
we're
we're
trying
to
you
know,
do
our
best
to
make
sure
that
Submariner
ends
up
being
something
that
is
usable
right.
So
it's
very
important
to
us
and
additionally,
if
you're
interested
in
contributing,
we
do
run
a
weekly
meeting
on
Tuesdays
at
8
a.m.
Pacific
or
I.
Think
it's
5
p.m.
A
B
Like
to
point
out,
if
you,
your
company,
is
working
network
plug-in
and
you
want
to
make
sure
that
we
are
compatible
also
with
your
network
plug
in
please,
you
know
our
community.
We
are
really
open
to
making
sure
that
we
are
compatible
or
if
you
are
a
close
cloud
provider,
and
you
want
to
make
sure
that
submariner
works
in
your
cloud.
Please
join
us.
D
B
In
the
architecture,
if
you
see
we
made
it
the
default
that
you
only
need
to
two
clusters
to
try
it
out,
so
unless
you
specify
them
not
at
the
plain
flag,
it's
going
to
deploy
the
in
the
deploy
broker,
it
will
deploy
both
the
broker
and
the
submariner
data
plane.
So
you
just
need
two
clusters,
but
the
one
run
in
the
broker
means
we
publicly
accessible
because
all
the
grasses
need
to
connect
and
exchange
information
there.
Okay.
D
B
A
So
we
haven't
done
specific
benchmarks
yet
and
that's
on
our
sort
of
list
of
things
to
do
reason
being
that
you're
gonna
be
limited
mostly
by
crypto,
because
of
well
it's
IPSec,
and
then
on
top
of
that,
it's
wherever
you're
running
your
clusters.
So
we
can
get
we're
looking
at
try
to
give
generic
information
regarding
benchmarking.
But
we
don't.
B
B
Somebody's
working
in
different
situations,
so
people
can
have
figures
and
maybe
compare
like
okay.
What's
the
difference
between
going
through
submariner
and
the
IPSec
tunnel
and
mapping
the
service
publicly
on
the
internet
and
connecting
to
the
service,
so
I
think
that
will
be
like
a
significant
number
that
you
can
use.
C
You
said
earlier
that
you
were
interested
in
use
cases
that
the
people
had
so
a
use
case.
I'm
interested
in
is
I,
have
an
on-prem
cluster
and
then
I
get
a
bunch
of
new
pods
that
I
want
to
schedule,
and
rather
than
queuing
them
up,
I
launch
new
clusters
on
cloud
providers
burst
onto
them.
They
like
put
the
workloads
there
and
then
shut
down
those
public
cloud
clusters
after
the
workloads
have
completed.
A
It
won't
actually
have
to
be
modified
past.
The
fact
that
you
know
it's
just
going
to
be
it's
just
going
to
be
hitting
that
local
service
happy
within
your
on-premise
cluster
on
and
that
should
actually
with
submariner
connecting
your
clusters
together.
Allow
you
to
burst
traffic
into
your
burst,
burst
clusters
in
the
cloud.
A
E
A
So
it
is
so
technically
when
it
was
first
marketed
or
like
when
it
was
first
written.
It
was
built
to
be
sort
of
modular.
Like
I
mean
it
is
it's
like
it
is
modular
right.
It's
it's
just
a
matter
of
the
actual
implementation
to
make
wire
drive,
work.
I,
know
wire
guards
come
up
before
in
the
issues
for
the
project.
It's
just
we
haven't.
You
know
we're
still
focusing
on
other
aspects
of
making
it
usable
before
we.
B
Focus
on,
but
we
are
going
to
be
accepting
contributions,
so
if
anybody
wants
to
contribute
a
cable
engine
for
whether
we're
completely
open,
we
also
looking
into
implementing
other
cable
engines,
for
example,
for
bx
lon.
I
know
the
explain:
it's
not
secure
for
your
traffic,
but
it
is
useful
if
there
is
already
a
VPN
between
your
casters
and
you
want
to
have
that
kind
of
connectivity,
and
you
don't
want
there.
Ipsec
overhead
yeah.