►
From YouTube: Administering Multi Cluster Service Meshes Securely - Eric Murphy & Eitan Yarmush, Solo.io
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon North America 2021 in Los Angeles, CA from October 12-15. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Administering Multi Cluster Service Meshes Securely - Eric Murphy & Eitan Yarmush, Solo.io
The majority of existing multi-cluster service mesh architectures require the distribution of Kubernetes API credentials (kubeconfigs) across clusters, typically by provisioning a service account in the local cluster and copying its access token to a process running in a remote cluster. This architecture requires that credentials for the Kubernetes API be shared with entities outside the cluster, exposing it to attack. Furthermore, scalability limits of the Kubernetes API Server make it less than ideal to serve an unbounded number of potential remote clients managing configuration and sharing access to a cluster. This talk will explore the downside of existing approaches in this model and propose a new approach based on a client-server management architecture inspired by Envoy which does not require sharing sensitive Kubernetes credentials with remote clusters.
A
A
So
before
we
jump
in,
let's
just
quickly
go
over
what
we're
going
to
discuss
in
this
talk.
First
we're
going
to
review
multi-cluster
service
mesh
and
how
it
has
evolved.
First,
focusing
on
single
cluster
service
mesh
and
jumping
off
from
there.
Next
we're
going
to
discuss
the
limitations
and
security
issues
with
the
current
approach.
A
So
before
we
jump
into
multi-cluster
service
mesh,
let's
just
quickly
take
a
look
back
at
single
cluster
service
mesh.
So
this
is
a
very
simple
single
cluster
service
mesh
example.
We
have
a
couple
of
different
workloads
running
in
cluster,
as
well
as
our
service
mesh.
In
this
case,
the
service
mesh
is
istio,
but
that
is
not
necessary
for
this.
A
So,
as
you
can
see,
all
of
the
all
the
workloads,
including
the
istio
control
plane,
are
communicating
directly
with
the
kubernetes
api
server.
In
the
case
of
the
control
plane,
it
communicates
with
the
api
server
to
get
the
endpoint
information
for
all
the
workloads
as
well
as
service
info
and
other
metadata
about
the
cluster.
A
So
let's
just
quickly
review
the
single
cluster
scenario
and
why
it
had
what
the
benefits
are
and
why
it's
more
simplistic,
so
the
control
plane
and
the
data
plane
both
live
on
the
same
cluster
authentication
with
the
api
server
is
handled
automatically
by
kubernetes.
So
the
service
account
tokens
are
generated
and
mounted
automatically
and
the
tokens
remain
safely
in
kubernetes
more
on
this
later,
most
likely
no
need
to
revoke
those.
A
A
Now,
in
this
scenario,
all
of
the
control
planes
still
need
access
to
the
cluster
that
they
are
running
on,
but
in
order
for
a
multi-cluster
surface
mesh
to
be
useful,
the
istio
control
planes
also
need
to
be
able
to
read
and
write
to
the
other
clusters.
So
what
does
that
look
like
for
us?
Well,
we've
quickly
drawn
up
an
example.
A
A
A
So
multi-cluster
service
mesh,
the
control
plane
and
the
data
plan
are
spread
out
across
many
clusters.
Each
control
plane
needs
access
to
each
other
cluster.
That
means
we
have
the
an
order.
N
squared
connections
where
n
is
the
number
of
clusters
authentication
is
not
handled
automatically
for
our
remote
control
planes,
so
service
accounts
must
be
created.
A
Coupe
configs,
containing
the
service
account
token
must
be
distributed
to
each
cluster.
Now
these
coup
configs,
which
must
leave
cluster
borders,
give
potentially
dangerous
levels
of
access
to
said
cluster.
If
any
kubeconfig
credential
were
to
leak,
replacing
it
and
redistributing
the
credentials
is
a
convoluted
process.
A
A
So
the
approach
that
I
just
mentioned,
with
order
n
squared
connections,
that's
one
way
of
doing
it,
that's
the
that's
that
was
the
istio
way
of
doing
it
beforehand,
but
a
new
way
is
starting
to
emerge,
which
we
think
is
a
lot
better
and
that's
the
external
control
point.
So
in
this
scenario
istio
is
running
or
the
service
mesh
is
running
on
a
management
cluster
and
managing
a
bunch
of
clusters
on
its
own
and
in
this
way.
A
A
A
So
the
external
control
plane,
the
control
planes
are
now
on
a
single
cluster
and
the
data
planes
are
still
spread
out.
Now
we
like
to
call
this
the
push
model
of
configuration,
there's
one
centralized
control
plane
with
access
to
all
data
plane
clusters.
So
that's
order.
N
connections
and
the
centralized
control
plane
will
be
pushing
config
out
to
the
remote
clusters.
A
So
what
are
our
concerns?
Our
main
concerns
about
administering
multi-cluster
seo
or
service
mesh
securely,
one
difficult
to
store
and
sync
all
of
the
coupe
credentials
using
traditional
git,
ops,
workflows
and
two
giving
a
cluster
read.
Write
access
to
many
other
clusters
leaves
a
potentially
large
security
threat.
A
So
how
can
we
get
around
some
of
these
issues?
Well,
the
way
that
we
thought
about
it
is
something
called
a
pull-based
method
of
config.
So
if
you
notice
in
the
previous
diagram,
all
of
the
arrows
were
pointing
out
to
the
to
the
remote
clusters,
whereas
now
all
the
arrows
are
pointing
in
at
the
management
cluster.
So
it's
still
an
external
control
plane,
but
it's
pull
base
this
time.
So
this
so
now,
all
of
the
remote
clusters
are
actually
creating
the
connections
to
the
management
plane.
A
So
the
management
plane
doesn't
need
to
have
access
to
any
of
the
remote
clusters,
and
neither
does
the
service
mesh
running
there.
So
what
benefits
does
this
bring
us?
Well,
this
pull
model
of
configuration,
as
I
mentioned,
there's
one
centralized
control
plane
which
need
act
which
needs
access
to
zero
data,
plane
clusters,
all
the
data
plane
clusters
connect
to
the
control,
plane
cluster
and
receive
updates
that
way.
The
config
updates
function
similarly
to
the
envoy
xds
protocol
for
those
familiar
and
for
those
unfamiliar.
A
We
will
go
over
that
in
a
minute
and
linked
credentials,
no
longer
give
the
same
direct
control
over
the
api
server.
Furthermore,
no
kubernetes
credentials
need
to
be
shared
outside
of
the
cluster
borders
and
since
we
are
no
longer
distributing
these
credentials,
the
the
remote
components
we
are
distributing
can
be
easily
stored
and
distributed
via
githubs.
A
So
how
does
this
really
work?
Well,
we
took
heavy
inspiration
from
envoys
xds
protocol
and
the
kubernetes
bootstrap
token,
which
I
will
explain
now,
but
those
links
are
available
from
the
slide.
The
way
that
the
xds
protocol
works
is
essentially
envoy
makes
a
connection
to
the
control
plane
or
the
management
server
and
says
hey.
I
want
some
config
updates
and
the
management
server
says
all
right.
A
Great
I've
got
that
for
you,
and
so
in
that
way,
each
remote
instance
each
envoy
is
responsible
for
creating
the
the
connection,
and
then
the
management
server
is
responsible
for
updating
envoy
with
the
config
when
necessary,
and
the
kubernetes
bootstrap
token
is
a
way
of
verifying
the
identity
of
different
callers.
So
more
on
that
in
a
second.
A
A
In
addition,
we
distribute
a
bootstrap
token
to
both
clusters
and
this
bootstrap
token
will
be
used
in
the
initial
request
with
the
token
so
that
the
server
knows
to
trust
the
agent
and
then
can
and
then
once
it
knows
it
can
trust
the
agent
it
will
respond
with
its
mtls
cert
and
it's
with
the
identity,
and
this
mtls
cert
is
how
the
server
knows
what
data
to
send
back
to
the
agent.
So
once
that
identity
has
been
provided,
then
we
can
open
our
our
config
streams.
A
So
can
the
pull
model
really
solve
all
of
our
problems?
Short
answer?
No,
but
it
does
alleviate
the
two
main
issues
that
we
began
to
discuss.
A
few
slides
back,
the
first
being
giving
a
cluster
read.
Write
access
to
many
other
clusters
leaves
a
potentially
large
security
threat.
Well,
since
our
control
planes
no
longer
need
direct
access
to
remote
api
servers,
the
kubernetes
credentials
no
longer
need
to
leave
cluster
boundaries.
A
Well,
luckily,
there's
no
longer
any
need
to
store
or
sync
these
coup
credentials,
on
top
of
which
the
the
components
that
do
need
to
be
synced,
mainly
the
agent
and
the
token,
can
easily
be
integrated
with
get
ops,
see
part
two
of
the
talk.
So
what
were
those
question
marks
that
we
were
talking
about
earlier?
Well,
it
just
so
happens
that
we
at
solo,
I
o,
have
created
a
product
called
glue
mesh
which
can
which
can
handle
this.
But
more
on
that
later,
for
now
I'm
going
to
kick
it
over
to
eric.
B
So
I
have
a
demo.
I
want
to
show
you
today
and
in
that
demo
I
have
two
clusters
running.
I
have
a
management
cluster
and
on
that
management
cluster
I
have
argo
cd
deployed,
I'm
going
to
use
argo
cd
to
deploy
configuration
from
a
git
repository
and
automate
the
deployment
of
the
glue
mesh
management
plane
on
the
remote
cluster.
I
have
a
separate
instance
of
argo
cd
running
and
I'm
going
to
use
configuration
there
to
deploy
the
agent
automatically.
B
So
there
are
multiple
benefits
here
to
this
approach.
Where
you
can
take
configuration
as
code,
you
can
store
everything
in
your
git
repository
and
you
can
have
a
repeatable
deployment
process
for
both
the
glue
mesh
management
plane
and
the
agent,
and
this
allows
you
to
automatically
update
your
glue
mesh
deployments
with
that
configuration
with
argo
cd.
B
B
So
what
I'm
going
to
show
here
in
this
demo
is
actually
how
to
physically
deploy
glue
mesh
using
argo,
cd,
okay
and
so
to
do
that.
I
have
some
configurations
checked
into
my
git
repository.
Everything
is
deployed
from
a
git
repository
using
argo
cd
and
in
that
git
repository
I
have
crds
that
are
specific
to
argo
cd,
and
so
this
is
an
application
crt
and
inside
of
that
application,
crd,
you
can
actually
define
values
for
a
helm
chart.
B
B
That
means
I'm
providing
the
certificates
in
advance
of
the
installation,
I'm
not
depending
on
glue
mesh
to
generate
certificates
on
its
own
okay
and
then,
if
I
go
over
to
this
other
application
crd,
I
can
see
the
helm
chart
for
deploying
the
agent,
and
so
here
the
chart
is
enterprise
agent
and
I'm
giving
it
a
server
address
for
the
management
cluster,
I'm
giving
it
an
authority
for
the
certificates
that
are
used
and
I'm
giving
it
a
cluster
name
which
is
simply
remote
cluster.
Okay.
B
B
B
Okay,
so
that's
done
so.
What
I
can
do
is
go
over
here
and
log
into
the
argo
cd
deployment
on
my
management
cluster
and
you
can
see
how
quickly
that
was
actually
deployed.
Everything
is
synced
and
healthy
already
in
a
matter
of
seconds,
and
if
I
click
in
here,
I
can
see
everything
that
was
deployed
for
the
glue
mesh
management
plane.
Okay,
so
you
can
see
there's
a
lot
of
different
things
deployed
here.
Right
everything
is
green.
B
Everything
is
deployed,
everything
is
healthy,
include
all
the
pods
for
the
dashboard,
the
enterprise
networking,
and
so
that
all
looks
good.
As
you
can
imagine
in
a
glue
mesh.
The
management
plan
is,
you
know,
a
little
bit
complicated
because
it
does
manage
multiple
clusters
with
multiple
meshes,
and
so
there
are
a
lot
of
capabilities
built
into
it.
Then,
if
I
step
out
here
and
look
at
the
rest
of
my
application
deployment,
I
can
see
configurations
for
the
glue
mesh
management
plane.
B
B
There
is
a
signing
certificate
also
stored
in
the
secret,
and
that
allows
the
glue
mesh
management
plane
to
generate
new
certificates
that
can
be
passed
to
remote
clusters
to
establish
an
mtls,
secure
connection.
Okay,
also,
there
is
a
token
that
is
created
and
I
provided
this
token
in
advance,
so
that
token
is
deployed
with
the
glue
mesh
agent
and
that
is
used
for
making
that
initial
connection
from
the
remote
cluster
to
the
management
cluster,
as
that
was
discussed
previously.
B
Finally,
I
have
a
configuration
called
kubernetes
cluster
and
that
simply
defines
the
remote
cluster
that
is
configured
to
be
registered
by
the
agent
and
I'll
walk
through
how
that
works.
Okay,
so
let
me
go
over
here
to
my
console
and
I'm
actually
going
to
kick
off
the
mesh
cuddle
command,
and
so
that
will
allow
me
to
see
what
is
currently
running
what
is
currently
registered
in
the
glue
mesh
management
plane.
Okay,
so,
as
I
mentioned
before,
there
is
a
kubernetes
cluster
crd.
B
Okay,
so
let
me
go
back
over
here,
and
so
what
I'm
going
to
do
is
I'm
going
to
change
my
cube,
cuddle
context
to
b
for
the
remote
cluster,
I'm
going
to
also
create
a
glue
mesh
namespace
there
and
I'm
going
to
deploy
that
other
application
crd
specific
to
the
glue
mesh
agent?
Okay,
so
let
me
paste
that
in
there
there
we
go
all
right.
So
now
let
me
go
over
and
log
into
the
other
instance
of
vargo
cd.
This
is
not
the
same
argo
cd.
This
is
the
one
deployed
on
the
remote
cluster.
Okay.
B
So
I'm
going
to
click
sign
in
here
and
here
you
can
see
that
everything
that
is
needed
for
the
glue
mesh
agent
is
deployed
right.
So
that
also
happened
very
quickly
in
a
matter
of
seconds.
You
can
see
that
there's
a
enterprise
agent
pod,
that's
running
right
here,
so
that
is
green.
That
is
healthy,
so
everything
looks
good
for
the
glue
mesh
agent
deployment.
Then,
if
I
go
over
here
and
look
at
the
rest
of
my
deployment,
I
can
see
the
actual
configurations
that
are
residing
in
that
git
repository
deployed
by
glue
mesh
okay.
B
So
if
you
look
here,
there
is
a
secret
that
is
deployed
with
that
wrist
certificate
right
and
there's
also
that
identity
token
secret.
So
that
is
the
same
token
that
was
deployed
on
the
management
cluster
and
once
again,
that
token
is
used
to
authenticate
the
remote
cluster
with
the
management
cluster
and
establish
that
initial
connection
with
the
management
plane
for
glue
mesh
okay
and
when
that
happens,
it
receives
back
an
mtls
certificate
and
a
new
connection
is
established.
So
there
is
a
fully
secure
connection
for
the
glue
mesh
agent.
B
Okay-
and
one
thing
I
want
to
call
out
here-
you
know
just
for
demo
purposes-
I'm
putting
the
certificates
into
git.
You
know
for
the
risk
certificate.
That
is
not
the
best
practice.
Instead,
you
could
use
remote
secrets.
You
could
integrate
with
hashicorp
vault.
You
can
integrate
with
your
cloud
provider
with
their
certificate
management
services.
B
So
once
again,
that's
just
for
demo
purposes,
okay,
and
by
the
way
you
know
all
this
is
stored
in
in
the
git
repository
as
I
mentioned.
So
if
you
go
in
here,
you
know
you
can
look
and
you
can
see
you
know
these
configurations
for
the
glue
mesh
agent.
You
can
see
like
the
token
the
token
value
right
here
right.
B
You
can
see
the
the
root
tls
secret.
You
can
see
the
ca
cert
right
here
right,
so
all
that
is
deployed
using
that
that
get
ops
approach.
Okay,
okay,
so
since
we
deployed
that
agent
it
should
be
registered.
So
let's
go
back
and
look
at
the
glue
mesh
console
again,
and
you
can
see
now
that
under
meshes
right
here
I
actually
have
a
mesh.
That's
registered
right!
B
It's
this
sdod,
istio
system,
remote
cluster
right
here
right
and
the
mesh
health
is
green,
and
I
can
see
that
there
are
some
destinations
that
were
automatically
discovered
on
that
remote
cluster.
Keep
in
mind
this
glue,
mesh
console
is
running
on
the
management
cluster.
So
if
I
go
in
here
and
look
at
the
mesh
details,
I
can
see
that
there
is
a
destination
for
the
pet
store
application.
I
deployed
that
pet
store
application
in
advance
of
this
demo.
B
B
B
We
also
have
something
called
the
glue
portal,
so
that
provides
a
developer
portal,
a
user
interface,
an
api
specification
discovery
that
integrates
with
glue
edge
and
also
istio
ingress.
And
finally,
we
offer
something
called
webassemblyhub.
So
that
is
a
place
where
you
can
build
webassembly
extensions
for
the
envoy
proxy.
You
can
publish
them,
you
can
deploy
them.
It's
compatible.
Both
with
istio
and
glue
edge,
so
with
that,
I
thank
you
for
watching
today
and
if
you
have
any
questions,
please
feel
free
to
reach
out.
Thank
you.