►
Description
Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Multi-cluster Service Mesh Workshop - Will McKinley, Solo.io
This workshop is based on Istio and Gloo Mesh (https://github.com/solo-io/gloo-mesh). It's a hands-on workshop where each participant has a dedicated VM. In the VM, you deploy 3 Kubernetes clusters using Kind. One cluster is a management plane where Gloo Mesh is deployed, while Istio is deployed in the 2 other clusters. Then, you federate the identity of the Istio clusters, configure cross cluster communications, failover, learn about Web Assembly, ... And before each lab, we go through some slides. For example, before the identity federation, we explain you what is SPIFFE, how it's used in Istio, ... The labs are publicly available: https://github.com/solo-io/workshops/tree/master/gloo-mesh
A
It's
not
necessarily
a
prerequisite
for
what
we're
going
to
learn
here.
So
if
you
miss
those,
don't
don't
fret
about
that?
If
you
have
some
experience
with
istio,
this
will
still
be
useful
for
you.
Who
am
I,
as
rob
said.
I
am
a
field
engineer
here
at
solo,
dot
io,
I
have
a
pretty
long
and
very
background
started
in
telecom,
moved
to
mostly
developer
work.
I've
done
a
little
agile,
coaching
and
most
recently
helping
our
customers
here
at
solo
with
service
mesh
and
our
glue
products.
A
A
A
That
gives
you
things
like
service
discovery,
mtls
being
able
to
do
fault,
injection
circuit,
breaking
retries,
all
this
kind
of
cool
stuff
right,
but
in
reality
you
know
what
we're
seeing
is
when
you
go
to
production,
you're
not
going
to
just
deploy
to
a
single
cluster
and
that's
a
typically
just
a
proven
concept,
kind
of
thing
and
most
customers
that
we
have
today
are
hedging
their
bets
against.
You
know
multiple
vendors,
and
so
they
don't
want
to
be
stuck
in
just
one
cloud.
A
Let
me
make
this
a
little
bit
bigger
all
right,
and
so
you
know
we
want
to
enable
our
customers
across
multiple
clouds
and
hybrid
environments,
right
things
that
you'll
get
with
glue
mesh
enterprise
are
one
you're
going
to
get
a
long
term.
Support
with
istio
you're,
also,
obviously
going
to
get
those
muscle
multi-cluster
operations.
A
That
will
help
you
control
how
your
teams
interact
with
sdo
across
these
clusters
and
you'll
get
extensibility
through
web
assembly,
as
well
as
our
advanced
filtering,
and
be
able
to
see,
what's
going
on
across
meshes
through
a
single
pane
of
glass,
with
our
observability
and,
of
course,
zero
trust.
Security
is
is
kind
of
top
of
mind
here,
our
our
version
of
enterprise
istio.
A
We
we
always
do
upstream
first
right.
So
if
there's
some
kind
of
fix,
we
will
always
put
it
in
the
community,
but
we'll
have
specialty
builds
available
as
well.
So
if
you
have
some
kind
of
special
requirements
there,
we
can
comply
with
those,
and
we
will
also
support
critical
security
patches
and
our
sla
is
is
quite
fast
there.
A
A
Being
able
to
you
know,
do
things
like
ensure
that
traffic
is
encrypted,
making
sure
that
you
know
you
have
the
the
right
resiliency
settings
and
high
availability
set
there.
The
the
problem
comes
in
that
istio
doesn't
really
mirror
these
abstractions
istio's
api
closely
resembles
envoy
instead,
and
so
there's
a
there's
a
contention
here.
What
glue
mesh
offers
is
is
abstractions
here
that
make
sense
to
these
teams
right
so
by
directly
dealing
with
resources
like
traffic
policy
or
access
policy.
A
A
The
other
problem
that
we
see
is
obviously
just
multi-cluster
traffic
right,
multi-cluster
traffic,
you,
you
know
you,
you
can't
just
configure
every
single
cluster.
Exactly
the
same.
There
has
to
be
intention
in
and
how
these
resources
are
defined,
so
that
you
know
if
you
want
to
deploy,
for
instance,
canary
deployment
of
the
review
service
here
v3
into
cluster
two
and
not
cluster
one.
A
A
You
know
so
so
these
are
some
of
the
common
problems
and
we
need
to
make
sure
that
you
know
as
we're
administering
a
multi-cluster
istio
system
that
we
have
the
correct
semantics
in
place
and
we're
ensuring
that
our
our
fault
domains
are
correct
and
we're
unifying
identity
across
all
of
these
right,
istio
upstream,
does
have
a
way
of
dealing
with
this.
Unfortunately,
it's
kind
of
an
n
plus
one
problem,
or
an
n
minus
one
problem
of
needing
access
to
the
kubernetes
api
of
every
remote
cluster.
A
A
A
A
A
A
A
A
Okay,
that
good
cool
all
right
thanks
all
right
so
just
make
sure
you've
got
a
glue
mesh
license
key
this.
This
stuff
won't
work
without
it.
This
is
enterprise
features
all
right.
So,
as
long
as
that
is
set,
then
the
next
thing
we're
going
to
do
is
we're
going
to
install
the
command
line.
Mesh
cuddle.
A
A
A
All
right
so
we'll
use
mesh
cuddle
to
register
our
agents
and
here's
where
the
convenience
of
mesh
cuddle
comes
into
play.
You
could
also
do
this
with
helm.
It's
just
a
little
more
work
on
your
side,
because
mesh
cuddle
is
doing
some
work
behind
the
scenes
to
create
those
objects
that
that
are
needed
on
the
management
plane
and
we'll
take
a
look
at
that
those
objects
in
just
a
minute
as
soon
as
these
get
installed.
A
A
A
And
before
I
hit
enter
here,
you'll
see
a
few
things
that
are
being
set
as
booleans
one
is
the
enterprise
agent
enabled
and
we're
going
to
turn
that
on
just
a
bit,
the
rate
limiter
enabled.
So
we
have
a
rate
limiter
feature
as
well,
and
then
external
auth
service
and
that's
another
feature
that
we
have
in
our
glue
mesh
gateway.
A
You'll
see
this
is
our
our
access
policy.
Here
we
are
using
a
source
selector
of
the
istio
ingress
gateway
on
both
cluster
one
and
cluster
two,
and
then
we're
matching
that
against
the
blue
mesh
add-ons
name
space
as
our
destination,
and
so
this
opens
up
the
authorization
within
istio
all
right.
So
now
we're
going
to
deploy
istio
and
is
there
anybody
that
needs
me
to
slow
down
or
space
okay.
At
this
point,.
A
The
istio
operator
for
cluster
one
can
be
found
in
our
files
here.
So
if
we
want
to
take
a
look
at
exactly
how
we're
configuring
that
this
is
probably
a
good
way
to
take
a
look
at
that,
some
of
the
some
of
the
key
aspects
of
this
issue
operator
install
for
these
clusters
are
one
we're
establishing
the
the
trust
domain
within
the
specific
cluster
right.
A
A
And
while
that's
processing,
let
me
just
take
a
look
at
the
cluster
two
yaml.
It
looks
very
similar
to
cluster
one
yaml,
but
you'll
notice
that
it
has
its
own
trust
domain
here,
but
it's
part
of
mesh
id
mesh
one.
So
if
we
go
back
and
look
here,
istio
operator
for
cluster,
one
is
also
part
of
mesh.
One.
A
A
A
A
A
All
right
cool
that
one
came
up
a
lot
quicker
and
and
if
you
want
a
kind
of
a
picture
diagram
of
what
we
did
there,
this
is
a
very
nice
display.
So
here
we've
got
both
ingresses
exposed.
Externally,
you
can
see,
we've
got
book
info
product,
page
reviews,
ratings
and
details
deployed
to
both
clusters,
but
the
cluster.
On
the
left
hand,
side
here
only
has
versions,
one
and
two.
A
A
It
could
make
some
decisions
for
us
that
would
you
know
that
would
alleviate
some
of
the
pains
that
we
would
have
of
managing
each
cluster
individually.
So
we're
going
to
talk
about
the
virtual
mesh
next,
all
right,
okay,
so
one
other
thing
to
see
is
that
right
now
we're
not
running
with
mtls
at
all,
and
if
we
inspect
reviews
v1
on
cluster
1,
we
can
see
that
basically,
it's
running
without
any
certificates
defined
so
no
mtls,
and
we
would
like
to
fix
that
pure
authentication
in
the
seo
system.
A
We
do
that
to
both
clusters.
Now,
if
we
take
another
look
at
that
same
reviews
pod,
we
should
see
a
certificate,
that's
been
deployed
and
you
can
see.
This
is
basically
just
the
you
know:
self-signed
generated
one
that
comes
from
istioca,
so
it
has
our
cluster
one
that
we
had
specified
for
our
trust
domain,
coming
directly
from
istio.
A
A
A
We
are
naming
our
remote
meshes
here,
this
geodestio
system
cluster
one
and
cluster,
two
in
the
blue
mesh
namespace,
and
we
for
our
mtls
config.
We
want
them
to
have
a
shared
room,
trust
right
so,
for
this
example,
we're
going
to
have
the
management
plan,
create
that
root
of
trust
and
allow
it
to
generate
the
certificates
for
the
remote
clusters.
A
So,
let's
just
take
a
look
at
this
picture
of
how
that
works
here.
So
what's
happening
on
the
glue
mesh
management
cluster
is
we've
created
a
shared
route
here
and
we
have
a
ca
along
with
a
csr
that
gets
configured
and
pushed
down
to
these
cert
agents
that
are
on
our
remote
clusters
through
our
enterprise
agent.
A
A
Now
what
we'd
like
to
do
is
see
did
this
take
effect
into
our
our
actual
pause
of
running
within
istio?
So
we
take
a
look
at
that.
We've
used
v1,
you
know
you,
don't
you
don't
see
that
our
issuer
here
is
that
same
cluster,
one
that
used
to
be
right.
It
looks
a
little
bit
different.
Now
we've
got
glue
mesh
as
as
the
owner
and
we've
got
the
corresponding
certificate
change
that
came
from
our
secret.
A
Now
what
we'd
like
to
do
is
do
a
little
validation
on
the
the
root
cert
right,
so
this
one's
a
little
tricky,
because
the
copy
and
paste
buffer
is
a
little
bit
small
for
this
one.
So
I
think
what
we
can
do
is
just
capture
what
we
just
got
and
send
it
into
temp,
cert,
okay
and
then
we're
just
going
to
edit
that.
A
And
you
can
just
see
for
yourself,
you
know
what
this
is
going
to
do
so
using
openssl.
We
can
take
a
look
at
that
insert
and
inspect
it
and
see
exactly
what
it's
got
going
on
in
there.
So
one
of
the
things
you'll
you'll
notice
here
is
this
sam
or
this
subject.
Alternative
name
is
our
spiffy
id
for
the
service
account
for
book
info
ratings,
and
that
is
specific
to
cluster
two
there.
So
we
know
we
have
identity
within
this
within
the
certificate.
A
A
A
So
one
thing
we
need
to
do
is
is
to
modify
our
virtual
mesh
quickly
to
enable
this-
and
this
is
kind
of,
I
think,
very
nice
if
you're
familiar
with
the
way
that
istio
works
natively
in
order
to
set
up
the
policy
where
you're
operating
out
of
zero
trust,
you
need
to
ensure
that
each
application
name
space
has
an
authorization
policy
that
will
deny
traffic
here.
We've
got,
you
know
as
part
of
the
virtual
mesh.
A
A
A
So
now,
what
we
need
to
do
is
create
access
policies
that
will
open
up
the
path
from
our
ingress
gateway
back
to
our
application
and
also
all
the
the
the
traffic
that
needs
to
go
from
service
to
service
all
right.
So,
let's
start
with
the
ingress
gateway
and
the
ingress
gateway
will
need
to
reach
to
the
product
page
so
for
both
cluster
one
and
cluster.
A
Two
we're
going
to
enable
this
for
product
page
and
note
that
in
our
our
destination,
selector
we're
not
specifying
a
cluster,
so
it
means
it
will
it'll
apply
to
both
clusters,
all
right.
Just
with
that
simple
fix.
If
we
go
back
and
refresh
our
book
info
page,
you
should
see
now
and
it
might
take
just
a
second
for
that
configuration
to
be
put
pushed
out.
A
You
should
see
that
we
can
now
get
back
to
our
book
info
page,
but
some
parts
aren't
working
right.
We
can't
get
to
the
product
details
we
can't
get
to
the
ratings.
We
can't
get
to
the
reviews
because
we
haven't
enabled
that
access
yet
all
right,
so
our
next
task
is
going
to
be
to
turn
that
on
as
well.
A
A
A
A
A
A
Let's
add
a
little
delay
there,
so
we're
gonna
have
a
fixed
delay
of
two
seconds
any
time
we
visit
version,
two
reviews,
and
so
you'll
see
that
percentage
there
is
is
a
hundred
percent
right
and
now
this
will
take
a
a
few
first.
We
have
to
set
our
port
forward,
but
this
will.
This
will
take
a
few
refreshes
potentially
of
our
book
info,
so
that
we
can
see.
A
A
A
A
An
error
on
the
page:
okay
and
there
we
got
it
so
sorry,
product
reviews
are
currently
unavailable
for
this
book.
I
think
in
an
earlier
an
early
workshop
that
we
did
when
we
were
checking
the
envoy
config.
We
got
to
see
that
there
was
like
default
retry
of
two
right,
and
so
that
could
delay
the
you
know
being
able
to
perceive
this
kind
of
error
here.
A
A
All
right,
let
me
copy
this
over
here
and
then
we
can
talk
through
this
before
we
apply
it.
So
here
we've
got
our
traffic
policy
and
and
our
destination
that
we're
going
to
is
the
reviews
application
on
cluster
one
in
this
case
we're
going
to
apply
a
traffic
shift
and
most
of
that
traffic.
In
fact,
75
percent
of
that
we're
going
to
divert
over
to
cluster
2
reviews,
okay
and
then
we'll
keep
15
or
25
percent
here
on
cluster
1
between
both
v1
and
v2,
all
right.
A
A
All
right,
so
actually
what
we're
seeing
is
kind
of
curious
we're,
seeing
rating
services
currently
unavailable
quite
a
bit
right,
so
maybe
that
75
isn't
working
so
what's
happening
well,
what's
happening,
is
we
didn't
actually
open
up
access
across
the
cluster?
So
when
we've
got
our
product
page,
that's
trying
to
reach
reviews,
we
didn't
allow
product
page
to
actually
go
and
talk
or
sorry
reviews
talking
to
ratings.
We
didn't,
we
didn't,
allow
it
to
talk
across
clusters,
so
we
need
to
turn
that
on.
A
A
A
A
Okay-
and
you
know
now
that
we've
done
that-
let's
just
verify
that
we're
not
seeing
red
stars
anymore.
We
should
just
be
on
cluster
one.
There's
version
one
there's
version
two
and
I
don't
see
any
more
red
stars.
So
it
looks
like
we're
good
back
in
this
steady
state,
all
right,
and
so
in
this
instance.
A
A
A
All
right,
each
of
those
is
patched
now
we'll
probably
need
to
refresh
a
few
times
here
on
our
book
info
and
take
a
look
at
what's
going
on.
But
what
I'd
also
like
to
do
is
on
cluster
two:
let's
take
a
look
at
what's
happening
with
our
logs
and
just
make
sure
that
we're
actually
hitting
cluster
two
okay.
So
let
me
create
some
space
there
and
we'll
go
back
to
our
book
info.
A
A
A
So
you
know
one
of
the
common
things
that
you'll
need
to
do
when
you're
looking
for
errors
in
the
system
or
just
want
to
observe
requests
that
are
coming
into
your
system
is
to
make
sure
that
you
have
accessible
access
logs
right
and
we
have
a
higher
level
abstraction
called
access
log
record
that
we
can
apply
in
the
management
plane
that
will
allow
us
to
capture
this.
So
let's
take
a
look
at
how
that
works.
A
A
We
have
a
plug-in
system
for
it,
so
we're
going
to
first
initialize
the
plug-in
manager
from
sctl
you'll
see
an
error
that
comes
up
that
doesn't
know
the
command
plug-in,
but
once
we've
gone
through
the
installation,
then
that
problem
goes
away.
So
that's
all
right
and
then
what
we'd
like
to
do
is
is
actually
install
the
access
log
plug-in
into
meshctl.
So
let's
also
do
that,
and
this
access
log
does
need
to
be
tied
to
the
specific
version
here.
A
A
You
know
any
kind
of
tls
properties
that
you
had
in
there
subject
names
that
your
yours,
san,
is
also
recorded
things.
You
know
that
that
would
be
critical
for
you.
If
you
were
trying
to
debug
a
problem
with
maybe
a
slow
running
app
or
maybe
some
back-end
access
wasn't
behaving
quite
quite
well,
then
these
metrics
would
be
very
helpful
there
as
well.
A
A
So
you
have
a
virtual
mesh
config
and
it's
across
clusters
1
and
clusters
2
here
we
can
also
you
know,
inspect
things
like
the
workloads
that
are
configured
against
that
or
policy
rules
that
we
have
applied
and
if
we
had
any
kind
of
extensibility
here
with
wasn't
filters.
Those
those
you
could
also
see
here
if
we
click
on
things
like
a
virtual
mesh
details-
and
this
is
actually
a
filterable
field
here.
A
So
if
we
just
want
to
look
at
reviews,
for
instance,
we
can
take
a
look
at
that
and
then
click
in
to
see
you
know
the
exact
policies
that
have
been
deployed.
So
so
we've
got
a
product
page
policy
here,
and
you
know
it's
allowed
to
hit
anything
basically
on
reviews
so
feel
free
to
play
around
in
there
a
little
bit.
A
A
A
A
A
Our
glue,
mesh
gateway
is
basically
an
enhanced
version
of
istio's
gateway,
where
we
add
a
lot
of
the
feature:
functionality
of
an
api
gateway,
and
so
you
can
do
things
like
rate
limiting
or
oidc
off
integration.
Very,
very
handy,
all
right.
So
first
thing
we
want
to
do
to
take
advantage
of
this
thing
is,
is
is
for
for
now,
what
we've
been
doing
is
just
using
istio's
gateway
to
expose
our
product
page
or
our
book
info
app.
But
what
we'd
like
to
do
is
take
advantage
of
the
mesh
gateway.
A
So
let's
take
a
look
at
first
deleting
the
gateway
and
the
virtual
service
that
we
had
created
for
the
book
info
and
now
we're
going
to
create
a
virtual
gateway.
So,
let's
take
a
look
at
that
what
it
looks
like
before
we
apply
it,
so
our
book
info,
virtual
gateway,
is
going
to
have
a
virtual
host,
that's
attached
to
it
and
it's
using
ingress
gateway
selectors
here
on
cluster
1
and
cluster
2
from
the
istio
ingress
gateway
that
we
had
defined
earlier
in
our
istio
operators.
A
Now
we
haven't
defined
the
virtual
host
yet
so
that's
the
next
thing
that
we
need
to
do
and
what
the
virtual
host
is
going
to
do
for
us
is
it's
going
to
match
up.
You
know,
domain
matching
and
any
kind
of
path
matching
as
well,
since
this
is
our
only
application.
We
have
a
very
simple
or
simple
example,
here
of
just
a
catch-all
domain
and
path
matching.
A
A
A
Now
we
should
be
able
to
see
that
this
worked
if
we
go
into
terminal,
2
and
port
forward
to
our
ingress,
and
you
can
see
that
book
info
is
exposed
and
we
can
refresh
this
to
make
sure
yeah.
It
looks
good,
so
our
virtual
gateway,
remember,
we
deleted
both
the
gateway
and
the
virtual
servers
that
were
existing
there
for
istio
system
created
this
virtual
gateway,
virtual
host
and
route
table
combination,
and
that
took
the
place
of
what
was
there
before.
A
A
A
A
What
I'd
really
like
to
see,
though,
is
being
able
to
do
multi-cluster
traffic
all
right
so
remember
in
one
of
our
previous
labs,
we
had
a
traffic
policy
that
allowed
us
to
shift
our
traffic
across
the
cluster,
so
we're
going
to
put
that
back
in
place,
as
you
know
very,
very
similar
to
the
earlier
traffic
policy
that
we
had
this
time
it
has
a
route
selector.
So
it's
taking
advantage
of
our
route
table.
A
A
A
Okay,
so
what
we
want
to
see
here
is:
did
our
traffic
shift
work
so
if
our
traffic
shift
worked?
What
we
should
see
is
that
we
get
ratings
back
with
our
red
stars
75
of
the
time,
and
you
can
see
that
that
is
the
case
most
of
the
time
we're
getting
red.
So
occasionally
we
get
black
small
percentage.
We
get
nothing
back
from
this
rating,
so
that
seems
to
be
working
which
is
good.
A
Did
you
say:
wrap
up,
rom,
yep,
okay,
all
right,
fantastic,
yeah,
we'll
wrap
up
here.
You
know
like
both
of
the
other
workshops.
We
run
our
workshops
pretty
regularly.
So
please
go
to
our
solo
site
events
and
take
a
look
at
you
know
when
another
one
might
be
offered.
If
you
want
to
review
what
you
saw
today,
I
want
to
get
more
out
of
it
and
yeah.
We
look
forward
to
seeing
you
there.
So
thank
you
to
the
helpers
in
the
room,
rom
and
eric.