►
Description
Don’t miss out! Join us at our upcoming hybrid event: KubeCon + CloudNativeCon North America 2022 from October 24-28 in Detroit (and online!). Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Multi-cluster Failover Using Linkerd - Charles Pretzer, Buoyant, Inc.
Failover across clusters is a great way to improve the overall uptime and reliability of Kubernetes applications. While whole-cluster failover can be accomplished at the global ingress layer, failing over individual services is a little more difficult. During this hands-on tutorial, Charles Pretzer, Linkerd team member, will walk attendees through how to use Linkerd, the CNCF graduated service mesh, to enable traffic failover for individual services across clusters. Attendees will learn how to combine service mesh metrics, traffic shifting, and cross-cluster communication in a cohesive and automated way using pure open source, while preserving fundamental security guarantees such as mutual TLS.
B
A
Is
on
so
welcome
to
this
session
on
multi-cluster
fellow
with
lingard,
I'm
not
going
to
say
much
other
than
if
you
have
any
questions,
there's
a
microphone
down
here
in
the
middle.
So
please
go
up
there
and
ask
your
questions.
I'm
sure
charles
is
going
to
instruct
how
he
wants
his
questions
as
well,
but
without
that
I'm
just
gonna
hand
it
over
to
ciao.
So
please
give
him
a
big
round
of
applause.
B
Well,
this
is
by
far
the
biggest
audience
that
I've
had
the
opportunity
to
speak
with.
So,
first
of
all
welcome
everybody
back
to
cubecon,
it's
so
exciting,
to
see
folks
in
person
I
you
know,
we
we've
done
the
virtual
thing
for
the
last
couple
of
years
and
I
am
just
really
energized
by
all
of
your
presence
right
now.
So
what
are
we
gonna
do
today?
B
B
B
B
B
B
Do
your
worst!
That's
what
I
have
to
say
so
again,
but
again
trust.
I
I'm
so
happy
to
see
you
all
here.
If
there
are
any
questions
during
this,
the
microphone
as
casper
mentioned
is
just
there.
This
is
meant
to
be
a
dialogue.
I
want
to
share
this
conversation
with.
You
share
the
space
with
you,
so
anything
that
you
want
me
to
stop
and
pause.
B
Let
me
know,
like
I
said,
I'm
from
california.
I
have
a
tendency
to
talk
very
quickly
if
I'm
talking
too
quickly.
Slow
me
down,
raise
your
hand
whatever
you
need,
I'm
here
for
you.
So
with
that
being
said,
let's
so
something
I
don't
get
to
do
anymore.
Have
you
ever
tried
to
during
a
zoom
meeting
be
like
so
who
here
is
has
worked
with
multiple
kubernetes
clusters.
B
B
B
B
It's
to
me
it's
a
category
of
of
circuit
breaking,
so
it
makes
sense.
When
something
fails,
you
don't
want
to
have
a
single
point
of
failure
which
what
I've
come
to
realize
last
night.
I
am.
I
am
a
single
point
to
failure.
If
I
didn't
show
up
today,
you
would
all
be
staring
at
this
screen
without
these
slides.
So
the
point
is
this
is
not
a
new
concept.
B
B
The
first
couple
slides
are
for
you.
The
rest
of
the
presentation
is
for
everybody
else,
yeah.
So
what
is
linked?
It's
a
service
mesh.
Why
is
it
important?
Why
do
people
care
about
it?
The
history
of
linkerd
is
that
it
started
outside
of
kubernetes
clusters.
It
was
a
standalone
jvm
application,
so
you
might
call
it
monolithic.
B
You
might
not,
and
it
was
a
response.
It
was
built
on
fnagle
nettie,
which
are
libraries
finagle's.
The
library
that
came
out
of
twitter
and
the
responsibility
of
it
was
to
manage
your
your
traffic
to
be
able
to
encrypt
well
not
only
to
encrypt
but
also
load
balances,
traffic
between
a
service
oriented
architecture.
B
So
this
is
not
new
for
some
of
you,
but
for
others
it
might
be
new.
The
the
really
interesting
thing
about
it
and
the
thing
the
reason
that
I
care
about
service
meshes
because,
as
a
former
application
developer
and
a
current
slide
developer,
I
develops
only
slides
now
when
I
was
writing
my
code,
I
would
have
to
consider
if
I
wanted
traffic
to
be
secured
between
my
my
services
or
whatever
my
endpoints
were.
I
would
actually
have
to
include
libraries
if
I
wanted
to
do
load,
balancing
or
retries
or
anything
like
that.
B
I
would
have
to
include
this
logic
into
my
code,
and
I
was
not
good
at
that
like
who
is
who
has
written
a
retry
block
that
looks
like
while
retries
is
less
than
retry
count?
Retry
anybody?
Yes,
okay,
good!
I
don't
feel
so
bad,
there's
a
smarter
way
to
do
that,
and
it's
not
to
say
that
you've
done
anything
wrong.
B
This
is
something
that
linkery
implements
so
we're
talking
about
linker
d1
we're
going
to
move
into
linker
d2
talking
about
the
kubernetes
space
specifically,
so
what
is
linkedi?
It's
a
service
mesh.
What
do
we
care
about
as
part
of
a
service
mesh?
I
mentioned
security
already,
reliability
is
that
load,
balancing,
piece
and
observability
is
being
able
to
collect
those
metrics.
All
those
things
work
together
to
give
you.
B
Like
some
of
you
who
are
running
a
a
service
mission
production,
I
I
think
it
I.
I
make
the
assumption
that
it's
obvious
and
it's
not
obvious
to
understand
that
really
understanding
the
latencies
between
your
services,
when
you
have
a
new
service
when
you
scale
replicas,
of
a
single
instance
of
a
service-
and
you
see
those
latencies
go
down.
B
How
do
you
observe
that,
where
you
get
that
information,
you
get
it
from
liquidity.
So
when
we
talk
about
a
service
mesh,
this
is
what's
incredibly
important.
To
understand.
Is
that
all
of
the
traffic
within
your
between
your
services,
within
your
cluster
and
across
clusters,
which
we'll
see
today
happens
thanks
to
linguity?
Your
services
are
not
communicating
directly
with
each
other
unless
they
are
what
we
call
unmeshed.
B
B
So
when
we
talk
about
failover,
what
we
are
gonna
dive
into,
and
it's
important
to
understand
that
this
failover
extension
there's
a
concept
there
that
I've
already
made
an
assumption
about
linkerd
supports
extensions.
So
you've
got
your
core
linker
d
control
plane
and
then,
on
top
of
that,
we
have
different
extensions.
Today,
we're
gonna
see
the
viz
extension,
the
multi-cluster
extension
and
the
failover
extension,
which
is
like
the
grand
poobah
of
this
whole
thing.
B
If
you
don't
know
what
a
grand
poobah
is
it's,
because
I'm
really
old
and
some
of
you
are
younger
than
me
and
the
grand
poobah-
I
don't
know
it's
it's
like
the
guy
who
is
important
anyway.
Today.
The
failover
extension
is
the
most
important
part,
I'm
probably
going
to
cruise
very
quickly
through
the
linker
d
deployment,
the
vis
extension
deployment
and
the
multi-cluster
deployment
to
focus
specifically
on
the
failover
piece,
because
I
expect
thousands
of
oohs
and
oz
when
you
see
failover
happening,
oh
we're
only
11
minutes.
B
In
goodness,
I
gotta
talk
slower,
okay,
yeah.
So
the
failover
piece
is
an
extension.
You
can
run
all
of
linker
d
without
any
of
these
extensions.
You
can
just
have
the
core
components
running
and
you
can
trust
that
your
application
is
secure.
The
traffic
is
secure
that
the
load
balancing
is
happening
properly,
but
to
really
understand
and
get
those
metrics
which
I'll
show
you
you're
gonna
want
to
include
some
of
the
the
extensions
that
we'll
have
so
let's
cruise
through
this,
I
promise
you
not
too
many
slides
here's
architecture.
B
I
actually
updated
this
slide.
It
used
to
look
terrible
now
it
looks
less
terrible,
so
you
have
services
a
b
and
c
the
way
that
linkedin
works
again.
These
proxies
are
injected
into
the
sidecar
or
sorry
injected
as
a
sidecar
into
your
pods
and
I'll
show
you
how
that
happens.
This
is
all
thanks
to
the
control
plane,
which
reads
an
annotation
who
knows
annotations,
I'm
going
to
keep
doing
the
raise
your
hand
thing
all
day,
because
I
missed
it
so
much
annotations
labels
who
knows
the
difference
between
annotations
and
labels.
B
B
Deployments
damon
set
staple
sets
most
of
your
workloads
that
you
care
about.
We
inject
with
an
annotation.
There
is
a
control
plane
component,
which
I
will
show
you
that
is
called
the
proxy
injector.
It
says
this
workload
is
annotated.
I
must
add
linkery
proxy
to
the
pod.
Who
knows
what
language
who
knows
about
the
linkedin
proxy?
B
Yes,
back
in
the
old
the
battle
days
of
kubecon,
people
are
like
yeah
linkery
uses
envoy
it
does
not.
The
linkerty
microproxy
is
a
purpose-built
proxy
for
the
linkrd
service
mesh.
It's
written
in
rust,
our
team,
our
main
team
of
maintainers,
spends
a
lot
of
time
in
the
rust
community,
contributing
giving
back
to
that
open.
Many
of
the
open
source
projects.
B
B
B
B
That
means
that,
like
it's
super
important,
but
at
the
end
of
the
day,
we're
going
to
have
two
separate
clusters
running
the
same
application
and
traffic
being
sent
across
those.
In
addition
to
that,
I
am
going
to
tear
down
this
emoji
service
right
here,
I'm
going
to
scale
its
replicas
to
zero
and
the
what
I
want
you
to
take
away
from
this
is
this
failover
that
you're
going
to
see
happen.
Failover
controller
is
going
to
say
there
are
no
end
points
for
this
service.
B
I
am
responsible
for
routing
traffic
to
whatever
the
backup
services
are.
The
backup
services
in
this
case
are
going
to
be
emoji
dash,
2
and
something
that
is
called
emoji
dash
service,
dash,
east
and
I'll.
Show
you
what
all
that
means.
Does
that
all
make
sense?
If
not,
let's
go
into
the
terminal
again.
C
B
B
C
B
B
B
Their
clusters
are
easy
to
spin
up
through
an
api.
Whatever
you
want
to
do
so
nothing
up
my
sleeve,
I
started
these
clusters
19
days
ago.
I've
actually
installed
deployed
unem
removed,
linker
d
multiple
times
from
this.
My
co-worker
used
this
yesterday
for
a
very
similar
presentation.
So
what
we're
going
to
do
now
is
deploy
linker
d
to
these
clusters
so,
like
I
said,
there's
nothing.
B
B
This
all
looks
good
okay,
so
certificates
are
important,
especially
when
we
talk
about
multi-cluster
communication,
the
certificates
that
you
deploy
with
linker
d,
especially
when
you're
deploying
using
helm
they're
a
requirement
if,
in
kind
of
like
a
dev
mode,
what
we
talk
about
with
linker
d,
you
can
use
the
cli
to
deploy
linker
d
it'll
generate
yaml
for
you
in
fact,
I'll
show
you
as
soon
as
this
is
done.
What
that
looks
like
when
you
use
the
linker
dcli
to
install
and
deploy
the
control
plane,
it
has
a
ca.
What
happened?
B
Oh,
no,
I'm
in
the
wrong
name,
space!
Okay,
thank
you.
Let's
do
right!
I
need
to
do
create
namespace.
I
think
let's
try
this
I'm
gonna.
Look
for
you
for
the
hand
up
again
in
case
something
goes
wrong
while
I
blabber
on
so
there
is
a
ca
that
comes
with
lingerie,
the
reason
that
their
certificates
are
important
and
that
it
all
matters
is
because
of
that
mutual
tls
that
I
talked
about
earlier.
B
B
Yes,
pki
means
to
me:
was
it
a
private
key
infrastructure?
Is
that
right,
I'm
guessing
so
we
create
a
root
certificate
and
we
have
issuer
certificates.
The
reason
that
we
have
issuers
those
two
levels
of
security.
The
root
certificate,
is
the
ca
that
is
used
by
the
linkedin
proxies
to
when
they
send
a
certificate
certificate
signing
request.
B
B
The
issuer
identity
are
intermediate
certificates
that
are
generated
from
the
this
root
ca
in
a
and
you'll
see
in
my
directory.
Excuse
me,
I
specified
west
and
east
it's
important
in
a
multi-cluster
linker
d
configuration
that
the
root
ca
is
the
same
ca
and
the
identity
issuers
are
intermediate
certificates
from
that
same
certificate
key
pair.
The
reason
for
this
is
because,
if,
for
some
reason,
your
identity
issuer
certificates
become
compromised
in
one
cluster,
that
is
the
blast
radius
of
that
compromised
security
boundary.
B
B
B
Okay,
so
I'm
gonna,
we're
gonna,
see
log
lines
that
are
gonna
overflow
and
it
might
get
a
bit
confusing,
but
lingerie
control
plane
is
deployed
in
h,
a
mode.
I
have
three
nodes
running
in
this
cluster.
In
this
one
cluster
and
you'll
see
I
have
three
replicas
of
each
of
the
control
plane
components.
This
is
default
for
linkery's,
high
bay,
high
availability
mode,
you'll
see
from
the
command
that
I
ran
that
I
have
values
dash
h,
a
dot
yaml
specified
there.
B
B
B
B
Okay,
so
that
binary
just
has
a
flag.
That's
dash
dash
context,
helm.
It
has
to
be
cube,
dash
context.
I
don't
know
why
but
yeah.
So
I've
got
the
control
plane
in
one
cluster,
I'm
getting
it
across
the
other
cluster.
At
the
moment,
the
the
clusters
are
not
communicating
with
each
other.
We've
got
a
few
steps
to
go
before
we
get
there
who's
following
along
by
the
way
like
do
you
who
wants
to
follow
along
who?
Okay,
good
everything
going
so
going?
Well,
so
far,
awesome!
B
Okay!
So
let's
talk
about
what
we're
building.
I
showed
you
a
a
diagram
that
had
a
few
things
missing
in
the
slides.
The
things
that
are
missing,
like
the
whole
lingerie
control
plane
is
meant
to
be
on
these
clusters.
It's
assumed
what
we're
building
is
identical
clusters
with
linker
d
deployments
across
the
thing
across
the
two
clusters,
so
one
of
the
really
interesting
pieces
of
the
linker
d
architecture-
and
you
all
can
tell
me
if
you
think
this
is
interesting
or
not.
B
The
the
clusters,
when
you
have
a
multi-cluster
environment,
are
meant
to
be
agnostic,
there's
no
one
single
control
plane
for
all.
If
there's
no
control
plane
of
control
planes,
each
cluster
has
its
own
control
plane
and
that's
what
we're
we're
building
here.
So
I
stalled
long
enough
to
make
that
happen.
So
po
is
my
alias
for
cube.
Control,
get
pods.
B
So
in
the
west
cluster,
in
the
linkery
context,
I
have
the
three
replicas
of
each
of
the
control
plane
components
in
the
east
cluster
same
thing,
so
we
have
identical
clusters
that
are
not
ready
to
communicate
with
each
other.
Next
we
are
going
to
and
I'm
going
to
do,
the
oh,
I
promised
to
show
you
we
did
the
linker
d
helm
installation,
which
is
the
100
recommended
way
to
do
this
in
production.
B
B
Cli
is
not
a
sub
command.
I
had
straight
word
mouth
to
finger
contact
there.
Lingerie
check
is
going
to
go
through
and
tell
me
whether
the
cluster,
the
control
plane
is
ready.
It's,
whether
it's
healthy,
there's,
also
a
linkity
check,
pre
command
linkedy
check
dash
dash
pre
before
you
deploy
the
control
plane.
It
makes
sure
that
you
have
the
correct
roles
and
permissions
to
be
able
to
deploy
the
linker
to
control
plane.
So
I
think
it's
worth
mentioning
here
that
we
consider
linker
d,
an
extension
of
kubernetes.
B
You
could
also
think
of
it
as
an
application
that
runs
in
kubernetes,
but
it
should
be
no
surprise
that
anything,
that's
managing
all
the
traffic
within
your
cluster
is
going
to
have
permissions
that
are
elevated
in
this
case.
My
specifically,
I
have
an
init
container
that
has
net
raw
and
net
admin.
Is
that
what
it
is
casper
net
raw
netadmin
capabilities?
B
B
B
B
These
clusters
have
been
true
to
me
for
the
last
20
days,
so
the
next
thing
we
want
to
do
is
deploy
linkedin.
So
as
it
stands
right
now,
in
fact,
let's
do.
Let's
do
something
fun
which
would
be
deploy
our
demo
application,
I'm
going
to
create
a
namespace
called
emojivodo
mojiboto
is
our
demonstration
application.
B
C
B
C
B
B
B
B
B
What
I
wanted
to
show
you
is
that
when
we
look
at
the
proxy
injector,
it
is
a
who
who's
familiar
with
mutating
and
or
validating
web
hook.
Configurations,
yes,
they're,
so
much
fun.
Proxy
injector
is
one
of
those
we've
got
another
one
called
the
tap
injector
anytime.
A
pod
is
scheduled,
it
reads
the
annotation
and
it
updates
the
ammo
in
real
time
injects
the
the
yaml
into
the
pot.
So
I
think
what
really
matters
here.
C
B
C
B
B
B
And
I
could
use
helm
for
this
as
well,
I'm
being
lazy,
because
my
fingers
are
tired
and
I'm
going
to
install
the
linkedin
vis
components
into
the
west
cluster
and
we'll
do
it
into
these
cluster
as
well.
This
is
going
to
give
me
the
ability
to
show
you
what
I
want
to
show
you,
which
is
the
secure
connections
between
the
services
and
as
well,
the
actual
metrics,
which
we,
if
you
remember
from
my
very
important
second
slide.
B
The
thing
that
linkedi
is
is
really
good
at
is
capturing
those
metrics
and
giving
them
to
you
in
an
actionable
way.
So
it's
one
thing
to
have
information
and
look
at
it
and
say:
okay,
I
see
my
latency
is
x
number
of
milliseconds
we'll
see
exactly
what
those
are
in.
Just
a
second
to
me,
it's
really
important
and
what
linker
d
will
open
up
for
you
is
exposing
these
prometheus
metrics
in
a
raw
format
that
you
can
then
take
action
on.
B
B
B
Solving
problems
with
them
and
coming
up
with
solutions
for
their
architecture.
What
we
get
into
oftentimes
is
these
raw
prometheus
metrics,
which,
like
I
said,
I'm
going
to
show
you,
but
what
I
care
about
is
that
we
can
do
something
with
those
metrics.
It's
not
just
it's
not
like
yeah
they're
there
they're
great.
B
B
C
B
B
B
B
B
One
of
the
really
awesome
things
that
folks
will
do
is
you
can
specify
a
specific
output
format
for
any
of
the
linkedin
checks,
including
the
viz
json,
specifically
so
some
of
the
operators,
people
who
are
using
linker
d
on
a
day-to-day
basis,
they
will
pipe
they'll,
just
run
this
as
a
cron
job
pipe
it
out
to
whatever
their
system
is
and
ingest
that
json
and
if
something
doesn't
look
right
like
their
network
is
slow,
then
they
will
alert
themselves
on
it.
So
again,
actionability!
That's!
What
is
important.
Is
that
a
word
actionability?
B
If
not,
I
coined
it.
I
didn't
coin
it.
Okay,
so
we've
got
d,
the
control
plane
deployed
and
we've
got
the
viz
deployed.
I
don't
know
what's
going
on
this
cluster.
Needless
to
say,
there
are
some
hamster
wheels,
spinning
very
quickly
trying
to
get
this
data
back
to
us
so
that
we
can
display
it
up.
There
goes
so,
like
I
said,
json
consume
it
parse
it
do
whatever
you
need
to
do
it
put
it
into
your
learning
system
and
be
able
to
understand.
B
B
It's
not
necessarily
what
we
see
and
linkerity
multi-cluster
doesn't
require
that
you
have
identical
services
running
in
every
cluster.
What
it
cares
about
is,
if
you
want
to
route
traffic
which
we'll
see
from
one
cluster
to
another,
that
service
just
has
to
exist
in
both
clusters,
so
it
can
be
one
single
service,
not
all
the
services.
Does
that
make
sense.
B
B
No
traffic
found:
why
not?
Why
not?
Let's
go
to
the
east
cluster
where
we
know
I
think
what's
happened,
is
I've
just
deployed
emoji
photo,
there's
a
traffic
generator
and
we
use
prometheus
metrics
to
display
these
statistics.
What
I
think
has
happened
is
there
just
aren't
enough
metrics
to
display
what
we
want
to
see.
B
B
Yeah
any
questions:
while
we
wait,
no
okay,
so
we've
got
linkery
the
control
plane
deployed.
We've
got
the
linker
d
viz
extension
deployed
here.
Are
my
metrics
lovely
save
the
day
and
often
what
I
do
is.
I
will
watch
these
because
it's
just
so
fascinating
to
see
the
requests
per.
Second,
all
of
this
information
and
again
this
is
you-
can
output
this
as
json.
If
you
want,
you
can
consume
it
by
whatever
you
need
to
consume
it.
Our
people
who
use
lingerie
in
production
have
x
use
different
apm
tools.
B
Anything
that
can
consume
prometheus
metrics.
You
can
export
everything
from
linkedin
out
to
your
provider
of
choice.
Does
that
make
sense?
Okay,
good,
so
we're
watching
we're
seeing
every
two
seconds
this
is
going
to
update.
I've
got
some
success
rates,
failure
rates.
These
look
like
the
golden
signals.
Who
knows
what
the
golden
signals
are?
B
B
I
use
the
term
bible
with
a
lowercase
b.
You
can
also
say
dictionary
your
guide,
your
guidebook.
Let's
call
it
that
yeah,
so
we've
got
metrics
here,
perfect,
linker
d,
vis
check
next
thing.
We're
going
to
do
is
deploy
so
again
our
clusters
are
identical.
Next
thing:
we're
going
to
do
is
deploy
the
linker
d.
Multi-Cluster
extension
mc
is
short
for
multi-cluster.
B
B
Yes,
okay,
so
we're
deploying
the
linker
dx
multi-cluster
extension
to
the
east
cluster
as
well.
So
again
our
clusters
we're
slowly
building
them
up
that
we're
building
them
up
to
be
identical.
You
will
see
a
couple
of
things
being
created
here:
we've
got
a
gateway
and
a
service
mirror
this.
This
is
all
that's
in
the
the
multi-cluster
extension,
so
it
doesn't
matter
which
cluster
I'm
in.
B
So
one
of
the
things,
if
you
all,
are
interested
in
helping
improve
an
open
source
project,
we
initially
talked
about
like
source
and
target
destination,
and
I
forget
what
like
local
and
remote
a
lot
of
terms
here.
That
really
just
mean
which
cluster
has
received
the
traffic
and
which
cluster
is
serving
the
response.
B
Today,
I'm
going
to
follow
our
docs
and
I
think
our
docs
say
local
and
remote.
So
for
the
sake
of
clarity,
our
local
cluster
is
going
to
be
the
west
cluster
and
our
remote
cluster
is
going
to
be
the
east
cluster
that
all
makes
sense:
okay,
good
yeah.
Okay,
so
you
will
see
that
I've
got
lingerie
gateways
in
both
of
my
my
clusters.
B
Mining
bots
ready
because
you
are
about
to
see
an
ip
address
for
my
cluster.
So
this
is
the
east
end
point
and
then
the
west
end
point
there.
Today,
I'm
not
going
to
do
this
today,
but
you
can
put
this
behind
in
ingress.
If
you
want
the
thing
that's
important
here,
I
think
it
actually
is
misunderstood
or
not
well
explained.
B
B
What
is
it?
418?
Look
up,
http,
418!
It's
awesome!
I
learned
about
it
a
couple
years
ago.
So
anyway,
this
is
all
secured
through
the
same
mechanisms.
Those
certificates
that
I
used.
When
I
deployed
linkery,
we've
got
the
the
trust
root
and
the
linkerity
gateways
themselves
are
linked
proxies
with
a
pause
container.
B
So
some
of
you
out
there
who
might
be
thinking
like
hey
this,
would
be
a
great
way
for
linker
d
to
act
as
a
gateway
because
it's
called
gateway
it's
coming
in
the
future.
I
get
to
tell
you
that
announced
at
cube,
gun
yeah.
We
talked
about
it
already.
It's
on
the
road
map
in
the
open
source
project
linkery
will
eventually
present
a
gateway
as
an
ingress
as
well
as
an
egress.
So
that's
on
the
roadmap,
something
that
folks,
like
you,
have
asked
for
for
a
very
long
time.
B
B
B
So
you'll
see
from
this
command
I'm
calling
liquidity
multi-cluster
link.
This
is
going
because
I
have
the
cube
config
and
I
have
cluster
admin
rights
on
both
clusters.
It's
going
to
the
cli
is
going
to
in
the
east
cluster,
generate
a
service
account
and
basically
a
cubeconfig
file
with
the
appropriate
permissions,
not
too
much
it's
the
goldilocks
not
too
much,
not
too
little.
The
appropriate
permissions
for
the
two
clusters
to
communicate
with
each
other
and
to
be
able
to
specifically
create
end
points.
B
B
B
I
hope
that
made
sense
again.
It's
the
the
folks
who
wrote
that
code
are
here.
I
can
read
the
code,
we
can
sit
down
and
read
it
together
and
we
can
understand
it
together.
I
cannot,
as
of
this
moment,
tell
you
all
of
the
things
that
are
actually
happening
there
to
make.
This
actually
work
is
that,
okay,
I
apologize,
they
told
you,
you
were
going
to
get
a
lingerie
expert.
They
lied.
B
What
we
should
see
is,
in
addition
to
making
sure
that
all
of
the
services
are
running
in
the
linkedin
multi-cluster
name
space.
It's
now
aware
of
a
cluster
named
east
that
has
a
mirror
and
that
it
can
talk
to
so
now,
unlike
five
minutes
ago
now,
our
two
clusters
are
able
to
communicate
with
each
other.
B
B
Linker
d
multi-cluster
supports
n
number
of
clusters,
so
this
architecture
that
we
talked
about
with
not
having
a
control
plane
of
control
planes
they're,
it's
a
trade
off.
So
now
you
can
have
every
cluster
looks
like
the
same
cluster.
If
one
cluster
goes
away,
you
can
still
have
that
cross
cluster
communication
and
the
important
thing
there.
The
thing
I
know
I
hope
you're
all
asking
yourselves
is:
does
it
do
circular
checks?
B
And
yes,
it
does
so
you
can
imagine
a
scenario
where
we're
going
to
export
a
service
in
a
minute
here,
if
you
have
bi-directional
communication
for
end
or
multi-directional
communication
for
n
number
of
clusters,
and
you
export
one
service
between
just
to
keep
it
simple:
we've
got
east
and
west.
So
let's
say
I
have
my
emoji
service,
which
is
the
one
that
we're
gonna
export.
B
If
I
have
that
exported
in
both
clusters
to
commute
like,
does
it
make
sense,
whatever
I'm
trying
to
get
here
with,
if
there's
a
circular
communication,
where
requests
would
just
fall
down
into
the
ether
and
never
ever
be
handled
with
a
response,
this
linkery
multi-cluster
check
checks.
For
that
raise
your
hand
if
you
get
it.
B
B
B
B
B
B
B
If
we
look
at
the
service
resources
in
the
emoji
photo
name
space
in
the
west
cluster,
what
we'll
see
is
that
service
mirror
created?
You
can
see
103
seconds
ago,
emoji
service
east
was
created.
This
was
automatically
created
by
the
service
mirror
because
I
added
that
label
straight
forward.
Does
that
all
make
sense?
B
B
That's
the
last
time
I'm
asking
you
to
raise
your
hand.
I
promise
yeah,
so
the
service
mesh
interface
specification
was
written
by
the
folks
at
deus
labs,
in
conjunction
with
the
folks
at
linkedin
and
buoyant,
and
I
think
some
of
the
google
folks
were
involved.
Anybody
else
know
do
you
know
who
else
was
part
of
that?
Okay.
B
So
what
the
specification
seeks
to
do
is
define
a
set
of
specifications
for
doing
things
like
traffic
policy,
traffic
management
and
linker
d
today
supports
the
traffic
split
specification,
so
I
have-
and
you
might
see
some
you
might
see
some
secret
things
here.
So,
okay,
good!
This
is
perfect.
This
is
exactly
what
I
want.
Okay,
this
is
a
traffic
split
specification,
api
version
split.smi.spec.io.
B
If
you
are
interested
in
more
about
the
service
mesh
interface
specification,
it's
smi
spec.io.
We
actually
support
v,
one
alpha
two.
I
don't
know
why.
I
have
v
one
alpha
one.
There
is
they're
up
to
v
one
alpha
four
at
this
point.
It
doesn't
make
a
difference.
Although
linker
d
only
goes
up
to
linker
d
or
sorry,
v1
alpha
2.
traffic
spit
split
resource,
it
has
a
name.
It
has
to
be
in
the
same
name:
space.
B
Who,
who
unders,
who
has
seen
this
before
because
it
took
me
like
it,
took
me
a
minute,
and
I
so
here's
the
thing,
there's
the
concept
of
an
apex
service
and
a
leaf
service
in
this
definition.
Our
you
see
that
service
right.
It
says
the
root
service
you
can
that's
synonymous
with
the
apex
service
is
the
one
that
decides
where
traffic
goes.
B
B
It
can
be
a
an
abstract
service,
which
means
you
have
just
a
service
definition
with
no
pods
backing
it
is
that
confusing
to
anyone,
don't
raise
your
hand
just
like
nod
we're
done
raising
hands
today,
so
it
confused
me.
It
really
confused
me
a
lot.
I
finally
sorted
it
out
any
time
that
I
do
traffic
splits
today
I
have
a
concrete
route
service,
and
this
is
the
part
that
can
be
confusing
the
back
ends
which
actually
handle
the
traffic
have
to
be
concrete
services.
B
B
So
this
is
actually
interesting.
It
doesn't
matter
that
I
have
it
in
the
east
context.
It's
going
to
ignore
it.
I
do
want
to
put
it.
Oh,
I
don't
need
to
specify
the
namespace,
because
that's
already
part
of
our
definition,
what
does
matter
is
that
it's
in
the
west
context,
this
traffic
split
is
configured
to
route
traffic
from
our
source
cluster,
the
west
cluster,
to
these
cluster.
So
we
now
have
everything
routing
or
we
have
traffic
going
between
the
clusters.
B
B
Ignore
this
part,
that's
fine.
This
is
talking
about
a
future
version.
Linker
d2.12,
where
there's
going
to
be
some
extra
steps
involved.
It's
not
important
today,
what's
important
today.
Is
that
we're
all
in
spain
hanging
out
together
there
we
go
okay.
So
let's
watch
this
one,
we're
still
waiting
for
traffic
to
come
through,
but
what
we
should
see
is
those
latencies
the
success
rates.
All
those
golden
signals
should
begin
to
populate,
as
those
prometheus
metrics
are
collected,
see
me
separately
I'll,
be
at
the
booth.
B
If
you
find
me
out
on
the
floor
anywhere,
I
think
personally,
I
think
how
to
get
to
the
raw
prometheus
metrics
in
linker
d
is
super
important.
I
don't
have
time
to
to
go
through
it
and
show
you
today,
but
it's
really
neat,
and
if
you
find
me,
I
will
show
you
how
to
do
it.
It's
also
a
very
handy
debugging
trick
for
for
liquidy,
so
guess
who
has
egg
on
their
face?
B
Brilliant
okay,
so
I
showed
you
how
to
annotate
the
name
space
which
is
going
to
do
everything
since
we're
here
doing
this.
We
could
actually
cube
control
edit
the
deployment
itself
to
add
the
annotation
one
of
like
the
simplest
easiest
things
to
do
when
you're
using
linker
d
is
to
put
the
annotation
in
the
wrong
space
or
in
the
wrong
part.
It
should
be
in
the
pod
spec.
B
B
B
B
B
B
So,
what's
interesting
here,
anything
that's
going
off
cluster
from
the
west
cluster
to
the
east
cluster.
We
would
expect
to
see
higher
latencies
overall
requests
per.
Second.
The
success
rates
are
interesting.
I
don't
know
why.
That's
that
low!
We
do
have
a
built-in
error
into
our
application,
but
it's
usually
around
83
85.
Something
like
that.
So
that's
interesting
to
me.
B
C
B
B
B
B
B
Yeah,
this
is
where
I
keep
files
for
dot,
dot,
demos,
doing
presentations,
github,
etc.
So
every
the
important
bits
are
in
this
linkedin
failover
directory,
I've
actually
forgotten
what
other
stuff
is
in
here.
There
might
be
something
interesting
there
might
not
be
so.
The
setup.sh
is
a
bash
file
that
will
use
k3d
if
you've
got
it
to
set
up
your
clusters
for
you
and
get
everything
in
the
right
state.
B
All
it
takes
to
do
this
failover
that
we've
talked
about
is
these
three
ammo
files?
I've
got
this
emoji
service
2,
which
is
actually
just
another
set
of
pods
that
do
the
same.
Excuse
me
the
same
exact
thing
as
the
emoji
service
and
the
the
part
that
really
matters
again.
Is
this
traffic
split.
So
this
is
how
we've
written
the
linkerity
failover
functionality,
the
part
that
matters
is
failover.linkedy.io
primary
service,
and
you
know
what
I
haven't
done
is
I
haven't
installed
the
link
defailover.
B
B
B
Just
so
you're
familiar
so
you
know
I
didn't
do
the
right
context
on
this,
so
we'll
have
to
go
back
and
do
that.
The
failover
controller
is
another
controller
written
in
rust.
It
looks
for
that
label
that
I
pointed
out
in
the
in
this
yaml
file,
failover.linkedin.I
or
sorry
annotation,
and
it
says
if
this
exists,
then
I
will
keep
an
eye
on
this,
the
health
of
the
service
and
make
sure
that
if
that
service
is
no
longer
available.
B
B
B
B
C
B
B
B
B
B
Traffic
split
statistics
when
there's
actually
traffic
being
sent
is
that
all
of
the
traffic,
when
I
scaled
down
that
deployment
of
the
emoji
service
on
the
local
west
cluster
would
have
been
routed
to
the
emoji
emoji
dash
2
and
emoji
dash
service
dash
east.
So
apologies
find
me
on
the
floor
and
we'll
get
this
sorted
out,
like
I'm,
probably
like
two
key
strokes
away
from
figuring
out
what
went
wrong.