►
From YouTube: eCHO Episode 41: Cilium Clustermesh
Description
Liz shows how Cilium's Clustermesh feature makes it easy to run services backed by pods distributed across multiple Kubernetes clusters
0:00 Introduction
5:42 Clustermesh introduction
6:35 Clustermesh for high availability services
8:23 Two-cluster demo
11:22 How Clustermesh works
14:38 Clustermesh connect joins the clusters
17:05 Marking a service as a global service
24:58 Splitting stateful and stateless clusters
26:40 Demo: delete the pods backing a service
28:28 What happens if you scale the deployment?
34:41 Applying network policy across multiple clusters
41:30 A classic network policy error!
42:20 Allow DNS in the network policy
--
Whether you are just starting to learn about eBPF, you're looking for further material or you're a seasoned contributor to major eBPF projects, the eBPF & Cilium Community is here to support you. Join the community on Slack 24/7 for help with, and discussions about eBPF and Cilium: https://ebpf.io/slack.
—-
Cilium is an open source software for providing, securing and observing network connectivity between container workloads - cloud native, and fueled by the revolutionary Kernel technology eBPF.
Learn more on the website: https://cilium.io/
Follow us on Twitter: https://twitter.com/ciliumproject
Download and contribute on Github: https://github.com/cilium/cilium
—-
eBPF is a revolutionary technology with origins in the Linux kernel that can run sandboxed programs in an operating system kernel. It is used to safely and efficiently extend the capabilities of the kernel without requiring to change kernel source code or load kernel modules.
Learn more on the website: https://ebpf.io/
#ebpf #cilium #cloudnative
A
Hello
and
welcome
to
episode
41
of
the
ebpf
and
psyllium
office
hours,
otherwise
known
as
echo
as
always.
Let
me
know
if
you're
watching,
I
always
love
to
know
that
there's
somebody
out
there.
I
always
enjoy
your
questions
as
well,
so
do
interact
type
in
some
comments,
I'll,
try
and
remember
this
time
that
I
can
actually
show
comments
on
the
screen
as
well
so
yeah
hoping
to
hear
from
you
soon.
A
We
have
a
document
for
every
episode,
so
you
are
more
than
welcome
to
go
in
there.
Click
on
the
links,
maybe
even
add
something
again.
If
we've
missed
something
you'll,
you
know
you're
great
you'll,
add
sensible
things
to
these.
These
show
notes,
I'm
sure
yeah.
So
this
week
I'm
going
to
be
exploring
the
psyllium
service
mesh
function
and
so,
let's
all
cross
our
fingers
for
the
demo
gods
to
be
with
us
today
for
that,
but
before
we
get
into
that,
let's
look
at
a
few
headlines.
A
So
a
few
things
that
have
caught
our
attention
this
week,
one
is
a
tool
called
ballast,
that's
been
described
by
uber
and
I
guess
released
by
uber,
and
this
is
kind
of
interesting
because
it's
using
bpf
it's
actually
using
p
cap
to
capture
traffic
and
then
use
that
traffic
to
kind
of
generate
load
testing.
So
I
think
if
we
we
look
in
here,
we'll
see
yeah
the
berkeley
packet
filter
and
I'm
sure,
if
you're
doing
load
testing
you
might
want
to
check
that
out.
A
Hi
to
evan
good
to
see
you
there
hi
all
right.
The
next
item
we
had
found
this
week
is
called
e-capture
and
this
it
reads
as
though
it
may
be
pretty
similar
to
ssl
sniff.
So
this
is
capturing
as
far
as
I
understand
it.
I
haven't
tried
it,
but
it's
capturing
traffic
before
it's
been
encrypted,
as
I
say
that
I'm
not
100
sure
if
it's
before
it's
been
encrypted
or
after
it's
been
decrypted
at
the
ssl
layer.
A
So
yeah
do
check
that
out.
There's
some
some
diagrams
here
which
can
help
us
to
understand
a
little
bit
if
our
chinese
is
not
up
to
reading
some
of
the
the
text
here,
but
I
think
we
can
see
this
is
using
ebpf,
it's
actually
using
the
syllabpf
library
by
the
looks
of
it.
So
that's
that's
great
might
be
one
to
try
out
and
compare
what
that
does
compared
to
ssl
sniff,
a
few
other
people
to
say
hello
to
hi.
A
To
I
don't
know
how
to
pronounce
this
I'm
going
to
go
with
bojan
shay
or
something
like
that.
I
apologize
if
I've
mangled
your
name
and
hello
to
moz
good
to
have
you
with
us.
Let's
see
what
else
we
have
on
our
show
notes
this
week,
yeah
we
have
egrets.
So
this
is
another
ebpf
based
tool
for
monitoring
egress
traffic.
A
A
And
I
also
put
in
a
link
here
to
the
third
episode
of
the
echo
newsletter,
so
if
you
haven't
already
signed
up
for
it,
there
is
a
link
to
that
in
the
show
notes-
and
this
has
been
compiled
by
my
colleague
bill
mulligan-
he's
doing
a
really
great
job
of
collecting
together
some
more
of
the
articles.
Some
of
them
will
have
referred
to
in
this
show.
Some
of
them
will
be
extra.
A
You
know
extra
details
that
you'll
get
from
this
newsletter,
so
you
can
see,
for
example,
e-capture
and
egrets
have
been
covered
in
that
that
last
episode
or
that
last
edition
of
the
echo
newsletter
so
yeah.
If
you
can't
get
enough
about
ebpf
sign
up
for
echo
news
and
you'll,
get
that
in
your
inbox
every
couple
of
weeks
or
so.
A
Okay,
so
my
my
plan
today
is
to
talk
a
bit
about
cluster
mesh
and
then
we'll
try
it
out.
I
have
got
a
couple
of
clusters
up
and
running
so
we'll
we'll
try
connecting
them
together
using
cluster
mesh,
so
I've
linked
a
couple
of
articles
here,
so
one
of
them
is
the
the
documentation
which
we'll
come
to
in
a
minute,
but
there's
also
a
blog
post.
A
Cluster
mesh
is
nearly
three
years
old
now
and
but
I
think
this
is
still
a
really
helpful
introduction
to
what
cluster
mesh
is
and
well
really
why
it's
so
helpful
if
you
want
to
share
services
across
multiple
kubernetes
clusters,
so
I'm
going
to
just
scroll
down
here
to
this
diagram.
I
think
it
makes
it
really
very
clear,
what's
happening
in
cluster
mesh,
so
you
might
have
a
number
of
pods
that
make
up
a
service.
A
And,
let's
just
forget
the
multi-cluster
for
a
moment
the
if
we
just
had
this
service
being
supported
by
these
two
back-end
pods,
the
front-end
would
make
a
request
to
back-end
service
and
it
would
be
load
balanced
between
those
two
with
cluster
mesh.
What
we
can
do
is
run
the
same
service
in
multiple
clusters,
advertise
them
as
a
global
service,
and
then
it
means
that
a
pod
in
one
cluster
can
reach
that
service
in
either
of
the
connected
clusters.
A
So
in
this
example
here
we
can
see
it's
really
useful
for
resiliency.
If
something
terrible
happens
to
the
back
end
pods
in
this
cluster,
the
front
end,
requests
to
the
service
can
be
routed
to
the
the
other
cluster,
so
that
could
be
really
useful
if,
for
example,
maybe
this
this
cluster
is
running
out
of
resources,
perhaps
these
back
end
services
got
preempted
for
some
reason,
you'd
still
be
able
to
share
that
service
across
across
those
clusters.
A
So
I
think
we
should
talk
a
little
bit
about
how
we
set
that
up.
In
fact,
no,
let's
look
at
my
look
at
my
clusters.
Let's
see
if
they're
still
there,
so
I
have
a
couple
of
clusters
called
there's
cluster
mesh,
one
and
glues
cluster
mesh
two,
and
I
can
do
things
like
sodium
status
and
if
I
do,
I
can't
quite
remember
what
my
context
is
called.
Let
me
just
check
okay
name.
There
we
go
yeah,
so
I
can
do
psyllium
status
context.
A
A
A
A
A
We
should
see
this
is
actually
going
to
be
in
the
context
one.
We
can
see
this
cluster
mesh
api
server,
that's
been
exposed
in
the
cluster
and
that's
how
our
two
clusters
are
going
to
be
communicating,
there's
a
similar
service
on
the
on
the
other
cluster.
So
if
I,
if
I
did
the
same
thing
on
context
context,
let's
spell
context
correctly.
A
So
we
can
see
an
external
we've
got
a
load
balancer
on
these
two
clusters
on
each
of
these
two
clusters,
so
these
they
can
talk
to
each
other
and
the
way
this
works.
I
wonder
if
I
can
find
a
good
picture.
I
don't
know
if
there
is
a
picture
in
this
yeah
here
we
go
so
the
way
this
is
working.
Is
you
probably
know
that
the
kubernetes
configuration
is
stored
in
xcd
and
that's
going
to
include
information
about
the
services?
A
What
we
do
with
cluster
mesh
is
we
run
these
xcd
proxies
that
can
look
for
interesting
information
that
needs
to
be
shared
across
multiple
clusters
and
essentially
use
these
external
load
balancers
to
communicate
that
information
so
with
cluster
mesh.
What
we're
going
to
do
is
mark
certain
services
as
global
I'll
do
this
in
a
second,
when
we
mark
them
as
global
they're,
going
to
be
sort
of
advertised,
if
you
like,
through
this,
the
cluster
mesh
service,
okay.
A
A
A
That
means,
if
you
have,
for
example,
clusters
in
two
different
service
providers,
you're
going
to
need
to
make
sure
there's
some
kind
of
connectivity
between
them,
maybe
you're
going
to
set
up
a
vpn
tunnel
or
something
between
the
networks
where
those
two
clusters
are
running
or
those
multiple
clusters
are
running
and
you
have
to
be
able
to
communicate
across
that
network.
You
can't
have
firewalling
blocking
that
communication.
A
A
A
A
A
A
A
A
I
don't
think
I
do
okay,
so
we
need
to
actually
deploy
some
sample
services
and
there
is
an
example
in
this
section
of
the
docs
load,
balancing
and
service
discovery,
and
this
is
the
key
thing
we
need
to
do
in
a
service
that
we
want
to
share
across
multiple
clusters.
We
need
to
deploy
it
into
both
those
clusters
and
we
need
to
mark
it
as
a
global
service.
A
A
A
A
A
A
A
A
A
A
A
Hopefully
I
can
also
curl
to
the
rebel
base
from
here,
and
hopefully
it
will
occasionally
go
on
show.
Yes,
there
we
go
sometimes
just
one
sometimes
clustered
two,
so
our
x-wing
fighters
are
kind
of
acting
like
that
front
end
that
we
saw
in
the
diagram
and
they're
able
to
send
requests
to
the
service
and
have
those
responded
to
by
the
back
end
pods
the
rebel
base
on
either
cluster
one
or
cluster
two,
so
that
actually
seemed
to
work.
That's
all
good
with
the
demo
gods.
A
Thank
you,
demigods,
okay,
so
good
question
here
from
moz:
does
cluster
mesh
help
with
the
replication
of
stateful
set
workloads
on
different
clusters?
So
it
certainly
means
that
you
can
run
the
workloads
across
different
clusters.
Now
I
think
it
might
be
in
the
same
blog
post,
I've
seen
a
really
interesting
diagram
is
it
here?
A
A
Applications
as
possible
in
into
let's
say
the
majority
of
your
clusters,
but
have
dedicated
clusters
for
your
stateful
storage.
So,
for
example,
you
could
be
running
these
clusters
in
different
cloud
providers.
You
could
be
running
them
with
different
availability
zones,
for
example,
but
your
data,
if
you've
got
a
significant
amount
of
data
that
data
has
to
be
stored
on
a
disk
somewhere
and
it
has
gravity
data
gravity.
A
A
A
A
A
A
A
A
A
A
A
A
Yeah,
so
this
would
be
really
helpful
would
be
if
I
could
show
exactly
which
pod
this
was
coming
from,
but
I
don't
think
I
quite
have
that
capability
today,
but
yeah
you
can
see
it's
still
load
balancing
across
those
two
clusters.
I
think
that
it
is
not
taking
into
account
anything
like
the
number
of
pods.
Somebody
can
correct
me
if
I'm
wrong,
but
I
don't
think
you
know
the
fact
that
there
are
five
pods
in
one
cluster
and
two
in
another
is
going
to
make
a
difference.
A
A
A
A
So
here
is
an
example
of
allowing
x-wings
in
one
cluster,
so
in
cluster,
one
to
contact
the
rebel
base
in
cluster
two
and
the
way
policies
work
they're.
If
you
have
no
policies,
no
network
policies
deployed
at
all,
then
all
communications
are
permitted,
but
as
soon
as
you
have
a
policy
in
place,
anything
that
isn't
explicitly
allowed
by
that
policy
is
not
allowed.
A
A
So
I
wonder
if
we
can.
Oh,
this
is
a
another
interesting
point
to
note.
The
security
policies
are
not
automatically
distributed
across
clusters,
so,
in
the
same
way
that
you
have
to
mark
a
services
global
in
both
clusters,
you
would
also
have
to
apply
a
psyllium
network
policy
or
a
kubernetes
network
policy
in
all
the
clusters
where
you
wanted
it
to
affect.
A
A
A
A
A
A
A
A
A
A
A
Resolves
r,
okay,
good,
so
we
are
now
getting
responses
from
cluster
two,
which
is
what
we
defined
our
our
network
policy
to
permit.
We
were
going
to
allow
the
x-ring
in
cluster
1
to
contact
the
rebel
base
and
cluster
2,
and
that
has
successfully
worked,
but
I
suspect
that
if
it
happens
to
load
balance
to
a
service
in
cluster
one
that
packet
is
actually
being
dropped
because
that's
the
way
that
network
policy
works
okay,
so
I
expect
that
this
is
eventually
going
to
time
out
because
it
was
trying
to
reach
cluster
one.
A
So
I
think
I
need
to
think
a
bit
harder
and
probably
consult
with
some
colleagues
about
what
the
best
practice
is.
If
you
want
to
have
a
local
service,
but
not
permit
traffic
to
it
because
dropping
packets
like
this,
is
you
know
if
we
happen
by
chance
to
send
the
request
to
cluster
two
things
are
great
if
it
happens
to
go
to
cluster
one
things
are
not
so
great,
so
yeah
that
really
needs
a
bit
of
thinking
about.
A
Okay.
So
if
I'm
not
mistaken,
what
we're
doing
now
is
the
same
with
existing
proxies.
Is
there
any
performance
benchmark
available
against
istio
and
envoy,
and
so
I
think,
there's
a
couple
of
things
here.
One
is
that
this
is
connecting
multiple
clusters,
so
I'm
not
doing
a
service
mesh,
I'm
just
using
regular
kubernetes
services,
but
distributing
them
across
multiple
clusters.
So
there's
no
there's
no
service
mesh
kind
of
layer
in
this
particular
example
that
I'm
showing
today.
A
I
think
that
would
be
actually
an
interesting
thing
to
look
at.
You
know
in
a
future
to
see
how
service
mesh
and
cluster
mesh
can
can
work
together,
but
that's
not
what
we're
doing
today.
I
guess
the
second
part
of
that
that
question
is:
are
there
any
performance
benchmarks
available
against
istio
and
envoy,
so
one
of
the
I
don't
know,
swings
and
roundabouts
when
you
have
a
service
mesh?
A
A
So
in
the
traditional
service
mesh
model,
I've
got
envoy
here.
It
wouldn't
necessarily
have
to
be
envoy,
but
it's
a
very
popular
proxy
for
service
meshes
and
when
you
deploy
envoy
as
a
sidecar,
it
has
to
be
injected
into
every
pod.
That's
part
of
your
service
mesh,
and
that
has
a
few
consequences.
One
of
them.
You
can
kind
of
see
in
this
diagram
that,
if
you
imagine
a
packet
coming
into
the
host,
it
has
to
go
through
the
host's
networking
stack.
Then
it
can
be
sent
across
the
virtual
ethernet
connection
to
a
pod.
A
It
knows
to
then
forward
that
traffic
onto
the
application
it
has
to
go
back
through
the
network
stack
within
the
pods
namespace
to
to
do
this
and
the
proxy
is
is
necessary.
The
proxy
is
there
to
kind
of
terminate
layer,
7
and
so
on.
So
you
know
we
want
the
proxy
there
in
order
to
offer
service
mesh
capabilities,
but
it
doesn't
necessarily
have
to
be
inside
the
pods
namespace
if
we're
using
ebpf,
and
so
one
of
the
things
that
we're
doing
with
citium
service
mesh
is
bringing
envoy
onto
the
node.
A
A
So
we
are
working
on
some
performance
comparisons.
I
mean
the
early
measurements
have
looked
really
good,
but
we
haven't
published
them.
Yet
we
don't
want
to
publish
some
reproducible
benchmarks
so
that
you
don't
have
to
take
our
word
for
it,
that
you
know
you'll
be
able
to
go
and
try
out
cilium
service
mesh
performance
for
yourself.
A
So
I
guess
that
was
a
little
bit
of
an
aside
talking
about
service
measure
other
than
cluster
mesh.
I
guess
the
word
mesh
is
doing
a
lot
of
abuse
today.
Okay,
let's
come
back
to
that
example.
Yes,
so
eventually
it
did
fail
to
connect
to
the
rebel
base,
because
the
connection
timed
out
and
afsal
has
asked
question
why
the
packet
being
dropped
is
well.
I
think
this
is.
Let
me
try
and
explain:
what's
happened
here,
so
I've
put
in
place
a
network
policy
that
says
we
can
only
send
packets
from
the
rebel
base.
A
A
I
hope
that
explained
what
was
happening
there
hi
to
yaron,
who
said
it
missed
the
cluster
mashed
potato.
I
didn't
realize
it
was
a
potato
okay,
I
think
that's
a
whole
other
custom,
crd
that
we
haven't
created
yet,
but
we'll
have
to
think
about
it.
Maybe
that's
something
for
april
the
1st
okay.
So
if
you
have,
I
mean
we're
coming
up
to
the
end
of
our
hour.
So
if
you
do
have
other
questions
about
cluster
mesh,
now
is
a
good
time
to
type
them
in.
A
A
Thank
you
all
for
your
excellent
questions
and
your
help
with
scaling
up
the
number
of
pods
behind
the
deployments,
you're
all
ready
for
your
cka.
As
far
as
I'm
concerned,
brilliant,
to
see
you
all
s,
I
will
be
back
in
two
weeks.
Time
do
join
duffy
next
week
and
we'll
see
you
all
again
soon.
Bye
now.