►
From YouTube: Service MESH without the MESS by Raymond de Jong
Description
Service meshes are becoming the secure, observable networking layer for distributed computing systems like Kubernetes. However, they are also known for their operational complexity and steep learning curve. This talk will help clear up the mess around service mesh. We will start by introducing what a service mesh is intended to do before diving into hands-on demos using Cilium Service Mesh powered by eBPF. The audience will learn how to monitor service-to-service connectivity, collect tracing data and golden metrics using standard Prometheus, Grafana, and OpenTelemetry with eBPF. The talk will close by discussing how eBPF eliminates service mesh sidecars to improve performance and reduce latency, operational complexity, and resource usage. By the end, the audience will be able to understand and implement a service mesh rather than mess.
A
Yeah
here
we
are
again
good
morning.
I
think
I
think
that
the
slide
is
misleading.
It
can
be,
there
can
be
a
mesh
without
a
mess,
but
Raymond
is
I'm.
Sure
Raymond
DeYoung
from
celium
is
going
to
show
us
it's
going
to
prove
me
wrong
Raymond.
Thank
you
give
it
up
for
him.
Please.
Thank
you.
Thank
you.
B
Good
luck.
Thank
you
good
morning.
Everyone,
my
name
is
Raymond
young
I'm
Dutch,
obviously
of
my
name.
I
work
for
ISO
valence
ice
surveillance
is
the
company
who
originated
psyllium.
How
many
of
you
know
psyllium
good?
How
many
of
you
know
what
ebpf
is
or
ebpf
in
general
cool
good
at
Aisha
fail
at
I'm
field,
CTO
I'm,
here
happy
to
present
about
psyllium
surface
mesh
without
the
mesh
and
I
hope.
I
can
explain
why
that
is
so.
B
First
of
all,
I
want
to
start
for
the
people
who
don't
know
so
much
about
ebpf
and
silly
and
what
these
are,
after,
which
I'm
going
to
talk?
How
surface
mesh
has
evolved
and
then
look
at
a
few
feature.
Psyllium
service
mesh
is
offering
today,
and
hopefully
I
have
time
for
a
small
demo.
So
you
can
see
it
in
action.
B
So
evpf,
what
is
evpf
ebpf
stands
for
external,
extended,
Berkeley
packet
filter,
and
that
doesn't
mean
anything.
It
doesn't
mean
a
lot,
but
how
we
use
it
is
ebpf
makes
the
kernel
programmable
in
a
dynamic
way.
We
like
to
say
what
JavaScript
is
through
your
browser.
Ebpf
is
to
the
kernel,
and
what
that
means
is
that,
based
on
kernel
events,
we
can
do
a
number
of
things
with
that.
So
today
we're
focusing
obviously
on
service
mesh,
so
we're
looking
at
a
lot
of
networking
capabilities.
B
Also,
on
the
runtime
side,
we
can
also
use
ebpf
to
secure
workloads
and
to
be
able
to
see,
for
example,
file
access
processes
being
executed
and
secure
and
enforced.
There
psyllium
uses
evpf
to
basically
provide
Advanced
networking
capabilities
today,
we're
looking
mostly
at
distributed,
applications
where
surface
mess
is
at
play,
but
you
can
also
use
psyllium
Standalone
for
high
performance
load,
balancing,
for
example,
using
ebpf.
We
can
accelerate
the
data
path
using,
for
example,
XDP.
So
synonym
is
a
broad
solution
for
networking,
observability
and
security.
B
B
And
finally,
we
also
have
you
know
things
like
egress,
Gateway,
bgp
overlay,
networking
on
the
networking
side.
On
top
of
that,
we
have
an
observability
layer
with
Hubble
that
allows
us
to
get
this
visibility
at
layer,
3,
layer,
4
layer,
7
to
see
what's
actually
going
on,
so
we
can
see
using
Hubble
using
the
UI
or
using
metrics.
B
B
On
top
of
that
is
right
now
psyllium
surface
mesh
with
Gateway
API
Ingress,
where
we
can
support
traffic
management
and
are
also
supporting
Gateway
API
as
such,
and
on
the
right
side
we
have
tetragon,
which
is
our
runtime
solution,
which
is
a
session
on
its
own.
I.
Won't
talk
a
lot
about
that
today
and
celium
can
run
on
any
Cloud.
It
can
run.
On-Prem
can
run
on
openshift
gke
enthus
uses
celium
under
the
hood.
B
Azure
is
currently
adopting
ceiling
and
default
data
plane
for
AKs
clusters,
eks
anywhere
uses
celium
by
default,
and
we're
hoping
that
AWS
is
also
switching
in
the
future
to
use
Celia
by
default
as
well
for
eks
clusters,
and
it
doesn't
matter
if
it's
on-prem
or
in
cloud
or
both,
and
you
can
connect
them
together.
Using
multi-cluster
capabilities.
B
So
today
we
talk
about
service
mesh.
So
where
did
service
mesh
came
from
when
looking
at
distributed
applications?
We
want
to
see
how
they
perform,
so
we
need
observability.
We
want
to
see
how
long
a
duration
is
taking.
We
want
to
see
the
return
codes.
We
want
to
effectively
troubleshoot
it.
We
also
want
to
secure
it
either
in
transit,
and
we
also
want
to
do
authentication.
Perhaps
we
want
to
use
mtls
to
authenticate
Services
talking
to
each
other.
B
We
also
want
to
do
layer,
7
traffic
management.
Perhaps
you
want
to
do
path,
redirection
or
TLS,
decryption
and
such
and
finally,
we
want
resilience.
We
want
to
have
availability
across
clusters,
even
across
clouds,
if
possible,
so
where
service
mesh
obviously
started
was
with
binaries
right
applications
were
having
codes
to
provide
that
kind
of
capabilities
at
scale
that
doesn't
work.
It's
hard
to
maintain.
B
So
today,
the
only
thing
which
is
not
yet
there
is
layer
7,
mostly
we
all
ready
for
a
few
years,
already
support
the
layer,
two
layer,
free
layer,
4
capabilities
in
kernel
using
psyllium
using
evpf,
for
example,
the
East
West
low
balancing
I
just
mentioned,
using
a
q
proxy
replacement,
that's
done
through
ebpf,
instead
of
iptables,
so
layer
7.
It
was
also
questioned
this
morning.
Ebpf
has
limitations
right
and
that's
for
good
reason
that
it
runs
in
kernel
right.
B
So
you
have
to
have
constraints
and
limitations
of
what
you
are
allowed
to
do
and
today,
for
example,
something
like
HTTP,
5,
redirection
or
TLS.
Decryption
as
such
is
costs
too
much
of
code
and
processing
to
do
in
ebpf
today,
but
it
might
happen
in
the
future.
But
that's
the
reason
why
layer
7
is
not
yet
there.
B
Although
psyllium
already
has
all
these
layer
7
capabilities,
we
already
see,
for
example,
using
network
policies,
we
can
enforce
traffic
on
the
API
layer.
We
can
see
HTTP
calls
methods,
return
codes,
we
can
load
balance
and
Route
grpc.
We
can
observe
traffic
using
grafana
and
Hubble
UI.
So
cilium
already
has
these
capabilities
and
we're
extending
these
capabilities
using
surface
mesh.
B
So
what
happens
with
service
mesh
so
as
soon
as
you
trigger
or
enable
a
service
mesh
capability?
That
means
that
on
the
Node
for
that,
given
namespace,
we
create
a
listener.
So
Envoy
has
specific
listeners
in
a
namespace
context
to
do
layer,
7,
load,
balancing
or
path
redirection,
for
example,
and
how
is
this
different
compared
to
other
service
meshes
well?
First
of
all,
we
want
to
reduce
operational
complexity.
We
want
to
make
it
easier.
You
already
need
a
cni.
B
You
may
most
likely
perhaps
already
use
celium
and
you
can
just
enhance
zillion
by
enabling
surface
mesh
and
you
don't
need
an
external
Ingress
or
Gateway
API
controller
also
goal
is
to
reduce
reduce
resource
usage.
So
having
not
running
sidecars
means
that
you
will
save
a
lot
of
resources
in
your
cluster.
It
will
provide
better
performance
for
a
number
of
reasons,
and
we
want
to
avoid
sidecar
shutdown,
startup,
latency
or
race
conditions.
So,
let's
dive
in
on
a
few
of
those
use
cases.
B
If
we
look
at
the
resource
usage,
obviously
with
sidecars,
you
have
for
each
spot,
where
you
enable
or
each
surface
where
you
enable
surface
mesh,
for
you
will
have
a
sidecar
running.
That
means
that
at
scale
this
means
a
lot
of
memory,
a
lot
of
CPU
and
also
a
lot
of
TCP
connections
being
tracked
for
those
sidecars,
so
that
will
cause
resources
moving
at
toward
the
kernel,
just
frees
up
a
lot
of
resources.
B
Obviously,
it's
not
free,
and
but
using
ebpf,
where
we
can
and
centralizing
resources
as
efficient
as
possible,
will
save
you
a
lot
of
resources
in
the
end,
there's
also
another
cost
of
sidecar
injection,
and
that
is
the
induced
latency
when
using
sidecars.
So
when
you
see
this
diagram,
an
app
is
trying
to
send
traffic
to
another
app
or
service.
That
basically
means
that
it
goes
three
times
through
the
TCP
stack
once
sending
the
packet
then
receiving
on
the
sidecar
side.
B
Processing
it
in
the
sidecar,
then
sending
the
packet
on
the
wire
and
the
same
happens
on
the
destination
again,
and
you
can
see
this
Loop
takes
time
and
that
will
induce
latency
how
we
solve
this
using
ebpf-
and
this
is
already
there
for
years-
is
that
using
ebpf
we
can
direct
directly
forward
traffic
from
the
socket
layer
to
the
to
the
network
interface
without
traversing.
This
DCP
IP
stack.
So
this
is
an
example
where
we
only
need
layer,
3
or
layer,
4,
load,
balancing.
B
We
don't
need
to
go
through
the
proxy
and
we
can
directly
do
using
a
q
proxy
replacement.
Do
our
load
balancing
and
send
it
on
The
Wire
in
case
we
need
layer,
7
processing.
Then
we
need
to
redirect
it
to
the
proxy
yes
on
the
Note,
but
we
do
it
through
the
socket
layer,
so
we're
avoiding
this
tcpip
stack
and
directly
forward
it
to
the
proxy
and
once
the
proxy
has
done
is
that
is
its
job.
It
will
forward
it
to
the
data
plane.
B
This
improves
latency
a
lot
in
our
measurements,
two
three
and
sometimes
four
times,
depending
on
what
you
use.
If
you
use
a
lot
of
network
policies
or
a
lot
of
layer,
7
path
redirections
here,
you
can
see
it
compared
to
SEO
based
implementation.
Avoiding
this
loop
on
your
TCP
stack,
improves
performance.
A
lot,
and
it
also
improves
throughput
slightly
and
also
thing
to
consider-
is
that,
obviously
at
scale
when
you're
scaling
out
your
applications.
B
That
also
means
with
the
sidecar
implementation,
that
you
have
to
wait
for
that
side,
guard
to
be
ready
to
receive
connections,
to
be
able
to
process
your
service
and
be
able
to
serve
your
application
with
cilium
service
mesh.
That
agent
runs
on
the
Node,
it's
ready
to
receive
connections.
It
creates
this
listener
and
it's
happy
to
serve
connections
when
it
needs
to.
B
So
today,
psyllium
service
Mass
can
do
a
number
of
things.
Traffic
management
is
first
for
layer,
7
traffic
management
path
to
redirection.
What's
what
we
are
looking
at
today,
observability
using
Hubble,
you
can
export
metrics
to
grafana,
which
gives
you
visibility
of
performance
of
your
application,
enforcing
security
through
cilium
Network
policies
by
itself.
It's
already
there
for
years
and
now
using
service
mesh,
we
can
extend
that
capability
and
resilience
through,
for
example,
cluster
mesh
topologies
and
such
we're
not
planning
to
develop
a
control
plane
on
our
own.
B
So
you
can
use
Ingress
resources
or
Gateway
API
resources.
Spiffy
is
on
the
roadmap.
We
already
have
the
data
plane
ready,
but
the
control
plane
needs
to
be
developed
as
such
and
for
power
users.
Today
we
can
also
do
and
advanced
Envoy
custom
resource
definitions,
but
with
Gateway
API.
My
my
vision
is
that
you
don't
need
death
anymore.
B
On
the
observability
side,
you
can
hook
up,
obviously
Prometheus
grafana.
You
can
instrument
your
applications
using
open,
Telemetry
using
evpf.
We
can
export
those
metrics
in
the
most
efficient
way
as
possible,
and
obviously
you
can
export
flows
and
data
to
simplatforms
fluid
D
or
elasticsearch.
Now,
if
you
want
to
get
started
with
psyllium,
you
have
two
choices.
B
Maybe
you
already
have
a
sidecar
implementation
based
on
istio
I,
strongly
encourage
to
also
run
psyllium
on
those
clusters,
because
when
you
look,
for
example,
at
the
sidecar
implementation
and
you're
thinking
about
mtls
between
sidecars
for
authentication
authorization
purposes,
what
a
lot
of
people
don't
know
is
that
the
actual
connection
between
the
sidecar
and
the
actual
Port
as
a
destination
is
unencrypted
using
psyllium,
because
we
shortcut
this
connection
on
layer
four.
We
avoid
that
this
connection
is
actually
going
through
a
virtual
interface
and
no
one
can
listen
to
that
traffic.
So
already,
there's
a
benefit.
B
So
last
year
we
released
psyllium
bundle
12,
which
meant
that
we
released
a
production
ready,
psyllium
surface
mesh
with
a
psyllium
Ingress
controller
conformant
using
kubernetes
increased
controller
resources.
You
can
use
kubernetes
as
your
control
plane,
simple
as
that
we
already
have
Prometheus
metrics,
grafana
dashboards
and
for
power
users,
the
celium
cluster,
Envoy
config
or
psyllium
Envoy
config,
and
last
week
we
released
psyllium
1.13,
which
enabled
Gateway
API
with
HTTP
routing
TLS
termination,
traffic,
splitting
and
header
modification.
B
We
have
a
shared
low
balance
for
Ingress
resources,
which
I
will
talk
a
bit
today
as
well
layer,
7,
load,
balancing
for
kubernetes
services
with
annotations,
so
plain
plus,
IP
Services.
You
can
use
for
low
benefit
purposes
and
a
ipam
solution
for
low
balance
of
service
and
bgp
advertisement
in
the
cloud
that
you
don't
need
it.
But
if
you
run
on-prem,
you
most
likely
need
to
connect
your
load,
balancer
IPS
to
the
network
or
advertising
through
bgp.
This
makes
it
really
easy
and
flexible
to
advertise
multiple
pools
of
ips
to
your
network.
B
So
let's
have
a
look
on
those
features,
so
for
starters,
Ingress.
This
is
just
following
the
kubernetes
Ingress
spec.
You
don't
need
an
external
Ingress
controller,
and
what
you
will
notice
is
that
you
need
to
specify
an
Ingress
class
name
as
cilium,
and
you
need
to
enable
psyllium
on
the
helm,
search
which
I
will
look
a
bit
later
on
as
well,
and
that's
basically
it
you
can
do
path.
B
Redirection
in
this
example
to
forward
traffic
to
a
specific
surface-
and
this
is
a
common
example
where
we
have
an
book
info
application,
where
we,
for
example,
forward
the
the
default
URL
to
the
product
page
and
the
details
URL
to
a
specific
service.
For
that
purpose,
we
can
also
do
Ingress
for
grpc,
so
we
can
specifically
forward
traffic
from
specific
dupc
calls
to
specific
surfaces,
and
we
can
also
do
TLS
termination
using,
for
example,
a
secret.
You
have
created
to
do
the
TLs
decryption
at
the
Ingress
and
then
forward
it
to
the
services.
B
B
Each
one
will
have
a
dedicated
load
balancer,
but
with
ceiling
1.13
you
can
specify
it
as
shared
using
an
annotation,
and
then
you
can
have
a
shared
single
load
balancer
to
do
have
a
lot
of
ingresses
running
on
top
of
that
Gateway
API
yesterday
was
an
excellent
session
by
our
for
you,
your
for
you
for
view.
Your
summary
about
Gateway
API
in
general
celium
already
supports
Gateway
API,
so
we
can
have
Gateway
apron
resources,
HTTP
routes.
B
How
that
looks
like
is
instead
of
if
you
on
the
look
on
the
left,
we
specify
a
Gateway
resource
of
Gateway
class
cilium,
and
then
you
create
one
or
multiple
listeners
for
the
specific
Gateway,
and
then
you
attach
HTTP
routes
where
you
specify
the
parent
gateway
to
link
it
to
that
specific
Gateway.
And
then
you
can
do,
for
example,
path,
matching
and
redirects
as
such.
B
B
What
we
also
enabled
is
layer,
7
load
balancing
for
cluster
IP
cumulative
surfaces
with
simple
annotations
without
sidecars.
So
that
means
that
you
can
specify
that
annotation
on
a
given
surface
as
shown
here
and
you
label
it
as
service.cellium.io
label,
lb,
Dash,
lc7
enabled
that
enables
that
feature,
and
then
you
can
specify
an
algorithm
so
in
this
case
lease
request,
but
we
also
support
round
robin
or
random,
and
this
allows
you
to,
for
example,
on
the
surface
level,
low
balance,
grpc
or
HTTP
traffic
as
such.
B
This
is
also
compatible
with
cluster
mesh,
so
cluster
mesh
allows
you
to
have
two
or
more
clusters
together
to
create
a
single
psyllium
identity.
Aware
data
plane,
on
top
of
it,
you
can
use
surface
mesh
in
those
clusters.
You
can
use,
for
example,
Ingress
or
Gateway
API
to
attract
traffic
inwards,
your
cluster,
but
then
you
will
forward
it
to
a
service
and
using
the
surface
you
can
also
decide.
I
want
this
specific
service
to
be
high
available
across
clusters,
so
Additionally
the
layer,
7
load,
balancing
capabilities.
B
We
can
also
load
balance
it
across
clusters,
so
super
flexible
and
Powerful.
You
can
mix
and
match
both
Ingress
Gateway
API
Resources
with
surface
resources
and
low
balance,
where
you
need
them
to
be
and
finally
observability,
so
we're
partnering
with
rafana.
That
means
that
we
are
creating.
As
of
today,
as
we
speak,
a
lot
of
new
dashboards.
You
may
already
be
aware
of
the
day
two
operational
dashboards.
We
have
already
available
on
the
grafana
marketplace,
to
Monitor
and
to
see
how
your
cost
is
behaving,
but
we're
expanding
these
dashboards
to
provide
meaningful
layer.
B
7
golden
signal,
metrics
based
dashboards
and
those
data
is
using
is
exported
through
ebpf
and
we
are
also
exporting
to
capabilities
in
grafana
such
as
mimir
Loki
and
Tempo
Tempo.
So
if
you
instrument
your
application
using
open,
Telemetry
using
ebpf,
we
can
extract
that
data
and
we
can
export
it
to
Tempo,
where
you
will
see
the
traces
and
you
can
see
exemplars
and
how
the
different
specific
span
is
taken,
and
this
is
One
dashboard.
B
This
can
be
a
session
on
its
own
as
well,
but
what
we
are
seeing
here
without
sidecars
using
psyllium
surface
mesh,
we're
already
able
to
see
HTTP
codes,
we're
already
able
to
see
latency.
We
can
see
request
duration,
we
can
see
we
can
detect
latency
as
such,
and
this
is
without
any
side
guard
without
instrumentation
of
your
application.
It's
already
there
because
we
have
this
layer,
7
visibility
with
network
policies
and
with
Hubble
metrics.
B
That
also
means
that
we
already
have
visibility
on
the
network
layer.
If
you
think
your
application
is
performing
fine.
We
also
have
this
visibility
in
the
network
layer
where
we
can
see
how
many
bytes
are
sent.
We
can
see
re-transmissions,
and
we
can
also
see
a
round
trip
round
round
trip,
time,
perhaps
being
increasing,
which
can
point
to
a
specific
Network
issue
or
specific
node
issue.
B
So
I
hope
I
have
a
little
bit
time
for
a
small
demo
to
show
how
this
actually
looks
like
in
for
real,
so
I
hope
in
the
back.
I
checked
it
a
bit,
it's
readable
in
the
back
good,
so
how
to
get
started
with
cerium.
Let
me
just
show
an
example:
I'm
running
this
demo
on
gke,
so
what
you
need
to
do
is
you
need
to
enable
an
Ingress
controller.
If
you
want
to
use
the
Ingress
resource
using
cilium
optionally,
you
can
enable
the
Hubble
metrics.
So
I
have
a
lot
of
examples
here.
B
This
is
all
documented.
You
can
find
all
these
examples
there,
but
what
I'm
wanting
to
show
and
what
I'm
wanting
to
see
is
DNS
drops
TCP,
HTTP
traffic
and
for
surface
mesh.
What's
important
is
that
you
also
enable
Gateway,
API
enables
equals
true
and
finally,
the
cube
proxy
replacement.
This
setting
has
to
be
set
to
to
strict
or
partial.
We
do
recommend
strict.
That
has
the
best
compatibility
and
performance
for
your
clusters.
B
B
So
now
you
see
my
environment
where
I'm
running
lgke
I'm,
hopefully
I
have
connection
still
it's
it's
a
bit
like
delaying.
Let
me
see
yeah
so
I'm
in
the
book
info,
namespace
I'm,
doing
a
a
cube,
CTL
get
services,
so
here
I
can
see
I
already
created
the
the
Gateway
I
can
also
do
Cube
CTL
gets
gateways.
B
This
should
be
able
yeah,
so
I
have
a
book
info
HTTP
and
https
Gateway
and
I
get
a
public
IP
from
gke
to
reach
it
from
the
outside
world
on
the
side
of
the
application.
I
also
have
this
HTTP
route,
so
I
have
the
details,
path,
forwarding
to
the
detailed
service
and
the
default
path
forwarding
to
the
product
page.
So
using
the
public
IP
I
should
be
able
to
connect
to
it.
Yes,
I
can
and
if
I
go
to
oops.
If
I
go
to
slash
details,
details
I
should
be
able
to
query
a
specific
entry.
B
So
that
means
that
if
I
refresh
this
on
https
I've
installed
this
certificate
on
my
laptop
as
well
so
I
trust
it
as
such.
So
the
connection
is
secure
and
going
to
this
seems
to
work
as
well.
So
that's
great
and
finally,
as
a
quick
example,
I
want
to
show
a
blue
green
Gateway.
So
in
this
example
you
do
a
blue
green
deployment
or
Cannery
release
using
weights.
Again,
you
create
a
gateway.
B
B
B
Back
to
the
slides
all
right,
so
that
concludes
my
demo
and
my
presentation
for
today.
If
you
want
to
know
more
and
try
it
out
yourself
feel
free
to
go
to
psyllium.io.
There's
a
lot
of
getting
started
guides
also
with
the
documentation
about
surface
mesh,
Gateway
API
with
examples
you
can
try
out
yourself
join
us
on
the
psyllium.select.com
channel.
If
you
have
questions
or
feature
requests
or
would
like
to
participate
or
learn
more.
What's
also
great
is
on
the
eyes
of
Fate
and.com
forward,
slash,
Labs
URL.
B
B
You
can
even
debug
things
if
you
like,
if
you
want
to
know
more
on
ebpf,
feel
free
to
go
to
ebpf.io
and
obviously,
if
you
want
to
know
more
about
Azure
valence
and
what
we're
working
on
or
want
to
look
for
careers
opportunities,
please
go
there
and
we're
hiring
so
I
have
a
have
a
look
there
and
feel
free
to
ask
me
any
questions
you
like
and
I'm
happy
to
stay
around.
Thank
you.
Thank
you.
A
B
A
A
The
the
second
one
is
we're:
gonna
have
some
q
a
with
Raymond.
Afterwards
we
are
going
to
have
a
little
break
and
lunch,
so
you
have
you
guys,
have
some
time
to
stretch
your
legs
and
to
speak
with
our
great
sponsors.
Please
speak
with
them.
The
lunch
is
going
to
be
served
in
the
sponsor
area,
and
that's
it
we're
going
to
restart
with
a
person
that
actually
very,
very
dear
to
me
my
brother,
my
brother
luconda,
is
going
to
start
according
to
here
now,
questions
for
Raymond.
C
Hello,
hey.
Thank
you
very
much
for
the
presentation.
I
would
like
to
ask
you
regarding
the
possibility
of
the
multi-cluster,
because
you
show
that
there
is
option
to
set
up
the
global,
but
I
don't
really
get
like.
How
do
you
do
that?
Do
you
have
a
separated
installation
for
Cinema
outside
the
Clusters
connected
to
multiple
cluster
audio
data,
yeah.
B
So,
first
of
all,
you
need
to
meet
a
few
requirements,
so
let's
say
you're
running
two
clusters
on
AKs
in
different
zones
or
regions
whatever,
and
you
want
to
connect
them.
You
need
to
have
layer
free
connectivity
between
them,
either
through
Gateway
or
or
through
VPN,
and
so
the
nodes
can
reach
each
other
and
that's
the
API
server
is
available
in
each
cluster
to
listen
to
requests.
Read
only
requests
from
another
cluster
once
you've
done
that
you
can
either
use
the
psyllium,
CLI
or
specific
Helm
values
to
enable
cluster
mesh.
B
And
what
it
looks
like
is
that
you
basically
create
a
unique
ID
in
each
cluster.
You
will
specify
the
main
CA
to
send
to
sign
certificates
for
the
agents
to
have
TLS
connectivity
with
each
other,
and
then
they
will
be
meshed
right.
So
the
CLI
is
super
easy
to
use,
but
we
also
have
documentation
how
you
can
use
that
with
helm
and
I'm.
Basically,
there's
one
data
plane.
B
One
cluster
learns
from
the
identities
from
the
other
cluster
and
the
way
around
and
when
you
create
Global
Services
in
each
cluster
in
the
same
namespace
psyllium
on
the
Dude,
using
ebpf
will
populate
low
balancing
ebpf
maps
with
those
IPS
of
endpoints
and
will
low
balance
traffic
across
and
through
the
nodes
to
the
other
nodes
and
then
forward
it
to
the
destination
port,
for
example.
So
super
easy
to
use
in
that
sense,
so
the
documentation
is
out
there
and
should
be
self-explanatory.
As
such.
D
First
of
all,
thank
you
for
a
great
talk
about
the
very
comprehensive
description
of
the
novel
service
mesh
approach.
I
I
have
a
small
question
because
you
mentioned
that
there
are
several
quirks
that
one
has
to
take
into
account
when
deploying
the
service
mesh.
What's
the
environment
that
you
would
recommend
in
order
to
try,
try
out
the
service
mesh,
for
example,
to
compare
with
the
other
Solutions
like
istio
or
Linker
D,
without
having
to
deal
with
quirks
of
the
deployments
of
let's
say:
Cloud
providers.
B
So
you
can
try
it
on
kind
clusters.
If
you
like,
on
your
laptop
or
mini
Cube,
with
with
kinds
you
may
need
some
metal
lb
implementation
or
you
can
use
the
celium
built-in
load
balancing
capability
to
try
it
out
yourself.
You
can
also
perhaps
run
a
little
cluster
on
VMS
install
psyllium.
You
don't
you
just
need
a
specific.
You
don't
just
need
a
4.9
plus
kernel.
B
So
if
it's
dbn
or
Ubuntu
doesn't
matter,
you
create
a
small
cluster,
install
psyllium
with
the
the
helm,
values
I
just
shown,
and
that
would
would
set
you
ready
to
use
service
mesh
as
such
and
again
on-prem.
You
will
need
this
load
balancer,
so
you
should
enable
the
psyllium
low
balancer
to
be
able
to
allocate
eyepiece
to
attract
and
attach
the
load
balancer
to
your
Ingress
resources.
As
such,
so
I
recommend
to
check
out
this
lab.
We
have
a
Gateway,
API
lab
and
a
service
mesh
lab.
B
E
Okay,
do
you
hear
me
correctly,
I
do
okay,
hey!
Thank
you
by
the
way
it's
a
nice
product,
but
now
I've
got
some
questions
and
I'm
not
trying
to
critique
you
here,
but
you
talked
about
the
sidecar
solution
and
that
this
is
an
alternative
yeah
and
then
you
showed
the
performance
Improvement
in
latency,
but
then
the
showing
us,
the
throughputs
it
didn't
relate
very
much.
Can
you
explain
that.
B
Well,
can
I
explain
it.
I
mean
you
get
a
little
bit
of
benefit
of
the
TCP
socket
layer
connectivity
through
the
sidecar,
but
you
still
have
to
get
the
throughput
on
the
the
same.
Tcp
stack
leaving
the
house.
So
that's
limited,
perhaps
by
speeds
or
as
such,
so
you
don't
gain
a
lot
there,
but
it
has
small
gains
for
sure.
E
E
B
Okay,
yeah
yeah,
so
so
one
best
practice
is
not
to
constrain
psyllium
and
psyllium
agents.
As
such
running
on
a
note
using
resource
pools.
Etc
don't
do
that
because
that
will
at
some
point
break
it.
But
obviously,
if
you
push
more
traffic,
if
you
do
more
load
balancing,
yes,
of
course,
you
will
see
an
increase
of
CPU
being
used.
B
You
can
track
it,
however,
using
our
already
available
psyllium
dashboards
using
grafana,
for
example,
and
you
can
see
how
an
agent
is
performing,
you
can
see
if
there's
BPF
map
pressure
or,
if
you're
running
out
of
memory
or
CPU
on
the
Node
level.
So
you
can
already
use
those
metrics
and
those
capabilities
built
in
to
monitor
the
performance
of
your
notes
of
psyllium
and
your
cluster.
B
Be
you're
breaking
yeah
yeah.
That's
that's.
One
reason
why
we
also
launched
tetragon,
because
we
can
also
use
tetragon
in
combination
with
psyllium
to
do
the
kernel
visibility.
So
that's
next
level
I
would
say
it's
not
something
I'll
yet
see
implemented
all
as
such,
but
we
can
actually
monitor
BPF
calls
and
BPF
and
kernel
as
events.
It
starts
and
is
to
Performance
using
tetracon.
F
B
You
you
have
a
choice
here
and
first
of
all,
you
can
use
psyllium
as
CPC
cni
chaining
molds.
That
means
that
you
leave
AWS
for
IPL
location,
the
eni
allocation
to
the
to
the
ETS
nodes
and
some
other
IP
related
things,
and
on
top
of
that,
cilium
can
do
the
enforcement
using
network
policies,
etc,
etc.
G
Question
is
so
in
the
proxy
based
service
mesh.
We,
as
a
user,
get
to
see
which
applications
use
service
mesh
yeah.
So
only
those
part
parts
of
only
those
applications
will
get
the
sidecar
inside
yeah.
In
this
case,
it
is
at
kernel
level
yeah
does
it
mean
that
we
lose
that
flexibility
and
whether
or
not
we
want
it
for
one
or
the
other
applications
sitting
on
the
Node
it
gets
applied
to
all
of
them.
No,
and
if
that
is
the
case,
then
what
about
the
performance?
No.
B
That
will
mean
that
only
for
this
given
namespace,
we
create
this
Envoy
listener
on
the
Node
on
the
agent,
but
not
for
everyone
else
right,
so
it's
dependent
on
where
you
enable
it.
So
it
just
unlocks
this
Ingress
controller
and
gated
API
controller.
You
can
use
it,
but
only
if
you
specify
it
either
in
as
an
Ingress
resource,
okay
to
the
API
resource
or
use
the
load
balancing
surface
cluster
IP
annotation,
then
it
will
use
it
not
not
before
that.
A
A
Is
packed
with
goodies,
I
have
played
with
psyllium
and
I
still
implemented
the
customers
a
lot
and
the
I
have
tried
out.
I've
tried
out
the
cluster
mesh
with
two
clusters.
Only
what
I
miss
is
a
a
single
pane
of
Glass
on
Hubble
and
a
third.
Something
that
allows
me
to
observe
the
service
mesh
into
one
place
is
in
the
same
look
and
feel
that
psyllium
always
gives
like
everything
mother
is
included.
Is
that
in
the
planning?
Well,.
B
That's
a
tricky
question
in
the
sense
that
we
do
have
it
available
as
an
Enterprise,
Hubble
UI.
So
yes,
Hubble
UI
is
limited
to
what
you
see.
You
can
see
the
identities
from
different
clusters.
You
can
see
you
can
secure
it
as
such
right.
You
can
use
the
cluster
ID
as
a
source
or
destination
in
studio,
Network
policies,
but
the
visibility
layer
is
in
the
Hubble
UI
Enterprise
solution.
Well,.