►
From YouTube: Service Mesh War Stories - Lessons Learned the Hard Way - Moderated by Mitch Connors, Google
Description
Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Service Mesh War Stories - Lessons Learned the Hard Way - Moderated by Mitch Connors, Google; Iryna Shustava, HashiCorp; Phillip Gibson, Microsoft & William Morgan, Buoyant
Failure is an excellent teacher, and the best failure to learn from is someone else's. This panel will take you behind the scenes in some of the biggest service mesh products around through the eyes of the engineers that build them. Hear from service mesh insiders about the greatest ideas that didn't work. Learn how to avoid adoption pitfalls that often end in disappointment. Get an inside look at how a service mesh product is made, and how service mesh maintainers run their own service mesh. Take away patterns and practices for successfully navigation your service mesh journey, regardless of the implementation you choose.
A
B
I'll
go
ahead.
First,
my
name
is
irina
schustava,
I'm
an
engineer
on
the
console
service
mesh
and
something
that's
unique
about
the
console
service
mesh.
Is
that
it's
kind
of
like
a
multi-platform
service
mesh,
so
you
can
maybe
call
it
platform
agnostic
service
manager.
You
can
pretty
much
run
anywhere
hand
it
over
to
you,
hello,.
D
A
Nice
and
well,
I've
already
introduced
myself.
What
I
think
is
my
favorite
thing
about
istio.
I
don't
know
how
unique
it
is,
but
I
just
love
our
community.
We've
had
315
companies
contribute
just
in
the
last
year,
and
11
of
them
are
involved
in
leading
the
project
as
a
whole.
So
at
conferences
like
this
or
any
part
of
the
world,
I
go
to
there's
almost
certainly
going
to
be
a
co-worker
nearby
that
I
can
go
and
grab
lunch
with,
which
is
just
a
really
cool
feeling
to
have
all
right.
A
So
first
question
for
the
panelists
and
by
the
way
I'm
I'm
a
panelist,
and
I'm
going
to
be
moderating
because
I
couldn't
find
anybody
else
too.
So
our
first
question
for
the
panelists
is:
why
should
you
use
a
service
mesh?
We've
heard
a
lot
about
that
today,
but
also
why
shouldn't
you
use
a
service
mesh.
C
Oh
yeah,
okay,
yeah.
I
think
what
we're
seeing
with
the
service
mesh
is
it's
becoming
more
like
a
new
platform
right,
so
when
we
say
utilizing
it
being
able
to
not
have
to
kind
of
keep
forth
legacy
apis
when
you
talk
about
circuit,
breaking
rate,
limiting,
etc,
it's
becoming
its
own
platform,
and
I
know
with
osm
I
mean
we're
seeing
we're
branching
out
in
this
ecosystem,
doing
a
lot
of
integrations
into
opa
and
other
products.
So
I
think
that's
a
good
reason
to
use
it
the
flip
side
of
it.
C
Yeah,
I
think,
there's
there's
some
misunderstandings
about
what
the
service
mess
actually
resolves
for
you
or
does
for
you.
I
think
most
of
my
communication
with
customers
is
about
like
network
policy
stuff.
You
know
that
like
they're
kind
of
thinking,
old
school
vlanning
and-
and
I
say
no-
that's
we're
a
little
above
the
stack
where
you're
at
so
just
just
make
sure
like
you're
doing
your
homework
and
you're
understanding
kind
of
what
what
the
service
message
is
going
to
provide
for
you.
D
Yeah,
so
for
me
you
know,
I
think
there
is,
I
think,
there's
exactly
one
reason
when
you
should
use
a
service
mesh
and
every
other
reason
there's
a
reason
not
to
use
it,
and
that
one
reason
is,
if
you
have
a
very
concrete
and
specific
problem
that
you
are
solving.
If
you
are
adopting
a
service
mesh,
because
you
feel
like
you
need
to
adopt
a
service
mesh
or
because
you
see
other
people
doing
it,
and
you
want
to
do
it
too,
then
you're
doing
it
for
the
wrong
reasons,
and
this
is
true
of
any
technology.
D
Almost
all
the
service
mesh
projects
will
provide
something
along
those
lines
with
a
lot
of
differences.
You
know
in
some
of
the
implementation
details,
but
the
value
prop
is
largely
the
same,
but
unless
you're
adopting
that
technology
for
a
specific
problem
that
you
understand,
then
you're
going
to
end
up
with
a
boondoggle
and
probably
every
answer
I
give
for
the
rest
of
this
panel
is
going
to
be
some
variant
of
that.
So
I
apologize
in
advance.
B
That
is
kind
of
automated
for
you
by
the
service
mesh,
but
it
comes
at
a
cost
of
and
complexity
of,
running
it
and
understanding
it.
And
you
know
essentially
it's
another
layer,
that's
kind
of
hidden
from
you,
so
every
organization
kind
of
has
to
understand.
You
know
this
is
a
relatively
new
technology,
and
so
you
have
to
do
the
cost
benefit
analysis
of
whether
it
makes
sense
to
you.
Do
you
have
applications
for
platforms
that
it
supports,
et
cetera,
et
cetera
but
yeah?
A
Yeah,
I
think
one
of
the
costs
that
often
gets
overlooked
when
people
are
coming
to
adopt
a
service
mesh
is
it's
the
same
for
any
software
component
as
engineers.
We're
all
really
excited
to
get
our
hands
on
new
technology,
but
every
piece
of
technology
that
you
add
has
a
cost.
Even
if
it's
free
and
open
source
it
has
a
cost
in
terms
of
maintenance
and
so
the
reason
to
not
adopt
a
service
mesh.
That,
I
would
say,
is
if
you
don't
have
intention
of
upgrading
and
keeping
up
to
date
with
whatever
service
mesh
implementation.
A
You
choose,
you
wouldn't
run
a
five-year-old
apache
server
exposed
to
the
internet
right,
there's
been
dozens
of
cves
over
the
years
like
there
are
with
any
networking
appliance,
and
so
it
would
be
completely
dangerous.
But
we
do
see
some
users
doing
that
effectively
that
with
their
service
mesh,
it's
managing
their
identity,
it's
managing
their
ingress
and
yet
they're
running
in
my
case,
I'm
looking
at
istio
and
they're
running
versions
of
istio
that
are
in
some
cases
three
years
old,
which
just
I
wouldn't
recommend
doing
that.
D
A
A
E
D
D
You
know:
cell
phones,
you
know
from
their
apps
and
stuff
and-
and
that
was
what
I
was
used
to,
but
you
know
we've
had
linker
d
be
used
to
for
like
train,
switching
and
stuff,
which
you
know,
hopefully
that
works
really
well.
We've
had
it
used
in
medical
devices.
You
know
that
people
rely
on
we've
had
it
used.
Actually,
I
gave
a
keynote
last
time
about
all
the
ways
that
had
been
used
to
help
combat
covet
19
during
the
middle
of
the
pandemic.
E
C
Probably
a
little
boring,
but
I
think
my
touch
is
on
some
of
the
integration,
so
we
were
working
with
a
customer
and
I
guess
what
service
mesh
kind
of
you
get
to
this
crossroad
between
the
ops
and
the
devs
right
and
who
actually
controls
policy.
So
this
one
particular
customer
actually
wanted
to
use
the
opa
integration
because
they
said
hey,
they
don't
trust
the
developers.
C
C
B
B
C
Well,
I
think
I
think
we
can
all
agree
that
a
service
mess
is
kind
of
like
the
canary
of
where
you're
at
on
your
maturity
journey
with
kubernetes.
You
know,
but
I
think
the
biggest
thing
we
see
is
people
always
want
to
scream
at
latency
when
the
service
mess
gets
turned
on,
and
then
you
kind
of
ask
them
well
what
was
it
before
and
they
don't
typically
have
those
metrics
prior.
So
what
I
would
say
is
hey
if
you're
on
this
journey,
just
just
do
your
homework
start.
B
I
think
one
of
the
challenges
that
we're
seeing
is
related
to
kind
of
different
deployment
architectures
and
how
different
organizations,
depending
on
their
size,
like
how
many
kubernetes
clusters
they
have
and
some
organizations
end
up.
Having
like
quite
a
quite
a
few
clusters.
And
if
you
are
in
the
situation,
then
running
a
service.
B
Mesh
control
plane
in
each
of
the
clusters
could
be
quite
a
bit
of
operational
overhead
and
what
we're
finding
from
from
those
users
is
that
they
kind
of
want
to
have
some
notion
of
multi-tenancy,
where
they
would
have
kind
of
a
shared
control
plane,
whether
that's
deployed
somewhere
else
or
maybe
you're
running
it
as
a
cloud
service.
Or
what
have
you
and
and
then
like
having
those
multiple
kubernetes
clusters?
D
D
D
Where,
where
are
the
end
user
talks
in
this
conference?
It's
like
all
vendor
talks,
there's
three
end
user
talks.
Every
other
talk
is
from
someone
who
wants
to
sell
you
a
service
mesh,
and
that
is
a
sign
of
a
sick
technology.
That's
an
unhealthy
relationship,
that's
a
junk
food
relationship,
that's
the
that's!
The
candy
bar
that
you
see.
You
know
you
see
the
commercial
and
you're
like.
D
Oh,
that
looks
so
good
and
you
taste
it
and
it
tastes
great
for
about
30
seconds
and
then
you're
left
with
empty
calories
and
a
sugar
hangover
and,
like
two
years
later,
you're
still
trying
to
get
the
service
mesh
to
work
and
it
never
lives
up
to
its
promises.
And
not
only
have
you
not
kept
your
promises
to
your
team
you've,
let
them
down
you've
left
things
in
a
worse
place,
because
you're
boondoggled
you're
saddled
with
this
technology
that
you
can't
adopt.
That's
that's
a
real
tragedy
because
it
doesn't
have
to
be
that
way.
D
It
is
a
difficult
path,
it's
one
that
is
not
for
the
faint
of
heart
or
the
weak
of
spirit,
but
it
is
a
path,
and
that
path
is
understanding
very
concretely.
The
problem
that
you
were
trying
to
solve
and
then
using
that
as
a
site
to
cut
through
the
waves
of
vendor
marketing
until
you
understand
exactly
what
you're
trying
to
accomplish
and
exactly
what
the
options
are
for
accomplishing
it.
A
You
know
I
mentioned
seeing
a
lot
of
users,
kind
of
install
and
forget
in
istio,
and
how
concerning
that
is-
and
we
started
asking
users
about
a
year
ago-
why
that
was,
and
what
we
found
was
that
upgrading
is,
is
a
really
hard
thing
in
a
service
mesh.
You
know,
you
know
the
best
thing
about
a
service
mesh
is
now
you
have
a
proxy
that's
running
everywhere
and
the
worst
thing
about
a
service
mesh
is
now
you
have
a
proxy
that's
running
everywhere
and
it's
you've
got
to
upgrade
it.
A
You've
got
to
patch
it
when
there's
cves,
when
there's
a
new
version,
you've
got
a
control
plane
that
you've
got
to
take
care
of,
and
so
the
whole
last
year
in
our
project,
we've
really
had
a
renewed
focus
on
sort
of
day
two
operations
and
what
it
costs
to
maintain
an
istio
installation,
one
of
the
biggest
things
that
has
changed.
I
mean
we've
worked
on,
making
it
easier,
but
we've
also
worked
on
making.
D
You
know
which
add
a
lot
of
complexity
and
there's
ways
of
introducing
features
which
add
a
small
amount
of
complexity
and
there's
different
types
of
complexity.
You
know
operational
complexity
versus
configuration,
complexity
versus
maintenance,
complexity
and
all
those
things
kind
of
go
into
a
big,
complicated
decision
matrix
so
I'd
say
as
a
whole
that
our
biggest
challenge
you
know
it's
is:
how
do
we
balance
building
the
set
of
features
that
are
going
to
actually
move
the
needle
for
someone,
but
not
saddling
them
with
something?
D
D
You
know
runtime
plug-ins
and
you
know,
had
this
nice
memory
model
and
it
was
clear
how
everything
worked,
and
you
know-
and
what
we
saw
was
a
lot
of
people
shot
themselves
in
the
foot
in
like
really
severe
ways,
because
you'd
introduce
something
into
every
request
path.
That
was
doing
something
you
know
and
most
of
the
time
it
would
work
and
then
every
thousand
requests
it
would
like.
You
know,
delay
for
200,
milliseconds
or
whatever,
and
so
that
experience
has
made
us
a
little.
Gun-Shy
of
you
know:
complete
data
plane
plugability.
D
C
C
This
stuff
yeah
so
yeah,
so
I
think
that
you
know
from
a
development
perspective
yeah
it's
really
kind
of
weighing
all
the
the
technologies
and
then
trying
to
figure
out
hey
like
we're
going
to
say
is
this
a
technology
that
we
can
attach
ourselves
to
and
that's
going
to
be
durable
and
really
be.
You
know
something
valuable
for
the
end
user
to
use.
D
Not
us,
we
ship,
cds,
like
we
cut
a
new
release
of
lingerie
on
friday,
and
then
we
go
home
like
go
ahead,
run
it.
Let
us
know
how
it
is.
You
know
that's
a
very,
very
different
relationship
that
we
have
to
our
software
from
what
the
user
has
to
our
software,
because
the
user
actually
has
to
operate
it.
Well,
it's
different
now
because
we
actually
run
linker
d
these
days,
but
for
a
long
time
we
didn't
run
it
at
all.
So
you
know
having
that
sense
of
empathy
for
the
end
user.
C
D
B
A
F
A
Storage,
which
has
an
amazing
system
for
version
management
and
maturity
of
your
apis,
but
what
we
learned
is,
once
enough
of
your
users,
have
adopted
an
api.
It
does
not
matter
what
you
called
it.
We
could
call
it
alpha,
don't
use
not
ready
for
production
virtual
service,
but
if
10
000
users
adopt
it,
it's
not
changing
we're
not
going
to
change
it.
A
A
D
C
Oh,
the
future,
I
don't
know
I
might
get
a
little
controversial
here.
You
know,
I
think
you
know
again
with
end
user
empathy
right.
You
know,
I,
I
see
a
growing
concern
from
the
community
about
some
of
this
functionality,
just
being
normal
primitives
inside
kubernetes
right,
so
that
that's
probably
a
really
really
long
debate
and
we
would
say
groups
and
etc.
But
you
know
I
could
see
a
you
know,
cross
road
where
those
conversations
start
to
get
very,
very
serious.
You
know.
A
D
And
we
had
in
one
of
our
kind
of
pre-panel
discussions,
we
had
a
really.
We
had
a
really
good
conversation
about
plumbing
and
how
you
know,
if
you
think
about
your
house
or
your
apartment,
you
know
basically
the
plumbing
just
works,
and
you
know
if
it
doesn't
then
like
it's
terrible
and
you
call
a
plumber
and
the
plumber
comes
in
and
does
something
and
for
the
most
part,
you're,
not
really
an
expert
in
plumbing.
You
know
that
it's
there,
you
get
the
benefits
of
it.
D
You
know,
like
you,
imagine,
there's
pipes
and
stuff
somewhere
in
there,
but
you're
not
developing
any
expertise
in
in
plumbing,
and
so
I
think
the
way
what
I
liked
about
that
is
kind
of
I
feel
like
whatever
the
war
stories
are
about
the
service
mesh
in
the
future.
Hopefully
they're
not
the
same
stories
that
we
have
now
and
hopefully
you
know
the
service
mesh
has
moved
to
the
level
of
plumbing.
D
So
it's
you
know
there
are
service
mesh
experts
out
there,
but
there
are
many,
many
more
service
mesh
users
out
there
who
are
not
experts,
and
today
you
kind
of
have
to
be
an
expert.
You
know
for
to
operate
it,
but
that
really
that
shouldn't
be
the
case
right.
You
should
it's
not
really,
unless
you're
working
at
a
company
like
buoyant
being
an
expert
in
the
service,
it's
not
really
your
job
description,
that's
just
like
an
unnecessary.
It's
an
unfortunate
side
effect
of
the
fact
that,
like
surface
mesh,
is
kind
of
complicated
right
now.
D
So
a
lot
of
what
we
have
been
trying
to
do
with
linker
d
and
also
on
the
buoyant
side
is
to
can
we
can
we
make
this?
So
it
really
is
like
plumbing,
and
so
you
get
the
benefits
but
you're,
not
paying.
You
know,
you're,
not
paying
the
price
and
there's
some
plumbers
in
the
world
and,
like
there's
still
poop.
D
I
guess
that
like
goes
through
the
tubes,
you
know
that's
the
traffic
and
gets
encrypted
and
whatever
else,
but
you're
not
really
thinking
about
that
for
the
most
part,
and
so
you
know
I
don't-
I
don't
really
know
what
those
war
stories
will
look
like.
I
hope
they'll
be
talking
about
higher
level
abstractions.
I
hope
we'll
be
talking
about.
D
Like
you
know,
oh
you
know
policies
and
how
we
got
these,
like
you
know,
server
policies
that
conflicted
with
each
other,
and
you
know,
therefore,
the
two
teams
got
really
angry
at
each
other,
because
their
name
spaces
couldn't
talk
or
something
but
like.
Hopefully
it's
not
talking
about.
Like
the
specifics
of
you
know,
our
proxy
implementation,
or
whatever
like
that
stuff,
should
all
fade.
If
we've
done
our
jobs
right
as
service
mesh
creators
that
should
all
fade
down
into
the
the
infrastructure.
A
A
D
You
know,
I
think
we're
getting
there,
but
I
I
still
feel,
like
you
know,
even
for
my
own
beautiful,
perfect
service
mesh
linker
d,
the
best
one
like
it's
still
really
hard
to
operate.
It's
unnecessarily
hard.
You
know
and
it'll
work
fine
for
long
periods
of
time,
and
then
you
have
to
do
an
upgrade.
Like
you
know,
then
you
got
to
load
a
lot
of
stuff
into
your
brain.
To
do
that
upgrade
so,
okay!
Well,
can
we
make
upgrades
like
totally
seamless
and
transparent?
Can
we
write
an
operator
that
does
it
for
you?
D
That's
all
stuff
that
we're.
You
know
that
that
we're
starting
to
build
out.
I
would
like
to
get
to
the
point,
and
I
think
we
can
get
to
that
point,
but
we're
still
ways
away
from
that,
and
so,
at
least
in
my
little
survey
you
know
I'm
not
happy
with
the
state
of
of
linker
d
today,
because
I
think
it
is
still
too
hard
to
run
and
the
people
who
are
really
successful,
who
have
like
these
amazing
talks.
We've
got
a
great
talk
tomorrow
from
in
tane
australia
about
how
they
10x
their
throughput.
D
Using
linker
d
is
awesome,
but
they
are
service,
mesh
experts
and
they
had
to
learn
a
lot
about
linkery.
To
do
that
and
I'd
like
to
have
those
same
stories
happen
without
anyone
really
understanding
the
internals
of
linkerd,
you
know
other
than
like.
I
guess
the
lingerie
experts
who
I
don't
know
who
are
the
plumbers
in
this
analogy.
A
E
B
But
I
I
do
like
really
like
this
analogy,
because
it
really
makes
me
think
of
like
what
would
it
be?
What
would
it
look
like
for
me
to
just
call
a
plumber
when
it
comes
to
like
a
service
mesh,
because,
right
now
it
feels
like
you
do
need
to
build
that
expertise
in
whether
you
know
this
is
something
that
you
do
internally
within
your
organization.
B
You
know
producers
do
to
make
it
easier,
and
is
it
really
automatable
or
is
this
a
problem
that
can
never
be
truly
automated?
One
thing
that
kind
of
going
back
to
your
question
mitch
that
we're
doing
on
the
console
side
is
trying
to
make
it
easier
for
people
to
like
diagnose
problems.
You
know
because,
right
now,
it's
sort
of
like
imagine
if
you
had
a
plumbing
emergency,
but
there
were
no
plumbers
to
call
and
you
would
have
to
figure
figure
it
all
out
on
on
your
own.
D
Yeah,
I
think
that's
a
really
good
point,
because
actually
plumbing
kind
of
sucks
right,
like
you,
have
no
idea
what's
happening
in
your
house,
water
is
flowing
around.
You
have
no
visibility
into
it.
There's
all
sorts
of
things
that
you
know
we
can
improve
plumbing
too.
So
maybe
that's
the
next
next
startup.
A
A
A
You
know
when
you
move
into
an
apartment,
you
expect
that
it
has
plumbing,
but
you
don't
really
ask
who
the
manufacturer
was
like
whether
they
used
plex,
connectors
or
copper
pipes,
or
I
mean
maybe
you
ask
about
lead
if
it's
really
old,
but
for
the
most
part
you
don't
care,
it
has
the
same
interface.
You
use
it
the
same
way.
I
think
the
way
that
the
industry
is
going
service
meshes
can
be
the
same
way.
A
You
don't
necessarily
need
to
think
too
carefully
about
whether
it's
istio
or
linker
d
or
console
or
osm
or
kong,
or
I'm
going
to
miss
some.
I
can't
name
all
of
them
under
the
hood
you're,
simply
able
to
use
like
the
kubernetes
gateway
apis
and
whatever
the
folks
at
kubernetes.
Networking
come
up
with
yet
next
to
use
the
service
mesh,
you
use
it
and
you
move
on
and
stop
thinking
about.
The
underlying
implementation.
A
A
I
think
the
shape
they're
in
now
is
probably
mostly
what
they're
going
to
look
like
as
they
move
forward,
but
there's
a
lot
more
to
be
done
right,
there's
a
lot
more
to
a
service
mesh
than
just
gateways
and
ingress,
and
so
I
I
haven't
talked
to
the
folks
in
cube
networking
recently
to
know
what
what
is
this,
the
next
api
they're
going
to
tackle
and
bring
new
ones.
But
I
I
like
what
they've
done
so
far
and
I'm
excited
to
see
what
happens
next.
G
A
A
B
A
I
think
I
think
you're
absolutely
right.
It's
it's
really
easy
to
do
that
at
runtime,
in
a
stable
state
within
your
service
mesh,
because
you're,
hopefully
running
your
application
already
in
a
highly
available
configuration
you've,
hopefully
got
horizontal
pod,
auto
scalers
all
configured
for
it
to
respond
to
demand,
and
so
your
service
mesh
data
plane
will
respond
to
demand
as
well,
where
it
gets
a
little
bit
more
complicated,
in
my
opinion,
is
during
an
upgrade
because
all
of
a
sudden
you
have
something
that's
infrastructure
and
plumbing
for
your
entire
thing
and
you're
gonna.
C
D
Different
here
than
than
any
of
the
other
service
meshes
it's.
You
know.
We
rely
on
the
same
kubernetes,
primitives
and
like
it's,
you
know,
replica
sets
of
immutable
pods
and
so
that
kind
of
informs
the
the
the
solution
space
there.
You
know
if
we,
if
kubernetes
changes
or
if
we
move
off
of
kubernetes,
then
like
you've
got
different.
You
know
you've
got
different
options.
You
know,
I
guess
you
could
write
your
whole
data
plane
as
one
big
wasm
plug-in
and
then
just
dynamically
load.
It
there.
C
A
C
C
Sure
sure
yeah
yeah,
I
would
probably
say
everyone's
in
in
line
here
I
mean
the
development
process.
I
mean,
there's
various.
You
know
soaked
tests
that
are
doing.
You
know,
corner
and
edge
cases.
You
know
we're
probably
piling
way
more
workload
on
than
what
the
usual
customer
would
do,
and
and
again,
if
you're
soaked,
testing
you're
running
it
for
a
long
time
right
so
yeah,
we
we
got
a
pretty
stringent
process.
As
far
as
you
know,
we
you
know,
enable
new
features
and
and
take
them
through
the
progression
testing.
D
Yeah
I
like
this
question
a
lot
because
open
source
is
horrible.
That's
you
know
we.
We
do
so
much
testing
on
linker
d
in
like
as
many
different
iterations
as
we
can
and
like,
and
we
ship
out.
You
know,
release
candidates
or
like
hey
everyone,
please
try
to
release
candidates
and
like
people
do
and
they
find
issues
and
we're
feeling
pretty
confident.
Then
we'll
you
know
then
we'll
ship
a
release
and
then,
like
someone,
will
upgrade
three
weeks
later
and
they're
like
well.
D
It
doesn't
actually
work
if
you're
on
this
special
variant
of
this
and
this
and
this
and
you're
like
oh,
if
you
had
just
tested
that,
like
you
know
earlier
or
if
we
had
known
about
that,
like
you
know,
and
now
we
have
to
cut
a
new
release
and
that's
annoying
because
there's
all
this
process,
we
have
to
redo
all
the
testing.
So
it's
just
it's
a
difficult.
D
I
mean
I
don't
think,
there's
an
easy
solution,
but
it's
but
it's
difficult
and
I
think
it's
particularly
difficult
for
the
service
mesh
because
it
sits
at
the
intersection
of
all
these
complicated
things
right.
The
network
is
going
through
it
and
all
of
the
layer,
three
layer,
four
like
underlying
substrate
there,
like
if
something's
weird
there
then
like
that's
gonna,
affect
your
surface
mesh.
It
relies
on
all
this
kubernetes
stuff
and
like
if
something's
weird
there,
whether
that's
going
to
affect
your
service
mesh,
relies
on.
You
know
it's
just
this
weird
super
complicated
intersection.
D
So
I
don't
know
that
we,
you
know,
at
least
from
my
perspective.
I
don't
know
we
really
have
a
great
answer.
I
did
mention
earlier,
and
this
has
made
actually
a
nice
difference
for
us.
We
started
running
linker
d
ourselves,
so
we
have
a
sas
product
called
buoyant
cloud,
that's
powered
by
linker
d,
and
that
has
really
changed
our
relationship
with
this
product,
because
now
you
know
we
have
a
team
that
is
like
you
know,
has
to
wake
up
at
three.
D
Just
shipping
cds
anymore,
we
actually
are
you
know,
living
the
whole
service
owner
lifestyle,
that's
helped,
but
even
with
that,
you
know,
even
with
that,
we
run
this
in
one
particular
environment.
You
know
and
there's
a
hundred
different
edge
cases
that
we
don't
get,
and
so
it's
just
a
it's
a
difficult.
It's
a
difficult
process.
Every
release
accumulates
tests,
but.
C
Mean
I
think
the
unfortunate
truth
is
like
we
can't
accommodate
for
every
permutation,
that's
out
there
right
and
then
I
mean.
I
think
what
we're
even
saying
here
is:
that's
why
we
like
to
push
rc's
out.
I
mean
we
tell
you
hey.
This
is
kind
of
experimental,
etcetera
test
it
out
and
we
actually
do
get
a
lot
of
feedback
from
the
community.
You
know
because
again
it's
it's
hard
to
know
every
type
of
configuration
that
that's
out
there.
C
A
I
think,
at
the
same
time,
though,
like
we
can't
proactively
test
every
conceivable
thing
that
you
can
do
with
a
network
right
networks
have
been
around
for
a
little
while
there's
a
little
bit
of
surface
involved
there
and
so
testing.
Every
combination
is
just
not
realistic.
What
we
can
commit
to,
though,
is
saying
we
will
never
break
you
the
same
way
twice
if
you've
experienced
an
outage
on
an
upgrade
in
any
of
these
projects.
Please
talk
to
us
open
up
a
github
issue.
A
B
D
B
C
Yeah,
I
think
the
too
many
patterns
that
I've
seen
is
kind
of
a
localized
spillover.
You
know
affinity,
spillover
and
then
the
other
is
kind
of
similar.
What
we've
said
here
is,
you
know,
I
think
most
of
the
enterprise
customers
are
trying
to
follow
the
kind
of
aha
patterns
of
the
clouds
that
that
they're
on
and
and
so
they
want
to
be
able
to
say,
hey.
C
If
this
this
region
goes
down,
can
we
still
operate,
and
so
there
is
kind
of
this
kind
of
synchronization
of
data
and
config
to
allow
you
know
someone
on
the
west
coast
to
still
operate,
something
if
you
know
the
main
cluster
was
on
the
east
coast,
so
yeah.
I
I
think
I
mean
it's
been
my
experience
that
a
lot
of
the
multi-cluster
patterns
are
following
kind
of
cloud
providers,
kind
of
high
availability.
You
know
processes.