►
From YouTube: Service Mesh - The New Single Point of Failure - Mitch Connors, Sabeen Syed & Thomas Rampelberg
Description
Service Mesh - The New Single Point of Failure - Mitch Connors, Google, Sabeen Syed, HashiCorp & Thomas Rampelberg, Buoyant
Interested in knowing why your favorite service mesh was implemented that way? Architecture decisions have real user impact. When building a service mesh, it is possible to fall into a trap of choosing implementation that is easier to build but makes it difficult to operate in the real world. While service meshes enable new levels of resiliency for users’ applications, they suffer from a chicken and egg problem: How do you build a resilient and scalable service mesh without having a service mesh to rely on?
Maintainers of Istio, Linkerd2 and Consul will walk through tradeoffs the projects have made during implementation and the impact on users. Topics will include:
- Why it is important to verify environments before installation.
- How to build a service mesh which can be safely upgraded.
- What regular security updates mean for upgrades.
- How to give users the same stability for config changes as they require for code changes
- What to do when the mesh breaks.
- Why the division of responsibility is important.
A
Good
morning,
servicemeshcon
welcome
to
servicemesh
the
new
single
point
of
failure.
In
this
talk,
we're
going
to
be
discussing
some
of
the
trade-offs
that
service
meshes
have
made
on
implementation
and
how
that
impacts
you
as
users,
I'm
here
representing
linker
d
mitch,
will
be
representing
istio
and
sabine
will
be
chatting
with
us
about
consul
sabine.
Why
don't
you
tell
us
a
little
bit
about
yourself.
B
B
C
Hi,
my
name
is
mitch
connors
and
I'm
a
software
engineer
with
google.
I
work
in
seattle,
washington
and
in
the
last
two
years
I've
been
working
on
the
istio
project.
In
particular,
I
work
in
user
experience
where
I
make
it
my
goal
to
really
deeply
understand
the
way
that
our
users
interact
with
istio
to
understand
what
its
strengths
are,
as
well
as
its
weaknesses
and
then
to
take
those
learnings
and
turn
them
into
meaningful
features.
That
users,
like
you,
can
take
advantage
of
in
istio.
C
Cubecon
is
one
of
my
favorite
ways
to
meet
and
interact
with
users
to
hear
about
the
creative
ways
that
you're
using
our
software
ways
that
maybe
we
would
have
never
anticipated
as
well
as
to
understand
how
we
can
better
serve
you
in
the
future.
So
I
look
forward
to
chatting
with
all
of
you
to
hearing
your
questions
at
the
end
of
the
session
and
interacting
with
you
in
the
lobbies.
Thanks.
A
And
hello,
everyone,
my
name
is
thomas
rampelberg.
I'm
a
software
engineer
here
at
buoyant.
The
creators
of
linker
d.
Linker
d
is
a
super
fast
super
lightweight
service
mesh.
That's
really
focused!
On
user
experience,
here's
a
little
bit
of
a
agenda
for
what
we're
going
to
be
covering
today,
I'd
like
to
call
it
turtles
all
the
way
down.
A
I've
never
heard
of
you
complexity
is
a
big
part
of
service
meshes.
Obviously
it's
something
we
want
to
fight
against.
Since
we're
talking
about
a
new
single
point
of
failure,
a
complex
single
point
of
failure
is
10
times
worse
than
a
normal
single
point
of
failure,
though,
obviously
we'd
prefer
not
to
have
a
single
point
of
failure.
To
begin
with,
to
get
me
to
get
us
started
on
this
one
mitch.
I
know
this
one
is
a
subject.
You're
particularly
passionate
about
tell
us
a
little
bit
about
how
istio's
been
managing
the
complexity.
C
C
These
things
we're
not
even
necessarily
talking
to
the
people
who
are
operating.
These
things
was
just
too
much
complexity,
and
so,
a
year
ago
we
moved
to
a
monolithic
model
for
our
service
mesh
and
we've
seen
substantially
reduced,
friction
in
terms
of
upgrade
and
maintenance
for
our
users.
As
a
result
of
that,
it's
been
a
sort
of
counter-intuitive,
maybe
eyebrow-raising
move
for
some,
but
I
think
it's
been
great
for
our
users.
C
A
C
Like
a
full
platform
for
running
software
and
services,
where
we
can
see
all
of
the
knobs
in
one
place
and
a
user
interface,
we
made
the
very
intentional
decision
not
to
be
a
platform
and
it's
an
interesting
decision,
because
it
means
that
istio
is
not
all.
You
need
to
run
your
services.
We
are
not
a
one-stop
shop.
The
intentional
decision
there
is
that
we
want
to
be
part
of
an
ecosystem.
C
We
know
that
we
are
not
going
to
be
the
best
at
everything,
so
we're
going
to
strive
to
be
the
best
at
a
very
core
set
of
service
mesh
features
and
then
allow
various
other
technologies
to
come
in
and
fill
the
gaps
such
as
canarying.
There's
a
number
of
technologies
out
there
today
that
are
doing
a
great
job
with
it
built
on
top
of
service
mesh.
So
that's
kind
of
how
I
think
about
complexity
and
how
we
manage
it
in
istio.
A
A
A
B
B
So
we
did
not
want
to
store
metrics
in
console,
but
we
also,
we
still
wanted
our
users
to
be
able
to
visually,
see
their
user
workflows
and
metrics
in
the
console
ui.
So
what
we
have
built
is
a
scalable
javascript
plugin,
and
this
javascript
plugin
allows
users
to
query
data
in
the
apms
that
they
use
and
apms
just
fyi.
For
those
folks
who
may
not
know
is
an
application
performance
management
tool,
and
these
are
as
an
example,
there's
prometheus
and
datadog.
B
So
right
now
we
have
a
plug-in
for
prometheus,
so
our
users
can
use
that
query
that
get
data
from
there
and
then
we
have
in
our
console
ui.
We
have
a
topology
view
that
basically
shows
the
data
model
and
then
it
supplements
it
with
metrics
data
coming
from
that
apn.
So
things
like
your
requests
per
second
latency
error
rates,
so.
B
A
Yeah
we
in
linker
d
we
looked
at
prometheus
and
said
everyone
in
the
world
is
integrating
with
prometheus,
so
we're
just
gonna
stick
stuff
in
prometheus
and
hope
for
the
best
mitch.
This
is
actually
an
interesting
conversation.
Given
istio's
history
tell
us
a
little
bit
about
where
istio's
relationship
with
the
prometheus
and
where
that's
gone,.
C
C
We
found
that
users
already
have
prometheus
installs
and
they
don't
want
to
fiddle
with
federation
levers
in
trying
to
get
our
prometheus
installed
to
work
well
with
theirs,
and
they
really
don't
want
our
prometheus
install
to
overwrite
theirs.
So
we
found
that
it
was
best
just
not
to
install
it
and
give
them
clean
instructions
on
how
to
plug
their
service
mesh
into
their
pre-existing
prometheus
install.
C
A
We
are
about
to
release
a
stable
version
of
linker
d
that
comes
with
our
bring
your
own
prometheus
solution
for
a
lot
of
those
same
reasons.
No,
I
was
actually
thinking
more
about
the
mixer
or
the
demixerification
that
has
happened
recently,
especially
since
we
were
talking
about
complexity.
I
thought
that
that
was
an
interesting
piece
there,
but
generally
speaking,
especially
as
cloud
native
projects,
I
think
we've
all
realized
that
metrics
are
better
served
by
the
community
to
go
back
to
the
theme
we're
kind
of
going
along
there.
Yeah.
C
Mixer
was
an
interesting
architectural
change
because
originally
mixer
one
was
one
of
the
key
motivations
for
running
istio's.
Microservices
mixer
was
the
only
layer
that
logically
could
block
the
data
plane,
so
it
had
to
run
extremely
light
extremely
fast,
and
it
was
because
of
policy
decisions.
We
also
put
telemetry
there
because
of
the
close
coupling
with
the
data
plane,
but
that
was
more
incidental
policy.
Decisions
are
really
the
hard
part
that
mixer
was
handling.
C
Fortunately,
over
the
last
year,
the
developments
in
webassembly
have
made
it
so
that
those
decisions
can
actually
be
written
in
arbitrary
code
that
executes
inside
of
the
data
plane
so
with
zero
latency
in
terms
of
network
cost,
and
that's
what
enabled
us
to
get
rid
of
mixer
and
ultimately
move
towards
monolithic
micro,
not
monolithic,
microservices,
monolithic
service
mesh.
There
we
go.
A
C
Yeah,
I
don't
think
anyone
has
fully
realized
the
capabilities,
including
the
istio
project,
the
capabilities
of
having
arbitrary
javascript
script
code
that
just
runs
ubiquitously
in
the
in
your
proxy.
I
am
really
excited
to
see
the
developments
that
come
out
of
that
over
the
next
two
years,
especially
now
that
webassembly
has
been
merged
back
into
the
envoy
upstream.
A
B
Yeah
so
there's
one
other
thing
that
that
we
decided
not
to
go
forward
with,
and
this
was
around
supporting
multiple
console
installations
on
one
kubernetes
cluster.
Yes,
so
some
users,
they
you
know
they
might
want
to
have
two
environments
on
their
k-8s
cluster.
So
you
know
a
dev
environment
and
a
production
environment,
possibly
on
the
same
kh
cluster,
and
so
that
can
result
in
a
number
of
issues
like
security
issues,
performance
issues.
So
just
like
around
the
security
issues.
B
If
you
have
two
environments
on
your
k-8
cluster
and
you
have
two
console
installations
as
well.
If
there's
a
console
client
that
you
want
to
register
with,
for
example
your
dev
environment,
but
you
do
it
by
accident
to
production
that
can
cause
huge.
You
know
very
obvious
security
issue
there
and
then
along
the
same
thing
that
can
also
cause
performance
issues
where,
if
your
dev
environment
or
your
staging
environment,
there's
a
ton
of
tests
running
and
all
of
a
sudden,
it's
using
a
whole
bunch
of
resources.
B
A
Yeah
we,
we
started
out
supporting
multiple
lingerie
installs
on
clusters,
with
this
idea
that
less
the
like
dev
prod
instance.
So
that's
a
really
good
use
case,
but
more
of
the
multi-tenancy,
which
I
think
I'm
going
to
ask
mitch
in
a
second
about,
because
I'm
super
excited
to
hear
there.
But
we
ran
in
directly
into
the
buzz
saw
of
crds,
because
you
can't
have
crd's
version
in
any
reasonable
fashion.
Having
two
installs,
especially
of
linker
d,
when
you've
got
a
couple,
crds
starts
to
become
totally
untenable.
A
Because
how
do
you
do
the
upgrades?
How
do
you
keep
from
stepping
on
other
people's
feeds
feet,
and
so,
at
least
on
our
side
of
things?
We've
really
been
pushing
the
multi-cluster
as
a
way
to
do
isolation
and
multi-tenancy
instead,
and
I
will
point
out
that
I
personally
would
never
operate
a
dev
and
prod.
C
Yeah,
so
multiple
installs
on
a
cluster
has
been
something
we've
been
working
towards
for
the
last
like
15
months,
or
so.
It's
been
kind
of
early
access
since
one
five
one
six
time
frame,
but
I
think
it's
safe
to
say
that
in
one
eight
the
standard
way
of
upgrading
istio
actually
involves
running
multiple
installs
concurrently.
C
Because
we've
consolidated
that
single
point
of
failure
into
one
resource,
the
kubernetes
deployment
model
of
updating
a
deployment
and
kind
of
crossing
your
fingers
that
health
checks
fail.
If
something
goes
wrong,
just
wasn't
enough
security
and
safety
for
our
users
in
particular,
when
you
upgrade
a
control
plane
in
a
service
mesh,
often
times
the
health
failure
doesn't
happen
within
the
control
plane
itself,
it
happens
within
the
proxies
and
kubernetes
doesn't
provide
the
kind
of
levers
and
granularity
that
we
needed
to
really
detect
failure
early
in
moving
to
a
control
plane.
C
So
with
revisions,
what
we
recommend
users
do
like
if
you're
moving
from
say
one
seven
to
one
eight.
Is
you
install
a
separate
one,
eight
control
plane
that
initially
does
nothing
but
serve
ingress
traffic.
It's
using
all
the
same
route
of
trust,
all
the
same
mtls
and
everything
else
as
the
other
control
plane.
It's
just
got
almost
no
proxies
connected
to
it
and
then,
as
you're
comfortable,
you
go
through
and
cut
your
proxies
over
one
name
space
at
a
time
until
you're
finally
able
to
shut
down
that
old
revision
of
istio.
C
So
I
don't
know
that
we
envision
a
lot
of
use
cases
where
people
are
permanently
running
multiple
vision,
revisions
in
one
cluster,
although
that
is
it's
possible,
we're
just
not
certain
it's
useful,
but
as
far
as
upgrades
go,
it's
been
a
very
beneficial
tool
and
I
think,
has
increased
the
predictability
of
the
upgrade
process.
A
good
deal.
A
Yeah,
the
it's
super
interesting
to
talk
about
upgrades
in
general
are
like
are
hard
I've
in
the
kubernetes
ecosystem.
In
particular.
I'm
really
excited
about
all
of
the
folks
who
are
working
on
workflow
related
projects,
because
especially
upgrading
service
mesh.
To
your
point,
you
really
need
gates
right.
It's
not
just
upgrading
the
control
plane.
A
It's
also
making
sure
that
the
proxies
are
at
the
right
version,
and
then
you
know,
okay,
I
can
upgrade
the
control
plane
this
far,
but
now
I
need
to
wait,
make
sure
that
the
data
plane
gets
rolled
before
I
go.
Do
the
next
steps
and
that's
and
that's
a
tough
it's
a
hard
problem
distributed
systems
I
hear
are
challenging
for
some.
C
A
All
the
way
down
all
right,
it's
super
awesome
to
hear
about
how
we've
all
been
managing
complexity
in
our
own
service
meshes.
But
let's
talk
a
little
bit
about
a
subject.
That's
near
and
dear
to
my
heart,
actually
operating
the
mesh.
Did
you
say
someone
needs
to
keep
the
mesh
up?
How
do
you
go
and
tackle
actually
managing
this
thing
in
a
cloud
native
world
where
there's
all
kinds
of
operators
and
users
sabine
tell
us
a
little
bit
about
the
console
side
of
things.
B
Yeah
so
on
the
console
side,
well,
specifically
around
getting
started
or
getting
the
mesh
up
and
running,
there's
a
few
things
that
we
do
to
make
that
easier.
So
the
first
is
around
automating
the
acl
setup,
so
acls
are
basically
access
control
limits
since
console
exists
outside
of
kubernetes.
It
has
its
own
authorization
solution
and
we
basic
so
basically
acls
are
to
console
what
kubernetes
is
to
our
back.
A
So
the
that's
really
cool.
This
is
something
I
had.
I
didn't
know
anything
about,
so
the
you
actually
have
an
rbac
bridge,
so
kubernetes
native
rbac
calls
get
translated
into
console
acls,
correct,
yep,
oh
that's
cool!
I
I
went
deep
on
the
gke
rbac
plug-in
once
and
was
I'm
not
gonna,
say
horrified,
I'm
going
to
say
impressed
by
the
effort
that
the
engineers
went
to.
A
I
can
only
imagine
how
much
that
took,
but
since
we're
talking
a
lot
about
complexity,
it's
so
important
to
have
a
single
store,
especially
for
something
as
important
as
our
back.
That's
really
cool
that
you
went
to
the
effort
for
that
cool
all
right,
I'm
going
to
I'm
going
to
do
a
little
bit
about
linker
d.
A
Here,
probably
one
of
my
favorite
things
about
linker
d
is
our
check
command
and
the
reason
for
that
is
that
kubernetes
clusters,
even
though
we're
talking
cloud
native
and
cattle
instead
of
pets,
I'm
I'm
gonna
say
kubernetes
clusters
are
snowflakes,
especially
once
you
start
talking
across
organizations.
For
example,
we
just
got
done
talking
about
the
console
r
back
to
acl
mapping.
Who
knows
how
that
maps
on
a
cluster
where
somebody's
got
installs
or
not,
and
so
we
spend
a
lot
of
time?
A
If
I
see
two
users
reporting
a
difficulty
setting
up
link
or
d,
we
go
create
a
check
for
it,
and
so
we
will
go
actually
do
validation
before
you
install
your
cluster
to
say
that
you've
got
the
right
rbac
to
be
able
to
go
and
create
it,
and
then
we'll
do
it
after
you
install
the
cluster
and
none
of
the
checks
at
the
moment
are
I'm
going
to
call
them
integration
tests
eg?
They
just
go,
make
sure
that
the
readiness
is
passed.
Liveness
is
passed.
C
You
know
something
that's
interesting
about
install
we're
in
a
kubernetes
world,
and
so
everything
needs
to
be
declarative,
but
what
we're
learning
right
is
that
upgrades
are
not
a
declarative
operation.
No,
I
don't
ever
want
to
declare
the
entire
state
of
my
mesh
in
the
next
minor
version.
Instead,
I
want
to
do
a
mutation
right.
I
want
to
say
I'm
on
istio1.x.
C
One
thing
that
we've
done
over
the
last
year
to
sort
of
assist
with
that
it
doesn't
directly
address
the
problem,
but
we've
introduced
analyzers,
which
it
sounds
like
are
a
little
bit
like
checks.
It's
a
suite
of
things.
It's
not
exclusively
focused
on
upgrade
or
install
time.
It
actually
can
run
during
runtime.
We
bake
them
both
into
the
control
plane
as
well
as
into
the
cli.
C
You
can
run
analyze
and
it'll
list
some
probable
problems
or
maybe
definite
problems
that
you
have
related
to
your
configuration.
For
me,
my
favorite
one.
I
always
fat
finger,
the
gateway
name
when
I'm
mapping
a
virtual
service
to
a
gateway.
I
mean
just
every
time
and
the
analyzer
catches
it
really
well,
and
I'm
excited
to
say
that
now,
there's
even
a
way
that
you
can
have
the
control
plane
begin.
Writing
those
analysis
messages
out
into
the
status
field
of
the
objects.
C
A
A
We've
actually
had
a
bunch
of
users
just
take
our
check
command
and
stick
that
into
their
alerting
workflow,
which
drives
me
crazy,
because
we
built
check
as
a
like
user
interface,
but
man,
it's
cool,
that
folks
are
doing
it.
That
way,
that's
that's
really
cool!
Here
we
go
so
sabine.
We
had
been
chatting
a
little
bit
beforehand,
and
one
of
the
things
you
said
mentioned
was
the
approach
that
console
had
around
federation.
It's
an
interesting
feature
set
that
at
least
we
don't
have
in
the
linker
d
world.
A
C
Fail
we
failed,
I
want
to
say,
like
this-
is
a
pivot
that
we
have
to
make
as
a
project.
We
take
responsibility
for
outcomes.
We
don't
just
deliver
really
cool
technology
that
this
one
time
we
got
running
inside
of
a
cluster
somewhere.
If
our
users
can't
do
it,
it's
not
done
and
that's
something
we've
been
focusing
on
actually
with
in
relation
to
multi-cluster,
in
particular
with
the
one
eight
release
we
are
finally,
and
it's
not
out
the
door
yet
so
I'm
not.
C
C
We
have
a
bit
of
a
feature,
maturity
problem
in
that
we
have
a
bunch
of
developers
who
love
shipping,
new
features
and
they're,
so
cool
and
shiny
and
they're
alpha,
and
we
don't
have
a
lot
of
developers
historically,
who
have
focused
on
doing
the
very
hard
productionization
work
of
promoting
that
feature
through
to
beta
and
then
to
generally
available.
So
I
think
that's
going
to
be
a
new
focus
and
theme
of
the
istio
project
over
the
coming
year.
A
Oh,
that's
like
the
it's
seriously
the
unsung
heroes,
too.
The
person
who
ships
this
feature
first
is
always
the
one
who
gets
the
like
kudos
and
then
that
that
poor
person
who
goes
and
spends
years
polishing
off
and
making
it
more
stable.
Just
we
in
linker
d,
we
actually
have
a
program
that
we
call
linkery
heroes
where
that's
kind
of
the
opportunity.
I
love
it
because
I
get
to
call
out.
A
We've
got
one
developer,
who
is
like
ci
machine
and
he
keeps
ci
running
and
again
like
it's,
not
something
you
see
in
the
change
log,
but
it's
the
only
way
that
we
ship
software.
It's
that's,
really
cool
to
go
back
to
sabine's
point
about
federation
and
the
multi-cluster
stuff.
It's
tough
like
in
linker
d
for
multi-cluster.
A
We
have
you,
pull
a
secret
from
one
cluster
and
push
it
on
to
another
and
that's
cool,
except
that
it
requires
kubectl
context,
games
and
this
that
and
the
other
thing
and
like
I
wrote
the
tutorial
for
a
multi-cluster
and
it's
basically
crazy
bash
for
loop
scripts,
and
there
was
nothing
I
could
do
about
it.
It
was
that
or
20
paragraphs
to
explain
how
it
all
fit
together.
I'm
super
excited
about
all
the
work
that
sigmulti
cluster
has
been
doing
to
kind
of
make
our
lives
a
little
bit
better.
C
I
think
that's
another
example
of
a
case
where
it's
hard
to
optimize
for
both
the
demo
use
case,
as
well
as
the
production
like
enterprise
grade
software
use
case,
because
in
an
enterprise
they
have
secret
management
systems
that
are
already
in
place.
They
already
have
probably
a
managed
certificate
somewhere
in
some
system,
whether
they're
doing
it.
You
know,
there's
a
hundred
different
ways
to
accomplish
that,
but
they,
the
last
thing
they
want,
is
my
bash
script,
generating
a
self-signed
cert
and
then
pushing
it
to
production
for
them.
But.
C
A
The
we've
definitely
run
into
this.
In
linker
d,
we
ship
with
a
jager
install
out
of
the
box
jaeger
installs,
are
insanely
complex
to
productionize,
not
because
jaeger's
a
complex
product
inherently,
but
you've
got
storage
as
part
of
it,
and
so
our
helm
chart
is
forcibly
minimalized
to
the
point
where
it
feels
awkward.
If
you
try
and
put
it
into
production
because
you're,
you
should
go
and
use
the
jaeger
operator.
If
you
want
to
get
that
into
production
for
sure
mitch,.
A
No,
I
think
we've
covered
everything
before
we
get
into
this
next
section.
I
want
to
tell
a
little
bit
of
an
anecdote
which
may
stick
my
age
as
being
an
old
person,
but
I'm
going
to
say
it
anyways
back
in
the
day
whenever
something
goes
wrong,
really
there's
only
one
reason:
it
went
wrong.
It's
dns,
it's
always
dns's
fault,
100
of
the
time.
A
Okay,
it
is
50
of
the
time,
but
the
service
mesh
gets
blamed
100
of
the
time,
and
so,
at
least
for
us,
we've
really
needed
to
design
for
failure
and
think
about
what
that
means,
not
just
for
the
control
plane
and
operating
the
service
mesh,
but
what
it
means
for
the
developers,
the
user
of
the
service
mesh
as
well
mitch.
I'm
sure
that
you've
got
some
interesting
insight
from
the
istio
side
to
share
along
those
lines.
Tell
us
a
little
bit.
C
Yeah
that
that
makes
me
think
of
kind
of
two
stories
within
the
project.
You
know.
Historically,
I've
only
been
on
the
istio
project
for
two
years
now,
and
before
that
I
owned
the
api
on
a
large
enterprise-grade
network
appliance,
and
we
became
the
catch-all
for
everything
that
went
wrong
with
that
network.
Appli
appliance
it's
like
well,
it
went
wrong
after
I
told
the
api
to
do
it
and
therefore
it's
definitely
the
api's
fault
and
it
took
a
while
to
really
embrace
that
role.
C
But
I
think
what
we
learned
is
that
that
can
be
done
really
really
well
and
what
it
does.
Is
it
forces
you
to
show
to
demonstrate
conclusively
that
your
layer
is
doing
they're
behaving
exactly
as
expected,
and
it
raises
the
bar
of
quality
to
such
a
high
degree
that
eventually,
your
users
stop
making
that
assumption,
and
so
I
think
it's
just
a
matter
of
maturity
in
service
mesh.
It's
something
that
we're
pursuing
in
istio.
I
don't
think
that
we're
there
yet
one
of
the
ways
that
we're
doing
that
is
historically,
you
know.
C
C
Then
the
control
blade
axle
propagated
out
to
every
data
plane
which
might
be
in
this
cluster
might
be
in
another
cluster,
a
different
region,
so
we're
raising
the
visibility
of
propagation
of
changes
throughout
the
service
mesh
so
that
our
users
can
very
easily
and
very
visibly
tell
yes.
This
component
is
behaving
as
expected.
I
can
move
on
and
start
troubleshooting
other
sections
of
the
application.
We
don't
want
them
spending
a
lot
of
brain
cycles
on
that.
A
Yeah
we've
we
have
a
little
bit
of
a
command
that
will
do
that
that'll
actually
reach
out
to
the
linker
d
proxies
and
dump
their
service
discovery
state.
For
exactly
that
reason
and
you're,
giving
me
flashbacks
to
my
istio
multi-cluster
experiment,
where
I
had
to
go
figure
out
the
how
to
dump
service
discovery
information
from
envoy
for
sure.
It's
super
interesting
that
you
went
that
direction,
because
that,
like
defensive,
this
is
healthy
is
a
big
reason
why
we
put
check
in
as
a
command
it's
that
defensive,
like
these
are
all
working
things.
A
B
Yeah,
so
in
that
from
that
aspect,
what
we
try
and
do
is
that
you
know
when
something
is
failing.
We
went
the
route
of
having
it
be
seen
in
our
console,
ui,
so
automatically
added
checks
to
each
of
our
components,
and
so,
when
something
fails,
you'll
see
it
as
red
in
the
ui
and
so
that
I
feel
like
that,
really
helps
our
users
just
automatically
know.
Okay,
this
piece
is
failing
all
right.
I
see
it
it's
spelling,
there's
something
that
I
need
to
do
so.
A
A
Questions
I'm
sure
like
I
said
it's
definitely
something
that
I've
wanted
to
do
for
a
long
time
now.
The
the
debugging
the
mesh
is
is
perhaps
one
of
the
subjects
I'm
most
passionate
about
in
linker
d,
we've
kind
of
had
to
throw
the
kitchen
sink
at
it
for
lack
of
a
better
term
and
kind
of
attack
it
from
a
multi-level,
multi-level
solution
and
that
we've
got
a
one
command
called
tap
that
actually
lets.
You
do
for
lack
of
a
better
term
wireshark
on
your
entire
cluster.
A
So
you
can
go
and
tap
and
see
the
live
requests
and
that's
pretty
helpful.
But
it's
only
really
helpful
to
show
when
someone's
app
is
misbehaving.
If
the
mesh
is
actually
misbehaving,
you
run
into
quite
a
bit
of
issues,
and
so
the
next
level
down
there
is
the
we've
got
a
debug
sidecar
and
the
ephemeral
container
features
in
kubernetes,
but
I
think
they
landed
in
116
and
they
might
be
beta
and
119.
A
A
Yeah
and
it's
like
it's
the
onion
right
like
as
soon
as
you
start
to
get
into
the
debugging
side
of
things.
You
start
out
at
that.
Like
high
level,
we
know
something's
wrong,
there's
an
alert
and
then
you
kind
of
have
to
pull
the
like
layers
of
the
onion
back
until
you
get
down
to
the
core
problem.
Like
I,
I
remember
running
into
an
issue
on
gke,
where
they
had
shipped
with
a
kernel
that
deadlocked
when
you
used
the
esso
original
desk,
socket
option
and
I'm
picking
on
gke.
A
C
C
Was
that
was
a
really
cool
tool
to
see?
I
love
the
way
that
you
guys
have
pulled
that
off.
I
think
one
of
the
things
that
we
like
to
see
in
istio,
we
talked
about
a
little
bit,
what
we
don't
develop
and
what
we
intentionally
don't
we've
seen
a
great
development
in
the
ecosystem
of
a
couple
of
different
tools
for
debugging
your
application,
using
a
service
mesh.
Now
that
the
the
network
is
completely
managed
that
actually
gives
you
some
really
cool
super
powers.
C
If
you
can
assume
the
network
is
working
properly,
if
you
can
go
through
that,
you
know,
distribution,
status,
checks
and
analyzer
checks
and
everything
else
to
say
my
service
mesh
is
working
well
now
you
can
actually
do
get
into
your
application
in
really
interesting
ways.
The
the
one
that
I've
played
with
the
most
has
been
squash
from
the
team
at
solo
and
other
than
ephemeral
containers
which
push
your
debug
tools
onto
the
kubernetes
node.
C
This
allows
you
to
run
a
microservice
locally
on
your
desktop
and
have
it
connected
using
istio
into
your
your
service
mesh.
So
it's
participating
in
the
mesh
as
though
it
were
running
in
kubernetes,
but
you
can
actually
still
have
it
plugged
into
your
ide
stepping
through
line
by
line
with
local
debugging,
which
is
just
it's
a
superpower
that
I've
wanted
for
years.
A
C
A
Have
to
look
through
the
docs
I'll
I'll
definitely
have
to
go
check
it
out.
That's
a
that's
a
new
one
for
me
when
you
mentioned
it,
the
tool
that
came
to
my
mind
at
first
was
k-sniff,
which
is
a
coupe
ctl
plug-in
that
will
go
and
set
up
tcp
dumps
for
you,
which
is
really
cool
as
well
sabine.
Do
you
have
anything
else
in
the
console
land
to
share
with
us
about
keeping
the
mesh
up
and
debugging?
What's
going
wrong.
B
A
It's
a
really,
especially
interesting.
In
my
opinion,
once
you
start
figuring
policy
into
it
with
linkardi,
we
kind
of
fail
open,
basically
to
make
sure
that
the
requests
go
no
matter
what,
but
you
know
again
once
you've
got
that
policy
in
there.
How
do
you
know
what
connections
are
valid
and
not,
and
how
do
you
start
putting
all
those
pieces
together?
I
I
guess
we've
mostly
talked
about
debugging
so
far,
since
thank
you
sabine
for
getting
us
on
to
that
like
designing
for
failure
mitch.
A
C
C
Your
control
plane
is
down,
but
your
application
traffic
is
still
flowing,
so
everything
should
be
healthy,
but
when
you're
really
talking
about
a
microservices
world
where
new
endpoints
are
being
added
to
services,
all
the
time
where
endpoints
are
being
marked
as
unhealthy
relatively
frequently,
and
sometimes
that's
done
by
the
control
plane
while
other
times
it's
happening
in
the
data
plane,
it's
sort
of
like
the
good
news
is
you've,
been
driving
a
car
at
200
miles
an
hour
and
you
didn't
hit
a
brick
wall.
It's
still
going,
but
there's
no
steering
wheel
and
so.
C
A
Oh
man,
that
that's
a
that's
a
really
good
way.
It's
that,
like
yes,
mr
customer,
I
understand
you're
kind
of
down,
but
not
really
down.
Is
this
an
a
serious
outage
or
a
semi-serious
like
it's
one
of
those
where
yes
or
no
is
a
much
easier
question,
especially
if
you've
got
the
checklist
together,
so
yeah
super
cool
to
think
about.
B
For
console
service
mesh,
we
would
love
to
continue
the
discussion.
If
you
all
have
any
questions,
feel
free
to
ask
us
on
our
discuss
forum
or
check
out
our
docs
at
console.io
docs,
or
we
would
love
if
you
all
were
interested
in
contributing
to
our
repos.
We
have
the
console
repo.
We
have
our
console
on
kubernetes
repo
and
our
console
helm
repo.
C
If
you're
interested
in
getting
involved
in
the
community,
I
highly
recommend,
starting
with
the
user
experience
working
group
where
we
would
love
to
hear
what
you're
using
istio
for
and
what
your
day-to-day
life
and
interacting
with
istio
is
like.
You
can
find
the
links
on
the
slide
to
join
us
that
we
also
have
a
community
meeting
where
we're
interested
in
hearing
all
sorts
of
different
use.
Cases
for
istio
hope
to
see
you
there
soon.
A
Oh,
that's
really
great.
This
has
been
a
fantastic
conversation.
Thank
you
mitch
and
sabine.
I
really
enjoyed
it
finally
to
call
out
linker
d
in
our
community.
We'd,
really
love
everyone.
Who's
interested
to
come,
join
our
community
get
started
and
check
things
out.
I've
got
github
slack
and
twitter
links
up
there
and
we'd
love
for
you
to
join
us.
However,
you
can.
This
is
a
great
community.
Thank
you,
servicemeshcon
and
have
a
great
rest
of
your
day.