►
From YouTube: Observing and Troubleshooting your Microservices with Istio - Isaiah Snell-feikema & Douglas Reid
Description
Want to view more sessions and keep the conversations going? Join us for KubeCon + CloudNativeCon North America in Seattle, December 11 - 13, 2018 (http://bit.ly/KCCNCNA18) or in Shanghai, November 14-15 (http://bit.ly/kccncchina18).
Observing and Troubleshooting your Microservices with Istio - Isaiah Snell-feikema, IBM & Douglas Reid, Google (Beginner Skill Level)
Istio, an open platform to connect, secure and manage microservices has become the de facto service mesh standard. As developers migrate their legacy or their newly built cloud native application to Istio, one of the top challenges is observability and troubleshooting when things don't go well. This session will describe and demonstrate how developers can gain observability of Istio and the microservices deployed onto Istio, along with common troubleshooting tips to get you use Istio easily.
About Douglas
Douglas is a core contributor to the Istio effort, primarily focused on Observability and Integrations. Prior to working on Istio, he spent 4 years working on the Google-internal precursors to Istio.
About Isaiah
Isaiah is a core contributor to Istio networking. Prior to Istio, he worked on precursor service mesh related projects whose concepts were later folded into Istio. Before his time working on open source, he worked on IBM cloud services and infrastructure.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
Join us for KubeCon + CloudNativeCon in San Diego November 18 - 21. Learn more at https://bit.ly/2XTN3ho. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
A
Yeah
we're
both
core
contributors
to
this
tío
project.
What
we
thought
we'd
do
today
is
sort
of
walk
through
just
a
little
bit
of
an
overview
of
what
ICO
is
just
in
case.
There
are
people
in
here
who
haven't
been
hearing
about
it
or
service
measures
in
general.
Talk
a
little
bit
about
the
ways
that
an
in-service
machinist
in
particular
allows
you
to
sort
of
get
better
insight
into.
What's
going
on
in
the
service,
even
what's
going
on
with
this
to
itself,
and
then
Isaiah
is
gonna.
A
So
so
I
think
you
find
now
I've
probably
seen
this
feature
slide
and
heard
people
talk
about
SEO,
so
I
won't
spend
a
whole
lot
of
time
here.
I
just
wanted
to
call
out
a
couple
things
about
service
meshes
that
I
think
help
with
debugging
and
observing
micro
service
architectures
and
in
particular,
like
you
know,
it's
very
easy
to
talk
about
metrics
and
logs
and
traces
I.
Think
that's
what
comes
to
mind.
A
One
of
the
powerful
things
that
I
think
SEO
provides
as
well
relates
to
traffic
control
and
I've
highlighted
fault
injection
as
part
of
this
right.
So
as
you're
developing
new
versions
of
microservices
and
ready
to
roll
things
out,
we
can
do
makes
it
very
easy
to
canary-
and
there
was
a
great
talk
earlier
today
about
canoeing
services-
do
traffic
shadowing
to
have
traffic
splitting,
but
you
can
also
test
failure
cases.
So,
if
I
want
to
see
what
happens
when
a
particular
service
takes
too
long
to
respond,
I
can
simulate
that
behavior
very
easily.
A
A
We
have
mixer,
which
is
an
integration
point
for
a
CEO
that
is
providing
policy
enforcement
in
telemetry
collection,
export
to
various
systems
that
you
might
want
to
do,
for
traffic
analysis,
etc,
and
then
there's
a
certificate
authority
called
Citadel
which
is
in
charge.
If
you
want
to
turn
on,
you
see
auth,
and
we
have
an
ingress
for
traffic
coming
into
the
mesh,
as
well
as
an
initializer
which
will
inject
steel
everywhere.
A
So
I
think
I
like
the
slide
just
because
I
like
to
show
what
SEO
is
and
I
think
it's
easier
to
understand
this
way.
So
I
don't
know
if
it
helps
other
people,
but
so,
if
you
think
about
it,
CEO
right,
you
have
a
user
client
application
that
no
notice
that
is
trying
to
call
say
this
angular
service
and
that
angular
service
notice
is
trying
to
call
it
back-end.
Everything
else
should
be
sort
of
invisible
to
each
of
those
pieces
and
then
there's
a
couple
things
that
I
wanted
to
point
out
about
this.
A
A
So
yeah
I
just
wanted
to
go
briefly
over
the
the
kinds
of
things
that
a
service
mission
is
to
in
particular,
give
you
to
help.
You
understand,
what's
happening
on
your
mesh
with
traffic,
so
Isaiah's
gonna
talk
through
a
demo
on
debugging
at
the
proxy
level,
so
I
won't
spend
a
lot
of
time
here.
I
just
wanted
to
call
out
that
their
envoy
and
there's
been
a
couple
of
great
talks
on
envoy
this
session
or
this
conference
as
well
about
how
was
built
around
observability
and
so
I.
Think.
The
admin
interface
is
a
great
place.
A
If
you're
just
trying
to
figure
out
what's
happening,
you
can
look
to
provide
stats
for
each
instance.
You
can
dump
the
whole
config,
which
you
know,
depending
on
how
complicated
things
get
is,
is
worth
while
I
like
just
looking
at
the
access
logs.
It
comes
straight
out
of
envoy
last
night
when
I
was
just
sort
of
looking
at
the
talk.
I
noticed
there
was
a
bunch
of
traffic
on
my
test.
A
Cluster
and
I
was
wondering
what
was
happening
and
so
I
just
went
and
saw
and
can
see,
I'm
just
getting
slammed
by
just
random
web
traffic.
So
that's
the
kind
of
thing
that
can
help.
You
understand
why
you
might
see
a
traffic
spike
they're,
a
bunch
of
dashboard
are
compatible
with
the
default
metrics
in
SEO
and
they're
worth
taking
a
look
at.
You
have
mesh
wide
dashboards
per
service,
dashboards
and
dashboards
on
mixer
and
pilot
themselves.
So
you
can
see
how
mixer
and
pilot
are
exporting
things
about
their
configuration
errors
in
their
configuration.
A
How
often
they're
calling
the
API
server
that
kind
of
thing.
So
please
take
a
look
at
those.
If
you
don't
want
to
use
the
pre-canned
queries
in
the
dashboards,
you
can
quote
directly
to
Prometheus
or
another
metrics
back
end
and
run
queries,
and
so
this
is
an
example
of
I.
Just
wanted.
I
was
playing
around
and
I
applied
a
traffic
splitting
rule
between
the
three
versions
of
the
review
service,
I,
think
and
I
just
wanted
to
see
that
that
was
actually
happening.
So
you
can
see
that
you
know
real
quickly.
A
I
can
get
a
feel
for
how
many
requests
are
going
to
which
place,
and
so
there's
there's
that
as
well,
oh
yeah
and
so
then
distributed
tracing
I.
Don't
know
how
much
time
I
want
to
spend
here,
but
I
think
this
is
provides
an
another
unique
way
to
look
at
traffic
flowing
through
your
service,
because
you
can
see
the
dependencies
between
services
pretty
easily
and
and
where
latency
and
errors
might
come
in.
A
So
that's
something
that's
worth
mentioning
and
finally,
logs
echo
saying
we
generate
access
logs
for
every
request
is
coming
through
at
each
point
that
it
goes
through
the
mesh,
and
so
we
have.
The
fault
integrations
with
things
like
fluency
and
you
can
hear
I'm
using
that
to
use
Cabana
to
analyze
logs.
So
that's
something
to
remember
and
the
final
piece
that
I
think
is
worth
pointing
out
for
when
you're
debugging
and
trying
to
figure
out
what's
happening,
is
you
can
use
coop
control?
You
can
use
it
for
a
number
of
things.
A
You
can
use
it
to
see
the
logs
of
any
of
the
processes,
but
also
you
can
use
it
to
get.
The
actual
config
like
I
was
saying
earlier
for
each
component.
Each
little
subsystem
with
an
SEO
and
I
find
that
very
useful
and
I.
Think
it's
neat
because
the
by
using
the
API
server
we
make
that
stuff
available
through
standardized
tooling.
The
other
thing
that
I
end
up
doing
sometimes
in
development
is
editing
deployments
on
the
fly
in
changing
logging
levels
and
scopes
for
various
parts
of
the
system.
A
B
B
It's
sometimes
not
entirely
clear
how
su
is
actually
doing
these
things
and
that's
usually
one
of
the
first
questions.
I
get
when
people
come
up
to
me
and
ask
me
a
lot
of
these
features
seem
like
magic,
and
you
know
magic
is
great
as
long
as
it
is
actually
working
and
doing
what
you
expect
if
it's
acting
in
a
way
you're
not
expecting.
You
need
to
have
a
little
bit
of
background
knowledge
to
understand
the
system.
Actually,
a
troubleshoot.
B
So
if
we're
going
to
talk
about
debugging
and
troubleshooting
in
SEO,
we
have
to
start
with
our
proxy
the
proxy
we
use
in
SEO,
which
is
in
boy
there's
a
whole
host
of
reasons.
We
use
envoy,
first
of
all,
it's
battle-tested
using
production
by
lyft
used
by
a
lot
of
big-name
companies,
it's
also
very
lightweight,
which
is
incredibly
important
for
a
proxy
for
a
service
mesh,
because
you're
gonna
have
to
deploy
that
proxy
in
all
your
pots.
B
B
So
in
this
example,
we
have
inbox
UI
service,
think
like
Gmail,
basically
the
front
page
from
Gmail,
and
it
talks
to
a
messages,
back-end
service,
where
the
emails
or
the
text
messages
or
whatever
else
live,
and
in
this
scenario
the
operators
have
decided
they
want
to
test
out
the
behavior
of
the
UI
when
there's
a
bunch
of
lateness
latency
in
the
network
and
make
sure
that,
like
it,
actually
behaves
in
a
reasonable
way.
When
that
happens
now,
traditionally
you
you
could
do
this
by.
B
B
It
just
says
that
all
requests
going
to
the
messages
service
from
the
source
of
the
inbox
UI
with
this
special
header,
the
special
test
header-
should
delay
by
5
seconds
100%
of
the
time,
and
so
the
way
this
actually
like
flows
is
the
requests
starts,
as
it
normally
would
inside
the
UI,
and
it's
sent
as
it
normally
would
be
to
the
messages
service.
But
it's
captured
by
the
inbox
you
eyes,
sidecar
using
IP
tables
and
NAT
magic,
basically
and.
B
B
B
B
Inside
the
guestbook
UI
service,
you
can
enter
phrases
and
the
phrases
get
sent
to
a
Redis
database
back-end
and
stored
there,
and
the
UI
can
also
show
that
I'm
like
a
little
wall.
What
we're
gonna
do
is
we're
gonna,
add
a
version
two
of
the
service
and
we're
gonna
add
in
a
tone
analyzer.
So
it
looks
at
the
tone
of
the
phrase
that
you're
entering
and
based
on
that
it
will
like
annotate
the
phrase
and
say
like
it's.
B
B
Very
simple
stuff,
so
what
I've
done
for
this
demo
is
I've,
actually
done.
Coop's
ETL
work,
forwarding
from
localhost
15:03
to
the
pilot
API
and
the
pilot
API
has
a
rest
endpoint,
which
is
actually
where
the
invoice
call
to
get
their
configuration.
So
if
we
take
a
look
at
it,
you
can
see
this
is
the
registration
API.
B
B
If
you
notice
here
inside
the
array
of
hosts
for
gifts
book,
we
have
this
dot
1
to
12,
or
that
this
dot
12
IP
address
and
just
to
illustrate
a
point.
If
we
do
coop
CA
I'll
get
in
points
for
guestbook
we
can
see,
is
the
same
IP
address
it,
and
this
is
because
it's
actually
in
terms
of
implementation,
it's
actually
talking
to
the
coop
API
server
to
get
that
information,
so
we're
gonna
go
ahead
and
deploy
this
new
version
of
the
guestbook
service.
B
B
B
So
the
first
thing
that
any
traffic
in
this
particular
micro
service
application
hits
is
the
sto
ingress
controller.
So
we're
gonna
look
at
the
route
configuration
for
that
ingress
controller
and
basically
the
easiest
way
that
I
know.
If
doing
it
is
just
to
go,
do
a
coop
see
TLX
AK
on
it
and
do
you
have
PS,
X
and
I'll?
Show
you
how
that
looks
in
a
second
here.
B
So
that
basically
shows
the
running
processes
inside
the
container,
and
you
can
see
there's
a
this
pilot
agent,
which
is
just
a
little
management
process
and
then
the
actual
Envoy
that's
doing
the
proxy.
It
has
a
huge
number
of
arguments
sent
into
it.
The
one
we
care
about,
though,
is
this
service
node
argument,
and
basically
what
the
service
node
is.
B
Is
it's
the
name
of
the
unique
identifier
that
each
of
the
odd
boys
is
given
when
when
it
started
up
inside
of
Sto,
and
if
we
grab
it,
we
can
use
that
ID
to
look
at
some
of
the
other
api's
that
are
exposed,
so
we're
gonna,
look
at
the
routes
API
for
port
80
as
a
ingress
type
of
proxy
and
for
this
particular
node.
So
this
shows
the
routing
configuration
that's
actually
being
used
for
the
ingress
controller.
It's
a
super
simple.
All
it
is
is
for
a
wild
card
domain.
B
Any
domain
we're
going
to
route
to
this
cluster
reference.
Basically
in
the
Envoy
nomenclature,
cluster
is
just
a
collection
of
endpoints
and
a
way
of
discovering
them
and
then
also
a
load
bouncing
them.
And
so,
if
we
continue
on
to
this
next
API,
which
is
the
clusters
API,
we
can
see
oops
clusters.
B
B
B
B
So
we
go
back
forward
back
to
the
clusters
API.
We
can
see
instead
of
that
original
one
cluster.
We
have
these
two
new
ones
and
if
we
grab
the
service
name
again
and
look
at
the
registration
API
with
that
particular
service
name,
we
can
see
that
it's
just
the
dot
13,
which
is
version
2
of
guestbook.
B
B
Like
normally,
what
you
do
here
is
you'd
start
looking
at
your
metrics
and
making
sure
that
the
guestbook
v2
is
actually
handling
the
traffic
properly
and
was
healthy
and
looked
like.
You
could
scale
up
and
you
slowly
step
up
until
you
could
replace
version
1.
But
we
don't
have
time
that
for
that
so
I'm
just
going
to
replace
our
routing
rule
with
a
new
one
which
just
sends
everything
to
the
new
version.
And
now
it's
all
at
the
new
version
and.
B
So
that
was
pretty
much
that
was
pretty
much
the
entire
demo.
The
point
of
it
wasn't
really
to
say
like
write
down
these
steps,
and
you
can
call
these
api's
and
figure
things
out.
It's
more
of
the
principle
of
being
able
to
look
at
the
configurations
of
these
different
proxies
in
order
to
debug
your
system.
B
And
then
sometimes
it
can
be
really
useful
when
you're
debugging
a
scenario
like
this
in
a
service
mesh
to
look
at
the
perspective
that
each
of
your
services
has
inside
that
that
service
mesh
by
looking
at
the
configuration
of
the
proxy
that
it
talks
to
when
it's
talking
to
all
the
other
services
and
just
to
touch
on
the
thing
I
started
with
magic
is
great.
But
you
know
a
little
bit
of
understanding
really
goes
a
long
way
in
terms
of
no
debugging
issues
when
you
hit
them.
B
So
right
now
all
the
envoys
use
the
version
1
API
for
obtaining
new
configurations,
and
it's
written
in
Raths.
It's
all
restful.
So
it's
very
easy
to
call
the
API
yourself,
but
it's
also,
you
know
it's
inefficient
because
you
have
to
send
the
entire
state
to
the
proxies
and
it's
also
they're
pulling.
Instead
of
in
the
new
version,
it
will
be
pushing
and
it's
going
to
be
using
G
RPC
and
version
2
of
the
Envoy
API
s.
B
So
with
version
2
we
can
now
look
for
specific
changes
and
only
when
there
is
a
change
that
actually
matters,
we
can
send
it
on
to
the
proxies
and
only
the
part
of
that
change
that
matters
is
sent
on
to
it.
So
bad
news
is
that
little
demo
did
you're,
not
gonna,
be
able
do
it
in
point
8,
but
it
good
news.
Is
it's
going
to
be
much
better
performance
and.
A
It's
a
nice
segue,
so,
like
you
saying,
better
visibility
is
coming
and
one
of
the
command-line
tools
that
will
help
is
we
have
a
tool
called
SCO
control
and
in
point
8
they'll
be
a
proxy
config
to
mend.
Did
you
can
provide
a
service
name
and
we'll
go
and
pull
that
information
that
you
saw
him
step
through
in
multiple
steps
so
that
you
can
look
at
it
all
at
once
without
having
to
know
api's
and
do
all
the
copy
pasting
of
names
from
different
parts
of
they
can
fake
I.
A
Think
that's
gonna
be
pretty
valuable
as
you
try
and
understand
why
proxy
is
behaving
and
why
routing
is
acting
the
way
it
is,
there's
also
going
to
be
a
metrics
command
that
we've
we've
added
as
sort
of
an
experimental
thing
so
that
from
the
command
line,
you
can
just
see
all
right.
Here's
what's
happening
across
these
services.
This
is
where
traffic
is
hitting
and
you
could
do
something
like
see
the
difference
between
v1
and
v2
of
a
service
depending
on
on
how
you
label
things,
there's
also
been
some
work.
A
That's
been
committed
already
that
should
be
in
the
next
release.
On
adding
pages
UI
pages
for
each
of
the
SEO
components
and
also
is
available
to
link
into
other
libraries
that
allow
you
to
control
behavior
of
and
visual
instance
of
something-
and
this
will
be
useful
for
turning
up
logging
scopes,
like
I
mentioned
at
the
beginning,
which
I
find
useful
in
debugging
for
targeted
parts
of
an
application,
as
well
as
seeing
the
metrics
for
that
particular
instance,
and
memory
and
CPU
consumption,
etc,
and
then,
as
far
as
telemetry.
A
Typically
in
SEO,
a
lot
of
the
telemetry
is
collected.
Server-Side
and
that's
great,
if
you're
concerned
only
about
your
service
and
how
its
behaving.
But
it
doesn't
help
understand
what
clients
are
seeing
and
if
there
are
other
things
that
are
happening
before
traffic
arrives
at
your
service.
So
there
is
work
going
on
right
now
to
add
client-side,
telemetry
and
add
metrics
support
so
that
you
can
see
the
difference
between
client
and
server
for
a
given
service.
A
There's
also
work
design
work
going
on
to
support
telemetry
around
config,
especially,
is
config
rollout
and
errors
around
config
so
that
we
can
make
that
exposed
in
the
same
way
and
in
a
first-class
way.
Throughout
a
CO
and
there's
always
work
on
more
adapters
that
can
push
this
information
to
wherever
you
need
it,
so
that
you
can
do
further
analysis.
A
This
is
just
sort
of
a
pictorial
representation
of
was
just
talking
through.
This
is
what
the
alpha
version
of
the
control
page
looks
like,
and
what
a
command
might
look
like
from
the
command
line.
Interestingly,
I
put
this
to
your
pilot
just
to
show
that
SEO
itself
is
available
for
inspection
with
all
of
these
tools,
and
you
can
see
how
it's
behaving.
I.