►
From YouTube: Debug Envoy Configs and Analyze Access Logs
Description
Can you use Istio without understanding Envoy config or logs? Probably not.
In this livestream, Denis and Greg who has been debugging Envoy and Istio across many releases will join Lin to discuss how to debug envoy config and analyze access logs, and do a live demo of their tools to help you to troubleshoot Envoy config and analyze access logs so you can run Envoy and Istio with confidence.
#istio #envoy #debug #accesslog
00:05 welcome + speaker intro
2:25 service mesh and Istio news
4:36 discussion related to debug envoy access log, envoy config
7:18 envoy access log analyze tool deep dive + Live DEMO
13:30 envoy config analyze tool deep dive + Live DEMO + answer live questions
28:33 Additional debug tips + answer live questions
41:58 wrap up
https://envoyui.solo.io/
https://github.com/djannot/envoyctl
https://github.com/GregHanson/engarde-viewer
To learn more about us, please visit https://www.solo.io
We are hiring! https://www.solo.io/company/careers/
Questions? https://slack.solo.io
Code Samples: https://github.com/solo-io/hoot
Suggest a topic to cover here: https://github.com/solo-io/hoot/issues/
A
A
A
This
is
len,
I'm
the
host
for
the
hood
live
stream
today,
where
we
bring
istio
envoy,
kubernete,
ebpf
and
graphql
technology
to
you,
so
you
can
be
very
well
prepared
at
your
job
and
be
the
best
I
am
so
excited
to
have
denis
and
greg
join
me
today.
They
are
our
expert
of
debugging
envoy
at
solo.
In
fact,
they
actually,
you
know,
created
some
of
the
toolings
to
help
us
to
do
the
job
better.
When
we
work
with
our
user
community
greg.
A
B
A
That's
awesome
yeah,
I
I
know
denise
is
one
of
our
experts
at
solo
that
he
really
makes
stuff
happen
and
he
can
always
pull
out
the
crazy
performance
numbers
regarding
envoy
and
istio.
So,
let's
get
to
the
news
for
this
week:
istio
khan
is
coming
next
week.
Can
you
believe
it?
So
if
you
haven't
registered
for
your
car,
make
sure
you
register
today
we
are
really
excited
about
the
agenda
of
instill
car
next
week.
It's
totally
free
and
it's
virtual
solo
has
released
general
availability
of
graphql
support
for
glue.
A
This
is
an
important
release
that
fix
a
critical
box
for
many
of
our
users.
I
believe,
denis
you
actually
running
into
the
spot,
working
with
our
users
when
you
have
a
class
namespace
of
virtual
system.
If
you
are,
you
know,
using
virtual
service
delegation
or
using
virtual
system
virtual
service
across
different
name
space,
different
cluster,
it's
going
to
be
ignored
without
this
fix,
so
the
fix
is
also
in
istio
1.12.6,
so
make
sure
you
update
to
these
seo
releases.
A
A
We'll
also
have
some
more
episode
planned
afterwards.
So
make
sure
you
hit
the
subscribe
button,
so
you
don't
get
so
you
always
get
notified
for
any
of
our
new
episodes.
That's
the
news!
Let's,
let's
go
to
some
of
my
questions
for
for
the
speakers
today.
Let's
see
so.
The
first
question
I
want
to
ask
is
for
you,
denis
and
greg:
how
long
have
you
been
debugging,
istio
and
envoy.
C
C
C
A
B
Yeah,
so
I
mean
I
I
I
guess
it
depends
on
which
debugging
you're,
referring
to,
because
I
mean
going
back
to
the
earliest
days
of
istio,
obviously
just
debugging
the
istio
code
and
making
sure
that
the
istio
control
plane
was
spitting
out.
The
correct
envoy
configuration
files
that
we
were
expecting
to
get
default,
routing
working
between
all
of
our
sidecars.
So
that's
one
type
of
debugging
and
then
of
course,
definitely
something
that's
picked
up
in
the
past
few
years,
as
istio
has
taken
off
is
debugging.
B
Basically,
my
expertise
in
istio
is
how
the
various
virtual
service
service
entries
destination
rules
and
all
those
other
fun
crds
translate
into
an
envoy
configuration
file.
So
when
something
doesn't
work,
why
is
my
request?
Failing?
I
became
that
guy
to
go
to
so
I've.
I've
been
parsing
access
logs
for
a
while
that
helped
debug
questions.
A
B
A
B
There
there's
a
lot
of
stuff
in
there,
and
this
is
a
really
good
picture,
just
kind
of
representing
all
that
information
jammed
into
a
single
line
and
everything
that
you
can
learn
from
it.
So
just
being
able
to
parse
that
into
a
nice
little
json,
pretty
print
structure,
you
can
actually
see
what
everything
is
in
there,
rather
than
having
to
look
up
the
hard-coded
format
in
the
istio
source
code,
but
so
I
saw
this,
I
thought
it
was
great
and
I
started
pointing
people
to
this
anytime.
B
They
had
debug
questions
that
they
were
pointing
my
way
and
basically
I
would
they
would
say,
here's
an
access
log.
Why
is
my
request?
Failing
I'm
just
like?
Well,
what
can
you
run
it
through
on
guard
and
tell
me
what
it
shows
and,
of
course,
the
next
thing
they
do?
Is
they
just
present
me
the
onguard
output
and
they
don't
actually
look
at
any
of
the
fields
and
figure
out
what's
going
wrong,
so
I
I'm
still
kind
of
a
middleman
in
that
situation.
B
So,
going
a
little
bit
further,
I
actually
wrote
my
own
little
ui
tool,
cleverly
named
onguardviewer
and
basically
it
combines
onguard
with
a
nice
little
ui
and
this
ui.
Basically,
it
prints
out
the
onguard
json
output,
but
also
it
goes
directly
to
the
envoy
docs
and
pulls
the
docs
directly
and
links
them
to
each
of
the
fields
as
you
click
on
them.
A
C
I
think
we
like
one
question
right
yeah.
I
think
we
can
go
through
that,
but
perhaps
we
should
like
just
go
through
the
perhaps
other
parts
of
the
access
logs
and
really
look
at
the
full
power
of
that
before
we
switch
to
this,
because
it's
a
really
good
topic
right,
like
by
degrees,
this
access
logs,
that
you
get.
That
tells
you
a
lot
about
what's
going
wrong
and
then
you
have
this
big
conflict
dump
like
is
speaking
about
right.
That's
you
want
to
look
at
when
you
want
to
understand.
C
C
You
don't
even
know
that
there
is
this
response
flag
right
and
this
response
flag
is
like
a
life
saver
right
when
you
are
working
on
history
right,
like
you,
have
this
five
or
three-
and
you
don't
know
where
to
look
at
this
response.
Flag
is
really
the
go-to
place
where
you
can
really
understand
why
I
have
a
503
right
and
I
think
it's
really
the
first
thing
that
you
look
at
right
and-
and
you
have
the
example
here
right.
B
Yeah,
this
is
one
of
those
things
where
I
would
get
on
guard
our
output
and
I
would
immediately
have
to
go
to
the
envoy
docs
just
to
see
what
exactly
these
response
flags
translate
to
and
figure
out
why
my
request
might
be
failing
based
on
the
response
flag.
So
I
I
know
I
I
said
I
wrote
this
for
customers,
but
really
I
use
this
all
the
time.
B
C
It's
exactly
what
I
was
thinking
about
like
in
my
case.
I
look
at
the
access
logs,
I
see
the
flag
and
I
never
remember
them
right
and
then
I
go
to
google.
I
think
my
one
of
my
top
searches
envoy
access
logs,
because
I
know
if
I,
if
I
type
n
voice,
exploits
I
directly
go
to
the
documentation
page
where
I
have
this
response.
Flags.
A
B
A
C
Exactly
that's
exactly
what
happened
like
yeah.
If
you,
if
you
don't
mind
sharing
my
screen,
I
can
go
through
a
quick
demo,
so
it
will
probably
be
easier
for
people
to
understand.
You
know
the
idea
behind
it
right.
So
you
are
probably
most
of
you.
You
know
familiar
with
the
booking
for
application
right
and
you
can
find
the
details
if
you
go
to
the
documentation.
C
You
know
you
have
this
product
page
talking
to
the
reviews
with
three
versions
and
the
details
and
the
reviews
talking
to
ratings
right
and-
and
it
looks
like
that
right
so
let's
say
you
have
deployed
this
and
you
want
to
change
something
right.
You
want
to
change
like.
Instead
of
having
like
a
half
and
a
half,
you
want
to
have
like
most
of
the
requests
going
to
the
v1,
for
example
right.
C
So
I
could
just
like
create
like
a
virtual
service
like
that
right
and
I
could
just
like
apply
it,
and
I
would
go
there
right
and
I
would
see
an
error
like
it's
not
available
right
and
for
that
purpose
that's
really
a
configuration
error
in
the
istio
side
right.
So
the
good
thing
is
that,
because
it's
a
configuration
error
in
store
right,
I
can
run
the
istio
cutter.
We
spoke
before
right
and
it's
very
obvious
right.
C
I
have
this
virtual
service
that
reference
some
subsets,
but
I
don't
have
a
destination
rule
for
them
right.
So
it's
good
right.
I
don't
need
like
to
go
and
pass
my
1000
lines
of
code
for
this
right.
I
can
just
you
know,
create
like
this
destination,
all
that
describe
this
subset
right
and
if
I
apply
it-
and
let
me
know
if
it's
too
small
lean-
and
I
can
you
know,
make
it
bigger.
C
Great
so
here
now
you
see
everything
is
fine
right.
I
can
have
like
most
of
the
time
it
goes
to
v1,
which
is
no
stars
and
10
percent
of
the
time
it
goes
to
v2.
And
again,
if
I
look
at
the
analyze
now,
it's
pretty
happy
right.
Everything
is
fine,
so
that's
kind
of
the
issue
that
you
don't
need
my
tool
for
right.
It's
like
an
istio
configuration
error
right,
but
now
let's
say
you
do
something
that
is
really
valid
in
term
of
istio.
C
Too
many
of
them,
if
I
apply
this
right
and
I
run
the
analyze
still,
everything
is
happy
right
because
I
didn't
make
any
mistake
in
the
configuration
of
istio,
but
now
the
behavior
is
not
what
I
expect
right,
it's
half
of
the
time
it
goes
to
v1
and
alpha
the
time
of
video-
and
I
don't
understand,
what's
can
be
wrong
here
right
and
basically
for
this.
That's
where
you
start
to
go
to
this.
C
Let
me
just
go
back
to
this
is
not
the
right
one
in
my
case,
it's
here,
so
you
you
get
like
the
all,
but
you
can
also
just
get
like
the
wood,
for
example
right,
so
you
can
see
what
are
the
roots
configured
here
you
can
see.
You
know
like,
as
I
said
all
here,
you
could
also
say
this
is
like
a
representation
of
the
configuration
that
is
not
too
complicated
to
understand
right.
C
C
You
could
also
do
like
this
or
minus
or
json
right,
and
that
starts
to
be
like
a
bigger
one
right.
So
that's
the
one
that
we
were
discussing
about
before
it's
pretty
small
configuration.
I
already
have
like
17
000
lines
of
car
right,
so
it
starts
to
be
quite
complicated
right.
So
that's
where
I
built
this
envoy
ctl
first.
So
there
was
like
a
kind
of
a
command
line
to
try
to
digest
this
and
show
that
in
the
table
view
so
that
it's
easy
to
understand.
C
C
I
do
it
on
the
product
page,
for
example,
oops,
sorry
and
then
I
need
to
take
it
a
bit
smaller
right.
So
what
you
can
see
here
is
that
it
shows
me
like
for
each
listener.
It
shows
me
like
the
roots
and
the
you
see.
I
have
like
some
good
amount
of
configuration
here
right.
Let
me
try
to
go
to
the
editor
of
it,
so
that
it's
easier
to
understand.
C
A
C
I
know
that
you
know
something
is
wrong
right,
that
I
understand
that
there
is
no
issue
in
my
istio
configuration.
It's
just
that
what
I
I
didn't
create
the
rule
for
the
right
target.
Basically-
and
I
can
let
me
just
also
capture
like
a
config
them,
so
we
can
use
it
later,
so
I
can
just
like
keep
this
config
down,
but
now,
if
I
go-
and
you
know
again
like
just
apply
what
was
this
file
here?
C
If
I
fix
my
problem
here
right,
I
put
like
9080
like
it
was
before
and
I
apply
it.
Then.
If
I
just
like
go
back
to
my
command
here,
I
can
see
I
fixed
my
problem
right
because
I
have
only
the
root
for
1980
now,
which
has
this
weight
right
and
you've
seen
it
before
working
right.
So
I
will
not
go
back
to
the
page
right,
so
I
started
with
this,
which
was
quite
good,
but
you.
C
To
add
this
minus
d
option,
I
don't
know
if
you
see
it
clearly
here
right,
so
I
started
to
add
this
minus
d
option.
You
know
to
to
filter
by
that.
But
then
I
wanted
to
filter.
You
know
by
the
root
or
finding
all
the
wood
that
have
a
specific
information
or
whatever,
and
it
started
to
be
like
really
complicated
for
a
cli
tool.
So
that's
where
I
basically
took
this
logic
and
build
the
envoy
ui.
C
C
So
you
just
click
previews
here
and
you
see
it
filters
easily
right,
but
then
I
could
do
like
1980
right,
so
you
can
have
like
it's
very
easy
to
navigate.
You
know,
because
you
have
all
these
filter
options
and
sorting
options
that
you
don't
really
have
in
a
in
a
cli
tool
right
and
obviously
the
the
rest
is
kind
of
similar
view
right.
C
Is
the
cluster
information
for
my
v1
subset
and
for
my
v2
subset,
and
here
I
really
have
access
to
all
the
details
and-
and
I
can
start
to
really
dig
into
the
the
configuration
right-
and
I
can
obviously
just
like
this-
is
the
cluster
config
them.
But
you
can
look
at
the
you
know,
listener
or
the
roots,
and
you
know
all
these
kind
of
things
and
just
before
I
I
I
see
if
you
have
questions
guys,
one
thing
that
I'm
people
ask
me
very
often
is
okay,
but
what
about
security?
C
Because
I
don't
want
to
send
my
config
them
to
you
guys,
because
you
know
it's
even
the
names
are
sensitive
right
in
the
company
in
many
many
cases
right.
So
I
just
wanted
to
show
you
that,
basically
everything
you've
seen
here
right,
like
if
I
refresh
you,
can
see
the
network
communications
happening
right.
It
just
download.
You
know
this
page
and
now,
if
I
you
know,
take
this
config
dump
you
can
see
here
there
is
nothing
that
is
sent
right.
So
basically
everything
is
done.
Client
side.
C
All
this
logic
is
implemented
in
javascript
in
the
client
side,
so
we
don't
have
any
data
sent
over
the
internet
and
completely
secure
for
that
right.
So
I
wanted
to
mention
that,
because
I
I
got
this
question
quite
often,
I
have
another
small
extra
thing.
I
can
show
you
after
but
I'll,
probably
pause
here
and
see
if
there
are
questions
and
if
you
want
me
to
jump
into
some
details
here,.
A
Yeah,
this
is
really
great.
Essentially,
users
should
use.
The
tools
is
still
provided
right.
It's
your
kind
of
analyze
to
help
you
with
basic
problems,
but
sometimes
it's
your
cardo
analyze,
not
smart
enough,
like
in
your
example,
you
give
a
port
doesn't
match
and
you
have
to
go
through
the
envoy
configuration
which
many
of
you
probably
have
gone
through,
and
this
is
a
really
nice
tool
to
help.
You
understand
the
envoy
configuration
to
be
able
to
filter
and
search.
I
think
I
personally,
like
the
ui
better
than
the
cri.
A
C
Yeah,
I
didn't
do
that.
We
didn't
really
open
source
that
part,
but
the
logic
is
there
right.
Let's
say
it's
just
that
you
know
the
cli
is
always
easy
to
maintain
and
push
a
new
version.
If
something
is
wrong,
the
ui
is
a
little
different
story
right
if
you
find
something
wrong
or
whatever
it's
a
lot
easier
to.
Let's
do
that
as
a
sass.
If
you
wish
right
and
as
I
said,
because
because
there
is
absolutely
no
data
going
through
the
internet,
there
is
no
real
point
in
having
running
it
locally.
C
An
extra
thing
very
quickly,
if
you
want,
is
just
like
because
even
with
this,
sometimes
it
was
not
solving
one
of
the
problem
I
am
which
is
more
like.
I
do
something,
and
I
don't
know
you
know,
I
create
something
like
a
new
resource
and
I
I
don't
know
what
what
it
is
doing
in
envoy
right.
It's
something
that
happens
quite
often,
I
feel,
and
if
you
go
to
this
tool,
you
can
check
out.
C
C
You
know
like
one
line
or
whatever
right,
so
we
can
share
that.
I
I
think
you
share
notes
after
the
roots
right,
so
I
can
share
with
you
the
snippet,
but
this
is
really
like.
You
know.
You
see
it
as
as
easy
as
that.
Right
so
basically
is
like
using
the
pc
or
json
right
putting
in
the
temporary
file
and
then
doing
a
using
this
json
diff.
It's
a
node.js
tool
to
just
do
the
div
between
the
two
and
that's.
B
B
If
you
define
a
destination
rule
and
you're
not
seeing
the
expected
behavior,
you
should
be
looking
at
your
clusters
so
just
beginning
to
understand
what
crds
you're
creating
and
which
parts
of
envoy
they
impact
is
really
giving
yourself
a
good
starting
point
for
tools
like
this,
because
then
you
know
I
need
to
look
at
the
listeners.
I
need
to
look
at
the
routes.
C
A
B
Yup,
true,
I
always
say
the
verse.
The
very
first
debugging
step
you
should
run
is
istio
ctl,
proxy
status
or
istio
ctl
ps,
because
you
could
create
as
many
rules
as
you
want,
but
if
something
else
is
broken
in
your
environment
as
preventing
a
config
from
being
pushed
you're,
not
going
to
see
any
of
those
rules
to
begin
with.
So
always
a
good
stepping
point.
C
But
that's
probably
because
you
are
a
very
early
adopter
of
istio
right
because
in
the
last,
like
five
or
six
versions,
it's
very
rare
that
you
have
this
kind
of
propagation
issue
right
compared
to
in
the
past,
where
it
was
probably
like
happening
a
lot
more
right.
So
for
me,
as
a
kind
of
user
or
like
that
is
only
really
intensively
using
easter
since
two
years.
A
A
C
B
A
Yeah
and
maybe
it
up,
we
all
prioritize
someday.
That
sounds
great.
So
so
we
learned
a
lot
about
denise
tool
about
onward
ui
and
the
only
ci
to
help
pass
the
reconfiguration
greg.
Do
you
want
to
show
us
something
about
your
tool,
because
the
way
now,
I'm
understanding
is
user
using
istiocado
to
and
analyze
to
analyze
the
config
after
the
config
is
pushed
to
istio
of
their
kubernetes
cluster?
B
B
Yeah,
there's
just
so
much
information
stored
in
an
access
log
that
can
definitely
provide
insights
into
an
environment
like
if
you're
ever
seeing
pass-through
cluster
on
any
of
your
access
logs.
That's
a
good
sign
that
you're
having
problems
or
if
you're
doing,
ingross
gateway,
routing
and
access
logs
on
your
osteoproxy
sidecar,
pointing
to
your
egress
gateway
that
you're
trying
to
route
the
traffic
to
that's
a
sign
that
something's
not
working
right
in
your
routing.
I
would
say
that
the
answer
depends
just
depends
on
the
exact
error
that
you're
hitting.
B
B
I
know
this
particularly
the
http
versus
tcp
log
fields
are
important
for
people
when
they're
debugging
a
log
because
a
lot
of
times
they'll
send
me
an
access
log
and
say
here's
the
failed
request
and
they'll.
Send
me
an
and
I'll
put
it
in
here
and
it's
a
perfectly
valid
request.
It's
just
a
tcp
access
log.
So
a
lot
of
the
fields
end
up
with
just
a
little
dash
in
it,
and
it's
just
like
at
that
point.
B
Is
this
an
error,
but
I
can
look
at
the
output
and
see
that
there's
some
bytes
sent
and
some
bytes
received,
so
a
request
technically
is
going
on
here.
It's
just
tcp
and
therefore
you're
not
going
to
see
a
lot
of
the
other
fields
that
are
there
by
default
in
your
access
logs
that
you
get
with
http
based
access
logging.
B
So
that's
always
something
good
to
make
note
of,
and
now
I've
noticed
as
I
had
to
update
my
tool,
because
now
they
have
udp
specific
access
log
fields.
So
I
had
to
update
my
logic
and
my
parser
to
make
sure
I
was
displaying
the
udp
field
specifically
so
always
something
to
keep
in
mind,
because,
depending
on
what
type
of
traffic
you're
having
in
your
mesh
http
tcp
tls,
your
access
logs
are
going
to
vary.
So
don't
expect
every
access
log
to
look
the
same.
C
Yeah,
I
think
that
the
config
dump
is
probably
the
the
best
place
to
find
that
right.
You
would
look
at
in
the
table
at
the
the
name
of
your
target
right
and
then
you
would
see
the
matcher
and
you
will
have
the
full
matcher
there
right.
You
will
not
have
to
navigate
by
yourself
in
the
config
them
to
find
it,
and
then
you
could
see
the
android
configuration
directly.
B
And
something
else
with
virtual
services,
you
can
actually
specify
a
route
name
in
the
virtual
service,
so
that
way,
that
field
will
actually
get
logged
in
your
access
log
and
it
will
show
up
where
is
it
route
name
so
by
default
for
the
http
traffic?
It's
just
istio's
generated
name
of
default,
but
in
your
virtual
service
you
can
actually
specify
a
route,
a
unique
route
name
and
in
your
access
log
you
can
verify
which
logic
it's
actually
using
on
that
particular
request.
C
C
No,
I
don't
have
any
other
in
mind.
I
think
it's
like
a
great
set
right.
You
you
it's
really
good
at
the
beginning
to
learn
the
studio
by
the
examples
and
looking
at
the
behavior
that
you
get
after
you
apply
something,
but
as
soon
as
you
start
to
be
really
serious
with
it,
it's
really
interesting
to
start
to
look
at
what
is
the
corresponding
android
configuration
you,
you
will
never
read
a
full
dump
right.
That
makes
no
sense.
It's
not
for
humans.
C
It's
for
machines
right,
but
looking
at
a
small
part
of
it
is
very
interesting
and
then
you
could
even
discover.
Oh
there
is
this
nice
setting
in
android.
Is
it
exposing
this
to
you
in
one
way
or
another
right,
and
then
you
can
start
to
research
and
you'll
find
new
options
in
seo
that
you
didn't
even
know
right.
So
it's
quite
interesting.
B
And
then
that
opens
up
the
door
to
envoy
filters
if
you
find
a
super
super
tiny,
small
xyz
field
that
isn't
exposed
to
non-voy
and
then
knowing
how
to
apply
that
with
an
envoy
filter
just
having
a
general
knowledge
of
how
the
listeners
clusters
clusters
routes
are
constructed,
so
that,
when
you
are
patching,
this
envoy
object
with
this
desired
field.
That
your
envoy
filter
is
adding.
B
A
Yeah
totally,
I
guess,
if
you
ever
need
to
write
your
own
onward
filter,
you
know
I
can
see
denise
config
analyze
tool
can
be
really
useful
for
you
by
the
way
I
just
sent
out
greg's
tool.
Also,
the
viewer
for
ingot
so
feel
free
to
check
them.
Out
with
that,
I
want
to
take
a
minute
to
thank
our
fabulous
speakers
today
to
show
us
their
tips,
thank
them
for
all
the
energy
they
put
behind
to
create
these
tours,
so
they
are
available
for
each
of
us
to
use.
B
C
Yeah
and
for
me
I
would
say,
like
the
same,
if
it's
like
an
issue
or
an
improvement
in
the
tour,
the
github
repo
is
the
best.
Otherwise
we
are
always
on
slack
like
we
are
kind
of
connected
to
slack
right.
So
just
go
to
the
solo,
I
o
slack
and
you
will
find
us
there
all
day.
So
it's
probably
the
easiest
way
to
interact
with
us.
A
Yeah
definitely,
we
are
all
on
the
solos
flag.
This
is
great.
Thank
you
so
much
again,
and
I
also
want
to
thank
you,
everyone
for
joining
us
and
watching
this.
I
hope
you
find
this
interesting.
If
you
do,
please
make
sure,
give
us
a
thumbs
up
and
also
click
on
the
like
and
subscribe,
so
you
don't
miss
any
of
our
future
episodes
and
also,
if
you
have
ideas
on
what
you
would
like
to
see
in
one
of
our
future
live
streams.