►
From YouTube: Envoy on Fire: A Practical Look at Debugging a Service Mesh - Lita Cho & Ryan Cox, Lyft
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Envoy on Fire: A Practical Look at Debugging a Service Mesh - Lita Cho & Ryan Cox, Lyft
In this talk, presenters will share lessons from several years of experience running Envoy in production at scale. They will explore practical techniques for triaging issues in a service mesh, along with the intuition behind them. The presenters will cover a broad range of topics including traffic capture, issues specific to GRPC, health checks, and techniques useful during incident mitigation. The talk will end with a deep dive into Envoy stats and their use in resolving issues.
https://sched.co/UaYD
A
I've
been
at
lift
for
a
few
years
and
I've
been
on
a
number
of
different
impro.
Related
teams.
I
currently
lead
a
team
that
works
on
resilience,
engineering
and
you
might
have
seen
some
of
my
team's
work
in
the
open-source
all-boy
repos
these
days
with
the
adaptive
concurrency
work,
that's
going
on
in
there,
and
so,
if
you're
interested
in
chatting
about
that,
I
can
hang
out
later
and
discuss
that.
But
let's
frame
the
talk
a
little
bit,
so
this
is
not
introduction
to
service
mesh
I
can
by
now
cube
con.
A
You
probably
had
a
dozen
of
those
already.
What
we're
going
to
do
today
is
look
at
what
to
do
when
your
control
plan
or
your
data
plane
is
not
behaving
in
a
way
that
you
expected.
So
this
could
be
during
dev
and
test,
or
it
could
be
a
production
incident,
basically
how
to
tease
further
information
out
of
on
boy
itself
and
we're
not
going
to
get
into
specifics
of
app
mesh
or
SEO
or
other
control
planes,
but
I
believe
that
most
of
what
we
will
convey
is
usable
for
any
control
plane
that
uses
envoy.
A
So
this
is
a
talk
in
three
parts.
We
will
first
start
with
the
consideration
of
config.
Config
can
be
a
very
complex
thing
in
Envoy
you
might
get
config
from
a
bootstrap
file
and
from
three
other
servers
that
combines
itself
into
a
single
Envoy
instance,
and
so
reasoning
about
the
origin
and
the
provenance
of
that
config
data
can
be
a
complex
thing.
A
B
Sounds
good,
let's
get
started
so
again.
This
is
specific
to
on
boy
configuration
and
I
start
thought.
We
should
start
off
with
a
high
level
picture.
So
humble
have
two
types
of
configuration.
One
is
the
bootstrap
file,
that's
a
static
configuration
that
lives
either
your
VM
or
your
container
and
envoy
will
read
that
and
then
start
up
and
we'll
use
that
configuration
for
the
lifecycle
envoy
running.
B
However,
managing
a
bunch
of
those
bootstrap
files
is
cumbersome
and
so
on.
Void
net
also
has
dynamic
configuration
known
as
the
xes
api
sourced,
our
DS
api's,
and
basically
that
connects
to
a
control
plane
that
I
sense,
dynamic
configuration
while
envoy
is
running,
so
there's
no
need
to
hop,
restart
or
restart
envoy.
You
could,
on
the
fly,
send
configuration
to
envoy
directly
and
then
have
quick,
quick
deployment
and
roll
backs
as
well,
and
so
when
you're
debugging
and
kind
of
need
to
figure
out
like
what
is
the
main
issue.
B
Is
it
a
xes
problem
or
is
it
a
problem
in
my
bootstrap
file
and
some
tips
I
have
at
a
high
level?
Is
that
usually
startup
issues,
especially
if
they're
global
tend
to
be
due
to
malformed
configuration
and
envoys
will
be
verbose
if
there
is
malformed
figuration,
giving
it
to
it?
The,
and
also
but,
however,
with
a
if
there's
a
issue
with
your
dynamic
configuration
envoy,
keeps
the
last
well-known
config.
So
you
won't.
B
If
you
don't,
have
the
popper
of
the
observability
in
place,
you
may
not
notice
until
envoy
scales
up,
which
is
also
not
a
good
situation
to
be
in.
So
what
is
our
tools
at
our
disposal?
So
the
tools
I
use
to
debug
configuration
issues
is
config
dump,
which
is
an
admin
endpoint
that
you
still
have
to
set
up
in
your
Envoy
proxy,
but
its
interface,
where
you
can
curl
and
I,
will
show
you
exactly
what
envoy
is
running
at
runtime,
both
static
and
dynamic.
So
it's
very
useful.
B
You
have
to
set
up
the
admin
interface
at
lift.
We
set
it
up
such
that
there's
a
local
host,
so
you
have
to
have
privileged
access
to
curl
this
endpoint.
But
then
you
can
see
all
this
data
once
you're
in
the
box.
The
other
ways
I've
seen
people
set
this
up
is
they
have
like
a
firewall
or
secure
network
to
curl
this
information
from
the
VM
or
contain
directly.
The
second
thing
that
you
could
leverage
is
Trace
level
logging.
This
is
super
useful,
especially
if
you're
trying
to
set
up
something
new
in
envoy.
B
So
let's
say
you're
setting
up
a
new
XDS,
API
or
a
new
feature.
Chances
are
you're,
not
gonna.
Get
that
right.
The
first
time
and
and
looking
at
the
trace
level.
Logging
will
tell
you
exactly
what's
going
on
and
where
the
problem
is
arising
so
and
oftentimes
they
were
doing
a
bug
report
the
maintainer
Zoar,
the
community
will
ask
you
to
post
a
trace
level
logging
along
with
your
issue,
and
so
this
is
you.
B
Okay.
So,
let's
go
into
like
a
realistic
scenario:
a
healthy
envoy
when
it
starts
up
usually
ends
with
this
log
line,
which
is
all
dependencies
are
initialized
starting
workers.
So
that
means
envoy
is
now
able
to
take
traffic
and
the
workers
are
the
request
workers.
So
that's
like
the
happy
path
and
what
you're
looking
for
when
you're
looking
at
the
logs.
These
lots
are
not
the
access
logs
that
Ryan
will
go
into
later.
B
In
the
talk
these
logs
are
the
sugar
logs
that
Envoy
minutes
in
the
main
thread,
but
now
you're,
seeing
this
you're,
seeing
like
an
issue
with
some
sort
of
config
you're
like
it
makes
a
charity
stream
call
and
like
realizes,
there's
no
healthy,
upstream
and,
let's
say
you're,
trying
to
set
up
your
control
plane.
This
is
the
first
time
you're
setting
it
up
and
you're,
seeing
this
error
and
Envoy
Haynes
says
amidst
this
log
line
a
couple
of
times
and
restarts
and
it
keeps
going
to
this
loop.
So
what
do
you
do?
B
This
log
is
not
enough
to
kind
of
figure
out
what's
going
on,
so
let's
just
try
running
on
void
with
with
a
with
trace
level.
Logging
and
so
I
concatenate
a
lot
of
a
lot
of
the
data
here
like
the
time
stamps
and
the
code
line
where
this
log
is
submitted.
Just
so,
it
will
fit
in
the
screen.
But
it
gives
you
a
lot
of
information
like
it
gives
you
debug
logs
as
well
as
trace
the
logs,
and
we
see
that
warning
again.
So
that
gives
us
a
form
of
reference.
But
ultimately,
oh
look.
B
We,
our
dns,
was
not
set
up
correctly
and
note
that
that
was
like
stated
in
the
debug
line.
But
what
was
happening
was
that
the
they're
so
healthy
upstream,
because
that
DNS
didn't
resolve
to
IP
addresses,
and
so
it
was
not
able
to
talk
to
the
config
server.
But
then
again
we
were
able
to
get
out
whoops.
We
were
able
to
get
more
data
through
these
traces
to
see.
What's
going
on.
B
So
let's
go
to
a
different
scenario
like
this
one
effects:
real
users-
let's
say:
you're
actually
running
envoy
in
production
and
you-
and
in
this
case
there
is
a
edge
proxy
that
your
customers
go
through
before
reaching
the
application.
And
you
notice
that
Oh,
certain
users
are
getting
404s
and
you
get
a
customer
report
about
it.
So
the
application
does
in
and
tries
a
triage
this
issue
and
when
she
like
debugs
the
endpoint-
and
let's
say
this
application
is
stateless.
She
gets
a
200.
She
cannot
repeat
this
404
or
we
produced
this
error
whatsoever.
B
So
now
she's
like
confused
and
goes
to
the
networking
team
Elia
must
be
a
networking
issue
goes
to
the
front
proxy,
and
so
now
it's
your
job
to
figure
out.
What's
going
on
so
what's
what's
what's
on
our
tool
belt
look,
can
we
do
to
figure
out
what's
happening
here?
So
let's
look
at
convict
up
so
like
there's
get
when
you
curl
this
endpoint
in
envoy.
There's
gonna
be
a
lot
of
configuration
here,
but
it
helps
to
kind
of
look
at
the
high
level
to
see
what
shows
up.
B
So
you
have
your
bootstrap
configuration,
you
have
your
cluster
and
listener
and
routes,
but
any
of
you
have
like
if
you
have
secret
discovery,
set
up.
That
will
also
show
up
here,
and
so
that's
a
good
thing
to
note,
like
you
should
separate
by
type
and
then
try
to
pick
out
where
to
dig
into
note
here.
B
Also
like
again,
it
shows
you
both
static
as
well
as
dynamic
configuration
everything,
that's
dynamic
will
have
a
version
info
with
it
and
that
version
is
set
by
the
control
plane
and
if
you
have
any
issue
with
your
config,
not
updating
or
like
your
XS,
not
updates
getting
rejected.
That
version
info
will
not
match
with
your
control
plane
and
that's
also
another
signal
that
something's
not
working
right
in
your
system.
B
So
when
you,
when
I
think
of
404s,
it
usually
has
to
do
with
routes
so
like
about
tables
misconfigured
in
some
way,
and
let's
say
you
look
into
the
Envoy
access
logs,
which
Ryan
will
go
into
later
and
they're
getting
you're
using
nr,
which
is
an
envoy
with
specific
response
and
Ryan
will
go
over
all
the
response
flags
later
in
the
talk
but
NRCS
for
no
route.
Usually
it
means
that
envoy
returned
a
404,
not
the
application,
meaning
that
there
was
no
route
table.
That
I
could
find
out.
B
You
couldn't
find
a
route
to
match
the
traffic
with,
and
so
so
let's
dig
into
the
route
table
and
see
what's
going
on,
so
you
look
at
all
you
like
aggregate.
All
the
route
table
are
like
the
dynamic
route
configs
from
all
your
hosts,
and
you
notice
that
the
old
ones
don't
have.
Oh,
the
old
ones
have
the
route
table
properly,
but
the
new
ones
don't.
So?
B
What
does
that
mean
to
admit
again
when
I
mentioned
earlier
in
the
talk
envoy
keeps
the
last
well-known
config,
that's
how
it
stays
resilient
to
bad
configuration
changes
in
XDS,
and
so
then,
when
the
new
boxes
don't
have
a
well
good
known
what
config
to
go
off
of
it's
good.
It's
always
still
gonna
run,
but
it's
not
and
take
traffic,
but
it
won't
route.
It
has
no.
It
has
no
routing
information
to
send
that
traffic
to
so
you
know
that
there's
something
wrong
with
your
xes
configuration.
B
So
knowing
that
like
there
is
an
issue
with
configuration,
you
could
see
that
looking
at
the
server
loss
you,
it
states
clearly
now
why
it
got
rejected.
So
there's
a
duplicate
entry
and
that's
why
it
rejected
the
update
and
we
and-
and
thus
you
know,
to
rollback
your
bad
access
change
and
then
you'll
be
able
to
serve
your
customers.
B
Traffic
note
here
that
there
is
also
stacks
that
you
could
like
alarm
on
this,
and
so
as
long
as
you
have
the
proper
observer,
you
shouldn't
beep,
you
shouldn't
get
to
this
state,
but
note
that,
but
also
note
that
all
the
way
we'll
tell
you
exactly
why
I
reject
it,
also
to
help
blue
cause.
Your
issues
and
the
other
thing
I
wanted
to
mention
on
the
side-
is
that
if
you
decide
to
statically
define
your
route
table
and
your
bootstrap
file,
that
would
be.
B
You
know
that
would
be
a
critical
error
and
boy
won't
start.
But,
however,
if
you
use
X,
yes
it
will,
it
will
run,
but
it
won't.
It
will
end
up
just
rejecting
the
update,
so
some
quick
config
takeaways
before
we
move
on
to
logging
they're
the
main
tools
or
that
I
leverage
is
the
config
dump
as
well
as
trace
level
log
E,
and
they
will
show
you
both
static
and
conferring
table,
will
show
you
both
static
and
dynamic,
dynamic
level
configuration
that
which
I'm
always
running
at
runtime
tray
slots
and
then
the
oh
sorry.
B
The
other
thing
I
wanted
to
mention
is
that
dynamic
configuration
envoy
will
use
the
last
known
well,
last
well-known
good
config,
that's
how
envoy
stays
resilient,
and
so
you
won't
see
any
dynamic.
It
config
issues
on
scale-up
unless
you
have
the
proper
observability
in
place
and
then
also
trace
level
logging.
It's
very
useful,
especially
if
you're,
enabling
a
new
feature
or
if
you
want
to
dig
into
what's
going
on
it,
gives
you
a
lot
of
data.
I
didn't
go
into
I
didn't
go
too
deep
into
it,
but,
like
a
will,
show
you
the
xes
payloads.
B
It
will
show
you
the
headers
that
are
being
set
in
the
data
plane
so
and
also
like
the
bytes
information.
If
you
want
to
so
definitely
use
that
if
to
your
advantage,
but
note
that
it's
very
verbose,
so
you
could
also
customize
the
level
of
tracing
to
this
component
of
where
you
care
about
whether
that's
like
the
upstream
section,
the
config
area
or
the
HDTV
connection
manager
like
you,
could
you
can
enable
tracing
for
those
specific
components?
A
So
to
motivate
this
section
just
a
very
small
toy
example
here
you
might
imagine
that
you
have
a
people
app
that
there's
a
weather
app.
It
calls
into
some
edge
proxy
that
transits
to
a
sidecar.
The
service
performs
some
weather
forecasting
or
data
collection,
and
as
part
of
that,
it
calls
out
to
some
external
api's
to
get
data.
So
I
guess
this
is
like
a
an
interplanetary
weather
app.
A
It
calls
out
and
gets
the
weather
on
Mars
here,
but
you've
got
some
problems
with
this
third
party
call
and
you
want
to
know
what's
going
on
in
each
of
those
calls,
and
so
you
want
to
know
a
lot
of
details
about
that.
So
we're
gonna
use
envoy
logs
to
do
that,
and
so
in
this
section
number
one
we're
gonna
look
at
how
to
configure
a
log
for
traffic.
That's
exiting
the
mesh,
so
the
point
I
want
it's
gonna
make
your
super
explicitly
as
you'll
notice.
A
There's
no
arrow
from
this
service
directly
to
the
API,
there's
only
basically
a
line
that
goes
from
the
service
to
envoy
and
then
out
to
this
third
party.
So
this
is
known
as
using
envoy
for
egressing
your
external
traffic.
Sometimes
people
call
this
I
believe
a
forward
proxy
to
proxy
traffic
outside
of
your
system,
data
and
data
center.
So
we'll
look
at
how
to
set
that
up
and
what
some
benefits
are
we'll
dive
into
the
filtering
capabilities
of
access
logs.
A
This
is
generally
done
by
setting
up
a
dedicated
listener,
and
so
you
might
have
some
listeners
associated
with
traffic
coming
in
and
out
of
your
service
mesh
and
in
this
example,
what
we're
saying
is
we're
going
to
allocate
port
7001
to
all
traffic
going
to
this
external
service
and
so
the
the
code
that
is
actually
reaching
out
to
this
API.
It
won't
use
that
that
URL
explicitly
it
will
call
it
a
local
hosts
port
7001.
It
will
specify
a
host
header
of
API
on
nasa.gov
and
then
envoy
will
route
that
traffic
out.
A
So
what
are
some
benefits
of
doing
this
big
big
benefit
is
that
you
get
all
the
stats
you
get
all
the
same
stats
you
would,
if
you're
using
envoy
at
the
edge
or,
if
you're,
using
all-boy
in
the
mesh,
and
you
get
it
in
a
sort
of
a
nice
unified
format.
Further,
you
get
the
explicit
retry
policies
of
envoy,
so
you
can
tweak
the
retry
policy.
A
Sorry
about
the
animal
wouldn't
be
a
cube.
Con
talk
without
pages
of
yeah
Mille.
In
fact,
I
asked
for
speed
back
on
my
slides
and
Lena
told
me
I
just
needed
more
Taylor,
Swift
I!
Don't
have
that
all
I
have
is
the
animal
for
you
sorry.
So
this
page
of
Yambol
defines
a
listener
and
it's
this
is
the
exact
type
of
config.
You
would
do
if
you
were
trying
to
set
up
this
type
of
egress,
and
so
you
can
see,
bind
this
localhost
dedicate
port
7001.
Listen
on
that!
A
If
you
look
down
to
the
bottom,
you
see
the
route
is
basically
saying,
send
everything
to
api
nasa.gov
and
then
here's
the
access
log
specifier.
So
even
if
you've
never
seen
one
of
these
before
you
can
probably
guess
what's
going
on,
this
is
kind
of
like
a
you
know,
printf
style
format,
string
here.
If
you,
google,
envoy
access
log,
it
will
land
you
on
a
really
excellent
page
of
the
documentation
that
outlines
what
each
of
these
fields
are.
All
the
possibilities
that
you
could
include
in
your
own
custom
format
strings.
A
A
So
that
gets
you
the
log
file
that
gets
you
lots
of
visibility
into
this
data.
That's
egressing
your
service,
but
maybe
you
realize
you've
just
got
too
much
data
now
and
you
need
to
slim
it
down
so
enter
the
filter
section.
So
the
filter
section
of
the
config
allows
you
to
specify
one
of
a
handful
of
very
powerful
filters
for
your
access
logs.
So
in
this
case
we're
saying
the
numerator
is
1
denominator.
Is
a
hundred
so
sample
effectively
1
percent
of
traffic
going
out
and
send
that
to
this
file
based
log?
A
You
might
notice
that
runtime
key
below
that
gives
you
the
interesting
capability
here,
which
is
to
turn
these
knobs
live
at
runtime
without
reloading
on
voice.
So
you
might,
for
example,
have
a
default
state
of
a
very
low
sample
percentage
and
during
an
incident
you
may
want
to
crank
this
up,
and
so
you
could
specify
a
higher
sample
percentage
in
real
to
get
more
fidelity
out
of
your
logging
system,
different
scenario.
Maybe
you
want
to
log
only
errors
for
external
traffic,
so
don't
care
about
the
to
hundreds,
don't
care
about
the
redirects.
A
You
want
to
know
about.
The
404
is
the
pipe
of
threes,
all
that
sort
of
thing.
So
the
first
thing
you
might
see
here
is
at
the
top.
We've
got
a
predicate
filter,
so
there
are
I
believe
two
of
these
that
are
that
are
supported
by
envoy
and
and
or,
and
so
in
this
case
we
have
the
or
filter
specified
by
a
list
of
other
filter
sort
of
nested,
underneath
the
first
of
which
is
the
status
code
filter.
A
It
basically
says:
look
at
the
status
code,
look
at
the
operand
and
if
it's
greater
than
or
equal
to
in
this
case
log
it
it
or
can
it
or
is
that
with
the
output
of
the
other
filter.
This
is
the
response.
Flag,
filter
I
actually
didn't
really
get
into
that
as
much
as
I
wanted
to
back
here,
I
I
did
I
did
circle
this.
You
might
notice
that
response
flags
in
the
in
the
access
log
format.
A
In
fact,
if
you
get
really
one
thing
from
this
talk
and
you're
not
familiar
with
the
response
flags,
it's
well
worth
your
time.
Just
spending
a
little
bit
of
time
studying
what
these
response
flags
are.
So
the
canonical
example
is
that
that
leta
alluded
to
earlier.
You
you
hit
an
end
point.
You
get
404.
The
natural
question
is:
was
this
on
where
they
gave
me
the
404,
or
was
this
the
remote
service
that
gave
me
the
404?
A
And
so,
if
you
all,
you
have,
is
response
code,
there's
really
no
way
of
reasoning
or
disambiguating
these
things.
But
if
you
have
that
that
response
code,
you
can
look
at
that.
You
can
reference
it
with
the
codes
that
are
documented
in
our
way
and
understand
precisely
who
was
responsible
for
returning
that
code
and
there's
there's
a
really
rich
set
of
data
in
there
that
can
accelerate
your
triage
capabilities
so
back
to
logging
external
traffic.
So
this
does
a
greater
than
or
equal
to
on
the
operand.
A
A
What,
if
you
want
to
log
slow
calls-
and
in
this
case
we'll
say
to
an
authenticated
external
service-
this
is
somewhat
contrived,
but
I
think
it
shows
the
power
of
the
filter
capabilities
very
clearly
again.
We've
got
a
predicate
filter
and
the
first
filter
that
we're
hitting
is
a
header
filter.
So
that
says,
look
at
egressing
traffic
does
it
contain
and
authorization
header
if
it
does,
and
the
duration
of
that
request
is
greater
than
or
equal
to
2,000
milliseconds
then
log
it.
A
So
you
can
probably
start
to
see
with
file
based
logging
with
these
filters
how
you
might
be
able
to
compose
some
really
sophisticated
and
powerful
logging
mechanisms
within
your
own
infrastructure,
but
simultaneously
you're,
probably
also
painfully
aware
of
the
limits
of
file
based
logging,
particularly
if
you're
deployed
in
an
environment
like
kubernetes,
where
maybe
you'd
want
to
keep
read-only
file
systems
or,
if
you're,
doing
something
fancy
like
with
all
these
filters
and
multiple
access
logs.
Maybe
you
have
like
five
logs
and
your
logging
system
really
wants
everything
on
standard
out
standard
error.
A
What
if
you
wanted
to
avoid
file
based
logging
altogether
so
envoy,
provides
this
mechanism
known
as
the
access
log
service,
and
it's
illustrated
here
in
blue,
so
the
general
idea
is
open.
Source
envoy
defines
an
endpoint
in
the
open
source
repo.
So
there's
this
proto
definition
there
and
then
you
as
a
as
a
customer.
A
You
implement
that
G
RPC
endpoint,
that's
what
the
blue
box
is
and
then
you
point
your
envoys
to
that
service,
and
then
it
streams,
VG,
RPC,
very
low
latency,
all
the
log
events
to
that
service,
and
then
you
choose
what
to
do
with
it
from
there.
So
this
is
kind
of
a
glue
when
I
was
probably
like
a
stateless
glue
service
that
basically
grabs
this
unpacks
it
and
then
hands
it
off
to
some
other
internal
system.
Maybe
it's
elastic,
or
maybe
you
treat
it
more
like
a
streaming
data
platform
problem.
A
We
do
this
at
lyft
that
I've
been
working
on
this
actually
over
the
last
couple
months
with
some
teams-
and
you
know,
few
observations
from
that
is-
is
recasting
this,
like
file
logging
problem
into
more
of
a
streaming
data
platform
problem.
It
lets
you
think,
a
little
more
clearly
about
what
you're
trying
to
do.
You
get
all
this
type,
safety
and
extensibility,
but
you
also
get
to
leverage
your
existing
system
so,
like
we
run,
we
run
flink
jobs
over
these
event
streams
in
real
time.
These
these
events
find
themselves
into
Druid.
A
So
we
get
near
real-time
access
to
these
events
via
a
sequel
interface.
They
find
their
way
to
hive.
So
we
can
query,
be
a
presto.
For
example.
Large
sets
of
this
data
and
different
teams
have
found
this
to
be
enormous,
ly
valuable,
and
it's
allowed
them
to
combine
these
data
sources
with
other
data
sources
for
like
much
more
powerful
use
cases.
A
So
it
sounds
a
little
bit
complicated,
but
it's
actually
be
straightforward.
So
this
is
what
a
config
looks
like
if
you're
pointing
an
envoy
at
the
an
access
log
service
implementation.
So
instead
of
saying
file
access
log,
you
say
HDTV
GRP
access
log,
and
then
you
give
it
a
cluster
name
and
that's
a
cluster
name
specified
like
any
other
cluster
or
an
envoy,
and
that's
all
there
is
to
it
you'll
notice.
A
The
filter
section
is
there
available
to
you
can
so
you
can
compose
those
same
filter
capabilities
together
to
give
you
not
non
file
based
access
logging
with
super
sophisticated
filtering
capabilities.
What
about
the
other
side
of
it?
So
what
about
what?
If
you're,
implementing
that
blue
box
that
access
log
service,
if
you've,
never
done
it
before,
might
sound
sort
of
daunting
how
to
implement
gr
PC
endpoint?
Well,
it
turns
out
there's
help
within
the
Envoy
repo
there's.
A
This
thing
called
the
go:
control
plane,
it's
kind
of,
like
you
could
think
of
it
like
a
Lego
set
for
composing,
your
own
control,
plane
and
so
there's
an
example
of
an
access
log
service
implementation
in
that
repo
and
I've
just
taken
this
directly
from
there.
So
you
can
see
really
just
like
a
half
a
page
of
code
is
that
is
the
meat
of
the
implementation.
There's
one
method:
you
can
see
what
it
does.
You
can
follow
it,
even
if
you
don't
know,
go
code,
basically,
loop
forever
go
pull
a
messages
off
this
stream.
A
Look
at
the
message
type
and
then
at
the
bottom.
You've
got
full
access
to
this
structured,
very,
very
rich
logging
event,
data
and
then
I,
say
rich.
It's
it's
extremely
rich
there's
all
these
are
very
sophisticated
timings
in
their
time
to
first
byte
time
to
last.
Buy
it
upstream
downstream
tons
of
super
useful
things
in
there.
I
did
I
did
kind
of
skip
past
this
too,
but
on
the
configuration
side
there's
these
optional
fields
where
it's
it's
like.
A
If,
as
these
events
are
coming
through,
you
can
choose
to
extract
request
or
response
headers
and
include
those
in
this
event
prior
to
it
being
dispatched
off
to
that
service.
You
know
so
again.
Maybe
you
have
some
headers
that
are
super
specific
to
your
environment,
but
very
vital,
maybe
identify
users
or
sessions
or
that
sort
of
thing
you
can
pull
them
out
of
here,
put
them
in
the
response
and
then
forward
that
along
and
have
full
access
to
that
in
the
pipeline.
A
So
that's
config.
That's
the
implementation,
not
actually
that
much
to
it.
So
some
takeaways.
If
you
wanna,
use
file
based
logging,
there's
it's
highly
configurable.
You
can
specify
the
format
to
your
liking,
very
flexible,
composable,
filtering
mechanism
for
access
logs,
as
you
saw,
we
can
combine
this
with
various
predicate
logic
to
get
really
sophisticated
use
cases
realized
and
then,
if
you
want
to
avoid
file
based
entirely,
you
can
use
this
access
log
service.
It
gives
you
tight
low,
latency
mechanisms
for
doing
pretty
fancy
things
in
terms
of
the
future.
A
I
think
there
there's
some
interesting
potentials
here
and
specifically
one
of
them.
My
team's
gonna
do
some
work
on
in
the
coming
year.
Is
this
notion
of
like
dynamic
fidelity?
So
maybe
you
have
your
logging
turned
way
down
or
almost
off,
but
dynamically
at
runtime.
You
can
have
a
knob
to
basically
throw
out
all
that
up
and
and
and
change
the
throughput
and
change
the
richness
of
that
that
logging
data
that's
coming
through
to
get
really
precise
information
about
very
specific
aspects
of
your
service
mesh.
A
Another
thing
that
occurred
to
me,
actually,
while
I
was
working
on
this
talk,
is
I,
think
there's
some
interesting
opportunities
for
collaboration
and
open
source
on
a
reusable
access,
log,
server
implementation.
So
I
think
you
don't
have
to
squint
too
hard
to
to
a
picture,
an
open
source
project
that
basically,
is
a
sync
for
these.
These
events,
it
grabs
it
in
via
G
RPC,
performs
perhaps
some
sort
of
optional
transformation
on
it
and
then
has
a
pluggable
back-end.
A
B
Actually
stature,
one
of
my
favorite
things
about
envoy
and
so
there's
a
lot
of
them
and
so
will
crack
a
little
bit
about
how
to
go
about
navigating
the
world
of
stats,
particularly
an
envoy,
so
I
decided
to
like
try
to
figure
out
a
taxonomy
of
stats.
If
that
was
possible
like
if
you
curl
the
admin
endpoint
slash
stats,
there's
overwhelmed
with
a
lot
of
data,
I
like
in
figure
and
that's
been
my
learning
curve.
It's
like
how
do
I
think
about
what
stats
I
care
about.
B
Where
should
I
be
looking
if
I'm
having
networking
issues
or
if
I'm,
trying
to
debug
or
I'm,
trying
to
roll
out
a
new
voice
feature.
So
this
is
what
I
came
up
with
I
well
with
the
help
of
the
docs
and
just
like
thinking
about
it
a
lot
but
know
you
I've
organized
them
to
like
boundary
stats.
So
these
are
your
listener
stats.
B
So
let's
go
into
the
each
one
of
these
little
sections,
so
down
down
shoot
stats
again.
These
are
requests
coming
in
to
the
proxy.
These
are
your
listeners
stats
when
you're
trying
to
set
up
TLS,
we
have
mutual
TLS
you'll,
be
looking
at
the
TLS
inspector
and
seeing
like
it's
my
it's
my
cert,
it
meant
as
long
as
the
traffic
correctly
am
I
getting.
What
are
the
certificates
I'm
getting.
So
those
are
the
things
where
were
these
stats
to
be
helpful
here?
Well,
you
when
you're
wanting
to
inspect
the
traffic
coming
into
your
proxy.
B
B
This
is
where
your
health
checking
will
live
and
service
discovery
would
live
again,
that
this
is
the
stats
are
like
you
care,
most
of
for
me
I
care
most
about
what
the
client
is
seeing,
because
at
the
end
of
the
day,
that's
like
that's
the
most
important
Wow,
like
so
I,
usually
check
the
view
of
of
5x
X
4x
ads
from
the
cluster
perspective,
which
is
I,
could
equate
it
to
the
client
perspective
and
the
process
that's
well.
This
is
the
actual
server
itself
or
the
process
of
envoy
self.
B
So
I
get
a
lot
of
questions
around.
Like
is
envoy
performant.
How
much
memory
does
it
take
how
much
CPU
utilization
this
is
take
on?
But
amidst
those
stats,
the
Envoy
has
that's
around
memory
usage
and
if
you
using
hot
envoys,
hot
restart
functionality.
This
is
where
the
stat
will
live.
You.
It
will
show
you
the
amount
of
connections
in
the
parrot
process,
as
well
as
the
child
process,
to
ensure
that
the
hot
start
is
working
as
properly,
but
also
this
would
be
good
to
monitor,
along
with,
if
you're,
using
like.
B
If
you
have
container
stats
for
your
VM
stats,
it
would
be
good
to
see.
Compare
those
to
see.
Do
they
match
up
is
something
funky
going
on
if
they're
not
munching
up,
and
so
that's
that's
extremely
useful.
That
envoy
admits
those
so
I
decided
to
go
into
some
of
my
youthful
stats
that
I
decided
to
share
with
you.
So
most
of
these
are
cluster
stats,
so
the
upstream
request,
pending
total
and
upstream
request,
pending
active.
B
So
if
usually
in
a
in
a
happy
path,
the
active
should
match
the
total
whenever
the
total
is
not
matching
the
unlike
the
total
requests
pending
total
is
a
lot
greater
than
request
pending
active.
That
means
envoys
request
to
is
backed
up
and
something
and
either
your
applications
not
responding
quickly
enough
to
your
requests.
So
there
there's
a
load
issue
there,
or
maybe
your
you
miss
configured.
How
much
active
requests
your
assistant
can
take.
B
So
that
maybe
there's
an
uni
degree
configure
there's
a
non-void
configuration
you
should
update,
but
usually
like
monitoring
those
when
you're
treating
issues.
These
are
the
stats
I'm
looking
at
to
make
ensure
that
your
system
is
running
along
pepper,
lay
or
if
it's
backing
up
and
can
lead
to
cascading
failure.
B
The
other
sets
I
look
at
is
service
discovery.
This
health
check
health
checking
is
a
very
important
aspect
of
my
job,
so
I
look
at
both
health
check.
We
use
active
health
checking
as
well
as
passive
health
tracking,
which
is
a
outlier
detection.
So
I
tend
to
look
at
that.
I
also
look
at
membership
stats,
so
that's
how
many
healthy
hosts
there
are
in
ideally,
membership
total
should
match
members,
membership
healthy
and
also
a
panic
routing.
B
So,
like
that's
on
voice
loud
way
of
saying,
your
mesh
is
not
converging
it,
and
so
we
page
on
this
if
this
ever
happening,
but
ultimately
there's
a
lot
of
you
again
like
I.
Didn't
I
wanted
to
put
a
list
of
all
the
stats
on,
but
it
didn't
fit
on
the
slide
so
and
I.
Don't
think!
That's
that
that's
the
experience
you
should
give
to
your
end
users,
which
is
your
application
developers.
So
at
least
we
provide
a
dashboard
to
make
the
network
understandable
and
transparent.
B
So
here's
a
normal
like
a
service
dashboard,
but
our
service
owner
can
see
like
what
are
the
network
requests
coming
in
and
what
a
network
request
I'm
admitting
out
like
what
service
and
we
show
what
services
you're
talking
to
as
well
as
what
services
are
talking
to
me.
If
we
have
that
data,
we
we
also
look
at
retry
accounts
as
well,
and
this
this
where
doesn't
show
it,
but
we
have
timeouts
as
well.
So
this
is
a
at
the
end
of
the
day,
like
I.
Think
most
people
do
that
most
companies
do
this.
B
Yeah
envoy
mitts,
a
lot
of
stats
and
you
can
do
a
lot
of
root,
cause
analysis
just
purely
on
stats,
which
is
pretty
amazing
but
again
like
it
requires
you
to
figure
out
where
we're
like
what
questions
like
am
I
worried
about,
like
it's
did
TLS
break
was
it?
Am
I
getting
a
bunch
of
500
in
my
question?
It's
my
request.