►
Description
In this Hoot live stream series we dig into Envoy, the cloud-native proxy technology that drives the data plane for major service meshes (Istio, Consul Connect, AWS AppMesh, etc) and API Gateways like Gloo. Starting from the fundamentals, each demo-filled episode focuses on one functional area like observability, customization, security and more -- to help you get a deep understanding of how Envoy Proxy works, it’s role in cloud-native application architecture, and how to use it.
Time codes:
0:00 Intro
2:10 Admin Interface overview (/stats, /clusters, /logging, /config_dump)
13:30 Debug logs
19:10 Access Logs
24:50 Metrics collection (Envoy+Prometheus demo)
35:25 Tracing (Envoy+Jaeger demo)
Hosted every other Tuesday starting on July 21st at 1pm Eastern / 10am Pacific Time.
* Envoy Architectural Overview and Fundamentals
* Observing Envoy: Monitoring Metrics and Logs, Proxy Performance, and Troubleshooting
* Securing Envoy: Understanding Available Security Configurations and Best Practices
* xDS Dynamic Configuration and Control Plane Interactions with Envoy Proxy Data Plane
* Envoy Filters: What are They and How They Work, Understanding the Filter Chain, and Lots of Examples
* Advanced Envoy Filtering and Build Your Own Filters with WebAssembly
Learn More
About Solo.io https://solo.io
Questions? https://slack.solo.io
About Envoy https://envoyproxy.io
Code Samples: https://github.com/solo-io/hoot
Suggest a topic to cover here: https://github.com/solo-io/hoot/issues/new?title=episode+suggestion:
A
A
Android
is
a
cloud
native
proxy.
That's
distributed
in
high
scale
needs
to
be.
You
need
to
be
able
to
debug
it
and
understand
the
current
state
of
the
system
and
that's
what
observability
gives
you
so
observability
is
kind
of
crucial
in
the
sense
that,
without
it,
you're
kind
of
shooting
in
the
dark,
and
you
need
it
to
understand
the
state
of
your
system.
A
We're
going
to
do
this
interactively
with
an
envoy
that's
running,
and
I
encourage
to
ask
questions
if
there's
any
questions
feel
free
to
ask
them
as
they
come
I'll,
be
looking
at
the
chat
and
it
will
answer
them
all
right.
So
without
further
ado,
let's
talk
a
little
bit
about
the
envoy
admin
interface.
A
A
So
what
can
you
see
from
there?
You
can
see
android
stats
and
we're
going
to
show
them
right
now
and
also
again,
when
we
talk
about
metrics
collection
with
prometheus
and
stats
are
essentially
counters
gauges
and
histogram.
That
tell
you
things
about
the
current
state
of
envoy,
and
we
will
see
that
in
a
second.
A
So
things
like
how
many
requests
happen,
how
many
of
them
succeeded
how
many
of
them
failed
things
like
that
you
can
sit
down
with
config
dump
and
that's
essentially
the
current
internal
android
configuration
and
I'll
expand
more
when
you
get
to
it.
You
can
see
that
if
you
remember
from
the
previous
talk
from
two
weeks
ago,
the
status
of
envoy
cluster,
so
angry
clusters
are
dynamic
and
their
membership
envoy
can
determine
on
its
own
using
one
of
the
discovery
mechanisms
that
it
has
built
in
and
in
the
clusters
endpoint.
A
We
will
see
that
you
can
see
the
actual
membership
of
the
cluster,
the
actual
endpoints
that
are
a
member
of
the
cluster,
and
we
also
see
the
login
page
where
you
can
set
the
android
log
level.
So
let's
leave
the
presentation
for
now
and
show
in
our
handy
terminal
and
how
it
looks
like
so
first
thing:
first,
I'm
going
to
start
a
bunch
of
background
services
just
so
we
have
something
that
doing
requests
and
we'll
have
some
data
in
the
stats.
A
A
A
A
And
we
can
see
here
that
envoy
serves
its
admin
page
in
port
9901
and
that
it's
bound
to
localhost
and
it's
very
important
to
bind
it
to
localhost
and
not
to
the
any
address,
because
from
the
interface
you
can
do
some
powerful
things
to
android,
for
example
like
shutting
it
down,
so
pay
attention
to
that
in
your
setup,
all
right!
So
let's
go
to
this
url.
A
A
All
right,
android
started,
and
now
the
admin
page
will
be
available.
Here
we
go
all
right.
So,
let's
review
this
is
the
envoy
admin
page.
It's
in
that
management
port
that
we
specified
in
the
configuration
and
let's
review
the
the
interesting
endpoints
here
that
are
related
to
observability,
there's
a
bunch
of
stuff
here,
I'm
not
going
to
go
into
all
of
it
because
it's
a
lot
to
to
take
in
I'll
just
review.
Those
end
points
that
I
mentioned
in
the
in
the
slide.
A
So
first
thing:
first,
we
can
see
the
android
stats
and
these
are
a
bunch
of
essentially
numbers
that
tell
us
what's
going
on
with
andre,
and
you
can
see
that
the
format
is
dot
separated
and
going
from
general
to
specific.
So
cluster
are
stats
that
are
related
to
an
android
cluster,
the
cluster
name
and
then
the
stat.
A
So,
for
example,
if
we
look
at
our
cluster,
you
can
see
that
we
have
here's
the
upstream
request.
All
right,
you
can
see
that
we
have
in
our
sum
cluster
64
requests
that
return,
200
and
14
requests
that
return
400,
which
is
about
a
ratio
of
one
to
one
to
four
now
this
is
the
these
stats
are
what
they
were
when
we
access
the
stats
page.
So
if
we
refresh
it,
we
can
see
that
these
numbers
change-
and
this
is
expected
so
android-
has
stats
related
to
cluster.
A
A
Additionally,
it
has
some
general
http
stats.
You
can
see
here.
These
are
generated
by
the
http
connection
manager
and
one
last
thing
that
I
wanted
to
show
in
the
stats
page
and
that's
something
called
a
virtual
cluster.
Now,
when
defining
an
unvoiced
stat,
you
can
see
that
it's
in
the
granularity
of
a
cluster
right
so
cluster
listener,
and
but
sometimes
it
is
useful
to
get
more
granularity
out
of
the
envoy
stats.
So
let's
say
you
have
a
path
that
you
are
very
curious
to
get
specific
start
from
it.
A
For
example,
if
you're
doing
an
app
that
does
online
shop,
it
might
be
the
purchase
page.
So
you
want
more
detail
stats
on
that
pack.
Android
offers
a
way
to
do
that
and
we
look
again
at
the
configuration
and
we
can
see
that
it's
very
similar
to
what
we've
started,
but
in
the
http
connection
manager
configuration
in
the
virtual
host,
we
defined
something
called
a
virtual
cluster.
A
Now
a
virtual
cluster
is
a
way
to
group
http
stats
under
their
own
name.
In
this
case
it's
called
a
virtual
cluster
because
it's
like
its
own
upstream
stats,
it's
not
a
real
live
stream.
It's
a
virtual
upstream
right,
so
a
virtual
cluster
in
the
android
terminology-
and
we
can
say
you
know,
create
a
visual
cluster
name
action
and
the
requests
that
are
associated
with
it
are
those
that
have
the
path
beginning
with
slash
full
and
we
can
see
it
in
the
stat
page
right
down
here.
A
A
Now
anway
has
three
types
of
stats
like
most
systems,
a
gauge
a
counter
and
a
histogram,
and
I'm
gonna
talk
about
the
details
a
little
bit
later.
I
just
want
you
to
see
that
most
stats
are
regular
numbers,
but
some
of
them
are
also
look
like
this,
which
is
a
histogram.
So
if
you
are
curious
about
p99
and
sap99,
these
are
where
you
get
them.
You
see.
This
is
how
a
histogram
in
envoy
looks
like
all
right.
A
Let
me
start
anway
again,
so
we
have
the
admin
page
available
again
and
you
know
what
I'm
going
to
start
it
in
the
background
and
keep
the
configuration
here.
If
you
guys
need
a
reference,
is
there
any
graphical
distribution
can
be
used
together?
Yes,
there
is,
and
we're
going
to
show
a
demo
in
two
more
slides
and
thank
you
for
the
question.
Jeffrey.
A
Specifically,
prometheus
will
do
a
demo
with
prometheus
and
jager
all
right,
so
the
next
endpoint
that's
of
interest
for
observability
is
the
envoy
config
dump
so
envoy
remember,
has
ability
to
be
configured
through
remote
server
and
we're
going
to
talk
about
talk
about
this
in
a
future.
Talk
right,
xds
configuration
via
api,
and
now
you
kind
of
don't
know.
If
you
just
look
at
the
static
config,
it
wouldn't
tell
you
much
right,
because
the
configuration
is
coming
from
a
remote
server.
A
That's
where
the
android
config
them
endpoint
comes
very
handy
because
it
allows
you
to
see
the
con
internal
android
configuration
as
it
is
currently
after
it
process
the
configuration
from
the
xds
server.
So
if
your
android
is
configured
through
an
xds
you'll
see
the
current
configuration
that
you
would
not
see
in
the
static
configuration
file
right.
So
this
is
a
bunch
of
information,
and
here
we
go.
We
get
to
the
interesting
part
the
listener.
A
Now
this
will
be
very
similar
to
the
static
configuration,
because
I
don't
have
xds
here,
but
it's
important
to
know
that
this
is
how
you
can
see
the
what
anwar
is
doing
right.
A
lot
of
times
when
we
need
to
debug
what
envoy
is
doing,
whether
in
glue
or
initial,
this
is
our
first
stop
to
see
that
it
is
indeed
configured
as
we
expect
it
to
be
configured
by
looking
the
control
plane.
A
Configuration
another
end
point
of
interest
for
observability
sake.
Sorry
about
that
is
the
cluster's
endpoint.
So
envoy
has
dynamic
cluster
discovery
right.
So
the
members
of
the
clusters,
the
endpoints,
you
can
discover
them
automatically
and
again,
there's
the
question:
what
is
the
current
state
right?
What
are
the
current
members
of
a
cluster
and
you
can
see
that
in
the
cluster
endpoint,
and
you
can
see
that
some
cluster
has
one
member
this
ip
at
this
port
and
a
bunch
of
stats
associated
with
this
specific
endpoint
all
right?
A
Why
can't
I
see
it
here?
It
is
here's
the
button,
and
this
allows
us
to
change
the
envoy
log
level
by
default.
Android
doesn't
log
much
so
you're
not
going
to
see
a
lot
of
stuff
going
on
and
if
you
want
to
increase
the
log
level,
we
can
do
that
by
interacting
with
this
interface
and
I'm
going
to
show
how
to
do
that
right
now.
So.
A
By
default,
android
logs
are
very
quiet,
as
you
can
see,
android
doesn't
look
much
right
and
sometimes
it's
convenient
and
helpful
to
see.
You
know
what
envoy
is
doing,
especially
when
debugging
it
and
for
that
and
we
can
enable
debug
logs
and
the
backlogs
actually
make
android
spew
a
lot
of
information.
So
it's
not
something
we
want
to
enable
globally.
A
It
will
have
an
impact
on
performance
right.
These
logs
don't
come
for
free.
We
only
want
to
do
that,
probably
in
a
non-pro
environment,
when
we
want
to
debug
something
understand
something.
What's
going
on
all
right,
so
there's
two
ways
to
do
this:
one
we
can
add
dash
l,
debug
log
when
we
start
envoy
the
shell
debug
to
set
the
log
level
to
debug
right
and
now
you
can
see
a
lot
more
logging
for
every
request
and
we'll
go
through
those
in
one
second,
the
other
way
the
dynamic
way.
A
Is
to
do
it
dynamically
we
can
turn
him
on
and
off
dynamically,
using
a
curl
command
to
the
admin
interface.
Now
I
have
to
use
curl
because
for
security
reasons
it
has
to
be
a
post
request.
I
cannot
do
it
just
through
the
browser,
because
we
mutations
to
android
through
them
in
interface,
require
a
post
method
and
are
not
allowed
using
a
get
method.
A
A
And
now,
let's
see
what's
going
on
in
in
this
log
and
we'll
go
through
a
life
cycle
of
a
request,
yeah,
let's
scroll
up
a
bit
here,
we
go
all
right.
Let's
start
here
new
connection
and
you
can
see
it
has
c112
and
that's
a
connection
id
just
a
integer,
but
the
logs
that
are
associated
with
the
same
connection
will
have
the
same
connection
id
right,
so
c112
new
connection
created
that's
curl
connecting
to
android
a
c112
new
stream.
A
That's
curl,
starting
the
http
request,
new
stream
created
and
you
see
now
we
have
s
and
a
long
number.
This
is
this
internal
envoy
stream
id
right.
So
if
you
want
to
follow
logs
that
are
related
to
this
specific
http
request,
you
can
follow
this
id
this
s
id
right
so
we're
in
a
connection
112
in
this
stream,
and
now
we
can
see
that
and
we
received
headers
from
curl.
A
You
can
see
the
user
agent
is
curl
path
is
full
all
is
well
and
moving
on
a
bit
more
a
request
and
it's
just
a
get
request.
So
there's
nobody
and
we
matched
their
request
food
to
a
cluster
named
sum
cluster
and
now
I'll
start
sending
it
up
upstream.
This
is
unvoice
sending
the
request
to
the
upstream
cluster
and
you
can
see
that
it
added
some
envoy
headers
and
that's
it
now.
A
It's
kind
of
going
back
in
the
response
path
and
this
time
the
upstream
returned
400,
because
every
now
and
then
it
returns,
400
and
onway
sends
this
downstream
and
that's
it
right,
and
this
is
the
end
of
this
stream.
In
this
connection,
you
can
see
that
the
connection
is
closed
right
after
this
log
message.
A
A
Oh
thank
you
for
the
compliment.
I'm
happy
you
find
this
useful.
If
you
have
any
more
questions
or
things
you'd
like
to
the
answer
on
in
this
format,
please
feel
free
to
alright,
so
we
can
x
this
out
and
now,
let's
talk
about
envoy
exercise
so
anyway,
access
logs-
and
I
have
here
of
course
of
course.
Of
course-
we
have
an
example
for
android
access.
Logs
android
access
logs
are
logs
that
are
meant
to
run
in
production.
A
Here
we
go
so
hey.
Let
me
do
this.
This
is
not
important
right
now.
Here
we
go
all
right.
Okay,
now
it
looks
good
all
right.
So
what
we're
gonna
show
you
right
now
is
how
we
configure
access
logs
in
android
to
output,
http
request
into
a
file
now,
in
addition
to
output
the
logs.
We
can
also
add
a
filter
that
tells
us
what
logs
to
output.
So
again
this
is
based
on
the
previous
configuration.
We
saw
pretty
simple.
We
have
a
listener.
A
Http
access
logs
are
defined
inside
the
http
connection
manager
and,
if
you
remember
from
the
previous
http
connection,
manager
is
what
deals
with
everything
http
related
in
invoice,
so
it
makes
sense
that
the
access
logs
for
http
would
be
defined
here.
So
we
have
an
accessor
field
that
accepts
a
list
of
logs,
and
you
can
see
that
we're
using
the
file
access
log.
A
A
A
I
hope,
if
there's
any
questions
or
feel
free
to
ask,
I
will
mention
that
there's
stuff
in
android
that
I'm
not
showing
in
this
example
for
simplicity,
you
can
customize
the
format
string,
you
can
have
it
log
in
json
format,
also,
and
you
can
check
out
the
android
docs
for
that
to
get
all
the
details
for
now.
I'll
just
run
this
configuration
and
what
I'm
going
to
add
to
this
configuration
with
when
we're
doing
file
access
logging
envoy
has
a
flash
interval
for
our
demo
purposes.
I'm
gonna
turn
it
into
one
millisecond.
A
A
A
Let
me
expand
that
eh
you're
not
gonna,
go
to
the
admin
page,
obviously,
and
just
hit
the
the
stats
all
the
time
right.
It's
not
feasible.
You're,
not
gonna,
go!
Oh!
Let
me
just
have
an
android
started
here:
you're,
not
gonna,
go
to
this
page
and
just
hit
refresh
refresh
refresh
right
it.
It
doesn't
make
sense.
You
gotta,
use
some
sort
of
system
to
collect
all
these
stats
and
allow
you
to
observe
them
in
a
in
an
aggregated
way.
That
makes
sense,
and
such
a
system
exists
in
the
world.
A
A
very
popular
one
is
prometheus
and
we're
going
to
show
how
to
use
prometheus
and
envoy
together
and
envoy
actually
has
native
support
for
prometheus
instead
of
stats.
If
I'll
go
to
stats
prometheus,
we'll
get
the
stats
in
a
prometheus
format
and
you
notice
they're
a
little
bit
different.
The
things
that
envoy
separates
with
dots
become
labels
in
prometheus
right.
So,
if
you
remember,
we
had
cluster
name,
a
cluster
dot
cluster
name.
A
Let's
open
this
one
here,
so
you
can
see
that
sorry
here
we
go,
you
can
see
membership
total
the
a
actually.
This
is
a
bad
example.
This
is
not
the
counter,
but
usually
here
we
go
connection
total.
So
this
is
the
encounter
for
the
total
amount
of
connections
that
we
had
today,
and
this
number
can
only
increase
right.
You
can
never
not
have
a
connection
that
you
already
had
right.
So
this
is
a
counter.
A
A
Scrambling
okay:
here
we
go
iq
active,
that's.
Why
so
currently,
there's
no
active
request
and
that's
because
curl
makes
requests
every
second
and
they
are
responded
very
quickly.
So
unless
I'm
very
lucky
I'll
see
here
zero
and
you
can
see
the
same
for
prometheus
and
we
cluster
upstream,
rq
active
for
some
cluster
is
zero.
So
this
stat
and
this
prometheus
formatted
stud
are
the
same
stat
they
just
formatted
differently.
So
prometheus
can
pick
that
up
all
right.
Close
this
window,
oh
and
histogram,
obviously
are
buckets.
A
A
A
Stats,
let's
run
that,
in
the
background
clear,
the
screen,
all
right
and
prometheus
is
a
go
program.
You
can
run
it
just
on
your
terminal.
I
have
here
a
prometheus.
I
that
I
downloaded.
I
had
a
config
file,
prometheus
yaml.
I
tell
it
to
listen
on
port
1990
for
the
web
ui
and
I
just
give
it
a
temporary
directory
for
its
data,
because
I
don't
really
care
about
the
data
at
this
point
and
before
we
run
it,
let's
just
review
the
prometheus
ymo
and
you
can
see
it's
pretty
basic
prometheus.
A
The
way
prometheus
works
is
that
it
will
go
and
scrape
this
page
and
aggregate
the
stats
that
it
sees
so
every
second
prometheus.
It
will
go
in
to
its
it's
go
to
this
envoy
and
scrape
this
metrics
and
save
them,
and
I
tell
it
what
to
scrape
right.
So
I
tell
it.
I
have
a
job
name:
android,
the
stats
are
in
stats
prometheus
and
the
target
is
the
invoice
stat
page
right
and
now,
let's
run
prometheus.
A
A
A
A
A
Maybe
we'll
give
it
every
10
seconds
here
we
go
okay,
sure
all
right.
So
what
what
this
query
does
and
again
it's
going
to
be
very,
very
brief:
it's
not
going
to
be
an
intro
to
prometheus,
just
to
illustrate
how
it
works.
I'm
telling
prometheus
to
take
the
stat
for
downstream
request
that
has
a
response.
Code
of
2xx
right
response
with
class
is
2
so
for
200
and
show
me
the
rate
of
this
request.
A
A
So
this
is,
you
know
how
to
use
prometheus
and
and
I'm
going
together.
This
is
a
very,
very
naive
way
right.
Usually
your
a
configuration
source
will
not
be
static.
It'll
be
some
something
dynamic.
Like
kubernetes
or
vmware,
or
openstack
or
whatnot,
so
you
have
to
configure
prometheus
accordingly
and
obviously
your
data
directory
cannot
be
a
temporary
folder.
So
obviously
this
demo
is
nowhere
in
your
production
ready.
A
A
A
A
So
when
you
do
this
http
request,
you
also
pass
along
a
correlation
id
a
tracing
system
and
we'll
look
at
jagger
in
a
second
can
correlate
all
this
information
together
into
a
single
trace,
a
single
distributed
trace.
So
the
way
this
work
is
that
each
micro
service
needs
to
be
aware
of
tracing
right.
It
needs
to
send
its
tracing
information
to
be
collected
and
it
needs
to
pass
along
this
correlation
id
and
envoy
natively
supports
that.
A
A
So
let's
get
it
started
in
the
background
here,
all
right
and
now,
let's
look
at
how
we
make
envoy
work
with
jagger
and
anway
has
two
possible
ways
to
work
with
jagger
one
jager
is
zipkin
compatible,
so
I
can
configure
it
to
work
a
very
similar
to
how
I
would
configure
android
to
work
with
zipkin.
The
other
is
jagger
native
using
open
tracing
which
will
show
in
this
configuration.
A
This
is
the
blob
that
defines
tracing
and
we
say:
okay,
we
want
to
use
dynamic,
open
tracing
because
jager
supports
open
tracing.
We
will
because
android
doesn't
have
open
tracing
code
inside
of
it.
We
have
to
load
a
shared
library
with
the
open
tracing
code
and
there's
the
information
about
this
in
the
android
docs
we
configure.
All
this
configuration
below
from
here
to
here
belongs
to
open
tracing,
so
android
is
not
really
aware
of
what
it's
doing.
A
It
just
passes
it
along
to
open
tracing,
so
the
service
name,
how
the
the
name
of
the
traces
coming
out
from
this
angle
would
appear
in
the
trace
and
we'll
see
it
in
the
ui
every
second
in
a
second,
how
frequently
to
sample-
and
this
essentially
a
strategy,
usually
with
tracing.
If
you'll
send
every
request
you
ever
got
to
the
trace
collector,
that's
going
to
be
a
lot
of
data.
A
A
So
this
is
the
sampling
strategy,
and
this
is
the
address
of
the
jager
or
open
tracing
collector,
essentially
the
the
one
we
started
with
in
the
container
in
the
other
tab
and
a
bunch
of
other
stuff
that
are
not
that
important
for
the
demo,
but
you
can
review
the
docs
for
them,
basically
how
some
prefixes
to
define
to
the
stuff
that
we
send
all
right.
In
addition,
we
define
on
the
route
a
decorator
with
an
operation,
and
this
operation
name
will
appear
in
the
trace,
so
that
kind
of
can
help.
A
If
you
have
a
lot
of
routes-
and
you
want
to
understand
better,
what's
going
on
now
to
make
this
trace
more
interesting,
I've
also
added
a
raid
limit
service
to
show
that
envoy
can
actually
give
you
the
exact
subtraction
when
it
sends
the
request
to
the
red
limit
service
and
when
it
comes
back
and
the
red
limit
service.
Also,
I've
added
support
there
for
jager
and
we'll
see
that
in
a
second.
A
So
this
is
all
the
android
configuration
needed
to
get
tracing
going,
and
this
is
the
rear
limit.
Configuration
doesn't
really
matter.
It
just
makes
envoy
use
this
extra
rate
limit.
So
right
now
the
request
will
go
from
envoy,
sorry
to
envoy,
then
from
android
to
do
a
rate
limit
check
and
once
the
rate
limit
check
returns,
it
will
go
and
forward
it
to
the
upstream
service,
unless,
of
course,
the
rate
limit
check
fails
and
then
it
will
return.
You
know
429
rate
limited.
This
is
how
the
rate
limit
service
works
in
anway
right.
A
So
the
request
comes
from
the
internet
to
envoy
it
gets
to
the
raid
limits
filter
the
red
limit
filter
stops.
It
sends
it
to
the
rate
limit
service
which
returns,
you
know,
allow
rate
limit
and
if
it's
allowed,
it
continues
upstream
to
the
cluster,
and
this
complexity
is
kind
of
illustrates
you
why
you
need
distributed
tracing
because
it
helps
you
see
this
exact
flow
in
one
place
in
one
system,
all
right.
So,
let's
get
this
started.
The
red
limit
service
that
I
have
set
up
it
sleeps
for
one
second,
just
to
keep
things
interesting.
A
A
All
right
so
now,
let's
review
our
android
request
and
you
can
see
it
arrives
like
before.
Curl
then
rate
limit,
filter,
engages
and
calls
the
rate
limit
service
and
asks
him
if
we
should
rate
limit
now,
in
addition
to
the
regular
headers,
you
can
see
that
we
also
have
this
uber
trace
id
and
that
thing
will
be
used
by
jager
to
correlate
all
the
different
requests
that
have
the
same
id
into
a
single
trace
and
from
here
the
flow
is
very
similar
to
before
the
request
is
once
it
is
approved.
A
It
sends
it
gets
sent
upstream
I'll,
probably
have
to
just
scroll
up
to
see
it
upstream
here
yeah
here
we
go,
and
this
is
the
request
sent
up
stream
and
you
can
see
it
also
have
the
same
trace
id
all
right
by
now,
zip
jagger
should
have
loaded
the
traces.
So,
let's
refresh
here
we
go,
we
can
see,
we
have
a
bunch
of
services,
let's
look
at
the
edge
proxy
find
traces
and
we
can
see
that
we
have
a
bunch
of
traces.
Let's
look
at
one
of
them.
A
Each
trace
is
now
because
I
always
sample
my
sampling
is
sent
to
100
for
the
demo.
Each
trace
is
essentially
a
request
and
you
can
see
the
trace
content,
and
this
is
essentially
a
timeline
view,
and
you
can
see
it's
request
start
at
zero
and
ended
after
one
second,
because
the
red
limit
service,
as
I
mentioned
sleeps
for
one
second
just
to
make
this
a
bit
more
interesting,
and
you
can
see
that
the
error,
a
request
arrived
to
envoy
and
to
the
fetch
content
route.
Remember
we
decorated
the
route
with
an
operation
name.
A
Then
envoy
made
a
request
to
the
rate
limit
to
the
rate
limit
service,
and
this
final
one
is
the
request
received
by
the
rate
limit
right.
So
this
bottom
section
here
this
bottom
span
came
from
a
different
process
than
these
two
spans
that
came
from
envoy,
and
you
can
see
that
they
contain
each
other,
because
this
started
after
this-
and
this
started
after
that-
and
the
difference
here
is
kind
of
the
latency
that
it
took
the
request
to
arrive
from
here
from
envoy
to
the
rail
limit.
A
So
there's
a
lot
of
insights
you
can
get
and
you
can
associate
more
values
and
logs
and
error
errors
and
get
a
full
a
view
of
a
lifetime
over
request.
So
this
is
a
very
basic
demo
with
just
two
services,
but
you
know
we
imagine
it
ten
micro
services,
where
request
goes
somewhere
waits
for
something
does
something,
and
now
the
whole
flow
is
very
hard
to
understand
and
tracing
gives
you
kind
of
like
the
way
it
would
have
looked
if
you
were
looking
at
a
debugger
in
a
single
monolith
application.
A
So
it
gives
you
that
holistic
view
of.
What's
going
on
with
your
traffic
another
cute
feature
of
jager,
you
can
see
system
architecture.
In
our
case,
you
can
see
that
edge
proxy
calls,
the
rate
limit
service
and
but
yeah.
The
the
main
thing
I
want
to
show
is
how
envoy
natively
supports
tracing
and
can
export
the
spans
for
you,
so
you
can
decorate
routes
if
envoy
does
a
call
out
to
authentication
or
rate
limiting.
A
It
will
add
a
spend
for
that
and
assuming
that
the
upstream
service,
like
great
limit
in
our
case
or
the
the
echo
server,
are
configured
and
support
tracing,
they
will
also
be
part
of
the
trace.
Now
this
is
one
disadvantage
of
of
tracing.
Is
that
every
program
needs
to
be
aware
that
it's
being
traced
at
the
minimum
it
needs
to
propagate
this
correlation.
I
think
right.
A
A
We
talked
about
the
admin
interface
and
the
stats
you
can
get
from
there,
the
config
dom
clusters
debug
logs
and
how
you
know
they
have
a
wealth
on
information,
but
they
also
hurt
performance,
and
so
you
shouldn't
just
turn
them
on.
It's
really
only
used
for
debugging
access
logs.
That
are
the
logging
mechanism
that
was
designed
for
production,
and
you
can
log
either
to
a
remote
service
using
jrpc
or
a
file
with
a
json
format
or
a
custom
string
format.
A
We
saw
metrics
collected
in
prometheus
and
we
saw
traces
collected
in
jager
and
yeah.
That's
it.
If
there
are
any
questions,
feel
feel
free
to
ask.
I
know
I
didn't
delve
in
deep
to
the
to
the
topics.
That's
intentional
because
I
didn't
want
to
cover
a
lot
of
ground
and
when
you
cover
a
lot
of
ground,
you
can't
really
go
deep.