►
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Solving Microservice Murder Mysteries with Envoy's Tap Filter - Mitch Kelley, Solo.io
There are 324 possible explanations for the great WhoDunIt question in the board game Clue.
If only debugging microservice failures were so simple. Learn how Envoy's Tap filter can be used to get to the bottom of your most intractable failure modes before they strike again. Through an interactive live demo we will show how to configure and apply Envoy Tap filters to capture, replay, and debug the types of intermittent failures that plague microservices. This debug workflow transforms bug reproduction activities from an expensive, team-wide investigation to a simple, instant database query. Whether you are using Envoy directly or through Istio, learn how you can take advantage of this powerful new feature to increase the resiliency of your distributed systems and the effectiveness of your teams.
Please bring your computer or smartphone to participate in this live investigation.
A
Hi,
my
name's
Mitch
Kelly
I'm,
a
software
engineer
at
solo,
I/o
started
in
Boston
we're
building
glue,
an
API
gateway
based
off
of
envoy,
it's
an
honor
to
be
here
and
celebrate
envoy,
and
one
of
my
favorite
new
features
the
tap
filter.
As
you
know,
this
has
been
a
big
year
for
envoy.
The
introduction
of
the
Envoy
mobile
as
Jose
and
Michael
presented
final
deprecation
of
the
v1
API
and
introduction
of
versioning
protocol
is
Harvey
reviewed
and
the
introduction
of
web
assembly
to
name
a
few
continuing
with
the
observability
theme.
A
My
personal
favorite
is
the
new
feature.
The
new
feature
of
the
tap
filter
I'll
show
you
how
to
use
it
and
encourage
you
to
do
so.
If
you're
not
already,
what
is
a
tap
filter.
Tap
filter
allows
you
to
capture
a
request
and
its
response.
When
certain
criteria
are
met,
the
capture
data
can
be
streamed
to
another
service
or
stored
in
a
file
system.
Unlike
other
tools
that
are
safe
to
make
widely
available
to
your
dev
teams.
The
tap
filter
captures
a
full
copy
of
all
your
data
included
in
the
request
and
response.
A
The
match
criteria
is
very
expressive.
So
you
can
pinpoint
the
traffic
that
you're
interested
in
investigating
you
can
limit
the
capture
size
to
avoid
wasteful
captures.
The
default
body
size
is
1gb
by
it
and
there's
also
a
depth
filter
for
the
TCP
traffic,
but
I'll
just
talk
about
HTTP.
Today,
here's
a
minimal
config
to
get
you
started.
A
This
tells
envoy
to
capture
all
the
HTTP
traffic
and
write
it
to
the
taps
directory
in
files
prefixed
with
any
string
naming
lots
of
things
as
much
of
the
fun
of
writing
software
I
think
the
top
filter
has
a
great
name
and
would
like
to
dive
into
its
etymology
to
discuss
what
it
can
do
for
your
development
team.
Imagine
that
your
service
is
embedded
in
a
complicated
architecture,
the
feeds
and
manages
a
large
reservoir
of
data
and
data
flows,
something
like
this
brewery
schematic.
A
A
Specifically,
you
can
configure
envoy
to
capture
taps
only
when
a
particular
request
header
is
set.
This
means
you
can
control
exactly
when
a
record
is
made
by
injecting
a
particular
header.
Coming
soon
my
teammate
Yuval
is
working
on
an
8
in
able
of
fractional
sampling.
In
addition
to
and
a
handset
a
mat
configuration
for
the
tap
filter
with
fractional
sampling,
you
can
easily
record
a
assistant
behavior
over
spent
up
time.
A
Imagine
is
said
that
your
system
is
a
highly
modular
reconfigurable
and
scalable
micro
service.
Many
systems
configurable
to
work
together
and
deliver
a
tuned
result
as
with
an
optics
bench.
You'd
want
to
be
able
to
assemble
and
disassemble
your
system,
swapping
out
and
reusing
components
to
deliver
solutions
faster
with
greater
accuracy.
A
Tapping
your
system
rather
than
riveting
or
welding
components
gives
you
traction
without
any
significant
overhead.
Like
a
mechanical
tapping
tool,
the
tap
filter
supports
a
highly
configurable
workflows,
as
shown
here,
you
can
include
multiple
tap
specifications
in
a
single
filter
chain.
This
may
be
useful
if
you
want
to
capture
a
standard
snapshots
across
your
organization,
in
addition
to
team
or
feature
specific
snapshots.
A
Alternatively,
you
can
deploy
the
tap
filter
on
a
sidecar
envoy,
this
mate.
This
allows
you
to
keep
your
route
config
separate
from
your
tap
specs
and
make
it
easier
to
reuse
common
configs.
My
teammate
Eitan
recently
implemented
a
web
application
firewall
filter
for
envoy
and
found
that
Sat
filter
useful
for
doing
development
on
avoid
itself.
We
anticipate
this
being
a
valuable
tool
during
the
creation
of
web
assembly
filters
as
well,
but
where
I
think
a
table
filter
best
takes
its
name
is
from
its
use
as
a
disaster
mitigation
tool.
A
Imagine
that
your
team
is
working
on
a
system.
That's
a
key
part
of
the
life
support
device
such
as
this
astronauts
life
support
system.
If
anything
goes
wrong,
you'll
be
haunted
by
the
failure.
Wouldn't
you
rather
that
spirit
take
the
form
of
a
helpful
of
poltergeist
and
tap
you
on
the
back
with
a
clue
towards
disaster
recovery.
This
behavior
can
be
achieved
easily
with
the
config
shown
here
when
a
response
could,
as
a
500
internal
server
error
status,
the
input
is
recorded.
A
Additionally,
it's
worth
noting
that
envoys
match
criteria
a
recursive,
meaning
they
support
highly
configurable,
highly
specific
filtering.
If,
for
example,
you
released
a
version
1.0
of
a
product
and
want
to
watch
our
is
use
produced
by
its
latest
release,
you
can
easily
do
so,
but
while
we're
talking
about
extending
your
software
features
and
how
to
adopt
them
into
your
workflow,
it's
worth
reflecting
on
some
of
the
associated
lore.
One
of
my
favorite
stories
of
software
engineering
mythology
is
the
real
reason.
A
But
then,
on
the
other
side
of
the
software
engineering,
lore,
there's
a
story
of
Steve
Jobs
that
I
can
never
forget.
He
asked
his
team
how
hard
they
would
work
to
shave
a
second
off
the
max
boot
time.
If
it
meant
saving
someone's
life
well
pretty
hard,
then
he
noted
that,
given
the
millions
of
users,
the
cumulative
time
spent
waiting
for
startup
amounted
to
several
human
lifespans.
A
In
this
sense,
even
if
we
are
not
developing
breathing
apparatus
for
astronauts,
the
systems
we
developed
concerned
the
realities
of
life
and
death,
the
faster
we
can
find
in
fixed
bugs
in
our
systems
the
fuller
everyone
can
experience
life
and
growth.
There's
a
lot
of
work
going
into
making
systems
more
robust
and
accelerate
in
the
future
delivery.
It's
what
everyone
came
to
coops
on,
to
find
and
still
whenever
something
goes
wrong
in
a
micro
service
environment.
You
need
to
become
a
detective,
and
this
is
where
the
top
filter
can
help.
A
A
A
Nothing
goes
wrong,
but
every
once
in
a
while
there's
an
intermittent
failure
and
if,
if
we're
not
looking
for
it,
we're
gonna
get
in
trouble,
I
think
Dylan
said
he's
on
call,
maybe
not,
but
fortunately,
I
had
the
tap
filter
set
up
in
the
background,
captured
a
file
to
my
system
and
just
for
expediency.
I've
got
it
on
the
slide
here
and
show
you
what
it
looks
like.
A
A
The
mime
in
my
favorite
part,
the
body,
is
captured
here,
it's
encoded
as
bytes,
but
you
can
also
encode
it
as
strings.
The
bytes
are
a
little
bit
more
robust,
then
just
decode
that
and
you
can
see
exactly
what
input
created
the
failure.
I
guess
they
in
room
B
with
object
C.
So
now
we
know
how
to
address
the
problem,
but
for
completeness,
let's
also
look
at
the
other
side
of
the
tap
filter.
It
record.
A
It
also
tells
us
the
response,
which
may
be
not
as
useful
for
reproducing
but
another
feature
of
the
tap
filter
worth
noting,
but
anyway,
thank
you
for
your
time
and
on
behalf
of
Solio.
Thank
you
for
the
excellent
work
of
the
onboard
community.
If
you'd
like
to
learn
more
about
the
tap
filter
or
solos
envoy
related
projects,
please
visit
us
at
koukin
or
on
slack.
A
We
have
a
lot
of
exciting
work
we'd
like
to
share,
including
in
loop,
a
tool
built
around
the
tap
filter
for
traffic
capture
and
replay
in
production
clusters
or
surface
mesh
environments,
glue
our
popular
Envoy
powered,
API
gateway,
which
just
released
version
1
and
service
mesh
hub
a
tool
for
coordinating
multiple
meshes
and
clusters.
If
you're
interested
in
joining
this
work,
let
me
know
we're
hiring
in
many
positions
and
the
source
code
for
the
game,
and
examples
from
this
presentation
are
available
on
github.