►
Description
#IstioCon2021
Presented at IstioCon 2021 by Venky Ganti & Rahul Lahiri.
Microservices applications rely on complex interactions among services. Engineering teams must create API tests with API mocks to shift testing left. Current approaches to mock creation are manual, which is expensive and inefficient.
We illustrate how Istio can be leveraged to significantly reduce engineering effort necessary for API testing.
API tests can be built using the following Istio capabilities:
Dynamic deployment of Envoy filters to capture relevant examples of API requests and responses.
Observability to trace request execution flows across all microservices.
Virtual services can switch traffic between live services and mocks with no code changes.
Developer benefits:
Ad hoc service testing locally leveraging API data to mock producer services
Create service tests with auto-created API mocks, eliminating costly manual API mock creation
Get visibility into failed API requests from end-to-end tests with no additional effort
A
So
we
all
know
that
the
number
of
applications
being
driven
by
apis
and
microservices
in
general
are
exploding
significantly
right,
and
this
migration
towards
microservices
and
distributed
services
in
general
has
started
placing
an
additional
burden
on
engineering
teams.
In
order
to
create
a
variety
of
api
tests,
it
could
be
things
like
end-to-end
api
test
of
the
gateway
layer.
It
could
be
component
test
for
specific
components
in
your
application
and
service
test
for
each
individual
service.
So
if
a
developer
is
changing
one
service-
let's
say
the
catalog
service
in
here.
A
He
or
she
needs
to
test
the
change
thoroughly
without
having
to
bring
up
all
the
producer
services
that
they
need
right.
So
all
of
these
different
kinds
of
tests,
with
a
service
test
and
component
as
an
end-to-end
test
is
placing
a
significant
amount
of
burden
on
engineering
teams
and,
as
the
number
of
apis
and
api
interactions
grow.
This
burden
is
only
increasing
right.
Let's
just
get
some
terminology
right,
because
different
people
have
different
understanding
of
what
a
service
test
is
and
what
a
component
test
is,
especially
in
the
context
of
these
distributed,
microservices
architectures.
A
So
what
we
mean
by
a
service
test
is
the
common
understanding,
also
we're
not
changing
it,
but
let's
just
be
clear
here
I
have
a
service.
I
want
to
issue
a
service
test
request.
An
api
request
could
be
grpc
could
be
a
rest
api
against
my
service,
and
my
service
depends
on
this
producer
services,
but
I
want
to
run
this
test
either
locally
on
my
laptop
or
in
sei
pipeline.
Without
having
to
have
these
producer
services
live,
so
I
need
to
mock
that
behavior.
A
So
that's
what
we
mean
by
a
service
where
only
the
service
that
is
under
test
needs
to
be
live
and
everything
else
is
simulated
and
mocked
right,
so
you
can
test
the
service
in
isolation
now.
Similarly,
a
component
test
is
no
longer
one
service,
but
a
group
of
related
services
that
all
of
them
need
to
be
live
while
the
rest
of
the
other
services
that
this
component
depends
on
will
be
mocked
right
and
typically,
services
are
developed
first
and
then
components
evolve.
A
The
entire
application
evolves,
but
once
the
application
is
in
in
life
different
components,
different
services
can
change
independently
and
move
forward
evolve
as
well,
and
what
we
are
observing
is
the
current
approach
is
to
create
the
services.
The
component
is
an
end-to-end
test,
do
not
scale,
as
I
already
highlighted
earlier
in
the
in
the
first
slide,
with
the
number
of
apis
and
api
interactions
right.
A
A
This
test
and
they're
all
created
independently
as
well
a
developer
may
create
the
services.
Someone
else
may
create
a
component
test.
Maybe
a
qa
team
will
create
the
into
an
api
test
and
these
are
all
independently
created.
They
don't
talk
to
each
other
closely,
that
they
can
reuse
the
data
and
so
on
so
and
that
places
an
additional
burden
when
it
comes
to
maintenance.
Because
if
an
api
changes,
then
I
have
to
recreate
update
my
services
independently
update
my
component
address
independently
update
the
internet
api
test
independently
as
well.
A
So
this
is
all
an
additional
headache
and
burden
to
the
team.
So
what
some
of
the
teams
end
up
is:
okay,
let's
just
focus
on
unit
tests
and
end-to-end
tests
and
and
ignore
the
component
and
services.
So
that's
really
bad
from
a
development
and
release
quality
perspective
because
you
develop
a
particular
service.
A
You
update
an
api
either
request
or
response,
and
you
wait
for
all
the
other
services
in
your
component
evolve
as
well
and
application
to
evolve
as
well
and
finally
test
the
integration
and
the
functionality
as
well
and
that's
when
you
identify
problems
and
then
you
have
to
go
back
so
that
violates
the
shift
left
paradigm
and
you
discover
your
bugs
late
and
all
of
these
bad
issues
happen.
So
that's
not
at
all
good
from
a
from
a
development
team
perspective.
A
In
that
context,
we
want
to
place
more
emphasis
on
unit
tests
and
services
and
component
tests
and
finally
end-to-end
tests,
which
are
much
more
expensive
to
create,
maintain
and
run
as
well
right,
and
we
want
to
do
all
of
this
with
as
little
engineering
effort
as
we
can
right,
and
that
is
the
ideal
goal
we
need
to
be
at
now
and
that's
what
the
rest
of
the
talk
is
going
to
focus
on.
How
can
we
achieve
that?
A
That
state
be
able
to
shift
your
testing
earlier,
achieve
a
balance,
this
pyramid
and
be
able
to
create
and
maintain
these
different
types
of
tests,
with
as
little
effort
as
as
needed
right
so
istio?
What
we
observed,
as
we
were
exploring
this
problem
is
istio
enables
this
pretty
nicely
right,
a
bunch
of
listener,
capabilities
that
we
can
leverage
to
actually
achieve
that.
So,
let's
consider
an
application
which
has
all
of
these
services
and
we
can
deploy
listeners
which
is
sort
of
envoy
filters.
A
We'll
talk
about
that
in
detail
later
at
each
of
these
service,
endpoints
right
and
as
the
traffic
comes
from
a
manual
qa
in
your
staging
environment
or
could
be
an
api
studio
where
you
can
call
requests
or
it
could
be
a
third-party
application.
It's
actually
calling
these
apis,
but
actually
listening
to
this
traffic
api
requesting
responses-
and
we
gather
this-
it's
captures
requests
and
responses,
and
we
add
a
bunch
of
sprinkle,
a
bunch
of
other
things
to
actually
convert
this
test.
A
A
You
can
leverage
this
data
and
the
test
ways
that
we
can
create
from
this
learn
from
this
data
and
run
any
of
this
service
test
component
test
application
test
entire
end-to-end
api
test
right
with
the
same
data
in
the
same
press
suite
and
the
mocks
are
auto
created,
we'll
show
you
how
that
mocks
can
be
auto
created
from
all
the
data
that
we
gather
right.
So
that,
in
a
nutshell,
is
our
approach,
and
this
enables
rapid
api
test
creation
and
not
just
end-to-end
api
tests.
A
A
A
That
is,
we
are
able
to
capture
any
of
the
api
interactions
going
through
that
each
of
the
services
right
and
then
we
drive
a
set
of
traffic
from
ui
from
a
third
party
application
from
your
api
studio
or
any
any
studio
that
allows
api,
curl
and
curling
apis.
You
capture
the
what
we
call
the
traces.
It's
not
just
one
request
to
the
endpoint,
but
it's
the
entire
trace
through
the
application
right,
that's
what
we
capture
and
from
there
we
can
create
tests
and
marks
for
all
of
these
api
endpoints.
A
Then
you
can
configure
your
system
so
from
the
same
test
data,
as
I
illustrated
earlier,
you
can
configure
various
components.
So
here
is
one
component.
There
is
a
different
component.
Here
is
the
entire
application.
Here
is
an
individual
service.
All
these
various
configurations
can
be
driven
from
from
the
same
set
of
tests
and
mock
and
we'll
show
you
how
to
mod
these
various
services
there.
So,
let's
get
into
the
first
step,
how
do
we
capture
the
api
interaction?
A
Let's
consider
these
three
services,
so
we
say
calling
service
d,
so
this
b
calling
service
c-
and
we
have
this
filters
lower
filters.
All
we
need
to
do
is
queue
partial,
apply,
minus
f
that
filter
file,
and
now
these
filters
are
deployed
on
the
on
y
proxies
and
your
services
are
listening
to
the
traffic
right
so
as
and
when
there's
an
api
request
going
through
the
service
a
and
service
a
calling
service
b.
A
Now
we
are
capturing
at
the
end
point
of
the
service
here,
the
endpoint
of
service
b
and
with
the
entry
point
of
service
c,
and
all
of
this
data
is
captured
independently
into
the
mesh
dynamic
services
right
once
you're
able
to
do
that.
We
now
can
leverage
the
tracing
right.
So
there
is
a
trace
id
that
is
introduced
at
this
endpoint.
The
same
psi
is
propagated
through
your
services,
you
could
be
using
data
dog
could
be
using
jaeger
or
zipkin
pick
your
favorite
tracing
provider
and
the
space
ids
are
propagated
through
the
services.
A
So
we
know
the
request,
a
is
being
is
being.
This
original
request
is
generating
a
request
from
a
to
b
with
the
space
r
and
span
s1,
and
the
service
b
is
calling
service
c
with
the
trace
r
and
the
parent
span
s1
and
the
new
span
s2
and
so
on.
So
we
can
reconstruct
the
space
request,
calls
request
a
b
request:
a
b
generates
a
request
from
b
to
c
and
so
on,
along
with
the
responses
as
well.
So
this
whole
trace
is
what
is
captured
and
reconstructed
by
mister
nmx
automatically.
A
So
here
again
note
note
that
we
are
leveraging
these
two
capabilities
to
deploy
these
lower
filters
on
demand,
with
with
no
with
very
little
effort
and
they're
able
to
constru
capture
this
data
right.
Whenever
you're
done
with
the
capture,
you
can
just
apply
or
remove
the
filters
and
then
stop
the
capture
whenever
you
need
to.
A
So
what
do
we
have?
At
this
point,
we
have
the
full
trace
of
every
request
that
went
through
this
service
application,
the
entire
application
and
it's
a
full
trace.
So
a
request
here
generates
a
request
to
this:
a
request
to
that
and
so
on.
All
of
these
are
tied
to
by
the
same
trace,
ids
and-
and
we
have
the
spams
as
well
and
from
this
data
now
we
are
able
to
capture
very
precise
marks
almost
at
the
request
level.
So
you
generate
the
same
request
here
and
this
request
is
calling
a
particular
service.
A
Now
we
can
add
the
trace.
We
can
use
both
the
request,
payload
and
the
trace
id
to
actually
precisely
mark
that
particular
request
that
so
this
service
is
live
right,
so
you
don't
have
to
have
the
services
in
live,
but
it's
actually
mocked,
and
these
mocks
are
also
created
by
from
the
capture
data.
So
that's
a
value
that
that
we
can
derive
from
this
ability,
combining
both
the
observability
slash
tracing
along
with
a
an
intelligent
mock
service.
A
So
once
we
have
that
now
we
are
able
to
create
service
tests.
Oh
yeah,
we
have
all
the
data.
Let
me
create
a
service,
just
repeat
for
the
catalog
service,
so
I
have
all
the
requests
that
went
into
the
catalog
service.
I
know
what
the
egress
requests
are
going
through
the
search
and
recommender,
but
that's
for
that
in
each
of
these
individual
requests
and
they
can
automatically
create
the
marks
right.
So
so
we
have
a
service
test
for
catalog
pick
any
service.
You
have
a
service
test
for
those.
A
Now,
let's
say
we
want
this
component
test.
This
component
talks
to
a
few
other
services
and
we
need
to
mock
those
services,
so
we
have
all
of
those.
It
doesn't
have
to
be
so
the
home
page,
catalog,
personal
history
or
live,
but
some
other
services
that
these
services
may
talk
to
are
being
mocked
and
that
those
marks
are
also
created
again
given
by
the
trace
right.
So
exactly
the
mesh
dynamics
are
also
creating
this
mock
services
and
both
the
trace
id
and
the
request.
A
Payload
are
used
by
them
by
the
mock
service,
and
that
enables
us
to
precisely
mock
the
state
of
any
service
to
that
individual
trace
level.
So
what
that
means
is,
I
can
take
a
trace,
perhaps
even
duplicate
it,
create
a
new
trace
and
change.
My
response
that
basically
means
when
a
service
is
being
mocked.
An
api
request
comes
in.
A
I
can
differentiate
between
two
different
responses
that
need
to
be
given
so
and
that
level
of
preciseness
and
robustness
of
the
marks
are
what
help
you
create
tests
at
scale
without
having
to
worry
about
stepping
on
each
other's
toes
around
around
mocking
so
and
as
far
as
mocking
is
concerned.
Again,
istio
has
a
nice
construct
of
this
virtual
service,
which
we
can
use
to
route
instead
of
to
a
live
service.
You
say
I
want
to
mock
this
particular
service.
You
just
change
your
configuration
and
say:
oh
this
prefix,
the
api.
A
Slash
reviews
needs
to
be
rewritten
to
this
mock
service
and
boom.
You
have
your
mocking
ready
to
go
so
your
service
and
the
test
is,
is
being
is
issuing
a
request
typically
to
your
live
service,
but
with
this
mock
virtual
service
configuration
it's
not
going
to
our
mesh
dynamic,
smart
services,
so
it's
as
simple
as
that.
So
again
we
are
leveraging
the
hto
to
create
this
on-demand
mocking
configuration,
whether
you
want
to
mock
one
service
or
all
services.
A
It's
up
to
you
and
you
can
configure
any
number
of
configurations
here
with
the
same
test.
Data
right,
so
that's
about
creating
mocks
and
configuring,
the
istio
application
with
various
configurations
you
choose
pick
and
choose
with
service
which
which
service
is
live
with
services
mocked,
and
you
can
have
multiple
configurations
right.
So,
let's
jump
into
the
test
creation
itself
right.
There
are
a
couple
of
other
things:
it's
not
just
taking
the
api
traffic
and
just
replaying.
A
A
You
need
to
enhance
them
with
these,
what
we
call
assertion,
rules
and
context
rules
which
I
will
talk
about
briefly
in
the
next
few
minutes,
and
that's
what
enables
you
to
create
test
suites
right
now,
you
can
create
take
those
test
suites,
you
can
run
them
based
on
your
test,
configuration
whether
you
have
one
service
or
a
component
or
an
entire
application,
live
and
and
generate
your
different
types
of
test
service
component
and
end-to-end
tests.
So
let
me
talk
about
the
context.
Rules
first.
A
Let's
take
an
example:
sequence,
where
there's
a
create
product
api,
followed
by
a
get
details
of
the
product
and
the
create
product,
always
generates
a
dynamic
product
type,
and
in
this
particular
case,
when
we
recorded
the
traffic,
it's
hdsn189,
something
so
on
right
and
the
same
product.
Id
value
is
being
used
in
this
upc
parameter
here
to
get
the
details
of
the
products
and
perhaps
the
stores
that
contain
the
products.
A
So
here
is
the
sequence:
now
when
we
play
the
test,
because
this
is
dynamically
generated,
the
product
id
could
be
different.
It
could
be
hdsn
one
two,
three
four
blah
right
now.
We
need
to
take
that
spsn,
one
two,
three
four
blah
and
insert
into
the
upc
query
parameter
so
that
the
test
actually
is
meaningful.
A
If
you
take
the
old
value,
this
one
may
or
may
not
produce
a
valid,
meaningful
response.
You
need
to
take
the
new
product
id
and
inject
here.
So
this
kind
of
contextual
dependencies
need
to
be
recognized
automatically
because
you
need
to
generate
this
test
as
automatically
as
you
can,
and
the
contextual
dependencies
are
what
we
enable
and
say.
First
look
at
the
data
that
we
captured.
A
We
learn
from
from
the
data,
so
there
is
a
contextual
dependency
here,
and
so
we
have
a
way
of
removing
false
and
avoiding
false
negatives,
and
we
have
these
context
rules.
Engineers
can
review
them
if
they
want,
they
can
customize
them
if
they
need
to
and
take
this
context
source
and
automatically
when
they
play
the
data.
Again.
With
this
context,
tools,
they'll
get
the
right
behavior
that
they
need
from
a
testing
perspective.
Right,
that's
number
one,
that's
the
assertion,
that's
the
context
source.
The
second
set
is
assertion
rules.
A
So
when
we
capture
the
response,
we
have
a
response
saying
order
id
and
order
value
order.
Id
is
something
and
order
value
is
58.75,
and
we
know
that
yeah,
this
order
id
is
always
dynamic.
When
you
play
the
test,
you'll
get
a
different
order
id
now
when
you're,
comparing
you
need
to
make
sure
that
you
compare
only
the
key
for
order
id
and
not
the
value
for
order
value.
A
Whichever
way
they
feel
comfortable
and
get
the
behavior
that
they
need
so
so,
once
we
have
this
assertion,
rules
and
the
context
rules.
The
data
that
we
captured
from
the
sto
envoy
listeners
of
the
lower
filters,
which
are
deployed
on
his
own
website,
called
proxies
they're
able
to
convert
this
data
into
real
test
suites
that
could
be
meaningfully
used
in
a
ci
pipeline
or
in
a
local
developer,
laptop
right
and
again
just
to
illustrate
that
or
sort
of
stress
that
again.
A
So
all
of
these
requests
and
responses
that
we
captured
from
either
the
in
this
particular
case
the
staging
environments
with
the
listeners.
This
could
also
be
envisioned
in
a
developer
laptop
as
well.
So
we
capture
that
api
traffic
into
this
api
catalog
and
from
this
catalog,
which
is
the
repository
of
api
requests
and
responses.
A
We
are
able
to
create
various
types
of
tests,
whether
it
be
component
and
end-to-end
tests
could
be
services
that
could
be
that
they're
running
in
a
ci
pipeline
or
could
be
just
local
services
when
a
developer
changes
a
service.
She
can
run
these
local
services
locally
without
having
to
worry
about
all
the
producers
service
services
being
live.
A
B
B
The
these
are
the
services
in
the
demo
application.
So
we
have
a
few
services.
It's
it's.
Obviously
it's
running
on
istio
and
all
of
this,
the
services
talk
to
each
other.
You
can
kind
of
see
the
flow
movie
info
is
the
gateway
service.
So
when
we
make
requests
from
the
ui
it
comes
to
movie
info,
it
makes
various
requests
to
these
services.
There
could
be
additional
communication
across
these
services.
B
Now
you
know
the
traditional
way
of
creating
tests,
of
course,
will
be
to
create
this
by
hand.
Here
we
have
the
ability,
because
of
istio,
as
venky
mentioned,
we
have
the
ability
to
capture
those
complete
traces,
so
you
can
just
go
to
the
ui,
go
through
expected
user
flows
and
we
can
capture
those
complete
traces
for
the
api
requests
that
are
generated
and
create
tests
out
of
this.
B
B
B
When
you
render
the
details,
pull
out
the
information
about
the
reviews
where
it's
await
available
for
rental,
etc,
we
capture
all
the
apis
behind
them,
complete
with
the
full
traces
and
then
you
can
go
ahead
and
go
through
a
process
to
rent
this
movie,
so
you
can
say
rent
and
once
again
we
capture
all
of
the
apis
behind
that
right.
So,
let's
kind
of
go
through
a
couple
of
use
cases.
Let's
say
the
first
use
case
is
just
listing
an
item.
B
So
here
I
clicked
and
listed
listed
an
item
and
let's
say
the
second
use
case.
I
want
to
create
api
test
test
from.
Is
somebody
lists,
an
item
and
then
go
through
the
rental
process
so
go
through
that?
Do
something
here:
servant?
Okay!
So
that's
enough!
That's
all
we
need
from
here.
We
are
done
with
that.
So
now
I'm
going
to
stop
the
recording.
B
So
what
we
did
here
was
we
could
have
also
created
all
of
these
api
requests
manually
from
the
tool.
Obviously,
it's
a
hundred
times
more
efficient
to
just
go
to
the
ui.
Go
through
the
user
flows
because
of
istio.
We
can
just
capture
the
entire
traces.
It
just
reduces
the
amount
of
effort.
That's
involved
very
significantly.
B
B
We
have
the
information
about
all
the
different
services
from
which
captured
api
requests.
Here
are
all
the
apis,
and
here
are
the
apis,
which
you
can
group
by
service,
so
I
can
say
movie
info:
what
are
the
list
movies?
So
here
are
all
the
instances
of
this
api
that
we
captured
and
if
you
dig
into
one
of
the
into
the
details
of
this,
you
can
just
click
them
here
and
you
see
all
of
the
details
over
here.
B
So
if
you
notice
here,
you
will
see
here's
the
request
to
that
service,
and
then
we
show
you
that
the
egress
request
test
now
we
internally
we
have
the
complete
trace.
If
there
are
multiple
levels
traversed,
we
have
the
complete
address
which
I'll
show
you,
but
here
in
part
this
particular
part
of
the
tool.
We
just
show
you
the
egress
request
from
a
single
service.
Okay,
now,
obviously
you
can
go
there.
You
can
edit
all
those
requests.
You
can
update
the
test.
Add
additional
requests
here.
B
B
We
are,
we
can
now
slice
and
dice
that
any
way
we
want
because
of
istio,
we
can
say
okay,
I
want
to,
for
example,
I
can
say
okay
here
I
want
to
test
just
the
review
service
and
our
replay
driver
can
just
pick
out
the
review
service
and
if
you
need
to
mock
something,
you
decide
what
you
what
you
want
here
I
can.
I
can
test
the
movie
info
service,
I
can
leave
all
the
services
live
or
I
can
mock
some
of
them
or
I
can
mock
all
of
them.
B
So
I
have
a
couple
of
configurations
set
up
here.
So,
let's
so
sorry
so
here
I
have
two
configurations
set
up
here
in
one
of
them.
I'm
mocking
two
of
the
services
in
the
other
test
configuration
I'm
mocking
all
the
services.
So
let's
pick
this
one
so
here,
if
I
run
a
test
for
movie
info,
I'm
mocking
all
of
the
services
and
once
again
that
got
all
automatically
created
from
the
data
that
we
did,
that
we
generated
so
let's
go
ahead
and
start
that
test.
B
B
Our
replay
driver
is
saying:
okay,
your
the
service,
you're
testing
is
movie
info,
you're
mocking
all
of
these
services,
so
we
are
replaying
the
requests
that
go
to
movie
info
to
the
movie
info
service
in
the
test
cluster
and
when
the
egress
requests
come.
We
have
configured
here,
obviously
for
this.
Once
again,
we
are
not
going
to
change
it
manually
here
for
the
sake
of
time,
but
here
we
have
configured
it
using
histo
where
all
of
these
services
are
being
mocked
and
the
requests
are
being
forwarded
to
our
mock
services.
B
B
B
I
will
just
interact
with
them,
so
this
is
the
component
that
we
are
testing,
and
so
I
will
pick
that
configuration
of
the
already
picked
that
configuration
and
now
see
once
again
it's
the
same
test,
still
I'm
just
going
to
say,
run
test
while
it's
running,
let's
go
and
start
looking
at
the
results,
we
got
from
the
other
tests
that
we
did
okay.
So
this
is
the
test,
the
previous
test
that
we
ran
at
1208
over
here,
you
can
see
run
with
the
test
file
istio.
B
So
it's
showing
you
all
of
the
various
services
that
were
part
of
that
test
movie
info
was
the
only
live
service
and
you
can
see
it's
showing
a
red
there.
Now,
if
I
expand
this
there
here
are
the
three
apis
that
we
captured
from
our
interactions:
didn't
capture
login,
so
two
of
the
apis
they
tested.
Fine.
This
api
is
showing
right
so
something
something's
wrong
here.
B
So
now,
if
I
look
at
this,
let
it
just
load
so
the
first
thing
I
want
to
show
you
is
the
trace.
So
here
you
can
see
what
we
captured
here.
This
is
a
recorded
request,
so
you
can
see
what
we
captured
was.
The
request
came
into
movie
info
movie
info
called
reviews,
review
scholar
ratings,
then
movie
info
called
restaurant
ratings,
etc.
B
In
order
now,
if
you
remember
for
this
one,
the
configuration
was
all
of
these
egress
services
were
mocked
and
it
turns
out
that
all
the
requests
were
actually
none
of
the
requests
to
the
service
has
changed.
So
obviously
the
responses
don't
change
either
so
everything's
good,
so
this
red,
whatever
it
is
happening,
is
happening
only
in
this
list
movies
api.
Only
at
this
level,
none
of
the
other
services
are
affected.
B
B
B
Okay,
so
here
the
1209,
this
is
the
second
test
we
ran
and
if
you
look
at
the
signature
here,
because
it
was
a
little
bit
different
here.
Obviously
there
are
a
lot
more
reds
here,
as
you
can
see
compared
to
the
other
other
configuration,
and
here
you
can
see
two
of
the.
If
you
remember,
we
only
mocked
two
of
the
services.
Other
two
services
were
live,
so
something's
changed
in
rent
movie.
B
So
now,
if
I
look
at
list
movies
here
and
see
what
the
signature
of
what
has
changed
so
if
I
once
again
do
the
trace,
so
the
recording
is
fine,
but
here
you
can
see
it
is
obviously
the
same
recording,
behavior
the
same
thing,
but
you
can
see
during
the
replay
this
detail
service
which
we
made
live
now
that
something
has
changed
in
the
response
of
that
and
to
look
at
if
you
want
to
dig
into
what
has
changed
here.
You
can
just
click
in
here,
so
we
are
showing.
B
Okay,
here
is
the
response,
and
you
can
see
what
has
changed
is
actually
the
the
response
from
the
service.
In
that
other
instance,
the
one
that
we
used
for
testing,
that's
actually
sending
a
much
more
abbreviated
response.
A
lot
of
the
things
that
should
could
probably
should
have
been
there
in
the
response.
Those
are
missing,
so
we
are
saying:
okay,
there's
a
problem,
and
now
so
you
can
essentially
go
down
that
trace
and
see
where
things
have
changed
and
if
you
go
back
at
the
top
level
now
here
I
suspect
yeah.
B
So
here
you
can
see
that
change
is
reflected
here.
We
use
that
piece
of
information
here,
that's
the
change
that
you
see,
but
there
are
also
other
changes
that
are
there
in
the
service
itself.
So
this
makes
the
whole
process
of
creating
the
tests
having
that
visibility
down
that
entire
trace,
where
things
have
changed,
makes
that
process
a
lot
easier
by
leveraging
history
and
the
ability
to
mock
all
of
these
things
or
keep
any
part
of
that
service
live
now
there
are
the
other
capabilities
that
we
have
built
here
if
you're
actually
using
this
now.
B
The
question
is:
what
do
you
do
at
this
point
right?
So
you
have
a
problem
here,
so
rate
ratings
may
or
other
details.
Maybe
something
has
changed
here.
Maybe
it's
a
problem.
So
what
you
can
do
here
is
from
a
real
user
perspective.
You
can
go
and,
for
example,
you
can
look
at
the
go
into
those
apis
over
here.
You
can
go
ahead
and
you
can
file
bugs
so
there
you
can
go
ahead
and
file
a
bug
over
here
to
jira.
B
B
So
if
we
go
back
to
our
previous
item
here,
so
if
we
look
at
the
list
movies,
which
is
the
api,
we
were
testing
so,
for
example,
if
excuse
me,
you
have
changed
something,
and
maybe
you
have
gotten
rid
of
this
actor's
last
name
error.
You
can
go
here
and
say
I
want
to
update
this
data
file,
the
the
reference,
what
we
call
golden
and
you
can
say-
okay,
I
want
to
just
remove
those.
B
So
similarly,
if
there
are
changes
that
you
can
you're
making
you
can-
and
all
you
need
to
do
is
change
the
data
you
can
do
them
from
here,
but
if
you
actually
need
to
edit
the
apis
make
changes
for
those
you
will
have
to
go
into
our
tool.
You
have
to
go
into
the
tool
here.
As
I
showed
you,
we
picked
up
a
couple
of
requests
here,
so
you
can,
you
can
go
in
here,
make
whatever
changes
you
need.
If
you
need
to
add
parameters,
etcetera,
you
can
add
them
here.
B
If
you
expect
that
the
result
will
change,
you
can
make
the
edits
over
here.
You
can
add
in
something
over
here,
for
example.
I
can
just
add
in
another
item
I
can
so
I
can
update
the
expected
results
put
in
whatever
and
and
then
I
can
up
save
this
all
this
that'll
update
the
test,
so
you
can
update
the
tests
using
all
of
this
inform
using
all
of
this
information
and
once
again
the
main
benefit
goes
back.
To
now
that
I
have
created
the
end-to-end
test,
I
can
just
use
it
for
service
testing.
B
I
don't
have
to
create
a
huge
amount
of
engineering
effort
writing
service
tests.
Where
I
test
that
service
in
isolation,
I
can
just
create
it
from
there.
The
component
is
that
I
can
just
create
from
the
same
effort
and
all
this
effort
is
reusable
if
I
go
ahead
and
make
a
change
to
one
of
the
things
that
gets
propagated
across
the
service,
taste
endurance,
it's
everything
and
you
can
leverage
all
of
this.
All
of
this
made
courtesy
of
istio
makes
the
whole
process
a
lot
easier.