►
Description
Join us for Kubernetes Forums Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Testing in A Distributed Systems World - Fernando Mayo, Undefined Labs
While microservices are becoming the norm due to advancements in development, deployment and monitoring techniques in the last few years, we are still using the same testing methodologies we used for monolithic apps. In this talk, we look at how distributed tracing can be applied to testing modern, distributed applications, from unit to end-to-end tests, to continuously give developers invaluable insight on how entire applications behave, and when and why they fail, before they are deployed to production. We'll also discuss the power of distributed context propagation and how it can be leveraged for testing purposes, from safely testing in production to failure injection.
A
Hello:
everyone,
my
name,
is
Fernando
Maggio
and
I'm.
Gonna
talk
about
testing
so
a
bit
about
me,
I'm,
the
CTO
and
founder
of
a
start-up
called
undefined
labs
or
undefined
labs.
We
build
tools
for
developers
to
make
them
more
efficient
and
we
are
focusing
right
now
on
solving
the
issue
of
testing
complex
micro-services,
previously
I
co-founded,
another
company
called
to
them.
That
was
a
container
management
platform
and
that
was
acquired
by
doctor
in
2015,
where
I
spent
a
couple
of
years.
A
So
as
enough
for
today
is
going
to
be
first,
why
testing
and
why
testing
is
cool,
and
you
should
do
it?
Why
testing
microservice
is
so
hard
a
bit
about
testing
in
production
and
different
techniques
that
we
can
use
to
test
directly
in
the
production
environment,
and
how
can
we
use
the
distributed
context,
propagation
system
of
tracing
in
order
to
help
with
that,
so
the
beginning
machine
production,
using
our
ability
tools,
is
great
right,
we're
all
here
because
of
that,
but
it
should
be
really
our
last
resort.
A
We
shouldn't
be
using
that
every
day
it
only.
It
should
only
happen
a
few
times
in
the
month.
Why?
Because
solving
an
issue
in
production
is
extremely
expensive.
If
you
are
solving
an
issue
in
production,
you
have
to
think
about
all
these
costs.
So
first
is
the
detection
cost.
Can
you
detect
that
your
application
is
not
behaving
as
I
suspected?
A
You
might
be
able
to
detect
a
spike
in
latency
and
a
spike
in
a
row
right,
but
do
you
do
you
know
if
it's
actually
returning
the
right
thing,
then
it's
travel
should
in
it?
Have
you
added
the
right
instrumentation
in
order
to
be
able
to
troubleshoot
that
issue?
Do
you
have
the
right
sampling?
Do
you
have
the
right
attention?
How
much
does
it
cost
to
store
an
analyzer?
All
data
for
troubleshooting,
then
is
the
fixing
course
can
any
rollback
I'm
fixed
are
temporarily,
do
I
have
to
roll
forward.
A
We
have
to
come
up
with
a
hot
fix.
Do
I
have
the
people
that
can
do
that
so
I
have
to
clean
up
any
data
stores
that
have
been
poisoned
verification
codes
can
I
even
verify
that
the
fix
actually
fix
the
issue,
or
do
we
have
to
ask
a
user
to
basically
verify
it?
For
me,
then,
is
the
user
impact
course,
of
course,
if
you're
debugging
an
issue
in
production
is
because
some
users
have
been
impacted,
so
how
many
for
how
long?
How
much
money
have
we
lost?
A
How
much
impact
has
our
brand
reputation
had,
and
it's
also
in
TV
burnout?
So
someone
will
get
page
because
of
this,
so
we
want
to
reduce
those
pages
at
a
minimum.
So,
as
hitches
get
closer
to
the
end
user,
they
are
more
expensive
to
solve.
So
how
can
we
avoid
that?
Well,
there's
a
thing
called
testing.
That
is
actually
it
will
actually
help
with
that.
So
testing
is
available
using
the
risk
of
your
application
not
performing
as
expected
at
the
lowest
possible
cost.
A
So
it's
about
reducing
the
risk,
because
the
thing
will
never
be
able
to
tell
you
that
your
application,
100%
correct,
that's
impossible,
but
it's
going
to
be
able
to
reduce
that
that
risk
in
a
very
cost-efficient
way,
and
it's
about
reducing
the
risk
of
your
application,
not
yet
forming
a
suspected,
not
performing
a
suspect
that
means
robustness.
So
that
will
be
probably
hitting
your
error
rate
as
allows
performance
will
probably
be
some
latency
as
allows
and
correctness
is
about
your
application.
Returning
the
right
thing,
which
is
not
so
obvious.
A
So
in
the
monolith
era,
testing
was
was
great.
It
was
easy.
It
was
good
times
we
had
a
three-tier
architecture,
for
example
I'm
developing
the
back-end
service,
so
I
could
really
replicate
this
stack
anywhere
in
my
laptop
in
seei
stage
in
production,
I
can
run
a
few
unit
tests.
Integration
tests,
internet
I,
can
run
all
this,
even
in
my
laptop
and
then
in
production
be
taking,
and
several
shooting
issues
was
releases,
just
monitoring
some
metrics
using
some
logs
to
to
troubleshoot
what
the
problem
is.
A
So
it
was
relatively
easy,
but
in
the
era
of
microservices
this
we
can't
have
any
more
complex,
now
I'm
the
service
owner
of
this
little
box.
That
is
part
of
our
greater
application
and
a
greater
thing.
I
still
have
to
test
that
service
in
isolation.
I
still
have
to
to
be
able
to
make
sure
that
it's
correct
I
can
run
unit
tests.
I
can
run
integration
tests,
but
now
because
I
don't
have
access
to
my
time,
streams
I
cannot
replicate
them.
A
A
A
So
if
we
can
along
the
replicate
production-
because
you
know
it's-
it's
not
the
number
of
services
that
we
have
in
production-
configuration
SSL
certificate
traffic
patterns,
scale,
data
release,
cadence,
so
different
teams
will
release
services
at
different
times
and
that's
very,
very
hard
to
replicate
in
any
environment,
less
directly
the
simple
action,
but
before
we
get
too
excited
about
that,
let
me
say
that
production
testing
is
not
a
replacement
for
people
reduction
testing.
We
should
still
test
in
pre-production
before
release
into
production.
A
It's
much
cheaper
and
we
can
do
that
in
an
automated
way,
so
think
about
testing
in
production
as
another
technique
that
you
can
use
next
to
pre-production
testing
and
that
it
requires
investment
to
do
it
right.
So
testing
production
is
not
free.
Is
not
shipping
my
code
to
production
and
see
what
happens?
We
actually
have
to
call
for
it.
We
actually
have
to
do
some
investment
to
do
it
right.
A
So
when
we
think
about
testing
in
production,
we
think
about
many
techniques
and
that
we
can
apply
to
to
make
sure
that
our
application
works
as
expected
when
deployed
in
production.
So
it
goes
from
the
moment.
We
deploy
the
new
service
and
that
service
is
running
in
production,
but
not
serving
any
real
user
traffic
to
the
point
where
that
new
service
is
actually
handling
a
hundred
percent
of
user
traffic.
A
So
this
all
these
techniques
have
some
risks
in
common,
so
the
first
one
is
user
impact.
If
you
are
testing
directly
in
production,
you
have
to
make
sure
that
that
users
are
not
affected
or
the
minimum
amount
of
users
are
affected.
The
second
one
is
stay
poisoning
if
you
are
releasing
an
untested
version
of
your
application
in
production,
and
you
start
throwing
tests
against
that.
You
have
the
risk
of
poisoning
your
data
stores.
A
So
then
you
have
to
be
able
to
clean
them
up
if,
if
something
goes
wrong,
which
is
not
easy
traffic
saturation,
if
we
do
load
test
and
stress
test
and
gas
production,
we
can
actually
separate
the
user
traffic
and
we
can
actually
cause
an
outage
telemetry
they
tes
Q.
So
if
we
testing
production-
and
we
have
these
versions
of
our
applications
that
are
being
tested,
we
also
have
the
risk
of
polluting
our
telemetry
data.
A
So
now
we
won't
be
able
to
differentiate
between
what
telemetry
data
is
related
to
end-user
traffic
handling
and
what
is
related
to
test
traffic,
and
because
of
that
we
have,
we
can
have
misfire
alerts.
So
if
we
have
a
lead
from
this
telemetry
data,
we
might
fire
pages
based
on
tests
that
go
wrong.
So,
in
order
to
mitigate
all
this
risk,
we
have
to
make
sure
that
our
application
is
aware
of
tests
being
made
in
production.
A
Will
that
solve
the
problem?
If
we
go
to
the
to
the
risks,
we
have,
for
example,
user
impact.
How
can
we
mitigate
the
user
impact
risk
by
using
this
label?
Well,
if
we
use
this
label,
we
can
actually
be
able
to
test
services
before
they
are
released
to
users.
If
I
were
routing
layer
is
able
to
differentiate
between
test
requests
and
and
user
requests,
we
can
actually
detect
those
requests
to
services
that
are
not
receiving
any
end
user
traffic
State
poisoning.
If
our
services
are
aware
of
test
traffic,
they
can
actually
handle
that
appropriately.
A
So,
for
example,
stateful
services
that
need
to
write
data
can
do
so
with
a
flag
or
offend
database
or
even
in
table,
in
order
to
separate
that
test
data
from
the
actual
end
user
data
traffic
saturation.
If
we
do
load
test
and
stress
test
against
production,
and
we
can
label
those
requests
as
test
requests,
we
can
make
sure
that
our
routing
layer
can
apply
quality
of
service
rules
to
that
traffic.
So
we
can,
for
example,
have
less
priority,
for
this
request,
have
rate
limiting
or
even
kill
it.
A
If
there's
something
going
on
in
production,
and
we
want
to
stop
all
the
load
testing
production,
let
the
limited
as
data
skew.
So
we
can
mark
all
that
elementary
data
that
we
produce
or
metric
loss
and
faces
that
come
from
test
requests
as
test
telemetry.
So
we
can
then
later
on
use
that
label
to
differentiate
to
how
different
dashboards
for
what
the
end
user
traffic
telemetry
looks
like
and
the
testing
traffic
coming.
Because
of
this,
we
can
make
sure
that
our
alerts
are
not
fired
because
of
testing
telemetry.
A
A
So
context,
propagation
comes
for
free
with
tracing.
So
if
you
are
using
up
and
trace
in
open
sensor
for
open
turn
and
read,
you
will
have
conscious
propagation
Milton.
He
has
a
very
developer
friendly
API,
so
the
developer
can
set
and
get
any
key
values
on
this
context.
At
any
point,
in
the
request
it's
compatible
with
resin
coating,
so
is
scope
aware,
so
it
will
actually
attach
that
context
to
the
thread
or
coroutine.
That
is
handling
that
request.
That
comes
with
the
library
and
it's
compatible
with
multiple
synchronous
and
asynchronous
protocols.
A
So
the
context
propagation
will
actually
work
through
synchronous
protocol
HTTP
and
the
RPC,
but
it
can
also
be
extended
to
asynchronous
protocols
like
message
queues,
and
we
just
have
to
be
aware
that
everything
that
we
add
to
the
context
will
actually
increase
the
request
size.
So
it
will
make
processing
those
requests
a
bit
a
bit
longer.
So
the
current
state
of
context
propagation,
for
example,
in
open
tracing,
which
is
a
an
API.
It
just
defines
the
context
propagation,
the
arbitrary
context
propagation
has
baggage.
A
So
you
can
attach
package
to
any
request
if
key
value
pairs
and
because
it's
just
an
API
that
phaser
is
the
one
that
will
actually
define
how
that
gets.
Idealized
and
deserialize
so,
depending
on
the
vendor
and
depending
on
the
object
register
that
you
use,
that
context
will
be
propagated
using
some
headers
or
others
or
some
format
or
other
in
open
telemetry.
This
is
still
working
progress,
but
as
of
this
morning,
he
was
called
disobey
the
context.
It's
still
key
value,
but
now
we
have
some
metadata
we
can
attach
to
to
that
baggage
like
TTL.
A
So
at
the
top,
we
have
the
format
of
the
trace
context.
So
that's
how
tracing
information
will
get
serialized.
This
is
just
trace.
Ids
param
span,
ID
sampling
information,
so
just
trace
a
specific
information
and
that's
actually
in
curriculum
recommendation
state
and
at
the
bottom
we
have
the
correlation
context,
draft,
which
is
how
our
arbitrary
metadata
will
get
sterilized
and,
as
I
said,
this
is
still
working
progress,
but
this
is
the
current
state.
A
So
we
use
this
project
propagation
technique
for
propagating
tracing
information.
So
then
we
can
troubleshoot
issues
in
production,
but
we
can
also
use
it
for
any
other
for
other
things
like,
for
example,
our
test
traffic
level.
We
can
use
this
this
system
to
mark
individual
requests
as
test
requests,
and
so
we
can
then
act
accordingly.
We
can
also
use
it
for
fall
injection,
so
on
every
request,
we
can
actually
specify
instructions
for
falls
in
line
that
will
only
affect
that
request
and
not
an
entire
node
or
an
entire
service.
A
A
If
we
point
our
service,
that
we
are
developing
locally
against
production
and
every
test
that
we
that
we
make
we
actually
attach
a
label
to
that
request,
assess
tenancy
equals
test,
so
I'm,
marking
that
every
request
I
make
to
my
downstreams
is
a
test
request.
So
how
do
we
do
this?
As
I
said,
the
API
is
very
easy,
so
as
a
developer,
I
just
have
to
start
a
trace
in
my
test.
This
is
an
example
using
go,
I,
start
a
trace
and
then
I
attach
that
package
to
the
tracer
in
this
test.
A
This
trace
is
a
test,
a
test
request,
so
this
context
will
be
propagated
across
all
the
all.
The
requests
are
due
to
my
downstream
dependencies.
Now
on
the
downstream
dependency
side.
In
the
server
side,
we
have
to
be
able
to
handle
this
request,
so
you
might
end
I,
have
a
stateful
service
and
I'm
making
a
request
that
will
actually
write
some
data.
A
So
in
order
to
do
so,
this
is
the
first
like
the
simplest
way
to
support
that
use
case,
which
will
be
deploying
two
separate
instances
of
my
service,
one
for
handling
end-user
traffic
and
another
one
for
handling
test
traffic
and
then
letting
the
routing
layer
be
the
one
that
actually
decides
where
to
send
the
request.
It
will
read
the
correlation
context
gather
and
which,
okay,
this
is
a
test
request,
so
I
will
redirect
it
to
the
test
version
of
the
part
of
my
service,
and
then
every
service
can
have
its
own
datastore.
A
A
We
also
have
to
manage
telemetry
data,
so,
as
I
said
before,
if
you,
if
we
don't,
we
are
not
careful
with
this.
We
have
telemetry
data
arrays
for
both
tests
and
production,
traffic
and
real
unusual
traffic.
So
we
have
to
be
able
to
mark
those
metrics
and
and
those
traces
and
logs
with
that
tenant
label.
So
this
is
an
example
of
go
which
is
an
API
that
is
still
working
progress,
but
the
idea
is
to
for
every
metric
that
we
create
add
the
tenancy
key,
add
the
tenancy
label.
A
So
when
we
omit
matrix
as
you
sorry
to
request,
they
are
actually
segregated
to
the
right
tenant,
they
are
either
end
user
metrics
or
test
metrics.
This
will
allow
us
later
on,
to
build
dashboards
that
are
for
end
user
data
or
test
data,
and
then,
in
order
to
manage
other
side
effects
that
you
might
have,
we
can
just
read
that
tenancy
label
from
the
disability
context.
A
So,
for
example,
with
this,
we
can
actually
implement
the
multi
tenant
storage
by
asking
to
our
tracing
library.
What
is
the
value
for
the
tenancy
in
this
request?
So
if
it's
a
test,
tenancy
I
can
actually
use
the
testing
the
test
database,
but
also,
for
example,
using
sandbox
accounts.
So
if
for
the,
if
our
application
is
using
a
third
party
service
like
stripe
and
we
are
charging
credit
cards,
we
can
actually
use
this
tenancy
in
order
to
use
a
sandbox
account
instead
of
a
real
account.
So
we
can
avoid
these
side
effects.
A
Another
example
of
using
conscious
propagation
is
in
in
testing
with
services
that
are
not
yet
released,
so
imagine
that
I
have
my
service
with
one
serving
initial
traffic
in
production.
I
deploy
but
I'd,
not
release
I,
don't
release
my
v2
of
my
service,
so
this
is
running
in
production,
but
it's
not
actually
receiving
any
inducer
traffic.
I
can
then
send
request
against
production
that
has
a
label
saying
I,
want
this
request
to
target
the
v2
version
of
my
service.
A
So
if
my
routing
layer
is
aware
of
tenancy,
like
the
the
routing
layer,
can
actually
redirect
that
that
request
to
my
service
when
it
goes
to
my
service,
so
it
will
use
all
the
production
environment
of
my
application,
but
before
it
goes
to
my
service,
the
service
that
I'm
responsible
for
it
will
actually
go
through
the
new
version
and
it
can
test
that
that
service
is
behaving
as
expected
without
affecting
any
user
traffic
like
ng.
So
traffic
doesn't
even
know
that
I'm
performing
tests
against
production.
A
Another
example
is
fall
injection,
so,
unlike
traditional
chaos,
engineering
where
the
whole
service
will
be
affected
or
a
whole
node
will
be
affected
using
context
propagation,
because
he's
tied
to
a
specific
request.
We
can
actually
inject
instructions
for
for
a
specific
test
request
to
inject
a
failure
somewhere
in
the
in
the
in
the
service.
So
in
this
case,
I'm
saying
I
want
this
test
request
that
goes
against
production
to
introduce,
for
example,
an
HTTP
delay
on
every
client
request
that
I
make
in
my
service.
A
So
this
test
request
will
go
through
all
the
normal
flows
of
production,
but
when
it
goes
to
my
service,
my
service
will
actually
understand
that
instruction
and
it
will
execute
it,
so
it
will
actually
introduce
that
delay.
So
with
this,
we
can
synthetically
test
that
my
service
will
and
the
entire
application
will
behave
as
expected
when
I
introduce
this
failure.
So
this
is
a
very
fine-grain
fall
injection
that
we
can
do
use
in
context,
propagation
feature
flags,
for
example,
so
so
I
have
a
some
code
in
my
service
that
is
behind
a
feature.
A
Flag
and
I
want
to
be
able
to
test
it
before
before.
We
listen
it
to
users,
because
I
want
to
mitigate
the
user
impact.
So
in
this
case,
I
can
attach
an
instruction
to
my
collection
context
and
I
want
the
feature
one
of
my
service
to
be
enabled
for
this
specific
request.
So
nobody
will
will
see
that
feature,
but
I
can
run
and
into
and
test
against
production
that
will
actually
go
through
that
that
code
and
test
it
proactively
before
we
release
it
to
end-users.
A
Another
example
is
using
test
accounts,
so
the
correlation
context
is
not
yet
is
not
only
set
at
the
beginning
or
outside
of
production.
Any
service
in
my
application
can
actually
add
that
label.
So,
for
example,
we
have
a
indication
service
in
our
application
and
that
looks
up
the
database
of
users
and
and
see
can
see
if
the
user
is
a
test
account
or
not.
You
can
actually
inject
that
information
into
the
into
the
request
that
will
be
propagated
across
my
application.
A
So
I
don't
need
my
services
to
be
looking
up
will
be
asking
my
authentication
service,
whether
that
you
said
is
a
test
user
or
not.
I
can
add
that
information
in
band,
so
every
service
in
my
application
will
actually
know
that
that
is
a
test
account
and
will
avoid
side
effects
like
charging
credit
cards
or
you
know,
sending
telemetry
to
a
different
different
tenant.
A
So
what
are
the
consequences
of
all
this?
So
the
first
thing
is
that
if
we
design
our
applications
to
be
to
be
able
to
be
tested
in
production,
we
avoid
the
need
to
have
to
replicate
our
application
stack
anywhere.
So
we
don't
have
to
run
it
locally.
We
don't
have
to
run
it
on
CI
or
even
a
stake
in
stay
in
its
heart.
A
So
by
testing
in
production
and
by
having
test
accounts
and
sub
box
accounts
in
in
production,
we
can
avoid
having
to
have
having
to
replicate
that
data
stack,
which
probably
is
impossible
to
replicate
by
segregating
telemetry.
We
can
actually
monitor
and
troubleshoot
test
like
we
monitor
and
troubleshoot
any
end
user
traffic.
So
this
is
much
more
rich
information
that
we
can
use
in
order
to
travel,
so
why
a
test
fail
instead
of
just
a
dump
of
logs
and
we
can
add
other
types
of
tests
of
traffic
to
our
application.
A
So
by
using
this
technique
of
using
condense
propagation,
we
can
actually
add,
for
example,
sandbox
accounts.
So
if
we
are,
our
application
is
actually
a
downstream
dependency
of
a
customer.
We
can
actually
mark
all
the
requests
that
come
from
the
customer
using
a
sandbox
account
as
sandbox
traffic,
so
we
can
have
different
quality
of
service
rules.
We
can
have
different,
segregated,
metrics,
different
alerts
from
the
actual
production
traffic
or
even
development
traffic.
Maybe
all
our
services
are
developed
against
production,
so
the
user
doesn't
have
to
replicate
anything
locally.
A
It
doesn't
have
to
connect
to
staging
or
anything,
but
they
can
actually
connect
to
production
where
all
the
services
are
there
anyway.
So
we
can
mark
all
the
requests.
I'll
go
through
and
I'll
go
to
production
with
a
development
label,
so
we
can
make
sure
that
there
are
no
alerts
filed.
If
something
goes
wrong
with
the
development
traffic,
we
can
actually
have
different
retention
rules
for
logs.
We
had,
we
can
have
different
levels
and
so
on.
A
So
the
key
takeaways
is
that
we
should
catch
issues
before
as
early
as
possible
in
the
cycle
through
proactive
testing,
because
it's
much
cheaper
than
actually
solving
that
issue
in
directly
in
production.
Testing
in
production
can
actually
be
the
most
efficient
way
to
test,
and
sometimes
the
only
way
to
test
complex
architectures,
because
we
just
can't
replicate
it
anywhere.
We
should
design
our
applications
to
be
able
to
be
tested
in
production,
so
it
needs
investment
is
not
free,
it
doesn't
come
for
free,
so
just
throw
something
production
and
and
do
some
tests.
A
We
actually
have
to
be
conscious
that
we
are
testing
in
production
and
we
have
to
be
able
to
code
for
that
and,
let's
make
sure,
let's
make
use
of
context
propagation.
We
have
instrumented
our
applications
of
ready
with
tracing.
So,
let's,
let's
use
that
context
propagation
system
that
we
have
in
our
applications
for
for
testing.
So
we
can
make
the
most
of
of
that
investment,
which
is,
is
a
big
investment.
And
with
that,
thank
you,
and
if
you
want
to
talk
about
testing
I'll
be
available
later.
C
A
Hopefully,
when
the
standard,
the
correlation
context,
standard
actually
gets
through
and
approved,
and
and
all
this
service
measures
will
actually
start
supporting
collection
context
in
a
better
way,
so
you
can
actually
understand
all
the
baggage
that
goes
through
them
and
you
can
actually
have
more
sophisticated
rules
based
on
that
baggage.
So
you
can
say:
okay,
I
want
every
request
that
has
a
test
label
actually
do
this
or
have
these
failure
or
we
can.
We
can
do
more
more
things,
but
either
one
because
there's
no
standard
is
based
on
HTTP
header.
D
Yeah
hi
I
think
you
forgot
the
demo
we
at
what
point
of
time
you
are
talking
about
like
at
the
testing
to
be
done
in
production.
Are
we
saying
like
immediately
releasing
it
to
the
production?
Are
you
talking
purely
from
the
GCP
I
h2o,
which
is
coming
up?
It's
like
a
new
kind
of
model
or
whatever,
so
at
what
point
of
time
are
we
talking
about
like
these
production
testing
needs
to
be
done
at
what
point
of
time?
Is
it
immediately
releasing
the
product?
Is
it
after
some
time
at
what
point
of
time
how.
A
So
we
can
test
in
production
across
all
these
steps,
so
what
I
would
do
is
I
will
try
to
test
after
deployment,
but
before
releasing
so
that's
the
moment
where
less
we
will
have
less
user
impact,
because
no
one
is
actually
exposed
to
the
new
version
of
the
service.
So
by
using
this
level
we
can
actually
very
direct
traffic
to
the
to
that
version
of
the
service
that
is
not
yet
exposed
that
that's
that's
the
moment
where
there's
less
impact,
so
I
will
try
to
do
that
as
much
as
possible.
A
If
we
can,
if
you
can
detect
it
in
that
in
that
step
is
much
cheaper
from
an
impact
perspective,
but
you
can
do
it
later
on,
like
you
can.
Actually
you
can
run
these
periodically
if
you
want,
so
you
can
run
it
again.
You
can
run
it
after
deployment,
but
you
can
also
have
CI
run
end-to-end
tests
with
this
label
every
five
minutes
every
hour
to
make
sure
that
your
application
in
production
still
works
as
expected
and
still
have
the
segregation
of
telemetry
and
and
avoiding
side
effects.
So.
D
A
To
make
sure
that
your
service
is
still
compatible
with
all
the
other
services
that
have
been
deployed,
so
production
will
change
over
time.
So
this
is
more
of
a
synthetic
monitoring.
Check
is
not
a
test,
but
you
can
do
synthetic
monitoring
using
requests
with
this
label
and
do
more
advanced
checking
of
if
your
application
actually
behaving
as
Specter
or
not
than
a
ping
to
an
endpoint.
So.
A
Will
it
would
produce
an
overhead
but
I
think
is
better
for
to
have
that
overhead
in
your
production
system
and
actually
catching
issues
before
users
do
I
think
it's
worth
it
and
especially
if
you
have
coded
your
application,
you
have
turned
your
application
to
not
have
side
effects
of
those
tests.
So,
yes,
you
will
have
a
bit
of
overhead,
but
I,
don't
I
think
the
cost
justifies
it
because
you
will
be
able
to
catch
issues
before
your
users
actually
start
complaining.
The.
D
A
B
A
Yeah,
that's
sorry!
That's
the
sole
question.
Yes,
do
you
have
to
make
sure
that
you
know
unauthorized
users
can
say
the
package
arbitrarily,
so
if
you're
inside
application,
that
should
be
okay
but
from
outside
you
have
to
have
some
way
of
authenticating
that
request
outside
of
the
of
the
baggage.
So
yes,
absolutely
you
have
to
make
sure
that,
like
no
random
user
can
adjust
a
net
request
with
a
package.
So
probably
your
hn3
paralyzer
edge
router
will
strip
that
out
and
you
have
some
other
way
to
submit
requests
against
production.