►
From YouTube: Diffy - Predicting and Preempting Regressions in Services - Puneet Khanduri & Prashant Khanduri
Description
Want to view more sessions and keep the conversations going? Join us for KubeCon + CloudNativeCon North America in Seattle, December 11 - 13, 2018 (http://bit.ly/KCCNCNA18) or in Shanghai, November 14-15 (http://bit.ly/kccncchina18)
Diffy - Predicting and Preempting Regressions in Services - Puneet Khanduri & Prashant Khanduri, Diffy (Beginner Skill Level)
Diffy is a tool used at Twitter to automatically catch regressions in our backend services before they are deployed to production. Diffy is open source and has been adopted into modern continuos deploy workflows by the larger community. This talk will cover the fundamental concepts behind Diffy and how its automation helps achieve peace of mind for service owners, devops, and QA folks.
About Puneet:
Puneet Khanduri developed Diffy at Twitter. He currently heads the ML&AI Services team within the larger Cortex group and also helped start Twitter's India Engineering office as the site lead for 1.5 years while driving a stealth project across engineering, product, and design. Currently, Puneet heads the ML Platform Services team at Twitter. Prior to Twitter, Puneet focused on small footprint JVM design and datacenter networking protocols as a researcher at Sun Microsystems (acquired by Oracle). Puneet has given multiple talks at various conferences including Google's GTAC and O'Reilly's Velocity.
About Prashant
Prashant is a solutions architect with over 8 years of experience in leading product design and implementation of technology solutions and processes. Prior to Diffy, he made significant technical contributions as a engineering manager / tech lead at Affirm, Jawbone, Hearsay Systems and as a data scientist at Microsoft (Core Ranking, Bing) and Amazon (Personalizations team). He holds a Masters degree in Computer Science (focus on ML / AI) from University of Southern California, US and a Bachelors in Computer Engineering from University of Waterloo, Canada. He is excited about eliminating logistical friction developers incur when trying to solve real world problems.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
Myself
we
need
can
duty.
This
is
my
co-founder
and
brother
Prashant
country
and
we're
here
to
talk
to
you
about
Diffie
and
before
we
talk
to
you
about
the
fee,
I
want
to
ask
the
audience
a
question:
it's
a
yes/no
answer.
So
if
the
answer
is
yes
put
your
hand
up,
November
2015
Twitter
stock
was
down
5%,
because
new
user
signup
was
broken
for
two
days,
and
users
were
trying
to
sign-in
trying
to
get
a
particular
handle,
and
they
just
couldn't
how
many
people
remember
that?
A
So
as
a
developer,
if
you
own
any
back-end
services,
your
shipping
new
features
all
the
time
right,
whether
you
follow
a
sprint
model
or
some
sort
of
a
Kanban
model,
you're,
always
creating
new
features
and
as
you're,
creating
new
features
in
your
code
base
your
refactoring
existing
code
to
make
it
you
know,
make
the
class
hierarchies
more
coherent
and
it
is
almost
always
impossible
to
ship
a
new
feature
without
modifying
code.
That
effects
other
features
as
well.
So
you're
constantly
worried
about
hey.
You
know,
with
the
two
new
features
that
I'm
shipping
into
production.
A
This
release
have
I
broken
anything
with
the
other
40
existing
features
that
are
already
out
in
production
have
I
accidentally
modified
their
behavior
and
if
you're
an
SRE
site,
reliability,
engineer
or
you're
a
DevOps
engineer,
then
you
have
an
even
bigger
problem,
because
you
don't
even
know
what
the
changes
are
that
are
getting
shipped
into
production
right.
You
have
this
abstract
docker
image
or
a
jar
or
an
artifact
that
you
need
to
listen
to
production,
and
you
know
you
have
all
the
risk
and
an
accountability
without
any
of
the
understanding
and
authority.
A
As
to
you
know
what
the
change
is
that
is
being
shipped
out
so
you're
worried
about,
like
you
know
how
do
I
make
sure
that
the
change
that
I'm
about
to
roll
out
in
production?
Is
this
going
to
cost
me
another
three
four
hours
to
sort
of
revert
back
roll
back
and
go
back
to
the
previous
version
and
how
can
I
prevent
that
from
happening
in
the
first
place
altogether?
And
you
try
to
address
this
using
a
bunch
of
approaches
like
canary.
A
So
you
send,
you
know,
1%
of
the
traffic
to
the
new
code
and
then,
if
there
aren't
any
exceptions,
you
move
forward
and
you
know
deploy
to
the
whole
cluster.
The
problem
is
that
if
the
business
logic
is
broken
in
a
subtle
way,
then
there
is
no
way
for
you
to
know.
You
know
at
the
high
level
by
looking
at
exception,
logs
or
other
kinds
of
metrics,
because
it's
it's
business
logic
right
like
it's
not
throwing
exceptions
or
anything
and
then
for
a
lot
of
organizations.
A
The
tipping
point
typically
happens
when
they
have
to
break
a
legacy,
monolith
and
move
into
microservices,
which
is
also
what
happened
at
Twitter
and
what
we've
seen
at
at
a
lot
of
customers
happening.
So
now
you
have
the
problem
that
you've
got
this
legacy
monolith
that
has
some
functionality
and
you
know
it's
horrible,
you
hate
it,
but
it
works
right
and
you
want
to
move
into
a
microservices
architecture.
But
how
do
you
prove
that
the
microservices
implementation
is
equivalent
to
the
monolith?
And
how
can
you
provide
those
guarantees?
A
You
know
before
you
remove
all
of
your
traffic
over.
So
you
know
these
are
sort
of
situations
when
the
risk
ends
up
being
extremely
high
for
companies
and
they
need
some
sort
of
specialized
mechanisms
to
get
large
amounts
of
coverage
and
guarantees
to
make
sure
that
things
won't
break
when
they
make
these
radical
changes
to
their
environments.
B
At
the
end
of
the
day,
so
far,
what
we're
trying
to
get
to
is
gain
confidence
over
the
coaching
that
we're
shipping.
How
do
we
gain
confidence
over
the
code
changes?
And
then
there
are
certain
standards,
some
standard
practices
that
developers
have
sort
of
developed
over
time,
the
basic
tr0
approaches.
Let's
say
we
write
a
function
or
a
method
to
a
class.
We
end
up
writing
unit
tests
for
it.
B
So
let's
take
this
example,
and
let's
make
the
assumption
that
we
have
five
independent
code
paths
to
a
function
that
we've
just
written
to
get
100%
coverage.
We
have
to
write
five
tests.
A
single
test
gives
us
20%
coverage
developers
typically
or
whoever's.
Writing
the
test.
They're
good.
At
this
level.
Getting
hundred
percent
coverage
is
relatively
easy.
The
next
step
the
next
year
is
within
a
service
boundary,
the
the
component
test
boundary.
B
So
if
we
have
five
such
functions
or
starting
six
such
independent
functions
each
function
having
five
independent
code
paths,
all
of
a
sudden,
the
the
problem
scale
just
expanded
from
five
test
cases
to
15,000
test
cases
to
get
hundred
percent
coverage.
A
single
test
is
just
giving
you
point
zero
six
percent
coverage.
This
is
where
things
start
to
break
for
us.
B
Not
a
lot
of
people
are
writing
15,000,
test
cases
for
a
single
service,
and
let's
take
it
a
step
further
at
the
integration
test
layer
when
you're
looking
at
the
request,
the
the
request
path,
where
you're
going
through
four
such
services,
each
service
having
six
methods
to
go
through
each
method,
having
five
independent
code
paths,
the
problem
just
explodes
60
quadrillion
tests
to
write
to
get
one
percent
coverage.
Even
if
you
put
a
QA
army
behind
to
write
tests,
the
this
is
just
not
scalable
whatsoever.
A
single
test
is
barely
giving
you
any
coverage.
B
B
This
is
why,
if
he
comes
in
so
diffie's
goal
here
to
get
more
coverage
for
free,
the
key
here
is
for
free,
but
to
understand
the
for
free
part
part.
We
first
have
to
understand:
where
is
the
cost?
Now,
if
we
look
at
the
anatomy
of
a
test,
a
test
is
expensive
because
of
two
variables
because
of
two
reasons:
the
input,
because
you
have
to
think
about
what
are
the
different
inputs
that
would
trigger
the
different
code
paths
and
how
would
each
one
of
those
triggers
behave
their
assertions?
B
These
are
three
components:
the
input
and
the
assertions.
So
if
we
could
somehow
get
free
input
and
free
assertions,
maybe
we
can
get
more
coverage
so
with
Diffie,
we
address
the
free
input
side
by
just
looking
at
production
traffic.
Your
production
users
are
actually
exercising
all
these
code
paths
for
you.
So
if
you
were
to
replay
that
live
production
traffic
in
a
safe,
separate
environment,
you
get
the
free
inputs
now
for
the
free
assertions.
B
If
you
replay
the
production
as
your
known
good
code,
let's
make
the
assumption
that
production
right
now
has
the
known
good
code
and
the
delta
you're
trying
to
ship
is
the
change
you
want
to
check.
If
you
look
at
these
two
responses
and
compare
them
together,
that
becomes
your
assertion:
did
it
the
behavior
change
or
not?
If
the
behavior
did
not
change
what
your
shipping
is
correct,
it's
solid
if
the
behavior
did
change,
you
need
some
sort
of
a
manual
check
that
says
this
change
is
a
required
change.
B
It's
a
wanted
change,
it's
not
an
unwanted
change.
Unwanted
changes
are
basically
bugs.
So
with
these
two
simple
comparisons,
there's
still
one
other
problem
that
lies
in
front
of
us,
which
is
what
these
two
responses
contain:
some
packet
information
that
are
noisy
in
nature.
What
if
there's
some
undetermined
istic
behavior
within
the
application,
and
that
noise
will
always
bubble
up
as
bug
as
unwanted
change?
Some
examples
are
survey
generated
time
stamps.
B
If
your
response
packet
has
a
time
stamp,
that's
being
generated
in
production
and
different
times
time
being
generated
in
the
gnome
change
code,
those
two
differences
will
show
up,
but
that's
just
noise.
We
don't
want
to
put
our
focus.
There
is
that's
not
your
business
logics
mistake.
Similar
same
goes
with
generators
or
downstream
non-deterministic,
behavior
or
any
race
condition.
So
let's
take
a
look
at
how
defeat
solves
that
by
first
taking
a
deeper
dive
into
the
defeat
apology
and
I'll.
Let
my
brother
come
back
in
for
describing
the
defeat
of
the
defeat
apology,
so.
A
As
we
source
traffic
from
our
production
cluster,
the
production
cluster
here
remains
undisturbed.
Right,
Diffie
is
not
in
the
critical
path.
Tiffie
is
in
a
separate
staging
environment
and
all
the
analysis
that,
if
he
is
doing,
is
in
an
isolated
staging
environment
that
has
nothing
to
do
with
production.
A
This
is
the
code
that
needs
to
be
tested,
and
you
know
it's
a
release
candidate
that
might
be
deployed
into
production.
The
primary
and
secondary
here
are
instances
of
code
that
is
already
deployed
in
production
right.
It's
the
same
code,
that's
deployed
in
production,
but
it's
deployed
on
two
instances
here
and
when
we
get
the
responses
back
from
candidate
and
primary,
we
compare
those
two
and,
of
course
like
these
are
the
raw
differences
that
we're
getting
and,
and
they
will
have
a
lot
of
noise
in
them
right.
So
how
do
we
cancel
the
noise
out?
A
Well,
we
compare
the
old
code
to
itself
so
the
code,
that's
in
production,
we're
comparing
that
code
to
itself
by
comparing
the
responses
that
are
coming
back
from
primary
and
secondary
here
and
by
definition,
any
differences
over
a
large
volume
of
requests.
That
surface
here
are
non
deterministic
noise
right.
C
A
The
old
code
can't
agree
with
itself
on
some
aspects
of
the
response.
Then
it's
okay
for
the
new
code
to
not
be
able
to
agree
with
the
old
code
on
those
things
right,
and
so
we
subtract
all
of
that
noise
out.
We
boil
it
down
to
a
filtered
set
of
differences
that
are
actionable
for
the
developer
right.
A
So
let's
do
a
quick
demo
to
to
see
what
this
actually
looks
like
and
get
a
better
feel
for
it.
So
keep
this
this
image
in
your
head.
We
have
a
candidate,
primary
and
secondary
that
we're
going
to
deploy.
We
also
have
a
diff
instance
that
we're
going
to
deploy
and
then
we're
going
to
send
some
traffic
to
it
and
then
we're
going
to
look
at
the
results,
we're
going
to
look
at
the
raw
differences
and
we're
going
to
look
at
the
filtered
differences.
So
let's
run
this.
A
By
the
way,
if
you
download
the
project
from
github,
which
I
encourage
all
of
you
to
do
and
play
around
with
Divi,
this
is
exactly
what
you
need
to
do
to
run
this
particular
example.
This
is
something
that
we
ship
with
the
code,
and
so,
if
you
decide
to
play
around
with
the
fee,
if
you
want
to
make
any
contributions
to
do
fee,
this
example
is
a
great
way
for
you
to
see
the
results
of
your
changes
and
also
run
defy
out
of
the
box.
A
So
now
des
filles
deployed
all
the
primary
secondary
and
candidate
instances
are
deployed,
and
we've
also
sent
some
traffic
to
the
defeat
instance,
and,
let's
see
so.
This
is
an
endpoint
whose
name
very
creatively
came
up
with
as
JSON
where
we
sent
a
bunch
of
requests,
we
sent
a
hundred
requests
and
we're
seeing
some
differences
here.
So
these
differences
are
aggregated
at
every
node
in
your
schema
right.
So
the
cardinality
of
things
that
you
would
see
in
this
UI
is
bounded
by
the
size
of
your
schema
right.
It
doesn't
like,
if
you
send
10,000
requests.
A
This
wouldn't
change
right,
because
this
is
an
aggregate
view.
So,
there's
obviously
something
weird
going
on
here,
because
you
know
the
expected
value
for
this
particular
field
was
all
lowercase,
but
the
actual
value
that
we
received
back
was
all
uppercase.
So
obviously
there's
something
weird
happening
here.
There's
another
thing
that's
happening
here,
which
is
the
timestamps,
are
never
agreeing
with
each
other
right,
so
those
are
also
potential
regressions
that
are
being
reported
now.
This
is
the
raw
differences
that
we
described
earlier
so
now,
let's
turn
on
the
noise
cancellation
part.
A
So
when
we
turned
the
noise
cancellation
part
on,
they
feel
smart
enough
to
figure
out
that
hey
these
timestamps
that
I
just
create
out
these
are
okay
to
ignore
and
I.
Don't
need
to
worry
about
them,
but
these
named
things
that
we're
looking
at
there's
something
weird
going
on
here.
So
you
know
the
developer
better.
Look
at
this
or
you
know
the
QA
better,
look
at
it
and
in
order
to
make
this
more
actionable,
you
know
we
want
to
give
concrete
ampuls
to
the
developer
right.
A
So
you
can
pick
any
of
these
examples
here
and
you
know
we
can
get
the
full-blown
requests
that
caused
this
behavior
to
trigger.
So
this
was
a
get
request
with
this
path
that
caused
this
regression
to
appear,
and
this
is
the
primary
response,
which
is
the
expected
response
that
that
we
received
from
the
primary
instance-
and
this
is
the
full-blown
response
that
we
received
back
from
the
candidate
right.
So
this
kind
of
insight
you
know
a
developer
is
able
to
look
at
this
and
say
hey.
A
A
So
it's
giving
developers
the
ability
to
see
the
future
right
master
is
going
to
be
deployed
either.
You
know
in
the
next
release,
whichever
whenever
it
happens,
it
could
be.
You
know
next
week
it
could
be
tomorrow.
It
could
be
in
a
few
hours,
but
if
he
allows
the
purrs
to
see
the
future
and
be
able
to
say
here's
how
the
world
would
change
when
this
release
actually
happens
without
the
release
ever
happening
right.
A
So
if,
in
these
email
reports,
you
see
any
of
the
regressions
after
later,
the
reports
include
all
the
regressions
after
noise
cancellation
has
been
applied.
Then
you
have
the
opportunity:
go
to
go
fix
things
before
they
make
their
way
into
production
and
in
sort
of
CI
CD
integrations
that
we've
done
you
know
and
our
customers
have
done.
A
A
They
just
wanted
to.
You
know,
try
a
few
requests
out
and
staging,
and
then
they
would
get
into
this
loop
off
hey.
Did
you
try
that
use
case?
Hey?
Did
you
try
that
use
case
and
that
ended
up
slowing
them
down
to
the
point
where
it
took
only
two
days
to
write
the
code,
but
then
two
weeks
to
actually
ship
it
out
into
production
because
they
were
stuck
doing
testing
so
in
this
sort
of
situation,
when
they
saw
me
shipping
code,
you
know
basically
within
two
days.
A
They
also
wanted
to
use
it
and
that
led
to
the
request
that
hey
you
know
it
works
for
your
service.
At
Twitter,
we
were
using
thrift
services
for
our
binary
service
interface,
so
other
thrift
services
also
wanted
to
use
this.
So
we
ended
up
making
it
generic
for
any
thrift
service.
In
a
first
iteration,
then
HTTP
service
owners
came
along
and
said
that
hey
you
know.
A
You
know
other
large
companies
have
also
started
using
it
and
they
started
reaching
out
to
us
for
support
and
pretty
soon
I
found
that
you
know
if
I
wanted
to
do
justice
to
two
Diffie
I
had
to
now
spin
this
out
of
Twitter
and
work
on
it
full-time
and
so
that
led
to
the
formation
of
our
company
around
a
fee.
So
Prashant
and
I
have
been
working
on
this
for
the
past
few
months.
A
Full-Time
gathering
requirements
from
users
and
focusing
on
the
open-source
version,
as
well
as
for
users
who
don't
want
to
run
this
on
premise,
offering
a
hosted
solution.
So
if
you
didn't
catch
anything
else
from
this
presentation,
one
thing
to
remember
is
that
go
to
github
and
check
out.
You
know
open
Diffie,
slash,
Diffie
and
you
know,
give
us
some
stars
show
us
some
love
and
we'd
be
more
than
happy
to
work
with
you.
If
you
have
any
questions,
take
it
out
for
a
spin
if
you
liked
it
give
us
feedback.
A
If
you
don't
like
it,
you
know,
your
feedback
is
even
more
valuable
to
us,
and
this
is
that
twitter
handle
where
you
can
contact
us
and
you
can
always
reach
out
to
us
for
any
kind
of
support,
help
feature
request.
Anything
you
need
at
info
at
the
fee,
dot
a
I.
So
thank
you
so
much
for
your
time.
At
this
point,
we'd
like
to
open
it
up
for
questions.
Thank.
B
D
A
So
they
can
see
that
a
question
even
though
the
requests
are
randomly
sample,
they
can
see
those
things
and
they
want
us
to
provide
a
feature
around
PII
scrubbing
right.
So
that
is
something
that
is
in
the
works
that
you
know
we
are
planning
on
releasing
which
will
allow
users
to
specify
criterias,
for
you
know
which
parts
of
their
schema
need
to
be
obfuscated,
so
sensitive
PII
data
doesn't
get
exposed
to
even
their
own
developers.
E
F
A
We
actually
don't
require
the
api
to
be
versioned
and
the
way
we
are
able
to
do
this
is
by
actually
having
two
separate
network
addresses
for
the
different
instances
of
the
code
that
get
deployed
right.
So
the
API
version
can
be
identical
right
and
the
code
can
be
different
and
because
the
request
is
being
multicast,
we
know
which
response
is
coming
from
which
version
of
the
code,
so
the
versioning
doesn't
need
to
be
part
of
the
API.
We
are
able
to
deal
with
that
just
by
having
isolation
at
the
network
level,
okay,
dearly.
C
A
So
that's
that's
an
interesting
one.
Sometimes
you
run
into
that
issue.
If
your
primary
and
secondary
instances
are
not
deployed
in
staging
and
you're
actually
using
live
production
instances,
because
the
latency
characteristics
can
be
very
different
in
production
versus
in
staging
right.
So,
as
you
pointed
out,
data
could
have
changed
between
you
know
the
responses
coming
back
from
production
and
you're,
just
you
know,
requests
going
into
staging
right.
So
that's
cue
in
the
the
time
can
cause
data
to
change.
And
what
do
we
do
to
eliminate
that?
A
When
you
see
differences
in
you
know,
in
the
view
of
data
across
your
candidate
code
and
your
primary
code,
you
will
also
see
over
a
large
enough
number
of
samples,
the
same
difference
across
the
views
that
are
seen
between
the
two
versions
of
the
old
code
itself.
So
primary
and
secondary
will
also
have
sometimes
different
views
of
the
same
underlying
data
right.
So
then,
noise
cancellation
is
able
to
take
care
of
that
and
automatically
eliminate
those.
Those
regressions.
G
I
really,
like
your
your
approach,
have
you
ever
thought
about
integrated
with
Apache
kafka,
so
I
think
it
could
play
very
well
with
the
event
sourcing,
where
you
could
even
go
further
and
not
just
dump
in
some,
let's
say
recorded
input
traffic,
but
you
could
go
back
to
the
the
beginning
of
time
of
your
application
or
of
your
whole
services,
landscape
and
just
replay
everything
through
it.
Yeah.
A
A
Absolutely
great
observation:
we
have
a
few
friends
at
confluent
and
some
customers
who
actually
are
using
Kafka
very
aggressively,
so
we
ended
up
supporting
Kafka
as
well
for
these
customers
so
they're.
Basically,
logging
requests
into
a
Kafka
stream
and
then
replaying
traffic
from
the
Kafka
stream.
So
there
is,
there
exists
a
version
of
this
which
supports
Kafka,
we
piloted
it
with
one
customer.
A
So
when
a
customer
already
has
Kafka
in
their
environment,
you
know
we're
happy
to
support
it,
but
if
even
if
they
don't
have
Kafka,
you
know
the
live
request
pattern
that
we
have
where
we
just
proxy,
like
through
instrumentation,
into
the
production
cluster,
we're
able
to
get
live
traffic
that
doesn't
require
any
Kafka
specific
infrastructure
right,
so
that
works
broadly
across
the
board
for
for
pretty
much
everyone,
but
for
customers
who
are
comfortable,
owning
Kafka
infrastructure.
We
definitely
support
that.
A
G
A
You
have
a
proprietary
message
broker.
We
can
have
a
conversation
about
how
to
do
that.
But,
yes,
the
code
is
definitely
extensible
enough,
where
at
some
layer
it's
possible
to
plug
it
in,
but
it's
we
we,
you
would
need
to
talk
to
us
a
little
bit
or
you're,
welcome
to
dive
into
the
code
and
and
and
do
it
yourself,
but
it's
definitely
possible
to
do
that.
Thanks.
H
Ok,
thank
you
very
much
was
really
good.
I
have
a
question
about
kind
of.
You
are
recording
production,
requests
and
kind
of
reproducing
it
on
your
staging
environment.
What,
if
that
production
request,
doesn't
actually
make
sense
in
the
staging
environment,
because
you
have
different
data,
for
example,
you
make
a
request
to
get
user
number
one
two
three
and
that
doesn't
exist
in
staging.
How
do
you
make
it
consistent?
How
do
you
deal
with
that
right?.
A
So
what
most
of
our
customers
do
is
that
you
know
the
services
themselves
are
stateless
and
then
for
any
kind
of
state
that
they
need
to
read
they're
accessing
underlying
data
stores
right
so
what
they
typically
end
up
doing.
Is
they
end
up
using
production
data
stores
and
have
their
staging
instances
talk
to
the
production
data
stores?
So
in
because
of
that,
you
know,
the
the
request
makes
sense
in
staging,
because
staging
is
also
pointing
back
to
production
downstream.
A
A
As
well
as
possible,
yeah
yeah
yeah,
so
we
definitely
believe
in
being
as
far
left
as
possible
and
I
don't
mean
that
in
a
political
sense,
I
mean
that
in
in
you
know
the
sense
that
you're
talking
about.
But
what
we're
seeing
is
that
you
know.
Initially,
it
was
the
sres
who
were
you,
know
big
fans
of
this
tool.
They
really
wanted
to
use
it
and
you
know,
because
it
gave
them
a
lot
of
confidence
for
release
over
release
like
what's
going
on.
A
We
also
noticed,
then,
that
you
know
any
people,
any
teams
that
had
embedded
QA
engineers
that
were
responsible
for
doing
a
lot
of
regression.
Testing
manually
then
started
using
defeat
to
do
this
kind
of
thing
and
then
give
feedback
to
developers,
and
then
what
we
saw
was
that
developers
started
asking
each
other
for
defeat
links
in
their
code.
Reviews
that
hey
can
I
see
what
this
difference
actually
means.
In
terms
of
you
know
the
change
in
production.
A
Can
you
run
a
diff
instance
and
paste
it
in
the
code
review
right,
so
it's
already
making
its
way
left
all
the
way
to
the
developer.
Where
developers
are.
You
know
comparing
their
unmerged
branches,
to
master,
to
show
what
the
difference
looks
like,
because
people
want
to
see
what
this
difference
means.
A
So
we
definitely
believe
in
in
the
same
view
that
you
describe
that
that
github
does
right
that
you
know
we
want
to
be
as
close
to
the
developer
as
possible,
because
obvious
the
cost
of
catching
a
bug
earlier
is
a
lot
lower
than
catching
it
later
in
the
process.
It
just
happens
to
be
the
case
that
you
know
if
the
developers
are
being
lazy
and
they
they're
not
using
defeat
then
at
least
before
production
sres
will
catch
it
right.
B
B
Okay,
so
I
think
another
like
difference
here
to
point
out
is
when
you're
sprinkling
something
in
the
code
there's
potential
of
leaving
blind
spots
in
the
code
itself.
So
if,
for
whatever
reason,
if
you
either
do
not
include
the
instrumentation
on
the
path
that
you
actually
want
to
trigger,
or
in
the
future.there
iterations
that
change
that,
if
we
should
be
able
to
catch
that,
but
if
you
miss
the
sprinkle
essentially
is
when
you're
going
to
miss
it.
If
I
understood
the
library
correctly
but
yeah,
we
had
another
question:
yeah.
E
I'm,
just
following
up
under
what
the
other
guy
said
regarding
a
staging
versus
production
data,
so
what
you
mentioned
would
not
be
possible
for
some
places
like
us.
We
work
in
banks
and
financial
institutions,
so
I
can't
just
use
the
staging
application
to
access
the
production
database.
So
what
what
other
options
do
we
have
I
mean?
Can
you
so
sorry.
D
D
J
Hey,
it's
really
interesting
so
like
in
our
case,
let's
say
we
have
like
a
hundred
micro
services
and
the
starting
kind
of
advocated,
like
sort
of
that
this
might
be
a
solution
that
could
be
used
to
test
an
entire
system
other
than
just
a
single
micro
service.
So
how
do
you
see
it
used
by
customers
today?
J
A
So
what
we've
seen
is
that
typically
Diffie
starts
off
at
the
top,
so
when,
in
terms
of
adoption
right,
a
company
will
typically
have
an
isolated
staging
environment
where
all
the
new
versions
of
the
services
are
are
being
maintained
right
and
what
they
do.
Is
they
test
the
top
level
API
right
and
through
that
top
level,
API
they're
able
to
see
some
top
level
regressions
that
ok,
there
might
be
a
regression
at
a
service.
A
That's
like
five
layers
deep,
but
it
is
indirectly
exposed
to
you
at
the
top
level
where
the
front
end
request
is
coming
in
right.
The
problem
then
becomes
that
you
know
they
need
to
trace
it
right
now.
You
know
that
there
is
a
regression,
but
you
don't
know
where
it
is
because
you
know
it's
you.
The
fee
is
not
able
to
see
beyond
the
API
boundary.
So
then
we
see
the
you
know
other
teams
downstream
starting
to
adopt
a
fee
right.
A
A
Is
that,
like
every
service
across
the
stack
ends
up
having
you
know,
I
directly
feel
air
so
that
they're
able
to
see
you
know
whether
it
was
their
regression
or
whether
it
was
another
team's
regression,
but
the
the
fact
whether
there
is
a
regression
or
not
can
be
determined
at
the
top
level?
If
you
want
to
know
where
exactly
it
is
in
the
stack,
then
you
have
to
go
instrument
all
of
those
services
with
if
you
yeah,
okay,
cheers.
K
A
K
So
do
you
kind
of
have
some
kind
of
annotations
at
the
code
level
as
well,
that
you
suggest
recommend
for
people
to
start
adding
test
cases
when
there
are
new
API
is
coming
in
as
well,
because
from
a
different
point
of
view,
any
new
API
is
essentially
different
from
the
previous
production
side.
Right
right.
A
Right
so
with
new
API
is,
unfortunately
we're
not
able
to
offer
any
coverage
because
the
first
time
a
new
API
gets
introduced
in
the
previous
version.
That
is
in
production
right
now.
There
is
no
traffic
for
that
API
right,
and
so
you
don't
have
any
traffic.
You
don't
and
you
don't
have
a
baseline
to
compare
against
right
so
for
new
API
is,
unfortunately
we're
not
able
to
provide
coverage,
and
that's
where
you
know
you
end
up
having
to
do
the
traditional
manual.
You
know
integration,
testing,
unit
testing
and
all
of
that.
Okay.
Thank
you.