►
From YouTube: Production review apps
Description
Discussion between Alessio Caiazza and Marin Jankovski (August 2020) on AB testing, and options for production review apps
A
Thanks,
okay,
so
we
are
talking
about.
How
can
we
do
something
like
a
b
testing
or
just
allow
ux
to
easily
test
and
experiment
with
new
workflows
of
any
kind?
So
one
suggestion
that
I
had
is
that
we
have
a
percentage
of
actor
and
one
of
the
actor
is
the
user,
so,
except
for
anonymous
user.
That
at
the
moment,
are
they
are
just
one
user,
so
everyone
that
is
logged
out
is
just
the
same
user,
but
it
can
be.
We
can
implement
this
distinction.
Also,
there
we
have
the
percentage
of
user.
A
It
has
a
nice
property,
that's
so
nice,
which
is,
if
we
don't
change
the
percentage
of
the
rollout.
We
still
get
the
same
people,
so
it
is
not
random,
based
it's
not
25
of
requests
but,
as
let's
say,
the
first
25
percent
of
user
is
not
really.
If
there's
a
cs
involved,
there's
some
hashing.
But
the
point
is
that
if
you
are
in
a
bucket,
let's
say
you
are
in
the
first
five
percent
of
user,
then
you
get
every
feature
flag
that
has
percentage
of
vector
above
five.
A
B
B
B
A
A
It's
this
I
mean,
I
don't
know
how
hard
it
will
be
at
the
infrastructure
level.
I
think
it's,
but
you
don't
have
to
do
anything
at
infrastructure
level,
because
it
becomes
an
application
problem,
because
the
application
code
reads
the
cookie.
So
it's
not
like.
Can
it's
like
cannery
in
terms
of
how
you
configure
it
so
opt-in
opt-out?
But
it's
not
like
cannery
in
terms
of
infrastructure,
because
infrastructures
doesn't
read
it,
it
just
sent
you
to
the
application
forward.
A
A
A
A
It's
the
problem
that
I
see
here
is
that
we
have
to
inject
the
cookie
so
that
you
get
marked,
as
you
are
part
of
the
25
percent
yeah.
So
get
cookie
and
you
and
so
you,
you
run
the
experiment.
Otherwise
you
have
this
again
the
mixing
situation,
where
maybe
a
certain
point.
You
are
marked
as
part
of
the
experiment
and
then
you're
no
longer
part
of
it,
and
I
will
probably
consider
having
some
kind
of
let's
say
time
out
to
the
experiment,
because
it's
really
hard
to
say
give
me
25
of
users.
A
What
does
it
mean
of
all
the
users?
So
you
just
just
you,
do
the
feature
flag
like
the
actor
and
say
if
you
are
in
25
right,
set
the
cookie
and
then
move
on
or
is
more
about
in
three
hour
times.
We
want
to
reach
1000
user
so
that
we
test
it,
and
so
we
know
how
many
we
marks
and
then
we
stop
and
we
disregard
the
cookie.
A
A
A
There
are
problems
like
how
can
we
clean
up
the
experiment?
Things
like
that
because
you
get
this
cookie
set
and
then
it
it's
it's
on
the
client
right.
So
maybe
you
just
do
one.
You
just
do
login
you
get
the
cookie
set
because
you
are
in
the
percentage
and
then
you
do
nothing.
You
just
log
out
and
get
out
so
you
just
yeah.
A
B
Like
it,
it
can
be
like
we
can,
we
can
so
what
I'm
thinking
is
like
there
are
like
a
bunch
of
rules
that
will
need
to
be
set,
for
example,
as
soon
as
you
see
a
db
change
like
abort,
like
that's
you're,
not
allowed
to
do
this,
and
probably
a
bunch
of
other
things
that
are
necessary
there,
like
more
complex
workflows
like
for
example,
you
know
like
you
need
to
update
italy.
You
need
to
update
like
satellite
projects,
like
all
of
that.
B
B
It
right
like
we
gave
the
click,
we
redirected
some
traffic
to
run
the
experiment
and
if
someone
else
comes
in
and
clicks
the
button
for
the
new
old,
they
their
other
merge
requests.
We
say
there
is
an
experiment
running
in
this
merge
request
or
something
like
that
right,
like
some
sort
of
id
that
will
redirect
people
just
to
to
know
who
to
reach
out
to,
and
then
this
way
we
can
have
a
box
that
is
independently
deployed.
B
A
Term,
but
is
what,
if
you
start
writing,
let's
say
I
json
b
encoded
column
with
a
different
format,
and
this
doesn't
work
with
the
production
stage.
So
you,
you
feel
empowered
to
do
something
like
this
before
getting
merged
to
master,
because
it
seems
to
be
safe
right,
but
you
may
affect
production
database.
A
B
A
B
So
I
expect
like
in
the
next
month
to
at
least
maybe
not
remove
it
completely
from
our
code
base
and
from
our
features,
but
at
least
not
run
it
in
gitlab.com
in
any
meaningful
way
that
is
shared
between
multiple
services
and
so
on,
because
gregory
is
working
on
that
okay,
tracy's
think
jakob
is
working
on
the
removing
pages
from
from
puma
dependency
right,
like
that
interaction
goes
to
psychic
isolated,
so
that
that
that
part
is
not
going
to
be
my
concern.
My
bigger
concern
is
polluting,
redis
and
database.
B
A
A
B
A
Maybe
we
have
this:
maybe
we
have
a
configuration
in
omnibus
which
may
be
very
interesting,
which
means
that
we
don't
have
to
wow.
This
is
super
because
it
means
that
we
don't
have
to
code
the
queue
selection
in
in
the
redis
cluster
wrapper.
Basically
because
you
have
a
middleware
that,
because
we
have
these
things,
that
we
no
longer
work
directly
on
queue,
but
we
can
do
the
cpu
bounds.
You
can
do
this
expression
right.
So
there
is
a
middleware
that
converts
this
into
regular
queue,
names
yeah.
A
B
B
B
The
thing
is
the
reason
why
I'm
asking
all
of
this,
like
I
told
you
to
beginning
of
this
conversation,
is
we
want
to
be
able,
like
remember
the?
Let's
call
it
an
incident
of
designs,
the
design
feature
right
it
is.
It
was
really
in
interruptive
it
is.
It
was
really
challenging
to
do
for
both
people
using
gitlab,
but
then
also
for
the
product
team.
The
product
team
didn't
do
anything
wrong.
Their
ux
team
didn't
do
anything
wrong.
They
within
their
own
investigation
and
within
their
own
design,
were
fine
right.
B
B
Is
really
yeah
yeah,
so
you
know
allowing
something
that
is.
You
know
like
non-destructive
non-interactive
even
like
this
would
be
really
a
good
opportunity
right
like
it's.
Not
only
that
it's
a
feature
for
you.
It's
feature
flag
becomes
complicated
when
you
have
like
tens
of
mrs,
like
it's
really
hard
to
keep
up
with
it.
B
But
if
you
start
with
designing
things
a
bit
differently
and
saying
all
right,
you
know
like
let's
just
roll
this
out,
deploy
it
somewhere
divert
some
of
the
traffic
like
like
you're,
suggesting
right
like
before
merging
everything
in
you
could
theoretically
do
that.
Much
like
you'll,
theoretically
get
more
valuable
feedback
and
you
don't
have
to
wait
for
kubernetes
for
and
even
when
kubernetes
arrives,
it's
not
going
to
be
trivial
for
us
to
do
this
easily.
B
B
A
B
B
A
A
And
this
has
also
another
nice
improvement
in
my
view,
which
is
this
pushes
through
the
idea
that
it's
better
for
you
to
merge
to
merge
database
changes
ahead
of
time
because
think
about
the
let's
think
about
the
design
upload
feature,
just
as
an
example
right
design.
Upload
feature
is
something
that
could
have
been
experimented
in
this
way,
because
you
want
to
tweak
something
and
just,
but
you
can
there's.
Obviously
some
sidekick
node
running
the
transformation.
There
is
back
end
to
sword
information,
there's
a
lot
of
things
there.
B
B
B
Let's
go
back
difference
in
this
case
would
be
you
don't
merge
everything
yet
or
you
prepare
the
territory
and
now
for
the
final
piece,
you
can
actually
experiment
on
the
go
there
right,
like
you,
tweak
the
interface
multiple
times
deploy,
deploy,
deploy
again,
get
some
traffic,
get
the
feedback,
get
some
traffic
get
feedback,
and
then
only
when
you
are
okay,
you
then
merge
things
and
then
you
can
decide
whether
you
want
to
play
with
the
future
flag
for
everyone
or
not.
But
up
until
that
point
you
have
much
faster
cycle
to
iterate
on
right.
B
A
B
A
A
B
Circumvent
the
whole
review
cycle,
the
whole
waiting
time.
That's
the
difference,
and
I'm
not
saying
that
I
want
to
circumvent
the
review
cycle
because
I
think
it's
not
working
or
blah
blah
blah.
It
is
because,
if
we
put
all
the
constraints
in
then
the
review
cycle
doesn't
need
for
this
type
of
information.
It's
not
strictly
necessary.
A
B
To
stage
where
you
want
to
ship
it
right,
but
when
you
want
to
get
the
data
back
from
your
experiment,
while
you're
working
on
it
the
two
hour
wait
time
is
fine
or
three
hour
wait
time.
Even
if
it's
four
hour
wait
time,
that's
a
half
a
day
cycle
in
which
you
can
experiment
and
not
wait
for
a
reviewer
yeah.
B
B
To
rubber
duck
yeah,
no,
I
wish
this
was
really
simpler.
It's
not,
but
I
think
there
are
a
lot
of
problems.
I
see
with
this
a
lot
of
problems,
but
then
this
is
a
sure
way
for
us
to
reduce
some
of
these
cycles
in
a
controlled
way
and
then
also
enforce
well
the
thing
that
we
need
to
enforce
anyway,
and
that
is
like
more
methodical
thinking
about
data
structure,
but
then
we
also
have
an
encouragement
of
why
we
are
doing
this.
For
this
specific
case.
A
Given
that
we
touched
this
topic,
the
the
real
challenge
that
I
see
now
and
this
and
this
the
major
source
of
pushback
in
the
thing
that
I'm
doing,
is
that
now
that
we
have
better
tooling
around
deployment
and
we
are
closer
to
continuous
delivery.
We
have
very
short
development
and
deployment
cycle.