►
From YouTube: WebPerfWG call 2023 03 02 - A/B testing update
Description
No description was provided for this meeting.
If this is YOUR meeting, an easy way to fix this is to add a description to your video, wherever mtngs.io found it (probably YouTube).
B
Right,
thank
you.
So
this
is
sort
of
the
second
update
about
a
b
testing
in
this
forum.
I
think
our
previous
presentation
was
actually
in
2022
March,
which
actually
marks
it
about
an
exact
one
year.
Recap
and
summary
of
updates
and
Amika
from
optimizely
will
be
co-presenting
with
me.
I
will
run
through
the
updates,
all
of
you
and
stuff,
and
then
Mika
will
take
over
from
the
demo
and
complete
the
rest
of
the
deck
now.
B
B
Related
changes
in
a
browser
and
typical
flow
is
that
the
the
page
that
would
like
to
AV
test
would
have
a
script
when
the
browser
loads,
the
script,
it
will
actually
fetch
the
a
b
testing
configuration
from
the
provider
and
the
scripts
would
actually
modify
the
Dom
and
any
make
any
changes
to
conduct
the
experiment
in
a
sort
of
a
blocking
fashion,
either
through
the
script
itself
blocking
or
depending
on
the
placement
of
the
script.
It
might
actually
hide
the
document.
B
There
are
a
couple
of
approaches
in
the
industry
that
are
followed
and
either
way,
and
once
the
changes
are
complete,
the
page
will
be
unblocked
and
that's
typically,
the
fashion
of
a
client-side,
a
b
testing.
Now
this
has
the
upside
that
this
experimentation
method
is
much
more
scalable
compared
to
server-side
experimentation,
so
which
is
the
reason
for
the
popularity
of
the
method,
a
small
scalable,
primarily
because
the
less
engineering
involvement
required,
which
makes
the
method
very
accessible
to
marketing,
Personnel
or
product
management
teams
who
doesn't
have
to
change
the
code
or
redeploy
the
application.
B
So
so
we
started
out
incubation
of
this
approach
with
a
prototype
and
a
proposal
which
was
which
is
linked
on
the
deck
and
the
link
linked
to
the
deck
itself
is
actually
in
the
agenda
document
of
this
meeting
so
and
that's
sort
of
the
recap
now.
Our
problem
statement
is
that
how
can
we
keep
all
the
good
parts
of
this
client-side
AV
testing
and
minimize
our
eliminate
the
negative
performance
in
back
in
at
bits?
B
Point
I'm,
hoping
that
client,
a
b
testing
would
be
a
a
tool
in
the
toolbox
of
any
web
engineer
or
a
web
application
so
that
smaller
tests
and
things
that
don't
require
architectural
fundamental
changes
to
the
application
that
this
would
be
a
very
viable
method
compared
to
how
widespread
it
is
already
today.
So,
hopefully
it
is
much
more
used.
B
So
we
started
with
some
initial
prototypes
that
involved
using
Edge
or
CD
and
compute
nodes
as
an
integration
endpoint,
and
that
was
the
last
year's
demo.
The
main
reason
and
the
highlight
of
this
method
is
actually
the
edge
allows,
parallel
Federation
to
origin
and
a
b
configuration
which
is
sort
of
sequential
once
you're
in
the
browser.
B
So
this
allows
us
to
reduce
the
latency
in
required
in
fetching
the
AV
configuration
it
makes
it
parallel,
and
the
second
part
of
the
solution
was
actually
standardizing
the
language
of
the
schema
by
which
we
can
describe
an
ABC
transforms,
which
today,
in
the
industry,
it's
mostly
a
combination
of
JavaScript
and
individual.
You
know
provide
the
specific
formats.
B
B
So
in
the
last
one
year
we
have
worked
together
to
standardize
about
10,
transform
operations
and,
and
those
are
detailed
in
the
external
I'm
not
going
to
go
through
that
right
now.
But
there's.
A
short
summary
here
is
that
there
are
about
nine
operations
which
are
interoperable
between
browser
and
Edge,
which
means
that
the
instruction
can
either
be
applied
on
the
compute
node
or
wherever
the
pre
UA.
B
We
call
it
pre-ua
because
it
is
the
changes
that
are
to
be
applied
prior
to
the
UA
encountering
the
page
or
they
can
be
pushed
down
to
client
and
that
decision
could
be
done
by
The
Entity.
That's
ordering
the
set
of
transforms,
so
it
could
be
the
a
b
testing
provider
or
it
could
be
somebody
curating
those
tests,
but
we
mostly
expect
the
instructions
to
be
created
by
a
system,
and
it's
not
that
human
friendly
by
Design.
B
There
is
also
a
client
only
operation,
which
is
a
custom
JS,
and
this
is
currently
limited
artificially
to
be
a
client
only
because
we
most
platforms
at
least
the
ones
we
are
prototyping
with,
which
is
cloudflare,
does
not
allow
arbitrary
JavaScript
execution.
So
this
could
change
in
future.
But
at
this
point
it's
actually
a
client
only
operation.
B
We
are
also
working
on
some
more
operations,
which
is
class
list
and
style,
attribute,
manipulations
and
also
relocating
HTML
element,
which
is
sort
of
a
corner
case,
but
it's
still
required
as
well.
You
know
optimizely
a
prior
AV
testing
experience
and
something
new
we
are
bringing
into
the
client.
B
B
Now
next
part
is
actually
a
demo
and
Mikhail
will
take
over
from
here,
but
we
thought
it
would
be
cool
to
wear
the
hat
of
a
marketing
team
and
come
up
with
a
campaign
to
be
running
on
a
real
sort
of
real
looking
site.
That
actually
is
a
back
to
work
campaign,
because
most
of
the
companies
are
actually
calling
their
employees
back
to
work.
So,
let's
promote
a
back
to
our
campaign
with
a
specific
product
to
be
injected
on
the
catalog
page
and
also
make
some
things
on
homepage.
A
Okay,
can
you
see
my
screen
all
right,
great,
so
yeah,
so
we
decided
to
use
a
site
called
umaticanbutton.us,
which
is
a
site
that
optimizely
has
built
for
kind
of
showing
off
the
functionality
of
experimentation
and
whether
it's
you
know,
client
AP
testing
using
our
snippet
functionality,
usually
right
here.
It
is
built
on
the
Shopify
platform
and
it's
intended
to
mimic
a
like
normal-ish
retail
site,
where
you
have
a
home
page
here,
landing
page
and
you
can
scroll
down
for
Featured
products
and
all
sorts
of
interesting
things
like
that
and
subscription
ability.
A
There's
like
searching
about
us.
We
have
a
Blog
and
a
catalog
page.
So
it's
navigable
here
and
you
can
filter
by
by
different
types
of
of
inventory
and
all
that
so
there's
retail
sites
are
a
big
users
of
experimentation
for
a
number
of
different
reasons
easily
for
for
experimenting,
but
also
for
personalization
types
of
reasons
like
the
back
to
work
campaign
and
so
to
show
what
we
have
done
with
the
with
our
demo.
Here
is,
for
example,
we
have
our
we.
A
We
are
going
through
the
cloudflare
worker
here
in
order
to
kind
of
front
the
attic
and
button
Us
site,
and
here
we
have
a
control
version
of
it
where
all
it
does
is
well,
it
doesn't
have
any
actual
variations
that
it's
applying
onto
the
page
and
just
returning
the
page
as
it
it
gets
a
formatic
us
and
then
returns
it.
It's
it's
navigable.
A
A
It
also
applies
the
a
change
where
we
inject
a
new
item
into
the
into
the
catalog
page
here
at
the
front
I'm
going
to
make
all
that
change
like
this
is
all
all
these
changes
are
done
on
the
client
side,
using
our
mutation
Observer
for
as
Dom
is
constructive
and
the
the
code
is.
Is
there
so
that's
that's
one
example.
A
B
A
Edge
worker
to
Federate
the
the
call
to
the
origin
use
it
as
a
we
grab,
formatican
button.us
and
at
the
same
time
we
grab
from
GitHub
the
transforms
that
we
want
to
apply
to
the
page.
We
can
do
that
in
parallel,
but
what
the
client
receives
at
the
end
is
simply
the
the
final
HTML,
along
with
the
variations
to
be
applied
in
the
in
in
scripts
themselves.
A
A
Then
this
is
an
example
of
the
actual
Transformations
that
we're
applying
on
the
page,
the
it's
a
list
of
Transformations,
where
we
have
three
of
them
that
we
we've
applied
here.
The
format
is
on
UA.
The
first
parameter
here
is
where
to
apply
it.
We
have
on
UA
versus
pre-ua,
pre-ua
being
on,
say
an
edge
worker
or
at.
A
If
you
can
handle
it
there,
the
second
parameter
is
the
selector
in
this
case,
for
the
hero
image.
On
the
page,
you
it's
it's
standard,
CSS
selectors,
the
third
parameters,
the
operation
to
execute
for
the
variation
and
the
any
any
other
parameters
are
Arguments
for
the
operation
itself.
So
in
this
case
for
the
set
attribute,
we're
changing
the
style
attribute
to
be
the
background
image
to
a
different
location
for
the
background
image
and
then
we're
also
changing
the
hero.
A
Title
we're
executing
to
custom
JavaScript,
in
this
case,
to
change
the
inner
HTML
to
be
ready
to
go
back
to
the
office.
This
could
also
be
done
through
a
different
operation,
but
we
wanted
to
show
off
the
custom
JavaScript
capability
and,
finally,
the
third
transform
that
we
have
is
selecting
the
specific
product
where
the
item
in
the
catalog
page
and
replacing
the
entire
HTML
or
H
entire
HTML
with
a
new
div.
A
We
did
a
few
performance
tests
with
the
control
versus
the
demo.
We
ran
a
few
web
page
tests,
comparisons
for
one
versus
the
other,
there's
a
little
bit
of
variance
in
the
in
the
results
themselves,
ranging
between
on
the
median
for
between
like
2.0
and
2.4
seconds,
but
overall
they
seem
to
be
pretty
comparable
on
both
the
home
page.
A
There's,
no,
no
real
performance
impacts
that
we
can
see
as
well
as
on
the
catalog
page,
and
there
are
links
to
more
to
the
web
page
test
results
down
at
the
bottom
here
if
you're
interested-
and
we
also
ran
additional
performance
benchmarks.
So
with
web
page
tests
it
usually
only
runs
three
tests.
You
know
by
default
those
three
loadings
of
the
page
and
it
runs
on
on
real
websites
or
real
devices,
which
is
excellent.
A
A
Performance
impact
in
our
results
on
the
demo
pages
yeah.
So
so
that's
where
we're
at
and
we
have
several
we.
We
have
a
lot
a
lot
of
discussions,
lots
of
things
that
we
want
to
work
on
and
improve
on
and
and
cover
in
the
next
year
or
so
so,
yeah
work
in
progress
and
what's
next
one
big
thing
is
to
improve
our
prototyping
and
performance
testing
Suite,
make
it
more
robust
and
more
consistent
and
and
and
stable
more.
A
One.
Big
thing
that
we
also
have
to
cover
is
incremental
transformation,
fetch
or
dynamically
receiving
or
obtaining
new
Transformations
as
a
visitor,
navigates
the
site
or
goes
you
know
through
through
a
site,
especially
during
the
life
cycle
of
an
of
a
single
page
application.
And
that's
that's
one
big
thing,
because
that
unlocks
a
a
large
part
of
experimentation
which
is
called
like
behavioral
experimentation,
so
that
if
a
user
goes
and
does
X
Scrolls
this
far
or
of
the
users
that
click
this
button,
for
example,
do
we
get
better
results?
A
The
we
also-
and
this
this
is
a
very
exciting
possibility,
as
well,
is
exploring
browser
native
implementation
of
the
applicator.
It's
we
are
we're
looking
into
bundling
the
actual
applicator
into
different
languages.
That
can
just
be
kind
of
you
know,
bundled
and
shipped
off
to
in
different
different
ways
and
having
that
also
implemented
in
the
browser
specifically,
would
hugely
help
in
terms
of
not
needing
to
include
that
in
the
payload
of
every
and
every
time,
you're
executing
a
an
experiment
on
a
page,
and
that
would
that
would
help
with
performance
tremendously.
A
And
finally,
we
are
so
the
the
transform
language
itself
and
applying
variations
is
is
certainly
a
you
know,
a
vital
and
crucial
component
of
running
experiments,
but
there
are
just
by
itself
it.
You
can't
just
run
an
experiment
and
avoid
and
expect
to
get
results.
You
need
additional
information
like
metric
collection
about
what
what
a
user
did
and
how
they,
what
the
results
are
from
your
experiment
in
order
to
actually
to
act
on
the
results.