►
From YouTube: WebPerfWG call 2022 03 17 - A/B testing
Description
No description was provided for this meeting.
If this is YOUR meeting, an easy way to fix this is to add a description to your video, wherever mtngs.io found it (probably YouTube).
A
All
right,
hopefully,
everyone
can
see
the
screen.
Okay,
good,
so
yeah,
it's
client-side,
a
b
testing,
and
I
want
to
give
a
brief
primer
on
this
topic
is
in
case.
Everyone
is
not
up
to
speed
on
that.
I
have
a
couple
of
discussions
in
the
performance
working
group
last
year
about
this,
but
client-side.
A
A
b
testing
generally
refers
to
making
a
changes
to
a
web
application
related
to
a
b
testing
right
on
the
browser,
and
I
call
it
a
cosmetic
level
application
because
it
usually,
these
changes,
are
not
integrated
to
the
primary
source
code
and
it's
you
know
it
could
be
from
an
av
provider,
but
the
the
key
key
differentiator
of
this
method
is
actually
that
most
of
it
gets
applied
on
the
browser
using
a
client
side,
scripting
and
usually
it's
javascript
and
and
because
we
are
using
javascript.
A
It
usually
has
a
performance
penalty
associated
with
it,
which
is
why
it's
of
interest
to
this
group-
and
that
happens
because
the
script
has
to
be
fetched
from
the
client
and
passed
and
executed,
and
if
we
do
that
in
a
blocking
way
where
we
are
actually
blocking
the
rendering
that
actually
incurs
the
penalty
in
the
first
rendering
of
the
frame.
A
If
we
don't
do
that,
what
usually
happens
as
a
side
effect
is
the
unmodified
or
unav
tested.
Control
document
is
visible
to
the
user
and
they
might
start
interacting
with
it,
and
then
the
script
would
be
modifying
changes,
so
that
could
also
be
a
very
jarring
experience
to
the
user.
A
The
second
thing
is,
it
actually
serves
a
ton
of
resources
in
you
know,
implementation
right
inside
the
code
based
testing
with
the
primary
team
and
redeployment
and
also
when
the
tests
are
you
know
done
with
somebody
has
to
clean
up
all
of
that
which
actually,
so
this
method
actually
somewhat
helps
to
reduce
the
tech
depth
introduced
into
a
code
base.
That's
a
it's
very
popular
in
the
industry.
A
A
Even
for
teams
which
are
actually
doing
server
side,
a
b
testing
solving
this
the
right
way
could
actually
help
them
offload.
Some
of
the
you
know
more
cosmetic
and
not
like
you
know,
full
stack
changes
to
this
method,
thereby
saving
effort
and
costs
required
to
do
a
b
testing,
and
that
also
usually
translates
to
more
tests,
which
might
benefit
which
might
be
beneficial
for
teams
which
are
actually
doing
active
experimentation.
A
This
is
a
recap
we
have
again
like
I
mentioned
before
two
sessions
which
were
recorded
and
I've
linked
them
out
on
the
deck,
and
we
have
a
general
conscience
of
the.
We
want
to
do
the
same
outcome
but
minus
the
performance
penalty,
if
possible.
A
The
the
key
thing
that
I
found
was
like
there
are
a
few
ideas
and
directions
emerged
out
of
that
call
and
they
were,
like
you
know,
kind
of
a
common
theme
and
some
of
those
were,
you
know:
can
we
standardize
the
language
used
for
these
transforms
and
the
other
one
was
like
could
be
used,
cdn
or
edge,
or
a
compute
capable
cdn
as
an
integration
point
which
could
be
in
the
network,
but
it
could
be
any
proxy
for
that
matter,
but
cdn
seems
like
a
great
point.
A
A
Do
we
need
this
transformation
language
to
be
javascript,
or
could
we
have
a
more
you
know
fine
granular
defined
spec,
and
that
was
basically
what
I
could
find
from
those
two
sessions
and
where
are
we
now
so
we
have
a
so
we
try
to
have
a
prototype
and
I
basically
took
some
of
those
ideas
and
explored
how
that
would
you
know
function
in
the
real
world,
so
we
have
a
out
of
that
experiment.
We
have
a
initial
spec
of
a
sort.
A
That
probably
is
enough
sketches
of
that
and
we
have
a
prototype
that
actually
shows
some
of
these
concepts
in
action
and
now
I'm
at
a
point
where
we
could
potentially
take
a
few
directions
from
here
and
that
that
makes
it
a
great
point.
You
know
point
in
time
to
actually
come
back
to
this
forum
present
the
learnings
and
maybe
you
know,
look
for
further
guidance
and
that's
the
idea
of
the
presentation
today.
A
So
into
the
details
of
it,
so
if
we
were
to
visualize
an
a
b
test
as
a
control
document
with
a
series
of
transform,
this
goes
back
to
the
same
transformation
language,
so
we're
modeling
nab
as
a
control
document
and
an
ordered
set
of
transformations,
and
these
transformations
are
expected
to
be
ad
important,
which
means
that
you
know
if
you
apply
them
multiple
times.
There
won't
be
any
unintended
side
effects,
for
example,
a
list
and
the
user
navigates
away
from
an
index
page
and
comes
back
to
the
home
page.
A
The
second
thing
is
that
we
broadly
classify
these
transformations
to
be
of
two
types.
One
is
pre-ua
and
one
is
on
ua.
A
pre-ua
refers
to
transformations
which
are
best
applied
before
the
browser
has
a
chance
to
parse
the
document.
So,
typically,
that
would
be
equivalent
of
a
server-side
transform
or
a
pre-browser
transform.
Now
some
good
examples
are,
let's
say,
swapping
out
a
style
sheet.
A
That's
best
done
before
the
browser
has
a
chance
to
parse
the
document.
The
all
newer
transforms
are
those
transforms
which
are
best
applied
on
the
browser
it
could
be
because
you're,
you
know
approaching
a
dom
element
that
was
created
using
javascript
or
client-side
single
page
applications
actually
fall
into
this.
A
lot
of
transforms
actually
fall
into
this
category.
A
A
With
that,
so
in
the
spec
that
is
linked
out
in
the
earlier
slide,
which
is
most
of
the
I
mean
actually
all
of
this
is
actually
in
the
open
source
right
now
and
linked
out.
So
we
can
examine
that
later,
but
this
is
a
very
early
draft
of
like
what
would
a
transform
look
like.
A
So,
let's
try
to
serialize
that
into
a
format
where
we
have
a
list
of
ordered
set
of
operations
or
transforms
the
first
one
is
a
flag
that
actually
defines
whether
it's
a
pre,
ua
or
only
ua,
and
because
it's
a
bit
field.
We
can
extend
this
further.
Let's
say
with
other
types
of
operations
as
well,
and
then
the
next
field
is
actually
a
selector.
A
That
is
a
typical
css
selector
that
looks
for
an
element
to
target
for
transformation
and
then
an
operation
and
a
payload
we'll
see
an
example
of
this
in
the
next
slide.
But
these
two
are
supposed
to
be
variable.
A
Based
on
what
support
we
add
into
the
spec
now
these
are
some
of
the
changes
modeled
out
in
that
spec,
for
example,
the
first
one
is
actually
to
change
the
body
background
to
a
different
color,
and
we
want
to
do
that
by
appending
a
style
tag
into
the
head
of
the
document,
and
this
is
marked
as
a
pre
ui
transform
by
this
one
so
that
it's
done
before
the
browser
passes
it,
and
this
also
is
a
potential
method
to
increase.
Cachability
of
these
transforms
that
now
we
could
cache
something
after
the
transform.
A
The
remaining
three
are
actually
client-side
or
on-uh
transforms
change.
The
you
know
title
of
the
page
to
a
different
h1
and
then
please
hold
a
text
of
a
text
box.
A
We
want
to
change
that
to
what
would
you
like
to
do
today
and-
and
these
are
actually
from
an
example
that
I'm
going
to
demo,
so
this
will
make
sense,
as
we
see
the
demo
and
the
last
one
is
actually
change
the
first
item
in
a
list
to
red-
and
these
are
done
purely
to
show
like
you
know
how
the
transformation
spec
would
look
like
for
some
of
those
common
changes.
A
Now,
with
that,
this
is
more
of
a
prototype.
Now
we
are,
you
know,
transitioning
into
a
prototype.
Where
can
we
see
this
in
example,
and
here
we're
trying
to
make
use
of
a
compute
capable
edge
and
for
the
prototype
I'm
using
a
cloudflare
worker
which
actually
proxies
the
request
to
origin
and
is
the
sequence
diagram
of
that?
A
So
the
page
request
comes
in
the
cloudflare
worker
and
it
federates
of
two
calls
out
one
to
the
origin
to
fetch
the
control
document,
and
apparently
it's
federating
an
ab
configuration
fetch
now
in
real
world
that
a
b
provider
could
be
one
of
the
a
b
testing
providers.
But
for
my
prototype,
I'm
actually
making
use
of
a
a
git
based
mechanism
so
that
it's
easier
for
me
to
deploy
changes
and
see
this
in
action
without
redeploying.
A
So
what
the
edge
compute
node,
which
is
the
pre-uaf
phase,
would
do,
is
you
know
it?
First
of
all,
it
applies
all
the
pre
ui
transforms
and
once
it
actually
gets
the
configuration
it
figures
out.
If
there's
an
a
b
test
in
progress
and
selects
the
request
into
that,
and
once
the
request
is
selected
into
a
particular
experiment,
it
would
apply
the
pre
pre-uh
transforms
and
then
packages.
A
All
the
remaining
client-side
transforms
back
into
the
response
and
sends
it
out
and
and
that
piece
of
code,
along
with
the
transforms,
is
actually
index
injected
into
the
head
of
the
document
which
we'll
see
why
it's
important
to
inject
it
into
the
head.
And
then
the
remaining
work
happens
on
the
browser
and
all
on.
Ui
transforms
are
applied
on
the
browser,
and
this
is
that
piece
of
code
that
I
talked
about,
which
is
basically
enabling
us
to
apply
these
own
ui
transforms
in
a
performance
and
the
key
to
doing
that
is
actually
one.
A
The
script
is
injected
into
the
head
of
the
document
in
the
industry
today.
It's
basically
both
approaches
are
sort
of
recommended
and
some
providers
are
recommend
to
apply
them
to
the
head.
Some
actually
pass
the
body,
but
in
this
prototype
we
are
actually
injecting
that
into
the
head
and
it
uses
a
mutation
observer
and
I've
linked
out
the
code
right
on
the
deck
it.
A
It
attaches
a
mutation
observer
that
listens
to
all
the
changes
on
the
dom,
and
that's
the
only
blocking
line
in
that
prototype
I
mean,
except
for
the
pricing
of
the
you
know
the
script
itself
once
that
is
done,
the
rest
of
the
work
happens
asynchronously
as
the
browser
is
creating
the
document
as
part
of
parsing
and
also
when
javascript
is
making
changes
to
the
dom.
A
Now
each
matching
element
that
you
know
in
the
selected,
whatever
css
selector,
is
actually
matching
us.
It's
finding
the
mutation
records
it
actually
up.
You
know
passes
all
of
them
through
the
supply
transform
function
and
these
transform
functions
are
currently
javascript
and
which
is
one
of
the
things
that
I
want
to
open
for
debate.
Whether
that
looks
good
or
do
you
know,
proceed
towards
the
most
finely
grained
spec.
A
This
is
the
general
architecture
and
sort
of
a
diagram
of
how
the
prototype
is
put
together.
So
we
have
cloudflare
edgeworker,
which
is
actually
doing
intermediary
work
and
internally
cloudflare
uses
a
html
rewriter,
which
is
based
on
low
latency
html
pricing
code
and
to
mimic
an
api
provider
or
an
ab
provider.
I'm
reusing
github
guest,
just
as
a
configuration
provider,
and
you
can
see
that
later
in
the
prototype.
What
that
looks
like.
A
So
to
the
demo,
so
sorry
for
creating
a
demo,
I
actually
chose
to
do
mvc,
which
is
a
react-based
to-do
program
and-
and
that
actually
looks
pretty
much
like
this-
and
this
is
being
sourced
from
the
actual
to
do
mvc
website
and
because
so
what
it
allows
to
do
is
again
first
of
all
architecturally.
This
is
a
single
page
application,
so
it
all
boots
up
on
the
client,
this
very
minimal
server
footprint.
A
It
allows
to
create
items
right
and
second
item,
all
right,
that's
pretty
much
the
application.
So
what
we
want
to
do
here
we
want
so
this
title
of
the
document
is
created
in
javascript,
along
with
the
rest
of
the
application.
So
we
want
to
change
some
of
these
things
which
are
created
in
javascript.
A
So
in
order
to
make
that
happen,
I
created
a
a
test
configuration
and
this
is
basically
the
github
just
that
actually
is
powering
this
a
b
test.
So
we
can
see
that
we
are.
A
You
know
fetching
document
from
two
mvc
examples
react
we're
creating
two
variants
here
and
both
of
equal
distribution,
where
the
control
actually
is
injected
with
couple
of
headers
to
make
the
page
functional,
because
this
is
being
sourced
from
a
different
domain
and
it
actually
is
very
similar
to
a
marketing
team
applying
a
b
test,
although
in
reality
it
won't
be
this
bad
because
the
team
would
have
the
same
domain
at
least,
but
here
as
a
prototype,
I'm
actually
hosting
an
entirely
different
domain,
so
which
requires
me
to
put
a
base
tag
so
that
the
relative
urls
continue
working
and
also
I
want
to
inject
a
canonical
url
so
that
the
prototype
doesn't
become.
A
You
know,
indexed
as
a
duplicate,
and
then
this
is
the
test
configuration
all.
Those
changes
that
we
mentioned,
which
is
we
are
injecting
canonical
here
again
base
tag
as
well,
but
these
are
actually
the
a
b
tests.
Like
you
know,
we
want
to
change
the
background
color
to
a
page.
A
We
also
want
to
change
the
heading,
and
you
can
see
that
this
is
actually
javascript
which
actually
gives
the
ultimate
flexibility.
But
there
are
two
problems
here:
one
the
language
between
the
pre-ua
and
on
ua
is
not
the
same,
and
that
was
done
deliberately
in
the
prototype
so
that
we
could
figure
out
what
is
the
best
direction
I
mean
we
could
unify
them,
except
you
know,
cdns
might
not
be
providing
arbitrary
javascript
execution
on
the
workers,
for
example
anyway.
So
back
to
that
test,
we
have
three
changes
to
be
made.
A
The
title
has
to
be
changed
to
a
b
test.
H1
the
placeholder
text
would
need
to
be
changed
to
what
would
you
like
to
do
today?
I
think
in
the
control.
It
actually
is
what
needs
to
be
done,
and
then
we
have
a
more
of
a
ongoing
test,
which
means
that,
as
folks
are
adding
to-do
items,
we
want
to
keep
the
first
item
color
to
be
read.
A
So
if,
if
somebody
deletes
the
first
item,
the
second
item
becomes
the
first
item
and
we
want
that
to
be
color
to
it
and
that's
the
ideal
method
of
doing
this
is
probably
using
css
on
the
page.
But
we
chose
this
as
an
example,
so
that
you
know
we
can
show
that
you
know
the
mutation
observer
based
approach
actually
continues
to
monitor
for
changes,
and
you
know
reapplies
them,
and
this
is
very
analogous
to
what
the
a
b
testing
industry
does
today
so
anyway,
the
demo
is
so.
A
I
created
this
a
worker
which
is
deployed
on
cloudflare
right
now,
and
we
are
picking
that
particular
example
that
we
discussed
about,
which
is
just
fetching
with
just
directly
from
the
github,
and
I
created
a
folds
parameters
so
that
we
can
force
us
ourselves
into
one
of
these
externally.
If
you
don't
use
that
it
would
automatically
select
one,
you
know
either
controller
test
for
you
based
on
a
random
function.
A
So
in
order
to
test
that,
let's
try,
so
we
can
see
that
we
just
apply
force
equal
to
one
now,
you
are
in
the
experiment
and
we
can
inspect.
First
of
all.
The
first
thing
to
notice
is
that
we
didn't
block
the
page.
There
is
no
flash
of
unav
tested
content
that
showed
up
and
the
way
it
work
is,
you
can
see
the
style
tag
was
injected
directly
on
the
preview
face,
so
even
before
the
javascript
booted
up.
A
This
line
is
already
there
that
pre-ua
also
injected
the
script
tag,
which
is
the
small
piece
of
code
that
actually
hosts
the
mutation
observer
and
then
the
serialized
javascript
functions
which
are
applied
right
now
as
an
ongoing
basis.
Now,
if
I
close
this
and
add
items,
one
see
that
item
one
actually
turned
out
to
be
right,
close
that
we
want
in
the
a
b
test
item
two.
If
we
happen
to
remove
the
first
item,
item
two
is
applied.
That
change
is
applied
on
an
ongoing
basis.
A
A
A
It
might
actually
have
a
bit
more
computation
to
do,
because
you
know
the
nutrition
based
approach,
but
for
the
most
part,
this
is
doing
much
better
than
what
we
are
able
to
see
in
the
industry
right
now
and
we'll
see
why
that
is
able
to
do
and
which
part
of
this
actually
makes
it
possible,
and
if
we
see
the
wpt
comparison,
which
is
linked
out
here,
most
of
the
performance
metrics
are
actually
on
par.
I
mean
there
is
a
little
bit
of
a
noise,
that's
created
from
wpt,
so
one
of
them
could
be.
A
You
know
either
faster
or
slower,
but
in
repeated
tests,
at
least
with
the
simple
test:
it's
not
showing
much
of
a
performance,
so
that
makes
it
a
very
promising
approach,
at
least
from
a
start.
Now.
What
are
some
of
the
key
lessons
here
in
this
the
edge
and
the
cdn
based
approach,
and
that's
one
of
the
reason
why
it's
able
to
perform
fast
because
of
the
parallel
federation
of
control
and
the
a
b
provider
thing?
A
If
we
were
to
do
that
in
the
browser
as
it
is
done
today,
it
is
still
somewhat
sequential.
So
this
is
sort
of
a
unique
approach.
A
To
I
mean
it
could
be
any
type
of
proxy,
but
as
long
as
we
place,
something
on
the
network
and
cdn
seems
like
a
great
place,
at
least
for
this
prototype,
that's
one
of
the
key
things
that
actually
makes
it
faster
and
second
thing:
is
these
transforms
being
available
prior
to
rendering
the
body
is
actually
key
to
making
the
document
performant,
which
actually
means
that
the
head
injection
is
what
we
should
aim
for.
A
The
other
thing
I
learned
is
cdns:
don't
allow
arbitrary
js
execution,
so
that
was
one
of
the
reasons
why
the
language
looks
different
between
pre
ui
and
only
ideally
from
a
spec
standpoint.
We
would
want
the
same
language
across
both,
and
that
makes
it
very
portable.
So
if
one
place
cannot
apply,
it
can
be
transported
to
the
other.
A
Now
that
brings
us
to
the
next
steps.
Where
do
we
go
from
here?
We
could
we
have
a
couple
of
options,
one
we
could
standardize
the
transforms
further
and
away
from
javascript,
because
a
lot
of
feedback
I
received
during
the
prototyping
was
that
we
don't
like
javascript
being
on
the
because
we're
integrating
a
third-party
javascript
directly
onto
the
page.
A
A
There
was
another
option
that
came
up
in
the
february
2021
session
we
had,
which
was
basically
using
mutation
records
as
one
of
the
means
of
standardizing
it
and
that's
actually
a
viable
possibility.
A
A
We
could
solve
some
of
that
problem
by
moving
the
applicator
implementation
into
the
browser
directly.
So
that's
one
option
in
front
of
us
also,
like
you
know
what
is
the
right
balance
between
flexibility
and
security,
what
that's
one
of
the
things
that
I'm
looking
forward
to
like
as
a
feedback
on?
Where
do
we
want
to
go
with
that?
A
The
other
option
is
that
the
edge
the
intermediate
player
that
we
introduced,
we
could
move
that
line
close
to
the
browser
or
close
to
the
origin.
So
what
does
it
mean
by
origin?
It
could
be
a
web
server
plug-in.
It
could
be
a
connect
middleware
that
actually
works
for
an
express
server.
It
could
be.
It
could
be
something
very
you
know
attached
to
the
origin.
It
doesn't
have
to
be
a
syrian,
because
cdn
is
also
kind
of
asking
for
a
architecture
change
in
at
least
one
time.
A
That
is
a
newer
development,
which
is
we
are
introducing
a
there
is
a
spec
in
progress
and
I've
linked
out
to
the
details
of
that
a
blocking
equal
to
render
specification,
that's
coming
out
in
the
browsers,
and
that
could
potentially
help
us
to
deploy
the
same
thing,
maybe
without
the
edge
or
maybe
a
combination
of
edge
of
that,
but
that
would
block
the
page
rendering
but
still
allow
document
pricing.
So
even
with
the
current
ab
implementation
in
the
industry,
adopting
this
particular
blocking
equal
to
render
would
be
a
significant
benefit.
A
I
think
that
that's
pretty
much
all
I
have
so
mostly
looking
for
you
know,
guidance
on
you
know:
do
we
want
to
see
more
things
in
this
prototype
or
which
direction
we
should
proceed
after
this?
Thank
you.