►
Description
No description was provided for this meeting.
If this is YOUR meeting, an easy way to fix this is to add a description to your video, wherever mtngs.io found it (probably YouTube).
A
So,
okay,
we're
back,
we
are
recording
again,
so
all
this
will
be
like
is
recorded
and
will
be
posted
online,
and
with
that
I
would
like
to
share
a
few
thoughts
I
had
on
on
the
on
that
front.
Can
you
see
my
screen.
B
A
Okay,
so
thoughts
about
client-side,
a
b
testing,
potentially
without
the
client.
So,
as
we
heard
client-side
a
b
testing
is
great.
It
enables
non-developers
to
try
out
changes
to
the
site
without
deploying
code
without
bugging
developers,
and
it
is
completely
decoupled
from
release
cycles
for
their
site's
production
code.
A
It
can
introduce
a
render
or
parser
blocking
scripts
and
in
cases
where
it
doesn't,
it
requires
a
bunch
of
anti-flicker
snippets
or
result
in
user,
visible
site
changes
that
can
be
jarring
and
more
often
than
not.
It
also
results
in
significant
javascript
payloads
to
the
site,
which,
as
we
heard,
are
often
not
cacheable,
because
the
av
testing
part
is
personalized,
can
vary,
per
user
can
vary
per
url
and
that
payload
has
to
be
downloaded
for
each
new
page.
A
So
the
question
is:
can
we
as
an
industry,
do
better
and
find
a
better
way
to
provide
the
benefits
without
the
downsides,
so
the
goals
that
I
had
in
mind
when
thinking
about
this
was
one
to
enable
code-free
experimentation
so
enable
those
experiments
without
having
to
actually
modify
the
site's
code
performance
delivery.
Obviously
so,
ideally,
we
want
the
html
to
be
transformed
before
it
reaches
the
client
whenever
possible.
A
At
the
same
time,
we
want
caching
to
be
like
we
want
better
caching
for
this
in
order
to
reduce
both
download
sizes,
as
well
as
reduce
server
load,
and
we
didn't
really
talk
a
whole
lot
about
server
side
av
experimentation,
but
when
experimenting
with
a
lot
of
server
side
like
when
running,
multiple
server-side
experiments,
we're
running
a
risk
of
exploding,
our
the
the
caching
granularity
and
forcing
our
caches
to
essentially
save
a
copy
of
the
transformed
html
per
experiment.
A
We
also,
ideally
would
like
the
solution
to
be
like
to
provide
us
with
cdn
flexibility.
A
Grishma
touched
on
like
the
edge
side
and
cdn
based
delivery
of
a
b
testing,
but
those
solutions
if
they
rely
on
a
specific
cdn
by
definition,
prevent
people
from
like
moving
from
once
again
to
the
to
the
other
or
provide
multi-cdn
solutions,
and
that
essentially
reduces
adoption,
because
not
everyone,
not
all
sites
that
use
a
certain
framework
are
on
a
certain
cdn
and
ideally
we
would
love
a
solution
that
works
for
all
cdns
or
no
cdns
at
all.
A
And
finally,
a
bonus
is
a
performance,
oriented
experimentation
right
now
there
are
many
many
front-end
techniques
and
front-end
ways
to
make
your
site
more
performant,
but
it's
very
hard
to
experiment
with
those
using
client-side
av
testing.
Andy
davies
wrote
a
post
about
running
various
performance,
related
experiments
using
a
cdn
and
edge
workers,
but
that
is
not
something
that
currently
a
b
like
client-side.
A
b
testing
frameworks
can
give
you
because
of
the
performance
impact
that
will
probably
throw
off
whatever
results.
A
You
may
get
so
think
about
this.
It
seemed
like
there
are
two
potential
places
where,
where
we
could
perform
transformations
before
they
reach
the
user,
one
is
at
the
cdn's
edge
and
the
other
is
before
the
html
leaves
the
origins
network,
but,
ideally
after
it
leaves
the
origins
caches.
In
order
to
prevent
the
caching
issues
that
I
talked
about
so
option,
number
one
would
be
to
somehow
figure
out
a
way
to
run
an
experiment
proxy
on
the
origin
network.
A
That
proxy
would
then
get
the
html
download
the
javascript
or
whatever
other
means
to
communicate
to
it.
What
transformations
this
html
needs
to
go
through
and
then
apply
those
transformations.
A
There
are
many
sub
options
here,
so,
theoretically
we
could
see
where
each
framework
implements
each
a
b
testing
framework
implements
a
proxy
that
does
that
for
them
and
encourages
their
customers
to
install
it.
A
We
could
imagine
a
common
code
base
for
the
proxy
itself
with
multiple
plugins
for
the
various
ap
testing
frameworks,
or
we
could
imagine
a
common
transformation
transformation
language
for
multiple,
a
b
testing
providers
that
a
single
proxy
implementation
can
then
apply
at
the
edge
or
not
at
the
edge
of
the
origin
network,
and
but
still
site
providers
will
have
to
manage
that
proxy,
which
is
the
main
downside
here,
every
site
that
we
would
want
to
adopt.
The
solution
we'll
have
to
install
and
maintain
that
proxy.
A
We
have
a
pest
experience,
for
example,
with
mod
paid
speed
that
showed
that
this
can
result
in
low
adoption
rates.
At
the
same
time.
This
also
has
advantages.
We
need
a
low
level
of
industry
consensus
in
order
to
prove
that
this
is
workable
and
it
re
doesn't
require
the
sites
to
be
hosted
on
a
cdn
or
covered
by
cdn.
A
The
second
option
is
for
the
cdns
to
provide
some
sort
of
a
transformation
service
where
they
could
either
provide
an
off-the-shelf
per
framework,
implementation
for
edge
side
transformations
for
that
specific
framework
or
if
we
were
to
manage
to
reach
some
sort
of
a
common
transformation.
Language
they
could
provide
an
off-the-shelf
implementation
for
that
transformation,
language
that
covers
multiple
frameworks,
and
here
the
pros
are
that
cdns
can
facilitate
in
a
way
facilitate
the
installation
and
in
a
way,
share
the
installation
cost
across
all
the
sites
that
will
use
that
service.
A
On
the
downside,
this
probably
required
a
higher
level
of
industry
consensus,
either
among
cdns
or
among
all
cdns
and
a
b
testing
providers,
and
it
is
likely
to
also
carry
some
per
site.
Maintenance
cost
because
or
the
cdns
are
likely
to
transfer
their
transformation
costs
to
their
customers,
and
that
can
also
impact
that
decision
and
because
of
that,
each
site
will
have
to
activate
that
solution
on
its
own
there's.
No,
it
doesn't
seem
like
a
solution
that
cdns
can
just
turn
on
overnight
and
make
the
internet
faster.
A
It's
somewhat
of
a
slower
adoption
curve.
Thinking
about
alternatives
to
that,
mostly
are
mostly
around
what
the
browser
can
do
to
help
frameworks
today.
So
browser
can't
really
impact
the
delivery,
but
it
can
provide
potentially
better
primitives.
A
So
one
thing
I
thought
about
is
what,
if
we
were
to
provide
a
render
blocking,
but
not
parser
blocking
script,
loading
primitive
so
that
instead
of
the
synchronous
script
that
is
blocking
the
parser,
as
tim
showed
earlier,
we
were
able
to
just
have
a
script
that
blocks
the
rendering
but
enables
the
parsing
to
continue.
A
I'm
also,
at
least
to
me,
it's
unclear
how
the
client-side
html
transformation
actually
happens
and
because
it
seems
like
the
script
can
deal
with
both
a
situation
where
the
html
is
already
there.
When
it's
running
and
a
situation
where
the
html
is
not
yet
there.
Maybe
we
can
provide
better
primitives
for
those
scripts
to
perform
those
client-side
html
transformations.
A
One
thing
that
came
up
is
implementing
the
network
side
solution,
but
as
part
of
a
service
worker
which
would
have
been
a
viable
option,
if
not
for
the
first
load
issue
in
which
service
workers
don't
work
on
first
load,
but
maybe
there's
we
can
think
of
different
primitives.
That
would
and
would
enable
those
client-side
html
transformations
before
the
html
reaches
the
parser
or
afterwards,
and
essentially
maybe
there
are
other
things
that
browsers
can
do.
A
That
will
be
able
to
help
us,
and
I
think
that
it's
not
necessarily
mutually
exclusive.
I
think
that
we
can
work
towards
solutions
that
cover
both
options.
A
We
can
see
what
browsers
can
do
to
improve
client
side,
and
I
think
that
from
what
we
saw
from
grishma,
even
if
we
were
to
like,
even
if
we
had
those
ed
side
and
server
side
transformations,
some
transformations
will
always
be
on
the
client
side,
because
there's
some
information
that
only
the
client
has
and
we
don't
want
to
send
to
the
server.
A
But
we
do
want
all
sites
do
want
to
modify
their
users
content
based
on
that,
so
we
will
like.
Essentially,
I
think
we
should
work
towards
both
of
those
different,
very
different
solution,
spaces,
and
with
that,
let's
brainstorm
and
yeah,
I
put
together
a
few
questions
here
that
we
can
discuss,
but
I
also
made
the
slides.
Essentially
anyone
can
comment
on
the
slide.
So
please,
if
you
have
more
questions,
you
would
like
us
to
talk
about,
please
as
add
them
as
a
comment
and
I'll.
A
I'll
put
them
through,
but
maybe
I
can
start
off
with
a
question
to
the
ab
providers
is
the
future
where
there
is
a
single
transformation
language
that
a
b
testing
frameworks
emit
and
that
can
run
either
on
the
client
side
or
on
a
generic
server
either
on
the
edge
or
the
origin
network.
Is
that
something
that
you
find
feasible?
Appealing
like?
C
So
we
actually
had
a
hack
project
associated
with
that
at
optimizely.
This
was
specifically
for
cms
so
to
have
the
advantages
of
a
server-side
testing,
but
have
the
advantage
of
client-side
testing
but
the
performance
of
server-side
testing.
So
we
created
a
language
which
cmss
can
hook
into
and
create
their
own
implementation,
but
you
still
had
the
busy
wake
editor
which
would
make
all
the
changes.
C
B
No
go
ahead
as
a
startup
language.
I
believe,
like
that.
Most
of
the
tools
are
indeed,
you
know,
have
a
set
of
primitives
that
are
equivalent
more
or
less,
and
if
that
language
would
be
something
that
the
browsers
could
you
know
implement,
then
you
know
at
least
that's
something
that
would
always
work
and
perhaps
in
many
cases
it
could
be
also
done.
A
Okay,
you
raise
your
hand
yeah.
E
Ishaan,
so
I
was
gonna
say
something
actually,
two
things,
one
which
is
very
similar
to
what
demetrius
just
said,
but
you
for
a
common
transformation
language.
You
want
it
to
work
flexibly
across
like
the
client
and
the
server
and
during
the
spa
life
cycle.
E
We
have
a
lot
of
clients
who
have
helped
with
edge
ap
testing
in
the
context
of
single
page
applications
or
jam
stack
like
things,
but
in
our
case
our
clients
are
I.t,
dev
team,
and
so
it's
a
little
easier
for
them
to
do
some
of
the
stuff
here
and
so
the
you
know,
I'm
going
through
this
I'm
hearing
you
know
we
still
also
want
a
wysiwyg
and
having
helped
a
bunch
of
clients.
E
Do
this
my
intuition
is
that
there's
not
going
to
be
one
solution
that
solves
all
the
constraints
we're
outlining
here
and
it
might
be
useful
to
try
and
prioritize
which
one
is
more
important
or
which
are
the
most
important
or
what
are
the
top
three
you
know.
Do
we
have
to
have
wysiwyg?
Do
we
have
to
have
you
know
the
ability
to
do
behavior
on
behavior
stuff?
So
that's
what
I
was
gonna
say.
A
A
Okay,
I
okay
paul.
F
Yeah
hey,
so
I
wanted
to
comment
on
the
the
part
of
your
question
of.
Is
this
crazy?
I
don't
think
it
is.
You
know
this
is
what
we
we've
been
doing
for
a
couple
years
now.
I
think
it
sort
of
goes
back
to
this
idea
of
there's
a
compromise
either
you
you
get
the
ease
of
use
and
that
comes
typically
with
a
higher
performance
cost
or
you
really
go
for
performance
and
then
you're,
typically
a
little
bit
more
on
the
on
the
developer
side
of
things.
F
But
you
know
this
experiment
that
you're
describing
we
we've
done
it.
We
still
do
it,
it's
it's
the
concept
of
a
proxy.
F
I
do
think
that
when
you
talk
about
how
you
might
manipulate
the
experience,
given
the
landscape
of
how
the
client-side
technology
is
to
me,
it
sounds
really
hard
to
rule
out
using
client-side
techniques
to
manipulate
or
create
an
experience
that
runs
in
a
browser.
I
just
don't
see
how
you
work
around
that
there's
a
part
of
how
do
you
manipulate
the
experience
before
it
gets
to
the
browser?
F
In
our
case,
you
know
it
sort
of
fundamentally
rejects
on
the
source
of
the
page
before
it
gets
to
the
browser,
and
so
that
unlocks
things
that
you
know
it's
one
thing
to
use
javascript
to
manipulate
an
already
constructed
page
dom,
is
rendered
you're,
you're
sort
of
on
top
of
that
applying
a
treatment.
It's
something
else
to
be
able
to
swap
the
js
before
it
gets
to
the
page.
F
I
think
that's
where
there's
there's
a
lot
of
game
that
can
be
done
in
terms
of
performance
while
maintaining
the
ease
of
use
of
these
this
this
language,
both
from
a
wizard
from
a
how
do
you
empower
the
the
developers?
I
think
there's
a
lot
there.
The
the
friction
we
see
is
you're
now
talking
to
it
departments
to
get
this
thing
in
place
where
the
end
user
consumer
is
not
necessarily
on
that
team
at
all
and
is
on
more
on
the
marketing
side.
A
Thank
you,
and
just
to
clarify
you
mentioned
you're,
applying
regex
on
the
javascript
before
it's
being
parsed.
Is
that
at
the
edge
at
the
origin
or
in
the
browser,
somehow
with
we're
attacks.
F
F
Those
are,
you
know,
other
interesting
scenarios
for
native
apps
and
else
so
it
it
it's
either
on-prem,
which
is,
I
think,
the
on
network
slide.
You
had
right,
that's
one
model
and
that's
right
after
origin.
Cache
apply
transformation
there,
so
that
would
be
typically
on
the
actual
container.
F
G
H
Thank
you
yeah.
I
just
wanted
to
say
I
know
some
people
were
saying,
like
there's
constraints
with
having
a
transformation
language
you
know
like
everything
being
represented
and
that
across
spas
and
so
forth,
and
I
agree-
and
I
also
wonder,
though,
if
we
couldn't
take
like
an
incremental
approach,
if
there
were
just
a
few
primitives
supported
at
it,
but
I
had
an
escape
hatch
for
running.
You
know:
custom
javascript
effectively
just
out
of
a
matter
of
being
pragmatic
for
two
different
reasons.
There
are
multiple
sides
of
support.
H
You
need
here
right
on
a
given
cms,
for
example,
also,
the
av
testing
provider
and
the
standards
group
to
have
implemented
that
given
type
of
transformation.
H
You
also
have
customers
who
will
insist
on
being
able
to
kind
of
just
paste
in
some
css
and
some
javascript,
and
you
know
taking
that
ability
away
from
them
will
result
in
them
not
adopting
this
at
all.
I
think,
and
so
if
we
had
like
an
escape
hatch
in
such
a
transform
language,
it's
like
here's,
some
arbitrary
css
and
arbitrary
javascript
that
could
be
used
to
mitigate
each
of
those
different
shortcomings
in
the
meantime,
but
pointing
them
and
steering
them
towards
a
happy
path.
H
I
A
In
terms
of
browser's
side
like
like
on
the
front
of
what
can
browsers
do
to
help,
do
y'all
have
any
ideas
beyond
what
you
know.
The
the
two
obvious
options
I
wrote
here
like
is
there
something
else
that
the
browser
can
do
to
enable
a
b
testing
frameworks
to
consume,
like
load
faster,
consume,
less
cpu
on
the
client
and
minimize,
ideally
to
zero,
the
the
cost?
A
Is
there
like
how
how
does
maybe
you
can
share
a
bit
about
like
the
current
implementations
and
how
they
operate
on
not
yet
loaded
like?
Presumably
the
the
data
was
downloaded
in
the
script,
the
script
is
blocking,
the
dom
is
not
yet
there
so
you
need
like.
Are
you
registering
mutation
observers
and
finding
the
when
dom
nodes
are
added
and
then
modifying
them
or
yes,.
C
Yeah
typically
have
it
how
we
do
it,
there's
mutation
observers,
you
register
it
and
you
also
mark
it
as
already
changed.
So
you
don't
reapply
so
there's
like
a
weird
down
manipulation
that
happens
during
that
time,
but
it's
mutation
observers
that
we
use
mostly
before
that
it
was
pulling.
J
H
I
I
have
a
sort
of
question
for
krishna.
Actually
would
are
there
additional
to
joe's
point
about
browser
support?
Are
there
additional
lifecycle
hooks
that
the
browser
could
provide?
That
would
allow
some
sort
of
like
mechanical
sympathy
here.
I
guess
so
that,
like
the
a
b
testing
snippet,
which
has
quite
a
bit
of
cpu
intensive
code,
could
execute
things
at
times
that
are
more
agreeable
to
the
loading
of
the
page.
H
Like
I
know
that
we've
we
struggled
a
lot.
You
know
with
controlling,
like
the
sequence
of
execution,
of
different
things
like
in
javascript.
Right.
Like
sometimes
you
know,
loading
in
the
head
synchronously
serves
as
something
of
a
workaround
for
not
being
able
to
get
it
in
exactly
where
we
do
need
it
right.
C
C
I
think
a
few
hooks
for
simple
things
would
be
like
I
can
think
of
tracking
for
tracking
itself.
There
should
there
could
be
a
hook
when
it's
not
necessarily
after
everything
is
done,
because
right
now,
most
cases
tracking
is
done.
Asynchronously
but
asynchronous
means
it
can
happen
at
a
very
critical
moment,
even
without
the
browsers
knowing
so
like.
Knowing
when
analytics
is
a
pretty
ever
present
thing.
So
knowing
when
analytics,
can
kick
in
and
start
its
tracking
bothering
anything
else.
That
would
be
one
thing.
A
Like
just
on
that
front,
if
we
were
because
I
agree
that
async
scripts
is
like
async
scripts
are
not
a
good
solution
to
the
loading
problem,
but
if
you
were
to
like
either
defer
those
scripts
or
even
load
them
at
a
later
point
in
time,
but
use
the
performance
apis
and
the
buffered
flag
to
give
you
what
you
need
from
the
page.
Is
that
sufficient
or
are
there
still
missing
pieces
there.
C
I
think
like
deferred
would
work,
but
deferred
is
probably
too
early
for
tracking,
even
sometimes
like
it.
It's
not
that
critical
that
it
needs
to
be
done
before
the
html.
C
So
yeah,
because
we
could
use
something
like
the
performance
apis
find
out
the
long
tasks
are
done.
It
doesn't
seem
like
a
clean
way,
especially
for
something
that
is
present
on
every
single
web
page.
There
should
be
a
very
clean
parameter
that
says
like
this
is
analytics
or
something,
and
so
the
browser
should
know
when
to
send
all
the
analytics
and
it
shouldn't
rely
on
how
a
specific
customer
implements
it
or
a
specific
vendor
implements
it.
H
J
H
Thank
you.
So
we
talked,
I
think,
probably
a
couple
years
ago
now
about
like
content,
addressable
javascript,
like
inline,
snippets
of
code
etc.
I
was
thinking
to
your
point
about,
like
you
know,
in
several
discussions
that
are
going
on
here
in
the
chat
about
proxies
and
and
kind
of
doing,
like
man-in-the-middle,
rewriting
of
the
contents
that's
being
sent
and
there's
like
a
compromised
approach
to
that
sort
of
where
the
proxy
injects
the
javascript
that
is
necessary
to
do,
especially
in
the
case
of
like
needing
spa
support.
H
Right,
like
you,
can't
do
the
transforms
yet
necessarily
to
the
page
if
it's
not
been
server
side
rendered,
but
a
problem
with
that,
obviously,
is
there's
a
some
sort
of
tipping
point
where
putting
it
in
line
is
worse
than
just.
You
know,
having
a
script
source
tag
that
loads
it
from
a
cdn.
H
I
I
think
that,
like
balances,
at
least
to
me,
like
very
opaque
and
I'm
sure
it
varies
in
all
sorts
of
conditions
about
when
it's
better
or
worse.
But
if
it
were,
you
know
cachable,
somehow
by
some
sort
of
like
attribute
in
the
script.
Tag
that
you
could
you
know,
identify
it
as
the
same
sort
of
chunk
of
code,
at
least
for
the
purposes
of
parsing
and
execution.
Obviously,
like
the
transfer,
the
bytes
themselves
couldn't
be
cached.
H
I
just
wonder
if
there's
there's
more,
that
could
be
done
here
and
it
kind
of
goes
back
to
the
same
points
about
the
browser
exposing
or
the
browser
taking
into
account.
Hence
that
you
know
javascript
content
on
the
page
provide
about
its
role
in
the
page
and
so
forth.
A
Yeah,
so
there
is
currently
a
proposal
called
web
shared
libraries
that
can
help
with
con
like
content,
addressable
content
delivery.
I
guess
the
main
problem
there
that
we
haven't
fully
solved
yet
is
the
privacy
implications
of
having
that
content
addressable
resource?
So
if
a
certain
user
has
some
libraries,
that
can
say
various
things
about
them,
and
even
if
it
doesn't
say
anything
specific,
the
collection
of
libraries
that
they
already
have
in
cache
versus
not
can
provide
a
good
fingerprinting
surface.
A
We've
been
thinking
about
that
in
terms
of,
on
the
one
hand,
trimming
the
list
of
things
that
can
be
shared
and,
at
the
same
time,
aggressively
prefetching
those
so
that,
essentially,
all
users
will
have
them
in
their
caches
to
some
extent,
but
it's
still
not
a
trivial
problem.
At
the
same
time,
I
I
hear
you
that
it's,
it
would
definitely
help
reduce
the
download
sizes
here.
H
B
B
I
think
I
saw
it
some
comments
that
it's
it's
very
few
of
them
that
support
and
that,
obviously
I
know
it's
a
fragmented
area,
but
I
think,
like
the
fact
that
there
is
not
some
sort
of
standard
for
a
b
tool
to
provide,
like
you
know,
transformation
instructions
or
that
could
help
on
one
hand
yeah
another,
because
here
another
totally
different
topic
here,
because
you
have
the
list
of
questions.
B
What
would
what
I
think
these
tools
are
looting
for
help?
Is
this
idea
of
persistence
of
experiment
variants
for
users
today?
I
think
most
tools
are
using
cookies
and
local
storage.
I
don't
know
if
that's
a
good
thing,
I
know
you
know
kind
of
raises,
raises
questions
about.
B
You
know
consenting
and
everything
there
I'm
wondering
if
it
makes
sense
for
the
browsers
to
help
in
that
area.
B
A
K
Chris,
I
see
you
next
hey
there.
I
had
two
questions,
so
the
first
one
is
that
we
keep
talking
about
a
split
between
different
responsibilities.
K
K
This
will
allow
the
groups
to
work
independently
and
not
require
client-side
javascript
to
execute,
and
then
the
second
thing
is
around
what
we're
trying
to
solve.
So
we
keep
mixing
concerns.
We've
got
people
trying
to
solve
for
single
page
applications,
people
solving
for
multi-page
applications,
people
solving
for
jam
stacks.
These
are
very
different
scenarios.
K
So
perhaps
maybe
these
solutions
wouldn't
tie
into
something
like
a
single
page
app,
which
tends
to
need
more
knowledge
about
how
the
app
is
constructed
in
the
framework,
you
could
imagine
a
solution
that
understands
how
your
react
hooks
work
and
injects
some
differential
values,
for
instance,
but
that's
pretty
complex,
whereas
the
multi-page
app
solution
would
be
more
simple
and
potentially
impact
a
very
wide
audience
at
a
much
faster
clip.
A
No,
it's
it's
very
useful
yeah.
So,
regarding
your
first
thought,
it
sounds
awfully
like,
but
the
common
transformation
language
implemented
in
the
browser
is
that
what
you're.
K
Yeah,
that's
kind
of
where
I'm
going.
The
idea
was
to
reduce
the
js
impact
and
because
it
would
be
side
channeled
through
the
main
document
request
in
a
tv,
determined
factor.
This
means
that
the
browser
could
actually
implement
it
on
c
side
rather
than
trying
to
execute
arbitrary
javascript
during
the
render
phase.
Even.
A
Yeah
so
so
the
browser
would
block
rendering
on
this
transformation
language
variant
to
avoid
flickr,
but
yeah,
then
the
execution
itself
would
be
significantly
faster
than
arbitrary,
javascript
and
then
arbitrary
javascript
can
be
added
on
top
of
that
for
things
that
cannot
be
accomplished
with
this
yeah
interesting.
Definitely
yeah.
B
Yeah
I
just
wanted
to
to
reply
back
to
christopher
about
this.
B
The
thing
the
thing,
though
there
about
something
that
works
only
on
the
server
side,
is
that
there
is
really
like
a
big
variety
of
ranges
in
client,
dynamic
sites
sps
is
extreme,
but
I
actually
had
done
a
look
up
in
a
big
sample
of
sites
recently
to
see
like
you
know,
what's
this
what's
the
state
there-
and
it
was
quite
like
sorting
like
a
lot
of
a
very
big
percentage
of
the
sites-
are
changing
their
dome
in
one
way
or
another.
B
You
know,
refining
your
card
thing,
so
there's
a
big
range
of
client,
dynamic
things
and
the
problem
from
the
av
tools
perspective
is
that
I
mean
you
know
you
can't
really
tell
the
user.
Yes,
you
can
change
the
header,
because
that
is
on
the
server
side
but
yeah.
I
know
you
can't
change
this,
because
it's
actually
and
sometimes
the
the
tools
wouldn't
even
know
how
the
content
really
came
up
there
right,
because
you
can
imagine
that
an
e-commerce
refinement
results
list
that
that's
something
that
happens
with
ajax
a
lot.
B
K
That
thing
right,
so
just
as
a
response
to
that
the
the
idea
here
was
that
that
transformation
language
is
is
baked
into
the
browser
right
and
is
actually
being
applied
on
mutations
as
well.
So
since
we're
opting
in
to
say,
I
want
the
following
changes
using
this
dsl
that
is
already
baked
into
web
specifications,
then
all
changes
that
happen
on
that
document.
Follow
that
specific
specification.
K
There's
probably
problems
with
that,
but
it
would
be
an
interesting
space
to
pursue
because
it
would
allow
the
browser
to
do
the
work
and
it
would
be
independent
of
the
the
back-end
technical
implementation
yeah.
That
sounds
great
yeah.
L
Cool
dave,
I
think
it
might
be
useful
to
describe
how
this
proposed
transformation
language
differs
from
css,
which
isn't
to
say
that
maybe
it
isn't
css,
but
if
it
isn't,
then
why
we
wouldn't
want
to
shoehorn
this
into
css,
because
we're
already
doing
things
like
media
queries
and
responsive
design,
which
is
very
similar
to,
I
think
the
kind
of
transformations
we'd
be
doing
for
a
b
testing.
K
On
that,
I'm
curious-
and
this
is
probably
ignorance,
but
I
don't
recall
a
way
for
css
to
be
able
to
change
the
attribute
of
say,
a
custom
element
and
that's
the
kind
of
thing
that
I
see
pretty
frequently
in
the
experimentation
that
runs
across
our
document.
Basis
of
about
150
billion
documents.
L
L
So
if
we
had
some
sort
of
like
cohort
attribute
that
could
be
based
on
you
know
some
identifier
stored
in
a
cookie
or
passed
as
a
header
or
local
storage
or
something
along
those
lines,
then
the
browser
you'd
you'd
saw
to
send
down
all
of
the
elements,
but
the
browser
could
choose
which
ones
they
were
going
to
display
the
same
way.
It
does
like
a
media
query,
but
just
based
on
the
value
of
a
cookie
or
whatever.
A
F
M
Yeah
I
mean
it
seems
like
it's
not
possible
to
add
a
new
button
somewhere
with
just
css
right.
That's
not
really
possible
like
if
you
wanted
to
add
a
new
banner
with
some
text
on
it.
Like
I
mean
yeah,
I
would
maybe
you
can
use
it
before,
but
then,
if
you,
what,
if
you
want
the
style
like
you
want
to
have
an
image
in
there?
You
really
can't
do
that
with
just
pure
css.
M
If
you're
the
xslt,
maybe
you
could
do
that,
but
then,
if
you
have
xslt,
that's
like
a
you
know
the
transformation
language
we're
talking
about
here
I
have
to
kind
of
drop
off
because
I
have
medical
appointment
up
too.
M
But
one
thing
I
wanted
to
add
is
that
it
seems,
though,
that
like,
in
addition
to
this
transformation
thing,
one
thing
that
we
might
be
missing
is
the
async
script
that
delays
paint
initial
painting
it
like
the
fact
that,
like
these
scripts
are
doing
this
thing
where
like
oh
like
I,
I
don't
want
to
be
synced
because
you
block
the
person,
so
we
want
to
be
async,
but
then
we
don't
want
to
paint
it
like.
Why
are
they
doing
it
manually?
M
Like
seems
like
that's
one
api
we
could
add
to
the
platform
without
you
know,
because
then
the
browser
could
decide.
Okay,
the
script's
taking
too
long.
So
that's
just
not
wait
for
the
script
and
just
display
the
content
right
in
the
case,
script
failed
to
load
like
one
of
the
examples
shown
over
like
page
complete
being
broken
to
the
user,
because
the
script
you
know
was
taking
too
long
to
load
or
like
it
never
enrolled
it.
A
Yeah
I
100
agree,
and
I
think
that
I,
the
only
thing
is
that
I
don't
think
that
async
is
the
right
primitive.
I
think
we
we
want
to
scr
like.
A
We
want
to
define
predictable
execution
points
rather
than
async,
which
is
racy
by
definition,
but
I
hundred
percent
agree
that
the
fact
that
people
nowadays
resort
to
blocking
scripts
is
a
failure.
That
seems
not
super
hard
to
correct.
M
Yeah
I
mean
I'm
not
necessarily
saying
async
is
the
right
semantics,
so
I'm
just
saying
we
need
a
sound
written.
That's
more
async
than
sync,
but
still
like
delay
is
the
painting
and
I
guess
the
default
somebody
was
saying
for
tracking
d4
is
too
early
or
something
like
that.
So
maybe
we
need
to
think
about.
You
know
what
what
the
exact
time
like
right
timing
for
that
type
of
scriptures
anyways.
I
have
to
get
out
yeah.
A
Okay,
where
are
we
on
the
queue
it
seems
like?
The
queue
is
now
empty,
so
yeah,
it
sounds
like
we
yeah
concluded
to
revive
xslt.
Is
that
it
yeah?
A
Yeah,
I
would
love,
I
would
love
to
develop
the
the
idea
of
a
transformation
language
further.
Chris,
you
mentioned
a
mutation
observer
output.
Is
that
currently
a
thing
that
I'm
not
aware
of,
or
is
that.
K
It's
the
way
that
the
tool
that
we
use
works
so
we
output
what
the
mutation
is
from
a
wysiwyg
like
editor
and
then
apply
the
mutation
observers,
apply
the
results
of
the
mutation
observer
on
the
client.
It's
it's.
I
think
there
is
a
an
interesting
path
there,
just
because
it
is
something
that
already
exists
in
the
web
platform.
I
don't
know
if
it's
necessarily
the
right
one
as
a
transformation
language.
It's
super
chatty.
A
K
Caches
as
part
of
its
experimentation
framework.
K
It
is
one
that
was
under
development
for
a
long
time
and
then
I
don't
know
what
happened
if
it
finished
or
not.
But
I
do
know
it's
in
use
by
some
people.
A
Okay,
cool.
Thank
you.
N
There
are
a
few
comments
on
the
chat,
so
there's
this
idea
that
maybe,
when
you
have
looking
script,
that
is,
like
all
things
being
considered
optional,
like
an
av
solution
or
even
like
fonts
fonts,
like
style
sheets
and
then
fonts.
I
think
it
would
be
useful
to
have
a
way,
maybe
for
developer
or
for
the
service
provider,
to
say
this
thing
is
not
supposed
to
take
more
than
xxx
milliseconds.
N
If
it
goes
over
that
it's
probably
a
better
idea
to
just
give
up
and
and
pretend
this,
this
just
doesn't
like
exist
at
all.
So
I
don't
know
if
it's
interesting,
but
that's
something
that
we
could
explore,
maybe
offline.
If
that's
something
that
people
want
to
use.
A
Yeah,
it
certainly
seems
like
an
interesting
primitive
to
have
I
so
yeah.
I
have
thoughts
on
script,
loading
that
I'm
yeah.
A
Are
not
super
baked,
but
it
seems
like
we
want
to
define,
at
least
in
my
view.
We
want
to
define
points
in
time
instead
of
like
what
developers
need
to
do
today
is
to
say
what
like.
How
is
the
script
supposed
to
be
loading?
Is
that
a
blocking
script
or
an
async
script,
or
the
first
script,
where
blocking
and
async
are
sub-optimal
and
defer?
A
It
gives
you
a
single
point
in
time,
and
I
think
what
we
need
is
more
of
a
like
more
options
for
the
defer
semantics
so
where
people
can
tell
the
browser,
we
want
this
script
to
run
like
to
block
rendering,
but
still
but
not
block
parsing.
We
want
this
script
to
execute
after
the
first
load
after
the
first
paint.
A
We
want
the
script
to
execute
when
the
fur
is
now.
We
want
the
script
to
execute
after
on
load
when
you
know
on
idle
time,
because
it's
not
super
important
but
essentially
move
from
what
semantics
to
when
semantics
and
give
people
more
more
control
over
the
when
and
then
browsers
can
have
more
flexibility
as
to
how
they're
loading
those
scripts
and
when
they're
loading
those
scripts.
A
In
order
for
them
to
be
ready,
when
the
developers
tells
the
browser
that
it
needs
to
be
ready,
but
yeah,
I
yeah,
I
think
it's
a
yeah,
it's
a
somewhat
tangent
to
the
or
yeah
having
that
will
also
enable
us
to
have
you
know,
script,
loading
that
enables
a
b
testing
loading
as
a
render
blocking
async
and
potentially
with
the
timeout
on
the
on
the
render
blocking
part,
but
yeah.
It's
something
that
I've
been
thinking
about,
that
I
need
to
develop.
A
K
I
think
there
was
one
sorry
raising
and
lowering
my
hand.
I
think
there
was
one
other
part
about
this.
That
was
not
really
covered
with
these
approaches,
which
is
the
actual
signal
of
what
variants
to
put
a
person
in
when
they're
visiting
a
page.
It
sounded
like
from
the
optimizely
and
google
optimize
approach
that
was
done
also
through
the
script.
A
I'm
not
sure
so
you're
talking
about
client-side
variants
or
server-side
variants.
K
Mostly
the
the
client-side
variant,
so
there
was
a
script
that
is
loaded
that
is
doing
dom
mutation
for
the
group
or
bucketing
of
the
user.
Is
that
how
does
if
the
browser
takes
over
that
work
and
does
it
in
some
way?
Does
it
receive
some
kind
of
standardized
information
about
what
bucket
a
person
belongs
to
for
that
visit
in
what
duration?
A
A
Presumably,
it
will
yeah,
so
I
I'm
guessing
and
like
a
b
testing
providers
can
correct
me,
but
I'm
guessing
it's
a
lot
of
that
is
cookie
based
and
then
grishma
mentioned
a
bunch
of
decisions
that
are
taken
on
the
client
that
are
not
available
on
the
server.
Is
that
correct.
C
So
I
think
it's
not
use
the
targeting
is
done
cookie
based,
but
the
decisions
themselves
which
are
tracked
are
usually
done
using
javascript
and
I
suspect
that
would
still
remain
javascript
from
what
I
understood
the
transformation
language
would
purely
be.
The
changes
applied
like
if
javascript
is
asking
change
the
color
of
this
element
to
red.
That's
the
part
that
the
transformation
language
will
be
doing,
but
the
fact
that
this
was
the
change
that
was
applied
would
still
be
in
the
control
of
the
client,
a
b
testing
vendors.
A
So
yeah
yeah,
so
the
the
goal
here
is
not
necessarily
to
eliminate
javascript
altogether,
but
to
limit
it
to
things
that
cannot
be
done
like
essentially,
ideally
eliminate
the
html
transformation
bit
from
script
and
delegate
that
to
a
separate
dsl,
but
the
the
point
that
chris
raised,
which
is
a
valid
one.
If
we
need
the
like,
potentially
we
need
client-side
code
to
decide
which
of
the
transformations
should
be
applied.
A
So
potentially
we
could
define
buckets
in
the
transformation
language
so
for
bucket
one.
These
are
the
transformations
applied
for
bucket
two.
These
are
the
transformation
applied
for
bucket
n,
etc,
and
then
the
client
side,
javascript
will
just
have
to
say
which
bucket
was
picked.
A
If
that
it
would
still
make
the
javascript
a
blocking
factor,
but
we'd
need
significantly
less
a
script
for
that,
and
this
will
only
be
something
that
will
block
the
the
decisions
that
have
to
happen
on
the
client
side.
That
cannot
happen
on
the
server.
C
Typically,
on
a
specific
page,
a
customer
has
maximum
five
experiments,
whereas
for
a
specific
project
that
the
customer
defines
they
can
have
any
number
so
like
the
application
of
changes
while
slow
is
not
the
biggest
factor
that
is
like
when
you,
when
I
I
pointed
out
some
numbers
on
edge
that
we
see
so
the
changes
applied
part
is
the
only
thing
that's
sent
to
the
edge
and
that
part
takes
on
an
average
five
milliseconds,
whereas
previously
it
was
taking
on
an
average
200.
C
I
forget:
was
it
150
milliseconds,
because
the
evaluation
logic
was
the
part
that
was
taking
the
longer
time.
That
is
specific
to
optimize
the
implementation.
It
can
be
optimized
based
on
different
factors,
but
that's
still
a
significant
portion
of
the
internal
javascript.
C
So
evaluation
logic
could
be
so
say
a
visitor
comes
from
www.google.com,
so
the
evaluation
logic
figures
out
which
url
the
vista
is
coming
from,
how
many
experiments
are
defined
for
this
and
how
many
experiments
are
valid
for
this
particular
visitor
based
on
the
url
based
on
ip,
whatever
else
the
customer
decides
to
define
and
if
the
experiment
is
valid,
how
do
you
which
variation
that
the
does
the
visitor
fall
into?
A
Okay,
and
so
that
is
the
part
that,
in
your
experience,
is
consuming
the
most
of
yeah
resources
on
the
client,
and
that
is
something
that
won't
change
in
this
particular.
C
A
And
is
there
a
way
to
delegate
that
logic
to
the
serving
infrastructure
so
make
the
decision
like
assuming
you
don't
need
you,
you
had
the
list
of
client-side
only
criteria,
but
if
people
don't
use
them,
is
there
a
way
to
delegate
all
that
evaluation
logic
to
the
server
and
decide
on
the
variant
at
serving
time
rather
than
on
the
client.
C
So,
like
performance
edge,
the
solution
that
we
have
kind
of
does
it
on
the
server
side,
even
though
it's
a
client
side
yeah,
that's
good.
So
that's
one
way
purely
server
side
has
the
negatives
of
like
not
being
visible,
not
having
the
capability,
and
you
have
to
wait
for
code.
Deploy
yeah
like
performance
edge,
falls
under
the
best
criteria
of
that.
Obviously,
it
lacks
certain
attributes,
like
local
storage,
which
is
used
for
behavioral,
and
we
have
decided
not
to
support
it.
A
Okay,
do
you
know
and
or
can
share
how
much
pushback
so
yeah?
So
essentially,
I
think
what
we're
describing
here
here
is
something
you
already
implemented
in
your
performance
edge
or
so
how
much
pushback
have
you
had
from
customers
for
not
supporting
the
client-only
evaluation
logic
in
those
scenarios.
C
C
Since
then.
The
major
pushback
is
with
respect
to
security,
which
is
always
a
concern
with
a
b
testing
software.
I
feel
and
also
like
we
don't
support
pci
currently
on
edge,
so
we
have
push
back
against
that
other
features,
essentially
not
a
whole
lot,
because
any
customer
who's
cares
about
performance
is
willing
to
make
that
change.
C
So
we
all
suggest
that,
on
the
landing
page
they
implement
performance
edge
and
on
their
subsequent
like
deeper
article
pages
and
example,
those
kind
of
pages
they
do
the
traditional
optimizely,
so
they
get
the
best
of
both
worlds.
C
Okay,
most
people
have
been
most
of
our
customers
have
been
pretty
receptive,
but
also
it
does.
We
do
charge
for
it.
So
not
all
the
customers
are
right
now
ready
for
it.
A
Yeah,
but
essentially,
if
like
what
I'm
hearing
is
that,
when
you
present
like
or
at
the
very
least,
the
the
customers
that
are
biased
towards
using
that
solution
are
willing
to
pay
the
loss
of
personalization
price
in
order
to
get
better
performance
and
yeah.
Okay.
So
that's
encouraging.
Thank
you.
A
Cool
anything
else,
because
I
think
we
added
a
bunch
of
a
bunch
of
interesting
ideas
to
the
to
the
slide.
So
thanks
all.
P
Going
back
to
melissa's
presentation,
beginning
we
talked
about
the
business
people
in
this
whole
equation
and
they're
the
people
creating
these
experiments,
but
we
should
we
need
to
have
a
or
we
need
to
persuade
testing
providers
to
expose
data
about
what
the
impact
of
their
tests
are
at
run
time
to
those
business
people
either
through
ga
or
the
run
product
they
got
so
they
so
they
know
some
of
the
what
the
decisions
or
what
actions
they're
taking
what
impact
that's
having
on
their
visitor,
because
at
the
moment
you
look
at
an
anti-flicker
snippet
and
it
runs
and
how
long
the
screen
was
blank
for
is
never
recorded
anywhere
unless
somebody
puts
in
a
bunch
of
work
to
instrument
it
manually.
P
So
there
are.
There
are
things
like
that
where
we
can
get
providers
to
help
people
make
more
informed
decisions?
I
think
the
other
thing
is
people.
People
are
doing
edge-based
a
b
testing.
At
the
moment
there
are
people
like,
I
think,
it's
love
holidays,
who
did
a
talk
at
fastly's
conference
about
how
they
were
using
edge
workers
to
do
a
b
testing
and
there's
people
doing
it
on
akamai
as
well.
It'd
just
be
interesting
to
understand
how
they're
getting
on
and
what
restrictions
they're
finding
from
that
approach.
A
Yeah,
so
if
I
try
to
summarize
your
first
point,
it's
essentially
we
need
to
surface
yeah
surface
the
implications
of
a
be
testing
decisions
to
yeah
to
the
people
who
make
those
decisions
yeah.
A
That
sounds
reasonable.
At
the
same
time,
I
think
it's
hard
to
do
in
today's
infrastructure,
where
it's.
A
I
think
hard
to
test
without
testing
together
data
without
the
a
b
testing
infrastructure
at
all.
N
I'm
wondering
if
there
could
be
an
agreement
for
like
for,
for
instance,
like
av
vendors,
to
agree
to
have
user
timing.
That
explain
what's
happening
because,
like
standardizing
like
those
moments,
could
have
the
tooling
like
surface
data
and
whatnot.
P
N
N
I
Another
thing
that
was
touched
on
a
little
bit
earlier
was
the
limitations
of
like
largest,
first
contentful,
paint
to
largest
contentful
paint
when
using
it
in
like
the
flashing
scenario,
and
maybe
we
could
take
that
back
to
the
working
group
and
try
to
see
if
you
know
we
can
evolve
those
metrics
a
little
bit
to
you
know,
help
us
out
help
help
run
providers
which
are
already
capturing
these
metrics
out
in
those
kind
of
cases
to
better
reflect
the
user
experience.
A
I
A
A
Cool
michael.
D
All
right,
yolov,
I
sensed
that
you
were
beginning
to
wrap
up
there
and
I
suspect
you
were
going
to
thank
everyone,
but
I
figured
that
risks
not
thanking
you
for
organizing
today
and
for
all
the
great
speakers
and
the
presentations
I
and
and
the
amazing
chat
between
everyone.
I
mean
I
learned
more
today
than
I
think
I
knew
on
this
topic,
so
this
was
an
excellent
use
of
time.
I
quite
enjoyed
it.
D
A
Thank
you,
yeah
is
sean.
E
Yeah
one
thing
that
I'm
that
occurs
to
me
in
terms
of
where
browser
standards
could
help
in
the
realm
of
edge-based
a
b
testing,
and
you
want
to
preserve
caching,
you
know
the
common
mechanism
that
the
server
side
will
indicate
stuff
is
through
a
cookie,
but
there's
no
good
standard
that
I'm
aware
of
you
can't
say
http
very
cookie
to
tell
the
edge
on
how
to
cache
specifically
for
a
specific
sub
cookie
within
the
entire
cookie
string,
and
that
would
make
that
type
of
splitting
a
lot
more
plausible
and
the
ability
to
have
better
cash
hit
rates
would
be
a
lot
more
achievable
in
this
kind
of
edge
experimentation.
A
Yeah
that
that's
a
good
point
yeah,
so
it
wouldn't
change
much
from
an
information
exposure
perspective,
but
it
would
make
yeah
caching
and
edge
and
server
side
handling
significantly
easier.
There
was
the
the
key
proposal
from
mark
not
again
for
a
while
that
would
have
been
able
to
do
that
over
cookies,
but
yeah,
that's
no
longer
a
thing.
So
that's
a
good
point.
Thank
you.
A
Cool
okay
yeah,
so
I
was
yeah.
As
michael
said,
I
was
yeah
starting
to
wrap
up
just
because
I
sense
that
there
is
yeah.
The
number
of
questions
starts
to
whittle
down,
but
if
anyone
has
any
just
for
like,
we
have
five
more
minutes
theoretically.
So
if
anyone
has
any
other
questions
or
things
you,
they
want
to
bring
up.
N
Maybe
I
guess
we'll
go
over
the
chat
and
the
video
to
make
sure
that
we
capture
everything,
but
this
slide
is
there
anything
missing.
I
think
there
was
a
discussion
about
holding
the
first
paint,
ideally
on
a
per
element
basis.
Maybe
that's
something
we
could
explore.
A
N
I
think
the
idea
is
like
obviously
a
rough
idea
would
be
to
have
a
way
in
javascript
to
say
I'm
about
to
paint
something,
but
I
don't
want
you
because
I
know
this
is
going
to
change
or
there
are
like
few
more
things
to
get
in
place.
First
and
doing
that
per
element
would
be
nice,
because
if
you
know
that
element
xyz
might
be
impacted
by
an
experiment,
you
would
just
like
retain
the
first
paint.
Don't
do
oh
right.
A
So
avoiding
avoid
painting
specific
elements
that
you
know
will
be
impacted
by
experimentation
yeah.
I
think
that's
a
good
way
to
put.
A
A
Yeah,
okay,
that's
interesting,
yeah
cool
yeah,
so
I
yeah
I
I
think
that,
yes,
we
will
like
there
was,
I
think,
a
parallel
track
in
the
chat
that
was
super
interesting.
I
tried
to
keep
up
with
it,
but
haven't
necessarily
succeeded.
At
the
same
time,
I
copied
everything
to
the
agenda
documents
and
nick
has
been
taking
minutes
throughout
the
whole
thing.
A
So
thank
you
nick
and
I
believe
the
plan
is
to
like
my
plan
is
to
go
over
the
recording
the
minutes
and
the
chat
and
somehow
curate
them
into
concrete
conclusions
and
then
publish
them
to
the
to
the
working
group
mailing
list.
I
think
that
the
discussion
was
extremely
useful.
I
think
that
the
like
my
conclusions
are
a
that,
the
as
a
format
it
seems
to
be
working
like.
A
I
think
that
the
despite
the
fact
that
we
weren't
all
in
a
single
room,
we
had
a
great
discussion
and
put
together
a
lot
yeah,
a
ton
of
new
ideas
that
will
enable
us
to
potentially
make
progress
here.
So
I'm
excited
about
that
and
yeah.
I
just
want
to
thank
everyone
because
the
yeah,
the
discussion
was
really
great,
and
none
of
that
would
have
been
possible
without
y'all.
So
thank
you.
A
And
yeah,
so
I
will
send.
I
know
that
not
all
of
you
one!
One
final
note:
I
know
that
not
all
of
you
are
working
group
members,
I'm
hoping
that
some
of
you
can
become
ones
if,
if
that's
at
all
of
interest
to
work
on
this,
like
on
the
solution
in
this
space,
if
it's
of
interest
to
you
feel
free
to
reach
out
to
me
or
nick
and
we'll
we'll
we'll
discuss
that
cool.
A
So
with
that
thanks
everyone
and
yeah,
I
will
send
the
summary
and
see
all
around.