►
From YouTube: Workshop: How to Use Powerful Feature Flags and Controlled Rollouts - Asa Schachar & Justina Nguyen
Description
Workshop: How to Use Powerful Feature Flags and Controlled Rollouts - Asa Schachar & Justina Nguyen, Optimizely
We'll focus on the beginning of the product experimentation journey -- feature management and feature rollouts. When used strategically and thoughtfully, feature flags can be powerful tools in mitigating risk in your development cycle. Feature flags give us a high degree of control over the features we release — but what ensures we have a high degree of control over our feature flags?
In this hands-on technical workshop, attendees will learn how to strategically create feature flags in multiple development environments and roll them out to a subset of their customers. We'll talk briefly about the possibilities feature flags open up, and then describe how to use best practices of visibility and accountability to align those different flags into a cohesive feature flagging system.
A
So
how
many
of
you
are
familiar
with
feature
flags?
You
can
raise
your
hands
cool.
How
many
of
you
use
feature
flags
at
your
current
company?
Okay,
I!
Do
okay,
okay,
it's
all
going
down!
You
can
do
that
at
scale
in
this
workshop,
so
feature
flags
enable
you
to
turn
things
on
and
off.
If
you
think
of
it.
At
a
very
simplistic
level,
if
this
feature
flag
is
on
show
this
user
X.
A
If
this
feature
flag
is
off,
show
user
Y
and
it
really
just
empowers
you
to
really
control
when
someone
sees
something
and
who
and
who
it
is,
that's
seeing
that
feature,
and
so
another
concept
within
feature
Flags
is
this
idea
of
controlled
rollouts,
or
some
of
you
may
know
it
as
canary
releases,
so
maybe
you're
building
this
feature
on
your
local
development
environment?
And
you
know,
maybe
you
want
your
PMS
to
test
that
feature
now
going
through
acceptance
criteria.
A
So
you
move
that
to
staging
then
after
you're
feeling
comfortable
with
its
performance
and
how
it's
performing,
then
you
want,
to,
you
know,
show
the
future
to
your
beta
users.
Maybe
it's
your
top
customers
who
get
early
access
to
that
feature,
so
only
a
small
subset
of
that
you
of
your
audience
base
is
seeing
that
feature.
Then
finally,
you're
feeling
more
and
more
confident
over
the
weeks
as
you're.
Releasing
that
feature
out
to
say
30
percent
of
your
audience,
50%
75%,
then.
Ultimately,
everyone
sees
that
feature.
A
If
that's
what
you
want,
but
we've
gotten
this
question
a
few
times
of
how
do
feature
Flags
fit
into
CI
CD,
continuous
integration
or
continuous
delivery,
/
deployment,
and
so
we
kind
of
made
this
chart
to
show
you
where
feature
Flags
fit
into
every
part
of
the
software
development
lifecycle
and
so
starting
with
building
and
testing.
It
really
saves
you
time
to
PI
things
behind
feature
flags
so
that
you
can
still
continue
working
on
them
without
getting
into
a
messy.
A
You
merge
conflicts
and
then,
once
you
get
into
the
deploy
part,
you
could
actually
de
couple
the
deployment
and
release
process
by
implementing
a
feature
flag,
really,
saving
you
a
lot
of
time
and
as
most
companies
like
maintaining
track
on
the
deployment
schedule
and
then
lastly,
but
most
importantly,
the
released
part.
You
want
to
empower
teams
to
manage
their
own
releases
and
what
means
is
we've
heard
engineers
say
we
want
to
focus
on
shipping
the
code.
We
don't
want
to
worry
about.
Who
sees
that
feature
when
they
see
it?
A
The
other
thing
I
want
to
talk
about
is
the
delivery
process.
Here
by
decoupling
deployment
from
release,
you
can
actually
have
more
flexibility
and
control
over
your
development
process,
something
that
we
just
went
through
so
Optimizely.
A
lot
of
our
retail
customers
saw
a
big
surge
in
traffic
during
say:
Black
Friday,
Cyber
Monday,
which
means
a
big
surge
in
traffic
for
us.
So
what
we
did
was
we
issued
a
deployment
breeze,
but
that
doesn't
necessarily
mean
that
we
froze
all
development
process
you'd.
A
Actually,
we
still
continued
to
pulling
behind
feature
flags,
so
we
didn't
actually
cause
any
incidences,
because
that's
a
pretty
big
prime
time
for
us
and
our
customer.
So
we
want
to
prevent
incidences
like
that
from
occurring
and
then,
when
the
release
process
here,
how
we
like
to
think
about
it
is
a
deployment,
is
a
technical
process
and
then
our
release
is
a
business
decision.
So
when
a
customer
sees
that
is
up
to
like
the
product,
team
or
sales
and
the
quality
of
the
code
and
when
it
gets
shipped
is
up
to
the
technical
side.
A
So
you
can
actually
control
and
have
everyone
be
able
to
see
that
visibly
with
feature
flags
and
then
the
other
thing
to
note
here
is
with
feature
Flags.
You
can
test
it
out
on
like
smaller
percentages
at
a
time
and
can
roll
that
back
so
something
we
do
that
Optimizely
is.
We
have
something
called
a
wall
of
work
and
every
week,
if
we're
deploying
any
feature,
we
make
it
to
that.
A
The
epic
owner
gets
up
and
says:
oh
we've
deployed
it
to
five
percent
of
our
audience,
ten
percent
of
our
audience
week
over
week,
and
then
sometimes
if
we
see
that
thirty
percent
of
our
audience
we
bump
into
a
bug,
we
actually
roll
that
back,
so
that
70
percent
of
our
audience
never
even
saw
that
bug
to
begin
with,
and
then.
Lastly,
I
recently
went
to
a
DevOps
conference
and
it
was
really
interesting
hearing
everyone
say
different
times
that
they've
been
paged
like
if
it's
like
2:00
a.m.
3:00
a.m.
4:00
a.m.
A
but
here
with
future
Flags.
You
actually
have
much
more
control
over
these.
Like
the
times
that
you're
being
paged,
you
can
actually
prevent
those
incidences
from
occurring,
something
that
we
have
that
Optimizely
is
within.
You
can
connect
feature
flags
to
your
tickets
and
we
have
actually
a
list
of
like
what
you
can
do
with
each
flag
and
it
specifically
says
like
this
feature,
can
be
rolled
back.
A
Do
it
this
way,
and
so
it's
pretty
organized
and
you
can
see
like
what
you
should
be
doing
with
each
feature
flag
and
how
you
can
address
those
issues
and
the
other
thing
with
roll
backs
is
you're
not
dependent
on
someone
else's
downtime.
So
if
you
know
like
you,
have
some
third-party
JavaScript
or
some
third-party
vendor
installed
and
they're
having
downtime,
that
doesn't
mean
that
you
have
to
have
that
you
can
just
roll
everything.
A
That's
attached
to
that
feature,
flag
back
and
then
the
last
thing
is
I
went
to
a
talk
yesterday
where
they
talked
about
where
the
speaker
mentioned,
they
fixed
something
in
production,
but
they
didn't
realize
they
broke
this
other
thing
and
then
they
fixed
another
thing
and
it
broke
three
other
things
with
feature
flags.
You
can
actually
prevent
that
from
happening
by
tying
each
of
those
features
behind
a
feature
flag
and
fixing
those
things
in
isolation
without
making
those
dependencies.
A
B
Great,
so,
as
you
get
your
phone
out,
some
phones,
you
can
pull
up
the
camera
app
and
just
point
it
at
the
QR
code,
if
you're
not
familiar
with
scanning,
and
it
should
show
the
URL
or
you
can
type
in
the
URL.
That's
next
to
that
code.
What
you
should
see
when
you
scan
that
QR
code
is
right
here,
you
should
see
kind
of
an
empty
little
moon.
Raise
your
hand.
If
you
see
the
moon
alright,
so
people
are
seeing
the
moon.
B
So
what
we
did
was
we
actually
put
this
little
astronaut
behind
a
feature
flag
to
kind
of
show
the
power
of
launching
a
feature
behind
a
feature
flag.
So
here
in
our
UI
we
have
this
slider
and
we
can
just
turn
the
feature
on
and
raise
your
hand
if
you
see
the
astronaut
now
nice,
so
you
see
my
astronaut,
so
everyone
saw
it.
So
that's
cool!
That's
like
a
nice
toggle,
but
let's
see
what
that
controlled.
Rollout
might
look
like
that.
Justyna
was
mentioning.
B
You
know
now,
I
put
it
2-0
percent,
so
again
keep
your
eyes
on
your
phone
and
let
them
raise
your
hand.
If
you
see
the
astronaut
so
we'll
do
25%
and
see
who
sees
it
any
hands
up,
no
up,
there's
one:
there's
one
over
there,
okay
couple
hands
up
all
right,
seeing
some
hands:
let's
go
to
50%.
You
should
keep
your
hands
up
if
you're,
seeing
the
astronaut
I'm
not
seeing
that
shot
summer,
all
right!
That's
about
half!
Let's
go
to
75
I'm,
still,
not
saying
all
right.
B
Very
neat,
so,
hopefully
you
saw
a
fun
little
demo
with
you
know
a
simple
controlled
rollout,
but
let's
take
it
even
a
step
further
for
this
pull
out
your
phone
and
scan
the
second
QR
code
and
in
the
second
QR
code,
what
we're
gonna
show
is
more
of
a
targeted
rollout
instead
of
doing
random
percentages.
Let's
be
a
little
bit
mindful
about
who
you
want
to
get
this
astronaut
feature,
so
everyone
all
set
with
QR
code
number
two.
There
should
be
nothing
cool
ever
I'm,
seeing
nods
all
right.
B
So
we
get
three
and
six
looks
like
we
got
three
and
six
very
neat
and
we
can
kind
of
go
back
to
the
audience
and
you
know
remove
that,
and
you
know
if
there
was
a
bug
and
we
realized
those
customers
didn't
actually
want
that
feature.
We
can
save
that
change
and
have
it
propagate
so
that
that
bug
is.
We've
mitigated
the
risk
of
that
rollout,
but
awesome.
So
hopefully
you
see
kind
of
the
power
that
this
tool
provides
both
with
sim
people,
kill
switches,
on/off
switches
as
well
as
targeted
rollouts.
B
B
There's
a
scan,
you
can
do
it
with
your
phone
to
get
the
URL
and
if
you
end
up
running
into
an
error
with
creating
account,
we
do
have
a
limit
on
the
number
of
IP
address
of
the
number
of
new
accounts
from
an
IP,
and
so,
if
everyone's
on
the
same
IP,
sometimes
we
can
see
errors.
Let
me
know
if
you
run
into
that
error,
but
it
seems
like
no
one
is
running
into
that.
Raise
your
hand
if
you're.
A
B
B
A
B
Cool
now
we
have
our
free
rollouts
account.
So
the
next
step-
and
let
me
know
if
anyone
raise
your
hand
if
they're
having
any
troubles
with
setting
it
up
cool,
so
everyone's
got
an
account.
Let's
go
to
the
next
step
of
the
workshop,
which
is
to
create
an
Express
application.
How
many
people
in
the
audience
use
Express
for
their
work?
Nice
good
amount,
anyone
not
using
Express.
B
What?
What
do
you?
What
do
you
use
them
back
there
happy
cool
any
other
other
frameworks.
People
are
using
out
there
in
a
house,
well,
very
cool,
very
cool,
maybe
it'll
open
source
one
day
and
we
can
all
use
it
based
on
that
expression
seems
like
no,
okay,
so
will
be
creating
an
express
application.
The
easiest
way
to
do
this
is
with
this
Express
application.
B
Generator
and
I
wanted
to
ask
the
framework
question,
because,
although
this
workshop
is
an
express,
you
can
use
rollouts
in
any
platform
but
we'll
be
using
Express
middleware
to
make
it
easy
to
install
so
in
the
Express
generator.
You
can
use
this
command,
Express,
View,
pug
and
then
the
name
of
your
application
so
go
over
to
a
terminal
and
I'll
name.
This
application,
no
js'
interactive,
and
that
was
super
fast
and
then
the
next
steps
to
follow
after
creating
it
is
just
a
change
to
that
new
directory.
B
So
let's
take
this
command
to
NPM
start
and
for
those
unfamiliar
with
this
command.
Hopefully
I
can
there
go.
This
is
saying
just
show
me
the
logs
from
NJ
si
the
application
I
just
named
so
now
we're
seeing
the
logs
and
we
can
go
over
to
our
browser
and
go
to
localhost
3000
and
see
our
Express
app
cool.
B
Creating
an
account
error
yeah.
So
if
you're
getting
an
account
creation,
error,
I
would
say
you
can
either
try.
If
you
have
cellular
service
on
your
phone
to
try
to
create
an
account
with
your
phone
to
avoid
the
IP
problem
that
I
mentioned
we'll
try
to
fix
that
later.
The
other
option
is
just
to
follow
along
with
the
rest
of
the
workshop
and
get
the
account
later.
B
B
B
B
B
B
You
and
then
we'll
want
to
initialize
that
SDK,
which,
as
you
can
see
from
the
blog
post,
there's
this
initialize
method
that
we'll
call
and
I'll
go
through
each
of
the
different
options
that
will
be
providing
so
we'll
do
our
Optimizely
equals
Optimizely,
Express
SDK
dot
initialize
and
provide
some
options.
The
first
and
most
important
option
to
initializing
is
providing
the
SDK
key,
and
this
is
what
connects
your
application
with
the
Optimizely
UI.
B
How
do
we
find
this
sdk
key?
We
go
to
our
newly
create
an
account
and
for
those
who
haven't
it,
created
an
account.
You
can
use
my
asset
k
key.
I
guess,
but
when
you
go
to
settings
in
the
rollouts
account
on
the
Left,
sidebar
you'll
see
that
there's
an
environments,
tab
and
under
here
you'll
see
the
SDK
key
and
we'll
use
for
this
workshop.
The
development
SDK
key
and
we
can
copy
and
paste
that
into
vs
code.
Here.
A
A
B
Have
to
come
back
to
that,
come
back
to
that
in
a
second
okay.
So
we'll
move
on
to
the
different
options
that
are
passed
to
the
s
CK
initialized
method,
which
is
that
data
file
I
mentioned
you
get
to
control
how
frequently
our
application
is
going
to
pull
for
that
JSON
file,
which
defines
the
feature
flags
that
we
create
in
this
UI.
B
So
our
data
file
options
will
want
them
to
have
an
auto
update
value
of
true
meaning
that
we
don't
want
just
the
state
of
the
data
file
with
the
current
state
of
the
feature
flags.
We
want
it
to
continuously
update
and
then
we
want
to
give
it
an
update
interval
and
for
these
workshop
purposes,
I
recommend
you
know
updating
every
second,
but
obviously
in
a
production
application
you
may
want.
You
know
a
little
bit
longer
of
a
time,
maybe
five
minutes.
We
also.
This
is
a
polling
mechanism.
B
B
B
So
you
see
these
feature
flags
it's
empty
right
now,
because
there's
nothing
here,
but
as
we
create
feature
flags,
it
will
get
populated
and,
and
so
what
the
application
is
doing
with
this
auto-update
is
saying:
fetch
the
latest
every
try
to
fetch
and
see
if
there's
any
changes
every
second,
you
can
think
of
it
as
a
remote
config
file.
If
that
helps
any
other
questions
here,
the
last
option
that
I'll
pass
to
this
SDK
is
the
log
level.
B
Our
log,
larger
kind
of
is
noisy,
so
I'm
just
gonna
set
it
at
the
warned
level
in
case
something
is
going
wrong
cool.
So
then
the
next
step,
any
any
problems
with
initializing
SDK
any
other
questions
cool.
The
next
step
is
to
apply
the
middleware
similar
to
other
middlewares
and
Optimizely
Express,
so
we'll
do
app
use
Optimizely
dot
middleware,
which
is
the
same
kind
of
instruction.
You'll
see
in
this
blog
post.
B
Right
here,
so
you
can
copy-and-paste,
if
you
don't
want
to
type
cool.
So
now
is
a
good
time
to
start
our
application
after
saving
these
changes
and
make
sure
we
didn't
break
anything.
So,
let's
start
that
seems
to
be
working
fine
and
we
can
go
over
to
our
Express
application
and
everything
is
working,
great
cool.
So
what
what
we
did
at
this
step
when
we
said
optimize
use,
Optimizely
middleware,
we're
now
giving
every
single
route
in
our
Express
application
access
to
optimize
the
information.
B
So
now
we're
ready
to
install
a
feature
flag
and
to
do
that
we'll
go
over
to
one
of
our
routes.
So
in
the
routes
tab
of
this
auto-generated
Express
application,
there's
an
index
file
in
the
index
file
you'll
see
the
route
which
controls
what
we
see
in
the
homepage
right
now,
it's
just
rendering
the
title
Express
and
we
can
see
that
here
Express
welcome
to
Express.
B
So
what
we'll
do
is
we'll
change
that
title
and
show
a
feature
flag.
So,
as
I
mentioned
and
also
is
mentioned
in
the
blog
post,
is
the
each
request?
Object
will
get
a
new
property
Optimizely,
which
will
have
the
information
like
the
SDK
client,
which
will
provide
API
is
like
is
feature
enabled,
which
controls
whether
or
not
the
feature
is
on
or
off.
So,
let's
go
to
the
code
and
in
the
request,
there's
going
to
be
enough:
Optimizely
property
and
in
that
Optimizely
property.
B
The
second
argument
to
is
feature
enabled
is
what
we
call
a
user
ID
in
an
application
where
you
actually
have
users
powering
them.
It
would
be
the
identifier
for
that
user.
In
this
case,
you
know,
I,
don't
have
users
in
our
back-end.
You
know
this
plane,
Express
app
that
doesn't
have
a
server
so
we'll
just
hard
code,
a
user
ID
for
now
to
simulate
what
it
would
be
like
in
a
production
application.
B
Once
that
call
is
made
we
can
save
this
to
a
variable
like
is
enabled.
So
this
is
now
a
boolean
that
we'll
get
back
from
is
feature
enabled
and
what
we'll
do
is
we'll
just
change
the
message:
that's
returned
when
the
feature
is
enabled
or
not
so
I'll
just
do
a
little
turnery
here
and
say
if
the
is
enabled
is
true,
we'll
do
a
feature
on
and
we'll
get
excited
with
an
exclamation
point.
B
B
Good
great
questions
so
that
data
file,
when
you
start
the
application,
the
data
file
will
get
loaded
as
soon
as
possible
and
then
is
trying
to
keep
up
to
date.
So
then,
when
you
call
so
that
data
file
is
now
in
memory
and
application,
so
when
you
call
is
feature
enabled
it's
just
going
to
that
in-memory
file
and
determining
whether
it's
on
or
off
so
it's
not
making
any
network
requests
and
it's
not
blocking,
which
is
why
we
don't
have
to
say
like
a
weight
or
anything
like
that
question.
B
Yes
great
question,
so
if
Optimizely
is
down,
if
the
data
file
is
not,
there
is
feature
enabled
will
default
to
false.
So
that
is
one
default
behavior,
but
you
can
also,
if
Optimizely
we're
down,
provide
your
own
kind
of
own
checked
in
data
file
to
say
we
never
want
this
feature
to
be
off.
That's
one
way
to
say
you
know
we're
gonna
have
our
own
defaults
and
another
way
is
to
you
know
if
you
really
don't
trust,
optimize
the
CDN
host
the
data
file
on
your
own
services.
B
B
So
we
called
the
feature
hello
world
and
so
we'll
have
to
remember
that
as
we
go
to
our
rollouts
project
and
since
the
data
file
didn't
look
like,
it
was
quite
up
yet
I'm
going
to
use
an
existing
different
project
so
bear
with
me,
while
I
change,
my
project,
one
that
I
know,
will
work
for
demo
purposes
or
I
hope
we'll
work
up.
That
might
be
wrong.
Why
I
think
I
got
signed
out.
B
B
B
B
B
This
feature
turn
on
so
we'll
turn
it
on
for
our
staging
environment
and
save
that,
and
we
can
at
this
point
either
see
that
change
being
reflected
in
the
CDN
file
or
we
can
go
to
our
running
application
and
we
can
see
when
that
data
file
gets
updated
from
the
logs
and
hopefully
we'll
see
that
we
can
go
now
to
our
application
and
which
I
don't
think
it.
Actually
there
now
it's
updated
and
we
can
go
to
our
application
and
now
see
the
feature
on
so
now.
B
B
We
can
see.
We
can
wait
as
our
application
pulls
for
that
change
from
our
CDN
to
get
that
update
there.
It
is,
and
we
can
go
to
our
application
and
we
can
reload
this
without
any
additional
code,
deploys
and
see.
Now
the
feature
it
is
off
so
hopefully
everyone
followed
with
me:
did
everyone?
Did
anyone
get
to
the
stage
where
they
are
able
to
toggle
a
feature
flag
on
and
off
I
see
one?
Oh
wow.
Lots
of
people
awesome
great,
any
other
questions.
B
What
I
will
do
now
is
ask
myself
a
question
which
is:
how
did
we
see
that
cool,
targeted
demo
piece
so
optionally
to
the
is
feature
enabled
API
call
you
can
pass
in
attributes
and
in
those
attributes
you
know
I
passed
in
maybe
like
a
table
attribute,
but
we
can
do
something
like
you
know.
Let's
say
you
have
certain
users
to
our
VIP
users
and.
B
Right
now
again,
we
don't
have
users
backing
our
application,
but
we
can
simulate
this
with
something
like
query
parameters.
So
what
what
I'll
do
is
say,
you
know,
like
let's
say,
is
VIP
and
if
this
is
a
true
query,
parameter,
let's
target
that
feature
to
be
on,
and
so
now
in
the
UI
you
can
go
to
audiences
to
target
people
and
you
can
see
the
special
rows
and
special
tables.
B
B
B
A
B
I
shouldn't
have
to
because
it
should
have
gotten
the
latest
data
file,
so
they
oh
good
point.
Yes
changing
the
code,
so
changes
to
Optimizely
doesn't
require
restarting
the
server
changing
the
code
does
require
restarting
the
server.
Thank
you
hard
to
demo
and
think
at
the
same
time.
So
now
when
we
were
to
load,
we
see
it's
true
and
when
we
turn
that
to
false
it's
off,
so
I
did
do
the
coding
right
cool,
and
that
is
the
workshop,
so
we're
pretty
early,
but
thanks.
Everyone
for
coming
we'll
be
here
afterward.
B
Just
as
a
summary.
Let's
go
over.
You
know
summary
of
some
things
that
we
learned.
We
created
a
an
Express
application
installed
Optimizely
seen
how
we
can
roll
out
toggles,
on/off
switches,
percentage,
rollouts
and
targeted
feature
flags.
This
application
is
free
to
use
unlimited
feature,
Flags,
unlimited
seats,
so
yeah
go,
have
fun
and
good
luck
with
better
deploys
thanks.
So
much.
A
A
B
B
In
the
you
are
in
the
page
that
you'll
see
you
can
see
the
the
different
platforms
we
support,
and
so
we
do
have
like
if
you
just
wanted
to
do,
client-side
browser
or
if
you
were
and
specifically
react.
We
have
SDKs
that
are
specific
for
those
platforms,
but
they
all
connect
to
the
same
data
file
in
same
type
of
way.
B
You,
oh
yeah,
so
I'll
show
another
cool
demo,
which
is.
You
may
have
wondered
why
my
account
has
more
stuff
in
a
sidebar,
and
you
also
might
be
wondering
like
Optimizely,
why
you're
giving
this
product
for
free
like
either
we're
really
stupid,
or
we
have
you
know,
there's
a
reason
why
we're
doing
it
and
the
reason
is
because
Optimizely
is
really
an
experimentation
platform
company.
B
In
full
sack,
which
is
our
sort
of
flagship
product,
so
here's
a
demo,
let's
say
you're
on
the
team
building,
flappy
bird
and
you
know
when
you
first
built
it,
you
didn't
build
any
pipes,
so
you're,
just
flapping
along
and
it's
kind
of
boring
and
so
you're
trying
to
figure
out
what
to
do
with
this
game.
The
next
thing
you
can
do
is
add
a
feature
like
we
did
just
now
in
the
workshop,
and
so
we
can
now
play
it
with
pipes
and
we
can
see.
B
Wow
like
Optimizely
gives
us
control
of
controlling
whether
the
features
are
on
and
off.
But
in
addition
to
that
in
the
paid
version
of
our
product,
we
also
have
variables
we
can
change
variables
like
remote
configurations.
So
let's
make
the
gravity
like
way
heavier,
and
you
can
see
this
game
like
changed
quite
a
bit
and
it's
kind
of
hard
to
do
on
this
podium
or
we
can
do
like
really
low
gravity
and
like
see
a
super
floaty
flappy
bird
and
the
point
of
this
is
not
just
greater
control.
B
But
now
you
start
to
ask
the
question
of
like
what
is
the
best
configuration
part
of
what
flappy
bird
success
was
was
like
finding
that
exact
right
configuration,
and
so
what
we
can
do
is
add
a
variation
and
create
an
experiment
to
run
on
live
traffic
and
see
given
a
different
variation.
Some
people
will
see
one
version
of
flappy
bird
and
another
will
see
another.
Can
we
see
the
metrics
coming
back
from
that
and
actually
run
an
experiment
and
see
which
got
the
most
engagement?
B
This
is
showing
kind
of
the
results
that
you
get
out
of
our
paid
product.
We
capture
results
from
the
different
variations
and
show
you
which
one
is
actually
the
winner
to
roll-out,
and
this
account
this
product
is
a
paid
version
so
and
this
this
demo
is,
you
can
play
with
it.
It's
a
knot
on
your
sheet,
but
optimized
at
github
do
a
slash,
aisa,
slash,
flappy
bird.
B
B
B
B
B
B
B
You
yes,
so
the
question
about
whether
we
can
use
this
kind
of
free
product
in
production,
the
nice
thing,
because
it
shares
the
same
SDK,
the
same
infrastructure
as
our
paid
product,
which
is
relied
on
in
production
by
very
large
customers.
You
get
the
enterprise
grade
with
the
free
product,
so
I
can
stand
confidently.
If
you,
if
you
want,
we
can
provide
kind
of
like
our
up
times
and
SLA
s
separately,
for
kind
of
the
paid
version
to
help
gain
confidence.
B
B
B
B
B
B
B
B
B
B
B
B
B
Yeah
we
would
pass
in
a
property
user-agent
get
it
from
a
request.
You
know
we
we
are,
you
know,
thinking
about
how
to
make
these
SDKs
better
and
easier.
So
you
know
if,
if
we
find
that
it's
it's
there's
a
common
attribute
that
everyone's
targeting
off
of
we
can,
but
this
is
the
way
to
do
it
right
now,.
A
B
B
B
B
B
B
Yeah,
so
I
think
there
are
different
ways
to
do
it.
One
way
we
redid
a
site-wide
redesign
and
we
wanted
to
see
what
that
redesign
looked
like
and
we
kind
of
did
the
same
thing.
I
did
with
a
query
parameter
of
like
if
this
query
parameter
is
on
yeah.
That
being
said,
if
you
don't
want,
like
a
public
query
parameter,
maybe
it's
a
query
parameter.