►
From YouTube: Keynote - Motivations and Incentives: An Evidence-Based Approach to Community Management
Description
Speaker: Jana Gallus
Organization: UCLA
Jana Gallus is an Assistant Professor of Strategy and Behavioral Decision Making at UCLA's Anderson School of Management. Her research interests lie in behavioral economics, innovation and strategy, with a focus on non-financial incentives and their effects on motivation and performance. She runs field experiments testing the effects of award schemes and other non-financial incentives for innovation with organizations and crowd-based communities including, among others, Wikipedia, NASA, and international code collaboration platforms.
A
I'll
be
talking
about
what
could
be
called
two
of
the
thorniest
challenges
in
community
management.
The
first
one
is
diversity,
and
inclusion
and
Zahida
mentioned
that
in
her
talk
also
this
morning
earlier
this
morning,
these
are
just
some
illustrative
figures
from
open
source,
but
also
from
Wikipedia,
showing
that
less
than
10
percent
identify
as
male
and
I
just
picked.
Gender.
Here's
one
example:
the
comparison
to
Wikipedia
is
interesting
because
Wikipedia
is
not
just
about
code.
In
fact,
even
the
different
topics
you
may
be
working
on
are
not
male
type.
A
The
readership
is
gender
balanced,
and
yet
you
observe
this
stark
pattern
of
a
predominantly
male
self-identified
group
of
editors,
and
so
that's
one.
The
first
challenge
I'll
be
talking
about
today
and
asking
what
community
management
practices
can
we
use
and
what?
What
do
we
actually
know
about
their
causal
effects?
Do
they
move
the
needle
in
during
participation,
a
more
equal
participation,
but
also
importantly,
inclusion,
because
this
these
figures
only
show
who's
in
the
room.
They
don't
yet
talk
about
who's,
actually
talking
up
while
being
in
the
room,
and
it
turns
out.
A
There
is
another
barrier
here
and
I
have
some
a
new
paper
where
we
look
at
Wikipedia
talk,
page
discussions,
and
we
actually
find
that
when
you
look
at
the
Edit
level,
an
even
lower
percentage
of
edits
come
from
from
females.
So
this
is
then
an
issue
of
inclusion.
I'll
be
talking
about
that.
The
second
challenge
I'll
be
addressing
in
today's
keynote
is
how
to
support
people's
motivations
and
in
particular,
that's
another
topic
that
actually
the
heada
also
mentioned
in
today's.
This
morning's
talk
is
retention.
A
How
do
you
retain
talent
and
I'll
be
talking
about
different
experiments,
one
particularly
about
that
I
ran
with
the
community
of
Wikipedians
because
they
face,
as
you
can
see
in
this
graph.
That's
one
of
the
most
well
known
graphs
on
this,
which
is
the
the
number
of
active
editors
in
the
community,
and
you
can
see
it's
been
declining
since
about
2007.
So
then
the
question
is
in
these
contexts:
which
tools
do
we
have?
A
Which
policies
can
we
implement
in
order
to
sustain
people's
motivation
and
keep
them
at
bay
and
keep
them
actively
involved
in
your
community?
Now
why
this
is
interesting
and
also
for
me
from
a
research
perspective.
Why
this
is
particularly
interesting
is
because,
in
these
contexts
of
open
communities,
we
don't
really
oftentimes,
have
the
incentives
that
we
know
quite
a
bit
about
which
is
mostly
monetary
incentives.
Why?
Well?
A
On
the
one
hand,
of
course
there
is
a
budget
constraint,
it
would
be
unthinkable
and
sheer
cost
prohibitive
to
pay
everybody
who's,
contributing
to
your
project,
oftentimes,
not
always
but
oftentimes,
even
if
you
are
able
for
some
reason,
you
have
the
funds
in
Wikipedia's
case.
That
would
be
unthinkable,
but
even
if
you
are
able
to
pay
so
that
budget
constraint
is
not
as
binding
in
your
case,
there
are
still
two
remaining
problems,
which
is
one
many
of
the
tasks
that
your
projects
need
are
complex.
A
It
goes
way
beyond
lines
of
code,
as
we
all
know,
and
so
problem
when
important
tasks
such
as
pro-social
behavior
cannot
be
measured
or
measured
accurately
or
as
well,
cannot
be
measured
as
well
as
other
dimensions.
Is
that
as
soon
as
you
start
introducing
high-powered
incentives
that
are
based
on
these
measurable
dimensions,
effort
provision
on
the
less
measurable
dimensions
is
being
jeopardized.
So
you
end
up
in
a
world
of
multitasking.
We
call
it
multitasking
where
people
just
focus
on
whether
whatever
is
being
incentivized
so
effectively.
You
get
what
you
pay
for.
A
So
people
who
used
to
do
a
given
task,
let's
say
used
to
play
the
violin
because
they
really
enjoyed
it
once
you
start
giving
incentives
and
tying
incentives
to
each
and
every
piece
they
play.
They
start
may
start
losing
interest
in
the
actual
underlying
task,
and
just
you
know,
their
motivation
shifts
from
an
intrinsic
motivation
to
an
extrinsic
motivation
and
as
soon
as
you
discontinue
the
incentives,
you
will
no
longer
see
effort
provision
coming
forth.
Another
form
of
motivation
that
risks
being
undermined,
in
particular
in
contexts
such
as
open
source
or
other
pro-social
activities.
A
I
mean
we're.
Pro-Social
activities
are
even
more
crucial
here.
Pro-Social
motivations
is,
of
course,
image.
Motivation
crowding
out
as
well.
People
see
that
you're
getting
paid
for
whatever
you
do
now.
It's
no
longer
clear
why
you
actually
engage
in
that
activity.
Maybe
you
just
wanted
to
volunteer
and
also
you
know,
benefit
from
that
positive
image
among
the
community,
but
now
you're
getting
paid,
and
so
it's
no
longer
really
clear.
A
What
am
I
talking
about
I'm
talking
about
recognition,
and
this
is
actually
what
I've
been
focusing
on
and
in
yeah
I
think
that
is
the
major
focus
of
my
research,
which
is
recognition
in
the
forms
of
awards.
You
can
see
I
just
picked.
Some
examples
you
all
know
awards
are
everywhere.
These
examples
illustrate
that's
not
just
a
phenomenon
in
the
US,
where
it's
very
they
are
ubiquitous
also
in
other
countries
and
actually
even
historically
awards
have
always
been
used.
Think
of
volition,
donor
in
France
and
many
other
countries.
A
You
also
see
they
take
different
forms.
We
don't
quite
understand
what
matters
which
different
dimensions
are
most
important.
Should
they
be
very
public
or
is
it
okay
to
deliver
recognition
in
a
more
private
form?
Those
are
all
questions
that
we
really
need
to
address
because
they
have
important
implications,
and
here
I
also,
of
course
highlight
two
examples:
just
two
from
the
online
community,
the
Mozilla
open
badges
project,
as
well
as
the
CIA
best
practices
badges.
A
So
you
can
see
this
recognition
and
awards
can
be
given
to
individuals,
but
also
to
projects
and
whole
organizations
right
and
the
question
now
is:
what
effect
does
this
actually
have?
Do
we
just
see
the
best
and
the
most
motivating
getting
the
awards
and
getting
the
recognition,
and
then
they
just
stay
the
most
active,
the
most
motivated,
so
the
award
had
no
effect
in
fact
near
what
could
have
even
backfired
and
reduced
their
motivation
and
they
might
still
I'll
perform
the
rest
right.
A
So
this
could
just
be
that
the
best
get
awards
and
they
just
stay
the
best.
We
don't
know
whether
the
recognition
overall
had
an
actual
causal
impact
on
their
subsequent
behavior
or
whether
it
has
even
an
impact
on
others
who
did
not
yet
receive
recognition,
and
that's
what
I'd
be
talking
about
today
as
an
example.
But
this
really
also
stands
for
how
you
can
think
about.
Other
community
management
practices
and
really
try
to
address
and
empirically
analyze
their
causal
effects.
A
A
Experiment
that
I
just
mentioned
Wikipedia
has
a
really
important
problem
with
newcomer
retention,
as
many
of
course,
communities
do
for
various
intriguing
reasons
that
I'm
happy
to
talk
about
offline,
but
not
in
this
talk
today
and
then
the
question
then,
is:
what
can
what
which
tools
do
we
have
to
support
people's
motivations
and,
in
particular,
to
keep
newcomers
engaged
and
active?
And
so
in
order
to
address
that
question
I
reached
out
to
the
community
of
Wikipedians
and
then
worked
with
several?
A
We
fantastic
people
from
the
community
from
the
in
particular
the
Swiss
community
of
Wikipedians
on
established
and
establishing
an
award
scheme
which
is
actually
at
a
national
level,
almost
and
figuring
out
what
the
causal
effects
are
of
such
a
newcomer
award
on
subsequent
motivation
and
behavior.
So
this
is
the
award
template
that
we
then
posted
on
the
newcomers.
Talk
pages
for
those
of
you
who
know
Wikipedia,
I'm
sure.
A
Also
it
contains
the
names
of
the
reputable
community
members
behind
the
award
scheme
because,
of
course,
with
recognition,
it's
important
sociologists
will
tell
you.
You
need
a
source
of
esteem
of
prestige
of
status
in
order
to
confer
status
right,
and
so
that's.
Why
also
one
other
reason
why
this
collaboration
with
the
community
was
at
the
heart
of
this
problem
of
this
project
and,
in
fact,
also
in
many
other
regards
was
really
a
fundamental
piece
stone
of
that
experiment.
Now,
in
order
to
address
cause
and
effect,
what
did
I
do
each
month?
A
I
would
scrape
the
list
of
the
previous
months
newcomers
and
apply
some
rigorous
rules
that
we
had
developed
together
in
order
to
filter
out
vandals,
for
example,
but
otherwise
keeping
the
barrier
quite
low
because
we're
talking
about
newcomers
right,
so
we
didn't
want
to
apply
any
criteria.
A
lot
along
the
lines
of
the
person
should
have
made
a
hundred
edits
because
it's
not
even
clear
that
quantity.
Here's
what
you
want
right.
So
it's
also
quality
you're.
A
Looking
for
and
since
those
are
newcomers
we
could
keep
the
various
relatively
low
and
don't
apply
to
many
criterias
and
from
that
final
pool
of
newcomers
each
month-
and
this
is
now
the
core
to
experimentation
each
month,
I
would
randomly
Alec
150
into
the
treatment
group
which
would
receive
the
award
and
the
remaining
150
roughly
each
month
into
the
control
group.
So
there's
no
way
in
which
those
two
groups
differ.
Other
than
one
group
ran
by
chance
happened
to
receive
the
award
and
the
other
didn't.
A
A
A
Some
people
like
to
see
some
academics
like
to
say
there
are
no
real-world
implications
from
this
award
well,
but
that's
not
true
right,
because
people
care
tremendously
about
even
pseudonyms
in
this
online
community,
and
you
see,
although
there
are
no
career
implications,
this
award
increased
the
retention
rate
in
the
subsequent
month
by
20%,
and
that
effect
is
not
only
substantively
significant
but
also
statistically
highly
significance.
Is
you
can
see
that
there?
It
has
a
p-value
of
point
O
and
then
there,
of
course,
many
other
analysis
that
I
did
in
this
paper
to
look
at.
A
You
know
how
it
impacted
different
forms
of
contributions,
for
example
tedious
maintenance
tasks.
So
did
this
award
also
increase
the
rate
of
newcomers
who
would
engage
in
these
tedious
tasks?
You
know
cleaning
up
behind
others.
Yes,
it
did.
What
about
this
is
just
the
the
subsequent
month?
What
about
the
following
year?
Well,
because
the
revision
process
is
in
academic
journals,
take
a
while
I
had
the
benefit
that
I
was
able
to
then
also
look
at
the
long
term,
treatment
effect,
persistence
and
here,
just
looking
very
simply
at
the
quarters
at
the
five
quarters
for
quarters.
A
Following
the
initial
award
we
stole,
you
can
see
that
again
the
retention
rate
in
the
treatment
group,
which
is
the
first
column
or
in
the
treatment
column
and
the
control
group
significantly
differed,
and
that
treatment
effect
continued
to
be
statistically
significant.
Actually
for
the
four
quarters
following
the
initial
award
pistola,
and
then
it
just
wasn't
statistically
significant
anymore.
A
That's
actually
the
very
purpose
of
awards
right.
These
external
effects
on
third
parties
as
well,
so
the
implications
from
this
first
study,
the
the
major
takeaways
of
what
I've
shown
you
now
is
that
yes,
purely
symbolic
awards
by
that
I
mean
they
don't
have
any
career,
material
or
career.
Related
implications
can
be
used
to
sustainably
motivate
contributors.
A
New
cut
new
contributors
in
this
case,
who
don't
even
yet
identify
with
that
online
community
as
much
they've
just
started,
had
their
first
edit
to
an
article,
the
pre
in
the
previous
month,
and
then
I
also
find
significant
positive
effects
on
doing
tedious
maintenance
tasks,
which
is,
of
course,
really
valuable
for
any
community.
And
then
they
also,
if
I,
look
at
what
are
the
mechanisms.
What
are
possible
reasons?
Why
I
observe
that
positive
effect?
There
are
two
that
I
want
to
point
out
here,
which
is,
on
the
one
hand,
confidence.
A
The
award
seems
to
have
increased
these
newcomers
confidence
to
make
valuable
contributions,
and
this
is
just
one
excerpt
from
an
award
recipient
who
posted
that
on
a
public
one
of
the
public
pages
on
Wikipedia,
others
would
go
on
and
think
their
mentors
for
their
great
mentorship
chips.
So
you
see
the
positive
spillovers.
They
are
also
in
a
qualitative
sense
here,
and
the
other
reason
why
awards
can
have
positive
effects
is
because
they
foster
identification
with
the
community.
A
You've
been
sort
of
labeled
as
a
community
as
a
valuable
community
member,
the
either
wise
Award
recipient
and
part
as
as
a
Wikipedian
right,
and
so
that
creates
the
sense
of
identity,
which
can
be
an
important
force
and
well
linking
individuals
to
projects
and
to
organizations
more
broadly,
that's
why
many
firms,
of
course,
also
try
to
use
awards,
although
they
come
at
the
risk
of
your
talent
being
poached
away
by
competitors.
Of
course.
A
Second,
let
me
talk,
take
a
break
and
talk
about
the
diversity
and
inclusion
challenge
that
I
was
referring
to
earlier.
Thank
you,
and
here
this
is
not
drawing
just
to
show
you
the
the
different
range,
the
different
types
of
experiments.
We
can
also
run
in
order
to
to
inform
our
practical
problems
that
we're
facing
in
the
field.
A
This
will
draw
on
lab
experimental
data,
but
the
problem
is
a
very
is
one
in
you
know
that
you
observe
in
practice,
which
is
that
oftentimes
teams
miss
out
on
high-quality
contributions
from
some
of
their
smartest
members,
and
that
is
even
though
those
smartest
members
are
sitting
around
the
table
and
are
among
the
co-founders,
are
sitting
on
the
board.
Are
in
your
seminar,
room
or
in
your
corporate
board
meeting,
and
yet
they
don't
speak
up,
and
so
there's
a
literature
on
knowledge
transfer
that
has
focused
largely
on
you
know,
motivational
reasons.
A
People
may
not
want
to
share
their
knowledge.
They
may
want
to
privately
benefit
from
their
knowledge
here.
In
this
paper,
we
actually
addressing
a
different
mechanism
or
psychological
channel.
If
you
want
to
call
it
such
which
is
self
stereotyping,
and
that
is
rooted
and
beliefs
in
people's
beliefs.
That's
why
it's
important
to
take
it
to
the
lab,
because
you
can
oftentimes
not
observe
those
beliefs.
So
this
is
not
a
story
about
motivation
and
people
wanting
to
benefit
from
their
knowledge
privately.
A
This
is
also
not
about
letting
the
knowledge,
because,
though,
we
actually
observe
people's
ability
and
we
see
the
high
ability
types
don't
speak
up.
This
is
also
not
about
social
discrimination
being
discriminated
by
others,
although
I
say
that
may
come
on
top.
Of
course,
it's
also
not
about
a
general
under
confidence,
but
rather
here,
in
this
case
it
is
about
not
speaking
up
because
you
don't
fit
the
stereotype
of
that
field
or
of
that
task.
Specifically.
What
I'm
talking
about
is
women
working
in
STEM
fields,
for
example,
who
don't
fit
the
stereotype
of
that
field?
A
You
can
think
of
code,
of
course,
as
well
and
sadly
I'm
laughing,
but
it's
of
course
not
it's
a
serious
issue.
So
what
we
then
do
is
we.
We
try,
we
first
document,
and
that
has
actually
been
shown
by
great
research,
previously
the
self
stereotyping
problem
relating
to
other
tasks,
and
we
then
ask
this
question
about
which
policies
we
can
use
in
order
to
correct
people's
beliefs
and
make
high
ability
types
as
we
call
them
because
of
the
lab
experiment,
high
ability,
individuals,
more
confident
to
speak
up
and
so
again,
focusing
on
recognition.
A
We
experimental
II
very
whether
that
recognition
is
just
in
the
form
of
private
feedback.
Only
the
individual
sees
it
only
the
best
performer
sees
it
is
it
virtual,
which
is,
of
course
inspired
by
the
Wikipedia
project
and
by
so
many
online
collaborations,
where
much
of
the
collaboration
takes
place
in
virtual
rooms,
and
you
might
not
even
see
the
others
face
to
face,
and
then
the
third
arm
would
be
public
introducing
face
to
face
contact
in
the
form
of
a
ceremony.
A
A
You
see
that
comparing
the
confidence
to
speak
up
before
the
award
we
stole
and
after
the
award
pistol
and
comparing
those
to
the
blue
group
of
non
recipients
with
the
recipients,
you
see
that
the
award,
in
fact
the
recognition
of
all
forms
of
recognition,
substantially
increased
the
recipients,
confidence
to
speak
up,
and
that
is
also
if
we
control
for
ability
and
the
regressions
which
I
won't
show
here,
just
to
focus
on
the
main
findings.
And
now,
of
course,
this
major
question
does
the
form
of
recognition
matter.
A
In
particular,
if
our
interest
is
closing
the
gender
gap
and
speaking
up
here,
I
show
you
the
different
treatments
before
the
six
columns
on
the
left
is
before
the
treatment
pretreatment
and
then
after
the
treatment.
You
see
that
we
actually
in
the
ceremony,
we
close
the
gender
gap
in
speaking
up.
So
male
males
and
females
have
at
least
statistically
indistinguishable
speak
up
confidence
to
speak
up,
and
that
is
just
in
the
ceremony
treatment.
The
others
increase
the
confidence
to
speak
up,
but
they
do
nothing
in
terms
of
closing
the
gender
gap.
A
That
recognition
increases
the
confidence
to
contribute
overall,
but
that,
in
fact,
in
fact,
for
the
gender
gap,
this
the
form
of
recognition
matters
and
that's
now
one
of
the
new
avenues
of
research
on
awards
also,
which
is
no
longer
to
document
what
the
effects
of
awards
are
writ
large,
but
to
find
out.
How
do?
How
can
we
design
recognition
schemes
so
that
they
are
the
most
effective
in
addressing
the
problem
that
we
want
to
address
with
them
right,
and
so,
in
this
case,
the
public
ceremony?
A
It's
there's
value
to
staging
a
public
ceremony,
and
why
is
that?
Well,
we
have
some
exploratory
and
I
reflect
that
exploratory
research,
where
we
look
at.
Why
why
we
find
that
differential,
gender
effect
and
the
closing
of
the
gender
gap
and
the
public
treatment?
And
it
seems
that,
on
the
one
hand,
being
seen
by
the
audience
as
a
high
ability
type
increases
women's
confidence
to
speak
up
in
this
gender
atypical
domain,
but
also
seeing
the
audience
increases
trust
in
the
signal.
A
So
it's
not
like
something
that
they
gave
everybody,
but
actually
I
can
see
that
I'm
among
the
few
who
got
the
award
and
there's
the
the
other
audience
of
non
recipients.
So
it's
also
trust
a
matter
of
trust
in
this
signal
now,
I've
talked
about
the
benefits
of
recognition.
I
want
to
end
with
pointing
out
there
limitations
as
well,
which
I
will
also
use
as
a
plea
for
empirical
investigations
trying
to
set
up,
in
particular
experiments
to
really
see
what
whether
your
incentive
scheme
has
the
intended
effects.
A
So
this
now
will
be
as
a
project
where
we
with
collaborators
at
Harvard
and
Stanford,
which
has
recently
been
published,
and
they
put
the
references
underneath
at
the
bottom,
where
we
test
two
of
the
most
prevalent
types
of
awards,
which
is,
and
in
both
cases
they
are
based
on
attendance,
which
is,
on
the
one
hand,
what
happens
if
you
promise
an
award
ex-ante,
which
is
the
prospective
award.
Can
you
incentivize
people
to
do
what
you
want
them
to
do
in
this
case?
A
Be
present
and
don't
have,
as
many
absences
ideally
have
perfect,
presents
a
perfect
attendance
or,
as
opposed
to
this
promised
and
explicit
incentive.
What
happens
if
you
recognize
them
as
ex
post
there's
some
psychological
research
from
the
70s
suggesting
that
you
know
this
exposed.
Surprise
recognition
is
the
most
effective,
so
we
tested
this.
We
put
this
out
in
a
large-scale
field,
experiment
in
the
US
and
with
more
than
15,000
participants
to
see
what
the
effects
are.
A
We
use
the
context
where
these
awards
are
as
ubiquitous
as
you
could
think,
which
is
schools
and
in
the
US
and
tested
whether
they
have
the
the
effects.
Now
let
me
show
you:
we
see
in
fact
great
a
significant
effect
in
one
of
the
treatment
groups,
which
is
this
promising
retrospective
surprise
award
group.
It's
a
statistically
significant
positive
effect
compared
to
the
control
group
and
also
actually
compared
to
the
other
award
group.
Now
the
problem
here
is
that
what
I'm,
showing
you
as
the
dependent
variable
or
on
the
y
axis,
is
actually
absences.
A
So
this
award
has
increased
the
number
of
absences
and
the
fraction
of
students
who
are
and
who
well
it
has
increased
the
number
of
absences
by
by
8%
in
the
subsequent
month,
and
that's,
of
course,
a
huge
issue.
So
here
you
see
it's
not
just
that
the
recognition
was
ineffective,
that's
been
used
and
that
is
being
used
so
broadly,
it's
actually
that
it
had
opposite
effects
from
what
he
wanted
it
to
do.
We
in
the
paper
we
of
course
also
addressed
now.
You
may
wonder
why
we
address
the
reasons.
A
I
won't
talk
about
them
too
much
to
create
some
curiosity
and
motivate
you
to
look
at
the
paper,
but
this
is
what
the
basic
message
I
want
to
get
across
here.
Is
that
even
though
there's
this
intuitive
appeal
that
recognition
is
is
great
because
it's
oftentimes
cost-free,
it
can
only
motivate
people.
Well,
that's
actually
not
true.
It
can
also
backfire,
and
here
we've
shown
this
in
the
rigorous
field
experiment.
Some
of
the
reasons
why
recognition
can
backfire
and
that's
something
to
keep
in
mind.
A
Also,
if
you
we've
had
a
conversation
earlier
about
rankings
and
you
may
think
about
ranking
individuals
or
ranking
or
entire
groups
or
organizations.
So
on
the
one
hand,
if
it's
at
the
individual
level,
you
risk
crowding
out
people's
motivation.
At
the
end
of
the
day,
it
is
an
extrinsic
mode
incentive.
Second,
you
can
just
as
similar
to
monetary
incentives,
you
can
actually
induce
strategic
gaming
and
multitasking
if
this
becomes
high-stakes,
and
you
clearly
show
these
are
the
criteria
you
have
to
fulfill.
A
A
You
probably
all
know
the
infamous
IBM
example
where
they
started:
compensating
coders
based
on
the
lines
of
code
with,
of
course,
the
the
obvious
consequences
now
in
hindsight,
so
think
about
the
tasks
that
are
less
visible
and
also
think
about,
and
there's
some
great
research
done
by
colleagues
of
mine
at
UCLA,
also
about
people
who
have
not
where
you
have
made
an
effort
in
the
past
and
you
you
may
want
to
recognize.
As
you
roll
out
a
recognition
scheme,
main
takeaways
recognition
can
be
used
even
if
it's
purely
symbolic
to
sustainably
motivate
contributors.
A
It
can
also
make
a
typical
users
more
confident
to
contribute
and
therewith
address
this
issue
of
diversity
and
inclusion.
But
there
is
no
one-size-fits-all
recognition
scheme,
and
so
you
really
have
to
take
into
account
your
specific
project
needs
and
also
the
people,
the
selection
of
people
into
your
project
who
contribute
to
your
project,
and
you
have
to
be
aware
of
possible
unintended
effects
and
set
up
the
stage
when
you
run
an
experiment
so
that
you
are
able
to
measure
those
effects.
A
So
this
is
a
plea
to
institutionalize
recognition,
design
it
wisely
and
importantly
test
so
by
using
these
experiments
that
I've
been
talking
about,
and
one
major
reason
why
I'm
particularly
excited
to
talk
to
you
here
today
is
that
I'm
sure
that
some
of
you
will
think
about
recognition,
schemes
and
other
community
practices
going
forward
and
if
you're
interested
in
testing
them
and
designing
an
experiment
around
them,
then
please
do
reach
out
and
let's
talk
about
it
also
offline
and
for
the
rest
of
the
day.
Thanks.