►
Description
The increasing ease with which disinformation or "fake news" is produced has been recently touted as a threat to democracy. One proposed solution is to teach people how to be better able to recognize fake news, and Facebook recently announced plans to attempt to crowd source fact checking. With this in mind, we have undertaken a large-scale study where for almost three months we sent out 5 articles per day, 4 days per week, that had appeared in the past 24 hours to be fact checked by 150 people each as well as 6 professional fact checkers. We will assess both the ability of individuals and the crowd to identify the veracity of news. I will also present plans for extending the study to cover Covid-19-related news, as well as how the pandemic may have changed the ability of individuals and the crowd to identify the veracity of non-Covid-19-related news.
Joshua A. Tucker is Professor of Politics, affiliated Professor of Russian and Slavic Studies, and affiliated Professor of Data Science at New York University. He is the Director of NYU’s Jordan Center for Advanced Study of Russia, a co-Director of the NYU Social Media and Political Participation (SMaPP) laboratory, and a co-author/editor of the award-winning politics and policy blog The Monkey Cage at The Washington Post. He serves on the advisory board of the American National Election Study, the Comparative Study of Electoral Systems, and numerous academic journals, and was the co-founder and co-editor of the Journal of Experimental Political Science. His original research was on mass political behavior in post-communist countries, including voting and elections, partisanship, public opinion formation, and protest participation. More recently, he has been at the forefront of the newly emerging field of study of the relationship between social media and politics. His research in this area has included studies on the effects of network diversity on tolerance, partisan echo chambers, online hate speech, the effects of exposure to social media on political knowledge, online networks and protest, disinformation and fake news, how authoritarian regimes respond to online opposition, and Russian bots and trolls. An internationally recognized scholar, he has served as a keynote speaker for conferences in Sweden, Denmark, Italy, Brazil, the Netherlands, Russia, and the United States, and has given over 100 invited research presentations at top domestic and international universities and research centers over the past 10 years. His research has appeared in over two-dozen scholarly journals and has been supported by a wide range of philanthropic foundations, as well as the National Science Foundation. His most recent book is the co-authored Communism’s Shadow: Historical Legacies and Contemporary Political Attitudes (Princeton University Press, 2017), and he is the co-editor of the forthcoming Social Media and Democracy: The State of the Field (Cambridge University Press, 2020).
Date: 05/07/20
Presenter: Joshua Tucker
Institution: New York University
A
A
Some
of
his
work
has
actually
looked
at
trying
to
popular
rumors
about
how
people
are
affected
on
social
media
in
terms
of
political
behavior
and
checking
out
whether
or
not
that's
actually
true
he's
the
rest
of
the
Jordan
Center
for
the
Advanced
Study
of
Russia,
so
his
other
half.
If
he
knows
everything
about
Russian
Studies
he's
also
the
co-director
of
New
York
NY
use
social
media
and
political
participation
lab,
which
is
better
known
as
the
famous
math
lab
and
under
that
guys.
Not
only
has
he
done
a
lot
of
work
with
social
media
politics.
A
B
Thanks
Kathleen
thanks
so
much
for
all
of
you
for
joining
us
today,
thanks
for
being
here,
thanks
for
the
invitation
and
in
this
brand
new
world
of
sort
of
virtual
talks,
one
of
the
exciting
things
about
it
is
it's
easier
to
bring
along
your
co-authors.
So
I
will
point
out
the
two
of
the
co-authors
on
this
paper.
B
B
So
this
is
so
this
the
work
I'm
gonna
be
showing
you
today
is
part
of
a
is
part
of
a
large
project,
and
today,
I'm
gonna
be
telling
to
give
you
and
we're
sort
of
the
research
questions
that
I'm
gonna
end
the
talk
today
by
telling
you
where
the
whole
project
is
going,
the
research
questions
we're
going
to
be
talking
about
in
the
talk
today
here
are
as
follows.
First,
10
individuals
distinguish
between
true
and
false
news
in
real
time,
or
can
individuals
assess
the
veracity
of
news?
B
That's
brand-new
news
stories
that
have
just
appeared
the
way
many
people
would
go
in
encounter
news
stories.
Second,
question
research:
press
we're
gonna,
try
to
answer
today.
Are
there
large
partisan
differences
in
performance?
The
third
is
going
to
be
what
makes
people
better
and
worse
at
this
task
and
are
there
interventions
that
we
can
take?
That
would
improve
people's
performance
in
this
regard.
B
Basically,
the
study
from
what
I'm
gonna
be
showing
you
today
is
was
conducted.
The
data
gathering
Parshin
of
this
was
conducted
during
from
November
21st
2019
to
February
8th
2020.
So
we
did
this
for
basically
about
three
weeks.
Individuals
were
shown
articles
within
24
to
48
hours
of
publication
and
asked
whether
they
were
to
rank
them
as
whether
they
were
true
or
false,
or
misleading,
and
lots
of
other
questions
about
them.
B
I'll
tell
you
more
about
that
in
a
minute,
and
all
the
articles
will
also
were
also
rated
by
professional
fact,
checkers
and
we
collected
a
whole
bunch
of
covariates
about
people
in
the
process
of
doing
this.
So
we
were
asking
them
survey
questions
as
well,
so
in
terms
of
the
types
of
evaluations
of
these
news
sources,
both
the
respondents
and
the
professional
fact
checkers
answered
two
types
of
questions
to
address
the
veracity
of
these
new
sources.
B
The
first
was,
we
asked
him
a
categorical
question,
so
was
the
article
and
we
gave
him
three
categories,
true,
false
or
misleading,
or
could
not
determine.
We
then
subsequently
asked
them
if
they
said
false
or
misleading,
whether
they
thought
it
was
false
or
it
was
misleading,
but
based
on
some
pre-testing
work,
we
did.
We
combined
that
into
a
single
category
and
everything
I'm
going
to
show
you
and
the
categorical
answers
will
be,
will
have
three
we'll
have
these
three
answers
there?
B
B
Most
of
what
I'm
gonna
show
you
today
is
about
our
assessment
of
how
they
behave,
how
they
performed
on
the
categorical
category,
but
will
I'll
show
you
some,
you
know
kind
of
you
can
think
of
it
as
almost
as
robust
as
test
or
subsidiary
research
on
the
veracity
scales
as
well
over
the
course
of
the
talk.
So
the
details
of
the
study
is
that
we
ended
up
with
about
a
hundred
and
fifty
total
articles
that
appeared
within
24
hours
of
publication.
B
The
way
we
did
this
over
that
13
week
period
with
some
breaks
for
vacation
was
we
sent
out
five
articles
per
day
and
one
of
the
things
that's
innovative
about.
What
we
did
here
is
that
we
took
ourselves
out
of
the
process
of
selecting
what
articles
we
would
send
to
people
for
fact-checking.
So
what
we
did
was
we
pre
registered
a
totally
replicable
selection
process,
because
any
of
you
could
go
ahead
and
Institute
the
same
selection
process
as
well.
B
So
we
got
the
the
researcher
choosing
the
articles
out
of
the
process
and
we
the
way
we
did
this
was
we
had
five
streams
of
news,
so
these
streams
were
from
what
we
call
sort
of
mainstream
news
sources
and
three
from
what
we
called
suspect
news
sources,
the
mainstream
news
sources.
We
then
divided
into
left
news
sources
and
right
right,
more
left-wing
and
more
right-wing
mainstream
news
sources
and
the
suspect
news
sources.
B
Each
morning
we
would
select
the
most
popular
article
that
had
appeared
in
the
last
24
hours
and
then
we
would
immediately
send
that
out
to
both
our
crowds
and
our
professional
fact-checkers.
Now
we
asked
ninety
risks.
We
asked
each
of
our
professional
of
our
crowd
members
to
evaluate
three
or
Nichole's,
so
we
went
out
to
so.
Each
article
ended
up
being
being
checked
by
90
people
from
the
crowd,
so
we
have
a
crowd
of
90
people
for
each
day,
so
we
went
out
to
150
people
per
day.
B
We
gave
them
three
articles,
each
that
gave
us
90
respondents
per
articles.
Then
we
used
our
fact
checkers
to
provide
the
ground
truth
of
whether
or
not
what
the
correct
answer
was
to
ranking
the
article
as
false
myths,
the
false
misleading,
true
or
could
not
determine,
and
on
top
of
this,
we
also
conducted
three
supplemental
experimental
studies,
one
of
which
was
a
two-by-two
study.
B
So
we
were
going
to
look
at
treatments
where,
for
two
weeks
we
ran
these
supplemental
experimental
studies,
where
the
main
study
functioned
as
a
control
group,
and
then
we
had
these
experimental
studies
which
I'll
tell
you
about
layered
on
top
of
it.
So
that's
basically
what
we
were
doing.
Here's
a
nice
graphic
to
give
you
a
sort
of
sense
of
this
that
I
use
for
a
more
public
facing
talk,
but
basically
we
have
the
five
new
streams
coming
in
every
day.
B
These
are
you
know
from
our
main
stream
left
wing
right
wing,
and
then
these
are
from
our
more
suspect
news,
sources,
right,
left
and
and
can't
tell
they
would
come
in
to
us
and
we
would
simultaneously
send
them
out
to
the
fact-checkers
end
of
the
crowd,
all
right.
So
our
experimental
surveys,
we
did
asked
essentially
three
and
we
did
four
experiments
in
three
studies.
B
So
we
did
this
two
by
two
seeing
what
would
happen
if
we
had
just
the
source
if
we
buried
whether
or
not
they
had
the
source
and
what
would
happen
if
we
gave
them
just
the
headline
as
opposed
to
the
whole
story.
Now
this
is
the
process.
I
told
you
about
a
minute
ago,
so
I'll
go
through
this
quickly,
the
article
sampling.
We
think
this
is
an
important
innovation
in
studying
false
news
or
studying
people's
ability
to
assess
the
veracity
of
news.
B
We
what
we
call
this
transparent,
replicable
process
and,
as
I
said
before
we
used,
we
got
two
articles
from
mainstream
news
sources.
Those
were
taken,
we
used
crowd
tangle,
which
is
a
tool
that
you
guys
can
use
as
well
by
just
by
dint
of
being
a
night
research
center,
and
then
we
also
used
for
our
suspects,
whose
news
sources
we
use.
The
list
existed
pre-existing
lists,
so
we
didn't
choose
the
list
from
al-qaida
at
all,
and
then
we
sort
of
cut
week.
We
coded
them
ourselves
as
false
news
weather.
B
Okay,
all
right
yeah
in
terms
of
I,
see
the
comment
about
whether
or
not
the
the
New
York
Times
is
a
left-leaning
publication.
We
have
a
sort
of
a
again
a
transparent,
automated
process
by
which
we
separated
into
sort
of
left
of
center
or
right
of
center
on
the
spectrum
of
political.
We
could
redo
that
so
yeah.
B
This
is
not
meant
to
be
a
normative
statement
here,
all
right,
so
these
are
examples
of
true
news
services
and
as
you
as
actually
as
I'll,
show
you
in
a
second,
our
designation
of
right-leaning
and
left-leaning
in
terms
of
mainstream
news
ends
up
making
very
little
difference
about
anything,
but
it
did
ensure
we
had
sufficient
variation
in
terms
of
news
sources.
So
these
are
some
examples
of
true
news
stories.
Interestingly
enough
Kanye
West
calling
himself
the
greatest
artist
that
God
has
ever
created.
B
That
was
a
true
statement
that
Kanye
West
did
call
himself
the
greatest
artist
that
God
has
ever
created,
whether
we're
not
weighing
in
on
whether
Kanye
West
is
the
greatest
artist
that
God
has
ever
created
and
Donald.
Trump
jr.
did
indeed
actually
kill
a
rare
endangered
sheep
in
Mongolia,
with
a
special
permit,
I'm,
here's
some
examples
of
things
that
were
coded
as
misleading
or
fake
news.
There
were
stuff
related.
B
Now
I
want
to
say
a
word
on
ground
truth
right
now,
and
this
is
what
I'm
going
to
come
back
to
at
the
end
of
the
at
the
end
of
the
talk,
cuz,
we're
thinking,
we're
thinking
a
lot
about
this.
For
today's
presentation,
I
am
going
to
use
as
the
ground
truth,
so
I
say
whether
someone
got
it
correct
or
wrong.
The
ground
truth
is
going
to
be
the
mode
of
the
professional
fact-checkers.
B
So,
at
the
end
of
the
talk,
I'm
gonna
come
back
and
I
love
your
feedback
on,
and
your
response
is
on
and
your
thoughts
on
different
ways
that
we
can
think
about
testing
the
robustness
of
our
conclusions
with
other
measures
of
ground
truth.
But
I
do
want
to
make
it
clear
right
now
that
for
now,
when
we
say
grant
when
we
say
if
you
got
it
right
and
what
the
ground
truth
is,
this
is
the
mode
of
the
professional
fact
checkers.
So
most
days
we
got
all
six
of
our
professional
fact.
B
Checkers
wrote
back
to
us
some
days.
Only
five
of
them
did
when
I
say
the
correct
answer
here.
It's
the
mode
of
the
professional
fact
checkers.
When
we
talk
about
the
veracity
scale,
then
that'll
just
be
the
average
of
the
professional
fact
checkers,
that's
much
simpler,
alright,
so
again,
article
designations
articles
can
therefore
be
the
correct
answer.
B
So
our
mainstream
news
sources
here
serves
as
a
useful
control,
but
in
all
of
these
cases,
pretty
much
with
one
exception
of
one
article.
We
found
that
the
mainstream
news
sources,
anything
that
came
in
the
mainstream
news
sources,
our
professional
fact
checkers
rated,
is
true,
alright.
So
what
makes
this
study
different
from
other
things
that
people
have
done
previously
about
individual
level
ability
to
identify
to
identify
the
veracity
of
news
sources?
There
are
two
very
kind
of
innovative
aspects
of
this.
B
One
is
that
we
are
doing
this
in
real
time,
so
we
built
this
pipeline,
so
we
could
send
this
out
and
we
could
analyze
and
we
could
have
these
stories
being
analyzed
in
the
first
24
hours
that
they
appeared.
We
think
this
is
crucial
for
a
couple
of
different
reasons.
First
of
all,
this
is
before
Snopes
and
polit
fact,
and
these
sorts
of
things
have
weighed
in
on
studies,
so
this
is.
B
B
If
we
are
going
to
use
the
signal
that
these
crowdsourcing
fact-checkers
can
can
provide
us
with
to
reduce
the
amount
of
false
information
in
the
overall
information
ecosystem,
we
would
want
to
intervene
quickly,
as
opposed
to
later
before
it
is
spread
widely
and
so
having
a
sense
of
what
this
happened.
What
happens
if
we
can
do
this
in
the
first
24
hours
that
news
stories
from
policy
perspectives
are
super
important,
but
we
also
think
from
a
sort
of
pure
science
thing.
B
This
is
our
best
way
to
balance
external
validity
with
really
seeing
the
extent
to
which
people,
when
they're
first,
given
the
access
these
articles,
that
they
can
identify
what
the
veracity
of
them.
The
second
thing
is
this
question
of
by
having
this,
by
having
this
pipeline,
where
we
pull
out
the
news
stories
automatically
we're
not
just
selecting
things
in
retrospect
that
went
particularly
viral
we're
gonna
capture
a
lot
of
stories
that
aren't
actually
particularly
viral.
B
We
capture
plenty
of
stories
in
this
design
that
are
actually
true
and
we're
not
introducing
reviewer
researcher
bias
into
the
selection
of
news
stories,
and
we
think
some
of
the
things
that
might
be
interesting
about
our
results
are
that
we
have
more
complicated
stories
in
here
than
researchers
might
normally
choose
if
they
were
choosing
a
story
to
evaluate.
So
we
think
it
adds
a
strong
degree
of
external
validity
to
what
we're
doing
alright.
So
the
first
research
question
now
I'm
gonna
walk
you
through
results.
B
Can
individuals
distinguish
between
true
and
false
news
in
real
time,
so
first
I'm
gonna
focus
on
the
categorical.
So
this
is
the
ground
we're
giving
ground
truth
based
on
the
on
the
fact
checkers
is
either
true.
False
or
misleading
could
not
determine
how
well
do
people
do
all
right,
so
this
is
across
our
entire
study.
B
Basically,
people
are
not
great
at
identifying
the
articles
that
are
professional
fact.
Checkers
have
rated
as
false
or
misleading.
They
do
ok,
identifying
the
true
story,
such
things
that
the
ground
truth
says
are
true
about
67%
of
our
respondents
categorize.
Those
is
true,
but
for
things
that
the
that
are
professional
fact
checkers
determine
are
false
or
misleading.
Only
about
a
third
of
our
respondents-
and
this
is
across
the
entire
study-
have
classified
those
as
false
or
misleading.
Here's
the
breakdown
by
true
articles,
it's
it's
less
worrisome,
I
mean
it
depends
what
you
define.
B
It
depends
what
your
baseline
is
for
being
worrisome
right.
What
what
we
see
is
about
only
about
where,
as
2/3
of
them
are
actually
kind
of
getting
it
right
that
they
are
at
identifying.
The
true
articles
is
true.
Only
about
15%
of
them
are
thinking
that
true
news
is
actually
false
and,
and
the
rest
are
in
this
kind
of
could
not
determine
category.
When
we
look
at
the
false
or
misleading
articles,
it's
it's
a
much
different
situation,
basically
for
things
that
our
professional
fact
checkers
have
ascertained
or
false
or
misleading.
B
We
have
about
only
about,
as
I
showed
you
a
minute
ago,
only
1/3
of
people
clack
correctly
classify
them
as
false
or
misleading,
but
even
more
troublesome,
a
third
of
the
people
class
of
incorrectly
classify
them
as
true
and
then
about
1/3
say,
could
not
determine
now
very
interestingly
source
matters
right
when
we
look
at
things
that
are
true
from
mainstream
sources.
People
are
over.
B
75%
of
the
people
are
getting
the
correct
answer
of
saying
that
these
are
true
when
we
look
it
true
from
fake
from
these
suspect
sources
about
50%
of
our
our
crowd
is
getting
it
right,
and
when
we
look
at
the
false
news
which
I
showed
you
before
from
the
fake
sources,
it's
about
1/3,
getting
it
right,
so
the
source
is
mattering.
Here
is
one
of
the
things
that
here's
all
right,
but
what
about
our
that
at
the
article
level,
are
there
some
fake
news
that
I'm
just
showing
you
at
the
individual
level?
B
Are
there
some
fake
news
articles
where
the
majority
respondents
are
right
or
our
respondents
as
a
group
rating,
most
fake
news
articles
incorrectly,
and
what
we
found
is
that,
even
when
we
look
at
the
article
level
again,
these
are
our
articles
that
are
professional
fact.
Checkers
have
rated
as
false
or
misleading.
We
don't
have
a
find
a
single
article
where
50%
or
more
of
respondents
actually
identifies
it
correctly
as
false
or
misleading
all
right.
So
what
about
the
veracity
scale?
B
This
is
just
to
show
a
totally
kind
of
different
way
of
measuring
it
on
the
veracity
scale.
How
would
you
rate
this
article?
This
is
from
one
to
seven?
Definitely
not
true,
and
the
veracity
scale
kind
of
looks
good.
Overall,
it
looks
a
lot
better
than
the
categorical
scale.
We
have
a
correlation
of
about
point.
You
know
about
point
eight,
but
we're
concerned
about
this,
because
this
seems
to
be
a
quirk
of
the
data,
that's
driven
by
the
source.
B
B
It's
kind
of
really
good
if
this
extends
to
science
related
information
as
well,
because
it
suggests
there's
a
pretty
good
vehicle
for
getting
information
out
to
the
public
that
the
public
will.
Trust
and
think
is
true.
However,
the
respondents
are
not
very
good
at
distinguishing
between
true
and
false
misleading
news
from
these
false
news
sources,
and
also
respondents
tend
to
trust
fake
news,
more
than
fact
checkers,
and
they
tend
to
trust
fake,
mainstream
news
less,
which
is
buried
in
those
veracity
tables
all
right.
B
B
Doesn't
matter,
then
there's
a
group
of
variables
that
we
found,
which
are
all
that
kind
of
like
having
more
of
a
sense
of
what's
going
on
in
politics
and
news,
so
political
knowledge,
trust
in
institutions,
digital
literacy,
news
consumption.
These
all
have
slightly
positive
impacts.
The
most
important
that
seems
to
have
the
biggest
effect
is
going
to
be
ideological
concordance.
So
let
me
walk
you
through
this.
So
what
we
have
on
the
x-axis
here
is
the
proportion
that
agree
with
our
fact
checkers
on
the
Left.
It's
for
all
news
on
the
right.
B
It's
for
fake
news
and
what
we
find
is
there's
not
a
huge
education
effect.
There
is
a
difference
between
people
with
less
than
high
school
education,
although
we
have
fewer
of
those
in
the
sample.
So
we
have
a
sort
of
larger
bar
around
that
and
people
once
you
kind
of
have
a
high
school
education
and
a
certainly
you
know
it
seems
to
be
fairly
close
on
all
news.
I
would
identically
identifying
the
fake
news.
B
B
Well,
not
quite
when
we
talk
about
different
explanations
in
that
paper,
but
it
suggests
that
the
issue
is
not
that
people
over
the
age
of
65
are
actually
having
trouble
figuring
out
the
veracity
of
news.
There
may
be
other
things
behind
why
they're
sharing
it
more,
but
so
this
was
a
very
interesting
and
surprising,
somewhat
surprising,
finding
to
us,
but
very
illuminative.
Ok,
so
now
we're
gonna
get
into
the
second
research
question.
B
B
This
is
just
our
you
know.
Conservative
sources,
and
these
are
things
that
are
fact
checkers
from
conservative
sources,
rated,
false
or
misleading.
We
had
fewer
from
liberal
sources
that
were
rated
false
or
misleading,
but
these
are
almost
completely
symmetrical
graphs.
I
mean
it's
just
kind
of
quite
striking.
B
You
start
with
extreme
liberals
and
go
down
to
liberals,
and
they
are,
you
know
much
better
at
correctly,
identifying
conservative
leaning,
false
news
as
being
false
or
misleading,
and
then
it's
the
exact
opposite
for
the
conservatives,
when
it's
a
liberal
lien
from
a
liberal,
leaning
source
that
our
fact
checkers
write
is
factor
misleading,
then
the
Conservatives
are
quite
good
at
identifying
and
a
false
or
misleading.
So
there
is
a
big
partisan
effect
that
are
seeping
through
individual
level
assessment
of
the
veracity
of
news,
and
this
fits
with
a
sort
of
long
line
of
motivated
reasoning.
B
Research
that's
been
around
for
decades
now
in
the
political
science
literature,
but
it
is
you
know,
but
what
we're
finding
here
is
a
sort
of
stark
level
of
this
shining
through
in
the
in
this
task
of
being
able
to
identify
whether
stuff
is
false
or
misleading.
It
also
matters
in
what
we
call
susceptibility
to
fake
news.
So
this
is
the
proportion
of
people
who
will
call
fake
news.
True
things
that
our
faculty
ecers
have
said
is
false
or
misleading,
and
they
will
label
it
as
true,
and
we
see
the
same
thing
among
the
conservatives.
B
We
see
it's.
The
effect
is
slightly
less
for
susceptibility
to
false
news
among
the
Liberals,
but
not
not
that
much
okay,
so
the
general
findings,
a
wide
array
of
characteristics
matter
when
matching
with
fact,
checkers
and
liberals
and
conservatives
both
exhibit
large
and
similar
biases
when
evaluating
the
veracity
of
news
all
right.
So
the
third
research
question
what
makes
people
better
or
worse
at
this
task-
and
here
was
where
we
tried
these
kind
of
four
different
interventions
to
see.
B
If
that
would
help,
so
the
interventions
are
going
to
be
our
first
experiment
was
we
asked
people
to
search
online
for
the
answers
now
in
our
original
study?
People
could
be
searching
online.
We
didn't
prohibit
them
from
searching
online,
but
we
didn't
ask
them
to
do
it
and
since
we
were
paying
people
to
take
these
studies,
our
assumption
is
most
of
them
were
not
spending
longer
than
they
needed
to
in
the
new
one.
B
In
the
experiment
we
had
people
and
encouraged
them
to
go
out
and
search
online
for
information
and
did
some
things
to
try
to
make
them.
Do
that.
The
second
experiment
we
paid
people
for
getting
the
answer
correct
and
then
in
the
third
experiment
we
buried
giving
in
our
normal
study.
We
give
everybody
the
full-text
of
the
article
in
the
in
a
experimental
treatment
which
sell
what
happened
if
we
gave
them
the
headline
just
of
the
article
and
the
blurb
instead
of
the
full
text
and
then
also
in
the
third
experiments.
B
We
made
it
two
by
two:
we
varied
whether
or
not
we
told
them
the
source.
So
we
for
the
experimental
group,
they
got
the
same
information,
but
it
was
without
the
source,
so
they
weren't
actually
clicking
on
a
URL
to
go
to
it.
They
were
just
taken
to
a
page
where
they
could
see
this
information
all
right.
So
this
is
the
sort
of
this
is
the
most
troubling
finding,
probably
that
we
have
in
the
study,
although
maybe
the
partisanship
one
deserves
that
thing,
but
also
kind
of
surprising,
given
what
one
would
normally
think.
So.
B
The
first
thing
is,
the
treatment
group
is
given
asked
to
search
online
for
information,
given
some
instructions
on
how
to
do
that,
and
given
some
ideas
about,
you
know
some
ways
to
try
to
make
sure
they're
doing
it
all
right.
Does
this
help
identify
fake
news?
No,
it
actually
to
false
news.
It
actually
turns
out
to
be
just
the
opposite,
so
does
searching
for
information
help
people
identify
false
or
misleading
articles.
B
B
So
this
is
for
the
true
articles,
so
this
is
the
articles
that
our
fact
checkers
rated
is
true,
and
we
found
that.
Actually,
when
people
went
out
and
searched
for
information
on
the
fact
checking
true,
they
were
more
likely
to
match
the
true
condition.
However,
the
press,
so
that
was
be
good
right,
like
on
true
articles.
Searching
for
information,
is
helping
people.
However,
the
same
thing
happened
with
our
false
or
misleading
articles.
B
Then,
when
we
didn't
ask
people
to
go
out
and
search
for
information,
so
what
we
suspect
might
be
happening
is
that
people
are
typing
these
headlines
into
Google
they're,
finding
things
popping
up
that
are
maybe
somewhat
similar.
Maybe
these
are
recycled
from
other
stories
that
have
been
out
there
before,
but
there's
something
about
this
activity
of
going
and
searching
for
information.
B
One
of
the
things
we've
done
in
this
in
the
study
that
I
haven't
talked
about
yet
is
we
actually
have
collected
sitting
every
time
when
we
enter
an
article
in
our
study
every
hour,
we
do
a
Google
search
with
that
article
as
the
headline
as
the
search,
and
we
record
what
comes
back
so
down
the
road.
We
could
actually
look
into
this
in
more
detail
and
see
what
those
Google
searches
were
recording,
but
this
seems
to
be
what's
going
on.
B
Is
that
going
out
and
looking
for
information
is
making
you
more
likely
to
say
things
are
true,
which
is
having
this
net
effect
of
decreasing
your
ability,
the
decreasing
the
group's
ability
to
get
things
right
on
these
false
or
misleading,
argue
articles,
not
the
group
individuals.
At
this
point
we
can
also
see
this
in
the
voracity
scale.
When
you
go
out
and
ask
if
you
just
look
at
this
graphically,
you
can
see
there
are
more
dots
above
the
zero
line
than
there
are
below
it.
B
Those
are
all
cases
where
our
fact
checkers
ranked
them
as
false
or
misleading
and
every
one
of
those
X's
that's
above
the
zero
is
in
our
treatment.
More
people
thought
it
was
more
true
than
thought
it
was
less
true
from
going
out
there
all
right.
What
about
paying
for
a
correct
answer?
That's
been
there's
some
interesting
social
science,
literature
around
political
knowledge,
questions
and
there's
been
a
lot
of
taik
about
how
you
know.
People
on
the
left
and
people
on
the
right
have,
you
know
radically
different
version.
B
You
know
beliefs
about
true
facts,
but
then,
if
you
get
you've
paid
them
to
get
the
answer,
some
of
those
differences
seem
to
go
away
pretty
quickly.
So
we
were
optimistic
about
this,
especially
for
the
ideological
bias
that
it
might
reduce
ideological
bias,
so
doesn't
help
to
identify
make
news.
No,
it's
just
the
bottom
line,
our
control
and
our
treatment
group,
but
look
exactly
the
same
in
terms
of
the
people
who
correctly
identify
the
false
or
misleading
articles.
Does
it
reduce
the
ideological
bias?
It
did
a
little
bit
somewhat.
B
So
if
you
look
here
for
conservative
articles
that
were
rated
false
or
misleading,
conservatives
were
slightly
more
likely
to
get
the
answer
correct
when
they
were
paid
than
they
were
when
they
weren't
paid.
And
if
we
look
at
liberals
on
liberal
articles,
rated
false
or
misleading
liberals
were
certainly
were
a
bit
more
likely
to
get
it
correct
when
we
paid
them
to
get
the
correct
answer
than
when
we
didn't
pay
them
to
get
the
correct
answer.
There
are
these
kind
of
bizarre
findings,
but
they're
fairly
small.
B
That
suggests
that
this
happens
in
the
opposite
direction,
which
kind
of
makes
no
sense
whatsoever,
but
but
for
the
for
the
people
whom
are
concerned
at
the
biased
conservatives
reigning
conservatives,
small
also
misleading
articles
and
liberals
reigning
liberal,
false
or
misleading
articles,
I
mean
these
levels
are
still
atrocious
right,
like
they're
still
like
at
the
best-case-scenario
we've
moved.
Liberals
from
you
know
slightly
under
twenty
percent
of
the
time
to
slightly
around
30%
of
the
time
Matt
on
these
questions.
B
Alright,
but
overall
and
again,
this
is
a
great
sort
of
example
of
a
kind
of
no
result.
Payment
doesn't
really
seem
to
have
much
effect.
You
see
how
much
more
bunch
this
is
around
the
zero
line.
Although
again,
if
anything,
you
look
at
these
X
are
these
false
misleading.
When
we
were
paying
people
to
get
the
right
answer,
these
false,
they
were
increasing
their
veracity
scores,
a
little
bit,
which
seems
troubling.
B
Ok,
the
final
experiment
we
did
was
this
two-by-two
experiment
where
we
looked
at
varying,
whether
or
not
we
gave
people
the
full
text,
which
is
what
we'd,
so
this
here
full
text
with
the
source
article
on
website.
This
is
everything
I've
shown
you
so
far,
so
this
is
the
sort
of
control
group
for
the
rest
of
the
study.
B
This
is
what
our
our
main
study
is,
so
we
varied
it
by
giving
people
the
full
text
without
the
source,
and
we
varied
it
by
giving
them
only
headlines,
as
opposed
to
the
whole
article
when
we
gave
them
the
source
we
kept
the
banner
in
there.
While
we
gave
him
the
headline
in
the
lead,
when
we
didn't
give
them
the
source.
This
was
just
like.
B
We
would
just
give
him
a
headline
and
lead
in
a
standardized
font,
and
this
was
the
full
text
in
a
standardized
format,
so
for
the
first
one,
if
we
want
to
look
at
the
difference
between
the
full
text
versus
just
having
the
headline
and
lead
we're
looking
at
the
difference
between
this
column
in
this
column-
and
this
is
what
say,
this
is
what
the
headline
and
lead
looks
like
that
way
they
would
get
if
they
got
the
source
version
of
it.
This
is
the
headline
and
lead
version
of
it.
B
If
they
didn't
get
the
source
version
of
it,
all
right
does
only
giving
people
the
headline
help
fact
checkers,
no,
not
really
how
much,
not
surprisingly,
when
we
only
gave
people
the
headline
they
did
slightly
worse
about
in
terms
of
agreeing
with
our
professional
fact
checkers,
and
that
held
whether
or
not
we
gave
them
the
source
or
not.
This
is
the
level
of
the
change
in
the
veracity
scores
between
the
case,
so
it's
a
sort
of
similar
finding
when
we
look
at
for
full
headline
when
we
look
at
source
versus
no
source.
B
So
this
is
now
comparing
the
difference
between
getting
the
source
either
with
the
headline
in
the
lead
or
what
we've
been
giving
people
already,
the
article
versus
the
sort
of
full
text
and
the
standardized
version
of
it
so
no
source
this
would
be
comparing,
for
example,
source
vs..
No
source
now
would
be
comparing
for
the
for
the
you
know,
just
the
headline
and
lead
it
would
be.
Comparing
this
version
of
the
headline
in
the
lead
versus
this
headline
and
lead
and
what
we
find
with
source
versus
no
source.
B
We
again,
we
don't
actually
find
that
much
difference
on
all
articles
when
they
have
the
full
text.
We
do
find
that
when
we
gave
we
didn't
give
people
the
source.
When
we
didn't
give
people
the
source,
they
did
slightly
better
when
we
just
gave
them
the
headline,
and
when
we
look
across
the
full
articles,
when
we
include
the
when
we
include
the
when
we
include
the
we
don't
give
them
the
source
I.
B
All
right,
so
the
finding
from
experiments
are
searching
for
information
increased.
These
are
the
key
takeaways
searching
for
information
increases
the
likelihood
that
individuals
believe
the
false
or
misleading
news,
as
well
as
increasing
the
likelihood
that
individuals
believe
the
true
news.
So
we
think
it
fits
in
with
this
truth.
Bias
idea
paying
individuals
for
correct
answers
does
not
increase
the
likelihood
that
individuals
correctly
identify
the
misleading,
false
news,
but
it
might
might
might
mitigate
a
little
ideological
bias
and
showing
only
headlines
decreases
the
likelihood
that
the
individuals
match
the
fact
checkers.
Okay.
B
B
Yeah
all
right
so
so
we
also
get
a
question
about.
Why
is
it
surprising
that
fact-checkers
are
ineffective?
Yes,
that's
a
good
question.
I'll!
Leave
it
to
you
to
have
the
norms
of
what
your
priors
were
on
these
particular
on
the
particular
issues
of
it,
so
surprising,
I
don't
want
to
weigh
in
on
whether
it's
surprising
or
not.
That's
a
in
the
eye
of
the
beholder
all
right
so
Kenda,
but
this
is
all
kind
of
leading
up
to
this
question
of.
B
Can
the
crowds
fact
check
news
in
real
time,
and
this
is
part
of
why
we're
doing
this
was
trying
to
figure
out?
Can
we
build
crowdsourcing?
Can
we
build
models
from
crowds
that
would
generate
valuable
signal
for
newly
appearing
articles
as
to
whether
those
articles
would
likely
be
rated
by
professional
fact-checkers
as
false
or
true
or
ultimately,
whether
they're
false
are
true
all
right?
So
why
might
we
be
thinking
about
this
right?
B
But
everybody
knew
that
the
sort
of
mainstream
news
sources
were
to
be
trusted
more
than
some
of
these.
Some
of
the
the
suspect
news
sources.
So
at
this
at
the
site
level,
the
crowd
seems
to
have
done
pretty
good
on
this
particular
topic
and
then
there's
this
whole
sister,
history
of
sort
of
crowds
being
able
to
do
these
sorts
of
things.
So
let
me
show
you
I'm
a
little
bit
of
kind
of
where
we're
going
in
terms
of
this
type
of
analysis
and
I'm.
B
We
can
sample
as
many
crowds
as
we
want
as
many
different
variations
of
crowds
of
three
people
to
see
how
crowds
of
three
people
do
over
10,000
simulations,
and
we
can
do
the
same
thing
going
up
to
crowds
of
thirty.
We
can
do
it
going
up
higher.
This
is
the
percent
of
our
crowds
across
our
entire
study
that
correctly
matches
the
fact
checkers.
So
we
see
when
our
crowds
are
small,
you
know
it's
quite
low.
B
B
B
We
could
do
this
and
when
we
look
at
the
difference
between
the
fake
news
sources
and
the
mainstream
news
sources,
what
we
find
is
for
the
mainstream
news
sources,
the
crowds
you
get
a
crowd
of
up
to
about
you,
know
10
or
15
people
and
they're.
The
mode
is
matching
over
80%
of
the
time,
but
again
for
the
false
for
what
we
called
our
sort
of
suspect
news
sources.
This
is
much
lower
all
right,
so
let
me
quickly
I.
Let
me
quickly
distinguish.
B
Let
me
quickly
go
back
to
our
first
original
research
questions
to
wrap
this
up.
Can
individuals
distinguish
between
true
and
false
news?
No,
we
see
a
big
problem
when
news
was
rated
false
or
misleading.
People
have
a
lot
of
trouble
with
this.
What
makes
people
better
or
worse
at
this
task
things
like
political,
not
more
knowledge,
more
institutional
trust,
more
digital
literacy
makes
things
better.
Are
there
large
partisan
differences
in
performance?
Yes,
but
it's
based
on
partisan
congruence
with
ideology
for
the
source
of
the
false
news.
What
can
we
do
to
improve
performance?
B
Paradoxically,
don't
ask
people
to
search
for
confirmation,
seems
to
be
better
for
identifying
false
or
misleading
news.
Paying
people
reduces
ideological
bias.
Can
fact-checking
news
be
crowdsource,
it's
still
too
early
to
say
that
there's
some
early
reasons
from
what
I've
showed
you
with
these
most
naive
models
to
be
concerned
about
the
ability
to
this,
but
there's
a
lot
more.
We
can
do
and
that's
what
I
want
to
talk
about
right
now.
B
So
one
of
the
things
that
we've
been
pushed
on
after
presenting
earlier
early
results
from
this
is
the
question
that
by
getting
the
mode
of
the
fact
checkers,
are
we
obscuring
the
fact
that
there
are
articles
out
there?
That
fact
checkers
have
trouble
getting
the
right
answer
for,
and
so
is
that
somehow
biasing
our
ability
of
individuals
to
get
the
correct
answer,
because
we're
these?
Are
these
just
really
hard
things
that
we
by
saying
the
mode?
Well,
maybe
the
mode
is,
isn't
the
correct
answer
in
some
cases
so
currently
we're
using
the
mode?
B
That's
what
we
pre
registered.
That's
when
we
design
the
study.
What
we
thought
of
was
by
far
going
to
be
the
best
way
to
get
the
ground
truth
out
of
this,
and
what
we
thought
was.
We
would
have
a
lot
of
things
where
it
was
like
five
to
one
or
maybe
four
to
one
to
one
or
something
like
that,
but
it
turns
out.
There
are
some
cases
in
these
false
news
stories
where,
if
we
changed
the
rating
of
just
one
fact
checker,
it
would
change
what
was
the
mode.
B
So
we've
also
come
up
with
this
alternative
method
that
we
called
the
robust
mode,
which
is
we
only
code
it
as
false.
If
changing
one
person
any
of
one
of
the
fact,
checkers
wouldn't
change
what
the
mode
was,
and
we
do
that
we
can
do
a
robust,
true
as
well
the
same
thing.
We
would
only
code
it
as
true,
if
changing
one
of
the
fact,
if
but
changing
one
of
the
fact,
checkers
wouldn't
change
that
categorization.
B
B
Then
our
ground
truth
will
be
that
that
article
is
false
or
misleading,
or
vice
versa.
As
long
as
nobody
said,
it
was
false,
our
ground
truth
will
be
that
that
article
was
true.
That's
gonna
lead
to
much
larger.
That's
gonna
lead
to
different
set
of
categories.
Oh
sorry,
but
they
all
yeah,
so
it
would
be.
All
two
would
be
like
that.
B
Sorry,
the
mode
of
it
would
still
have
to
be
false,
but
then,
if
somebody
said
it
was
true,
we
wouldn't
include
it
in
our
false
group
and
seeing
how
well
people
do
with
false
information
all
right.
Then
we
have
this
kind
of
any
of
the
first
three
outer
goals
where,
if
it
meets
matches
for
any
of
these
categories,
we
would
we
would
code
it
as
false.
B
That
would
lead
to
the
most
articles
being
coded
as
false,
because
the
no
true
one,
sorry,
the
no
true
and
doesn't
have
to
include
it
in
the
mode
it
would
just
have
to
include
it.
As
nobody
said,
it
was
true,
so
that
would
leave
the
most
coders
as
also
or
then
we
could
have
the
sort
of
most
robust
way
of
thinking
about
this
or
the
sort
of
most
demanding
way
is
that
it's
got
to
meet
all
of
these
four
categories.
B
B
We
want
to
think
about
lots
of
different
ways
that
we
could
extract
signal
in
different
ways.
That
might
be
more
helpful,
so
our
current
version
is,
we
are
going
to
we're
using
the
mode
of
the
crowd.
What
we're
now
looking
at
is
some
alternate
ways
we
can
use
the
crowd,
so
one
was
to
remove,
could
not
determine
to
just
say,
like
let's
get
rid
of
the
people
in
the
crowd.
Who
said
they
couldn't
determine
we'll
take
them
out
of
it,
and
then
we
take
the
mode
of
the
remain
to
true
or
false
or
misleading
right.
B
That
would
be
an
alternative
to
saying
it
has
to
be
the
mode
of
the
crowd,
because
right
now,
the
mode
of
the
crowd.
Sometimes
this
could
not
determine
so
this
is
we
remove
the
people
who
can't
determine
and
then
we
essentially,
then
the
crowd
will
say
that
everything
is
either
false
or
misleading,
or
it's
true,
unless
there's
not
a
majority,
unless
there's
a
tie
between
those
two,
the
second
option
we
could
do
for
the
crowd
is
that
we
could
count
the
could
not
determine
category.
B
We
could
say
all
right,
people
are
only
gonna
say
could
not
determine
if
there's
some
something
problematic
about
the
news,
and
so
we
could
then
just
sort
of
bounce
the
could
not
determine
category
for
the
crowd
into
the
false
or
misleading
category.
So
in
that
case
we
would
say
the
crowd
is
saying
that
the
article
is
true.
Only
if
a
majority
of
the
crowd
says
the
article
is
true.
Otherwise
we
say
the
crowd
is
sending
a
signal
that
there's
something
suspicious
about
this
particular
article.
B
That's
a
second
way
to
do
voting
rules
then
we're
also
playing
around
with
altering
the
way
we
sample
from
our
reviewers.
So
right
now
we're
just
sampling.
When
I
showed
you
a
crowd
of
three
people,
we
were
just
sampling,
a
random
draw
from
the
crowd.
So
our
first
thought
in
this
regard
is
well
knowing
that
there's
this
partisanship
bias
we
could,
instead
of
sampling
a
random
draw
from
the
crowd.
We
could
sample
partisan
bias
crowds.
So
we
know
people
self-reported,
conservative,
liberal
and
an
independent.
B
We
could
draw
crowds
that
have
one
conservative,
one
liberal
and
one
independent
in
them
and
see
if
those
crowds
do
a
better
job
of
matching
the
professional
fact
checkers
and
then
the
third
thing
which
I
haven't
even
gotten
into
here,
is
that,
of
course,
we
can
use
machine
learning
models
on
this
and
we're
going
to
be
doing
this
moving
forward
and
we're
playing
we've
started
playing
around
with
that
now,
but
I
didn't
want
to.
It's
are
still
quite
preliminary.
I
will
just
say
going
to
machine
learning
models
over
simple
voting
rules.
B
Our
initial
aspects
of
this
has
been
not
surprisingly,
that
they
perform
a
lot
better
and
that
they
are
that
they
are.
We
were
more
optimistic
that
the
machine
learning
models
are
going
to
actually
be
matching
the
professional
fact
checkers
more
often,
however,
the
downside
of
machine
learning
models
is
that
if
we
were,
if
we
were
going
to
try
to
push
anyone
to
implement
this
as
policy,
it's
much
much
easier
to
explain
to
the
public
from
Apollo
perspective,
a
simple
rule:
we
take
the
mode
of
the
crowd.
B
So
the
first
thing
we're
gonna
do
starting
perhaps
as
early
as
next
week
is
we're
gonna,
restart
this
whole
apparatus
again,
but
we're
going
to
do
for
the
first
month,
two
days
a
week,
we're
gonna
go
out
and
we're
gonna
only
do
Cove
in
nineteen
articles.
So
when
we
go
to
our
five
streams,
we're
gonna
look
at
the
most
popular
article
if
it's
related
to
the
pandemic,
we'll
do
it.
B
If
it's
not
related,
the
pandemic
will
go
to
the
next
one
and
then
find
the
first
one,
that's
related
to
the
pandemic,
and
then-
and
this
is
depending
the
funding
part-
we
our
plan
is
to
do
this
through
the
entire
election
season,
and
so
in
the
second
study.
What
we
would
do
is
we
would
do
one
day
a
week.
B
We
would
do
this
Cove
in
nineteen
category
and
the
second
day
a
week,
we'd
go
back
to
our
normal
thing
of
just
pulling
the
most
popular
article,
which
will
turn
out
to
be
Koba,
19
related
in
lots
of
the
cases
anyway.
But
then
we
would
do
this
throughout
the
entire
election
campaign
and
see
if,
as
more
people
stop
socially
isolating
at
home.
B
So
those
are
the
next
steps
and
then
finally,
last
thing,
I'll
just
say-
is
sort
of
just
to
give
you
a
sense
of
where
we're
going
with
this
in
terms
of
I've
showed
you
bits
and
pieces
of
a
larger
study.
The
sort
of
four
planned
papers
at
this
point,
all
of
which
are
in
progress
right
now-
is
individual
level
analysis
of
susceptibility
to
false
news.
B
So
a
lot
of
what
I've
showed
you
here
today,
but
really
focusing
on
this
question
of
who
codes
these
articles,
that
the
professional
fact-checkers
label
is
false,
whose
coding
them
is
true
who
thinks
they're
true?
The
second
is:
can
we
use
crowdsourcing
to
provide
valuable
signals
as
to
what
a
team
of
professional
factors
would
likely
say
is
true,
false
or
news?
B
What
was
the
real
spread
of
this
information
in
life
and
then
real
crowds
that
we
can
draw
upon,
so
we
could
say
in
the
agent-based
model,
if
say,
we
were
able
to
suppress
the
the
prevalent
the
spread
of
an
article
by
75%.
If
our
crowd
used
this
rule,
we
could
then
see
based
on
these
real
crowds,
that
we
have.
B
How
often
that
would
end
up
suppressing
true
news
and
how
often
that
would
end
up
suppressing
false
or
misleading
news,
and
then
we
can
set
these
rates
of
spread
based
on
how
the
articles
actually
spread
in
the
real
world,
because
we've
been
tracking.
All
of
that.
So
we're
very
very
excited
about
that.
B
Think
in
Kevin
can
speak
to
this,
but
I
think
the
very
preliminary
results
on
that
is
that
there
doesn't
seem
to
be
a
huge
amount
of
difference.
Three
months
later
among
the
professional
fact
checkers,
as
there
was
originally
alright,
so
I'm
now
done,
we've
got
a
little
bit
of
time
for
Question
and
Answer
thanks
very
much
for
your
patience
and
it's
been
a
pleasure
sharing
this
information
I
look
forward,
answering
your
questions
so
I'll
hand
it
over
to
Jeremy,
and
you
can
moderate.
However,
you'd
like
it.
C
B
C
Yeah
well
I'm,
not
so
I'm
new
to
this
community
I'm
actually
was
looked
into
this
a
lot
while
I
was
a
program
manager
at
DARPA,
okay,
so
and
I
reached
out
to
these
folks.
We
didn't
get
that
program
started
in
this.
Although
there
are
some
folks
who
are
pursuing
that
area
yeah,
it
seems
like
you're.
Looking
at
a
particular
snapshot
of
you
know,
find
me
and
it's
an
interesting
I
mean
getting
a
real-time
fact-checking
capability
would
certainly
be.
C
Actually
Jerusha
Vasuki
was
looking
at
the
time
history
a
lot
of
these,
so
that
you
could
find
you
know,
detect
the
fake
news
before
it
hits
the
mainstream
and
in
fact,
a
lot
of
the
stuff
that
is
fake.
That's
that's
not
trivial.
To
fake
like
not
by
not
not
obviously
fake
has
some
truth
in
it,
I
mean
it's
written,
it's
maligned
information
campaigns
and,
and
it
starts
in
offline.
C
You
know
reddit
right
8chan,
whatever
the
heck
is
called
4chan
whatever
it
is,
I
mean
it
starts
way
before
that,
and
if
you
could
find
the
content
and
find
that
seed,
then
you
know
we
started
to
look
at
that
and
then
I,
just
wonder
of
you.
You
know
you're
in
this
business,
whether
you're,
you're
yeah.
B
No
I
mean
I
think
that,
as
I
said,
we've
I
mean
that's
the
point
of
this
third
paper,
because
we
have
now
for
every
one
of
the
articles
in
our
sample.
We
have
the
spread.
I
would
spread
on
reddit
how
it
spread
on
Twitter,
how
it's
I
don't
know
if
I
read
it
but
Twitter
we
have
twitter
facebook,
Instagram,
we've
looked
at
all.
You
know
we're
collecting
this
data
about
how
its
spread.
So
the
idea
of
this
would
be
like
right.
B
So
if
we
can
get
that,
if
we
can
know
how
these
things
spread
down,
the
road
is
our
data
would
allow
us
to
model
what
interventions
look
like
what
interventions
might
do
to
that
spread
if
they
came
early
on
in
the
process?
And
the
question
here
is
that
the
question
here
is
that
can
we
actually,
you
know
what
is
the
content?
We're
gonna
get
some
stuff
wrong
if
we
do
it
early
on
in
the
process
right,
but
so
from
that
time
perspective,
that's
like
where
I
see
our
work
fitting
into
this
like?
B
Could
we,
you
know
ex-post,
have
identified
the
false
news
based
on
the
pad
their
patterns
of
spread
in
the
first
I
mean
the
problem
with
you
know,
if
you,
if
you
have,
if
you're,
really
good
at
identifying
false
information
based
on
their
patterns
of
spread
over
three
months,
then
by
the
time
you
run
that
on
things
that
are
three
months
old
they've
already
spread
all
over
the
place,
so
the
trick
would
be.
Could
you
do
it
earlier
on?
We
can
actually
use
our
data
to
look
at
that.
We
haven't
done
that
yet.
B
Yeah,
thank
you
yeah.
That's
why
we're
excited
about
this,
like
we
feel
like
most
of
what
we
had
known
previously
about
people's
ability
to
identify
false
and
misleading
information
came
from
articles
that
you
know
were
chosen
by
scholars
to
put
in
studies
that
had
already
been
out
there
for
a
long
time
and
that,
ultimately,
if
you
want
to
build
a
sort
of
apparatus
out
of
this,
the
best
time
to
do
it
is
going
to
be
really
early
on.
So
is
that
feasible
at
all
other
questions,.
C
Looks
like
I'm
sorry,
but
put
your
name
a
senior.
You
did
your
hand
up.
Hi
hi,
my
name
is
I
should
know.
Thank
you
very
much
for
the
presentation.
Dr.
thicker
on
East
was
reading
to
think
I
PhD
student
and
sake
University
and
I'm
interested
in
the
these
studies.
So
I
have
two
questions.
The
first
one
is
about
the
finding
that
age
didn't
matter
in
I.
Don't
find
false
news
right.
B
C
B
C
C
So
so
I
was
very
surprised
to
see
that
age
didn't
affect
identifying
fake
news.
Can
it
be
about
I
mean,
can
can
it
be
about
the
confidence
that
old
all
the
people
might
have
in
in
sharing
something
online
and
I
think
this
is
also
about
the
literacy
just
which
connects
to
my
second
question.
So
when
searching
for
news
for
evidence
doesn't
decrease
the
likely
the
island
I,
don't
find
false
identifying
in
news
I
think
so.
This
is.
C
This
is
about
this
right
so,
but
the
question
is
how
to
identify
a
fake
news
based
on
evidence
because,
like
a
Google
search
results
in
the
other
links
that
shares
the
same
article,
so
there's
a
availability
bias
as
well,
so
I'm
not
sure
like
how
to
improve
digital
literacy.
In
that's
like
what
your
opinions
on
this
with
me
right.
B
Yeah,
so
just
gonna
pull
up
so
okay,
yeah
well
I'll
just
answer
the
things.
I'm
gonna
slide
that
I
applaud
but
I.
So
the
first.
So
let
me
answer
the
first
of
those
two
things
right.
So
if
you
read
the
original
paper
that
we
wrote
about
the
sharing
of
fake
news
on
Facebook
in
2016,
we
said
that
the
Occam's
razor
explanation
for
this.
The
simplest
explanation
is
digital
literacy.
Right
I
tell
my
mom
to
get
on
Facebook
to
see
pictures
of
her
grandchildren.
B
She
gets
on
Facebook
and
at
2016
the
New
York
Times
of
the
Denver
Guardian
are
both
like
a
headline
a
picture
and
a
blurb.
Only
the
New
York
Times
is
the
New
York
Times,
the
Denver
Guardian
doesn't
exist
right,
and
so
that
was
the
sort
of
that
was
the
Occam's
razor
explanation.
But
we
do
mention
in
that
paper
and
I'm
sure
I
brought
up
in
the
talk
at
coach
Wright
that
there
are
alternative
explanations
for
this
right.
B
One
might
be
that
this
is
identity,
signaling
right
that
people
are
sharing
these
fake
news
stories,
as
I
have
to
say,
with
that
fake
meet
with
that
study,
I've.
Never
gotten
so
many,
you
know
anecdotal
comments
from
people
when
I
present
it
to
people
and
one
of
the
ones
that
resonated
with
me
was
what
a
scholar
who
and
I
was
presenting
it
once
was
saying
she
kept
sending
her
mother
Snopes
articles
and
saying
this
isn't
true.
B
This
isn't
true
and
the
mother
finally
wrote
back
and
said:
I
don't
care
if
it's
true
I
just
want
my
friends
to
know.
I
hate,
Hillary,
Clinton
right,
and
so,
if
it's
that
kind
of
identity
sharing
right
there's
all
we
would.
That
would
be
a
different
reason
why
people
share
fake
news
and
in
the
sense
that
there
are
so
there
would
be
different
reason
why
people
share
fake
news.
B
If
it's
this
kind
of
identity
sharing,
then
in
order
to
have
that
explain
our
results
that
we
found
in
the
science
advances
paper,
it
would
have
to
somehow
be
correlated
with
age
and
you've
mentioned.
One
thing
like
older
people
might
just
be
more
confident
in
their
opinions.
They
may
want
to
engage
more
in
terms
of
this
signaling
activity,
signaling
their
identity.
Other
people
have
brought
up
that.
B
It's
costly
for
people
who
are
still
in
the
job
force
or
the
job
market
to
be
sort
of
vehemently
in
public
sharing
this
fake
stuff
just
to
signify
their
identity.
So
there
might
be
something
about
that.
So
we
went
into
this
study,
which
is
designed
for
totally
different
purposes.
I
was
very,
very
interested
in
that
age
finding
and
to
the
best
of
my
knowledge,
this
is
the
first
time
we've
ever
had
a
study
where
we've
been
able
to
say
all
right,
based
on
that
first
science
advances
study.
B
You
know
you.
We
should
expect
that
these
people
would
be
worse
at
identifying
the
veracity
of
these
news
stories
and
in
particular,
we
should
expect
that
older
people
would
think
that
false
news
was
true.
More
often,
the
fact
that
they
didn't
to
me
signifies
and
I'm
I
got
to
say,
I've
been
pushing
this
since
the
get-go,
because
I
think
that
there
may
be
different
explanations
with
sighs
digital
literacy.
B
The
fact
that
they
didn't
is
to
me
really
interesting
because
it
suggests
some
of
these
alternative
explanations
that
aren't
the
simplest
explanation
might
have
been
behind
the
findings
that
we
had
that
we
had
previously
and
I'm
just
now,
but
the
one
thing
I
will
say
that
cuts
against
that
is
that
and
I'm
trying
to
pull
up
this
slide
really
quickly.
Is
that.
B
Is
that
digital
literacy
does
predict
higher
levels
of
concordance
with
fact
checkers?
This
was
part
of
this
series
of
different
variables.
We
also
looked
at
like
trust
and
institutions,
political
knowledge,
news,
consumption,
anything
that
we
were
kind
of
getting
at
like
at
it
like.
How
familiar
are
you
with
stuff?
Be
it
politics,
Viet
news
be
at
the
Internet
that
kind
of
thing
the
more
familiar
you
were,
the
better
you
do
on
this
right,
it's
not
nearly
as
dramatic
and
effect
as
as
we
found
for
the
the
Co
partisanship
finding,
but
it
is
in
effect.
B
So
if
it
really
was,
the
digital
literacy
had
nothing
to
do
with
the
sharing
of
fake
news.
You
might
have
expected
this
to
be
totally
flat,
so
we
have
a
little
bit
of
contradictory
evidence
here.
We
have
old
people
are
not
worse
at
this
task
than
younger
people
and
now
the
one
other
thing
we
have
to
say
here-
and
this
is-
we
have
some
other
work
about
this
thing
with
we're.
Do
you
know
these
are
people
who
are
taking
Internet
surveys?
Now
we
did
this
through
Qualtrics.
This
isn't
like
just
it
can
be.
B
This
is
an
EM
Turkish
or
anything
like
that.
If
I
showed
you
this
with
M
Turk
I'd
be
very
concerned
about
it,
but
nevertheless,
Qualtrics
seeks
to
have
representative
samples
of
the
population
on
whole
Lots
levels.
That
being
said,
these
were
still
people
who
were
able
take
an
Internet
survey.
That
being
said,
people
who
are
on
Facebook
are
still
people
who
know
how
to
use
Facebook.
So
whether
you
think
those
are
different
levels
of
things
there,
all
right
so
I
know
I,
see
we're
running
over
the
designated
time.
So
I
want
to
like
stop.
B
Do
you
want
to
type
in
the
chat,
or
we
can
we
can
follow
up
later?
If
you
want
alia
is
what
has
is
what
is?
Is
a
student
at
NYU,
Abu
Dhabi,
who
has
been
an
undergraduate
research
associate
of
our
SMAP
lab
so
again,
showing
the
the
value
of
these
virtual
online
online
fluorophore
present
presenting
work
Hey.
A
Okay,
thank
you
very
much
Josh.
That
was
a
great
talk
and
I'm
sure
we're
all
extremely
appreciative
both
of
the
research
and
of
sharing
it
with
us
for
the
rest
of
you
stay
tuned.
We'll
have
another
talk
in
the
series
next
next
month.
Meanwhile,
as
a
point
of
announcement
for
the
group,
the
case
of
seminar
series
will
be
summer.
Institute
will
be
going
on
as
a
remote
training
session
that
will
be
happening
in
June,
2nd
in
June.
A
The
second
thing
is
that
we
want
to
remind
you:
we
have
the
social
cybersecurity
webpage
where
we
are
putting
where
we
have
links
to
various
events,
but
also
have
links
to
the
various
people
in
this
area,
and
so
please
sit
and
links
to
various
tools
and
so
on
and
some
of
the
and
it
actually
points
to
some
of
the
smashed
tools.
So
please,
if
there's
things
that
are
missing
from
there,
that
you
want
to
see
there,
please
let
me
know,
and
we
will
get
them
up.
B
Guys,
thanks
for
the
invitation,
thanks
all
of
you
for
your
time-
and
you
know,
keep
an
eye
out
we'll
have
these
paper
drafts
of
papers
coming
in
the
coming
months
on
all
of
this
stuff
so
and
if
anyone
has
any
thoughts
on
any
of
these
questions,
I
raise
feel
free
to
reach
out
email
and
again
huge
thanks
to
Zieve
and
kevin
for
joining
today
and
for
all
the
work
they've
done
on
this
project.
Thanks
very
much
thanks.