►
Description
January 7th: Social Cybersecurity Working Group
Presenter: Emma S. Spiro
Title: Election Disinformation: Real-time Analysis and Response
Abstract/Description: NA
Bio: Emma S. Spiro is an Associate Professor at the University of Washington Information School. She is also an Adjunct Associate Professor in the Department of Sociology, and an affiliate of the UW Center for Statistics and the Social Sciences. Dr. Spiro is a Data Science Fellow at the eScience Institute. Dr. Spiro is a PI and Co-Founder of the Center for an Informed Public at the University of Washington. Dr. Spiro’s research involves the collection and analysis of large-scale social and behavioral data to answer key questions within the areas of sociology, information science, and social computing. Since early 2012, her work has focused on understanding social and information-related behaviors in the context of crisis events, including rumors, misinformation and collective sensemaking in online environments. Her work also explores the structure and dynamics of interpersonal and organizational networks in both online and offline environments. Her research has been published in leading journals such as PNAS, Social Networks, Field Methods, Demography and Information, Communication & Society. She also participates in premier conferences such as ACM CHI, ACM CSCW, and AAAI ICWSM. Dr. Spiro earned her Ph.D. in Sociology from the University of California, Irvine. She also holds a B.A. in Applied Mathematics and a B.A. in Science, Technology, and Society from Pomona College, as well as an M.A. from the Institute for Mathematical Behavioral Sciences at University of California, Irvine.
A
A
Once
we
add
more
specifics,
especially
as
claims
become
more
falsifiable
through
analysis
and
response,
they
became
more
relevant
to
our
project,
so
the
elections
being
stolen-
and
I
saw
postal
workers
burning
ballots-
gets
a
little
bit
closer,
but
we
were
what
we
were
really
interested
in
is
things
along
the
line
of
the
elections
being
stolen.
Here
is
a
video
of
postal
workers
burning
ballots
with
some
sort
of
attached
video.
These
are
the
kind
of
like
high
highly
in
scope,
pieces
of
information.
A
Misrepresentations
of
information
that
we
wanted
to
tackle-
and
you
know
out
of
scope-
were
things
like
my
opponent:
it's
criminal
criminal.
You
know
we
didn't
want
to
get
into
the
he
said
she
said
kinds
of
battles.
We
wanted
to
tackle
these
specific
pieces
of
content,
especially
those
that
were
getting
a
lot
of
attention,
so
just
to
give
you
a
sense
of
sort
of
the
the
sort
of
scale
of
this
operation
right
we
had.
This
is
not
a
comprehensive
list,
but
we
have
lots
of
different
partners
across
different
stakeholder
groups.
A
So
this
one
of
the
things
that
we've
learned
over
the
past
year
is
it's:
it's
a
it's
a
learning
process
to
put
together
these
kind
of
multi-stakeholder
collaborations,
especially
on
the
research
side
of
things.
So
all
four
of
these
major
stakeholder
groups
have
some
sort
of
influence
on
what
we
looked
at
right.
They
were
able
to
put
things
almost
like
in
our
intake
queue,
and
then
we
had
a
team
at
the
eitp,
which
was
which
was
based
mostly
at
stanford,
that
was
doing
on-call
data
collection.
A
Triage
sort
of
responding
and
organizing
some
of
these
tickets
as
we
called
them
initially
and
then
things
would
go
into
an
analyst
queue
and
here's
where
our
role
at
the
cip
at
the
university
of
washington
really
came
into
play.
Oops
sorry,
so
I
want
to
talk
a
little
bit
about
the
work
that
we
did
in
this
investigations
and
analysis
component,
but
I
do
want
to
say
that
you
know
the
work.
A
The
the
outcomes
of
this
work
were
not
only
sort
of
rapid
response
analysis,
things
that
came
out
as
blog
posts
or
other
things
or
twitter
threads,
but
they
are
also
served
directly
back
to
some
of
these
stakeholder
groups,
and
you
know,
as
a
researcher,
it's
oftentimes,
you
know
rarer
to
find
cases
where
you
can
work
and
actually
see
the
impact
of
your
results
almost
in
real
time
right.
So
you
do
this
analysis.
A
A
Okay,
so
this
is,
this
is
really
the
phase
that
I'm
gonna
talk
about
today
and
you
know
just
in
case
you
didn't
think
this
was
a
timely
topic,
as
I
was
putting
together
my
slides
yesterday
morning,
sort
of
finalizing
these
slides.
I
just
went
and
took
a
look
at
some
of
the
things
that
were
going
on
and,
of
course,
these
are
still
relevant
topics,
even
though
we
are
post-election
and
now
post-vote
certification.
A
This
is
not
a
topic.
I
think
that
is
going
away
and
there's
a
really
interesting.
I
think
mix
of
factors
at
play
that
are
socio-technical
right.
It's
the
people,
it's
the
platform,
it's
the
algorithms,
it's
the
policy!
All
of
these
things
are
intersecting
in
this
space
and
there's
so
much
to
do,
and
so
many
open
questions
that
I
think
this
particular
group,
especially,
is,
is
well
suited
to
tackle
some
of
these
things.
A
A
As
very
broad,
you
know,
crisis
events
are
characterized
by
high
levels
of
uncertainty,
sort
of
rapidly
evolving
information
spaces,
especially
online,
and
so
there's
no
reason
you
have
to
necessarily
think
of
natural
disasters
or
moments
of
political
or
civil
unrest
as
crisis.
But
a
lot
of
the
things
that
we've
seen
over
the
past
few
months
really
do
feed
well
into
this
framework
that
we've
been
developing
for
understanding
how
information
moves
and
spreads
and
is
received
by
and
enter
and
engage
it
people
engage
with
that
information
over
time.
A
So
one
of
the
things
that
we
first
did
as
part
of
this
project
is
kind
of
just
think
back
on
those
experiences
and
some
of
the
initial
preliminary
work,
we
were
doing
in
the
project
to
understand
what
we
might
expect.
Okay,
and
we
know
that
our
democracy
depends
on
our
trust
in
elections
and
we've
seen
many
indications
over
the
past.
A
A
You
know
place
on
top
of
all
of
this
that
we
are
now
operating
during
a
pandemic,
and
so
there's
a
lot
of
uncertainty
and
anxiety
about
what
are
the
ways
that
covet
the
sort
of
kova
19
dangers
would
potentially
deter
voting.
You
know
people
posting
images
and
videos
of
long
lines
and
saying
don't
go
to
the
polls
you're
going
to
get
covered.
There
was
a
lot
of
different
things
that
that
were
going
on
right.
A
So
in
doing
this,
what
what
we
were
thinking
about
a
little
bit
more
is
what
were
the
major
narratives
that
we
would
expect
to
see
that
were
already
emerging.
We
were
already
seeing
the
groundwork
being
laid
for
those
narratives,
as
we
were
engaging
in
our
preliminary
research
and
and
to
support
these
narratives
of
of
delegitimization
really
of
the
election.
Most
of
these
narratives
were
surrounding
d,
legitimate
jason.
What
were
the
stories
or
evidence
that
we
were
being
compiled
and
curated
to
speak
to
that
right?
A
A
We
knew
there
were
going
to
be
affidavits
follow
filed,
we
knew
there
were
going
to
be
allegations
of
foreign
interference
and-
and
I
think
one
of
the
other
interesting
aspects
of
our
project
is
we're
all.
We
were
also
interested
in
when
are
bad
statistics
being
highlighted
right
and
when
were
people
making
sort
of
data
science
based
or
statistically
based
claims
that
were
selectively
highlighted
in
different
ways
to
to
be
reported
as
evidence
of
a
rigged
election
or
a
fraudulent
election?
A
And-
and
of
course,
you
know
this-
this
particular
project
was
very
public
facing,
so
you
know
to
the
best
of
our
ability.
We
tried
to
offer
some
advice
to
journalists
and
the
public
not
only
to
prepare
for
this
kind
of
uncertainty
on
election
day
election
night
and
those
sort
of
days
and
weeks
to
follow,
but
also
to
emphasize
that
the
vast
majority
of
ballads
and
polling
stations
wouldn't
experience
experience
these
kind
of
issues
right.
A
A
I
just
chose
three
that
are
kind
of
different
and
sort
of
give
you
a
sense
of
what
we
were
working
on
and
some
of
the
interesting
things
that
we
found,
so
I'm
sure
almost
everyone,
if
not
everyone
saw
some
of
the
narratives
that
were
emerging
around
mail
being
dumped
okay.
So,
for
this
is
one
example
here
where,
on
september,
23rd,
I
think
was
the
the
date
of
the
event.
A
There's
a
batch
of
mail,
that's
found
improperly
disposed
of
in
a
ditch
in
wisconsin,
and
the
mail
was
turned
over
to
the
postal
service
who
who
was
reporting
on
this
case-
and
this
is
just
one
of
multiple
sort
of
cases
around
mail
dumping
that
we
investigated
and
it's
interesting
to
think
about
how
the
two
different
cases
or
many
different
cases
compare.
But
in
this
particular
case
they
weren't
reporting
on
the
number
of
ballots
that
were
here.
A
A
A
We
can
use
some
basic
kinds
of
clustering
algorithms
to
find
what
we
might
think
about
as
community
audience
communities
within
this
network,
and
so
here
what
we
actually
saw
in
a
lot
of
our
election
related
content.
Is
we
have
sort
of
four
main
clusters
which
would
be
like
a
mainstream
liberal
cluster?
A
You
know
things
like
gateway
pundits,
fox
news
contributors
are
are
really
the
ones
who
are
pushing
and
engaging
with
this
particular
incident
after
sort
of
local
news
outlets
initially
reported
on
it,
an
another
sort
of.
So
I
guess,
let
me
back
up
and
say
that,
so
I
also
wanted
to
sort
of
illustrate
for
you.
You
know
one
of
the
things
that
we've
been
thinking
a
lot
about
in
our
team
is:
how
do
we
use?
A
So
the
sort
of
size
represents
the
audience
estimated
by
that
the
tweeting
accounts,
followers
follower
count
and
what
these
plots
have
helped
us
to
sort
of
think
about
in
a
more
qualitative
sense,
is
sort
of
how
information
gains
attention
and
how
it
spreads
and
to
look
at
if
there
are
places
where
you
see
the
narrative
getting
amplified
by
particular
kinds
of
accounts
online.
So
here
is,
in
the
lower
left
hand
corner
this.
A
Big
blue
circle
is
actually
this
narrative
gets
early
amplification
from
eric
trump
and
then
you
know
as
it
moves
along.
It
doesn't
get
a
whole
lot
of
attention
until
the
upper
right
corner
here,
where
there's
sort
of
a
new
burst
of
attention.
Almost
as
as
the
to
this
event,
and
what
happened
at
the
end
of
this
timeline
here
is
that
some
media
outlets
and
and
actually
reports
that
were
made
by
the
white
house
press
secretary,
mentioned
this
particular
incident.
A
So
you
know,
one
of
the
things
that
we
have
have
found
is
how
these
kind
of
false
narratives
that
began
in
sort
of
very
right-wing
partisan
media
outlets
with
very
sort
of
false
misleading
claims
that
frame
an
actual
event.
That
happened
eventually
make
their
way
into
sort
of
more
mainstream
political
discourse
and
then
come
out
of
the
mouth
of
the
white
house.
Spokesperson,
okay
and
then
another
thing
I'll
just
sort
of
seed
as
something
is.
A
We
also
wanted
to
understand,
sort
of
who
and
who's
participating,
not
only
in
single
events
but
across
events.
So
in
many
of
our
analysis,
we
also
look
to
see
you
know
what
are
the
most
tweeted
domains
in
this
particular
case.
What
are
the
actors
that
are
engaging
the
sort
of
accounts
that
are
engaging
over
and
over
again,
and
especially
with
some
of
these
mail
dumping
narratives
right?
A
Okay.
So
as
we
were
working
through
our
analysis,
one
of
the
things
that
became
very
evident
and
potentially
powerful
was
the
fact
that
we
at
the
eip,
so
this
election
integrity
partnership
we're
seeing
hundreds
and
hundreds
of
these
incidents
reported,
and
you
know
one
of
the
obvious
questions
I
think
to
ask
then
is,
if
you
look
across
miss
and
infer
disinformation
incidents.
A
What
kind
of
repeat
patterns
do
you
see?
So
we
decided
to
look
into
this
after
recognizing
the
same.
You
know
names
and
same
domains
over
and
over
again
and
and
the
way
that
we've
been
sort
of
thinking
about
this
or
characterizing
this,
as
repeat
offenders
of
misinformation
and
disinformation
spread.
A
What
can
they
do?
What's
allowable?
What's
not
allowable
and
something
we
don't
see.
Very
often
is
sanctions
against
these
repeater
repeat
offenders
right,
so
we
tend
to
add
labels
to
particular
pieces
of
content,
but
not
labels
to
particular
kinds
of
accounts.
You
know
it
was
a
fairly
significant
over
the
past
24
hours
that
you
know,
trump's
content
was
removed
from
platform
accounts
were
suspended,
but
that's
not
something
we
saw
systematically
sort
of
over
the
past
few
months
across
the
election,
and
so
you
know
as
you're.
A
Looking
at
this
list
of
our
sort
of
top
repeat
offender
accounts,
you
can
see
some
of
the
things.
You
would
probably
recognize
a
lot
of
the
names
on
this
list.
If
you've
been
following
some
of
the
current
narratives
on,
especially
on
twitter,
okay,
we
do
have
some
anonymous
I'll
say
we
do
have
some
anonymized
things
here.
Our
practice
has
been
to
anonymize
non-verified
users,
things
that
have
some
expectation
potentially
of
of
privacy,
okay.
A
So
then,
this
this
video
becomes
something
that
depicts
shredded
mail
and
ballot
applications,
specifically
those
filled
out
for
donald
trump,
and
then
you
can
see
as
the
this
this
piece
of
content
moves
across
these
different
actors.
You
know
it
becomes
ballot
fraud
on
video.
Some
of
these
things
received.
You
know
thousands
of
retweets,
even
though
the
tweet
was
later
removed,
and
you
get
you
get
to
a
point
where.
A
A
What's
going
on
here
and
again,
you
know
something
that
starts
out
relatively
factually
correct
becomes
much
different
right,
so
as
things
get
reshared
as
they
get
deco
de
contextualized,
the
these
kind
of
repeat
offender
accounts
and
repeat
offender
news
outlets
in
particular.
Partisan
outlets
rewrite
the
news
stories
right
and
they
republish
them
with
their
own
new
and
misleading,
specifically
partisan
frames.
A
A
So
things
like
this
tweet
that
says,
wisconsin
has
more
votes
than
people
who
are
registered
to
vote
right
and
you
know
not
surprising,
because
people
can
register
to
vote
the
day
of
or
or
different
kinds
of
things
like
that.
But
you
know
one
of
the
one
of
the
things
that
was
talked
about.
A
lot
was
benford's
law,
and
so
we
using
some
of
the
expertise
on
our
team,
who
think
a
lot
about
statistical
and
numerical
literacy.
A
One
of
the
things
that
we
looked
at
was
the
use
of
these
kind
of
scientific
arguments
right,
scientific
arguments
to
support
and
deal
claims
of
fraud
and
delegitimize
the
election
results.
So
this
actually
comes
from
a
simulation
from
one
of
our
led
by
one
of
our
postdocs
at
the
cip
and
those
familiar
with
benford's
law
might
sort
of
recognize
some
of
these
distributions.
But
the
basic
idea
here
is,
if
you
look
at
the
vote,
tallies
projected
against
benford's
law.
A
So
what
we
sort
of
think
about,
as
you
know,
armchair
investigators
during
the
election
had
begun
to
argue
that
too
many
of
the
submitted
vote
totals
begin
with
larger
single-digit
numbers
like
seven
eight,
and
this
was
being
spun
as
evidence
of
fraud,
and
you
know
our
our
aim
here
was
to
really
caution
against
that
kind
of
conclusion,
because,
as
these
simulations
show,
those
the
assumptions
of
these
laws
are
often
violated
in
these
kind
of
cases.
Right.
A
You
know
how
is
it
that
these
local
news
outlets,
who
have
initially
very
little
audience
in
some
of
these
platforms,
become
the
sort
of
center
of
attention
or
central
to
these
larger
narratives
of
delegitimization
during
the
election?
And
so
how
is
it
that
you
know
all
of
a
sudden?
The
spotlight
is
shown
on
accounts
or
individual
pieces
of
content
that
were
sort
of
maybe
in
sort
of
more
obscure
page
places
in
the
social
network
and
all
of
a
sudden.
A
They
are
sort
of
moved
across
that
network
and
become
you
know
something
that
comes
out
of
the
white
house,
press
secretary's
mouth.
So
these
kind
of
attention,
dynamics
and
amplification,
I
think,
is
one
of
the
places
where
there
are
lots
of
open
questions
as
we
move
forward
in
this
space.
The
other
thing
that
we
saw
very
clearly
in
this
project
is
about
cross-platform.
A
It
is
sort
of
the
emphasis
on
cross-platform
information
spread,
which
again,
I
think,
is
something
that
our
community
hasn't.
It's
hard
to,
study
right,
it's
hard
to
sort
of
think
about
that
cross-platform
information
spread,
especially
especially
now
that
we
have
many
of
these
more
peripheral
platforms
that
have
very
little
visibility
to
researchers.
A
So
you
know
things
that
get
taken
down
or
for
violating
platform
policies
on
facebook
or
twitter
and
are
moved
to
things
like
parlor
or
or
other
platforms.
How
do
we
think
about
that?
And
how
do
we
investigate
it?
You
know
in
this
particular
project
we
did
a
lot
of
sort
of.
Almost
our
partner
teams
did
a
lot
of
forensic
work
in
this.
These
spaces
youtube
in
particular,
I
think,
is
one
of
the
places
that
we,
our
team,
is
sort
of
thinking
more
about
as
a
place
for
content.
A
A
You
know
we
started
this
project
with
the
idea
that
there
was
really
an
opportunity
to
have
impact
by
doing
this
rapid
response,
analysis
and
lending
our
sort
of
way
of
thinking
about
misinformation
and
social
media
platforms.
To
this
case,
where
we
could
directly
re
report
back
to
platforms,
have
them
take
action
right
is,
I
think,
again,
fairly
unique
in
terms
of
the
research
aspect
of
this
project.
But
how
do
we
think
about
that
impact
and
intervention?
A
You
know
there
were
many
cases
where
we
had
discussions
about.
Do
we
even
want
to
highlight
this
particular
incident?
Is
that
going
to
make
it
worse
than
it
actually
is
now,
and
so
there
was
a
lot
of
sort
of
tracking
and
trying
to
sort
of
make
heuristic
calls
on
whether
we
should
respond
to
particular
events
or
not
right.
A
You
know,
there's
been
some
evidence
that
repeating
some
of
these
false
narratives
just
makes
people
believe
them
more.
You
know,
and
I
think,
as
we
think,
about
rapid
response
right.
How
are
we
set
up
as
a
community
to
think
about
rapid
response
science
in
these
cases
where
it
becomes
really
critical
and
the
crisis
content
is
one
of
those
you
know
the
the
kobit
19
pandemic
has
also
illustrated
this.
A
Lots
to,
I
think,
dig
into
there,
and
I
I
also
wanted
to
recognize
the
fact
that
this
kind
of
project
takes
a
significant
toll
on
project
participants.
So
everybody
you
know
from
the
pis
to
the
students
you
know
spend
hours
a
day
on
some
of
these
platforms.
Looking
at
content
that
can
be
very
emotionally
taxing
and
disturbing,
and
we
experience
this
with
our
students,
in
particular
on
this
project
and
even
at
the
beginning.
You
know
we
knew
this
was
gonna
happen
after
working
in
the
crisis
context.
A
For
over
a
decade,
we
try
to
be
very
careful
about
the
content
that
we
ask
our
research
project
participants
to
look
at
and
explore
and
in
in
some
ways
you
know
doing
this
as
part
of
a
team
is
almost
necessary
so
that
you
have
others
with
you
in
that
experience,
but
it
takes
a
significant
toll.
You
know
at
the
end
of
this
project,
while
everybody
felt
I
think,
good
about
the
amount
of
work
and
impact
that
was
accomplished
on
this
project.
A
You
could
tell
everybody,
had
sort
of
experienced
the
mental
strain
and
a
mental
and
emotional
strain
of
that
as
well,
and
you
you
sort
of
worry
about
the
the
effects
of
that
for
this
and
and
long
term.
So
I
think
that's
another
area
where
we
need
to
spend
more
attention
and
think
about
sharing
resources
to
ensure
the
sort
of
safety
and
well-being
of
researchers
who
are
in
this
space.
So
with
that
you
know
I
I
want
to
save
our
last.
A
You
know
10
15
minutes
or
so
I
wish
I
could
have
been
more
but,
like
I
said
I
could
talk
about
this
for
hours
to
sort
of
discuss
this
a
little
bit
I'll
I'll
sort
of
leave
you
with
some
of
our
center
for
an
informed
public
contact
information.
That's
where
you
can
find
most
of
our
work
and
the
eip
work
as
well
is
you
can
find
it
from
there
and
everything
that
I
talked
about
here
again
was
collaborative
work.
B
Thank
you
emma.
This
is
so
fascinating
and
timely,
to
say
the
least,
and
I'm
sure
I
I,
as
I
see
a
lot
of
people
have
raised
their
hands.
So
I
know
there
are
a
lot
of
thoughts
and
questions,
but
just
just
before
we
open
the
floor
for
q,
a
I
just
wanted
to
point
out
in
your
last
slide.
You
mentioned
that
there
is
a
need
for
studying,
cross-platform
and
also
looking
at
youtube,
and
I
totally
agree
with
that,
and
this
may
be
sounded
like
a
shameless
self-promotion.
B
So
if
anyone
is
interested,
I
would
be
happy
to
share
the
link
of
the
book
and
similarly
for
the
youtube,
we
have
a
tool
called
youtube
trackers
and
I
would
be
happy
to
speak
with
you
emma
or
anyone
else
who
would
like
to
use
that
tool,
which
essentially
looks
at
youtube
content
and
allows
you
to
examine
how
information
campaigns
shape.
Not
necessarily
misinformation
could
be
any
even
legitimate
information
campaigns
if
you
run
a
youtube
campaign
or
ad
campaign.
B
You
can
use
that
to
understand
how
your
campaign
is
doing
in
in
different
ways,
because
we
we
know
that
youtube
data
is
quite
humongous
and
high
velocity.
So
there
are
several
smart
techniques
to
process
all
the
information,
but
that's
that's
all.
I
also
have
a
lot
of
the
question,
but
I
think
it
would
be
better
if
we
open
the
floor
for
some
other
q.
A
from
anyone.
B
I
see.
Uber
has
his
hand
up
over.
Please
go
ahead.
Sure.
Thank
you
emma.
It
was
a
great
talk.
Thank
you
very
much
and
it
was
very
timely.
So
I
have
one
question
actually
and
maybe,
if
you
have
any
insight
on
the
type
of
users
involved
in
these
type
of
campaigns,
so
it
can
be
about
or
it
can
be
humans,
but
they
have
some
particular
goal.
B
A
A
You
know
a
term
that
I
don't
like
very
much
but
like
influencers
on
social
media,
partisan
influencers
in
particular,
so
we
didn't
see
as
much,
I
think,
bought
activity,
but
that
could
very
well
be
a
function
of
the
kinds
of
incidents
that
we
were
exploring
right.
So,
as
I
sort
of
mentioned
at
the
beginning,
the
sort
of
eip
project
as
a
whole
tried
to
scope
the
kinds
of
incidents
we
were.
A
A
But
the
way
that
we
were
looking
at
many
of
our
incidents,
you
know
and
in
a
more
sort
of
strict
kind
of
statistical
sense,
makes
it
hard
to
think
about
causal
claims
in
a
lot
of
these
settings.
So
I
think
that
yeah-
that's
particularly
challenging
and
sort
of
probably
more
suited
to
some
of
our
other
analysis
partners
who
do
the
kind
of
more
qualitative
forensic
work
to
try
to
understand
the
sort
of
origins
and
where
things
come
from
and
how
they're
coordinated.
A
B
A
Yeah
that
that
is
a
really
great
question
and
actually,
unfortunately,
one
that
I'm
not
really
well
suited
to
tackle,
because
most
of
our
work
has
been
very
us
centric
and
mostly
that's
just
that's
a
that's
a
function
of
our
kind
of
research
team
right.
We
don't
have
a
sort
of
cross
language,
ex
extensive
cross,
language
expertise,
and
so
you
know
in
some
ways
we're
kind
of
forced
to
focus
on
the
things
that
we
can.
We
can
look
at
you
know.
A
We've
we've
done,
I
think
a
little
bit
in
the
past
with
with
sort
of
cross
language
work,
but
I
think
nothing
that
it
really
does
address
your
question,
and
so
you
know
I
mean
I
think
that's
that's.
A
really
big
open
question
is
to
think
about.
You
know
how
this
sort
of
applies
in
the
global
south.
How
this
applies.
You
know
outside
of
the
u.s.
A
B
Thank
you
so
much,
and
we
can
definitely
discuss
more
about
that.
I
know
that
we
have
some
more
questions,
particularly
martin.
Please
go
ahead.
I
am
this.
Is
martin
halbert,
I'm
at
the
national
science
foundation,
great
presentation.
I've
got
a
a
kind
of
a
broad
question
that
maybe
you
could
just
reflect
on
a
little
bit,
especially.
This
is
been
on
my
mind
as
I'm
on
federal
task
forces
that
look
at.
I
guess
challenge
issues
in
sharing
information.
B
My
particular
remit
is
public
access
within
the
nsf
and
there
are
seems
to
me
you
know
clear
issues
that
we
need
to
grapple
with
between
the
tensions
in
providing
public.
You
know
information
or
you
know
open
these
open
social
media
systems,
and
we
want
to
do
that,
but
then
the
kinds
of
problems
that
you're
highlighting
as
well
as
just
access
to
you
know
personal
information
and
you
know
restricted.
B
You
know,
defense
information
are
also
tensions
with
you
know
what
we
want
to
achieve
in
an
open
society,
and
I'm
just
you
know,
because
of
your
the
area
of
work
you're
in
you're.
I
think
you're
uniquely
situated
to
be
able
to
comment
on
these
tensions
and
it's
a
that's,
a
pretty
broad
unspecific
question,
but
I'm
just
curious
what
your
thoughts
are
about.
It.
A
Yeah
I
mean
I,
I
think
it
is
a
broad
question,
martin
and
you're
right.
I
think
this
is
a
really
important
area
for
sort
of
also
a
field
to
think
about
moving
forward,
and
I
actually
think
there's
you
know
you.
You
touch
on
kind
of
two
different
components
here.
One
is
that
it's
it's
also
very
challenging
to
work
and
study
in
this
domain
when
a
lot
of
the
data
that
you
need
or
would
be
really
powerful,
is
sort
of
locked
down
and
unavailable.
A
Right
and-
and
you
know,
we've
we
run
into
this
kind
of
challenge
all
the
time
in
our
own
work.
Trying
to
come
up
with.
You
know,
guidelines
and
best
practices
of
you
know.
When
is
it
important
to
actually
highlight
these
repeat
offenders,
for
example?
And
when
is
it
important
to
preserve
privacy
or
expectations
of
privacy
in
our
our
work?
I
don't
you
know,
I
don't
have
all
all
the
answers.
A
I
think
to
this
particular
challenge,
but
I
think
you're
right,
martin,
that
this
is
this
is
going
to
be
an
area
that
sort
of
again
sort
of
collectively
as
a
field.
It
would
it's
really
important
to
develop
some
sort
of
standards
or
ethical
guidelines
of
how
to
do
work
in
this
space
and
then
in
a
sort
of
transparent
but
privacy
preserving
way
yeah.
I
think
you
know
for
us.
A
B
Thank
you
and
I
know
emma
as
you
mentioned,
we
can
go
hours
on
this
topic.
We
can
conduct
five
day
conference
for
this.
It's
it's
such
a
fascinating
and
timely
topic
and
it
keeps
evolving.
You
know
it
keeps
us
going
things,
keep
changing
and
and
as
researchers,
we
are
always
fascinated
by
these
behaviors.
B
I
know
that
we
are
three
minutes
over
the
hour
and
people
probably
getting
hungry.
So
we
would
stop
here,
and
I
would
like
to
thank
you
so
much
for
making
yourself
available
and
speaking
to
the
working
group.
I
would
also
like
to
thank
all
the
participants
for
hanging
around
and
listening
to
to
our
great
presentation
today.
Thank
you
so
much.
Everyone.