►
Description
Social media, online news, blogs, etc. – in conjunction with widespread access to the internet from multiple devices including cell phones – is creating a new information environment. In this environment, the norms of social interaction, conversation, public discourse, and news reporting are being rewritten.
In this panel, we explore how Trolls and Bots can shape the conversation, shift public discourse, and build bridges among groups who might not otherwise connect. We explore how fake news can spread and the role of different types of actors in creating, spreading, countering and monitoring such news.
Panelists were: Nitin Argawal of the University of Arkansas at Little Rock; Kathleen M. Carley of Carnegie Mellon University; Huan Liu of Arizona State University; and Rand Waltzman of RAND Corporation.
A
A
The
hub
with
Georgia
Tech
for
those
of
you
who
may
not
be
familiar
the
South
big
data
hub
works
to
catalyze
and
strengthen
partnerships
between
academia,
industry
and
government
to
apply
data
science
and
analytics
to
real-world
challenges
before
we
get
started.
We
ask
that
those
in
the
room
silence
your
phones,
please
and
those
that
are
watching
online.
We
ask
that
you
mute
your
mics
and
we
will
be
monitoring
the
chat
box
in
WebEx.
So
if
you
have
a
question
you
can
type
in
your
question
there
and
Sienna
who's
in
the
back
of
the
room
here.
A
Monitoring
will
raise
your
hand
and
ask
your
question
on
your
behalf
for
those
of
you
who
tweet
do
we
have
the
tweet
slide
up?
Yes,
I
won't
write
before
that
hashtag
SB,
dh4
South,
big
data
hub,
so
hashtag
SB
th17.
We
also
will
be
trying
to
monitor
Twitter
as
well
for
that
hashtag.
So
if
you
have
a
question
for
the
panelists
or
a
comment
you'd
like
to
make,
please
also
use
that
and
you
can
follow
the
hub
on
Twitter
at
South,
big
data
hub,
so
welcome
to
the
round
table.
This
is
a
monthly
series.
A
We
host
on
data
science
topics
exploring
the
emergency,
are
merging
research
challenges
and
potential
solutions.
Today's
roundtable
is
also
being
co-hosted
by
Meredith
Lee
who's,
the
co-executor
or
the
executive
director
of
the
West
hub
as
one
thank
you,
one
is
in
Arizona
and
around
is
in
California,
so
we've
got
the
West
hub
represented
and
we
also
have
the
Northeast
hub,
which
is
here
in
Pittsburgh,
so
Renee
vest
on
is
the
co-executor
based
at
Columbia
University.
A
So
our
discussion
today
will
focus
on
anti
social
computing
and
the
new
information
environment.
Our
panelists
will
discuss
the
changing
norms
and
social
interaction,
conversation,
public
discourse
and
news
reporting
around
that.
We
have
four
distinguished
panelists
with
us
today,
so
I'd
like
to
introduce
dr.
nitin
Argo
wall.
He
is
the
Jerry
Malden
Entergy
endowed
chair
and
distinguished
professor
of
information
science
at
the
University
of
Arkansas
at
Little.
Rock
nitin
is
the
director
of
collaboratory
'um
of
social
media
and
online
behavioral
studies.
A
His
research
interests
include
social
computing,
deviant,
behavioral,
modder,
modeling
group
dynamics,
influence
trust,
collective
actions,
social
cyber
forensics,
which
we're
going
to
ask
you
about
later:
health,
informatics
data,
mining
and
privacy.
Dr.
Kathleen
Carly,
who
many
of
you
here
in
the
room,
know
it's
professor
of
computation
organizations
in
society
in
the
center
director
for
computational
analysis
of
social
and
organizational
systems
in
the
institute
for
software
research,
school
of
computer
science
at
carnegie,
mellon
university.
Her
research
combines
cognitive
science,
network
science,
agent-based
modeling
social
media
analytics
and
language
technology
to
address
complex
social
and
organizational
problems.
A
We
have
dr.
Huang
Liu,
who
is
a
professor
of
computing
science
and
engineering
at
Arizona,
State
University.
His
research
interests
are
in
data
mining,
machine
learning,
social
computing,
artificial
intelligence
investigating
interdisciplinary
problems
that
arise
in
many
real-world
data,
intensive
applications
with
high
dimensional
data,
a
disparate
form
such
as
social
media.
That
to
me,
sounds
like
big
data
and
dr.
Rand
Walsman,
who
is
currently
senior
information.
Scientist
at
the
RAND
Corporation
of
Santa
Monica,
so
r
&
at
Rand.
A
He
prior
to
joining
round.
He
was
acting
chief
technology
officer
software
engineering
Institute
in
Washington,
DC
of
Carnegie
Mellon
University,
and
before
that
he
did
a
five-year
tour
of
duty
as
program
manager
of
information,
innovation
officer
at
DARPA,
where
he
created
manage
the
social
media
and
strategic
communications
program
and
the
anomaly
detection
at
multiple
scales,
insider
threat,
detection
program
and
so
I
met
this
cast
of
characters.
These
merry
band
several
years
ago,
oh
through
a
mutual
colleague
dr.
A
Rebecca,
goolsby
who's
at
Office
of
Naval
Research,
and
we
put
on
a
workshop
at
the
Woodrow
Wilson
Center,
where
I
was
at
the
time
on
human
computation
and
had
a
group
discussion
on
what
is
the
role
of
anti
social
computing,
and
so
this
roundtable
kind
of
the
idea
for
that
sprung
from
that
meeting.
So
with
that,
I
would
like
to
hand
it
over
to
Kathleen.
C
Today
so
I'm
going
to
describe
to
you
some
of
the
findings
on
bots
and
fake
news,
that
from
our
group,
and
that's
been
done
by
the
students
and
myself
here
so
today,
the
warbeck
ideas
is
being
waged
in
the
cyber
mediated
environment.
That's
like
the
Wild
West
we
founded
vigilantes
like
anonymous,
are
actually
changing
and
self-policing.
The
environment
data
providers
like
Twitter
and
YouTube,
are
creating
policing
and
evolving
an
environment
that
is
changing
on
a
daily
basis
that
is
ripe
for
abuse
and
that
consensus
is
often
treated
with
greater
consequence
than
the
truth.
C
We
find
viral
marketing
strategies
being
used
to
actually
shape
the
conversation
and
to
build
communities.
Consider
the
following
three
facts:
first,
news
agencies
no
longer
have
the
financial
resources
to
do
deep
investigative
reporting,
so
they
rely
on
citizen
reporters,
media
releases
and
press
events
to
spread
the
news.
Social
media
has
changed
the
way
news
is
being
consumed
in
the
way
it
is
being
delivered
so
that
now
short
sound
bites,
140
character,
tweets
and
scroll
through
technology
are
directing
the
traffic
to
the
news
and
providing
only
high-level
information,
making
it
difficult
to
get
more
deep
information.
C
Trolls
and
bots
are
being
used
to
create
groups
and
spread
information,
and
they
actually
are
shaping
the
conversation
too
dense
community
creation
techniques
using
a
combination
of
social
network
analytics
and
machine
learning.
Our
group
has
done
the
number
of
studies
to
look
at
these
kind
of
issues.
For
example,
we
did
a
Twitter
study
where
we
actually
collected
information
on
Syrian
extremists.
We
began
with
a
set
of
known
members
of
Isis
and
used
those
to
scroll
out
and
find
a
set
of
other
individuals
who
are
engaged
in
the
same
conversation.
C
Many
of
those
individuals
are
became
known
as
members
of
Isis
and
others
were
thought
to
be
members
of
Isis
by
companies
like
anonymous.
We
kiddin
within
this
dense
community,
which
was
driven
by
new
events
that
were
going
on
at
the
time.
There's
this
bot
called
the
therapy
gnome
bought
the
fair,
be
known
bot
is
actually
a
collection
of
bots
that
work
together
to
keep
mentioning
and
retweeting
each
other
and
to
keep
mentioning
people
like
Conan
al-sabah,
who
is
a
who
was
taught
about
all
by
al-qaeda
as
one
of
the
crucial
motivators
for
foreign
fighters
tricking.
C
The
members
followers
of
also
by
two
following
the
core
ferebee
bought
those
supporters
were
then
given
a
lead
into
another
site
which
was
a
benefactor
for
the
fair,
be
bought
this
particular
benefactor.
It
was
actually
an
app
that
let
you
sign
up
for
and
don't
provide
donations
to
a
charity
for
the
children
of
Syria.
This
particular
charity
was
a
well-known
money
laundering
site
for
Isis,
so
this
particular
bought
and
BOTS
like
to
see
social,
bots
or
collection.
C
Bots
are
actually
often
part
of
these
dense
communities
and
they're,
actually
so
involved
in
it
that
they
actually
will
affect
how
those
groups
operate,
what
they
think
about
and
what
they're
saying
as
they
did
in
this
particular
case.
This
particular
study,
however,
cannot
be
replicated
many
of
the
members
of
this
community,
we're
actually
suspended
by
Isis,
and
the
data
is
no
longer
available.
C
The
next
time
I
want
to
describe
briefly
involves
two
different
studies,
and
it's
showing
more
about
what
these
social
BOTS
can
do
on
your
left.
You
actually
see
a
number
of
dense
communities.
These
are.
These
are
communities
who
are
engaged
in
political
conversations.
There
are
lots
of
different
political
agendas
and
do
various
kinds
of
things.
For
example,
one
of
them's
involved
in
pornography,
one
of
them's
following
various
other
kinds
of
things,
and
so
on,
but
social
BOTS
lurk
within
these
and
on
the
fringes
of
these
and
do
a
number
of
things
one.
C
They
they
take
to
different
communities
with
two
different
agendas
and
they
pull
them
together.
They
act
as
a
bridge
that
links
the
communities
together,
so
they
all
get
the
same
kind
of
information
once
they
built
the
bridge.
It's
hard
for
the
individuals
who
remembers
those
communities
to
tell
what
is
coming
from
one
community
and
what's
coming
from
another.
C
An
example
of
this
was
a
bot
that
worked
to
actually
combine
and
the
alt
right
with
an
in
that
event,
Jellicle
group
creating
a
common
linkage
between
them
and
then
tricking
Twitter,
effectively
into
rank
ordering
certain
tweets
from
certain
individuals
so
that
it
appeared
that
all
of
the
middle-aged
women
who
were
part
of
the
VF
Jellico
group
were
completely
in
favor
of
the
alt
right.
Behavior.
Dense
communities,
of
course,
are
used
once
they're
formed
often
use
things
lot
often
linked
be
on
Twitter.
It's
not
just
a
Twitter
story.
C
These
both
the
bots
and
the
in
members
of
these
communities,
will
actually
send
out
the
messages
to
and
get
messages
from.
A
variety
of
other
sites
youtube
just
pasted
a
facebook
or
examples,
and
many
times
what
they're
referencing
and
their
URLs
are
what
appear
to
be
news
sites,
many
of
which
are
actually
fake
news
sites.
C
What
this
does
is
it
creates
a
ground
swelling
what
appears
to
be
strong
support
for
common
ideas,
even
though
it's
not
you
know
it's
again
just
fiction,
so
you
get
a
resulting
mixture
of
truth
and
fiction
being
spread
to
people
so
fast,
or
a
series
of
selling
bites
that
there's
not
an
ability
for
them
to
go
for
them
to
check
it
and
causing
emotions,
then
to
skyrocket
out
of
control
in
these
groups.
So
these
kind
of
echo
chambers
that
are
formed
with
a
false
news,
propagating
Ralph
rhythm,
get
this
increase
in
emotions.
C
There
can
be
consequences
far
beyond,
of
course,
just
opinion
formation.
One
of
our
other
studies
show
found
that,
as
you
got
these
groups
echoing
out
of
control
and
leading
to
high
negative
sentiment
on
Twitter,
that
you
would
get
follow-ups
in
actual
cyber
attacks
on
the
country.
So
if
you
have
these
echo
chambers,
basically
forming
they
start
saying,
o
rushes
bad
rushes
bad,
then
you
will
see
an
upsurge
of
cyber
attacks
on
Russia.
So
that's
an
example
of
some
of
the
kinds
of
behaviors
that
we're
seeing
the
ways
they
be
go
beyond
it.
All
in
all.
C
What
we're
saying
is
that
these
social
bots,
these
collection
of
bots,
that
act
as
a
group
well
will
come
in.
They
will
build
communities
and
direct
them
toward
particular
accounts
causing
those
accounts.
You
know
to
get
more
followers
and
also
often
getting
you
to
apply
to
be
part
of
it.
They
may
get
you
to
come
on
to
their
site
to
allow
them
to
tweet
from
your
site,
and
so
you
become
part
of
the
supposed
to
ground
swelling
of
support,
and
you
may
not
even
know
that.
C
That's
coming
from
your
account
because
of
the
frequency
with
which
the
tweets
are
coming
out:
they
build
bridges
between
separate
communities,
making
them
appear
like
a
common
community.
They
develop
echo
chambers
in
which
emotions,
REM,
rapid
and
so
on.
Standard
approaches
for
finding
bots
do
not
cannot
find
these,
and
many
of
the
ways
in
which
data
by
Twitter
is
normally
collected,
prevent
people
from
seeing
that
these
exist
new
data
collection,
new
techniques
are
needed
to
find
them.
D
You
thank
you,
explain,
grace
thank
you,
so
Thank
You
Kathleen
for
putting
the
nice
segue
between
or
you
have
been
doing,
and
what
I'm
going
to
talk
about
and,
as
you
can
see
here,
the
cover
is
basically
from
the
wired
magazine.
This
was
a
few
weeks
ago
and
they
had
actually
interviewed
a
Macedonian
teenager
where
the
average
salary
monthly
salary
is
371
dollars.
D
But
this
guy,
just
by
willing
to
fake
news
blog
sites,
earn
sixteen
thousand
dollars
within
four
months
and
which
is
the
main
motivation
for
him
and
some
of
his
peers,
who
are
doing
the
same
work.
What
they're
doing
is
they're
not
even
creating
the
content,
they're,
just
lifting
the
content
from
some
all
tried,
or
some
extremely
conservative,
fake
new
sources,
and
they
take
those
content
and
put
it
on
their
docksides.
D
The
impact
that
this
has
has
been
felled
by
the
survey
that
was
done
by
a
Pew
Research
Center,
which
mentioned
64
so
I,
34%
Americans
trust
the
information
they
get
from
social
media
and
fourteen
percent
Americans
considered
that
in
mation
as
very
trustworthy,
very
important
right,
which
is
a
heat
percentage.
When
you
think
in
terms
of
the
electorate's
1%
make,
could
make
a
huge
difference
there,
so
our
work
is
basically
looking
at
how
blogs
are
used
and
framing
a
narrative
and
shaping
the
political
discourse
where
there
are
millions
of
bloggers.
D
How
our
only
very
few
of
them
become
very,
very
successful,
a
very,
very
popular,
very,
very
influential.
They
certainly
do
not
represent
the
entire
electorate.
They
are
highly
under
represented
the
population,
yet
they
are
able
to
drive
the
discourse
through
various
ways
that
we
are
going
to
discuss
next
and
make
sure
that
they
can
dominate.
The
voices
of
many
Internet
is
the
first
medium
in
history
that
has
native
support
for
group
communication
and
conversation
at
the
same
time,
which
means
that
if
you
look
at
television
space,
radio
space,
it's
one
too
many.
D
If
you
talk
about
telephones,
that
is
one
to
one.
When
you
talk
about
social
media,
it
is
many
to
many
which
brings
all
these
conversations
together.
Helps
in
developing
this
discourse.
The
political
communication
further
internet
is
Internet
has
become
the
mode
of
carriage
for
all
the
other
media,
meaning
everything
comes
to
internet
everything,
digital
migrates,
to
the
internet,
space,
all
the
videos,
all
the
memes
images
they
are
documented.
They
are
curated
on
it,
they're
further
distributed,
which
is
the
third
point
here.
D
Consumers,
the
previous
consumers,
have
now
become
the
producers,
but,
more
importantly,
distributors,
so
I
see
something
I
can
easily
and
very
conveniently
distribute
to
all
my
friends,
all
right,
which
can
be
further
propagated
or
shared
and
can
be
seen
by
many
others
right.
So
here
the
need
is
not
just
for
the
financial
incentive.
It
is
also
to
feed
the
hunger
that
we
have
for
the
political
or
the
online
political
hyper-partisanship
that
we
see
in
internet
spaces.
D
Some
of
the
findings
I've
shown
here
we
have
done
some
studies
with
Kathleen
Rebecca
I
one
and
many
of
the
researchers
on
studying
the
blog
blogosphere
space
is
in
Ukraine
and
Russia
conflict,
and
what
we
have
observed
is
some
of
these
images
that
are
on
your
left
that
were
propagated
on
some
of
the
very
high
influential
bloggers
and
they're
blocking
spaces.
So
these
are
the
memes
that
travel
on
blogs
and
further
shared
through
sources
or
through
platforms
like
gab.
D
How
and
if
you
have
heard
of
something
called
gap:
AI,
okay,
it
is
an
invitation-only
social
network,
but
they're,
very
liberal,
in
giving
these
invitations
or
in
handing
out
these
invitations,
but
themselves
proudly
say
that
we
are
the
ones
who
are
rejected
by
Twitter,
so
we
own
this
free
speech
space.
So
if
you
have
time,
please
feel
free
and
check
out
the
gap,
dot,
AI
platform,
the
others
are
readied
discord.
Discord
is
like
a
social
media
channel
for
gamers
slack
for
gamers.
D
If
you
will,
if
you
have
heard
of
what
slack
is
its
lack
for
serious
gamers,
so
you
have
developed
these
tools
called
block
trackers
and
focal
structures,
analysis
which
track
these
blocks
and
identify
who
are
the
influential
bloggers?
What
are
the
leading
narratives
in
these
spaces?
How
fringe
there
it
has
become
mainstream
and
so
on?
What
are
the
blog
that
subscribe
to
those
ideologies
and
how
these
are
further
shared
in
spaces
like
Twitter,
Facebook,
Vkontakte
and
so
many
other
platforms?
Mobile
vector
analysis
is
another
tool
that
tries
to
identify
the
coordination
of
this
information
campaign.
E
C
E
Yes,
hey
all
right,
so
it's
obviously
social
media
data
is
big
and
it's
so
big
and
and
also
everywhere.
But
we
know,
big
data
is
not
just
a
simply
big
right.
So
usually
we
talk
about
to
like
a
4,
vs
and
social
media.
Has
all
data
has
all
the
four
beats?
Ok,
then,
people
would
like
to
lot
of
researchers
would
all
to
use
social
media
intelligence
to
study
human
behavior
and
to
understand
what's
going
on
in
the
world,
so
it
provides
a
new
lens.
I
have
been
gained,
picked
there
for
maybe
20
some
years.
E
So
when
so,
my
original
research
is
about
teacher
selection
to
how
to
lie
can
reduce
the
big
data
to
manageable
size,
but
conventional
methods
really
find
that
they
are
the
bottleneck
and
big
data
challenges,
traditional
think
so
now,
for
example,
I'm,
even
at
a
CMU
still
like
it's,
not
everything
is
equipped
abide,
big
data
but
I
very
soon.
Okay,
even
the
room
like
oh.
If
people
ask
the
smartest
person
in
the
room
is
so
anybody
has
the
answer
in
the
room,
though.
Okay,
this
smart
is
the
person
who
cookie
is
the
room.
Okay.
E
This
is
the
part,
2
scaring
pot
and
so
bear
say
you
have
heard
about
this
computational
thinking,
but
now
we
also
have
seen
the
big
data.
Big
data
has
produced
the
so
much
influence
and
impact,
and
it
seems
like
when
we
have
big
data.
We
can
solve
all
the
problems.
You
have
heard
about
the
deep
learning
right
and,
of
course
everybody
is
now
doing
machine
learning,
but
you
have
also
heard
about
al
Oliver,
scaring
news
about
AI
like
a
a3
intelligence.
It's
a
beats
like
a
human
conscience.
E
Ain't
a
scene
girl,
hooker
djibouti,
it
seems
like
AI
is
going
to
overtaking
everything,
but
that's
not
true.
Actually,
ok,
I
will
discuss
more
later,
but
in
this
sin
in
this
context,
anti
social
computing
come
hey.
I
can
make
a
boat's
more
intelligent
and
socially
social
media
intelligence,
less
intelligent
and
now
nexus
light
parts
and
indifferent
to
forums
and
very
the
roles
they
are
everywhere.
E
E
It
hasn't
a
lot
of
a
negative
impact.
If
you
heard
about
a
temperature-
and
I
think
recently,
they
posted
that
I
can
they
help
to
trump
to
wing
her
the
presidential
election.
And
but
if
you
really
read
about
this,
you
will
see
big
data
alone,
actually
couldn't
wing
everything.
If
you
read
about
this,
the
blog
about
a
news
report
about
the
rise
of
the
open,
I,
weaponized
AI
propaganda
machine
and
it
really
painted
a
picture
light.
E
Big
data
can
solve
everything,
but
now
I
would
like
to
move
on
to
the
last
slide
when
AI
meets
social
media
intelligence.
Ai
is
a
double-edged
sword,
so
it
can
help
us
too.
Okay
and
because
human
are
easy
to
fool
already
mentioned,
we
can
try
to
do
to
AI
system
to
battle
them,
and
nowadays
we
also
heard
about
to
that.
The
bots
are
fighting
parts
they
can.
The
defining
can
last
for
years,
okay,
but
obviously,
if
falsifying
parts
that
means
they
are
not
intelligent
enough
right.
E
The
last
point
I
would
like
to
make
is
we
need
to
enable
user
control
the
information
filters
and
checkers
the
way
to
counter
anti
social
computing
is
really
to
promote
and
build
divers
collective
intelligence.
I
would
like
to
emphasize
divers
and
diversity.
Diversity
is
so
important
or
us
to
detect
verify
and
exposing
misinformation,
and
with
with
very,
very
important
okay,
we
cannot
blindly
can
town
big
data
and,
at
the
end
of
dry,
to
mention
to
tutorials
we
give
last
year
or
they.
E
F
Yes,
okay,
so
there's
a
second
I
training.
Oh
oh
yeah,
so
there's
certainly
been
a
lot
of
discussion
in
the
news
lately
about
fake
news.
Well,
and
when
you
listen
to
all
these
discussions,
you
get,
you
might
have
the
feeling
that
fake
news
is
something
new.
Well,
fake
news
is
actually
really
really
old
and
the
whole
idea
of
doing
this
is
really
really
old
and
it
affects
the
entire
information
environment
and
what
the
technology
today
allows.
F
All
sorts
of
people
to
do
is
to
get
in
and
modify
the
environment
surrounding
environment
in
people's
heads
and
so
on,
and
fake
news
is
an
example
of
what
used
to
be
charmed
and
what
still
is
turned
I
mean
a
KGB
term
of
called
active
measures,
and
you
might
have
heard
the
term
active
measures
thrown
around
and
I
just
thought.
I
would
somebody
s
today
that
we
can
you
tell
us
in
60
seconds,
you
know
what
what
active
measures?
What
does
that
really
mean?
So
here's
a
little
statement,
I
wrote
up
and
I
just
thought.
F
I
would
throw
that
out
to
you.
It
has
active
measures.
Are
techniques
used
to
manipulate
groups
of
people
anywhere
from
10
to
a
billion
using
information
as
a
weapon
into
doing
whatever
the
perpetrator
wishes,
while
making
those
people
think
it
was
their
idea?
It
does
not
matter
whether
the
information
is
true,
half
true
or
false,
as
long
as
it
gets
the
job
done.
Active
measures
include
use
of
forgeries,
false
front
organizations
targeted
messaging
campaigns,
using
whatever
combination
of
I've
said
Nene
that
will
be
most
effective
on
the
targeted
audiences.
F
Active
measures
are
extremely
effective
for
everything,
from
large-scale
criminal
activity.
The
mass
marketing
campaigns
to
massive
scale,
political
and
social
influence
operations.
There
are
active
measures,
mercenaries
who
have
no
allegiance
and
no
moral
scruples
at
some
point
and
I
think
we're
pretty
much
almost
there
active
measures
completely
dominate
the
information
environment
and
objective
truth
and
reality
become
almost
meaningless
or
relevant
concepts.
At
that
point,
Jack
London
same
is
quote
only
the
strongest
wings
gives
way
to
the
tube
to
the
new
statement.
F
Only
the
best
bullshitter
wins
you
know
and
I
would
refer
you
to
a
spectacular
little
book
written
by
Professor,
Harry
Frankfurt
at
Princeton
called
on,
and
in
this
book
he
makes
a
very
important
distinction
and
the
distinction
between
a
liar
and
a
or
a
con
man.
So
a
liar
is
somebody
who
will
tell
you
try
to
convince
you
of
something.
If
they'll
try
to
convince
you.
Something
is
true
that
they'd
believe
that
day
themselves
believed
to
be
false.
F
It
doesn't
matter
whether
it
really
is,
but
they
believe
is
footballs
and
they're
going
to
try
to
convince
you.
It's
true
a
con
man
or
are,
on
the
other
hand,
has
no
regard
for
the
truth
or
falsity
of
what
they're
saying
it's
irrelevant.
The
only
thing
that's
relevant
is:
is
it
going
to
get
you
to
do
what
I
want
you
to
do?
That's
the
only
real
question
and
we've
come
to
a
situation
where
our
entire
information
environment
is
basically
dominated
by
this
kind
of
conversation.
So
what
you
do
about
that?
F
A
The
nitty-gritty
of
the
research
you're
doing
and
questions
there.
One
of
the
things
that
came
up
in
the
discussion
is
with
the
prominence
of
this
in
the
news.
Lately
we're
now
seeing
a
rapid
increase
in
the
amount
of
research
in
this
area,
and
it's
not
necessarily
all
good
research,
because
there
may
be
a
lack
of
understanding
of
the
limitations
of
the
tools
that
were
using
in
limitations
of
the
social
media
platforms.
So
so
for
the
panel
he
here.
How
does
the
social
media
platform
affect
the
data
that
you're
using?
And
how
do
you
address
that.
C
So
one
of
the
things
too,
that
you
have
to
recognize
the
mostly
social
media
is
that
the
data
providers
themselves
have
put
constraints
on
what
information
is
provided
and
how
it
is
provided
and
how
much
you
can
get
access
to.
So
a
lot
of
this
work,
that's
out
there
on
Twitter,
for
example,
doesn't
even
recognize
the
fact
that
when
you
tweet
and
he
retweets
you
and
I
retweet
him,
my
retweet
of
you
goes
to
him.
So
there's
no
way
to
trace
the
path
of
the
real
diffusion.
Okay,
that's
one
thing.
C
Secondly,
a
lot
of
the
data
is
not
geo
tagged,
so
understanding
where
it
is
coming
from
is
extremely
difficult.
Thirdly,
the
data
sampling
which
most
people
and
most
secondary
data
providers
use
which
is
doing
keyword,
search
or
using
these
bounding
boxes
like
geographic
things,
don't
wreck
it
they're
there
they
think
they're
getting
random
samples
within
those,
in
fact
they're,
not
at
all
random,
because
the
data
that's
being
collected
as
being
vetted
in
a
certain
ways.
It
actually
gives
you
bias
samples.
C
So,
like
the
results,
we
gave
you
if
we
had
used
the
standard
way
people
collect
data
from
Twitter,
which
is
just
using
keywords.
We
would
never
found
those
BOTS.
We
had
to
do
a
deep
dive
onto
the
conversation,
so
you
really
have
to
understand
the
technology
and
understand
the
way
it's
api's
are
shaping
the
data
that
you're
getting
and
then
you
have
to
use
anti
biasing
data
techniques.
Do
an
ad
I.
D
D
So
it's
not
representative
sample
that
we
get
through
AP
is
there
are
other
workarounds
that
if
you
have
access
to
the
fire
hose,
then
you
can
get
larger
datasets,
but
those
also
have
certain
limitations,
such
as
location
information
is
not
available,
and
if
you
want
to
geo
inferencing,
so
it
may
not
be
as
accurate
as
possible.
The
other
thing
that
I
would
like
to
highlight
here
is
most
of
the
research
that
I
have
come
across,
looks
at
these
platforms
in
isolation.
D
So
if
you
are
doing
research
and
Twitter
you're,
focusing
only
on
Twitter,
if
you're
doing,
research
and
blogs
are
focusing
only
on
blogs
or
YouTube
and
such,
however,
there
is
a
need
for
a
more
concerted
effort.
Then
you
can
analyze
the
whole
entire
social
media
ecosystem
as
a
whole
organism
and
see
how
things
evolved
in
social
media
space,
maybe
just
bring
in
a
couple
of
in
a
couple
of
social
media
channels
such
as
blogs
and
Twitter
blogs
in
YouTube
and
so
on.
D
Where
you
can
understand
how
a
narrative
is
built,
how
it
is
shaped
and
then
further
how
it
is
disseminated
or
travels
on
the
social
media
platforms.
So
looking
at
multiple
channels
is
very,
very
critical,
especially
when
you're
talking
about
things
like
misinformation,
fake
news,
propaganda
and
so
on.
One.
B
E
All
right
about
the
platform:
okay
I-
can
start
from
the
individuals
first.
So
how
many
of
you
use
the
personalized
information
filters?
Oh,
how
ray
I?
Can
the
news?
Do
you
have
automatic
news
feed
for
you?
Okay,
so
with
the
big
data
with
the
big
thing
and
we
are,
we
are
inundated
by
so
much
information,
so
it's
impossible
for
us
to
look
at
all
of
them.
E
So
the
info
that
that
means
the
platform
can
feed
you
the
information
they
want
to
reach.
That's
number
one.
But
if
you
look
at
the
from
the
platforms
point
to
be
it's
even
more
scary,
because
it's
a
steer,
it's
a
machine
operated
right,
so
they
received
the
Ocelot
of
information
and
how
could
they
rank
and
the
recommend
that's
why
this
is
the
danger
has
imposed
by
the
box
and
the
box.
They
can
game
the
system
and
they
can
make
a
fake
news
as
random
ation
on
the
top.
E
Then
if
you
use
the
filter-
and
you
will
only
receive
those
the
machine,
the
platform
site
that
they
consider,
they
are
trendy-
they
are
hot
and
they
feed
to
you.
So
this
is
this
also
relate.
This
is
also
related
to
what
I'm
talking
about
the
diversity.
So
then
the
lawyer
use
the
filter
and
less
diverse.
It
is,
and
then
you
are
here
the
to
the
news
and
the
information
they
feed
you.
So
that's
the
problem,
and
now
we
need
to
figure
out
a
way
I.
Just
still
I
also
mentioned
with
that.
E
A
B
F
I
think
that
so
so,
in
a
part
of
the
thing
with
the
data
is
that
you
you
need
to
it's
not
only
the
contents
themselves,
but
you
need
to
be
able
to
put
the
data
in
the
context.
So
very
few.
People
I
should
talk
about
the
larger
context,
social
media
itself
or
all
the
platforms,
and
you
know
in
total-
don't
exist
in
a
vacuum
there,
they're
embedded
in
a
larger
reality,
and
so
in
order
to
actually
use
the
platforms
correctly
or
make
or
exploit
them
in
a
reasonable
way.
F
You
have
to
be
able
to
also
exploit
the
contour.
You
know
take
into
account
the
larger
context
in
which
they
exist,
and
so
far
people
aren't
really
doing
that.
So
that's
one
issue
that
you
need
to
worry
about,
so
that's
a
limitation.
The
limit,
81
limitation
is
people
are
not
using
context
in
which
the
platforms
are
embedded
in
in
the
largest
ends.
Another
another
thing
is
people
almost
of
the
analytics
to
date
have
been
sort
of
check:
centric,
okay,
but
that's
not
the
future.
The
future
of
social
media
really
is
multimedia.
So
it's
you
know.
F
If
you
look
at
snap,
for
example,
snapchat
snapchat
is
an
example
of
the
first
step
in
the
direction
of
multimedia
centric,
social
media
and
right
now
there
aren't
really
any
really
good
analytic
tools
available
to
do
social
media
analytics
on
multimedia
data.
It
fits,
but
you
know
you
can't
rely
on
the
metadata.
You
can't
rely
on
anything.
You
have
to
actually
work
with
with
the
media
itself,
with
images
with
video
with
a
speech,
and
you
have
to
find
a
way
to
actually
integrate
those
together
because
that's
clearly
the
trend
for
the
future.
F
So
so
that's
really
so
that's
another
issue.
So
that's
not
a
question
of
the
platform
relates
a
question
of
you
know
what
the
analytic
technology
behind
it
is,
but
but
still
I
would
say,
the
biggest
limitation
of
the
platforms
is
not
being
able
to
take
context
into
account,
and
you
know
you
want
to
also
be
able
to
as
been
pointed
out
these
things
that
they're
all
biased
in
different
ways,
and
you
also
have
to
have
technology
to
sort
of
black
box
them
and
understand
to
sort
of
reverse
engineer
or
what
bias
are
they
using?
F
E
I
may
I
got
something:
okay,
so
yeah
and
add
this
year's
a
trippy
I
conference.
That's
a
one
of
the
two
prominent
to
AI
conferences
in
the
world
and
we
presented
their
tutorial.
We
present
a
tutorial
on
social
data
bias
in
machine
learning,
an
impact,
evaluation
and
correction,
and
this
is
a
if
you
visit
to
my
web
page
and
just
search
my
name
either
use
Bay
Google.
You
will
be
able
to
find
my
home,
create
the
homepage
just
roll
down.
E
It's
a
trip,
eh
I
2017
tutorial
on
social
data
bias
and
it
discusses
many
biases
due
to
basically,
we
talked
about
at
the
platform.
One
is
about
the
platform.
The
second
one
is
about
the
manipulation.
So
actually
people
can
attack
the
platforms
to
make
their
data
more
available.
Then
it's
supposed
to
be
that's
all.
A
C
C
Cup,
an
additional
one,
okay,
so
one
of
the
additional
examples
that
we
saw
was
in
looking
at
what
were.
We
were
at
that
point.
Calling
ukrainian
elves
as
opposed
to
the
russian
trolls,
get
have
fun
in
science
and
we
were
actually
able
to
find
some
of
these
bots
that
were
actually
actively
engaged
in
discussions
that
we're
linking
various
groups
from
different
communities
it
in
russia
and
so
on
with
various
communities
in
the
US
and
we're
actually
directing
them
to
think
about
and
talk
about
various
extremist
ideas.
C
That's
one
thing:
there's
another
example:
in
another
case,
again
back
with
the
Ukrainians
up,
there
is
one
group
where
they
were
actually
using
these
BOTS
to
support
their
discussions
and
that
led
the
people
to
start
communicating,
and
then
they
brought
in
a
fake
images
spread
by
BOTS
and
for
it
spread
by
people.
And
some
of
those
images
were
you
know:
lots
of
pornography
trying
to
get
young
men
to
recruit
be
recruited.
C
In
other
cases,
they
were
using
things
like
images
of
soldiers,
suppose
they
were
supposedly
US
soldiers
who
were
raping
someone
else,
and
they
were
actually
photoshopped
images,
and
so
they
were
transmitting
these
images
and
getting
people
riled
up
by
them.
Even
though
the
images
were
false,
two
examples
so.
A
Nitin
you
had
talked
to
me
a
little
earlier
before
we
got
started
about
some
of
the
complexity
of
how
how
to
track
what's
going
on
in
blogs
and
how
different
blogs
are
influencing
the
the
conversation.
Could
you
step
me
through
just
maybe
an
overview
of
the
methodology
you
use
and
may
be
using
an
example
sure.
D
So
logs
is
a
very
complicated
space
as
we
were
discussing,
and
the
main
reason
is
because
you
won't
find
an
explicit
network
in
blogs
in
essence
that
when
you
look
at
Twitter
space,
you
have
followers
and
friends
and
you
can
build
a
diffusion
graph
or
a
social
network
communication
network,
and
you
can
understand
who
is
tweeting
or
retweeting
on
whom
and
how
the
information
memes
are
travelling
in
blogs.
However,
you
have
separate
web
pages
and
they
are
very
rarely
connected
to
each
other
right.
D
B
D
Highly
meticulous
and
careful
and
mostly
involves
human
effort,
where
you
start
with
a
very
carefully
selected
keywords
and
start
snowballing.
Your
keywords
through
the
information
that
you
have
discovered
right,
just
look
into
it
relevant
entities
that
may
be
helpful
to
get
to
the
the
right
blogs.
Once
you
do
that,
then
you
start
looking
into
something
called
cyber
forensic
or
social
cyber
forensics,
where
it
can
help
you
build
that
underlying
network
among
these
blogs.
D
So,
as
I
mentioned,
there
is
no
explicit
network
on
logs,
but
you
can
use
things
like
who
is
registered
on
these
blogs
if
they
have
used
the
same
email
addresses
on
various
other
platforms
like
Twitter
like
Facebook,
and
you
use
these
techniques.
In
many
cases,
Ukraine
Russia
are
conflicts
and
NATO
exercises
that
have
we
done
and
how,
after
media
puts
a
hand,
hire
us
or
anti-western
spin
on
those
on
those
exercises,
Isis
related
stuff.
So
there
we
are
looking
at
how
these
blocks
are
somehow
connected.
D
You
can
explore
how
these
bloggers
are
using
traffic
analysis
tools
or
traffic
or
reps
website
crackers,
and
each
of
these,
where
crackers
have
unique
codes
which
are
associated
with
a
unique
email
address,
and
you
can
use
those
to
identify
if
a
blogger
has
multiple
blog
sites
and
what
they're
talking
about
on
one
blog
may
be
different
than
the
other
and
how
you
can
piece
all
that
information
together
to
build
a
more
comprehensive
profile
for
a
blogger
or
someone
who
is
of
interest.
So.
B
D
On
the
bottom
right
corner,
they
have
a
tool
or
cocoa
structured
analysis,
and
this
is
a
very
small
network
of
blogs.
These
are
not
going
to
account.
These
are
blog
bloggers
who
are
connected
through
such
mechanism
that
I
just
described,
and
these
connections
let
us
to
identify
who
are
the
most
coordinating
set
of
bloggers
in
an
information
campaign-
and
this
was
one
of
the
one
of
the
bloggers
here
that
was
identified-
was
Graham
Phillips,
who
was
supposed
to
be
an
internet
celebrity
when
he
went
to
Ukraine
and
through
his
live
blogging
style.
D
He
shot
to
instant
fame
and
other
people
started
following
him
and
started
believing
what
he
had
to
say.
But
his
rhetoric
was
all
through
Russian
and
anti
Ukraine
and
high
best,
and
he
was
quickly
identified
by
the
Indian
government
and
he
was
banned
from
entering
evening.
He
was
originally
a
British
citizen
and
then
he
moved
to
Russia.
Now
he
has
a
completely
new
blogging
identity
and
he
has
a
new
blog,
in
fact,
new
blogging
site
and
he
blocks
from
Russia
his
equipment.
His
computers,
whomever
owns
everything,
was
seized.
He
every
NATO
conference
every
NATO
summit.
D
D
This
was
identified
by
the
two
that
we
have
developed.
This
focus
after
analysis
based
on
how
he
has
been
coordinating,
how
he
has
linked
through
all
that
methodology
and
when
we
shared
this
with
our
liaison
at
NATO,
and
he
practically
jumped
from
the
chair
is
a
comedy
movis.
You
know
the
context
behind,
while
you
are
so
surprised,
so
he
shared
all
of
that
information
with
us
and
that
led
us
to
believe.
Yes,
this
is
something
that
we
need
to
follow.
A
You
have
one
question
on
the
twittersphere
from
from
India
Joe
Cole
ho,
so
he
says
you
mentioned
that
there
are
different
platforms
emerged
to
serve
different
user
needs,
but
have
now
morphed
into
becoming
more
news
feed.
He
says
what
happens
when
some
of
these
merge
into
Kenny's
merge
into
a
platform
and
become
harder
to
control.
That's.
D
Interesting
so
as
some
of
the
other
panelists
have
also
mentioned,
that
choice
of
platform
has
to
be
well
also
with
the
context
that
lies
behind
it,
so
it
does
make
it
very
complicated
to
analyze
joint
forms
of
analysis.
You
looking
at
multiple
platforms.
However,
if
we
follow
a
scientific
and
rigorous
methodology,
we
can
eat
these
things,
our
bikinis,
these
individual
communications
out
from
one
space
to
the
other
and
then
build
a
diffusion
network
or
communication
network
of
some
sorts,
knowing
how
information
is
travelling
from
one
platform
to
the
other.
D
Also,
there
is
a
caveat
here
because
they
have
been
talking
very
specifically
about
platforms.
Now
we
always
keep
observing
new
platforms
on
the
horizon,
so
we
have
something
called
gap
dot.
I
we
have
read
it
and
the
methodology
that
they
are,
we
hope
to
develop,
should
be
genetic
enough
or
should
be
grounded
in
theory,
so
that
we
can
take
those
methodologies.
Take
those
software
tools
to
other
platforms.
It
should
not
be
very
specific
through
one
platform
and
couple
of
platforms,
and
then
it
breaks
down
when
you
take
it
to
other
platforms.
A
B
That
you've
done
a
lot
of
work
just
purely
on
the
sort
of
looking
at
the
graph
and
the
network
analysis,
and
you
can
often
find
sort
of
various
behavior.
Do
you
think
that's
a
fairly
effective
way
of
looking
for
these
botnets
and
other
sort
of
artificial
injection
of
misinformation,
or
will
those
people
just
get
more
clever
and
and
hide
themselves
better?
As
you
be
more
aggressive
about
it,.
C
So
if
we
just
look
at
like
like
say
the
mentions
network,
whose
matching
whom
eventually
I
think
that
won't
be
where
you'll
see
these
groups,
we
are
seeing
them
there
now,
but
even
now
we
have
to
find
them
in
what
they're
talking
about
the
hashtag
network
and
the
mention
graph
at
the
same
time.
So
we
have
to
use
multi-dimensional
graphs
to
find
them,
and
then
we
have
to
look
at
them
over
time.
So
I
think
that
will
continue
to
be
effective
as
these
structures.
C
A
F
Well,
you're,
the
only
additional
comment
I
would
make
is
that
you
know
when
you're
looking
for
things
different
types
of
activity,
it
isn't
what
you're
tryna
you're
not
trying
to
find
a
false
statement
or
a
true
statement
or
whatever
you
know
what
you're
trying
to
look
for
is
intent.
What
is
the
intent
of
the
narrative
that's
being
pushed
so?
F
First,
you
need
to
be
focused
on
narrative
detection,
all
right
and
then
there
it
is
today,
but
that
when
they're
done
by
people
who
really
know
what
they're
doing
I
actually
transmedia
the
transmedia
storytelling
transmedia,
a
narrative
pushing
is
really
is
what's
becoming
really
important
right.
So
these
are
people
pushing
narratives
across
multiple
types
of
platforms,
multiple
types
of
media,
and
you
have
to
be
able
to
track
that
collectively.
Otherwise,
you're
really
missing
something.
That's
one
issue.
F
The
other
issue
is
you
have
multiple
players,
so
it
isn't
just
one
group
or
one
person,
that's
trying
to
push
a
narrative
that
you're
you're
trying
to
detect,
but
there
are
many
happening
simultaneously,
and
so
you
have
to
try
start
to
think
about.
You
know
what
are
the
dynamics
of
different
people
who
are
trying
to
seize
control
of
the
narrative
in
different
topics.
So
so
these
are
things
that
you
need
to
keep
into
account.
So
the
problem,
that's
that's!
That's
one
problem
in
limiting
yuvutu
one
platform
or
one
type
of
messaging.
E
I
would
like
to
okay
Oh.
What
would
one
point
I
would
like
to
add
is
yes,
it
is
true
that
buds
are
becoming
more
intelligent
and
more
difficult
to
detect
and
in
the
in
the
past,
it's
very
easy.
Just
by
looking
at
a
simple
network
structure
like
a
hominid,
how
big,
how
lonely
they
are,
you
can
do
it,
but
now
because
they
become
more
intelligent,
so
we
can
have.
We
have
to
use
like
more
nuanced.
E
The
methods,
for
example
in
one
of
the
projects
we
worked
on,
is
about
the
sarkas
in
detection
on
Twitter,
so
we
found
that
early
days.
We
just
need
to
look
at
each
treat
butter.
Then
it
becomes
less
effective.
You
have
to
look
at
it
as
the
random
mention
narratives.
You
have
to
look
at
the
a
sequence
of
tweets.
Basically,
you
have
to
look
red
to
the
conversation.
Then
you
will
be
able
to
more
like
effective
to
detect
sarcasm.
E
C
I
just
wanted
to
pick
up
on
rands
point
and
point
out
that
one
of
the
ways
we
know
we
find
these
dense
communities
with
graph
detection
techniques
and
so
on.
There
are
linking
out
to
cross
media
to
multiple
platforms
and
the
most
effective
ones
of
them
in
the
are
actually
linking
to
some
of
the
same
different
sites.
So
you
connect
so
they'll,
actually
all
linked
to
the
same
youtube
channel
or
the
same
facebook
page
or
the
same.
You
know
whatever
page
and
a
lot
of
the
those
you
know
conversations
will
look
like.
C
G
G
G
G
F
That
Rand,
yes,
I,
did
bill.
Thank
you
I.
So
so
here's
a
big
deal
right.
So
people
talk
about
countering,
but
it
isn't
about
it's.
Not
it's
not
trying
to
counter
something
as
much
as
it
is
as
finding
is
providing
a
no
viable
alternative.
You
need
to
replace
one
narrative
within
other
narratives
exist.
Mythology
exists.
F
All
of
these
things
are
just
because
they
fill
in
need
of
some
sort
right,
so
you
can't
just
discredit
one
or
try
to
fight
against
one
without
providing
the
main
way
you
do
is
by
providing
an
alternative,
and
then
the
question
becomes.
How
do
you
actually
inject
the
alternative
and
get
it
to
replace
the
one
you
don't
like
and,
of
course,
you're
doing
that
in
the
environment,
where
there
are
multiple
actors
trying
to
do
exactly
the
same
thing,
so
you're
saying
you're,.
F
Similar
techniques,
it's
actually
providing
a
replacement,
that's
the
only
thing
you
can
really
do
I
mean,
of
course
you
could
try
to
discredit
the
other
people,
but
but
but
it's
much
more
effective
to
provide
a
replacement
because
remember
the
other,
the
other
guy
who's
trying
to
push
a
narrative
he's
succeeding
because
he's
selling
something
people
want
to
buy
I
mean
the
absolutely
essential
ingredient
of
a
good
con
right.
One
famous
con
man
said
was
to
have
a
willing
victim
all
right,
so
narratives
are
successful
because
they're
selling
something
that
people
fundamentally
want
to
buy.
F
So
if
you're
not
happy
with
that
one,
then
you
have
to
provide
an
alternative
that
they're
going
to
be
even
more
happy
to
buy.
So
that
means
that
you
have
to
understand
what
is
exactly
the
need
that
you're
trying
to
fulfill
that
you
know
that
somebody's
attacking
you
know
you
can't
be
making
this
stuff
up
out
of
nothing
for
no
reason
right,
so
you
always
have
to
build
on
something
which
exists.
Otherwise
you
never
get
anywhere.
So
yes,
so
it's
it's
finding
a
replacement
and
saying
yes,
look!
F
E
I
I
would
like
to
add,
is
in
order
to
effectively
handle
misinformation.
Counter
countering
counter-argument
want
to
be
effective.
Basically,
people
realize
that
you
have
to
just
expose,
but
I
don't
engage
and
that's
the
that's
one
of
the
best
strategies,
and
so
as
ran
mentioned.
I
also
mentioned
this
in
my
three
slides.
E
We
need
to
detect
the
important
point
is
we
need
to
detect
a
misinformation?
Then
we
should
offer
alternatives
to
for
karen's
occasion
and
and
also
point
out
to
the
sources
of
facts.
So
it
is.
It
is
very
difficult
to
change
people's
mind.
Usually
it's
a
there's,
something
called
a
group
of
polarized
polarization
right.
So,
as
a
matter
of
fact,
if
you
engage
in
the
discussion,
the
discussion
becomes
so
heated,
but
at
the
end
the
two
parties
will
be
farther
apart
to
not
think
they
get
a
closer.
E
F
So
there
was,
it
was
a
great
example
in
the
last
election
right
when
Hillary
Clinton
called
basically
called
people
who
disagree
with
our
deplorable.
Well,
you
know
what
that's
not
a
way
to
win
friends
and
influence
people
all
right,
though
it
is,
isn't
it
attacking
people
attacking
something
that
they
fundamentally
believe
that
that
doesn't
work
I
mean
it
can
never
work
so
just
by
just
trying
to
discredit
something
that
doesn't
work
even
exposing
something
even
that
doesn't
really
work.
F
A
F
F
Not
only
are
we
terrible
but
we're
terrible
on
purpose,
okay
and
with
turbos
on
purpose,
because
the
United
States
government
basically
doesn't
want
to
have
anything
to
do
at
issue.
So
when
I
say
Whedon,
we
I'm
talking
about
in
the
government
right.
So
if
you,
if
you
refuse
to
even
participate
well
you're,
not
going
to
be
very
good.
So
that's
where
we
are.
C
The
other
thing
is
that
what
we
haven't
talked
about
much
is
that
even
wear
a
lot
of
this
stuff
is
being
spread
by
BOTS,
and
you
know
special.
You
know
AI
tools
and
so
on.
Those
things
can
be
used
and
reversed
also
spread
counter
messages,
and
that
has
not
been
explored
much
except
by
marketing
firms,
but
I
think
they
all
said
question
from
the
audience.
Yes,.
B
Matthew
been
any
one
of
Kathleen
students,
so
my
question
is:
if,
if
the
way
forward
is
to
replace
the
narrative
and
for
lack
of
a
better
word,
our
information
adversary
is
clearly
using
deceptive
means
to
promote
their
narrative.
What
would
you
think
from
our
case
either
that
be
government
or
academia
if
we're
going
to
counter
that?
How
can
we
compete
in
that
space
in
a
manner?
That's
moral.
F
Okay,
well,
if
you
want
my
honest
opinion,
you
got
to
take
morals
and
throw
them
out
the
window.
Sorry
nah,
that's
it
I
mean
if
you
listen.
If
you
show
up,
if
you
show
up
to
it,
if
you
show
up
to
a
fight
with
a
knife
and
your
opponent
has
a
submachine
gun,
you
lose
sorry,
that's
it!
Yes,
if
you
want
to
take
the
more
if
you
want
to
take
the
moral
high
ground
and
that's
the
ground,
you're
gonna
end
up
buried
in.
B
F
Well,
so
you
see
so,
first
of
all,
you
have
to
do
that
in
case
the
case
by
case
basis
and
when
and
when
I
say
you
know,
it's
not
like
I
said
it
isn't
about
false
and
everybody's
focused
on
things
that
are
false.
First
of
all,
any
take
the
idea
of
force
and
throw
it
out
the
window
right
I
mean
it
goes
back
to
what
I
was
saying.
Is
you
tell
it's
telling
people
whatever
you
need
to
tell
them
to
make
them?
F
To
put
your
point
across
falls
true,
irrelevant,
it's
not
doesn't
even
come
into
the
question
all
right.
So
if
you
look
at
the
way
the
Russians
operate,
for
example,
the
Russians
can
change
his
story
on
an
hourly
basis
if
it
suits
their
needs,
doesn't
matter
and
that
they
could
be
saying
true
things
that
you've
been
saying
have
true
things.
They
became
completely
false
things.
I
mean
what
you
know.
What
is
false,
I
hear
a
lot
of
discussion
about.
Well,
is
there
some
set
of
facts?
F
Can
we
you
know
just
put
the
facts
for
what
are
facts?
I
mean
that's
a
bogus
concept
actually
for
the
most
part,
because
anything
that
I
tell
you
any
information
that
I
provide,
or
that
comes
your
way,
you're
going
to
interpret
it
in
your
own
personal
context.
It's
got
to
fit
into
your
own
world
and
once
you
start
interpreting
it,
it
loses
a
lot
of
its
objective
value.
F
Do
so
it's
you
can't
be
going
on
objective
value
of
Jacob
is
not
really
the
point
anymore,
though
you
know
credibility,
I,
don't
know,
I,
don't
even
know
if
credibility
is
an
issue
anymore.
In
that
way,
it's
long
as
you're
saying
something
that
somebody
will
buy.
Then
you're
credible.
That's
that
it's
a
short
said.
It's
a
functional
thing.
This
point.
A
C
Okay,
I,
don't
I
think
at
this
point,
we're
still
trying
to
understand
even
what
it
would
mean
to
act
ethically
in
some
of
this
area.
It's
not
clear
what
ethics
mean
in
some
sense.
You
know,
given
the
way
that
you
know,
because
it's
all
consensus
formation
and
so
on
so
I
think
it's
a
very
hard
issue.
C
I
also
think
that
where
you
sometimes
are
overreacting
to
the
information
that
appears
false
when
it's
really
meant
originally
mantas
humor
and
sarcasm,
and
we
need
to
begin
to
understand
how
to
disseminate
sarcastic
remarks
from
from
actual
spread
or
false
information
until
when
one
is
one
versus
the
other,
and
without
that
I
think
we
could
overreact
to
a
lot
of
things.
I.
D
Have
another
question
that
he
has
previously
about
how
we
can
address
some
of
these
misinformation?
So
there
are
many
many
efforts,
computational
as
less
crowd-sourced
that
are
out
there.
So
if
someone
is
interested
in
knowing
how
we
can
tackle
the
problem
of
this
information,
there
are
these
computational
AI
based
solution,
machine
learning-based,
aerations,
you're,
trying
to
train
a
model
and
decide
decide.
Limited
information
is
false
or
true,
but
also
there
are
efforts
like
stop
fake
or
God.
D
D
A
I'd
like
to
add
we're
talking
about
kind
of
hostile
information.
We
have
colleagues
at
the
peace
innovation
lab
at
Stanford
who
are
looking
at
how
social
media
can
be
used
to
build
bridges
between
parties,
for
example,
they're
doing
research
in
Israel
between
the
Israeli
and
Palestinian
populations
and
how
to
build
those
positive
bridges.
It
also
can
be
used,
as
so
does
any
other
questions
from
the
audience.
I.
F
Would
make
one
comment
on
that?
You
know
so
it's
it's
great
to
join
anything
like
that.
What
you
just
described
to
build
bridges
of
that
way.
The
question
is:
how
do
you
do
that
in
a
completely
hostile
environment
and
the
problem
with
the
people
who
are
doing
research
on
these
kind
of
things?
Is
they
seem
to
ignore
the
fact
that
all
these
things
are
happening
in
very
hostile
environments
and
that
the
type
of
interference
that's
going
to
be
perpetrated
is
going
to
be
enormous
and
it's
going
to
come
from
multiple
directions?
F
G
Media
campaigns
and
counter-offensive
media
campaigns
and
I.
It
does
seem
to
me
from
my
uninformed
perspective
that
perhaps
one
thing
that
we
can
do
in
the
Academy
is
to
try
to
understand
a
little
bit
better.
The
tools
that
are
used
to
launch
these
campaigns
and
and
and
kind
of
take
an
infrastructure
based
approach
a
little
bit
and
I'm
just
curious
and
that's
part
of
the
purpose
of
this
discussion.
G
Obviously,
but
given
the
reality
of
what's
going
on,
and
and
given
the
kind
of
intuitive
resonance
of
Rands
comments
like
it
or
not,
I'm
just
wondering
what
how
we
might
at
how
we
might
address
this
situation
from
our
perspective
of
trying
to
develop
an
understanding
and
also
trying
to
improve
the
tools
and
capabilities
that
we
have.
You
know,
maybe
even
to
advance
the
possibility
of
doing
better
at
these
counter
counter.
Narratives.
C
In
organizational
theory,
people
often
point
to
restaurants
as
like
the
e.coli
of
organizational
research,
because
you
can
go
outside
just
about
anything
about
organizations
with
restaurants
because
they
come
and
go
so
fast
and
in
social
media.
There
are
equivalent
eco
light
areas.
Okay,
and
so
one
of
those
for
right
at
the
moment
tends
to
be
things
that
we
found
associated
kind
of
by
doing
searches
on
the
all
right.
There's
this
kind
of
an
e.coli
area
where
there
are
lots
of
little.
C
F
C
There
just
on
different
parts
of
web-like
we
found
was
one
area
by
looking
through
the
by
looking
through
the
all
right
and
we
found
a
bunch
of
little
these
little
pockets
of
dense
topic
groups-
okay,
that
are
not
your
big
special
ones
around
Isis
or
whatever,
but
there.
But
there
is
a
whole
bunch
of
them
and
they're
kind
of
and
they
can
be
linked
together
and
pulled
apart
and
so
on.
So
they
become
just
kind
of
a
nice
test
bed
area
and
you
can
find
other
ones
like
this
on
social
media.
A
C
Perhaps
I
kid
well,
while
you're
thinking
of
a
question
I
just
want
to
mention
that
I
want
to
bring
back
up
this
issue
of
sarcasm
and
images,
and
so
on
I
mean
for
a
long
time.
People
thought
that
we
could
get
at
this
by
just
looking
at
a
single
tweet
or
a
single
blog
post
or
something,
and
then
he's
at
0
sentiment
is
going
to
help
us.
C
Of
course,
sentiment
tools
are
terrible,
but
what
we're
seeing
now
is
that
you
actually
have
to
track
full
conversations
and,
like
you
know,
I
think
we're
trying
to
talk
about
here.
Is
that
there's
a
whole
new
approach
that
you
need
to
take,
which
is
focusing
on
the
discourse
focusing
on
the
conversation
and
by
focusing
in
on
the
conversation
as
opposed
to
taking
a
tweet
out
of
context
or
Facebook
page
out
of
context.
You
actually
realize
that
something
that
you
thought
was
oh
highly
negative
becomes
like
neutral.
C
Okay,
something
you
thought
was
really
funny
is
not
funny
anymore,
that
the
image
that's
attached
to
it
completely.
Inverts.
What's
said
before
that,
and
so
it's
this
kind
of
discourse
centric
conversation
following
techniques
that
we
need
to
develop
new
technology
for,
and
it's
very
hard
with
current
AI
software.
A
So
then,
the
question
earlier
was
and
ran.
You
mentioned
the
in
previous
years.
The
government
hasn't
necessarily
focused
on
this
area
and
doing
research
in
this
area.
Where
do
we
need
to
look
now?
Where
do
we?
What
are
the
priority
areas
for
both
academia
and
for
government
and
industry
going
forward.
F
Well,
I
be
didn't,
grow
up,
I
mean
it's
really,
but
you
know
how
you
do
narrative
detection
I
mean
that's
really
the
Ducks,
that's
the
biggest.
That's
an
immediate
problem
because,
like
like
tetany
was
saying,
you
know
what
one
tweet
or
one
item,
even
one
article,
it's
very
difficult
to
sarcasm,
only
means
something
in
a
larger
context.
Right,
there's
no
such
thing
as
sarcasm
in
isolation.
It's
sarcasm
relative
to
something.
So
we
need
is
better
tools
to
detect
what
they
saying.
F
You
know
all
of
these
narratives
and
to
actually
be
able
to
disentangle
one
narrator
from
another
and
to
really
get
a
clear
picture.
You
know
in
a
broad
sense
of
what's
going
on
so
those
tools.
You
know
we
really
don't
have,
and
that's
that's
what
we
need
to
focus
a
lot
of
effort,
I'm
going
to
ask
the
panelists
to
give
closing
statements
and
then
I'll
have
a
few
announcements
to
make
at
the
end.
Thank
you
ran
so
as
a
closing
statement,
I
would
say
that
it's
a
completely
new
information
environment.
F
We
need
new
technologies,
we
need
to
follow
the
conversation
and
we
need
to
watch
out
for
social
bonds.
I'll
just
add
that
this
is
a
very
infant
information
environment
and
this
can
because
it
has
a
lawless
wireless
of
information
space.
So
we
need
to
think
more
scientifically
more
rigorously
and
develop
more
capabilities
that
can
address
the
issues
that
we
have
stepped
into.
F
So
what
my
closing
statement
is
so
we
need
to
promote
and
the
boot
divers
collective
intelligence
and
then,
as
a
computer
scientist
we
should
be.
We
should
enable
user
control.
The
information,
filter
and
checkers
ran.
One
word,
I
think
I
said
enough,
so
we'd
like
to
extend
a
warm
thank
you
to
our
panelists
and
to
our
audience
for
their
time
and
today's
engaging
discussion
before
we
close
I'd
like
to
mention
some
events
that
are
coming
up
for
the
south.
F
Big
data
innovation
hub
applications
are
being
accepted
until
March
fifteenth
for
the
data
science
for
social
good
program
in
Atlanta.
The
Atlanta
data
science
for
social
good
program
is
an
intensive
10-week
paid
internship
experience
that
blends
data
science
and
technology
design
interested
graduate
undergraduate
students
should
visit
WWD
SSG
dash
al
I,
say
tlio.
F
There's
a
smart
grid
workshop
on
April,
eighteenth
Texas,
A&M,
University,
College
Station,
which
is
a
South
hub,
spoke
grant.
Recipients
is
holding
a
hands-on
workshop
on
smart
grids
big
day
to
the
next
steps.
If
you're
interested
in
attending
please
visit
HTTP
colon,
slash,
slash
smart
grid,
Center,
TAMU
edu
the
next
south
hub
data
sharing,
an
infrastructure
working
group
will
be
held
on
March
31st
from
3
to
4.
30
p.m.
eastern
there'll
be
demos
from
internet
to
and
side
s,
and
you
can
go
to
South
big
data
hub
org
to
get
more
information
on
that.
F
The
southern
data
science
conference
will
be
held.
April
7th
in
Atlanta.
Those
interested
in
attending
should
go
to
Southern
data
science
com.
Our
south
big
data
hub
round
table
for
next
month
will
be
held
on
Thursday
April
13th
from
noon
to
one
thirty
eastern,
it
will
feature
a
panel
on
data
science
challenges
of
cancer
immunotherapy.
You
can
visit
bit
ly
forward,
slash
self
hub,
roundtables
for
more
information
and,
lastly,
I'll
let
Kathleen
and
Nitin
plug
the
conference.
F
They've
got
coming
up
in
the
summer,
so
in
July
fifth,
through
eighth,
there
will
be
the
conference,
the
SBP
brims.
The
data
challenge
this
year
is
on
fake
news.
So
those
of
you
who
are
interested
in
this
topic,
please
see
our
website.
Please
send
in
a
submission-
and
we
hope
to
see
you
there
website
is
s
BB
dash
brains,
org,
and
there
are
a
few
graduate
student
scholarships.
So
if
you
have
something
in
mind,
if
you
have
something
cooking
up,
please
submit-
and
please
visit
us-
it's
a
really
nice
young,
thang
cierto.
F
This
is
the
10th
year
now,
so
we
wanted
to
bring
in
some
cake,
and
so
for
those
of
you
who
were
sorry
for
him,
lick
lick
lick
can
I,
can
I
just
put
in
one
more
plug
yep.
Okay,
so
I
was
just
like
to
inform
everybody
that
we
have
created.
A
new
organization
called
the
information
professionals,
association
and
its
is
founded
by
me
and
a
a
guy
who
was
former
director
of
io
at
the
u.s.
DP
and
a
few
other
people,
and
so
this
new
Association
is
about
that.
F
We
already
wearin
incorporated
just
501c3,
we
have
already
forced
corporate
sponsors
and
our
website
is
about
to
go
live
and
we
are
going
to
be
holding
our
first
symposium
on
cognitive
security
at
the
end
at
the
sort
of
endor
August
beginning
system
time
frame.
So
please
look
out
for
that,
one
whose
interests
they
can
contact
me
and
I'm,
giving
the
exact
details
when
it
comes
announced
grand
where
will
it
be
held
in
DC,
oued
south
hub
alright.
So
let's
talk
definitely
washington,
d.c
fantastic!
Well
with
that.