►
Description
Jeff Hawkins explains how introspection can be a helpful tool in neuroscience research. He first gives an overview of what a column in the neocortex needs to know to recognize an object. To recognize an object, a column must simultaneously infer object, location, orientation and scale. The team then extensively discusses how scaling works in the columns. Lastly, Jeff proposes four possible explanations for how the neocortex can represent objects with different orientations.
- - - - -
Numenta is leading the new era of machine intelligence. Our deep experience in theoretical neuroscience research has led to tremendous discoveries on how the brain works. We have developed a framework called the Thousand Brains Theory of Intelligence that will be fundamental to advancing the state of artificial intelligence and machine learning. By applying this theory to existing deep learning systems, we are addressing today’s bottlenecks while enabling tomorrow’s applications.
Subscribe to our News Digest for the latest news about neuroscience and artificial intelligence:
https://tinyurl.com/NumentaNewsDigest
Subscribe to our Newsletter for the latest Numenta updates:
https://tinyurl.com/NumentaNewsletter
Our Social Media:
https://twitter.com/Numenta
https://www.facebook.com/OfficialNumenta
https://www.linkedin.com/company/numenta
Our Open Source Resources:
https://github.com/numenta
https://discourse.numenta.org/
Our Website:
https://numenta.com/
A
All
right
so
you're
good,
so
I
I
just
have
a
few
topics
today,
really
just
like
two
slides
but
could
go
long.
If
there's
a
lot
of
discussion,
I'm
gonna
do
some
review
of
some
information
that
we
know.
I'm
gonna
elaborate
on
some
things
we've
talked
about
before
and
then
I'm
going
to
introduce
a
new
problem
which
I've
been
thinking
about
before
I
did.
I
just
for
some
reason
felt
like
putting
the
slide
up
here
in
defense
of
introspection,
and
what
do
I
mean
by
introspection?
A
Introspection
and
neuroscience
is
essentially
that's
my
definition,
using
self-observed,
perceptual
state
perceptual
states
to
infer
facts
and
theories
about
the
brain,
and
I
put
this
up
there
because
I
remember
a
long
time
ago,
probably
a
few
decades
ago,
I
heard
a
talk,
oops.
Sorry,
I
heard
a
talk.
Someone
said
you
should
never
use
introspection
if
you're
a
neuroscientist,
it's
just
false
and
I
found
just
the
opposite.
Every
introspection
is
very,
very
useful
and-
and
I
I
originally
accepted
this
person's
premise,
but
now
I
said
no,
that
wrong
inspection
is
great.
A
So
I'm
talking
a
lot
about
introspection
today
on
its
own.
It's
never
sufficient
or
rarely,
but
it's
really
very
helpful
yeah.
I
had
a
couple
examples
of
things
I
use
introspection
all
the
time,
I'm
not
sure
it's
worth
going
through
these.
Some
of
these
have
become
obvious,
but
a
lot
of
them
with
today
is
regarding.
B
A
I
think
it's
wrong,
and
and
again
I
think
you
can't
only
use
introspection,
but
it's
one
of
the
tools,
that's
very,
very
valuable
and
I
even
originally
accepted
it.
As
I
said,
but
then
later
I
said,
wait
a
second.
This
is
really
useful.
This
is
not
right,
so
these
are
well.
These
are
two
examples.
You
know
I
use
it
all
the
time,
but
this
you've
heard
me
talk
about
when
good
cells
remap,
I
I
can
say
that's.
A
A
Another
example
is
like
noticing
that
when
our
eyes
move,
we
don't
we
have
perceived
stability,
but
we
don't
notice
the
movement
through
the
cods
and
that
immediately
that
perceptual
observation
led
to
the
conclusion
that
there
must
be
cooling
going
on,
as
you
have
to
go
from
these
changing
states
to
a
constant
state
and
a
mapping
and
a
pooling.
So
this
is,
these
are
the
kind
of
things
you
can
just
think
about
to
say.
Yeah,
I
observe
this.
I
don't
notice
that
my
eyes
are
moving,
but
I
notice
the
world
seems
stable,
so
there
must.
B
C
B
A
Here
is
a
perfect
example
that
you
know
that
we
know
that
when
the
eyes
move
that
the
inputs
of
the
brain
are
changing,
we
know
the
neurons
are
changing.
Yet
you
have
perceived
stability
under
many
different
scenarios,
not
all
scenarios,
and
that
just
tells
you
it's
a
fact
that
there
has
to
be
a
pooling
operation.
Going
on
that
is.
These
changing
patterns
are
being
mapped
into
a
stable
pattern.
There's
no
way
around
it.
It's
just
that's
what
it
is.
A
Don't
they
argue
otherwise
would
just
say
that.
Well,
your
perceived
stability
is
not
really
the
action
of
neurons.
It's
it's
something
else
which
is
like
a
dualistic
point
of
view.
So
anyway,
let's
get
into
it.
We're
going
to
get
a
lot
of
examples
as
I
go
through
the
next
slide
here.
So
I'm
going
to
I'm
going
to
sort
of
work
through
a
review
and
adding
things
to
it
in
a
moment
and
and
sort
of
elaborating
some
things
we've
talked
about
in
the
past,
but
I've
got
clear
definitions
on
them.
A
So,
let's
just
talk
about
thousand
brains
theory,
it
was
originally,
as
you
all
know,
this
idea
that
hey,
I
was
observing
my
finger,
touching
a
coffee
cup
and
they
said
well.
What
does
it
require
for
the
cortex
to
predict?
What
my
finger
is
going
to
feel-
and
we
said:
oh,
it
requires
that
the
cortex
or
column
knows
the
object,
identity,
the
coffee
cup,
and
it
knows
the
location
of
the
object,
relative
location
of
the
sensor
relative
to
the
object
which
implies
reference.
A
We
then
learned
subsequently
that
wasn't
sufficient
and
I'm
surprised
we
didn't
realize
this
initially,
but
it
required
the
simulation
that
lewis
was
doing,
that.
We
also
need
to
know
cortex
needs
to
know
the
orientation
of
the
sensor
to
the
object.
That
is,
the
position
of
the
finger
could
be
at
some
location
x,
but
the
orientation
of
the
finger
at
that
point
determines
what's
going
to
be
sensed,
and
so
this
makes
the
problem
more
complicated.
A
That
a
column
needs
to
know
this
additional
piece
of
information,
the
orientation
in
space
relative
to
the
object
sense,
and
I'm
going
to
add
one
more
to
this
one
we've
talked
about,
but
I'm
really
confident.
Now
I've
put
much
more
clarity
on
this,
which
is
scale
and
I'm
going
to
go
to
depth
in
this
in
a
second.
A
This
is
a
coffee
cup.
That
is
what
we
perceive.
We
don't
perceive
the
movement
of
the
eyes
or
the
finger
as
much.
You
can
perceive
the
movement
of
your
of
something
like
if
you
attend
to
some
sub-piece,
but
if
I'm
just
looking
at
a
scene
or
just
casually
touching
a
coffee
cup,
my
perception
is
the
coffee
cup
is
stable
and
my
perception
is
the
thing
I'm
looking
at
is
stable,
and
so
we
don't
really
perceive
generally
we're
totally
unaware
that
our
eyes
are
moving.
A
So
then,
now
we
have
an
explanation
for
that
that
these
long-range
voting
neurons
that
do
this.
Now,
let's
talk
about
scale,
you
define
what
I
mean
by
scale
scale
is
a
scalar:
it's
a
ratio
between
a
motor
vector
and
path
integration
relative
to
the
object.
So
imagine
my
fingers
touching
the
side
of
the
coffee
cup
and
I
move
my
finger.
A
The
movement
leads
to
a
prediction
of
a
new
location
relative
to
the
coffee
cup.
I'm
going
to
move
it
up,
I'm
predicting
a
new
location,
but
how
much
a
movement
translates
to
a
change
in
location
depends.
If
I
had
a
small
coffee
cup
like
I
reached
my
finger
into
into
a
box
and
I'm
pitching
an
object,
but
now
I'm
touching
a
very
small
coffee
cup
at
first.
A
I
don't
know
that,
but
I
kind
of
say
hey:
this
is
a
coffee
cup,
but
it's
really
small
once
I
know
that,
then
I
adjust
all
my
movements
relative
to
that.
Basically,
I
can
manipulate
the
coffee
cup
or
any
object
at
different
scales
once
I
know
that
the
relative
scale
of
it
and
what
it's
basically
doing
is
just
changing
when
you're
doing
path,
integration,
you
have
some
movement
vector
and
then
how
much
path,
immigration,
how
much
movement
in
a
reference
frame
is
the
scale.
That's
what
I
mean
by
scale.
A
It's
not
necessarily
the
size
of
an
object.
In
the
example
I
just
used
with
the
coffee
cup,
I
could
have
a
miniature
coffee
cup
and
my
finger
touches
it
well.
That
is
a
smaller
object,
but
if
I,
if
I
was
looking
at
a
bicycle
and
the
bicycle
was
four
feet
away
from
me,
I
would
know
how
to
move
my
eyes
over
to
predict
what
I'm
going
to
see
at
different
locations
like
how
far
do
my
eyes
have
to
move
to
go
from
wheel
to
wheel
or
from
the
wheel
to
the
seat.
A
If
the
bicycle
was
eight
feet
away,
the
movement
of
my
eyes
would
be
much
less
to
achieve
the
same
movement
in
the
optic
space.
I
don't
perceive
those
two
objects
as
being
different
sizes.
Yes,
I
they
are
different
sizes
on
my
retina,
but
perceptually
they're,
not
different
sizes.
I
I
know,
maybe
I
know
it's
further
away,
but
even
if
I
didn't
was
looking
a
picture
of
a
bicycle
on
a
piece
of
paper,
I
don't
and
my
eyes
are
cutting
over.
I
don't
say
hey.
This
is
a.
This
is
a
miniature
bicycle.
A
I
don't
do
that.
I
said
it's
a
bicycle,
and
so
it
doesn't
always
translate
to
until
it's
not
an
object
of
different
size.
It's
basically
just
saying
how
much
should
I
move?
If
I'm
moving
my
sensor
in
the
in
the
in
body
space?
How
much
should
that's?
How
should
I
scale
that
to
the
movement
in
the
path
integration
in
the
object
by
the
way?
Interrupt
me?
If
you
have
questions
about
any
of
these
things,.
E
I
I
have
one
question
yeah
given,
given
that
kind
of
continuum
you
just
described,
would
you
the
notion
of
whether
something
is
within
re?
I'm
sorry
to
turn
on
my
video
whether
something
is
within
reach
or
not
would
be
a
critical
thing,
because
it's
like
you
make
an
instantaneous
decision
as
to
whether
I
can
reach
out
to
touch
it
or
whether
it's
something
a
remote
for
me,
it
would
seem
like
you
would
dedicate
a
certain
amount
to
just
evaluating
whether
whatever
this
thing
is
is
with
intention.
A
Remember,
remember
the
the
what
in
the
where
path
was
incorporated
right
and
the
idea
of
something
which
is
in
reach.
That's
a
wear
pathway
thing
right,
that's
a
that
is
a
mapping
in
body
space
not
in
object
space,
so
there's
separate
parts
of
the
cortex
that
are
dedicated
to
that
problem
than
the
problem
of
inferring
an
object.
A
So
I'm
only
talking
right
now
about
inferring
and
recognizing
an
object
or
learning
an
object,
and
so
you
can
assume
for
the
moment
that
I'm
already
touching
this
thing
and
now
I'm
just
trying
to
recognize
what
this
thing
is
and
therefore
I'm
moving
within
a
reference
frame
relative
to
the
object.
Whereas
if
I
want
to
reach
for
the
thing
and
know
if
I'm
going
to
reach
for
the
thing,
that's
aware
pathway
thing
and
we
believe
that's
a
reference
frame
relative
to
the
body
and
therefore
it's
like
okay.
A
E
Okay,
so
so
let
me
mention
this
so
if,
in
the
sense
of
an
object,
was
available
to
you,
so
you
could
tangibly
touch.
You
could
learn
about
it
through
touching,
as
well
as
by
seeing
versus
an
object.
You've
only
seen
remotely
never
been
able
to
touch.
Do
you
want
to?
Is
there
any
distinguishment
between
object,
recognition
capability
in
those
cases?
Well,.
A
I
think
this
has
been
a
confusing
aspect
for
us,
because,
obviously,
when
you
touch
something
your
finger's
right
on
it
right,
there
is
the
location
of
your
of
your.
Your
finger
is
in
the
is
actually
on
in
the
point
on
the
object
reference,
but
when
you're
looking
at
something
your
eyes
are
remote
right,
and
this
has
puzzled
us
quite
a
bit
in
the
past.
You
know
trying.
A
A
It's
not
a
complete
solution
to
that
problem,
but
it's
part
of
the
solution
to
that
problem,
and
so,
if
you
think
about
this
way,
you
can
say
yeah
and
I'm
moving.
I
have
to
move
a
certain
number
of
degrees
to
move
to
a
new
location.
An
object
will
help
that's
the
scale.
How
far
do
I
have
to
move
to
do
that
depends
on
how
either
how
big
the
images
I'm
looking
at?
How
far
away
it
is,
but
I
can
just
separate
those
out.
I
can
just
say:
okay,
I've
got
something.
A
A
A
So
this
idea
that
I
I
were
able
to
just
map
motor
behaviors
in
one
space,
the
body
space
to
to
movement
in
a
object.
Space
is
sort
of
a
general
principle
and
it
can
relate
to
the
object
being
a
different
size
or
it
can
relate
to
being
further
away,
or
you
know
it's
sort
of
a
general
principle
of
trying
to
sort
of
not
think
about
it.
Exactly
like.
Oh,
it's
a
you
know,
I'm
trying
to
reduce
the
differences
between
sense
of
vision
and
touch.
I
know
that
helped
at
all.
A
B
One
can
analogy
I
sometimes
use
to
understand
this
is
just
think
of
a
transformation
matrix
in
in
computer
graphics,
they're,
not
the
same
thing,
but
a
transformation
matrix
tells
you
how
two
reference
frames
align
with
one
another,
and
you
know:
there's
a
displacement
component
to
the
transformation
matrix,
there's
a
rotation
component
and
there's
a
scale
component,
and
these
are
kind
of
these
number
two.
Three
and
four
here
now
they're.
Not
it's
not
going
to
be
the
same
here,
but
it's
it's
sort
of
analogous
to
that.
Yeah.
A
All
right,
so
I
think
that's
the
same
idea,
yeah,
it's
exactly
the
same
now
in
the
past,
I
have
sort
of
sort
of
floated
around
the
proposal
multiple
times
that
this
is
the
scaling
is
occurring
in
the
thousands.
I'm
now
much
more
certain
that
that's
true.
I've
moved
myself
along
the
scale
of
certainty.
A
Here
I
put
it
in
the
sort
of
98
99
likelihood
category
and
there's
a
number
of
reasons
for
that,
but
perhaps
the
best
is
that
we
know
of
a
mechanism
that
would
do
something
and-
and
that
involves
thomas
and
specifically,
we
think
about
path.
Integration
using
grid
cells,
almost
all
the
the
theories
that
we're
considering
these
days
involve
the
the
theta
rhythm
and
and
the
beating
or
the
sort
of
the
two
frequencies
of
theta,
one
representing
a
baseline
theta
rhythm
and
one
representing
that
same
frequency.
A
A
Another
thing
we
can
can
say
is
that
the
scaling
factor
has
to
be
applied
to
every
column
in
a
sensory
modality.
That
is,
if
my
fingers
touch
my
hand,
is
grabbing
a
coffee
cup.
Some
of
my
fingers
can't
be
working
at
one
scale
and
some
have
another
either
the
coffee
cup
is
big
or
it's
small,
but
it's
not
like
half
as
big
and
half
as
small.
A
Don't
know
that
I
don't
know
if
we
know
the
answer
question.
Maybe
someone
does.
I
do
know
that
it's
surprisingly
sensitive,
if
you,
if
the
whole
the
whole
sort
of
theta
concept
for
path.
Integration
requires
very
small
changes
in
frequency
because
we're
basically
moving
the
the
interference
frequency
the
beat
of
these
two
different
frequencies,
and
if
you
just
change
the
frequency
a
little
bit,
it
makes
that
difference
quite
large.
So
it's
quite
a
sensitive
system.
It
wouldn't
require
much
change
in
the
frequency
to
affect
a
fairly
large
scale
change.
E
Well,
I
guess
I
I
think
you
did
answer
in
the
sense
that
you
can
have
a
small
delta
on
the
theta
and
it
it
if
you're
having
interference
patterns.
They
can
shift.
You
know
radically
over
that,
so
so
you
can.
But
I
was
also
wondering
about
the
the
the
paradigm
that
was
presented
in
that
paper
that
we
looked
at
recently
with
the
cohonin
maps
for
the
v1
v2.
You
know
sections
of
the
of
the
optical
cortex.
E
A
So
so
this
is
the.
This
is
like
that.
That
paper
was
consistent
with
the
idea
we
propose,
which
is
that
you
know,
columns
in
v1
and
v2
are
are
basically
dealing
with
different
scales
on
the
retina,
but
within
those
scales
you
still
need
variation.
It'd
be
like
saying
you
know
I
can
quantize
this
into
two
buckets
or
three
buckets
of
scale,
but
within
that
I
still
have
to
speed
things
and
slow
things
down,
because
any
particular
object.
I
have
to
match
this
yeah.
E
A
Yeah
well,
the
thalamus
has
the
scale
could
apply
to
all
those
regions.
It's
just
that.
Maybe
some
of
the
reasons
aren't
going
to
be
essentially
modified.
Some
of
the
reasons
may
not
be
able
to
model
the
thing
that
you're
looking
at,
but
as
far
as
I
know,
the
matrix
cells
in
the
thalamus,
assuming
that
those
are
the
ones
that
are
doing
this
they
like
in
for
visual
reasons,
they
project
the
v1,
v2
and
b4
the
same
cells.
E
A
A
A
So
if
I
you
know,
but
it's
it's
not
like,
I'm
not
sure
that
there's
a
voting
like
I'm
not
I
don't
know,
I
don't
know
the
actual
actual
mechanism.
That
would
do
this.
How
how
the?
How
did
the
calm
play
around
with
these
things
to
reach
the
right
answer,
reach
the
right
scale?
I
don't
know
yet,
but
I
don't
think
it
would
be
too
hard
to
figure
that
out,
but
it's
kind
of
like
voting,
but
it's
not
like.
I
bet
it's
different.
Obviously
it's
happening
in
the
thalamus
itself.
A
E
Could
could
it
be
that
there
is
some
notion
of
where
there's
a
locus
in
one
of
these
scale
spaces
that
it
says
I'm
doing
a
little
bit
of
movement?
I'm
getting
this
kind
of
reaction,
and
I
should
be
localizing
my
attention
in
in
in
some
weird
sense
in
and
turning
off
the
other
inputs
and
saying
this
is
the
scale
that
I
want
to
be
operating
in
and
that's
a
shifting
concept.
Yeah.
A
I
think
in
the
general
idea,
that'd
be
right
right
in
general,
you
say:
hey,
I'm,
making
a
prediction,
but
the
prediction
was
either
overshot
or
undershot.
Something
like
that
and
so
you
just
somehow
you
have
to
detect
them
who's,
overshooting
or
undershooting
and
then
adjust
the
theta
frequency.
Accordingly,
I
think
that's
what
you're
saying
kevin.
A
E
That's
that's
it
right.
Well,
I
was
thinking
what
would
happen
is
that
if
you're
grossly
mismatched,
then
you're
going
to
get
random
firings
in
that
region,
there's
nothing
going
to
be
stable
about
it,
but
in
the
region.
If
you
pick
the
right
scale,
you're
only
going
to
see
you
know
small
variations
within
that
region,
maybe
that
that
reinforcement
says
okay,
that's
the
one
I
should
pay
attention
to,
as
opposed
to
one
that's
overshooting
grossly
over.
A
Yes,
yeah
yeah.
I
think
that's
right,
but
it's
it's
that's
an
area.
I
think
that's
right,
but
there's
a
lot
of
unknowns
here.
I
want
to
make
one
of
the
observation
here.
The
last
point
on
this
box
is,
I
don't
I'm
not
aware
that
the
hippocampus
in
the
antarctic
cortex
has
anything
equivalent
to
the
thalamus.
I'm
not
aware
that
they
project
to
the
thalamus
the
way
the
neocortex
does
my
impression.
Is
they
don't?
I've
never
heard
anything
like
that?
Maybe
someone
else
needs
a
marker.
A
There
is
no
equivalent
relationship
to
the
balance
in
this
regard,
and
so
the
question
then
that
comes
up
is,
do
do
play
something
grid
cells
have
this
ability
to
scale
and
I
think
there's
some
empirical
evidence
that
they
don't,
and
this
goes
back
to.
If
you
put
a
rat
in
an
environment
and
you
you
then,
and
that
learns
this
environment,
then
you
change
the
size
of
the
environment.
You
start
stretching
or
making
it
smaller
within
a
small
variation
of
size.
A
The
grid
cells
sort
of
adjust,
they
kind
of
move
in
or
move
out
a
little
bit,
but
if
you
go
beyond
even
a
modest
amount
of
scaling
of
the
environment,
the
grid
cells
remap,
which
would
tell
us
that
the
rat
again
introspection
they're
at
is
saying
this
is
not
the
same
room
anymore.
This
is
a
different
room.
You
know,
even
though
it
looks
exactly
like
the
other
rooms
I've
seen,
but
it's
it's
just
a
little
bit
too
big.
Therefore
it
is
not
the
same
room.
A
I
am
not
able
to
scale
this
in
any
significant
way
where
you
and
I
can
scale
things
very
easily.
I
can
make
a
dollhouse
and
I
can
look
in
the
room
and
say
yeah.
That's
my
living
room.
Oh,
let's
move
the
furniture
up
again,
there's
no
problem!
I
can
see
that
as
my
living
room,
even
though
it's
in
a
dollhouse
right,
I
mean
I,
I
would
use
the
same
map.
I
wouldn't
I
don't
it's
not
that
I'm
fooled
that
I'm
in
that
room,
but
I
would
use
the
same
macbook.
B
A
Yes,
I
think
that's
right
in
the
sense
that
I
think
you're
right
that,
and
we
do
know
that
many
of
the
sort
of
the
area
representations
that
occur
in
the
hippocampal
complex
also
occur
in
prefrontal
cortex.
So
you
can
see
that
in
the
those
are
the
those
fmri
experiments
we've
talked
about,
so
we
have
the
ability
to
do
this,
but
my
only
point
is:
I
think
that
this
is
an
ability
that
didn't
exist
in
the
original
map,
environment
mapping,
mechanism
of
good
cells
and
place
zones,
and
it
wasn't
really
necessary.
A
A
Well,
I
think
it
clearly
has
to
because,
because
I
gave
the
examples
of
of
touching
a
small
coffee
cup,
you
know
I
or
just
get
a
stapler
right.
I
have
a
big
staple
little
staple,
I
don't
say:
oh
that's
a
completely
new
object,
a
big
staple
says:
yeah.
It
works
like
the
same
as
a
small
stapler.
Here,
I'm
gonna
use
it
right.
I
don't
get
confused.
A
I'm
aware
that
it's
a
different
size,
but
I
use
I'm
using
the
same
math
and
the
same
grid
cells
and
the
same
movement
behaviors,
and
I
can
scale
them
all
up,
so
I
think
that's
happening.
I
also
give
the
example
assigning
your
name
and
different
motor
commands
and
using
tools.
You
know
it's
very
interesting.
I
can.
I
can
use
a
tool,
that's
a
short
tool
to
do
something.
I
use
a
long
tool
and
I
have
to
adjust
my
motor
behaviors
but
perceptually.
A
I
don't
think
it's
any
different,
like
I'm
prying
something
open
with
a
screwdriver,
it's
a
short
one.
In
the
long
run,
I
change
my
behaviors
just
natural.
You
know
you
just
scale
up
your
behaviors,
so
I
think
I
think
this
matter
century
cortex
is
doing
this.
I
think
this
is
going
on
in
in
all
cortical
regions.
I
mean
if
the
matrix
cells
are
doing
this
in
the
thalamus,
like
I'm
pretty
sure
confident
they
are.
A
As
far
as
you
know,
they
exist
everywhere
in
the
cortex,
so
every
part
of
the
neocortex
will
be
dealing
with
scale
one
way
or
another,
and
I
think
it's
a
universal
property
of
of
well
the
universal
property
of
transforms,
as
you
just
pointed
out
again,
so
I
think
it
was
interesting
to
think
of
like
how
a
rat
may
not
see
a
room.
It's
the
same
once
it's.
If
it's
you
know,
30
bigger,
even
though
it
has
all
the
same
features
in
it.
You'll
say:
oh
there's
no
room.
A
I
have
to
learn
it
over
again.
That
would
be
the
implications
for
the
experiments
there
so
to
recognize
an
object
not
only
to
make
a
prediction
but
to
recognize
an
object.
The
column
has
to
simultaneously
for
all
these
things
estimate
for
what
the
object
is.
The
location
of
the
sensor
on
the
object,
the
orientation
of
the
center
to
the
object
and
the
scale
beyond
it.
These
are
all
unknowns.
A
If
I
stick
my
finger
into
a
black
box
and
I'm
going
to
touch
something
with
my
single
finger,
I
don't
know
any
of
these
things,
and
so
they
all
have
to
be
infer
through
movement.
So
there's
a
it
just
says:
okay,
the
column
has
to
do
all
these
things.
I
haven't
been
able
to
think
of
anything
else
that
column
has
to
do,
but
I
mean
well
in
this
list.
A
Like
scale
in
one
dimension,
that
would
that's
weird
you're
right.
That's
a
good
point.
Like
you
take
a
look.
I
made
it.
Let's
say
I
took
a
coffee
cup
and
I
and
I
stretched
it
out,
so
it's
the
top
of
like
an
oval
yeah.
I
could
probably
I
would
well.
I
would
know
that
it's
a
cup
I
would
would,
I
think,
it's
the
same
coffee
cup
at
a
different
scale,
or
would
I
say
like,
for
example,
what
if
I
was
looking
at
it
I
was
looking
at.
A
I
showed
you
a
picture
of
a
coffee
cup
and
I
showed
you
a
picture
of
the
same
coffee
cup
at
a
different
scale
and
a
different
orientation.
I
wouldn't
say
that's
a
different
coffee
cup
yeah,
that's
the
same
coffee
cup.
However,
if
I
stretched
it,
I
would
say
no,
that's
a
different
cup.
There's,
there's
no
question,
that's
different!
That's
not
the
same
one
that
I
saw
before
now.
I
would
still
be
able
to
see
similarities
between
it.
A
A
E
Would
you
say
these
are
kind
of
the
same
balloons
you
know
despite
the
distortions,
I
think
there's
there's
an
expectation
component
about
there
of
when
you
see
an
object,
whether
you
believe
it
to
be
rigid
and
should
rigidly
transform
and
those
things
make
identity,
whereas
a
stretched
version
of
it
would
be
somehow
it
would
be
a
different
object.
So
I'm
wondering
how
much
how
much
you're
modeling,
what
the
expectation
is
of
what
kind
of
distortions
fall
into
the
same
way?
Well,.
A
I
guess
what
I'm
saying
here
is:
if
we,
if
the
matrix
cell
idea
is
correct,
then
there
there
is
no
mechanism
for
distorting
in
one
dimension
versus
another
dimension.
This
doesn't
appear
to
be,
and
now
we
can
do
that.
But
my
point
is
that
the
system
would
say
if
I
try,
if
I
had
an
object
that
was
distorted
in
one
dimension
in
all
dimensions,
then
this
point
I'm
making
here
would
say:
well
that's
a
different
object.
It's
not
the
same
object
anymore.
A
A
I
would
say
yes,
this
is
a.
This
is
an
object.
This
is
similar
to
an
object
I've
seen
before,
because
there's
some
of
the
sub
features
arranged.
Similarly,
but
some
of
them
are
not.
Therefore,
I
I
would
I
would
be
able
to
say
yes,
this
is
similar
to
a
copic
upper.
This
is
similar
to
a
particular
thing
I've
seen
before,
but
it
isn't
different,
but
here's
the
similarities
based
on
these
attended
two
sub-components.
A
So
if
I
think
I
saw
something
that
was
highly
distorted,
like
a
very
you
know,
like
a
oval
shaped
coffee
cup
at
first,
I
would
say
what
is
this
and
then
I
attended
the
different
features
of
it
and
I
say
yeah.
These
features
are
arranged
similar
to
how
they've
arranged
and
other
things
I've
seen
and
therefore
I'd
see
the
similarity
between
them,
but
I
wouldn't
be
fooled
into
thinking
it's
the
same
thing,
so
I
would
create
a
new
reference
frame
for
that
object.
A
A
I
don't
know,
I
guess
I
don't
know
if
you
say
learn,
I
guess
what
we've
we've
argued
in
the
past,
and
I
wrote
about
this
in
the
book.
Is
it
and
is
that
when
you
to
learn
the
structure
or
something
you
first,
you
have
to
attend
to
the
different
components
as
you
attended
them.
One
and
another
you
say:
oh,
I
recognize
here's
the
handle,
here's
the
answer.
This
is
this
as
you
do
that
you're
building
up
their
relative
positions
to
each
other
in
a
reference
frame.
So
it's
just
the
learning.
A
An
object
is
literally
just
attending
the
features
of
the
object
and
assigning
it
to
locations
in
the
reference
frame,
and
now
a
new
object
comes
along.
It
doesn't
quite
fit,
but
as
you
attend
to
some
subset
of
the
features
they
seem
similar
to
subset
of
features
in
a
previously
learned
object.
There
say
this
is
similar.
It
has
similar
behaviors,
I'm
not
sure
what
you
mean
by
learn.
Everything
has
to
be
learned
and.
B
E
A
I
guess
I'm
trying
to
answer.
I
think
I
tried
to
answer
that
question.
If
all
the
components
fit
within
the
same
reference
frame,
then
it's
the
same
object.
If,
if
there
are
different
components
at
different
locations
or
the
components
are
different,
you
know
if
they're
stretched
in
the
reference
frame
in
some
sense,
then
I
would
see
it
as
different
but
everything,
but
I'm
arguing
here
that
everything
scaled
up
for
scales
down
like
like
looking
at
a
picture,
that's
close
or
pictures
further
away.
I
would
not
see
the
same.
A
C
A
E
A
I
think
it
goes
back
to
this
the
issue
of
the
rat
in
the
room.
If
you
can
assign
the
features
the
observed
features
to
a
reference
frame
and
they
all
fit.
It's
the
same
object.
If
it's
not.
If
you
have
to
re,
if
you,
if
the
system
flips
over
and
it
remaps
the
grid
cells,
that
is
re-anchor
the
grid
cell
modules,
then
perceptually
it's
a
different
object.
That's
the
different
genus,
a
common
object.
A
E
A
A
I
might
do
that,
but
I
would
still
think
it's
a
different
room,
but
if
it
was
just
like
I
I
I
don't
know
you
have
to
be
careful,
but
my
my
argument
would
be
that
the
razzies
is
a
different
room
and
therefore
it
doesn't
know
anything
about
that
room,
but
he
might
start
seeing
similarities
and
useful
similarities
in
the
same
way.
I
talked
about
how
we
can
do
that,
but
it
would
be
a
different
room
that
wouldn't
just
I
mean
here's,
here's
one
way
to
put
it.
A
If
the
rat
was
in
that
room
and
and
then
you,
but
then
you
turned
off
the
lights.
I
don't
think
the
rack
could
do
path
integration
to
get
to
the
food,
whereas
if
it
was
the
same
room
rat
could
do
that
rat
could
say.
Oh,
I
know
I'm
in
this
room.
I
know
how
to
get
to
the
food,
even
though
I
can't
see
anything.
But
if
it's,
if
the
rat
says
this
is
a
different
room,
then
I
wouldn't
have
any
idea
how
to
get
to
the
food.
A
A
It's
both
a
transformation
based
on
orientation,
so
I
was
like
okay,
you
know
I
have
to.
I
have
to
use
the
orientation
of
the
finger
to
the
object
to
know
you
know
which
direction
am
I
moving
in
the
object
space
and
I
have
to
scale
it
by
this
scale
factor.
So
these
two
things
have
to
occur
to
make
a
proper
transformation
of
a
movement
in
one
space
to
a
movement
in
the
other
space,
and
it
has
to
go
back.
A
The
cortex
has
to
the
column
has
to
do
both
ways,
because
I
could
say
I
want
to
get
to
a
particular
location
in
object
space.
What
is
the
movement-
and
I
need
to
do
to
create
that,
so
it
has
to
go
back
and
forth
between
these
two.
So
I
thought
to
me.
This
is
useful
to
know
that
this
is.
These
are
more.
A
This
makes
the
problem
more
complicated,
but
these
are
more
constraints
on
the
problem
to
think
about
scale
and
orientation
and
how
they
have
to
be
how
they
have
to
play
in
in
both
transformations
going
from
one
space
to
another.
But
again,
I'm
always
thinking
about
in
terms
of
path.
Integration,
you
know
movement
in
one
space
versus
movement
and
and
trying
to
take
out
abstracted
to
that
level.
C
C
Exactly
yeah,
but
but
what
you
are
in
it.
You
know
that
you
know
that
coffee
cup,
even
though,
as
you
turn
your
head
around,
sometimes
it's
going
to
look
a
little
bit
different.
It's
going
to
be
maybe
elongate,
elongated
or
flat,
or
something
like
that,
but
you
still
know
that
it's
the
same
coffee
cup
in
the
same
place.
C
A
C
No,
no,
my
point
is
that
we,
the
object,
may
change
not
not
just
like,
may
have
a
weird
transformation,
but
you
still
know
that's
the
same
object.
I
think
in
that
case,
for
example,
you're
building
different
reference
frames
each
time
the
object
is
transformed
a
bit,
but
you
can
still
navigate
that
environment
right.
So
just.
A
Well,
I
guess
it
gets
back
to
this
question:
can
the
cortex
handle
transformations
in
certain
dimensions
and
not
in
other
dimensions?
You
know,
you
know
the
idea
of
stretching
out
the
coffee
cup
is
some
example
of
what
you
might
see
in
a
virtual
reality
world
right.
You
know
you
move
around.
Perhaps
the
copycat
looks
elongated
now,
because
they
didn't
do
the
transformation
correctly
and
the
question
is.
A
I
don't
see
how
the
neural
mechanism
can
support
that,
but
clearly
we
have
some
capabilities
to
do
that
and
I
was
arguing
that
perhaps
it's
doing
at
this
level
of
you
know
sort
of
seeing
the
similarity
but
two
objects.
I
mean
again.
If,
if
I
see
that
elongated
coffee
cup,
I
know
it's
not
a
regular
coffee,
that's
what
that's
why
you
know.
There's
an
error
right.
You
just
know,
there's
an
error.
A
A
D
Yeah
I
wanted
to
ask
something
about
scale.
So
have
you
I'm
curious
if
a
thought
has
crossed
your
mind
that
you
could
choose
to
unbundle
the
scale
into?
There
are
really
two
different
kinds
of
skills
here
that
go
with
two
different
kinds
of
movements:
eye
movements
which
are
rotating
your
eyes,
and
you
know
body
movements
or
something
like
like
if
you
actually
move
the
location
of
the
eye
where
it
is
in
the
reference
frame
of
the.
D
If
you're
staying
still,
if
you're
moving
your
eye
over
an
object,
it
doesn't
really
matter
how
far
away
it
is
and
how
large
it
is
like
you
might
be.
Looking
at
a
coffee
cup,
that's
close
by
and
small
or
you
might
be
looking
at,
like
a
giant
coffee
cup
in
the
sky,
a
mile
away,
I'm
bringing
up
a
silly
example,
but
you
could
come
up
with
better
examples
where
there's
an
ambiguity
there
there's
a
distance
ambiguity
that
doesn't
need
to
be
solved
if
you're
only
moving
your
eyes.
A
E
A
I
recognize
it
no,
but
you
I
I
I,
let's
just
let's
say
I
separated
out
two
scenarios.
One
is
you're
looking
at
pictures
on
a
piece
of
paper
right
sure,
that's
an
easy
one,
because
you
can
say
look.
I
have
never
never
think
like.
Oh,
it's
a
different
object,
because
it's
one
size
one
pick
the
copic
of
one
moment
is
big.
A
Another
one
is
small,
not
even
aware
I
just
think
of
like
that's
a
coffee
cup,
I'm
looking
at
a
picture
of
coffee
cup
no
problem
and
I
do
have
to
scale
to
to
to
recognize
it
and
to
infer
and
so
on.
But
in
reality,
if
I
see
if
I
were
to
see
a
coffee
cup
on
my
table
and
one
outside
and
one
outside
had
the
same
angle
of
displacement,
you
know
size
as
the
one
on
my
table.
I
would
say:
well
that's
a
huge
coffee
cup
out
there.
A
That
is
one
hell
of
a
coffee
cup.
You
know
it's
six
feet
tall,
and
so
we
are
aware
of
that.
I
mean
in
that
scenario.
Unlike
the
piece
of
paper,
the
piece
of
paper,
I
don't
go,
wow,
that's
a
six
foot!
Coffee
cup-
or
this
is
a
coffee
cup's,
only
an
inch
tall,
I'm
looking
at
a
piece
of
paper
and
it's
just
like
no
that's
a
copycat.
D
A
Yeah,
maybe
I
I
think,
you're
actually
getting
to
my
question
in
the
next
bullet,
I'm
going
to
make
here,
which
is
an
observation
which
I
certainly
haven't
dealt
with
before,
which
relates
to
this.
Perhaps
so,
maybe
can
you
just
keep
that
thought
in
your
head?
Marcus
go
on
to
the
next
one
here,
there's
a
problem
of
this
scenario
that
we
we've
got
here.
There
could
be
lots
of
problems,
but
there's
one
that
seems
really
jumping
out
of
me
right
now
and
here's
the
problem.
A
I
have
argued
that
these
these
voting
neurons
between
columns
are
basically
representing
the
object.
You're,
perceiving
the
object
that
you're
looking
at
and
I've.
I've
argued
that
that's
what
we
perceive,
but
that's
not
what
we
perceive.
What
we
perceive
is
an
object
at
an
orientation,
not
the
object.
So
when
I
look
at
a
coffee
cup
on
my
table
here,
I
don't
just
say:
oh
it's
a
coffee
cup.
I
perceive
the
orientation
of
the
object.
A
We
just
say:
oh,
the
voting
neurons
are
just
determining
the
object,
but
that's
not
what
I
perceive.
I'm
perceiving
the
object,
plus
orientation
the
object.
At
the
same
time.
It's
not
like.
I
have
some
just
generic
coffee
cup
image,
it's
like
no,
it's
and
so
and
here's
another
issue.
The
perceived
orientation
is
relative
to
the
body,
not
the
sensor,
so
it's
not
the
same
orientation
that
we're
talking
about.
A
I'm
not
aware
of
the
orientation
of
my
finger
relatively
cut,
but
I
am
aware
of
the
orientation
of
my
body,
even
though
I'm
touching
the
object
with
a
single
finger
in
a
black
box.
I
have
this
and
then
I
have
this
mental
image
of
the
of
the
coffee
cup
at
some
position
in
orientation
and
rotation
relative
to
my
body,
and
so
we
don't
really
have.
We
have
we've
never
really
dealt
with
this.
A
F
A
Already
have
an
explanation
for
that.
You
know,
and
I
wrote
about
in
the
book-
is
it
that
once
some
of
the
columns
are
voting
and
these
people,
these
columns
are
saying?
Oh,
it's
a
coffee
cup,
but
half
the
coffee
cup
is
excluded
right
or
occluded.
Excuse
me:
well,
those
voting
neurons
cover
the
entire
region,
and
so
all
the
columns
know
what's
being
perceived
even
if
they're
not
getting
any
sensory
input.
A
A
They
would
say
yeah
well,
I
know
I'm
not
seeing
anything,
but
this
is
what
I
should
be
saying,
or
I
wouldn't
be
saying
if
I,
if
I
was
getting
input,
so
I
think
that's
a
different
problem.
It's
I
think
it's
a
problem.
We
just
have
a
partial
answer
to,
whereas
what
I'm
talking
about
here
is
a
new
problem.
I
think
it's
a
new
problem
where
we
didn't
have
any
concept
at
all
about
orientation
as
being
a
shared
feature
before
that.
Does
that
help
yeah
yeah
yeah
thanks.
A
A
That's
obvious
next
slide
all
right,
so
I
asked
myself:
okay
there's
an
assumption
here,
because
I
am
perceiving
objective
orientation,
that
there
are
stable,
neurons,
representing
object
and
orientation,
and
the
question
is:
how
does
the
cortex
do
that
all
right,
they're
stable
my
eyes,
are
moving,
but
the
coffee
cup
seems
stable,
its
orientation
seems
stable,
its
identity
seems
stable.
So
what
are
the
possibilities?
I
came
up
with?
Well,
the
spoilers.
I
don't
know
the
answer
to
this
question,
but
I
came
up
with
four
possibilities
and
I
thought
I'd
just
mention
them.
A
One
possibility
is
that
we
still
have
a
single
layer
of
neurons
that
a
single
layer
of
neurons
is
stable,
representing
both
object
and
orientation.
Just
like
we've
been
arguing
that
an
object
can
be
represented
by
the
stable
neuron,
but
this
this
has
a
problem.
All
my
potential
solutions
have
a
problem.
If
there
was
a
single
layer
of
neurons,
representing
both
object
and
orientation,
remember
the
voting
layer
has
to
learn
right.
The
voting
layer
has
to
the
columns
when
they're
voting
have
to
say
yeah,
we've
all
learned
what
a
coffee
cup
is.
A
Therefore
we
can
all
form
these
connections.
Therefore,
now
we'll
know
how
to
vote
on
coffee
cups
so,
but
this
would
imply
that
this
single
layer
of
neurons,
representing
orientation
object,
would
imply
that
it
would
be
very
difficult
to
learn
how
to
vote,
because
you'd
have
to
learn
all
combinations
of
objects
and
orientation,
and
that
can't
be
learned.
That's
too
many
things,
so
you
couldn't
learn
or
like.
Oh
I'm
going
to
learn.
You
know
coffee
cop
at
this
orientation
corporate
government.
This
orientation
call
because
no,
it's
not
going
to
work.
A
A
A
There's
another
possibility
is
that
you
might
have
two
layers
of
neurons,
one
for
object
and
one
for
orientation,
and
these
these
ones
could
vote.
We
could
learn
all
orientations
in
the
orientation
columns
layer
and
we
can
learn
objects
in
the
object
layer
that
would
actually
work
and
they
would
vote
independently.
A
But
then
it's
not
clear
to
me
why
I'd
have
a
singular
percept.
A
single
percept
of
objected
orientation.
I'd
have
one
perceptive
object,
another
perceptive
orientation
where
I'd
have
one
set
of
cells
representing
object,
one
said
cell
different
orientation,
it
would
solve
everything,
but
this
is
where
introspection
could
be
failing
me,
and
I
don't
want
to
get
hung
up
on
this.
A
You
know,
but
it's
it's
it's
a
little
bit
more
tricky
now,
because
if
I
wanted
to,
for
example,
remember
what
I
saw
I
want
to
say
hey,
I
saw
that
coffee
cup
today
and
if
I
were
to
remember
what
I
saw
very
often,
I
would
remember
the
orientation
with
it.
It's
like.
Oh,
I
saw
you
know
this
car
that
was
facing.
That
was,
you
know
upside
down
today
or
it
was
facing
away
from
me
or
something
like
that.
A
Well,
I
would
remember
that
I
don't
just
remember:
I
saw
a
car,
I
could
remember
the
orientation
of
the
car.
Therefore
you
would
somehow
have
to
be
broadcasting
these
two
signals
throughout
the
brain
and
in
some
place
like
the
thermocourse
or
the
hippocampal
complex,
which
can
form
a
memory
of
this
we'd
have
to
remember
both
of
those.
At
the
same
time,
it
makes
it
look
complicated.
A
E
E
E
What
I'm
trying
to
do,
because
it's
an
I
haven't,
said
what
the
second
one
is,
because
it's
in,
in
contrast,
what
I'm
trying
to
see
is
whether
it's
the
components
of
the
object
that
are
being
tracked
for
orientation
as
opposed
to
the
gestalt
of
the
object,
so
the
other
one
would
be.
You
take
a
feature
of
the
object
like
the
loop
of
the
of
the
coffee
cup,
the
top
rim,
the
bottom
rib,
the
other
side
of
the
coffee
cup
and
display
those
displaced
in
time.
E
A
I'm
not
sure
the
distinction
between
those
two
kevin.
The
point
is
imagine
we
can
do
this
with
a
single
column
right.
You
can
do
all
of
this,
with
a
single
finger
tip
or
looking
through
a
narrow
straw,
and
so
in
that
scenario
with
a
single
fingertip.
Clearly
you
do
this
through
time,
but
it's
it's
the
time
match
with
the
reference
frame
location
right.
It's
not
it's
not
just
that!
A
There's
a
sequence
of
things
it's
like,
because
that's
just
a
bag
of
features,
it's
like
each
feature
has
to
be
assigned
to
a
reference
frame,
location
and
that
that
is
what
determines
what
the
object
is.
So
I
and-
and
so
that's
the
way
I
think
about
it
with
in
my
mind,
the
way
you
think
about
it
is
like
yeah.
I
can
do
this
sequentially
in
time,
but
it's
not
that
the
time
element
is
not
the
important
part.
E
A
Yeah,
but
this
I
think
this
gets
back
to
the
whole
voting
mechanism
in
general
right.
The
system
is,
is
inherently
time
based.
Everything
is
time
based,
but
and
so
to
recognize
something
to
learn
something
you
have
to
move
the
sensors
over
time
in
a
reference
frame,
but
we
can
do
you
can
do
this.
You
know
the
gestalt
inference
with
multiple
columns
by
voting
right,
but
it
it
doesn't
mean
that
the
system
is
not
time
based.
It
inherently
is
it's.
I
mean
that's
the
whole
point
of
that's
the
key
part
of
this
whole
theory.
A
If
you
really
know
this,
I'm
convincing
your
question,
but
a
key
part
of
the
whole
thousand
brains
theory.
It
explains
why
we
can
look
at
an
image
in
a
flash
and
know
what
it
is.
Even
though
you
can't
learn
it
that
way,
you
have
to
learn
it
through
time
and
general
inference
is
through
time.
So
the
system
is
inherently
time-based
entirely
movement-based,
but
we
can
do
these
costal
things,
it's
a
trick,
the
voting
track.
Now,
maybe
I
misunderstood
your
observation.
Well,.
A
But
the
thousand
bank
theories
does
do
that
kevin.
I
mean
it
explains.
Well,
that's
one
of
the
beauties
of
the
theory
is:
it
explains
how
we
can
do
flash
inference
without
movement.
It
explains
that
without
throwing
away
the
idea
that
the
entire
system
is
sentry
motor,
it's
just
it's.
The
voting
mechanism
says.
B
A
A
Grab
a
I
touch
a
coffee
cup
with
one
finger.
I
have
to
move
to
recognize
the
coffee
couple
measures
a
very
unique
feature
and
I
can
touch
the
couple
cup
of
three
fingers.
I
may
be
able
to
recognize
it
without
movement,
but
if
I
do
move
it,
I
can
usually
move
less
because
they,
you
know
so
there's
this
continuum
from
like
yeah.
I
have
to
do
a
lot
of
movement
to
recognize
something
to
less
movement
to
no
movement.
If
I
have
enough
columns
voting,
I
think
it
covers
that
whole
spectrum
very
nicely.
E
It
you
know,
even
though
he
didn't
see
the
entire
thing,
you
could
snap
and
say
this
is
a
picture
of
a
man's
collar,
and
I
could
go
back
to
the
previous
images
and
look
at
them
and
saying
I
you
know,
I
look
at
that
and
I
can't
you
know
I
even
though
mentally
I
know
that's
where
it
came
from.
I
can't
see
that
as
a
man's
collar,
but
at
a
certain
point
when
enough
features
were
available
and
it
was
an
incomplete
image
I
could
say.
E
Oh
you
know
I
can
recognize
that
as
a
man's
collar
yeah,
so
that
that
tends
to
that
tends
to
you
know
when
we
talk
about
occluded
objects
and
we
talk
about.
You
know
what
we're
looking
at
as
far
as
a
gestalt.
The
thing
there's
a
certain
point
where
there's
a
critical
number
of
features
that
are
available.
That
is
enough
to
say
I
can
recognize
it.
I
can
explain
this
image.
A
With
that
right,
exactly
it's,
because
you
have
these
features
that
relative
positions
to
each
other,
that
there's
a
resolution,
you're
saying
if
I,
if
I,
if
I,
if
I
discover
the
relative
position
and
in
a
reference
frame,
then
the
proper
reference
you
know
the
object
will
pop
out
because
there's
only
a
single
solution
at
this
point.
There's
enough
of
these
features
at
these
locations,
where
it's
unique
to
the
object
and
it'll
pop
out.
E
A
No,
no
in
that
scenario.
Well,
it
depends
if
you're
saying
I'm
trying
to
show
these
features.
At
the
same
time,
then
there
are
multiple
columns.
Getting
them
like
different
columns
are
getting
different
features
and
that's
the
voting
right.
That's
where
voting
occurs,
they're
saying
I'm
getting
one
com
saying
I'm
seeing
a
little
bumpy
thing
here,
but
you
know
so
I've
got
this
kirby
thing.
I
don't
know
what
it
is.
A
Another
guy's
saying
saying
I'm
seeing
like
a
buttonhole,
but
I
don't
know
what
that's
from
another
person's
saying
this,
but
then
when
they
they
vote
and
they
know
that
they're
in
some
relative
position
to
one
another.
They
say:
there's
only
one
thing:
that's
consistent
with
this
and
that's
what
the
voting
is
all
about.
The
voting
says
they
can.
They
all
have
ambiguity,
but
when
they
vote,
there's
no
ambiguity
because
there's
only
one
solution
that
fits
all
of
them.
E
A
Well,
something
that
something
is
devoting
I
mean
again,
it's
not
just
it's
not
just
you
know
a
bag
of
features.
These
are
the
features
we
have
enough
features
to
identify
the
object.
We
didn't
explain
this,
but
marcus
was
the
first
one
to
point
this
out.
That
columns
need
to
know
their
where
their
relative
position
is
developed
to
each
other
as
well.
A
So
so
they're
saying
we
have
these
features
in
relative
positions
to
each
other
and
and
that
is
sufficient
to
reduce
the
ambiguity
about
what
all
the
objects
that
have
those
features
relative
positions
to
each
other.
At
some
point
you
don't
have
enough
of
that.
You
could
say
well,
it
could
be
still
these
three
different
objects,
but
at
some
point
it
becomes
unique
enough.
You
could
say
bingo,
I
get
it.
E
A
A
We
don't
know
we
don't
know
yet,
which
is
the
right.
We
don't
know
yet
how
to
anchor
the
grid
cells.
If
you
will,
we
don't
know
which
exact
object.
This
is
right.
We
know
enough
to
say
by
fact
I've
got
a
number
of
features
in
relative
positions,
one
of
those,
then
that
tells
us
how
to
anchor
the
reference
range
and
the
object
pops
out.
E
So
can
we
can
we
then
say
that
the
the
orientation
output,
if
we're
trying
to
you,
know
if
the
question
was
asked?
What's
the
orientation,
this
thing
that's
something
which
could
be
deduced
by
the
relative
positions
of
the
features
themselves.
In
other
words,
it
it
could
be
an
emergent
property
as
well.
A
I'm
trying
to
I'm
the
point
of
this.
This
slide,
I'm
working
right
here
right
now
is,
is,
I
think,
there's
a
separate
problem.
There's
this
there's
the
relative
position
of
the
features
in
the
reference
name
of
the
object.
That's
what
you
would
have
just
talked
about.
That's
what
we've
been
talking
about
in
the
thousand
brain
theory,
but
our
perceptual
experience
is
that
we
recognize
the
object
in
a
relative
orientation
to
our
body,
which
is
not
what
the
columns
are
dealing
with.
A
The
columns
are,
we
haven't,
had
any
need
for
the
columns
themselves
to
say:
where
is
this
object
relative
to
the
body
we
just
we
just
have
all
we
need.
Each
column
has
to
know
where
its
orientation
position
is
relative
to
the
object,
and
so
I
was
trying
to
make
this
distinction
between
what
we've
been
talking
about.
Columns
need
to
do,
which
is
all
object,
reference
frames
and
what
our
perceptual
experience
is
of.
Like
that's
a
coffee
cup
angled
towards
me
at
a
certain
position
that
orientation
is
different.
A
It's
a
different
orientation,
it's
it's
a
body,
centric
orientation!
Imagine
I'm
going
to
move
on
here
for
a
second
imagine.
I
have
this
coffee
cup
in
front
of
me
and
I'm
moving
my
hand
around
the
back
side,
the
front
side,
all
over
the
place,
touching
with
multiple
fingers
right.
The
the
orientation
of
my
fingers
to
that
coffee
cup
are
all
changing
radically,
as
I
do
this,
and
each
column
needs
to
know
that
orientation
of
its
particular
input
relative
to
the
reference
frame
of
the
coffee
cup.
A
But
my
perception
of
the
orientation
of
the
cup
to
my
body
is
stable
and
therefore
there's
a
separate
representation
of
coffee
cup
to
my
body.
There's
absolutely
nothing
to
do
with
where
my
fingers
are.
At
that
moment
I
mean
there's
a
obviously
a
set
of
transformations
that
have
to
occur
to
make
this
this
pop
out.
So
all
I'm
saying
is
it's
not
clear
to
me
where
this
perception
of
orientation
relative,
my
body
comes
from
that,
doesn't
exist
in
the
columns
themselves.
Is
it
created
elsewhere?
A
I'm
just-
and
this
is
a
big
confusion
here.
Let
me
just
go
finish
my
my
list
of
things
I
talked
about
here.
I
did
number
one.
I
said
you
could
have
a
single
layer
of
neurons,
representing
both
object
and
orientation,
but
I
didn't
know
how
those
learners
could
could
learn
to
to
vote
and
then.
Secondly,
you
have
these
two
layers
of
neurons,
one
for
object:
one's
orientation.
A
I
don't
even
know
how
that
would
work,
but
but
I
don't
know
how
you'd
end
up
with
a
single
percept
from
that
there's,
another
variation
on
that,
which
is
what
something
super
tight
brought
up
earlier,
which
is
like.
Maybe
it
has
to
do
with
what
columns
and
where
columns,
maybe
what
columns
and
where
comes
divide
this
problem
up.
Somehow.
A
This
is
a
variation
in
number
two,
where
perhaps
what
columns
could
vote
on
object
id
and
where
columns
could
vote
on
orientation
relative
to
the
body,
because
we
think
of
where
columns
are
generally,
they
represent
locations
relative
to
the
body
in
some
sense,
but
it
again
it's
like
a
centralized
part
of
my
body.
It's
not
like
relative
to
my
hand,
it's
like
relative
to
my
entire
body.
I
don't
think
about
where
the
coffee
cup
is.
Unless
I
tend
to
it.
A
I
don't
think
about
what
a
cup
of
coffee
is
relative
in
my
hand,
I
just
perceive
it
at
some
or
relative
my
eyes.
I
see
it's
perceptually
relative
to
my
body
and
then
there
could
be
other
solutions
to
this
and,
I
say
recall
the
perceived
orientations
relative
to
the
body,
not
the
sensor,
so
this
implies
some
sort
of
pooling
to
construct
orientation
relative
to
the
body.
I
could
take
all
these
different
column
orientations
and
somehow
they
relate
to
the
body
orientation.
A
But
you
know,
and
marcus
again
was
the
first
person
to
point
out.
There's
other
cortical
areas
related
that
that
highly
connected
to
the
cortex
you're
gonna
be
playing
this
role
to
be
involved
in
this,
whether
it's
the
mystery
atom
or
the
retrospinal
cortex,
there's
other
areas
that
could
be
involved
in
this
perception
of
orientation
modes
of
the
body.
A
So
I'm
going
to
end
with
my
presentation.
We
can
continue
the
discussion,
but
I
felt
these
are
more
constraints
on
the
problem.
I've
introduced
today
the
constraints
of
scale,
the
constraints
of
trying
to
merge
the
vision
and
the
touch
sensations
together
in
some
way,
and
also
this
whole
idea
that
object
orientation
relative
to
the
body
is
part
of
our
perception
which
we
haven't
really
talked
about.
As
far
as
I
recall
in
the
past.
A
A
E
That
would
argue
that,
at
least
for
that
problem
you
need
a
blending
of
what
and
where
to
come
up
with
the
resolution,
I
mean
you're,
looking
at
a
different
problem,
you're
looking
at
body
orientation
with
respect
to
everything
else,
but
it
would
seem
like
you
still
need
what
and
where,
even
if
you're
trying
to
assemble
the
components
of
the
features
and
saying
these
things
make
a
coherent
hole,
you
need
both
of
those
things
too.
I.
A
A
A
B
A
And
in
all,
including
everything,
all
the
things
we've
talked
about
eye
movements,
you
know
locations
orientations
all
that
stuff
is
all
taken
care
of
in
the
in
the
what
pathway.
But
if
you
don't
have
the
wear
pathway,
then
you'll
say
there's
a
coffee
cup
at
this
orientation,
but
I
don't
know
where
it
is
relative
to
my
body.
That's
what
people
with
the
damaged
worst
pathway
say
they
say
I
can
see
the
coffee
cup.
I
can
see
its
features.
I
know
what
it
is,
but
I
don't
know
where
it
is.
I
can't
reach
for
it.
A
E
A
What
that's
what
the
the
empirical
evidence
we
have
says,
that's
what
our
theory
is,
what
we
propose
say:
yes,
so
it's
all
about
if
the
reference
name
is
relative
to
the
body
or
the
reference
relative
to
the
object,
and
this
is
why
this
idea
that
I
recognize
the
orientation
of
the
object
in
my
perception
of
recognizing
an
object.
You
know
even
just
sticking
my
finger
into
the
black
box
feeling
around
sensing
a
coffee
cup.
I
now
have
this
orientation
of
the
entire
coffin.
A
I
perceive
the
coffee
cup
as
being
a
coffee
cup
at
some
orientation
to
my
body
that
seems
to
occur.
You
know,
I
don't
that's
my
first
step.
I
don't
need
to
know
that,
as
far
as
I
can
tell
to
recognize
the
coffee
cup,
but
that's
what
I
perceive
so
I'm
just
saying
that
somehow
this
pops
out
when
we
recognize
something
and
I'm
suggesting
that,
maybe
then
the
where
columns
are
involved
in
that
perceptual
experience,
if
not
the
recognition.
A
Somehow
again,
I'm
separating
two
orientations:
the
orientation
my
finger
to
the
coffee
cup,
which
is
very
different
than
the
coffee
cup.
In
my
body
I
don't
perceive
the
orientation.
My
finger
to
the
coffee
cup
unless
I
attend
to
my
finger,
but
I
always
perceive
the
orientation
of
the
coffee
cup
to
my
body
and
that's
the
thing
that's
weird
about
it.
Perhaps
if
I
had
a
damaged
wear
pathway,
I
wouldn't
perceive
the
orientation
of
the
coffee
cup
in
my
body,
I
would
just
say:
there's
a
coffee
cup
out
there.
I
don't
know
it's
orientation.
F
I
was
going
to
say,
I
think,
there's
one
of
these
classic
studies.
I
remember
it's
like
milner
at
all
or
something
with
this
case
study
df.
I
think
she
had
a
a
parietal
kind
of
wear
lesion,
but
she
could
she
could
describe
the
orientation
of
bars,
for
example,
and
and
or
kind
of
their
angle
and
and
had
some
kind
of
information
about
that,
but
she
couldn't
like
orient
a
an
item
to
put
through
like
a
letterbox.
F
A
A
So
in
that
theory
I
mean
this
is
all
right.
Theoretically,
why
I
would
think
about
okay?
Well,
they
would
they
would
be
able
to
perceive
the
the
the
orientation
of
a
feature
relative
to
some
object,
but
again
they
wouldn't
be
able
to
know
where
that
object.
Where
that
feature
is
relative
to
their
body,
so
they
would
not
be
able
to
reach
and
pass
a
letter
to
the
spot,
just
in
the
sense
that
they
can't
they
can't
reach
out
and
grab
anything
if
you've
damaged
the
whereway,
the
general
rule
is
you
can't
reach
it?
A
F
A
F
A
A
A
I
think
if
you
had
a
damage
to
the
wet
patch,
but
you'd
still
see
the
bar
the
stripe-
and
you
say:
oh,
that's
a
horizontal
stripe
or
a
vertical
stripe
or
an
angle
swipe,
but
you
wouldn't
be
able
to
tell
me
what's
its
relative
position
to
the
to
the
object
that
it's
on.
So
if
I
just
showed
you
a
striper
I'd,
say:
yeah,
it's
vertical!
It's
horizontal
diagonal!
A
F
A
Another
problem
with
a
lot
of
these
studies
is
that
when
they're
lesion
studies
like
then
they're
not
very
well
controlled,
and
so
you
don't
know
what
damage
extent
of
the
damage,
and
you
know
how
much
of
its
damage
was
the
damage
on
the
other
side,
it's
just
a
lot
of
things.
They
don't
know
so,
especially
with
humans
right.
You
just
they're,
not
going
to
go
out
and
and
say:
oh
well,
you
you
know
you're
you're,
this
region
was
damaged.
We're
going
to
remove
the
whole
thing,
just
make
sure
something's
still
operating
right.
A
A
They
seem
like
one
thing
and
it's
it's
puzzling
to
me
how
that
comes
about
and
both
the
mechanisms
and
the
perception
of
it,
and
it's
also
interesting.
Obviously,
if
I'm
seeing,
if
I'm
observing
the
cup
at
this
particular
orientation
to
my
body,
I
can
still
identify
the
cup
right.
I
can
independent
of
its
orientation.
A
I
would
say:
that's
this
particular
coffee
cup,
so
I'm
able
to
extract
out
that
information
either
from
the
the
dual
representation
or
maybe
there's
two
sets
of
neurons,
the
ones
that
the
object
and
I
could
just
label
the
object
and
then
that
again
bring
that
number
two
and
that
in
number
three
and
that
again
brings
up
the
question
as
well.
Why
I
perceive
them
as
this
thing?
Why
do
I
have
one
person?
You
know?
I
don't
know.