►
From YouTube: Jeff Hawkins on Object Modeling in the Thousand Brains Theory (Part One) - September 3, 2021
Description
In continuation of the last two meetings where Subutai discussed voting in the Thousand Brains Theory, Jeff looks at the overall theory and focuses on the missing elements - things we haven't implemented, things that have changed due to new insights or things that are still unknown.
He first presents the two core ideas proposed in our Columns paper (2017), and then walks through a list of shortcomings of the hypotheses and simulation. He then gives an overview of our Frameworks paper (2019) where we hypothesized that objects are modeled based on grid cell modules. He highlights how the paper solved a few issues in the Columns paper but points out that there are still previous shortcomings and problems that we haven't addressed.
He wraps up the meeting by discussing the ways we’re currently trying to address these problems in our research. He proposes new ways of thinking about how we model objects, how cells represent location and orientation of the sensor and how columns detect movements and path integration. He expands these ideas further in the next meeting.
➤ Columns paper: https://numenta.com/neuroscience-research/research-publications/papers/a-theory-of-how-columns-in-the-neocortex-enable-learning-the-structure-of-the-world/
➤ Columns+ paper: https://numenta.com/neuroscience-research/research-publications/papers/locations-in-the-neocortex-a-theory-of-sensorimotor-object-recognition-using-cortical-grid-cells/
➤ Frameworks Paper: https://numenta.com/neuroscience-research/research-publications/papers/a-framework-for-intelligence-and-cortical-function-based-on-grid-cells-in-the-neocortex/
- - - - -
Numenta is leading the new era of machine intelligence. Our deep experience in theoretical neuroscience research has led to tremendous discoveries on how the brain works. We have developed a framework called the Thousand Brains Theory of Intelligence that will be fundamental to advancing the state of artificial intelligence and machine learning. By applying this theory to existing deep learning systems, we are addressing today’s bottlenecks while enabling tomorrow’s applications.
Subscribe to our News Digest for the latest news about neuroscience and artificial intelligence:
https://tinyurl.com/NumentaNewsDigest
Subscribe to our Newsletter for the latest Numenta updates:
https://tinyurl.com/NumentaNewsletter
Our Social Media:
https://twitter.com/Numenta
https://www.facebook.com/OfficialNumenta
https://www.linkedin.com/company/numenta
Our Open Source Resources:
https://github.com/numenta
https://discourse.numenta.org/
Our Website:
https://numenta.com/
A
A
A
A
Today,
I'm
going
to
talk
about
the
stuff
we
don't
know,
so
I'm
going
to
look
at
the
overall
theory
focusing
on
what
is
missing
what
we
haven't
implemented
or
haven't
figured
out
yet
some
of
the
stuff.
We
know
pretty
clearly
what
it
has
to
do.
We
just
don't
know
we
haven't
done
it
something
some
of
the
stuff.
We
don't
know
how
to
do
it.
Some
of
the
stuff
we
just
reclue
this
some
of
it's
dynamically
changing.
A
So
that's
the
overall
picture,
there's
a
lot
of
material.
I
have
not
a
lot
of
slides,
but
a
lot
of
material
and
it'd
be
really
best,
as
we
did
in
the
last
two
meetings.
If
you
are
confused
by
something
I
don't
understand
the
word
I'm
using
or
something
like
that.
Just
jump
in
interrupt,
there's
no
point
going
forward
if
we're
not
all
understanding
the
same
terminology,
so
if
we
don't
get
it
through
today,
we
won't
get
through
it
today,
but
hopefully
we
can
we'll
see.
A
A
I'll
count
the
three
okay
we'll
just
go.
I
thought
I'd
go
somewhat
sort
of
chronologically
through
this
thought,
experiment
and
then
two
papers,
we've
written
and
use
those
as
a
basic
framework
for
the
conversation,
so
you're
all
familiar
with
the
thought
experiment
of
touching
the
fingers
of
the
coffee
cup.
This
was
a
real
eye-opener.
For
me,
it
was
a
big
big
deal,
because
a
lot
of
things
became
clear
that
I
just
never
knew
before
and
I'll
just
list
them
here.
A
A
So
that
was
that
was
the
basic
insight
like
you
know,
when
your
your
finger
knows
what
it's
going
to
sense
and
therefore
it
knows
it
needs
to
know
where
it's
going,
where
it
is
relative
to
the
object
that
it's
scenting
in
order
to
to
know
where
it
is,
it
has
to
have
some
sort
of
reference
frame.
That's
attached
to
the
object,
and
you
know.
Otherwise,
how
would
how
would
you
represent
that
it
has
to
have
some
representation
for
that?
We
also
knew
that
path.
A
Integration
is
occurring
because,
as
you
move
your
finger
relative
to
the
object
again,
not
even
looking
at
it
just
moving
you,
you
could
predict
in
advance
what
you're
going
to
feel.
Therefore,
you
have
to
be
able
to
predict
in
advance
where
your
finger
is
going
to
be
so
there's
some
sort
of
path.
Integration,
that's
occurring
there.
This
idea
that
there
was
a
reference
frame
attached
to
an
object
in
a
single
column
was
a
revelation
for
me.
It's
like,
oh,
I
could
never
even
refer
to
it.
A
If
you
ask
people
like,
oh
the
reference
frames
of
the
brain
sure
all
over
the
place,
we
knew
they
had
the
solvent
bulkhead
the
motor
problems,
but
the
idea
that
every
single
column
had
this
was
was
the
thing
that
was
surprising
to
me,
and
then
we
quickly
infer
that
you
can
given
this
information.
You
know
where
the
finger
finger
knows
where
it
is
relative
to
the
object
and
it's
sensing
different
things.
Then
a
column
can
learn
a
three-dimensional
model
by
integrating
location
and
sensation.
A
Over
time
I
say
nd
because
I
don't
think
it's
restricted
to
three
dimensions:
it
can
learn
whatever
it
is,
but
we'll
think
about
three-dimensional
objects
and
and
again
following
mount
castle's
hypothesis,
which
is
sort
of
bedrock
for
us
every
column,
column
is
doing
the
same
thing.
This
means
every
quarter.
Column
works
this
way.
A
So
all
of
a
sudden,
we
have
this
idea
that
every
quarter
of
a
column
has
its
reference
frames
and
is
integrating
over
time,
and
that
is
going
to
be
the
basis
for
everything
not
just
touch
and
seeing,
but
language
and
high
level
thought,
even
if
we
didn't
understand
that
yet,
but
these
were
going
to
be
components
of
that.
So
the
first
paper
we
wrote
about
this
was
the
columns
paper
which
came
out
in
october
2017,
and
the
columns
paper
made
two
major
claims.
I
mentioned
this
the
other
day.
A
The
first
major
claim
was
every
each
and
every
column
was
complete
models.
This
is
something
that
I
don't
think
anyone
had
thought
about
before.
Maybe
they
did,
but
we're
not
aware
of
it
to
me
was
a
really
big
idea,
and
it's
really
why
we
had
to
get
the
paper
out
there,
even
though
our
model
for
how
this
worked
was
very
impoverished,
as
you
see
in
a
moment.
A
A
The
second
big
idea
we
introduced
in
the
columns
paper
was
the
voting
between
columns
and
this
this
solved
a
whole
bunch
of
mysteries.
The
two
primary
ones
were
the
stability
of
perception
and
again,
when
you're
looking
at
the
world
you're,
not
aware
your
eyes
are
moving
bruno.
Also,
someone
said
to
me
says
you
know
if
you
can
solve
the
stability
perception
problem.
You
know
that
deserves
a
nobel
prize
when
he
recently
understood
that
we
did
this.
A
He
goes
that's
pretty
interesting,
but
I
didn't
remind
him
what
he
said
earlier,
and
it
also
explained
so
many
of
these
connections
that
we
see
in
the
cortex
that
are,
we
I'll
call
them
lateral
they're.
Just
these
are
these
non-hierarchical
connections
and
they
go
all
over
the
place
and-
and
there
was
no
explanation
for
them,
and
so
these
are
two
big
ideas
that
the
voting
implemented.
A
A
So
let
me
just
walk
down
through
these.
I'm
not.
I
tried
to
be
inclusive
here
to
capture
everything,
but
I
probably
didn't
so,
but
there's
a
lot,
so
we
didn't
know
how
neurons
could
represent
reference
frames
and
locations.
We
when
we
wrote
the
paper,
we
had
absolutely
no
idea
so
what
we
did
in
the
paper.
We
said:
okay,
we'll
just
we'll
just
assign
random
sdrs
to
locations
and
use
those
and,
at
the
very
end
of
the
paper
we
kind
of
we
kind
of
knew
that
we
were
about
to
submit
it.
A
A
So
when
you
touch
the
cup,
what
you
feel
is,
depending
on
the
orientation
of
the
finger,
it's
not
just
the
location,
you
can
rotate
your
finger
around
that
point
and
you'll
feel
different
things,
and
we
can
predict
that
and
so
there's
this
concept
of
orientation
of
the
sensor.
That
also
has
to
go
under
path,
integration
and
you
need
to
comp.
You
need
to
know
about
the
orientation
of
the
sensors
in
order
to
make
predictions,
and
we
didn't
have
that
anywhere
in
the
paper.
I
think
we
might
have
mentioned
at
the
end.
A
I
don't
remember,
but
we
certainly
learned
this
later,
so
that
was
a
big
hole
at
this
time.
We
knew
that
objects
were
composed
of
other
objects.
That
was
something
that
you
know
it's
kind
of
been
known
for
a
long
time,
but
it's
not
heavily
discussed
in
neuroscience,
and
this
this
model
didn't
have
anything
to
do
with
that.
This
is
point
number
three.
A
A
We
also
knew
that
objects
had
to
have
behaviors,
meaning
that
they
they
they
they.
We
have
to
learn
how
they
morph
over
time,
how
they
change.
You
know
when
we
interact
with
them,
how
they
move
on
their
own
examples.
We,
of
course,
use
like
staplers
and
doors
and
smartphone
apps
and
things
like
that.
That's
all
part
of
how
object
modeling-
and
there
was
no
concept
of
that
in
this
paper-
a
really
really
big
hole.
A
Number
four
here
is
it
didn't
work
with
vision,
it
kind
of
glanced
over
it
instead
of
pretending
to
work
with
vision,
but
it
really
didn't
because
the
problem
with
the
vision
is
the
sensor
is
at
a
distance.
Your
ar
your
retina
is
not
touching
the
object
with
the
finger
on
the
coffee
cup,
we
can
say
yeah
the
location.
The
finger
is
in
the
reference
frame
of
the
coffee
cup.
That's
what
the
sensation
is.
We
can
just
record
that,
but
with
without
with
your
eyes,
it's
not
like
that.
A
A
You
can
also
see
it
in
touch
if
you're
like
holding
a
tool
in
your
hand-
and
you
know
you're
using
the
tool
as
opposed
to
your
finger,
you
can
you
can
sort
of
do
all
the
things
you
can
do
with
your
finger,
and
yet
your
fingers
are
no
longer
at
that
at
the
location.
So
that's
number
four
number
five
relates
to
the
to
the
orientation
issue
is
that
we
talked
about
voting
representing
object
id,
but
but
we
know
that
we
perceive
not
just
the
object
id.
A
When
I
touch
something
or
look
at
something.
I
don't
just
see.
Oh
it's
a
copycat,
I
know
its
location
and
its
orientation
relative
to
me,
and
so
those
are
things
that
have
to
be
resolved
as
well,
so
voting.
You
know
we
have
to
those
things
have
to
be
resolved,
they're
going
to
be
solved
by
voting,
so
voting
was
incomplete.
We
just
talked
about
object
id.
A
There
was
no
mention
of
how
motor
generation
could
be
used
to
solve
goals.
You
know
and
and
then
there
was
I
I've
used-
the
word
transfer
learning
here
and
I'm
afraid
to
do
that,
because
I
know
it
means
something
in
machine
learning
and
it
probably
means
something
completely
different
in
machine
learning.
So
this
is
the
term
that
we
were
using
the
other
day
for
disjoint
pooling
it's
what
we
used
in
the
past.
Let
me
just
explain
what
it
is
and
and
then
I'll
stop
the
questions.
A
A
This
column
learns
them
out
of
the
coffee
cup,
but
now
I
can
I
can
say
I
can
learn
a
bunch
of
objects
that
way
using
my
index
finger
or
my
right
hand,
but
now
I
can
reach
into
a
box
with
a
bunch
of
objects,
and
I
can
touch
it
with
my
finger
on
my
left
hand
and
I'll
be
able
to
infer
those
objects,
I'll
be
able
to
say
yep.
This
is
the
this
is
the
object,
and
this
is
b
even
I
learned
it
with
my
right
hand,
or
I
could
do
it
with
my
toe.
A
I
could
put
my
toe
in
the
box
and
rub
it
around
the
coffee
cup
bureau.
What
it
is,
or
you
know
I
can
do
any
part
of
my
body
and,
of
course,
this
composed
across
modalities
as
well,
when,
if
I
learned
where
the
several
objects
looked,
you
know
felt
like
with
my
finger,
but
I
never
saw
them
and
then
I
showed
them
to
you.
A
A
How
it
is
that
knowledge
about
a
mod
object
model
was
learned
in
one
set
of
columns,
but
we
know
that
the
other
columns
have
to
be
able
to
be
able
to
use
that
model
or
have
their
own
model,
or
something
like
that.
So
that's
what
I
mean
by
transfer
learning
in
this
case,
and
I
don't
know
if
it's
the
same
as
other
people
use
it.
So
why
don't
I
stop
here
for
a
second
and
see
if
this
question
is
about
any
of
this
so
far,.
B
C
A
What
do
I
mean
by
motor
generation,
every
column
in
the
cortex
has
has
has
cells
that
generate
behaviors
all
right
and
they're
in
layer
five
and
they
every
column
has
them
as
far
as
we
know,
and
so
it's
not
that
these
are
just
inference
engines.
These
are
inference
motor
generation
engines.
You
know
the
the
columns
create
my
speech.
Problems
make
your
fingers
move
columns,
make
your
eyes
move,
so
we
didn't
talk
about
that.
Yeah
at
all.
We
just
talked
about
how
a
column
could
could
use
incoming.
A
A
A
Everything
just
think
about
all
the
things
you're
in
your
projects
does
through
movement.
We
have
to
explain
that
right.
It
generates
it
generates
speech
it
types
on
computers,
it
opens
doors
it
you
know,
cooks
on
the
stove,
I
mean
we
do
all
these
things
and
you
know.
Ultimately,
we
have
to
explain
all
that,
so
I
don't
want
to
get
it
yeah.
It
could
be
exploration
too.
You
know
I'm
trying
to
not
go
down
to
that
level
of
detail
right
now,
just
it's
a
big
black
hole.
We
didn't
talk
about
motor
generation,
okay,.
C
A
A
Remember
in
the
in
the
conversation
we
had
the
last
meeting,
we
were
talking
about
voting
for
object
id,
so
you
can
see
this
in
the
upper
right
picture
here
and
that
there
was
a
set
of
cells
that
represent
the
the
object,
the
coffee
cup
and
as
your
finger
moves,
the
input
to
the
finger
would
change
its
location
on
the
coffee
cup.
Would
change,
but
the
object
id
would
not
change
right.
It's
still
the
coffee
cup.
It's
not!
It's
not
a
different
object,
just
because
your
finger
is
moving
and
sensing
something
different.
A
You
follow
that
so
far,
quran
yeah
got
it
now.
This
is
important.
I'll,
come
back
to
this
point
later
you
are
only
able
to
perceive
consciously
or
able
to
perceive
the
voting.
That
is,
you
can
say.
I
am
seeing
a
coffee
cup
but
you're
you're,
not
aware
that
your
individual
columns
are
whacking
all
over
the
place.
You
know
changing
on
and
off
as
they
move
as
your
eyes
are
scanning
over
this
thing
you
know
it's
incredible
activity
but
you're,
just
not
conscious
of
it.
A
It's
not
available
to
you
if
that
activity
is
locked
up
inside
the
columns
and
the
only
thing
you're
aware
of
what
is
spread
out
beyond
that
in
in
the
voting
signal.
So
your
perception
what
you
perceive
is
stable,
even
though
the
activities
of
most
of
the
column,
the
cells
and
the
columns
are
not
stable.
A
D
A
We
assumed
I
went
through
this
less
I'll,
then
I'll
try
to
do
it
again.
So
if
I
pick
jump
in,
but
we
did
discuss
this
so
this
is
really
good.
We're
picking
these
things
up,
I'm
not
saying
this
say:
hey
how
come
you
didn't?
Learn
it
it's,
because
these
are
hard
concepts
and
it's
hard
to
learn.
So
we
have
to
do
it
multiple
times.
A
A
A
They
want
to
all
they're
happy
to
you
know
it's
a
self-reinforcing
group
and
they're
like
yep,
we're
all
when
we,
when
we
think
of
the
coffee
cup,
we're
all
going
to
be
reinforcing
each
other.
They
will
also
reinforce
the
coffee
cup
sdr
in
other
columns
nearby
or
far
away,
and
so
it's
it's
a
mutually
reinforcing
mechanism
where,
if
they
get
input
from
well,
basically,
that's
it
it's
a
self.
It's
a
it's
a
cell
assembly
in
the
old
language
of
heb
that
that
reinforces
itself
and
I'm
not
that's.
D
I
I
guess
I
I
don't
know
the
the
I
I'm
just
trying
to
cleave
two
concepts
here,
so
you
have
the
notion
that
there
are
multiple
aspects
or
contributing
aspects
to
an
object,
identity
that
can
come
from
multiple
columns
and
then,
as
you
get
different
percepts
of
that
same
object,
there's
a
confluence
where
it
says
it's
still
the
same
object.
It's
still
the
same
object.
D
So
I'm
looking
at
the
case
where,
okay,
so
I've
got
an
amount
of
coffee
cut
up.
There
looks
like
in
the
middle
coffee
cup.
You
know
it's
got
the
metal
on.
It's
got
the
logo
and
stuff
like
that,
and
I
rotate
it
around
and
suddenly
I
see
a
target
symbol
like
like
the
like.
The
the
big
box
store
target
on
the
back.
Okay.
Well,
that's
different!
A
Basically,
why
does
an
object
become
something
different?
If
you
recall
super
tight
talked
about
these
cells
in
the
voting
layer,
get
input
from
two
sources:
there's
a
feed
forward,
input
which
is
coming
from
the
column
itself.
This
is
sensory
driven
right.
This
is
like
saying,
oh
at
this
location
in
this
paper,
we
would
describe
it
the
following
way:
we'd
say
at
this
location:
there
is
a
feature
or
sense
thing
and
I
would
represent
that
and,
and
that
does
not
fit
into
the
set
of
sensory
feature
pairs
that
we
ascribe
to
the
coffee
cup.
A
The
metacoffee
cup
does
not
have
a
target
logo
on
it.
So
if
I
see
the
target
logo
that
is
now
going
to
try
to
activate
something,
it's
not
going
to
reinforce
some
momentum.
In
fact,
the
the
input
to
the
object
layer
will
be
different.
It
will
be,
it
will
be,
not
be
one
of
the
things
that
belongs
there
and
it
might
say,
oh
well,
then
this
is
a.
This
is
a
target
cup.
B
D
A
A
How
it
is
that
it's
you
know
going
between
multiple
columns
and
wipe
lids
is
to
to
recognize
something
with
fewer
movements
of
the
sensors.
These
are
important
concepts
for
right
now,
it's
not
important.
Actually,
it's
not
even
important
the
mechanism
of
voting
right
now
that
was
that
was
discussed
because
two
days
I
don't
mind
going
over
it
again,
but
it's
not
important
for
what
I'm
trying
to
get
across
today,
because
there's
lots
of
little
mechanisms
in
here.
A
So
I
want
to
stick
to
sort
of
conceptual
ideas
here
and,
if
anyone's
confused
about
some
conceptual
ideas
that
would
be.
I
want
to
know
that
and
we
can
we
can
bring
offline
later
having
questions
about
like
okay,
what's
the
threshold
of
difference,
you
know
that's
a
very
well
interesting,
but
very
new
question.
In
some
sense,
I.
D
A
A
We
did
not
talk
about
that
at
all.
I
will
talk
about
that
later
vivian,
so
I
didn't
mention
you
know.
I
said
this
is
a
list
of
shortcomings
and
I
said
I
tried
to
be
inclusive,
but
I
didn't
make
it
inclusive
of
everything.
Well,
the
attention
is
a
lot
of
things
I
didn't
put
in
there.
So
we'll
just
put
that
in
the
list
number
eight
okay
attention,
because
it
is
yeah.
I
can
do
it
right
now.
There
we
go
never
just
put
attention
in
there.
A
F
A
A
That
is
that's
wrong.
As
I
pointed
out,
models
are
composed
of
objects
relative
to
other
objects,
and
so
now
the
question
is:
is
how
many
you
know
how
many
compositional
objects
can
a
column
learn
or
how
many
relationships
between
compositional
components
right?
If
you,
if
you're
into
motorcycles-
and
you
see
a
new
motorcycle,
you
don't
have
to
learn
what
the
chain
does
and
you
don't
have
to
relearn
what
the
wheels
do
and
you
have
to
relearn
what
the
throttle
does
you
know
you
know
these?
A
Are
these
are
concepts
you
already
have
and
you're
just
rearranging
them
again
so
and
in
our
simulations
in
this,
I
think
it
was
in
this
paper.
No,
I
think
it
was
a
columnist
plus
paper.
Maybe
I
forget
we.
We
showed
that
it's
very.
It
was
all
fake
data.
It
was
all
made
up
data,
but
we
showed
that
a
column
could
put
hundreds
of
sort
of
complex
objects.
A
It's
not
that
a
column
can
learn
everything.
It
can't
and
that's
an
interesting
question,
but
at
the
moment
it
seems
to
be
much
more
capable
than
you
might
think,
and
we
don't
want
to.
We
don't
want
to
just
say:
oh,
we
can't
do
this.
We
have
to
do
it
someplace
else.
We
understand
how
it
is.
Let's
just
give
me
one
second
here
I
just
want
to
mute
myself
for
a
second.
B
Okay,
maybe
I
can
make
one
point
there
as
well:
a
very
common
kind
of
confusion
we
get
is
that,
even
though
we're
seeing
saying
that
each
and
every
column
learns
complete
models
of
object,
we
are
not
saying
that
every
column
learns
every
object
right.
It's
there's
a
distributed
knowledge
here,
so
every
object
will
have
hundreds
or
thousands
of
columns
that
have
models
of
it,
but
there's,
let's
say
a
hundred
thousand
columns.
So
there's
you
know
99
000
columns
that
don't
have
models
of
that
object.
B
So
a
very
you
know
it
may
be
that
some
parts
of
your
motorcycle
are
stored
in
some
parts
of
some
subset
of
cortical
columns,
some
other
parts
of
the
motorcycle,
maybe
in
some
other
parts
of
some
other
cortical
columns,
we're
not
saying
that
every
cortical
column
has
models
of
every
single
object
that
you
know.
Yeah.
A
A
So
there's
that
division
and
there's
even
within
the
somatosensory
system
and
the
visual
system,
there
would
be
division
as
well.
You
know
some
visual
columns,
we'll
cover.
We
talked
about
different
scales
and
so
on.
I
think
I
think,
I'm
very
comfortable
with
the
idea
that
when
I
say
they
learn
complete
objects,
what
I
mean
by
that
you
know
is
it.
This
goes
against
the
traditional
thinking
of
neurosciences,
and
there
is
nobody
in
neurosciences,
I
knew
of
would
have
said
a
column
in
b1
or
v2
or
even
v3
would
know
what
object
it's
sensing.
A
They
just
say.
No,
that's
not
possible,
though
those
columns
are
just
learning
features.
They
don't
really
have
any
concepts
at
all
about
what
the
overall
bigger
picture
is
that
happening
somewhere
else
in
the
brain
somewhere
else
in
the
neocortex,
so
that
why
I
say
this
is
a
big
idea
when
I
say
complete
model,
I'm
not
saying
a
column
knows
everything
like
super
dice
said
what
I'm
saying
is
that
this
idea
is
so
crazy
to
most
neuroscientists
just
that
he
can
learn
any
kind
of
model
it
is
is
like
a
you
know.
A
It
just
was
against
every
kind
of
thinking
that
people
were
thinking
about
how
sensory
information
is
processed
in
the
cortex.
You
know,
columns
just
extracted
features,
and
somebody
else
did
the
recognition
and
that's
how
they
viewed
it,
at
least
for
the
for
the
several
first
levels
of
sensory
cortex.
So
we're
saying
that's
not
true.
Even
a
one
level,
v1
system
can
learn
objects
and
know
and
build
three-dimensional
models,
and
that
is
a
real
that's
why
I
say
this
is
a
big
idea.
No
one
thought
of
this.
F
Yeah
yeah
I
got
I
was
alerted
to
this
when
you
said
that
you're
surprised
or
that
somebody
should
be
surprised
that
they
can
feel
something
with
their
left
hand.
Let's
say
that
they
learned
with
their
right
hand,
but
then,
basically,
that
the
the
explanation
of
shubutai
that
these
things
are
let's
say
in
multiple
places
and
somewhat
distributed,
then
yeah.
Then
then
it
then
that
would
not
yes,.
A
The
point
the
point
I
brought
up
under
problem
number
seven
was
we
had
no
explanation
for
how
how
a
column
that,
representing
your
left
hand,
could
learn
a
model
that
was
only
you
know,
learned
by
your
right
hand,
there
was
no
explanation
for
it.
It
was
like
we
lacked
any
kind
of
theoretical
mechanistic,
any
explanation
whatsoever.
How
how
multiple
columns
I
you
know
I
could
learn
the
type.
A
A
I
think
no
one's
surprised
that
we
can
recognize
something
by
touching
with
your
left
hand,
after
you've,
blended
with
your
right
hand,
but
it
just
where
is
that
model
and
how's
it
done,
and
how
does
it
get
to
the
left
hand?
That
kind
of
thing
that
was
that
was
a
sort
of
different
fundamental
question.
A
E
Yeah
to
to
what
you
just
talked
about,
how
much
would
you
say,
the
modeling
capability
of
a
single
column
is
impacted
by
top-down
feedback
from
from
other
columns
or
areas
and
also
another
feedback.
So
if
you
have
a
column
and
then
you
separate
it
from
all
the
columns
around
it,
would
it
still
be
able
to
model
a
full
object
just
from.
A
Area
yeah,
I
mean
again
we're
theorists
here
baby,
and
so
you
know
we
go
to
extremes
on
things
right
and
then
in
the
real
biology.
It's
much
messier
than
this,
of
course.
But
in
the
theory
we
would
say
a
single
column
is
sufficient
to
learn
anything
like
kumar,
I'm
not
anything,
but
it's
efficient
to
learn
a
model
of
its
input,
our
that's
how
we
modeled
it
in
our
simulations.
A
A
The
other
connections
we've
already
talked
about
how
we
don't
know
I'm
going
to
talk
about
in
a
moment,
but
how
transfer
learning
occurs
because,
obviously
I
can
learn
about
my
left
hand,
can
learn
a
model
that
my
right
hand
learned.
But
how
did
it
do
that
we
haven't?
We
didn't
propose
that
in
this
paper,
but
clearly
it
can
happen,
but
the
theory
doesn't
say
you
have
to
do
that.
A
The
theory
says
I
could
have
a
one
column
neocortex
and
it
wouldn't
be
very
interesting,
but
it
would
still
learn
a
model
of
the
world
and
make
predictions
and
and
and
that's
what
it
would
do,
and
when
you
talk
about
things
like
top
down,
that's
a
loaded
term.
It
could
mean
lots
of
things
it
could
be
for
other
regions
of
the
cortex
it
could
be
from.
You
know:
projections
subcortically,
emotional,
you
know
things,
you
know
neuromodulators.
A
I
don't
want
to
go
there
right,
we're
just
from
a
theory
point
we
were
saying
put
a
column.
Can
a
column
learn?
Yes,
it
will
and
we're
arguing.
Yes,
they
do.
They
learn
single
models.
Nothing
else
is
required
from
a
theoretical
point
of
view.
Biologically
a
lot
of
things
are
required.
Blood
is
required,
energy.
You
know
oxygen
whatever
you
know,
but
but
from
a
theory
point
of
view,
nothing
else
is
required.
It
would
be
a
limited
system,
but
it
would
work.
A
A
They
basically
just
have
to
learn
them
in
this
very
simple
example:
they're
learning
the
morphology
of
the
shape
of
the
object.
It's
a
three-dimensional
model
and
it's
complete,
I'm
not
when
I
say
complete,
I'm
not
saying
everything
that
we
know
about
the
world
of
coffee
cups.
It's
just
it's!
It's!
It's!
It's
complete
in
its
own
right
right!
It's
it's
not
like.
I
only
know
features,
I
don't
know
coffee
cups,
it's
no.
B
G
No,
that
was
kind
of
my
question.
I
guess
yeah,
that
that
does
kind
of
fly
in
the
face
of
my
intuition
here
with
how
like,
for
example,
like
columns
in
v1,
when
I
think
of
a
column
and
view
one,
for
example,
being
able
to
understand
what
it's
going
to
expect.
When
you
look
at
a
coffee
cup,
I
think
of
that
signal
of
a
coffee
cup
coming
in
from
other
like
higher
level
areas,
or
something
like
that.
Yes,.
A
Yes,
yes,
now
I
am
by
the
way
I
you
know
it
may
not
be
worth
anything
to
do,
but
I'm
very,
very
confident.
So
I
realize
this
is
flies
in
the
face
of
what
people
thought,
but
what
people
thought
really
didn't
make
any
sense
either.
No
one
had
any
idea
where
how
anywhere
in
the
car
in
the
u.s
and
neuroscientists,
where
in
the
brain,
is
the
structure
of
a
coffee
cup
store,
they
wouldn't
be
able
to
answer
that
question.
A
They
would
say.
Well,
you
got
these
features
coming
in
the
process
of
passed
up
the
hierarchy
and
some
place.
They
assembled
into
some
model
of
the
coffee
cup
and
what
mount
castle
told
us
is
the
most
amazing
discovery,
and
it's
true
is
that
the
columns
are
all
doing
the
same
thing.
So
if
the
cortex
is
going
to
learn
the
coffee
model,
the
coffee
cup
model
of
something
someplace,
it's
going
to
learn
models
everywhere
anyway.
This
the
whole
point
of
this
thought
experiment
to
explain
how
this
happened.
A
A
H
A
I
mean
we're
not
ruling
out
the
idea
that
the
input
to
another
cortical
column
could
be
output
of
one
quarter
column.
If
that's
what
you're
saying
yeah
yeah?
No
we're
not
really
back.
In
fact.
That's
almost
a
fact:
it's
a
certainty,
not
all
cortical
columns,
get
input
from
the
eyes
or
the
ears
or
any
sensor,
and
they
get
improved
from
other
things.
But
all
cortical
columns
are
learning
models.
It's
just
what
is
their
input?
It
can
vary
right.
One
can
get
input
from
the
eyes.
A
A
The
connections
we
talked
about
briefly
going
through
the
thousand
thomas,
represent
that,
but
it's
it's
not
denying
that
there
isn't
any
hierarchy:
people
structure
to
the
cortex.
It's
just
saying
that
every
column
is
a
complete
modeling
system
and
it
will
do
the
best
it
can
to
model
that
input,
given
whatever
input
it
has.
A
A
A
A
A
A
A
I
forgot
about
that
three
months,
three
or
four
months
later,
or
something
like
that
anyway,
I
I
wasn't
going
to
go
through
the
comms
plus
paper
carefully.
What
the
columns
plus
paper
I
interpreted
as
it
was
it
was.
It
was
following
up
on
the
on
the
the
paper
I
just
talked
about
a
moment
ago:
the
columns
paper
with
simulation
mathematical
analysis,
etc.
A
And
but
when
we
propose
this,
we
didn't
know
of
any
evidence,
and
we
didn't
know
of
it.
There
was
some,
but
we
didn't
know
it
any
evidence
that
good
cells
would
exist
in
the
ear
cortex
and
we
not
only
said
they
would
be
found
in
your
projects.
We
said
it'd
be
found
in
every
single
cortical
column
and
that
this
was
a
core
feature
of
cortical
columns
and
core
feature
than
your
cortex.
That
is
a
big
idea
and
it
certainly
was
again
a
surprising
prediction.
A
Now
it's
not
terribly
surprising,
because
we
know
grid
cells
exist
in
the
antaranna
projects,
and
so
it's
not
like
it's
unknown
thing.
But
at
this
point
we
didn't
know
about
any
evidence.
Otherwise
it
turned
out.
There
was
some,
we
didn't
know
about
it
and
now
there's
there's
more
and
more
evidence
coming
out
about
this,
and
this
is
almost
certainly
true
this
this
claim
in
this
paper,
and
so
we
went
further
than
grid
cells
in
the
grid
cells
are
found
in
the
anterional
cortex
and
the
whole
hippocampal
internal
cortex.
A
Complex
has
been
studied
a
lot
and
there's
a
whole
bunch
of
types
of
cells
in
there
and
mechanisms
that
are
very
well
studied
and
there
are
place
cells,
there's
grid
cells,
there's
these
type
of
vector
cells,
there's
head
direction,
cells
and
so
on.
We
made
the
proposal
in
the
framers
paper
that
many
of
those
mechanisms
would
be
recreated
in
cortex.
It
wasn't
just
grid
cells.
We
said
you
know,
there's
a
there's,
an
ensemble
of
things
that
are
going
on
over
in
the
in
the
hippocampal
complex.
A
A
A
But
if
you
have
multiple
grid
cell
modules,
as
shown
in
this
upper
right
figure
here,
that
shows
two
grid
cell
modules,
a
green
one
and
a
red
one
that
are
slightly
different
scale
or
different,
slightly
different
orientation.
If
you
read
cells
from
both
of
them
or
multiple
ones.
At
the
same
time,
you
get
a
much
higher
resolution,
representation
of
space,
and
so
we
said,
okay,
we
propose
multiple
goods
and
modules
represent
location
of
a
sensor.
A
We
now
believe,
that's
probably
wrong,
we'll
get
into
that
in
a
moment,
and
what
this
did
it
was
interesting
is
that
when
you,
when
you
formed
a
representation
of
an
object
right
by
location,
so
excuse
me
every
location
was
unique.
That
is
there
wasn't
a
common
location
like
you
know,
one
one.
You
know
one
one
one
or
two
four
six,
oh
my
god,
it
would
apply
to
every
object.
The
neural
representation
for
locations
were
unique
for
the
object
and
the
location.
A
A
A
So
clearly,
we've
taken
something
we've
known
before
the
an
object
called
the
mental
logo
that
says
previously
learned
object
and
we've
taken
something:
a
coffee
cup
which
is
a
previous
zoom,
an
object
and
we're
able
to
link
them
together
in
a
very
fast
efficient
means
to
form
a
new
compositional
object,
and
we
knew
at
this
time
that
pretty
much
all
objects
were
composed
of
compositions
of
other
objects.
That's
kind
of
the
way
it
works,
that's
the
way
the
world
is
structured.
A
In
fact,
I
think
I
even
talked
about
this
in
on
intelligence,
but
but
how
did
that
happen?
It
was
really
tricky
to
understand
and
I
a
number
of
momentums
went
out
and
came
back
and
I
always
I
don't
know
who
actually
did
the
work
with
marcus
or
most
probably
marcus.
But
you
know
a
bunch
of
people
came
back
and
said
we
have
a
proposal
and
it
was
really
cool
and
the
proposal.
I
won't
walk
you
through
the
details
of
it.
A
We
don't
want
to
do
that
today,
it's
complex
and-
and
it's
probably
not
right
so,
but
we
came
up
with
this
idea
that
there
were
not
only
grid
cell
modules,
but
there
were
the
displacement
cell
modules
and
individual
displacement
cell
modules
would
sort
of
represent
the
displacement
between
one
reference
frame
and
another.
And
if
you
had
a
set
of
them,
then
you
would
have
you
see
you
could
very
rapidly
form
a
representation
of
where
is
the
nomental
logo
relative
to
the
copper
cup?
A
And
this
is
where
we
first
start
introducing
attention
to
vivienne.
The
idea
here
is
it
was
the
idea.
Is
you
attend
to
one
object
like
the
coffee
cup
and
then
you
attend
to
the
other
objects
like
the
mental
logo
or
any
two
objects
that
the
displacement
cells
would
automatically
every
time
calculate
the
displacement
between
those
two
objects?
A
So,
as
you
go
around
the
world
attending
the
things
in
which
what
you
do
all
the
time,
whether
you're,
wherever
or
not,
you're
going
around
looking
at
little
things
all
the
time,
one
after
another,
that
your
brain
is
constantly
determining
the
relative
position
of
those
objects
to
each
other.
So
as
you
attend
to
one
object,
then
another
displacements
automatically
calculate
that's.
What's
really
cool
and
most
people
aren't
even
talking
about
how
neurons
could
do
this
at
all,
and
we
had
now
had
a
neural
mechanism
for
how
it
would
happen.
A
We
also
then
proposed
a
mechanism
for
learning
how
object
behaviors.
We
use
the
example
of
stapler
and
it's
a
very
simple
concept.
The
stapler
essentially
has
multiple
components
and
some
of
those
components
move
relative
to
other
components
and
so
the
displacement
of
the
top
of
the
stapler
to
the
to
when
it's
open
to
the
display,
from
the
top
of
the
scraping
to
the
bottom
of
the
stapler
changes
as
you
open
it.
A
This
way,
and
that's
just
going
to
go
through
a
series
of
displacement
cells
activations,
because
this
is
that
one
displacement
there's
another
and
these
are
in
between,
and
this
is
just
a
sequence
and
we
could
use
our
sequence
memory
in
its
original
form,
in
some
sense
to
just
learn
a
sequence
of
displacements,
and
that
would
be
representing
how
the
components
of
an
object
move
relative
to
each
other
over
time.
And
that
is
what
the
definition
of
object
behaviors
are
and
how
things
are
changing,
not
just
how
they
move,
but
how
they
change
over
time.
A
A
We
also
introduced
the
term
thousand
brain
theory
and
the
the
the
big
idea
behind
that
is.
We
made
a
very
weak,
but
we
still
made
the
argument,
but
not
with
a
lot
of
force
to
it,
that
these
mechanisms
for
learning
compositional
structure
through
movement
and
learning
how
objects
behave
over
time
that
they
apply
to
all
parts
of
the
air
cortex
and
therefore
they
apply
to
all
parts
of
cognition
that
everything
we
do
is
going
to
be
built
on.
A
These
everything
we
think
about
is
high
level
thinking
and
planning,
and
language
is
going
to
be
built
on
these
same
basic
mechanisms,
because
if
these
mechanisms
are
in
one
column,
they're
in
all
the
columns-
and
we
couldn't
really
make
it
it's
hard
to
make
that,
but
that
was
a
very,
very
top-down
theoretical
argument
for
that
doing
this
paper.
We
introduced
new
problems.
A
So
then
they
go
through
the
new
problems
that
we
we
introduced.
It
required
too
many
grid
cell
modules.
Get
this
to
work.
You
had
to
have
a
whole
bunch
of
grid
cell
modules,
not
like
hundreds,
but
you
have
to
have
at
least
you
know,
eight
or
ten,
ideally
20.,
and
that
didn't
seem
possible
for
a
lot
of
reasons.
I
won't
go
through
the
reasons
why,
but
it
just
didn't
seem
right,
and
there
was
no
evidence
that
this
was
happening.
A
It
was
very
little
evidence
that
that
was
a
similar
type
of
thing
was
happening
over
the
toronto
project,
so
that
seemed
to
be
a
problem.
The
spice
themselves
had
many
limitations,
although
they're
really
cool
the
way
they
work,
they
didn't
really
work
for
most
situations.
They
had
no
way
of
handling
scale.
I
mean
like
representing
the
size
of
the
mental
logo
if
it
was
smaller
or
larger,
it
had
no
capability
of
doing
that.
It
did
not
have
the
ability
to
represent
the
orientation
of
the
mental
logo
on
the
coffee
cup.
A
So
if
the
logo
was
angled
at
45
degrees,
displacement
cells
could
not
handle
that
there's
no
concept
of
that.
It
didn't
have
the
ability
to
handle
distortions,
so
the
nomencla
logo
it
sort
of
wraps
around
the
cup.
It's
no
longer
its
original
shape
and
that's
a
simple
example
of
a
problem
that
occurs
in
a
lot
of
situations.
A
You
know
you
can
take
an
existing
object
and
distort
it
and
form
a
new
object
and
in
a
very
simple
way-
and
you
know
it
just
it-
there
was
no
explanation
for
that.
You
know,
and
so
we
could
learn.
We
knew
we
could
learn
these
things
very
quickly.
I
could
learn.
You
know
a
coffee
cup
with
this
small
logo
or
a
big
logo
or
an
angled
one
way
or
the
other
just
just
as
fast.
I
can
learn.
A
A
You
know
that
was
a
big
question
in
as
they're
typically
described
in
the
other
honor
courses.
They
don't
work
in
3d.
We
needed
3d
representations,
mercus
of
mercury,
marcus
and
merkel,
and
others
wrote
a
nice
little
paper
about
this,
where
you
could
have
a
lot
of
grid
cell
modules
and
show
you
how
they
could
represent
n-dimensional
spaces.
But
again
it
requires
a
lot
of
good
cell
modules
and
we
don't
think,
there's
really
a
possibility
for
a
lot
of
good
cell
modules
in
a
particular
cortex
protocol
column.
A
We
then
exposed
the
problem
we
knew
earlier,
but
we
now
kind
of
exposed
it
more
here
we
said:
okay,
imagine
look
at
this
upper
left-hand
corner
this
picture
up
in
the
left-hand
corner.
We
said.
Oh,
we
have
this
location
layer
made
of
multiple
grid
cell
modules.
That's
what
these
little
trapezoidal
things
are
representing,
and
it's
getting
some
motor
input
that
updates
this.
This
information,
so
like
okay!
Well,
how
is
that
movement
detected
and
where
does
it
come
from
and
how
does
it
update
location
and
orientation?
And
all
these
things
it's
like.
A
A
There,
I'm
sure,
there's
other
ones
and
of
course
we
had
our
previous
shortcomings,
which
we
didn't
address
this
this
paper
or
the
frameworks
paper.
We
still
didn't
model
talk
about
how
we
handled
the
orientation
of
the
sensor
in
the
world.
We
still
didn't
solve
the
problem
of
vision,
which
is
sensing
at
a
distance.
A
We
still
didn't
really
deal
with
all
the
issues
of
voting
and
we
didn't
include
motor
generation
and
again
no
transfer
learning
or
the
disjoint
pooling
thing.
So
those
were
not
attempted
here.
So
we
still
had
a
big
list
of
issues.
Now,
let's
stop
here
again
and
do
the
same
exercise.
We
did
before
kudos
for
the
first
person
to
ask
the
question
that
they
don't
understand
something.
I
Well,
one
thing
not
so
much
a
question
but
just
kind
of
a
look
behind
the
curtains
like
on
the
research
side.
Here,
often
in
these,
when
these
things
come
up,
these
flaws.
We
know
about
these
flaws
from
day
one.
It's
like
I'm
like
hey
look
displacement
cells
can
do,
can
do
compositions,
but
it's
not
gonna
handle
scale
and
orientation,
but
this,
but
what
they
can
do
is
so
cool.
I
Let's
explore
it
for
a
while
and
and
so
often
these
all
these
cases,
where,
like
just
pointing
out,
there's
like
a
flaw
in
it
afterward
it's
like
we
knew
about
the
flaw
from
day
one,
but
we
we
decided
to
just
run
with
it
for
a
while
and
see
if
the
flaws
would
work
themselves
out
and
that's
a
good.
That's
the.
A
Case
here
that's
a
good
point,
marcus
and
in
fact,
in
our
papers
we
often
call
them
out.
We
called
out
the
fact
that
we
didn't
understand
orientation
and
we
called
out
some
of
these
things,
but
you're
right,
I
mean
most
of
these
things
we
kind
of
knew,
but
it
was
it
was
like,
but
we
didn't
know-
and
we
just
said-
but
these
are-
these
ideas
are
so
important
anyway,
so
they
like
take
displacement
cells.
I
agree
it
was
a
really
cool
idea.
A
It
is
a
cool
idea
and
it
is
like
the
first
attempt
even
solving
a
problem,
a
fundamental
problem
that
most
people
don't
even
think
about,
and-
and
so
it's
worth
putting
it
out
there,
because
it
shows
that
what
the
problem
is.
It
shows
the
nature
of
the
problem.
It
shows
you
might
be
able
to
solve
it.
Neurons
could
do
these
things
yeah,
and-
and
so
yes,
that's
why
we
did
it.
It's
not
like.
Oh
it's
a
bad
idea.
No,
it's
a
great
idea.
It's
just
okay!
A
B
I
don't
know
if
this
is
important
for
people
to
understand
or
not,
but
I
don't
my
guess
is
most
people
here,
don't
understand
how
multiple
grid
cell
modules
represent
locations
of
sensors
because
in
fact,
most
neuroscientists
that
I've
spoken
to
even
those
who
study
grid
cell
modules.
Often
don't
understand
this
concept.
So
it's,
but
we.
A
It
it's
it's
important
only
in
the
sense
that
it
exposes
some
very
interesting
theoretical
concepts
and
it's
and
and
grid
cells
work.
On
the
I
mean
displacement
cells
also
work
on
those
very
interesting
theoretical
concepts.
So
I
touched
on
one
of
them.
The
idea
that
like
locations
can
be
unique
per
object.
You
know
what
I'm
saying
like
like
we
think
about
like.
If
I
wouldn't
design
a
cad
system,
I
would
use
a
cartesian
coordinate
system
and
I
would
say:
okay,
everybody's
going
to
have
the
same
cartesian
coordinate
system.
A
A
B
G
I
have
a
question
about
number
three
here:
this
mechanism
for
learning
object:
behaviors
yeah.
I
guess
this
ties
into
my
like
original
thesis
statement,
the
other
day
of
like
not
doing
vision,
but
this.
This
is
like
a
very
visual
example
and
it
makes
sense
with
the
stapler,
where
you
have
something
that's
physically
moving
and
you
can
represent
that
as
displacements.
A
Yes,
yes,
yeah
it
changing
again,
let's
take,
for
example,
using
a
smartphone
right.
The
morphology
of
the
smartphone
does
not
change
as
you
use
a
smartphone,
so
the
location
of
things,
don't
change
really
what
mostly
changes
what's
at
different
locations
on
the
smartphone.
So
what's
on
your
screen,
you
know,
oh,
I
can
have
an
icon.
Icon
could
move
but
more
likely
the
icon
just
changes
or
something
else
pops
up,
and
so
you
have
a
model
if
you
think
about
a
model
of
a
set
of
objects
relative
to
each
other.
A
A
A
But
we
need
to
do
both,
I
mean
you
know.
Just
if
you
solve
one
and
not
the
other,
then
you
haven't
solved
the
problem.
I
always
felt
that
the
that
the
physical
movement
like
the
staple
one
was
harder
to
me.
It
was
hard
for
me
to
figure
out.
That's
why
I
wanted
to
work
on
that.
One
you
know,
there's
a.
How
are
neurons
would
represent
that
kind
of
thing.
It's
really
hard
to
imagine
on
their
own
could
do
that
where
you
can
easily
it's.
A
You
might
come
up
with
simpler
explanations
for
like
oh,
how
does
the
color
of
something
change?
Maybe
I'm
wrong
about
that,
but
that's
how
it
felt
to
me,
but
you're
right.
It's
both
of
those
all
right.
What,
if
you
think,
if
you
think
of
the
world
as
a
compositional
structure,
it's
what
are
the
components
and
what
are
the
relationships
to
each
other
and
a
single
object
like
a
stapler
or
a
smartphone,
can
change
the
components
and
change
the
relationship
to
each
other.
A
Yet
we
still
think
it's
the
same
thing
and
we
don't
think
the
smartphone
is
a
new
smartphone,
because
I
have
a
different
app
on
this
display
and
I
don't
think
this
is
a
new
physical
object
on
my
desk,
because
the
stapler
is
open.
So
then
we
have
to
that's
the
general.
That
is
the
general
problem
of
how
we
very
fundamental
problem
like
how
do
you
represent
these
different
things,
and
sometimes
there
is
a
trajectory
to
the
movement
right.
A
If
I
just
if
I
change
the
icon
on
my
display,
you
know
maybe
that
my
my
display
is,
I
cycling
through
pictures.
Well,
I
don't
have
to
do
anything.
It
just
happens,
but
but
often
there's
an
interaction
with
the
user
as
well.
You
know
you
touch
it
and
it
does
this
and
things
like
that.
So
this
is
this
that
all.
F
A
Well,
I
used
to
remove
the
word
movement.
I
know
very
specifically,
I'm
referring
to
movement
of
movement
of
your
body
or
your
sensor.
That's
the
movement
I'm
referring
to
when
I
like.
You
know
when
I
talk
about
movement,
I'm
I'm
talking
about
you
know,
like
your
eyes,
are
moving
your
fingers
moving.
A
This
is
a
property
of
the
model.
This
displacement
is
the
property
of
the
physical
world
and,
as
we
just
talked
about
it
could
be
a
color
change
or
an
icon
change.
It
doesn't
have
to
be
physical
movement
so
anyway,
one
is
a
property
of
a
model
of
something
and
the
other
is
like
your
physical
body
moving
through
space.
A
So
you
could
call
this
movement
if
you
want,
but
don't
confuse
it
with
the
other
kind
of
movement
to
represent
the
stapler
of
the
iphone
icon
on
your
displays,
you
have
to
have
a
model
of
that
object.
That
model
has
to
be
independent
of
your
current
position
with
it.
It's
just
it's
a
property
of
the
object
itself
and
has
nothing
to
do
with
how
your
body's
moving.
A
E
Yeah,
I'm
not
really
sure
how
to
phrase
this
question.
First,
I
guess
I
still
struggle
a
bit
with
understanding
how
how
the
object
behaviors
are
represented.
So
if
I
see
an
object
and
immediately
recognize
some
affordances,
for
instance,
is
this,
like
part
of
the
voting
signal
like
additional
to
the
object
id
or
is
this.
A
Oh
we're
still
talking
we're
still
talking
one
column
here,
conceptually
we're
talking
one
column
and
we
haven't
gone
to
voting
yet
here
that,
if
a
mod,
if
a
column
is
going
to
model
an
object,
it
has
to
model
its
behaviors
as
well.
This
is
not
something
that's
going
to
happen
elsewhere,
so
voting
is
not
part
of
this
yeah
voting
would
be
important,
maybe
for
learning
in
a
practical
system,
but
at
the
moment
we're
not
talking
about
that.
We're
just
talking
about
single
columns.
So
our
argument
that
all
these
properties
exist
in
a
single
column.
E
A
Well,
I
I
missed
the
last
part
of
your
question,
but
let
me
the
first
part
it
is
over
time.
Obviously
an
object
doesn't
exist
in
multiple
states.
At
the
same
time,
it's
not
outside
of
quantum
theory,
and
so
you
know
so
in
some
cases
like
like
the
stapler
opening
up,
there's
a
physical
movement.
That's
going
to
happen
at
some
rate,
you
know
things
like
that.
The
changing
of
the
icons
on
your
display
happens
instantaneously
perceptually.
E
A
First
of
all,
you
you'd
have
to
learn
this
right.
You
you'd
have
to
learn
it.
You
just
wouldn't
know
it
if
you've
never
seen
anything
like
this
at
all.
It's
the
first
time,
you've
seen
anything
you're
calling
me
just
seeing
something
for
the
first
time
and
you're
you're
exploring
it.
That's
your
finger
and
you're
touching
it
and
you're
moving
around
you're
going
to
learn
the
morphology
of
it.
You
wouldn't
know
that
it
can
open
until
it
opens
or
you
push
on
it
and
it
changes
or
something
like
that.
A
So
it's
not
like
you
just
instantly
know
this.
Maybe
this
stapler
doesn't
open.
Maybe
you
know
who
knows
I
mean
these
are
all
these
behaviors
have
to
be
learned
now.
Obviously,
if
you've
seen
similar
things,
we
could
talk
about
analogy
how
you
might
deduce
that
the
stapler
could
open
or
how
you
might
deduce
how
the
cell
phone
would
behave,
but
a
priori,
if
you
didn't
know
anything,
you
just
have
to
learn
it.
A
You'd
have
to
be
you'd,
have
to
experience
it
and
it
would
have
to
be
learned
and
then
it
would
become
part
of
the
model
of
the
object.
It's
just
a
very
conceptual
idea:
you
have
a
model
of
something
that
model
includes
its
morphology,
its
features,
how
it
behaves
over
time
if
it
has
behaviors
over
time.
A
lot
of
you
know
honestly
it
does,
and
and
and
these
are
just
point
examples-
the
stapler,
the
iphone
or
smartphone.
This
is
just
examples
of
the
kind
of
things
the
behaviors
might
manifest
themselves.
E
A
It's
it's
think
about
the
model
before,
like
the
one
we
had
from
the
previous
page,
which
was
just
like
it
features
at
locations
all
right
features
that
locations.
That
was
a
very
simple
model.
It
didn't
include
any
behaviors
like
that,
and
and
until
you
have
like
there's
this
feature
at
this
location.
This
feature
with
that
location.
This
sense
hp
sense
feature
at
this
location
so
on.
Well,
it's
a
predictive
model
in
the
sense
that
if
you
move
your
finger
to
a
new
location,
the
model
will
predict
what
you'll
feel
say.
Okay,.
A
Well,
now
we're
just
extending
the
model
to
say
like
well.
Maybe
if
I
push
on
this
location,
then
the
relative
displacement
of
the
components
will
change.
So
if
I
push
down
on
the
top
of
the
stapler,
it's
going
to
move
and
it's
going
to
spring
back.
So
it's
it's
not
like
the
model,
just
making
a
prediction.
It's
more
like,
if
you
were
to
execute
something
like
the
stapler,
it's
not
going
to
do
anything
on
its
own,
so
with
the
stapler,
the
model
would
say.
A
Well,
if
I
were
to
push
on
the
top,
it
would
go
down.
The
staple
would
come
out,
but
if
I
don't
imagine
pushing
down
on
the
stapler,
then
the
model
is
silent.
In
the
background
it
doesn't
say
anything:
it's
not
making
a
prediction.
It
only
makes
a
prediction
when
you
actually
think
about
moving
your
finger
and
pushing
down
on
the
top
it
doesn't.
You
know,
while
I'm
doing
that,
it's
not
predicting.
You
know
that
you
know
the
rubber
feet
are
gonna
not
slide
or
something.
I
don't
know
what
it
is.
A
You
know
it's
just
like
it's
a
momentary,
momentary
thing:
it's
the
model
is
sitting
there.
Silent
in
the
background,
it's
stored
knowledge
and
as
your
as
your
sensor
moves
through
the
world.
Your
finger,
for
example,
moves
to
the
world
and
and
interacts
with.
It
basically
invokes
the
different
parts
of
the
model
and
the
model
says:
oh,
if
you
press
here
this
will
happen.
You'll
feel
this,
and
if
you
press
here,
you'll
feel
something
else.
E
A
A
This
is
not.
This
is
a
a
flow
diver.
Okay,
it's
not
like
colorful
column.
This
is
not
like
layers
of
cells
or
anything
like
that.
This
is
a
miracle
flow
diagram,
and
I
want
to
sort
of
point
out
where
we
are
currently
working
on
these
issues.
What
are
we
up
to
right?
How
are
we
trying
to
solve
these
issues
that
I've
just
identified?
A
So,
let's
just
talk
about
the
model
of
the
object.
Marcus's
has
sort
of
proposed
a
new
way
of
thinking
about
modeling
options
and
part
of
it's
new
and
part
of
it's
not
new.
So
his
and
marcus
correct
me
by
getting
this
wrong.
Okay,
because
I'm
doing
this
from
memory,
not
from
recent
experience,
so
he's
saying,
look,
let's
just
go
all
out
a
model
is
a
set
of
displacements
of
sub-objects
and
he's
now
referring
to
it
as
a
graph.
Wait:
okay,
it's
a
graph,
but
it
is
essentially.
A
A
That's
a
pretty
big
change
in
how
to
think
about
modeling
is.
Is
he
introduced
these
vector
cells,
which
there
are
many
of
them,
and
we
had?
We
totally
ignored
them
in
the
past,
because
we
didn't
know
what
to
make
of
them,
and
so
now
that
we've
run
the
limit
of
grid
cells
and
visuals
don't
seem
to
work.
A
I
No,
that's
all
accurate,
and
I
do
think
you
explained
kind
of
you
say
you
said
earlier.
We
as
theorists.
We
choose
like
the
extreme
thing,
where
reality
might
be
a
little
more
complicated
at
some
point
in
the
future.
We
might
bring
back
some
notion
of
maybe
location,
but
right
now.
The
extreme
version
of
this
is
the
let's
throw
out
the
locations
and
talk
completely
about
displacement.
Great.
A
I
am
happy.
I
got
that
right.
Okay,
there's
another
idea
that
that
it's
kind
of
been
we've
been
dealing
with
for
quite
a
while,
but
I
think
marcus
made
it
more
explicit
here,
which
is
great,
and
this
is
this-
have
to
get
back
with
the
idea.
I
told
you
remember,
none
of
our
models
worked
with
vision.
A
We
had
no
explanation
how
an
eye
which
is
not
actually
touching
the
coffee
cup.
You
know
the
fingers
at
the
coughing
up,
but
the
eye
is
not
the
coffee
cup.
So
how
do
we
handle
that
right?
So
we've
introduced,
or
you
know
the
idea
that
there's
a
pose
of
the
sensor
the
sensor
is,
is-
is
at
some
location
in
orientation
relative
to
the
object,
that's
being
sensed,
it's
not
at
the
location
on
the
object,
we're
not
even
thinking
about
that.
It's
at
some!
It's
it's
it's
at
some.
A
You
know
distance
or
location
and
orientation
and
and
there's
still
a
role
for
grid
cells
grid
cells.
Again,
I
think
I'm
trying
to
understand
what
markets
have
proposed.
I
think
this
is
it
is
that
grid
cells
are
still
used
for
path.
Integration
of
the
location
of
the
sensor,
but
it's
more
like
a
temporary
scratch
pad.
It's
not
like
the
basic
fundamental
representation
scheme
in
the
column.
It's
like.
A
Oh,
we
have
a
grid
cell
because
when
I
move
I
have
to
know
where
my
finger
is
going
to
be
so,
let's
figure
that
out
using
grid
cells,
but
it's
and
but
grid
cells
aren't
this
permanent
reference
frame
of
high
fidelity.
They
don't
have
to
be
high
fidelity.
They
just
have
to
work
good
enough
for
these
path.
Integration
at
the
moment
and
and
then
we'll
use
them
again
later
and-
and
this
gives
us
the
ability
to
sort
of
get
rid
of
the
idea.
A
You
need
to
have
lots
of
grid
cells
and
and
a
bunch
of
other
issues
associated
with
them,
but
they're
still
there
and
we're
still
going
to
find
them
in
every
cortical
column,
but
they're
playing
a
different
role,
they're
playing
it's
more
of
a
temporary
path,
integration
role,
it's
not
fundamental
to
the
representation
of
the
object
model
itself.
You'll
see
no
grid
cell
representations
in
the
model,
as
we
did
in
our
in
our
columns
pick
where
the
location
and
feature
were
part
of
the
model.
A
Here,
the
location
is
not
part
of
the
model,
it's
only
the
relative
positions
of
the
different
components,
but
we
still
need
to
track
the
sensor.
It's
movement,
it's
posed
to
the
object
as
your
as
your
fingers
moving
through
space
or
your
eyes
are
moving
around
and
things
like
that
and
there
may
not.
We
don't
really
understand
this.
Yet
in
marcus
has
figured
it
out.
There
may
not
really
be
an
object-wide
reference
frame
anymore.
We
don't
really
understand
this.
A
It
could
be
very
temporary
and
and
used
moment-to-moment,
and
I
was
encouraged
by
this
idea,
because
that
was
that
paper
we
saw
the
rats
of
the
bats
flying
in
the
long
tunnel
and
the
grid
cells
remap
their
size
and
scale
changed
throughout
the
the
traversing
of
the
of
the
of
the
tunnel,
and
it
was
like
wow.
They
can't
really
been
representing
a
roughage
frame
for
the
tunnel.
A
D
I
I
do
have
one
question,
so
we
know
grid
cells,
I
mean
exist
in
around
context.
They're
they're,
at
least
in
some
sense,
there's
a
morphology
where
you
can
place
them
when
you
said
that
we
didn't
know
what
to
do
with
vector
cells.
Do
vector
cells
exist
as
a
concept,
or
is
this
a
imputed?
They
must
be
vector
cells.
A
A
That's
the
same
techniques.
They
find
anything
else,
they
find
cells,
they
record
from
them.
They
see
what
makes
the
cells
respond
and
they
test
that
and
come
up
with
theories
and
test
it
again.
We
could
go
through
that.
It
would
be
maybe
a
useful
exercise
sometime
to
review
the
literature
on
these
things.
We
have
done
a
paper
review
on
this,
so
I
think
we
do
have
a
couple
of
review
papers
about
vector
cell
logic,
vector
cells
and
other
types
of
vector
cells.
A
A
C
A
Okay,
okay,
they're
kind
of
in
the
way
to
think
about
this.
The
way
I
kind
of
think
of
it,
maybe
it's
not
the
right
way,
but
the
way
I
think
about
it
between
the
hippocampus
and
the
subiculum
and
the
antarctic
cortex,
and
then
there's
medial
antimicrobials,
these
on
tonight,
projects,
there's
all
these
different
subsections
you
can,
you
can
get
pretty
far
for
just
ignoring
where
the
actual
cells
are.
It's
not
that
helpful,
all
the
time,
they're
all
kind
of
working
together,
they're
highly
interconnected,
and
so
you
can,
you
know
often
I
just
say
yeah.
A
A
D
A
A
A
We
haven't
published
anything
on
this
yet,
but
I
think
that
would
be
a
logical
thing
in
a
paper
about
this
to
say
yes
well
internally,
but
say
yes,
we
would
agree
with.
According
you
know,
the
latest
modeling
that
marcus
has
come
up
with
we'd
expect
expect
to
see
various
flavors
of
vector
cells
in
the
cortex.
A
A
A
A
A
It
could
be
just
one
original
module.
That
would
be
the
evidence.
We
have
that
a
column
has
one
grid
cell
module
and
that
grid
cell
module
would
be
re-anchored
under.
I
would
assume
so
because
that's
the
way
they
work
in
the
antorano
cortex
and
I'm
assuming
they're
going
to
work
that
way
in
the
cortex
too,
so
they
would
be
re-anchored.
I
I'll
jump
in
because
like
here,
though,
I
think
the
word
scratch
pad
might
actually
be
conveying
a
wrong.
Well,
just
sometimes
we
use
that
word,
but
here
this
this
is
like
one
of
the
questions
of
good
cells
is
that
it
could
have
been
a
scratch
pad.
It
could
have
been
the
case
that
if
an
animal
comes
into
a
room,
it
doesn't
matter
which
grid
cells
are
active.
It
just
it's
this
path,
integrating
system.
Next
time
it
comes
into
that
room,
maybe
a
different
grid,
cell's
active
for
a
location
that
would
be
a.
I
Be
like
that
here
there
there
is
some
memory
component
that
there
is
like
here.
The
the
same
grid
cell
is
going
to
become
active
when
the
rat
comes
back
into
a
room,
which
is
an
interesting
vital
fact
like
it
could
have
been
different
from
that.
So
so
that
is
an
important
detail.
So
I
think
of
it
more
as
the
grid
cells
are
providing
additional
spatial
information
that
was
somehow
not
captured
by
the
vector
cells.
A
A
I
haven't
gone
to
the
third
box
here,
yet
any
questions
still
on
the
first
two
again.
Let
me
just
review
again
before
I
give
you
a
chance.
Big
idea
is
that
we
now
are
going
to
model
objects
completely
as
displacements
we've.
Just
we've
accepted
that
completely
abandoned
the
idea
of
sensor
features
at
locations.
A
The
spicers
are
now
based
on
vector,
polar
coordinates
versus
cartesian
coordinates.
That's
that's
a
really
nice
idea,
and,
and
now
we've
we've
sort
of
introduced
the
idea
that
a
sensor
is
not
at
the
location.
It
doesn't
have
to
be
at
a
location
on
the
object
to
send
something.
It's
it's
at
some
pose
to
the
object,
and
we
will
now
have
to
do
various
kind
of
calculations
to
determine
the
displacements
of
two
sub-objects.
A
When
my
sensor
is
at
a
distance,
so
my
eye
is
two
feet
away
from
the
things
I'm
looking
at.
I
have
to
be
able
to
calculate
the
displacement
between
those
two
things
as
I
attend
to
them
one
after
another,
and
I
have
to
incorporate
the
pose
of
the
eye
relative
to
the
object,
I'm
viewing
for
the
object
I'm
giving.
So
those
are
the
big
ideas
and
then-
and,
as
I
said,
you
know,
grid
cells
are
still
there,
but
they're
not
playing
the
same
role
as
before.