►
From YouTube: Jeff Hawkins on Minicolumns - September 23, 2020
Description
Jeff Hawkins brainstorms some ideas on minicolumns, in a continuation of a recent concept he presented in Numenta’s July 27, 2020 research meeting. In the previous meeting, he hypothesized that minicolumns represent movement vectors. In this research meeting, Jeff suggests a new mechanism for calculating reference frame transformation that ties into his minicolumn hypothesis. He suggests that the movement vectors of the minicolumns’ upper layers are allocentric, while that of the lower layers are ego-centric.
July 27, 2020 Research Meeting: https://www.youtube.com/watch?v=yceJeKf-ad4
- - - - -
Numenta is leading the new era of machine intelligence. Our deep experience in theoretical neuroscience research has led to tremendous discoveries on how the brain works. We have developed a cohesive theory, core software technology, and numerous software applications all based on principles of the neocortex. Our innovative work delivers breakthrough capabilities and demonstrates that a computing approach based on biological learning principles can do things that today’s programmed computers cannot do.
Subscribe to our News Digest for the latest news about neuroscience and artificial intelligence:
https://tinyurl.com/NumentaNewsDigest
Subscribe to our Newsletter for the latest Numenta updates:
https://tinyurl.com/NumentaNewsletter
Our Social Media:
https://twitter.com/Numenta
https://www.facebook.com/OfficialNumenta
https://www.linkedin.com/company/numenta
Our Open Source Resources:
https://github.com/numenta
https://discourse.numenta.org/
Our Website:
https://numenta.com/
B
So
this
is
a
very
from
a
very
high
level
perspective.
We
have
these
two
sort
of
neuroscience
goals.
It's
and
one
is
understanding
the
functions
of
cortical
column,
and
that
means
mapping
it
that
functionality
under
cortical
anatomy
and
physiology,
and
the
second
is
to
understand
how
columns
work
together
with
other
parts
of
the
brain.
B
So
I'm
just
focusing
on
a
single
column
again
today
and
I've
been
working
on
various
aspects
of
this
over
the
last
few
months,
and
I'm
just
going
to
tell
you
about
one
idea
I
have
been
pursuing,
which
hasn't
been
fruitful
yet,
but
I
want
to
share
with
you
anyway,
so
just
just
again
and
we'll
start
with
some
basic
review.
It's
always
good
to
do
that.
If
we
think
about
a
column
that
generally
or
the
near
cortex
as
a
whole,
the
generally
accepted
view
is
that
you
have
some
sort
of
sensory
input.
B
There's
also
an
idea
that
there's
a
motor
input,
which
is
typically
referred
to
as
a
reference
copy,
that's
coming
from
someplace
else
in
the
brain
and
the
cpg
is
a
central
pattern,
generator
that
would
be
a
subcortical
structure
and
then
the
cortical
cortex
also
has
a
motor
output
which
goes
projects
back
to
those
cpgs
and
controls
them.
I
think
most
neuroscientists
would
agree
with
this
basic
description,
although
they
don't
think
about
it
too
much
a
lot
of
them.
What
we've
done
in
the
thousand
brain
series
is
every
individual
column.
Does
this?
B
It's
not
just
the
near
cortex
as
a
whole,
but
every
individual
column
learns
models
through
sensing
and
moving
and
sensing
and
moving.
This
is
the
basic
layout
for
how
things
in
the
world
are
learned.
We
move,
we
sense,
we
move
and
sense.
We
build
a
model
of
the
thing
that
way
and
then
that
model
lets
us
generate
behaviors
to
achieve
certain
goals.
B
Recently,
I
propose
an
alternate
view
to
this.
Oh,
I'm
sorry
before
I
go
on
so
this
basic
idea,
if
you're
gonna,
if
this
is
true
which
we're
pretty
certain
it
is
then
basically
a
sense
that
every
column
has
some
sort
of
has
to
have
some
sort
of
reference
frame
and
path,
integration,
there's
a
motor
input.
B
Again.
These
are
ideas
that
are
not
really
controversial,
but
most
people
don't
think
about
them
occurring
in
every
column
and
most
people
don't
think
about
them
too
much
but
they're,
not
in
general,
not
too
controversial.
Recently.
I
I
presented
this
slide
of
a
month
or
two
ago.
I
don't
maybe
six
months
ago,
I
can't
remember
where
I
said
this.
I
proposed
an
alternate
way
of
thinking
about
the
motor
inputs
or
column,
and
that
alternate
way
looks
more
like
this.
B
There
is
in
vision
we
would
think
of
it
as
the
part
of
a
cellular
and
the
magnetocellular
inputs
and
one
represents
flow
or
in
because
in
this
case
would
be
optic
flow
and
the
other
rep
is
more
of
a
static
input,
and
that
actually-
and
I
made
the
argument
that
this
is
the
main
motor
input
that
a
column
uses
to
learn
it.
It
doesn't
start
with
the
with
the
with
the
reference
copy
from
from
the
cpg,
but
it
starts.
B
Basically
it
infers
movements
and
infers
and
infers
movements
by
looking
at
a
large
scale
flow
of
input
bits.
I
won't
I'm
not
going
to
go
for
all
those
slides,
but
I
presented
this
idea
recently.
I
don't
know
if
everyone
was
there.
I
assume,
maybe
maybe
nils
you
weren't
there.
For
that.
I
don't
know
but-
and
I
think
there's
a
lot-
there's
a
lot
of
evidence
supporting
this,
and
I
think
it's
actually
right
and
I
think,
even
though
I
talked
about
here
in
terms
of
vision,
I
think
it
occurs
it.
B
It
applies
to
anything
you're
modeling.
You
have
a
sensory
organ
or
you
have
some
bits
of
information
that
are
coming
from
some
place.
It
could
even
be
another
cortical
region
and
and
by
observing
how
those
those
bits
flow
you
can
determine
what
are
the
basic
movement
vectors
that
are
being
operated
on
in
that
case.
B
In
this
case,
then,
there's
still
a
motor
reference
copy
and
there's
still
a
motor
output
as
before.
These
are
documented
features.
So
in
vision,
you
you
get.
The
the
visual
cortex
gets
input
from
this
protocoliculus
indicating
how
it's
moving
the
eyes
and
and
the
cortex
can
project
back
to
the
speculars
in
this
particular
example,
but
I
I
I
put
those
as
secondary.
That
is
the
way
the
column
learns.
B
The
model
that's
objects
is
not
by
first
observing
those
things,
the
efforts
copy
it
observes
by
observing
the
sensory
data
itself,
and
then
it
can
then
learn
to
associate
its
model.
It's
it's
its
representations
of
movement
with
projections
from
the
from
the
subcortical
projections,
and
it
can
learn
to
do
this.
No,
it
can
learn
to
basically
anticipate
movement
from
reference
copies
where,
but
I'm.
C
B
B
I
can
just
take
the
visual
input
and
learn
that
same
model,
the
world
of
darwin,
and
that
told
me
that
there
was
sufficient
input
in
the
in
the
visual
stream
to
to
do
that
once
up
one
second
here
guys,
I'm
just
gonna,
I'm
gonna
ask
my
wife
to
be
a
little
quieter.
There's
my
I
can't
get
my
one
second,
I'm
trying
to
find
I'm
trying
to
find
my
mute
button.
Well,
I
can't
find
it
hey
janet.
B
D
Okay,
so
so
you
have
a
static
input,
so
I'm
just
wondering
when
you
say
the
sense
flow
input.
Is
that
just
is
it
literally
something
that's
processed
the
thing
and
says:
here's
what's
different
about
this
thing,
and
this
is
the
flow
or
is
it
something
says,
here's
something
different
and
inside
that
your
rectangle,
it
senses
it.
It
generates
a
flow
because
you
have
a
reference
versus
something.
That's
done.
B
B
They
represent
change,
they
will
not
represent
a
static
input
and
and
what
I'm
arguing
that
they're
going
to
argue
in
a
moment
that
those
are
those
are
ideal
for
for
detecting
flow
and-
and
I
presented,
I
never
should
have
done
the
whole
presentation
again,
but
I
presented
earlier
papers
that
showed
that
complex
cells
and
visual
cortex
really
are
not
detecting
movements
of
edges,
they're,
just
flow
patterns,
they're
just
optic
flow,
and
you
can
give
them
any.
They
actually
respond.
B
Better
to
random
bit
patterns
than
they
do
to
structured
patterns,
they
just
they
they're,
really
just
determining
flow.
So
on
its
own,
you
don't
need
anything
else.
You
just
look
this.
This
magnificent
pathway
says
hey,
I'm
I
can
detect
when
there's
large
flow
patterns
over
the
visual
field
and
the
power
cellular
cells
are
the
opposite
of
that.
They
they
do
respond.
Statically
they're
slower
responding,
so
they
wouldn't
be
triggered
very
easily
if
something
was
moving
and
they
are
much
smaller,
it's
up
to
field
sizes.
B
It's
it's
coming
right
from
the
retina,
the
retina
doesn't
encode
flow,
but
it
encodes
these
two
different
types
of
inputs,
so
the
evidence
we
have
right
now
is
that
the
actual
axons
that
are
representing
the
magnitude
do
not
represent
flow.
They
just
represent
very
fast,
changing
bits
that
are
that
represent
change,
and
but
when
you
get
to
the
cortex,
there
are
multiple
types
of
cells.
We'll
talk
about
a
second
there's,
these
complex
cells
and
simple
cells
and
there's
two
different
types
of
complex
cells
and
they
seem
to
represent
flow.
B
B
This
is
not
how
people
think
about
it,
they
think
about.
You
know
everything
in
the
cortex
is
the
visual
cortex
is
all
about.
You
know
orientations
and
things
like
that,
but
actually
that's
not
the
evidence.
The
evidence
is
that
the
complex,
also
not
like
that.
It's
just
people
forgotten
it's
old
data
anyway.
So
that's
this
idea.
I
I
think,
there's
I'm
really
this
all
the
more.
I
think
about,
I'm
very
convinced
that
this
is
actually
happening,
even
though
it's
speculative
and
it's
a
hypothesis.
B
B
One
of
the
things
is
that
if
you
think
about
it,
this
way
the
these
inputs
coming
in
this
case
again,
I'm
going
to
also
focus
on
vision,
but
I
don't
think
it
has
anything
to
do
with
vision.
You've
got
two
different
reference
frames
going
on
here.
If
I,
if
this
is
a
primary
sensory
region,
you'd
have
an
egocentric
reference
frame,
which
is
where
your
data
is
coming
from.
B
So
if
I'm
detecting
input
on
the
retina
and
I'm
detecting
flow
due
to
my
eye
movements
or
due
to
the
object,
moving
well
at
least
due
to
my
eye
movements,
then
that's
an
egocentric.
If
I
move
my
eye
to
the
left,
then
that
movement
is
an
egocentric
movement,
it's
not
in
the
object
space
and
similarly
movement,
the
motor
inputs
and
outputs
of
the
cortex
are
going
to
be
an
egocentric
coordinates,
but
we
want
to
build
a
model
of
something
not
an
egocentric
coordinates.
B
We
want
to
build
it
in
all
essential
coordinates,
and
so
there
has
to
be
a
a
reference
frame.
Transformation
going
on
and
and
other
people
have
written
about
this-
and
it's
become
more
and
more
clear
to
me
that
this
is
this
is
happening
all
the
time
in
every
cortical
column
that
there's
a
reference
frame
transformation
going
on,
even
if
I
just
consider
like,
oh
you
know,
what's
the
if
I'm
moving
my
eyes,
my
eyes
are
moving
relative
to
my
head,
but
then
you
know
I
move
my
head.
B
It's
all
to
my
body
and
my
move,
my
fingers
rolled
to
my
hand
and
so
on.
There's
this
this
constant,
you
just
every
time
a
the
cortex,
a
column
of
the
cortex
model,
something
it
has
to
take
a
set
of
inputs
in
one
space
and
convert
it
into
a
set
of
a
set
of
coordinates
or
a
set
of
properties,
one
space
converting
the
settlement
on
the
space.
So
I've
now
come
to
believe
this
is
a
fundamental
component.
B
What's
going
on
in
every
quarter
column,
we
haven't
always
talked
about
it
that
way,
but
you
know
we
knew
this
had
to
happen,
but
I've
I've
come
to
realize
that
this
is
like
a
key
thing
that
is
going
on
everywhere.
So
we've
had
a
here's,
a
list
of
things
that
we've
talked
about
multiple
times
in
the
past,
that
that
we
know
cortical
columns
must
do
and
I'll
just
just
go
through
them
very
quickly.
B
Here
I
just
talked
about
the
bi-directional
reference
frame
transforms
by
the
way
it
has
to
be
bi-directional,
because
I
need
to
go
from
the
model
to
make
a
prediction
and
go
from
the
model.
If
I
want
to
make
a
movement
relative
some
object,
I
have
to
then
convert
that
into
a
movement
relative
to
my
body,
and
so
it
just
go
both
ways.
B
B
We
also
know
there
have
to
be
displacement
cell
modules
and
those
would
also
be
1d
just
to
review.
We
haven't
talked
about
displacement
cells.
A
lot
here,
although
it
was
in
our
2019
paper,
you
need
displacement
cells
for
two
things.
You
need
it
to
calculate
the
movement
to
get
from
one
location
to
another
location
and
you
need
to
learn
compositional
structure
of
objects.
So
if
I
have
two
points
on
two
different
objects,
then
the
displacement
tells
tells
me
how
to
go
between
those.
B
B
So
the
way
again,
this
will
be
new
for
some
of
you,
but
when
you
walk
around
you're
constantly
attending
the
objects
constantly
constantly
constantly
just
several
times
a
second
and
you
do
that
vision
and
touch
and
hearing
and
so
on,
and
what
we
believe
is
going
on
each
time.
You
turn
to
an
object.
You
you're
actually
calculating
its
displacement
from
a
larger
parent
object.
So
if
I
walk
in
into
a
room-
and
I
see
a
table-
I
quickly
say
oh,
where
I
quickly
calculate
just
look
at
the
table.
B
My
brain
calculates
where
the
table
is
relative
to
the
room,
but
I
then
look
at
a
dish
on
the
table.
It'll
calculate
where
the
dish
is
relative
to
the
table.
If
I
calculate
if
I
look
on
a
pattern
on
the
dish,
it'll
calculate
where
the
pattern
is
relative
to
the
dish,
this
happens
all
the
time
you
just-
and
this
is
how
the
model
of
the
world
is
built-
you're
just
constantly
just
building
this
other
hierarchical
structure.
B
As
you
look
at
as
you
go
around
and
tend
to
things
it's
always,
it's
always
calculating
its
position
all
to
something
else,
and
what
we
have
to
do
is
remember
those
displacement
cells
and
that's
your
model
of
the
things
you're
looking
at.
I
know
that's
a
lot.
If
you
haven't
been
exposed
to
this,
you
have
no
idea
what
I'm
talking
about.
Probably
we
also
know
we
have
to
learn
the
sequences
of
displacement
vectors
because
some
objects
have
behaviors
and
that's
how
you
learn
behaviors
in
the
paper.
B
We
use
the
example
of
a
stapler
and
how
the
stapler
opens
and
closes,
and
that's
something
you've
learned.
How
do
you
know
how
to
what
does
it
mean
to
know
that
a
stapler
opening
closes?
It
means
that
you
have
to
learn
the
displacement
between
the
different
parts
of
the
stapler
and
how
those
displacements
change
over
time.
So
sequence
of
displacements
represents
this.
B
Basically,
the
segment
displacement
vectors
represents
a
behavior
of
an
object,
and
so
we
know
it
has
to
do
that,
and
we
also
know
that
the
column
has
to
maintain
a
state
of
the
object.
It
specifies
its
current
set
of
displacements.
So
again,
the
staple
could
be
open
or
closed.
My
cell
phone
have
one
app
on
it
or
another,
app
on
it.
It's
still
the
same
cell
phone.
B
I
have
the
same
reference
frame,
but
I
need
to
know
the
state
of
the
object
in
order
to
know
what
exactly
is
going
to
be
visible
or
what
its
position
and
morphology
would
be
at
any
point
in
time.
So
this
is
a
hell
of
a
lot
of
stuff
to
go
on
in
the
cortical
column
and
over
the
years
we've
talked
about
this
a
lot,
some
of
you.
This
is
a
lot
of
this,
should
be
new
to
you.
We've
talked
about
all
these
issues
and
how
they
might
get
resolved,
and
what
I'm
trying
to
do.
B
Basically
we're
trying
to
map
all
these
functions
onto
a
cortical
column.
So
what
I've
been
doing?
The
last
few
months
is
just
thinking
about
these
very
broadly
trying
to
gel
some
sort
of
scheme
by
which
we
can
make
all
this
happen.
I'm
just
going
to
talk
about
one
idea.
I've
been
working
on
this
last
couple
weeks
and
it's
frustrating
because
I'm
stuck,
but
I'm
really
stuck,
I'm
just
like
it's
puzzling
and
I
haven't
really
been
able
to
decide
whether
it's
a
great
idea
or
not
yet,
but
it's
exciting
to
think
about.
B
So
I'm
just
going
to
talk
about
that.
One
idea
and
then
I'll
be
done
for
today,
and
so
I've
been
asking
myself
how
this
bidirectional
reference
training
transform
could
occur.
We
haven't
spent
a
lot
of
time
in
that
in
the
past,
so
it's
like.
Okay,
that's
a
big
thing.
How
could
that
occur
and
the
more
you
think
about
it,
the
harder
it
is
to
imagine
it's
like?
How
does
you
know
how
does
a
single
column
take?
You
know,
coordinate
from
one
reference
frame
and
transfer
the
chord.
It's
another
reference
thing.
So
here's
the
idea.
B
I've
been
working
on
recall
that
this
is
a
slide.
I
showed
you
once
before.
This
is
just
showing
what
a
mini
column
is
a
minicom
is
a
set
of
cells.
Basically,
it
goes
across
all
layers
of
cortex
and
the
hypothesis
that
I
had
proposed
that
many
columns
represent
a
1d
movement
vector
and
the
complex
cells.
These
are
cells
that
respond
to
movement
in
some
direction,
represent
a
velocity
in
a
1d
movement
vector
it's
like
okay,
my
eye
is
moving
this
way
or
my
body's
moving
forward,
or
it's
going
at
some
angle.
B
B
However,
there's
a
problem
with
this
idea
and
if
a
minicom
contains
some
cells
that
are
egocentric
and
other
cells
that
are
allocentric
which,
by
definition
since
all
the
cells
are
minicoms,
that's
going
to
have
to
be
the
case,
because
we
know
that
the
inputs
of
this
column
are
going
to
be
in
ecocentric
form
and
yeah,
and-
and
we
know
this
is
going
to
build
an
algebraic
model.
Then
then,
then,
all
the
cells
in
the
mini
column
cannot
have
the
same
basic
orientation
preferences.
B
They
all
can't
represent
the
movement
in
the
same
direction,
because
you
know
how
I
move
on
the
objects
is
going
to
be
different.
How
you
know
if
I
move,
if
I'm
moving
straight
in
relative
or
returning
leftover
to
the
up
to
an
object,
they
might
in
a
different
dimension
depending
on
the
orientation
object,
so
it's
essentially
which
suggests
that
one
of
the
the
basic
principles,
I
think
people
think
about
mini
columns.
They
say.
B
Well
all
the
cells
in
the
minicom
in
b1,
for
example,
all
have
the
same
orientation
preference,
and
this
would
say
that
can't
be
true.
Now,
there's
a
whole
bunch
of
caveats
here.
First
of
all,
when
they
talk
about
that
they're
talking
about
less
than
50
of
the
cells
a
little
bit
less
than
50
percent
of
their
cells,
the
other
50
cells,
they
don't
know
what
they
do
at
all,
so
they're,
not
exposing
that.
But
one
of
the
one
of
those
older
papers
that
I
talked
about
before,
which
was
talking
about
the
optic.
B
The
flow
idea
pointed
out
that
they
found
sometimes
cells,
complex
cells
that
had
a
different
orientation
preference
than
let's
say,
layer,
four
cells
in
the
same
income.
It's
not
like
you
know
you
stick
the
probe
in
and
the
idea
is
that
all
these
cells
are
the
same
orientation
preference.
That's
not
true,
and
it
may
be
true
under
artificial
conditions,
but
they
really
found
some
evidence
under
some
conditions.
That's
not
true.
They
find
that
they're
completely
different,
so
this
would
be
an
important
sort
of
a
guess.
B
So
the
idea
I'm
working
on
right
now
is
this
idea
that
the
perhaps
the
upper
layers
are
allocentric
and
the
lower
layers
are
egocentric,
meaning
that
they're
they're
they're.
They
are
they're
converting
a
movement
from
one
into
another,
it's
sort
of
in
the
mini
columns,
and-
and
let
me
just
give
you
one
more
slide-
just
oh
just
to
remind
you
just
maybe
get
back
to
something-
let's
go
back
something
earl
I've
showed
earlier.
These
are
complex
cells
in
v1.
B
These
are
representing
their
cell
responses,
and
the
important
point
in
this
slide
is
that
the
cells
in
the
lower
layers
have
much
broader
receptive
fields.
In
general
than
the
cells
in
the
upper
layers,
these
cells
in
the
upper
layers
also
tend
to
be
much
more
end,
stop
meaning
they.
The
darker
they
are
here,
means
that
if
input
gets
even
wider
than
they're
shown,
that
gets
a
bigger
book.
You
know
wider
amount
of
the
retina
visual
field.
The
cells
stop
responding,
they
like
they
say.
B
I
only
want
to
respond
to
small
things
that
are
moving.
I
don't
want
to
spawn
the
big
things
that
are
moving,
whereas
the
lower
layers
tend
to
be
like.
We
want
to
respond
to
big
things
and
the
bigger
the
better,
and
so
they
so.
The
idea
that
I
proposed
before
and
I'm
working
on
again
now
is
that
these
these
lower
cells,
the
lower
layers,
essentially
imagine
they're,
representing
the
broadest
receptive
field
possible,
and
so,
if
I
detect
optic
flow
over
the
entire
visual
field,
that
tells
me
that
I'm
moving
relative
to
the
world.
B
B
So
it
says
I'm
looking
at
something
that's
on
a
much
smaller
part
of
the
receptive
field,
and
I
only
want
to
focus
on
that
now.
This
gets
back.
I
think
the
question
you
ask
kevin
early
kevin
is
like
well
one.
Under
what
conditions
would
these
things
agree
and
what
conditions
would
they
disagree?
There's
there
seems
to
be
a
possibility
of
calculating
something
here
I
can
say,
there's
something:
that's
moving.
That's
a
small
part
of
microsoft
field,
but
my
large
part
of
receptor
field
is
not
moving.
B
Then
you
know
what
can
I
calculate
based
on
the
how
much
I'm
moving
in
a
particular
direction
to
the
lower
cells
here
versus
how
much
the
thing
I'm
looking
at
is
changing
and
under
different
orientations.
If,
if
the
upper
idea
that
the
upper
cells
represented
some
sort
of
allocentric
movement
vector,
then
then
the
amount
that
you
know
how
much
th
there'd
be
a
varying
grade
of
differential
response
between
these
upper
and
lower
cells.
Perhaps
this
will
become
clear.
I
mean
what
did
I
say
here?
B
Let
me
see
here
first,
I
want
to
point
out
that
this
is
sort
of
a
ubiquitous
property
of
the
cortex.
These
large
and
small
rfs,
meaning
receptor
fields
in
the
upper
layers
are
small
and
the
lower
layers
are
large.
This
is
a
huge
clue.
This
is
not
there
for
nothing
this.
This
is
like
a
ubiquitous
property
of
the
cortex
there.
It
must
be
there
for
a
reason.
You
know
it's
not
it's
it
and
it's
an
important
thing.
B
It's
a
huge
feature
of
the
cortex
and
now
I
already
said
what
I
said
a
moment
ago
and
stop
smaller.
After
represents
movement
of
a
subset
of
the
field
of
view,
and
the
question
is:
can
we
can
the
differential
firing
of
the
of
large
and
small
rf
complex
fields
be
usage,
perform
reference
frame,
transform?
B
That
is
the
idea.
I'm
working
on
here's
one
more
slide
to
talk
about
it.
This
is
sort
of
the
the
standard
view
of
how
a
chord
v,
one
quarter
of
a
column,
looks
where
in
one
dimension,
so
we're
looking
at
a
slice
of
the
cortex
here,
little
cortical
column.
In
one
dimension,
you
see
a
change
of
orientation,
that's
in
one
dimension,
but
if
you
go
in
the
other
dimension
of
thalium
would
mention
we
have
what
they
call
iso-orientation
slabs
all
the
cells.
B
There
seem
to
respond
to
the
same
basic
orientation,
and
so
we've
talked
about
this
quite
a
bit.
I've
even
proposed
how
this
could
be
play,
a
role
in
in
in
learning
voltage,
controlled
oscillators
for
grid
cells,
but
this
is
a
pretty
standard
feature.
We
see
what
I'm
exploring
the
idea
now.
Is
this
different
thing?
B
It's
the
idea
that
the
that
the
that
the
lower
layers
have
eagle
movement
vectors-
that
is,
these
layers,
five
and
layer,
six
cells
that
respond
to
these
broad
areas
and
the
upper
layers
are
somehow
or
aloe
movement
vectors
and
we're
using
this
crossbar
sort
of
orientation
just
to
figure
out
how
to
calculate
how
to
transform
from
one
to
the
other.
A
few
observations
we'll
make
about
this
because
we
see
these
slabs
and
they're
very
prevalent.
There
must
be
mechanisms
that
make
it
happen.
B
This
is
not
going
to
happen
on
its
own,
so
there
has
to
be.
There
has
to
be
a
cellular
mechanism
to
make
these
things
behave.
This
way,
you
know
whether
it's
an
inhibitory
cell
or
not.
We
don't
really
know
yet,
but
it's
not
something
that's
going
to
happen
on
its
own,
and
so
that's
an
interesting
question,
because
if
we
can
figure
out
what
that
mechanism
is,
I
can
ask.
Does
that
mechanism
apply
equally
to
upper
and
lower
layers?
Do
we
see
a
difference
in
how
that
works?
B
It's
likely
that
slabs
play
a
functional
role.
I've
already
said
that
it's
it's
not
like
they're
they're,
you
know
they're.
The
brain
wouldn't
be
organized
this
way.
If
they
didn't
do
something
and
then
the
question
I'm
asking,
could
there
be
orthogonal
slabs
in
upper
lower
layers,
which
is
a
very
speculative
idea?
No
one's
proposing.
That
generally,
I
think,
but,
as
I
said,
there's
evidence
that
that
there
are
differences,
that
the
lower
layers
do
respond
to
different
orientations,
the
upper
layers
on
the
most
experimental
paradigms.
B
You
wouldn't
notice
that
perhaps
so,
if
you
just
you,
know
anesthetize
animal
and
you
expose
it
to
some-
you
know
sinusoidal
gradient.
You
might
not
see
that
it
requires.
You
know
actively
attempting
to
object
to
different
orientations,
so
I
said
there
was
evidence
for
that,
and
so
the
question
is:
could
this?
Could
there
be
this?
Be
macula
calculating
the
reference
frame
transformation.
B
Remember
the
paper
I
reviewed
a
few
weeks
ago
or
a
month
ago,
if
I
forget
who
the
author
is,
I
think
it
was
a
bird
just.
I
can't
remember
they
they.
It
was
a
review
paper
about
the
the
vector
cells
and
the
different
types
of
vector
cells.
You
know,
object,
vector,
boundary,
vector
cells
and
so
on,
and
they
had
this
proposed
sort
of.
I
don't
have
the
picture
with
me
right
now.
B
They
had
this
proposed
way
that
you
could
convert
between
reference
frames
of
like
allocentric
reference
cells
and
egocentric
reference
cells,
and
they
argue
that
both
types
vector
cells
and
they
argue
that
both
type
effector
cells
exist.
They
give
evidence
for
that
which
says
these
egocentric
vector
cells
and
these
allocentric
vector
cells,
and
they
propose
a
sort
of
almost
like
a
crossbar
proposal.
B
It
looks
very
much
like
I'm
showing
here
on
the
right,
where,
as
a
mechanism
for
how
you
could
transfer
between
allocentric
and
egocentric
reference
frames,
and
so
I
was
inspired
by
that-
that
figure
and
and
said
hey,
you
know,
could
that
be
happening
in
the
cortex
here
as
well.
So
I
didn't
totally
make
this
up.
B
I
don't
know
how
to
do
that
today
so
anyway.
This
is
the
this
is
the
idea
I'm
working
on.
I
would
say
it's
highly
speculative
at
this
point
in
time,
but
I
think
there
must
be
a
very,
very
basic
mechanism
for
doing
reference.
Training
transforms
that
ties
into
the
whole
mini
column,
hypothesis
that
many
columns
represent
movement
vectors
because
I
think
that's
true,
and
so
there's
got
to
be
an
answer.
Looking
here.
That
looks
something
like
this.
B
B
That's
the
basic
of
the
basis
of
the
model.
I
think
that's
like
that's
correct,
I'm
not
sure,
but
I
think
that's
correct
so,
but
we
don't
have
to
necessarily
have
grid
cells
in
the
in
the
in
the
ego
sensor
reference
chain,
so
the
thing
I'm
thinking
about
right
now
is:
can
I
convert
a
movement
vector
on
its
own?
Just
like
hey,
I'm
moving
this
amount
in
this
direction
relative
to
my
body?
B
F
E
B
A
B
Of
the
heart
of
every
sort
of
column
it
has
to
be
during
these
transforms
back
and
forth,
and
there's
a
lot
of
things
that
smell
right
about
this.
You
know
they
just
like.
Okay,
we
everyone's
for
50
years,
I've
talked
about
upper
and
lower
layers
and
how
they're
sort
of
separate
circuits
and
how
they
interact
with
one
another.
This
idea
that
they're,
you
know
the
size
of
the
receptor
fields
are
quite
different
in
the
upper
and
lower
layers.
B
Now
we
know
that
there's
two
sets
of
complex
cells
in
the
upper
and
lower
layers,
they're,
both
independently
derived.
We
know
that,
for
the
lower
complex
cells
do
not
require
the
the
upper
complex
cells
to
and
the
upper
complex
does
not
require
lower
crimson
cells
and
the
upper
complex
cells.
Don't
require
input
from
layer
four,
if
you
disable
disable
layer.
Four,
you
still
got
complex
cells
in
layer,
two
three,
so
so
this
and-
and
so
this
is
like
ubiquitous
ideas
and
anyway,
just
there's
a
number
of
things
like
that.
B
It's
not
it's
it's
hard
to
think
of
ways
that
we
could
do
this
transform
using
neurons.
As
you
just
said,
it's
not
easy.
Yet
it
has
to
be
done
ubiquitously
throughout
the
cortex,
and
so
this
is,
you
know.
I
also
don't
think,
there's
something
to
be
done
elsewhere.
It's
not
like
I'm
going
to
ship
this
information
over
to
striatum
or
something
and
this
radio
is
going
to
ship
it
back
it
doesn't
that
doesn't
seem
right
to
me.
It's
like
it's
got
to
be
happening
everywhere,
so
yeah,
more
questions,
more
suggestions.
E
It's
interesting
what
you
were
saying
before
about
the
experiments
being
done
in
static
paradigms.
I
think
there's
a
bunch
of
evidence
now
that,
as
soon
as
you
start
introducing
movement
and
things
like
that,
all
of
those
classic
receptive
field
properties
just
go
completely
out
of
whack,
and
I've
heard
this
also
anecdotally
from
neuroscientists.
They
just
don't
know
what
to
make
of
it,
because
any
theory,
they
add,
doesn't
hold
anymore
as
soon
as
they
start
as
soon
as
the
animal
has
started
to
move
allowed
to
move.
No
matter
almost
doesn't
matter
what
the
movement
is.
B
Well,
that
and
that
yeah-
and
it
makes
sense
again
if
you
think
about
like
I'm
an
experimentalist
and
I
have
an
industrialized
animal
and
I'm
trying
to
figure
out
what
or
even
if
they're,
not
in
this
size,
I'm
just
showing
them.
You
know
sinusoidal
gradings
or
you
know
bars
that
are
moving
or
something
what
they
would
classically
do.
B
Obviously,
if
I
have
a
if
I
have
an
n
stopped
complex
cell
in
layer
3,
if
I
show
it
a
large
bar,
it
doesn't
respond,
so
they
should
narrow
the
bar
down,
say:
oh
look,
it
responds
at
this
orientation
and
then
they
go
down
to
the
lower
layers
and
they
say:
oh
look.
This
cell
responds
much
better.
If
I
make
it
wider
and
I
move
it,
you
know
so
they're,
basically
changing
the
input
to
to
to
match
and
that's
what
you're
going
to
get.
You
know
it's
like
until
you
have
this
difference.
B
The
idea
is,
you
have
a
differential
response
between
the
the
that's
the
thing
I'm
working
on.
How
do
I
use
the
differential
response
between
the
upper
complex
cells
and
the
lower
complex
cells?
So
I'm
trying
to
imagine
an
object
and
I'm
I'm
rotating
the
object
or
I'm
moving
my
eye
over
the
object,
the
object's
moving
on
its
own.
What
would
be
the
differential
responses
of
those
two
cells
and
what
could
I
infer
from.
C
B
Think
that
is,
I
think
that
is
the
you
know
the
classic
view.
There's
a
there's,
a
there's,
an
input
from
the
thalamus
to
the
lower
layer,
five
right
at
the
you
know,
and-
and
that
would
be
that
would
be
the
the
thing
that
generates-
that
complex
cell
respect
response
right
and
what
I
think,
what
I
think
and
those
in
layer.
Five,
of
course,
is
the
motor
output
layer
right
right.
B
B
These
divisions,
I'm
showing
on
the
lower
layer,
would
be
a
different
movement
vector
so
with
the
cortex
is
saying:
okay,
give
them
all
my
flow
patterns,
I'm
going
to
do
like
a
spatial
pooler
thing
and
figure
out
what
I'm
coming
up
with
the
set
of
flow
movement,
vectors,
and
so
each
of
these
each
of
these
slabs.
If
you
will,
would
represent
a
movement
vector,
that's
basically
just
a
complex
cell
and
that
complex
self,
you
can
now
project
right
back
to
the
cpg
and
say
I'm
gonna,
I'm
representing
the
movement.
B
You
just
did
I'm
representing
the
movement
you're
you're
performing
right
now
your
eye
is
moving
and
I'm
representing
it,
because
I'm
firing
and
and
and
so
we're
going
to
learn
the
connection
between
layer,
5
and
the
cpg,
we're
going
to
say.
Okay,
the
layer
5
cell
is
representing
this
movement
as
detected
by
flow,
and
I
we're
going
to
form
synapses
to
the
cpg.
B
That's
performing
that
movement
right
at
that
moment,
and
so
this
is
how
the
cortex
learns
to
control
movement
by
saying
I've,
modeled
the
movement
and
now
here
it
is,
let's
form,
connections,
whoever
cells
that
are
making
that
movement
happen
at
this
time.
So
it's
a
way
of
we
don't
have
to
assume
any
knowledge
of
what
the
cpg
looks
like
in
the
cortex.
It
just
projects
it.
I
think
I
went
off
in
a
rant
there,
but
yeah.
So
I
think
max
this
review
here
very
clearly.
There
are.
B
There
is
a
lower
layer,
three
and
a
lower
layer
5.,
and
what
we
have
speculated
is
that
that
the
that
the
lower
layer,
3
and
the
lower
layer
5
are
getting
magnesium
input.
Although
I
do
not,
I
don't
think
we
had
evidence
for
that,
but
that's
what
I'm
speculating,
because
that
would
explain
why
those
cells
or
complex
cell
behaviors
exist
and
they
don't.
As
you
said,
they
don't
require
layer.
Four,
if
you,
if
you
disable
layer,
four,
these
complex
they'll
still
respond.
B
So
this
idea
you're
right
that
this
idea
that
flows
from
four
to
three
to
five
stuff.
That's
not
it's
true,
but
it's
not
necessary
yeah!
It's
not
a
requirement!
So
these!
So
again,
this
idea
that
the
cortex
is
figuring
out
these
movement
vectors
on
its
own.
It's
doing
the
transformation
on
you
know
on
its
own
and
then
later
it
can.
Then
it
can
take
the
input
to
layer,
four
and
and
and
start
using
that
as
like
okay,
what
am
I?
What
am
I
seeing
at
this
location?
B
A
On
the
topic
of
location
and
orientation,
not
so
much
on
the
topic
of
movement,
sometimes
the
egocentric
alicentric
terminology
can
be
ambiguous
in
the
following
sense.
Sometimes
it's
more
useful
to
think
of
is
this
a
there
are
two
types
of
location
you
can
have
the
spatial
relationship
between
the
viewer
and
the
and
the
object,
or
you
can
have
a
spatial
relationship
between
the
object
and
another
object
and
the
first
kind
spatial
relationship
between
viewer
and
object.
A
B
A
C
A
I'm
I'm
saying
I'm
saying
collect
collectively
the
location
and
orientation
grid
cells
and
head
direction.
Cells
are
sick,
are
well
supposed
in
3d,
it's
a
six
dimensional
representation
of
the
spatial
relationship
between
the
animal
and
the
room,
and
if
you
split
it
apart,
yes,
the
grid
cells
on
their
own
are
more
alice-centric,
but.
A
Yes,
this
good
the
grid
cells
on
their
own.
It's
like
the
system,
has
decided
to
split
the
six
dimensions
into
something
that
is
allocentric,
but
the
space.
The
fundamentally
the
spatial
relationship
is
between
the
animal
and
the
room
and
that's
like
one
fundamental
type
of
look
of
of
of
spatial
relationship.
A
The
other
fundamental
type
is
between
two
external
objects
between
the
logo
and
the
coffee
cup,
the
carpet
in
the
room
and
and
sometimes
it's
more
useful,
to
think
to
draw
the
line
right
there
as
like.
That's
how
you,
rather
than
thinking,
egocentric
and
allocentric,
think
of
external
spatial
relationships
and
and
sorry
and
and
racial
relationships
between
you
and
the
outside
world.
B
I
just
been
thinking
about,
and
I'm
wondering
how
you
think
about
this,
and
I'm
bringing
this
up
to
say
this,
this
sort
of
address,
what
you're
thinking,
I'm
thinking,
there's
just
there's
just
reference
frame,
one
in
reference
frame.
Two,
it's
like
you
know
dr
seuss
thing,
one
and
thing
two
and,
and
I
just
I
need
to
go
back
and
forth
between
two
reference
frames.
I
don't
really
care
what
they
are,
because
then,
where
columns
and
what
columns
other
columns
who
knows
what
they
represent?
B
E
B
Okay,
so
maybe
I
can
start
doing
that
because
I
didn't
do
that
here
because
I
just
wanted
it.
It
helps
initially
to
start
thinking
about
like
oh,
what
are
you
talking
about
reference
frame,
1
and
reference
0.?
It
helps
to
think
about
ego
and
aloe
as
a
sort
of
a
placeholder
for
that.
So
if
I
just
started
doing
that
marcus,
would
that
sort
of
address
what
you're
talking
about?
Are
you
talking
about
something
different.
A
A
So
one
one
thing
that,
where
jeff
hinton's
work
kind
of
changed
over
the
years
was
back
in
the
80s,
he
would
talk
a
lot
about
about
both
the
both
types
of
representation,
the
viewer,
centric
one
and
the
object-centric
representation,
and-
and
these
two
types
and
one
interesting
change
that
happened
when
capsules
came
along,
is
he
no
longer
has
the
object-centric
representations
in
the
neurons
anymore?
So
he
no
longer
has
them
in
the
activations
they're.
A
All
that
information
is
only
in
the
weights,
the
the
spatial
relationship
between
parent
objects
and
child
objects.
Spatial
relationships
between
things
is
only
captured
ever
in
the
weights
and
and
so
like
a
capsule,
a
version
of
the
brain
or
whatever
wouldn't
require
neurons
to
represent
allocentric
locations.
B
But
I
have
to
make
centric
pred
I
have
to
make.
I
have
to
do
calculations
in
the
allocentic
space
like
if
I
I
want
to
get
from
point
a
and
point
b
in
allocentric
space.
I
have
to
be
able
to
calculate
that
right.
I
mean
it's
not
like
and
there
for
to
me
to
do
that.
I
think
I
need
activations.
I
can't
just
say
there
are
no
activations.
It's.
A
A
They
would
represent
that
as
part
of
the
state
of
the
object
that
they
would
have
like
a
they
call
them
instantiation
parameters,
and
they
would
update
the
state
of
the
object
in
a
way
that
captures
whether
the
stapler's
opening
or
the
the
animals
moving
its
limbs
or
sorry,
the
the
the
object
you're
viewing
is
is
moving
in
some
strange
way.
That's
the
best.
I
can
say
there.
I
think
that
there
isn't
a
a
a
nice
insight
in
capsules
that
that
you
don't
necessarily
always
need
to
use
activations
to
store
this
information.
B
B
I
was
gonna
mention.
There
was
something
that
max
presented
the
other
day
that
caught
my
eye
and-
and
I
didn't
mention
it
here,
but
I've
been
thinking
about
it
max-
has
proposed
that
the
layer,
six
simple
cells
were
a
prediction
of
the
next
input,
and-
and
so
I
I
know,
I'm
not
sure
I'd
buy
that
yet,
but
the
idea
that
they
represent.
B
A
different
point
in
time
of
the
input
right,
so
you
would
say:
oh
the
layer
forward,
simple
cells
would
be
the
current
input,
but
then
layer,
six
cells
would
be
prediction
of
the
next
input.
I
think
that's
what
max
presented
correctly.
If
I
got
that
wrong
and-
and
that
was
an
interesting
idea-
and
because
one
of
the
things
that
displacement
cells
requires
displacement
cells
require
that
we
compare
two
locations,
that's
that's
what
you
need
to
do
to
calculate
a
displacement
cell
is,
you
have
to
say?
B
Okay
now
those
two
locations
could
be
two
locations
in
the
same
reference
frame,
meaning
the
same
object
space
there's
in
that
case,
you
know
I
haven't
re-anchored.
I've
only
moved,
and
in
that
case
it
represents
a
distance
or
movement
vector
if
the
two
locations
are
in
different
objects,
meaning
it's.
I
can't.
I
didn't
just
move
from
one
to
the
other,
but
I
re-anchored.
B
Then
I
would
then
I
could
compare
the
prediction
to
my
current
input
and
that
would
that
would
be
a
displacement
or
perhaps
layer,
6
represented
the
last
input,
so
maybe
layer
6
is
delayed
input.
It's
like
what
I
just
had
and
now
I
get
a
new
input,
and
so
that's
the
opposite.
It's
t
minus
one
versus
two
plus
one,
and
so
now
I've
been
able
to
calculate
this.
B
The
point
is
the
calculating
displacements
which
we
know
has
to
occur
in
some
form
requires
to
comparing
these
two
different
locations
and
it'd
be
really
nice
if
those
locations
were
coincident
in
time,
as
opposed
to
just
a
transition
which
seems
harder
to
manage,
and
so
the
idea
that
that
you
could
have
two
different
representations
of
the
input
and
different.
I
again
it's
very
confusing.
C
Have
you
read,
do
you
have
any
thoughts
on
because
in
hippocampal
recordings
as
a
ratchet
versus
a
track,
you
don't
actually
see,
in
my
understanding,
a
single
place
cell
activating
what
you
see
is
a
sequence
on
the
theta
rhythm
of
the
place
cells
just
prior
and
the
expected
place
cells
in
the
future.
You're
talking.
C
E
C
B
Well
it
what
we've
talked
about
here
is
that
all
the
mechanisms
I
mean
place,
cells
and
grid
cells
show
this
precession
phenomena.
Now
there's
two
things
there.
One
is
just
that:
a
cell,
as
you
approach,
you
know,
the
cell,
where
the
cell
fires
in
the
theta
cycle
tells
you
where
you
are
in
some
sense
the
other.
The
idea
is
that,
in
a
very
short
period
of
a
part
of
a
cycle
of
the
theta
cycle
that
you
played
through
a
sequence,
I
think
that
might
have
been
what
you
were
just
talking
about.
C
So
there's
there's
two
there's
two
things
that
you
see
in
the
hippocampus
in
my
base
of
what
I've
read.
There's
these
sharp
wave
ripples,
which
is
a
rapid
replay
of
a
sequence
either
forward
or
backwards,
not
not
the
ripples.
Just
as
I'm
traversing
on
the
theta
rhythm,
the
first
gamma,
let's
say:
there's
place
a
b
c
and
d.
If
I'm
at
place
c
it'll
quickly
replay
a
b
and
then
it
will
also
replay
c
and
d
and
the
c
will
be
at
the
center
of
the
theta
rhythm.
But
it's
actually
playing
both.
B
Yeah
that
wouldn't
be
right,
it's
different
different
place
cells
are
well
you
can
you
can
call
it
the
sequence
or
you
could
just
say
these
place.
Cells
are
firing
early,
and
these
platforms
are
firing
late.
Well,
I
think
we're
always
going
to
get
with
this
everything
that
we've
ever
read
about
how
grid
cells
come
about
in
place
cells,
but
is
that
it
requires
this
differential
firing,
frequency,
theta,
theta.
B
That
you
know
the
whole
the
the
prominent
mechanism
is.
You
got
two
cells
that
are
firing
lockstep
and
theta,
and
when
you're
moving
one
starts
firing,
a
slightly
faster,
very
slightly,
faster
rhythm
and
and
therefore
which
cells
are
in
sync
in
which
cells
are
out
of
sync
becomes
it
changes,
and
this
is
the
basic
way
that
grid
cells
come
about
and
and
that,
and
that
explains
all
the
pre-session
stuff
that
people
observe.
B
B
I'm
not
happy
with
any
one
of
those
at
the
moment,
but
I
do
think
it's
it's
when
I
say
that
quarter
column
has
to
create.
You
know
an
allocentric
grid
cell.
My
assumption
is
that
it's
doing
that
using
a
mechanism
where
there's
a
essentially
a
beating
effect
between
two
theta
rhythms
and
that
that
has
to
exist
here
and
whether
there's
there's
this
one
idea
is
that
you
have
a
set
of
cells,
each
firing
at
a
different
phase.
B
B
You
know,
but
it
has
to
occur
somehow
some
way
by
the
way,
just
one
more
thing
that
that
I've
mentioned
in
the
past-
but
you
know
it
comes
into
this
as
well-
is
that
you
know
when
we
observe,
let's
say
vision
and
I'm
looking
at
some
object
and
I'm
moving
my
eyes
to
attend
to
the
different
parts
of
the
object.
Now,
if
I
say
oh,
I
need
to
tend
to
the
coffee
cup
handle.
I
need
to
come
to
the
lip.
B
B
I
could.
I
could
essentially
by
changing
the
theta
rhythm,
I
can
have
the
eyes
move
further
or
less
depend
on
the
same
movement.
It's
sort
of
like
that
frequency
could
be
the
encoding
of
distance
if
you
will
and
we-
and
we
do
this
another
example
we've
given
is
like
you
know,
I
can
pick
up
a
coffee
cup,
but
then
I
can
pick
up
a
little
demi-test
coffee
cup,
which
is
small,
but
it's
a
small
version
of
the
coffee
cup,
but
I
can
still
manipulate
it.
Just
like
I
can
do
the
bigger
cups.
B
So
how
do
I
do
that?
You
know
it's
like
you
know.
It's
like
it's.
So
it
seems
in
this
inherent
ability
to
scale
is,
is
an
important
capability.
Another
example
is
like
you
can
write
your
name,
you
know
with
a
large
lettering
or
you
can
find
it
with
a
very
small
lettering.
It's
the
same
motor
behavior.
Yet
somehow
I
can
make
it
big,
and
so
now
I
can
make
it
small
right
and,
and
so
it's
not
like
I've
memorized
it
big
and
small.
B
C
About
yourself,
the
idea
of
scaling
is
interesting
too,
because
it
also
is
an
alternative
explanation
for
various
visual
illusion
phenomena.
Like
I'm
sure
you
came
along
with
the
bigger
circle,
with
smaller
circles
and
the
same
size
circle
with
smaller
circles
around
it,
you
perceive
the
the
center
circle
relative
to
the
surrounding
circles.
That's
based
on
its
relative
size,
not
its
absolute
size,
which
would
be
explained
by
what
you're
describing,
which
is
your
mind,
creates
an
invariant
object,
representation
that
can
be
scaled
up
and
down.
B
Of
it
in
terms
of
optical
illusions,
but
it's
an
interesting
observation.
So
again,
this
is
like
a.
I
think.
The
scaling
factor
is
something
that's
it's
it's
it's
ubiquitous,
and
so
it
has
to
be
a
major
architectural
feature
of
the
cortex
I've.
I
propose
that
this
is
this
could
be
done
with
the
matrix
cells
and
the
thalamus,
which
project
broadly
to
layer,
one
and
other
places
and
could
encode
timing.
I
think
it's
somewhat
consistent
with
you
proposed
to
I
forget
now,
but
but
anyway
yeah.
B
So
these
again
that's
another
requirement
that
the
cortex
has
to
be
able
to
do,
and-
and
so
you
know,
I'm
trying
to
resolve
all
these
things
at
once.
If
I
could
possibly
do
it
because
we've
been
spending
years
talking
about
these
things
individually
and
then
you
know,
how
do
we
integrate
them
all
together?
It's
tricky.