►
Description
Karan Grewal reviews the paper "Self-Organization in a Perceptual Network" from 1988, and argues that the use of Hebbian learning rules (1) is equivalent to performing principal components analysis (PCA), and (2) maximizes the mutual information between the input and output of each unit in a standard neural network, more commonly referred to as the InfoMax principle.
“Self-Organization in a Perceptual Network" by Ralph Linsker: https://ieeexplore.ieee.org/document/36
Other resources mentioned:
• “Linear Hebbian learning and PCA” by Bruno Olshausen: https://redwood.berkeley.edu/wp-content/uploads/2018/08/handout-hebb-PCA.pdf
• “Theoretical Neuroscience" textbook by Dayan & Abbott: https://mitpress.mit.edu/books/theoretical-neuroscience
• “Representation Learning with Contrastive Predictive Coding” by van den Oord et al.: https://arxiv.org/abs/1807.03748
• “Learning deep representations by mutual information estimation and maximization” by Hjelm et al.: https://arxiv.org/abs/1808.06670
- - - - -
Numenta is leading the new era of machine intelligence. Our deep experience in theoretical neuroscience research has led to tremendous discoveries on how the brain works. We have developed a framework called the Thousand Brains Theory of Intelligence that will be fundamental to advancing the state of artificial intelligence and machine learning. By applying this theory to existing deep learning systems, we are addressing today’s bottlenecks while enabling tomorrow’s applications.
Subscribe to our News Digest for the latest news about neuroscience and artificial intelligence:
https://tinyurl.com/NumentaNewsDigest
Subscribe to our Newsletter for the latest Numenta updates:
https://tinyurl.com/NumentaNewsletter
Our Social Media:
https://twitter.com/Numenta
https://www.facebook.com/OfficialNumenta
https://www.linkedin.com/company/numenta
Our Open Source Resources:
https://github.com/numenta
https://discourse.numenta.org/
Our Website:
https://numenta.com/
A
Okay,
we're
recording
so
I'm
going
to
talk
about
hebbian
learning
and
the
connection
with
the
infomax
principle.
Most
of
most
of
what
I'm
going
to
discuss
came
from
this
paper
from
1988,
which
I
went
through,
and
I
was
written
by
ralph
linsker.
Now,
there's
a
lot
of
nowadays.
There's
a
lot
of
momentum
behind
using
mutual
information
as
an
objective
to
learn.
A
A
So
in
everything
I
talk
about,
this
is
the
underlying
framework
that
we're
going
to
assume.
So
we
have
a
we
have
a
bunch
of
pre-synaptic
cells,
they
all
have
activations
x1
through
xn
and
then
an
output
which
is
which
is
y
and
they
all
get
summed
together
in
a
pretty
usual
way,
which
are
all
which
we've
all
seen
before
so
the
heb
rule,
which
I'm
assuming
most
people
are
also
familiar
with
and
I'll
review
it
again
here.
It's
just
the
change.
A
But
for
now,
let's
just
talk
about
how
heavy
and
learning
is
connected
to
pca
here,
so
pca
is
a
dimensionality
reduction
technique
and
what
what
happens
is
when
you
have
a
bunch
of
data
in
in
a
potentially
high
dimensional
space
you're,
finding
the
you're
finding
the
axes
that
give
the
largest
variation
in
the
data.
So
if
you
were
to
project
all
these
data
points
onto
e1
here,
then
you'd
get
you'd,
get
the
maximum
spread
in
the
data.
A
So
most
of
the
information
is
preserved
and
if
you
try
to
reconstruct
the
data
projecting
onto
just
e1,
that's
the
one:
that's
that's
the
one
direction
or
that
you
can
protect
onto
that,
will
give
you
the
best
reconstruction
error,
and
you
can
you
don't
just
have
to
project
onto
one
dimension.
You
can
project
onto
multiple
dimensions,
but
the
constraint
there
is
that
these
these
different.
A
I
guess
these
different
eigenvectors,
which
you're
projecting
onto
all,
have
to
be
orthogonal
to
each
other,
so
the
eigenvectors
are
just
the
or
these
vectors
that
you're
projecting
on
to
that
define
the
the
surface
that
that
you're
projecting
onto
are
the
eigenvectors
of
the
covariance
matrix
of
the
of
the
data
points
that
you
have.
So
if
you
just
project
on
to
the
first.
A
So
so
there's
a
relation
between
this
and
and
heavy
and
learning
so
far.
So,
let's
just
so.
This
is
the
hebbian
rule
that
I
showed
a
little
earlier,
and
this
is
all
it's
saying
is
the
change
in
the
weights
is
proportional
to
the
output
and
the
pre-synaptic
activation.
So
the
and
I
and
I
have
angle
brackets
here
what
the
angle
brackets
represent
is
just
this
is
averaged
over
a
over
a
large
batch.
A
I
know
and
all
the
references
I
looked
at
this
is
a
very
common
notation
to
use
so,
instead
of
just
having
one
example,
you'd
have
multiple
examples
and
you
just
average
it
over
this
item,
and
so
now,
with
a
bit
of
manipulation,
we
get
a
new
we're
going
to
end
up
with
a
new
learning
rule
here,
and
so
what
I
did
here
is
to
get
from
line
1
to
line
2.
y
is
just
simply
equal
to
this,
this
linear
transformation
of
x
and
then
and
then
I
pull
I
pull
the
w.
A
I
pulled
the
the
weight
outside
these
angle
brackets
because
this
the
weights
change
at
a
much
slower
on
the
much
slower
time
scale
than
the
x's
do
and
then
ultimate
and
then
what
we
get
is
something
that
looks
like
this
and
you'll
notice.
You
have,
we
have
the
x's
being
multiplied
with
each
other,
and
that
is,
and
that
can
be
interpreted
as
the
covariance
between
multiple
individual
data
points,
and
so
what
we
get
is
the
change
in
a
particular
weight
is
just
the
covariance.
A
Between
between
that,
between
a
certain
data
point
between
that
weight
and
another
one
summed
up
and
this
so
this
is
called
the
correlation
based
plasticity
rule,
and
this
is
basically
equivalent
to
the
hebbian
learning
rule.
Except
it's.
I
guess
it's
a
it's
a
it's
a
different
way
to
state
it
over
a
large
number
of
examples.
So
the
so.
The
the
relationship
between
the
data
points
tells
you
quite
a
bit
about
how
the
weights
are
going
to
change.
A
Right,
yeah
yeah:
I
should
mention
that
so
there's
an
assumption
here
about
everything
having
zero
mean,
because
otherwise
you
couldn't
you
couldn't
take
the
covariance
matrix
like
this
now,
okay,
so
that
that's
all
nice,
that's
like
a
nice.
I
guess
something
we
have
in
our
back
pockets,
but
now,
let's
actually
try
to
relate
it
to
something
else.
A
So
so
by
the
way,
there's
there's
a
lot
of
math
and
derivations
these
next
couple
slides,
but
I'm
trying
I'll
try
to
make
it
as
intuitive
as
possible
and
not
go
on
too
deep
into
it.
Okay,
so
we
know
y
is
the
output
of.
Why
is
the
postsynaptic
cell?
Now
we
can
now
now
we
can
ask
the
question
of
what
what
is
the
variance
of
y
right
and
so
as
super
time
mentioned.
A
If
we
assume
that
y
has
zero
variance,
then
it's
gonna,
we're
gonna,
get
something
that
looks
like
this,
because
this
this
y
bar
here
just
becomes
zero
and
so
and
then
we
can
rewrite
y
itself
as
w
transpose
x.
We
get
something
that
looks
like
this.
Now,
with
a
little
bit
of
manipulation
I
can
get.
I
can
pull
the
x's
together
and
get
x
x,
transpose,
which
is
which
just
turns
into
the
covariance
matrix.
A
A
A
C
C
A
A
No
no
worries,
so
that
was
one
argument
for
heavy
and
learning
basically
doing
pca,
and
I
have
another
argument
for
that,
which
is
a
bit
it's
much
much
more
mathematical
and
I'll
try
to
go
over
it
as
smoothly
as
possible,
and
this
is
not
from
the
from
the
from
the
linsquare
paper.
This
is
taken
from
another
source,
but
I
really
liked
the
way
this
was
fleshed
out.
A
So
so
now,
how
does
this
work?
So?
The
correlation
based
plasticity
rule
which
we
got
on
the
last
slide
written
in
written
in
vector
form,
is
basically
basically
looks
exactly
like
this.
Okay.
Now,
what
we
can
do
here
is
diagonalize
the
covariance
matrix
so
that
you
have
q
written
as
e
times
lambda
times
e
transpose,
where,
where
lambda
is
this
diagonal
matrix
of
the
eigenvalues
across
the
diagonal
and
big
e
here?
Is
this
matrix?
Where
each
column
is
the
is
corresponds
to
an
eigenvector
of
of
q
of
the
covariance
matrix?
A
Now,
once
you
have
that
we
can
introduce
a
change
of
basis
so
now,
instead
of
working
with
w
we're
going
to
be
working
with
v,
so
v
is
just
basically
the
each
eigenvector
multiplied
by
the
inner
product
of
each
eigenvector,
with
with
the
with
our
weight
vector
w.
So
once
we
have
that
the
update
rule
for
v
instead
of
w,
because
now
we're
interested
in
v
just
becomes
change.
The
delta
v
is
lambda
times
v
and
now
this
is
an
ordinary
differential
equation.
A
So
we
have
the
tools
to
solve
that
and
all
this
becomes
is
I
write
it
as
a
function
of
time
so
since
it
is
being
updated
over
time
so
v
at
time,
t
is
the
initial
value
of
v.
At
time,
0
times
e
to
the
power
of
lambda
times.
T-
and
so
what's
important
here
to
note
is
that
is
how
this
would
change
over
time
right.
A
So
the
lambdas
are
the
eigenvalues
and
they
all
they're
all
strictly
positive,
because
that's
just
an
assumption
based
on
based
on
the
covariance
matrix,
and
so
the
one
that
is
the
largest
will
tend
to
dominate
as
t
goes
to
infinity.
So
if,
if
lambda
1
is
the
largest
eigenvalue,
then
this
vector
v
will
converge
to
the
value
one:
zero,
zero,
zero,
zero,
zero
and
and
since
and-
and
so
if
v
is
going
to
one
zero
zero,
zero
zero.
A
Then
that
means
in
our
original
reference
frame,
which
is
that
for
w
w
would
would
converge
to
the
the
principal
eigenvector
itself.
So
that
might
be
a
lot
to
digest
here.
But
this
all
this
is
doing
is
showing
that
mathematically,
that
w
the
weight
vector
converges
to
the
principal
eigenvector
and
so,
and
so
that
means
that
doing
heavy
learning
would
basically
is
the
same
as
doing
pca.
B
A
A
B
D
A
So
in
I
guess
here,
one
one
cons,
one
constraint
is
that
we
want
these
eigenvectors
to
be
orthonormal,
so
they
do
have.
They
are
constrained
to
have
a
norm
one,
and
if
you
use
the
oya
rule
instead
of
the
the
vanilla
head
rule,
then
you
do
get
yeah.
The
weight
vector
tends
to
have
a
unit
norm.
So
in
that
case,
it's
more
clear
how
this
would
converge
when
you
have
multiple,
when
you
have
multiple
output,
post-synaptic
cells
right.
E
B
E
A
A
Okay,
so
now
so
I
talked
so
far
about
how
heavy
learning
picks
up
the
first
principal
component
and
does
in
in
is
in
some
sense,
analogous
to
principal
components.
Analysis
now,
I'm
going
to
try
to
I'm
going
to
make
the
transition
from
going
from
pca
to
the
info
to
preserving
mutual
information
between
input
and
output.
So
the
info
max
principle
is
basically
stated
in
two
parts.
A
So
one
goal
is
to
maximize
the
total
information
conveyed
by
the
output
message
y
and
the
second
one
is
to
minimize
the
information
that
y
conveys
to
to
one.
That
already
knows
the
input
message
x.
So
one
way
to
think
of
this,
or
at
least
how
I,
how
I've
been
thinking
of
this,
is
that
going
back
to
our
pca
diagram.
If
you're
maximize,
if
you're
doing
number
one,
which
is
maximizing
total
information
conveyed
about
the
output
message,
then
you'd
want
to
find
that
principal
component.
A
A
So
that's
kind
of
how
I'm
reading
this
now
now
there's
a
hand
waving
argument
that
I've
sort
of
convinced
myself
is
true
from
going
from.
So
by
the
way.
I
just
want
to
say
that
I
didn't
come
across
anything
too
formal
for
why
pca
is
in
fact
equivalent
to
infomax,
but
but
I'm
sure,
there's
stuff
out
there.
I
just
haven't
gonna
come
across
it.
A
Yet,
and
and
in
the
few
cases
where
linscore
was
getting
rigorous,
there
were
a
lot
of
assumptions
made
about
the
out
the
structure
of
the
the
inputs
and
the
outputs
which
I'll
get
to
on
the
next
slide.
But
for
right
now,
just
to
give
an
intuition
as
to
why
I
see
pca
being
being
very
related
to
infomax
is
for
the
following.
So
here's
our
this
is
what
mutual
information
is
right.
It's
the
entropy
in
the
output
y
minus
the
entropy
minus
the
conditional
entropy
of
y
given
x.
A
A
I
guess
this.
This
second
part
here
could
relate
to
could
relate
to
this
item
number
two
in
the
information
right
right
so
for
for
minimizing
the
information
that
y
conveys
to
one.
That
already
knows
the
input
message
x.
Then,
then,
this
this
item
is
minimized
right.
So
this-
I
guess
this
having
this
second
term,
be
really
small-
is
exactly
the
same
as
number
two,
but
but
now
I'm
not
relating
it
back
to
pca.
So
I
don't
know
I
I
had.
A
Now
the
second
one,
which
is
a
special
case
which
linscore
talked
about
and
also
I
found
a
nice.
I
guess
a
sentence
in
this
in
this
textbook
about
it
basically
says
for
gaussian
input
statistics.
So
now
we're
assuming
that
x
is
distributed,
gaussian,
an
output
that
is
corrupted
by
noise.
That
is
also
gaussian.
So
now
we're
assuming
y,
is
also
gaussian,
maximizing
the
variance
of
y
by
a
hebien
rule
maximizes,
not
only
the
output
entropy,
but
also
the
amount
of
information
that
y
carries
about
x.
A
So
this
here
now
we're
assuming
that
the
inputs
and
the
outputs
are
both
gaussian
distributed,
and,
in
that
case,
using
a
heavy
and
learning
rule
would
maximize
the
mutual
information
between
y
and
x,
and
so
linscore
talked
about
this
in
the
paper
he
went
into
some
just
some
details
of
it.
He
skipped
a
lot
of
proofs,
but
the
main
idea
was
there,
but
this
is
a
very.
A
This
is
a
special
case
where
you
are
making
this
assumption,
and
so
I
I
haven't
come
across
anything
yet
which
shows
that,
regardless
of
what
we
assume,
the
distribution
to
be
over
the
input
and
output,
that
it
is
indeed
maximizing
mutual
information,
okay,
so
there's
also
so
I
I
mentioned
near
the
beginning
that
there's
been
a
lot
of
reasons.
Probably
I
mean.
B
B
F
B
C
A
G
Your
optimization
objective
is
maximizing
the
variance
and
if
you
explicitly
solve
that,
you
will
get
like
the
first
eigen
vector,
but
you
can
also
derive
pca
as
a
maximum
likelihood
estimation
on
a
gaussian
and
so
like
that
assumes
it's
a
gaussian
right,
because
you're
you're
parameterizing
the
distribution
as
your
gaussian
and
then
doing
maximum
likelihood
estimation
and
then
you'll
get
like
an
equivalent
like
it's
an
equivalent
way
of
deriving
it.
But
one
doesn't
assume
a
gaussian
one.
Does.
A
B
A
Okay,
so
there's
been
a
lot
of
work
that
uses
mutual
information
for
representation
learning,
and
so
this
so,
for
example,
one
example.
One
popular
example
is
cbc
paper
from
two
three
years
ago,
which
is
building
up
representations
of
inputs
using
using
mutual
information
as
their
objective
and
that's
later
being
used
for
downstream
classification
tasks.
F
I
think
the
the
one
thing
I
can
possibly
think
of
is
when
you're
asked
when
you're
wondering
like
why
pca
doesn't
work.
I
mean,
as
you
say,
it's
trying
to
maximize
the
amount
of
information
that
y
is
carrying
about
x,
but
it
doesn't
necessarily
mean
that
it's
good
at
trying
to
maximize
that
amount
of
information,
as
opposed
to
other
techniques.
A
F
A
A
I
know
under
the
under
the
under
the
assumptions
which
I
showed
on
the
slide
before
this
about
inputs
and
outputs
being
gaussian
distributed.
Then
it
is
equivalent,
but
otherwise
I'm
not
sure
if
it's
I'm
not
sure
if
it's
always
equivalent
or
if
there
are
edge
cases
where
it's
not
necessarily
equivalent.
B
I
mean
I
thought
the
contrastive
predictive
contrast
of
stuff
did
something
more
than
just
maximizing
the
mutual
information
they're,
also
minimizing
the
mutual
information
across
different
classes,
which
the
pure
information
theory
stuff
doesn't
have
any
notion
of
classes
in
there
right.
So
the
idea
being
a
pure
information
maximization
is,
you
know,
you're
just
trying
to
you're
it's
sort
of
a
compression
mindset
that
you're
reducing
the
representation
but
you're
carrying
as
much
information
as
you
have
about
the
inputs,
you
in
theory,
you
can
decompress
it
and
reconstruct
the
input.
B
I
mean
that
was
the
whole
kind
of
original
motivation
for
shannon
and
whereas
the
contrastive
stuff
says
we
don't
the
the
most
important
thing
is
to
make
sure
we
well.
We
want
to
create
maximize
information
with
the
input,
but
we
also
want
to
minimize
information
with
inputs
that
come
from
other
classes.
B
B
You
know
not
exactly
sure
what
that
would
be,
but
maybe
the
stripes
on
the
face
or
something
like
that,
and
so
those
features
would
get
extra
weight
that
if
you
just
care
about
you
know
reconstruction,
you
wouldn't
give
those
features
extra
weight
necessarily
right.
So
I
think
this
contrastive
piece
is
really
key
and
in
the
past
I
think
people
have
tried
to
just
stack
pure
information
present.
B
A
B
Had
variations
of
it,
which
is
still
in
theory
unsupervised,
but
it
relies
on
the
supervised
thing
like
this.
You
know
some
of
the
cpc
stuff.
If
you
have
a
thousand
categories
in
imagenet,
if
you
randomly
pick
images
from
the
from
your
batch,
you
can
assume
with
high
probability
it's
from
another
class.
You
don't
actually
have
to
know
it's
which
class
label
it
is
all
you
need
to
know
is
that
it's
another
class,
and
so
you
can
separate
try
to
separate
those
representations
out.
B
C
I'm
making
I'm
sorry
I'm
making
this
comment
without
fully
understanding,
contrastive
predictive
coding,
but
is
there
a
formulation
of
it
that
tries
to
push
the
mutual
information
to
an
earlier
layer
and
preserving
the
differentiating
information
and
the
current
or
pushing
it
forward?
In
other
words,
a
separation
of
mutual
versus
non-mutual
information
in
layer
wise
or
is
it
just
kind
of.
B
B
C
B
Yeah
I
mean
it's
interesting,
you
know
the
you
know.
Koran
walked
through
this
in
a
relationship
with
heavy
learning
and
pure
information
preservation.
So
there
might
be
some
small
variation
to
have
been
learning
that
gives
you
the
contrastive
piece
and
it'd
be
very
interesting
to
see.
If
there's
something
like
that
in
in
biology.
C
Well,
you
kind
of
made
the
case
that
there
has
to
be
because
otherwise
everything
would
gravitate
toward
one
shot.
So
I
think
that's
that's
the
essence
of
of
it's
got
to
be
essential
to
it,
because
otherwise
you'd
be
hot
firing
all
the
time
on
just
one
channel,
and
then
everything
else
would
after
fee
away.
So
there's
got
to
be
something
on
those
other
components.
B
Well,
okay,
so
there's
two
separate
things:
one
is
picking
up
successive
principal
components
for
which
you
need
competition.
That's
one
thing:
that's
completely
separate
from
the
contrasted
predictive
coding
piece,
that's
a
completely
orthogonal
concept,
and
so
the
first
part
I
think,
there's
pretty
decent
evidence
that
something
like
pca
is
going
on.
There's
competition
across
units
and
so
on,
jeremy
might
know
more
about
this,
but
the
contrastive
piece,
I'm
not
sure,
and
I
think
there
has
to
be
something
like
that.
It's
it's
just
such
a
powerful
principle.
Yeah.
C
C
B
B
C
Right
but
they're
not
going
to
show
up
on
the
first
principal
component.
All
those
are
the
right,
yeah
components
beyond
that
point:
yeah
yeah!
So
I,
when
I'm
the
I
was
trying
to
drive
off
your
comments,
saying
that
people
just
tried
to
stack
linear.
You
know
linear
layers
and
had
problems.
It
would
be
like
you
only
had
blob
detectors.
A
Speaking
of
just
having
blob
detectors,
linscore
did
talk
a
lot
about
how
you
know
we
have
you
have
different
sort
of
you
have
orientation
selectivity
and
all
these
different
things
happening
in
the
in
the
visual
processing
system,
and
he
tried
to
relate
that
to
this
sort
of
competition
that
you
have
among
different
post-synaptic
cells
and
and
based
on
based
on
this
information
principle.
That's
how
everything
that's,
how
you're
getting
different
cells
that
are
recognizing
different
different
things.
A
B
B
You
know
it's
an
information
reduction
that
you're
creating
if
you're
given
you're,
creating
the
most
compact
representation
that
can
best
reconstruct
the
input.
What
bruno
kind
of
showed?
Well
suppose
you
relax
that
you
allow
having
an
over
complete
basis
and
that's
you
know
that
he
he
introduced
that
using
sparsity,
then
the
feature
detectors
you
get
end
up,
looking
a
lot
more
much
closer
to
what
you
get
in
the
visual
cortex
than
if
you
just
do
pure
pca.
B
It's
not
doing
it's
not
creating
the
most
efficient
representation,
it's
actually
creating
a
redundant
representation,
an
overcomplete
representation,
and
that
has
benefits
in
and
of
itself.
But
if
you
do
that,
then
that's
how
you
get
perceptive
fields
that
really
look
very
closely
to
what
you
see
in
b1.
D
G
B
Kind
of
related
for
us,
you
know
a
lot
of
this
started
from
oh
yeah
and
linsker,
and
I
remember
when,
when
this
paper
came
out,
because
I
was
just
a
graduate
student
at
that
point,
it
had
a
huge
impact
on
the
community,
just
the
thought
that
there
could
be
a
tie
from
between
heavy
and
learning
and
information,
maximization
and
pca,
and
all
of
this
kind
of
quote-unquote,
optimal
notions
of
optimality
and
that
something
like
the
heb
rule
and
local
learning.
Rules
that
are
in
biology
might
actually
be
doing.
B
I
I
mean
depends
what
you
mean
by
short-lived.
I
mean
we're
still
talking
about
it
here
today
and
we
said
it
carried
on.
I
mean
there's
bruno
stuff,
which
had
a
huge
impact
on
the
field
and
I
don't
think
I'm
not
too
familiar
with
the
whole
lineage.
But
to
me
not
much
happened
between
what
bruno
did
until
this
contrastive
stuff
came
into
play.
B
B
D
The
newer
methods
also
don't
rely.
The
nearest
self-supervised
methods
don't
rely
on
the
contrastive
piece,
so
I
know
you're
talking
about
the
importance
of
negative
pairs,
but
if
you
take
byol,
they
don't
require
negative
piece:
they're,
just
learning
from
a
slow
moving
average
learning
like
a
target
network
in
an
online
network,
and
they
only
rely
on
kind
of
positive
samples
right,
like
so
you're
like.