►
Description
In this research meeting, our research intern Alex Cuozzo reviews some notable papers and explains high level concepts related to learning rules in machine learning. Moving away from backpropagation with gradient descent, he talks about various attempts at biologically plausible learning regimes which avoid the weight transport problem and use only local information at the neuron level. He then moves on to discuss work which infers a learning rule from weight updates, and further work using machine learning to create novel optimizers and local learning rules.
Papers / Talks mentioned (in order of presentation):
• "Random synaptic feedback weights support error backpropagation for deep learning" by Lillicrap et al.: https://www.nature.com/articles/ncomms13276
• Talk: A Theoretical Framework for Target Propagation: https://www.youtube.com/watch?v=xFb9N4Irj40
• "Decoupled Neural Interfaces using Synthetic Gradients" by DeepMind: https://arxiv.org/abs/1608.05343
• Talk: Brains@Bay Meetup (Rafal Bogacz) : https://youtu.be/oXyQU0aScq0?t=246
• "Predictive Coding Approximates Backprop along Arbitrary Computation Graphs" by Millidge et al: https://arxiv.org/abs/2006.04182
• "Identifying Learning Rules From Neural Network Observables" by Nayebi et al: https://arxiv.org/abs/2010.11765
• "Learning to learn by gradient descent by gradient descent" by Andrychowicz et al: https://arxiv.org/abs/1606.04474
• "On the Search for New Learning Rules for ANNs" by Bengio et al: https://www.researchgate.net/publication/225532233_On_the_Search_for_New_Learning_Rules_for_ANNs
• "Learning a Synaptic Learning Rule" by Bengio et al: https://www.researchgate.net/publication/2383035_Learning_a_Synaptic_Learning_Rule
• "Evolution and design of distributed learning rules" by Runarsson et al: https://ieeexplore.ieee.org/document/886220
• "The evolution of a generalized neural learning rule" by Orchard et al: https://ieeexplore.ieee.org/document/7727815
- - - - -
Numenta is leading the new era of machine intelligence. Our deep experience in theoretical neuroscience research has led to tremendous discoveries on how the brain works. We have developed a framework called the Thousand Brains Theory of Intelligence that will be fundamental to advancing the state of artificial intelligence and machine learning. By applying this theory to existing deep learning systems, we are addressing today’s bottlenecks while enabling tomorrow’s applications.
Subscribe to our News Digest for the latest news about neuroscience and artificial intelligence:
https://tinyurl.com/NumentaNewsDigest
Subscribe to our Newsletter for the latest Numenta updates:
https://tinyurl.com/NumentaNewsletter
Our Social Media:
https://twitter.com/Numenta
https://www.facebook.com/OfficialNumenta
https://www.linkedin.com/company/numenta
Our Open Source Resources:
https://github.com/numenta
https://discourse.numenta.org/
Our Website:
https://numenta.com/
A
That
would
be
great,
I
say
all
question
marks,
because
there
seems
to
be
evidence
that
there's
multiple
learning
rules
and
there's
multiple
different
learning
machines
and
different
brain
structures
might
require
different
learning
rules
et
cetera.
So
there's
there's
a
lot
of
complexity
there
and
it
would
be
nice
if
this
started
to
work,
because
currently
hebbian
rules
do
not
generalize
to
deep
networks
and
another
problem
associated
with
this
is
that
it's
really
hard
to
get
all
the
learning
rules
from
biology.
So,
as
I
just
said,
biology
is
very
complex.
A
The
third
point
here
is
that
either
way
we're
pretty
sure
that
the
brain
is
not
doing
back
propagation
as
it
exists
in
computer
hardware
today
and
it
it
might
approximate
back
prop
in
some
way,
and
that's
definitely
up
for
debate.
It's
it's
very
likely
that
something
something
that's
sort
of
ground.
A
So
I
was
debating
whether
to
conclude
this
slide,
because
I
think
we
should
be
pretty
familiar
with
it,
but
for
I
guess
the
folks
watching
on
youtube
or
people
who
aren't
super
familiar
with
backpropagation.
This
is
just
a
very
brief
overview,
so
what's
happening
here.
Is
you
have
a
bunch
of
inputs
on
the
left?
That's
x1
through
xn.
A
So
one
immediate
problem
with
this
is
that
it's
called
the
weight
transport
problem,
and
so
this
formulation
on
the
left
here
is
exactly
what
I
was
showing
on
the
slide
before,
and
it's
that
the
activations
of
layer,
l
plus
1,
are
the
matrix
multiplication
of
the
weights
of
that
layer.
Multiplied
by
the
activations
of
the
previous
layer,
with
the
bias
pass
through
an
activation
function,
and
then
the
equation
below
that
is
the
error
function
right.
So
you
have
the
error.
B
A
Yeah,
okay
cool.
So
then,
this
red
arrow
corresponds
to
this
gray
arrow
going
forward,
and
so
the
transpose
of
this
weights
matrix
corresponds
to
these
red
errors
here,
and
so
what
this
looks
like
in
biology
is
something
like
this.
These
are
neurons,
but
they're
represented
as
clouds
just
for
lack
of
visual
tools
that
I
could
find
in
google
drive,
but
yeah,
so
neuron
a
would
be
connected
in
on
b
and
then
in
order
to
propagate
the
errors
backwards.
A
If,
if
forward
is
to
the
right
and
backwards
to
the
left,
then
every
forward
connection
from
a
to
b
would
also
have
a
corresponding
backwards
connection,
and
this
is
just
shown
not
to
be
the
case.
It's
not.
It's
not
the
case
that
every
sort
of
forward
connection
in
biology
has
corresponding
backwards.
Connection.
C
A
So
that's
why
this
is
kind
of
biologically
implausible
and
a
reason
why
that
propagation,
isn't
you
know,
as
it
currently
is
formulated
a
very
reasonable
explanation.
So
one
thing
that
was
proposed
a
while
ago
and
what
jeff
hinton
called
a
miracle
was
that
random
synaptic
feedback
weights
support
air
back
propagation
for
deep
learning.
So
what
does
that
mean?
A
This
is
the
same
formulation
on
the
left
here
with
the
activations
in
the
error,
but
you
just
can
swap
out
the
weights
matrix
transposed
for
a
just
completely
random,
matrix
b,
so
b,
this
capital
b
matrix
is
just
completely
random
and
this
will
still
work
as
long
as
you
are
careful
about
how
you
update
the
weights
of
your
matrix.
So
you
can
I'll
go
into
this
more
in
detail
on
the
on
the
next
slide.
A
The
reason
it's
called
feedback
alignment
is
because
the
if
you
are
careful
about
how
you
update
your
weights,
then
the
forward
weights
will
come
to
align
themselves
with
the
backwards
plates,
which
is
b,
and
that
would
be
that's
good,
because
then
you
can
actually
start
to
update
and
learn
using
these
random
matrices.
So
here's
a
visual
alex.
B
Yeah
yeah,
it's
a
question
with
that.
So
here
in
this
connect
in
this
scheme,
you
still
need
a
one-to-one
connection
like
every
forward
weight
still
needs
a
corresponding
backward
weight.
It's
just,
they
don't
have
the
the
two
weights.
No,
don't
need
to
be
the
same
right.
The
backward
weight
can
be
random,
but
at
least.
B
A
But
as
long
as
as
a
whole,
this
matrix
is
sort
of
pointing
in
the
same
direction
as
this
one.
Then
it
will
work
yeah,
so
then
yeah
the
reason
it's
called
feedback
alignment
is
because
you
want
to
sort
of
update
your
weights
such
that
these
get
closer
and
closer
together,
and
the
surprise
is
that
you
can
change
only
the
forward
weights
to
bring
them
into
alignment.
So
this
was
this
experiment
was
actually
done
as
a
control
for
a
different
experiment,
just
to
say,
okay.
This
was
definitely
not
going
to
work.
A
Therefore,
you
know
we
can
prove
that
the
other
method
will
work,
but
this
ended
up
working
because
you
can
change
these
forward,
not
w
transfers.
You
can
change
the
w
matrix
to
align
it
with
the
b
matrix
and
then
it
will
work
and
that's
supposed
to
be
sort
of
a
good
way
to
have
local
errors
propagate
without
needing
the
sort
of
biological
impossibility
of
point-to-point
forward
and
backward
connections.
E
That's
a
quick
question:
it
kind
of
looks
like
to
me
if
you
have
a
random,
sparse
matrix
being
the
backwards
matrix.
It
kind
of
implies
to
me
that
that
represents
a
subspace
and
that
somehow
only
the
elements
of
that
subspace
are
the
ones
that
need
to
be
brought
into
alignment
with
your
forward
direction.
E
So
I'm
I'm
just
wondering
you
know.
I.
I
understand
that
you're
showing
a
two-dimensional
representation
of
this,
but
it's
if
it's
a
high-order
space,
then
it's
almost
like
you're,
saying
I'm
trying
to
find
the
principle
component
and
that's
the
most
important
thing.
As
long
as
I
can
conspire
to
get
my
random
guys
to
more
or
less
be
aligned
with.
You
know
the
principal
component
of
what
I'm
trying
to
do.
I'm
okay,
is
that
a
fair
assessment.
A
A
Okay
yeah,
so
then
the
yeah
so,
as
I
said
before,
it
actually
worked,
which
is
pretty
interesting.
This
is
a
non-linear
problem
on
a
shallow
network.
So
it's
not
actually
that
difficult.
This
experiment,
I
don't
recall
exactly
what
the
data
set
was,
but
you
can
see
here
that
in
these
four
panels
the
sort
of
the
first
and
the
fourth
one
show
that
this
feedback
alignment,
which
is
in
green,
actually
approximates
backdrop
in
terms
of
its
performance.
A
I
wouldn't
pay
attention
to
the
fact
that
it's
better
here,
I
think,
that's
probably
just
noise,
but
the
point
is
that
it's
pretty
good
and
empirically
what
ends
up
happening.
Is
that
your
alignment,
this
this
b
panel
is
the
alignment
ends
up
going
and
converging
around
45
degrees
or
a
little
bit
less,
and
so,
ideally
it
would
be
zero
degrees,
but
it
kind
of
bottoms
out
at
some
non-zero
degree,
and
this
is
just
the
structure
of
the
network.
A
So
it's
only
a
few
layers,
deep,
which
is
sort
of
ominous
for
the
future
of
this
feedback
alignment,
because
this
is
kind
of
as
deep
as
it
will
pretty
much
get
the
another
cool
experiment.
That
was
done
with
the
feedback
alignment
algorithm.
Was
this
multi-layer
stochastic
mnist
network?
So
this
is
kind
of
very
reminiscent
of
a
spiking
neural
network
and
psp
is
post-synaptic
potential,
and
this
is
the
error
signal.
A
But
what
this
is
showing
is
that
you
can
use
this
feedback
alignment
on
a
completely
different
architecture
and
it
will
still
work
and
it's
it's
like
solid
in
theory,
and
it
can
be
used
on
like
again.
Mnist
is
actually
not
like
that
difficult
and
this
network
was
not
very
large,
but
even
still,
it
was
able
to
learn,
and
you
can
see
that
it's
classifying
these
numbers
correctly
on
the
right,
because
the
output
unit
is
a
six
and
this
is
a
six
on
the
input.
A
So
I
thought
that
was
pretty
cool
but,
as
I
said
before,
it
doesn't
scale
to
deep
networks.
So
this
is
on
the
safari
data
set
and
the
top
one
error
is
still
like
pretty
bad,
and
this
is
just
using
feedback
alignment
which
is
fa.
Bp
is
back
prob
and
then
so
what
what
ends
up
happening
is
that
modifications
have
to
be
made
for
feedback
alignment
to
work
on
deeper
architectures
and
so
there's
a
few
different
proposals.
A
That
goes
from
the
very
first
layer
to
the
very
last
layer,
and
so
there's
a
bunch
of
different
ways
to
sort
of
modify
this
another
one
is
just
uniform
sign,
concordant
feedback,
which
is
a
very
complicated
way
of
saying
that
you
just
set
b
to
be
the
sine
of
w
transpose.
This
should
be
transpose.
But
one
thing
that
kind
of
annoys
me
with
with
these
proposals
is
that
all
of
a
sudden
it
becomes
less
biologically
plausible.
A
So
if
you're,
you
know
converting
b
to
be
the
sine
of
w,
then
all
of
a
sudden
I
mean
propagating
or
copying
the
sign
between
the
b
matrix
and
the
w
transverse
matrix
is
kind
of
like
just
copying.
The
real
value,
there's
a
lot
of
information
is
in
the
sign
the
fact
that,
like
the
weight
exists,
even
well.
B
D
B
C
A
A
A
Okay,
so
that's
kind
of
it
for
the
first
method,
feedback,
alignment
and
one
thing.
Another
proposal
which
is
by
joshua
benjio,
was
that
you
can
swap
the
errors
for
targets
with
different
target
propagation.
So
to
put
a
visual
to
what
I'm
saying
here,
here's
your
your
network
and
traditionally
you'd
have
in
feedback
alignment.
We
try
to
approximate
this
w
transpose
matrix
with
b
and
what's
propagated
back
is
the
errors,
but
if
you
swap
the
errors
for
targets
and
propagate
back
the
targets,
the
formulation
comes
different.
A
That
comes
along
with
this,
where
you
have
a
perturbation
phase
and
you
propagate
forward,
and
you
try
to
make
sure
that
when
you
reconstruct
it
it's
similar,
even
though
it
was
perturbed,
and
then
you
have
this
reconstruction
loss.
So
it's
very
much
like
an
autoencoder
in
order
to
get
to
a
working
difference,
target
propagation
rule,
and
recently
this
is
actually
in
2020.
A
A
So
there's
you
know
second
order
methods
that
work.
You
know
if
you
had
first
order.
Second
order.
Third
order,
approximations
to
the
weight
updates,
then
that's
even
better,
and
so
it
doesn't
we're.
Not
all
I.
What
am
I
trying
to
say
it
doesn't
always
have
to
be
just
a
single
first
order,
gradient
descent,
and
so
this
was
a
nice
way
to
reconcile
that
with
another
sort
of
local
learning
rule.
A
B
C
C
A
Steps,
because
what's
going
to
have
to
happen,
is
that
this
loss
is
going
to
have
to
propagate
to
the
layer
backward
one
time
step
later
and
then
the
next
time
set
it
will
get.
You
know
one
layer
backward
and
it'll
keep
progressing
backwards,
as
you
have
more
and
more
backwards
passes
or
more
and
more
weight
updates.
I
should
say-
and
so
it
takes,
you
know
three
updates
for
the
loss
to
get
back
to
layer,
one
as
opposed
to
with
back
crop.
It
just
takes
one.
A
You
just
go
straight
through,
so
why
would
you
do
this?
The
reason
why
they
did
it
is
because
they
wanted
to
separate
out
each
layer
and
their
gradient
updates
into
like
individual
modules.
So
what
this
allows
you
to
do
is
run
this
whole
thing
in
parallel,
even
though
it's
a
feed-forward
network
and
then
you're
simultaneously
training
this
network,
which
is
the
gradients
actually
there's
one
of
these
boxes,
this
gradient
network
for
each
layer
or
there
can
be
there
can
be.
You
know
arbitrarily
many.
A
A
But
if
you
do
that,
then
you
can
start
separating
out
into
different
layers,
so
activations
come
in
gradients
go
out
and
the
way
that
you
train
this
network
is
by
getting
the
gradients
from
the
layer.
Above
so
you
get
the
gradients
from
the
layer
above
you
back
propagate
through
layer
two
and
then
you
can
train
the
synthetic
gradient
network
from
the
layer
below
and
that's
why
it
takes
a
lot
of
time.
So
you
gotta
propagate
all
the
way
back
by
training
all
these
networks
in
succession.
A
A
I
think
it's
like
the
reconstruction
loss
with
respect
yeah.
It's
just
a
reconstruction
loss
with
respect
to
the
gradients
that
so
like,
for
example,
this
one
it
gets
activations
in
and
it
gives
it
a
gradient
out.
But
then
it
figures
out.
Okay,
what's
the
gradient,
I
actually
should
have
given
layer
one
from
layer
two
and
so
then
it's
reconstruction
loss.
I
think
it's
like
the
squared
error
between
you
know
what
it
predicted
and
what
was
the
actual,
and
so
then
you
can
back
propagate
through
that
network.
D
C
A
A
What
would
have
been
back
propagation,
and
so
this
was
kind
of
proven
out
in
a
theory
paper,
and
then
there
are
simple
examples
that
have
been
sort
of
tested
as
well
with
this
sort
of
structure,
but
a
lot.
So
there's
there's
been
a
few
like
major
criticisms
of
this
work,
and
that's
that
you
have
to
have
error,
nodes
and
value
nodes,
and
they
must
correspond
like
one
to
one
right
so
for
every
error:
node,
you
have
a
value
node
and
then
they
must
be
connected
as
well,
and
that
is
somewhat
implausible.
A
So
the
good
news
is
that
there's
actually
been
even
more
like
further
progress
and
that
assumption
has
been
removed.
You
don't
have
to
have
exactly
one-to-one
correspondence
between
error,
nodes
and
value
nodes.
You
can
also
relax
the
weight
transport
assumption
here
with
something
similar
to
feedback
alignment
and
then
there's
other
work.
That's
extended
the
predictive
coding
networks
that
map
to
like
dendrites.
So
this
is
like
giving
a
one
of
these
predictive
coding.
A
Networks
can
be
formulated
as
being
implemented
by
dendrites
on
different
neurons,
which
is
interesting
and
then
the
yeah,
and
then
it's
worth
noting
that
all
this
predictive
coding
stuff
is
happening
on
hebian
learning
rules.
So
the
weight
update
is
a
function
purely
a
function
of
pre
and
postsynaptic.
A
So
we've
talked
about
various
different
learning
rules,
but
one
thing
that
you
can
do-
and
that
is
this
is
work
that
just
came
out
recently,
so
you
can
infer
learning
rules
from
the
weight
updates.
So
let
me
explain
a
little
bit
more
here.
What
you
have
is
a
bunch
of
different
networks
and
a
bunch
of
different
tasks
with
a
lot
of
different
data
sets
and
you
train
those
networks
using
various
different
sort.
Optimizers
right,
so
you've
got
atom.
A
So
you
can
find
out
what
optimizer
is
being
used
just
by
looking
at
the
trajectory
that
the
weight
updates
take,
which
is
really
cool,
and
that's
that's
what
this
is
here.
So
with
99
accuracy,
you
can
classify.
You
know
when,
when
a
neural
network
is
being
trained
with
atom,
when
it's
being
trained
with
sgd
and
information
alignment
or
information
element
and
feedback,
climate
yeah.
G
G
A
So
yeah
all
there's
the
the
next
few
slides,
I
think,
are
a
little
bit
more
useful
in
terms
of
like
actually
why
you'd
want
to
to
infer
a
learning
rule.
This
is
so
so
I
guess
I'm
setting
this
up
to
say
that
you
can
classify
which
learning
rule
is
being
used,
but
in
later
in
the
next
few
papers
that
I'm
going
to
show
you
can
actually
generate
optimizers
right.
You
can.
You
cannot
just
say
which,
which
one
is
which,
but
you
can
generate
a
new
optimizer.
Okay,.
A
Yeah
yeah,
but
this
is
to
show
that
it's
definitely
possible
to
do
this
so
yeah,
here's
here's
kind
of
what
I
was
hoping
to
get
to
and
that's
finding
new
learning
rules
and
there's
a
bunch
of
different
ways
that
people
have
done
this,
and
so
this
paper
is
kind
of
a
fun
title.
It's
learning
to
learn
by
gradient
descent
by
gradient
descent
and
what
what
this
is.
Is
you
take
your
standard
weight
update,
so
you
say:
well
the
weights
at
you
know,
time
step.
A
T
plus
one
are
going
to
be
equal
to
the
weights
of
time,
step
t
and
then
you
change
them
by
a
learning
rate.
And
then
you
take
the
gradient
of
f
of
theta,
of
t
where
f
is
your
network,
so
your
network
is
parametrized
by
theta
and
just
look
at
this
is
sort
of
the
standard
gradient
descent
rule,
but
instead
of
using
regular
gradient
descent
rules,
this
paper
argues
that
we
should
actually
learn
an
optimizer.
A
We
should
say:
okay,
we
know
what
we're
supposed
to
do,
what
the
gradients
of
these
weights
are,
but
we're
going
to
take
that
and
just
throw
it
into
another
network.
Given
parameters
phi
and
that
will
tell
us
how
to
update
these
these
weights
so
yeah,
it's
basically
just
saying
this
shouldn't
be
just
a
linear
update.
We
shouldn't
just
go
directly
along
the
gradient.
We
should
plug
it
into
another
function
and
then
learn
this
function
and
then
the
way
that
they
learn
it
learn.
A
I
do
not
know
how
it
scales
to
larger
networks.
I
suspect
it
probably
doesn't
work
super
well,
but
I
mean
that's
just
a
suspicion.
It
could
work
really
well
and
it
actually
outperforms
the
other
optimizers
on
on
this
data
set
cool.
So
then
this
is
the
topic
that
I
presented
in
the
deep
dive
a
couple
weeks
ago
and
it's
the
idea
of
looking
for
new
learning
rules
based
purely
on
local
data
right.
So
this
is.
This
is
how
you
characterize
this
function.
Delta.
A
W
is
how
you
change
your
weight
and
that's
the
weight
from
you
know:
neuron
I
to
neuron
j
and
it's
a
function
of
the
local
variables
of
the
at
synapse
I
to
j,
and
so
maybe
some
global
variables,
theta
and
the
way
this
is
done
in
this
paper
is
actually
by
genetic
algorithm
and
there's
all
sorts
of
different
ways
that
they
parameterize
this
function
here.
This
function
delta
w.
This
is
something
actually
it's.
A
The
idea
of
genetic
programming
versus
genetic
algorithms,
but
then
there's
also
just
regular
back
propagation,
as
you
can
see
that,
as
you
add
more
tasks
during
optimization,
then
your
optimizers
get
better
and
better,
and
so
then
they
start
to
approximate
back
propagation
or
they
don't
approximate
back
propagation.
They
approximate
the
performance
of
back
propagation
as
you
start
to
look
at
many
different
tasks.
A
A
I
think
these
are
in
2000,
so
just
after
I
was
born
but
yeah,
so
then
people
have
tried
other
other
ways
of
finding
new
learning
rules.
Usually
they
involve
some
sort
of
genetic
algorithm
because
it's
somewhat
complicated
to
try
to
learn
a
learning
rule,
but
I
mean
I
I
I
definitely
think
you
can
you
can
back
propagate
through
it.
It's
just
somewhat
complicated
and
also
these
were
done
before,
like
autograd.
A
Maybe
the
top
10
percent
of
the
previous
generation
of
individuals
and
it's
just
a
combination
of
all
their
traits
and
the
traits
can
be
like.
You
know
the
parameters
given
the
parameters
of
a
function
right,
so
they
can
be
the
parameters
of
a
neural
network.
They
can
be
parameters
of
like
a
parametric
function.
You
know
the
weights
of
the
weights
of
several
basis
functions.
These
are
all
things
that
I
think
in
this
and
this
one
they
did.
Actually
it
was
this
one
on
the
right.
A
But
it's
very
computationally
intensive
because
you
have
to
compute
each
like
you
have
to
compute
each
individual
fitness
for
every
generation
and
they
have
to
do
many
generations,
so
it
takes.
It
takes
a
long
time
before
it
to
converge,
but
yeah.
So
what's
interesting
here
is
that
they
actually
took
the
evolved.
A
Yes,
so
they
evolved
learning
roles
and
so
the
delta
rule
is,
I
think,
the
regular
sort
of
heavy
learning
rule
that
would
have
been
just
like
yeah,
just
a
very
simple
heavy
learning
rule
and
then
the
evolved
learning
rule
is
what
they've
learned
and
there's
something
somewhat
different.
I
think
they
used.
A
I
think
this
is
a
parametric
function.
It's
come
out
very
smooth.
So
that's
why
I'm
saying
that,
but
it's
a
function
of
the
pre
and
post
synaptic
activity
and
they
get
this
interesting
sort
of
landscape
for
weight,
updates
that
are
dependent
on
the
pre
and
post
synaptic
activity.
And
you
can
see
here-
this
is
very
small,
but
the
a
and
the
f
rule
are
both
different
weight,
update
rules
and
the
light
dotted
line
up
here
is
the
delta
rule,
which
is
just
a
given.
A
Like
heavy
learning
rule,
and
so
what
they're
showing
here
is
that
they
can
actually
learn
better
learning
rules
by
using
genetic
algorithms,
and
that's
that
work
is
continued
over
here
which
references
this
paper,
but
they
do
both
a
just
parameterized,
heavy
and
learning
rule
which
is
given
by
these
five
parameters.
Abcd
and
then
this
data,
I
think
it's
called,
and
then
they
do
another
weight,
update
rule
used
using
a
neural
network
for
comparison
and
then
that
that
also
is
trained.
A
I
think
using
a
genetic
algorithm-
and
this
is
this-
is
trained
on
a
task
where
I
think
an
agent
is
running
around
collecting
food.
So
it's
a
little
bit
different
than
something
like
a
supervised
learning
task.
It's
not
actually
reinforcement
learning,
I
don't
think,
but
what
they
show
is
that
you
can
sort
of
update
these
weights.
I
I
think
this
is
the
equivalent
of
a
policy
network,
so
these
weight
updates
are
used
on
a
policy
network,
but
it's
not
all
the
bells
and
whistles
of
rl
and
yeah.
A
A
No,
this
is
this
is
no
learning
the
blue,
so
the
heavy
layering
will
actually
converge
as
to
what
no
learning
would
have
been
after
2.5
times
10
to
the
fourth
time
steps,
and
then
this
neural
network
weight
updates
gets
better
over
time.
So
they
argue
that
this
is
just
a
favorable
way
of
doing
it,
instead
of
keeping
it
in
this
simple
parameterized
format.
So
it's
just
more
complex
than
this
one
over
here.
A
Okay,
yeah,
so
that
that's
kind
of
the
the
main
papers
that
I've
looked
at
and
tried
to
give
the
high
level
summary
of
there's
just
a
ton
of
great
research
out
there,
and
I
had
to
leave
out
a
bunch
of
it,
and
so
that's
that's.
That's
always
pretty
sad
and
then
another
concern
that
I
have
is
that,
as
we
stray
further
from
biological
architecture,
it
might
be
harder
and
harder
to
find
simple
local
learning
rules.
A
So,
for
example,
these
are
the
some
of
the
questions
that
I
think
are
you
know
pertinent
here
is
that
do
we
need
multiple
learning
rules
for
different
structures
right?
Does
it
make
sense
that
we
just
have
one
learning
rule
that's
applied
to
every
single
weight
in
a
neural
network?
Maybe
not
you
know
just
adding
grid
cell
modules
or
more
complicated
structures
that
you
might
find
in
the
neocortex
change.
The
way
that
we
want
to
use
these
learning
rules.
A
Maybe
certain
structures
require
some
learning
rules
and
certain
other
structures
require
other
learning
rules
and
maybe
sparsity
matters
right.
So
maybe
these
learning
rules
don't
make
sense
unless
you're
really
close
to
biology
and
in
biology.
There's
you
know
tremendous
amounts
of
sparsity,
so
this
is
sort
of
a
hypothesis
of
mine
is,
as
you
get
closer
and
closer
to
biology,
the
learning
rules
will
get
easier
and
easier
to
find,
and
maybe
your
genetic
algorithms
will
converge
faster.
A
This
is
just
kind
of
a
general
concern.
After
reading
a
bunch
of
these
papers
is
that,
whenever
a
biologically
biologically
plausible
local
learning
rules
proposed
the
authors,
you
know
prove
that
it
can
work
in
theory
and
then
they
just
go
ahead
and
violate
it
to
get
good
results
on
some
sort
of
testing
data
set,
and
so
this
is
just
a
general
problem
with
ml
research.
Is
that
everyone's
trying
to
get
to
you
know
really
good
performance
on
standard
standard
benchmarks?
G
G
G
Biological
networks
are
very
structured
and
complex
compared
to
typical
deep
learning
networks,
and
so
we
have
all
these
different
types
of
inhibitory,
neurons
that
have
different
rules
at
which
they
operate
and
they
force
they
force,
in
some
sense
to
structure
the
networks
and
the
types
of
different
types
of
neurons
forced
things
to
happen
that
you
don't
get.
If
you
don't
have
that,
you
know,
if
you
just
have
a
multi-layered
network,
you
don't
have
any
of
that
enforced
rules.
G
So
I
think
your
last
point
there
is
we're.
The
current
systems
are
so
far
from
biology
that
it's
not
clear
how
much
you
can
bring.
Those
like
a
local
thing,
like
a
local
learning
role
into
the
picture,
and
maybe
you
can
maybe
can't
I
don't
know,
but
it
wouldn't
be
surprising
that
you
know
yeah
you're
gonna
run
into
limits,
because
this
network
itself
is
just
totally
different.
A
Yeah
yeah,
I
totally
agree,
I
think
the
structure
of
the
network-
probably
you
know,
matters
a
lot,
especially
considering
that
like
for
for
some
of
these
things,
like
the
predictive
coding,
you
need
to
actually
modify
the
structure
of
the
network
for
a
learning
world
to
make
sense
so
yeah
I
mean
I
wouldn't
be
surprised
if
that
was
true
for
a
lot
of
different
cases
where
your
learning
rule
is
just
not
going
to
work.
Unless
you
have
a
very
specific
kind
of
structure.
G
Yeah
I
mean
we're
talking
about
like
the
role
of
the
thalamus.
Now
is
this
reference
rate
transforms
and
so
on.
So
that's
a
whole
part
of
the
system
that
it's
just
missing
completely.
You
know
even
the
way
grid
cells
come
about.
You
know
the
good
cells
are
the
the
neurons
in
the
grid.
Cells
may
have
some,
they
may
have
very
typical
learning
roles
or
but
there's
other
structure
that
make
them
into
grid
cells.
You
know
there
are
there's
differentiation
in
the
cell
types
in
these
networks.
G
So
it's
it's
an
interesting
question
like
how
far
can
you
go
applying
biological
rules
to
these
networks
that
are
not
biological
and
you
know,
I
think,
you've
shown
so
far
with
sparsity.
You
can
do
some
pretty
cool
things,
so
that's
good
and
I
think
we're
I
think
you
know
we
believe
we
can
get
a
bunch
of
other
stuff
going
on
with
the
dendrites
and
reference
rate,
but.
G
G
B
B
C
A
Yeah,
I
did
that's.
That's
definitely
part
of
the.
You
know
the
corpus,
research
that
I
had
to
leave
out,
but
there's
yeah
one
one
way
that
a
lot
of
people
try
to
tackle.
That
problem
is
with
with
this
global
neural
modulator.
So
if
you
have,
you
know,
if
you
have
a
network
or
even
something
like
learning
to
learn
by
gradient
descent,
then,
like
that
that
I
can
go
back
here
like
this
phi
might
include
some
sort
of
global
modulator,
and
it
might
say
you
know
what
every
weight
update
is.
A
A
What
I
haven't
seen
is
like
a
a
big
distinction
between
like
kinds
of
weights.
It's
usually
something
like
this,
where
it
usually
is
like
a
global
modulator,
as
opposed
to
you
know
the
distinction
between
like
fast
and
slow
weights
or
like
different
kinds
of
weights
and
different
update
rules.
It's
usually
all
kind
of
shoehorned
into
one
one
like
big
function,.
B
Yeah,
I'm
just
thinking
about
like
our
dendritic
networks,
you
have
the
feed
forward
weights
and
then
you
have
the
context,
weights
or
contacts
and
those
might
have
to
evolve
at
different
time
scales.
I
think
we
did
that
even
in
our
htm
work,
you
know
the
assumption
was
the
spatial
pooler
stuff
was
sort
of
slower
in
some
sense.
B
C
C
H
You
said
on
your
very
last
slide:
one
of
your
points
was
how
you
know
it
doesn't
make
sense
to
apply
some
of
these
learning
rules,
because
we
don't
have
the
right
structure.
Did
you
come
across
any
work
where
they're
trying
to
improve
the
structural
aspects
so
that
these
learning
or
that,
so
that
more
biologically
plausible
learning
rules
couldn't
be
applied.
A
That's
a
good
question.
I
think
the
way
that
I
went
about
this
was
by
looking
along
a
trail
of
papers
that
were
all
very
related
to
each
other
without
kind
of
yeah.
I
I
think
there's
got
to
be
something
that
that
does
try
to
answer
that
question
with
different
structures,
but
I
guess,
given
the
way
that
I
went
doing
this
literature
review,
I
didn't
come
across
any.
A
F
Or
sorry,
did
I
cut
someone
off?
No,
you
go
ahead,
oh
yeah,
so
one
question
I
had
was
like.
I
feel
like
deep
learning,
architectures
nowadays
kind
of
structure
the
network,
so
it
gives
the
possibility
for
something
to
happen
so,
for
example,
like
transformers
it
gives
the
network
the
chance
to
like
wait
different
parts
of
like
a
sequence,
and
then
it
kind
of
hopes
that
through
gradient
descent,
some
it
happens
upon
the
solution
that
we
think
of
you
think
or
do
any
of
the
papers.
A
Not
not
exactly,
I
think,
a
lot
of
the
papers
that
I
were
reading
I
was
reading
were
more
general
and
trying
to
tackle
like
does
this
learning
really
make
sense?
Can
it
learn
in
simple
scenarios,
and
the
problem
with
a
lot
of
these
things
is
that
it
doesn't
scale
super
well
right,
let's
scale
to
like
I,
I
haven't
seen
anything
with
specifically
like
transformers
and
local
learning
rules,
or
maybe,
like
you
know,
another
example
might
be
like
these
memory,
augmented
neural
networks
and
local
learning
rules.
A
Maybe
you
just
add
that
in
there,
but
I
haven't
seen
yeah,
I
I
haven't
seen
anything
that
has
to
do
with
like
trying
to
prove
that
certain
behaviors
will
exist.
Given
a
certain
update
rule,
it's
kind
of
been
like.
Can
we
make
a
general
update
role
or
a
general
optimizer
that
approximates
backdrop
or
does
like
better
than
backcrop
in
some
way,
as
opposed
to
like
for
specific
structures?
If
that
makes
sense,.
F
G
E
E
Was
there
anything
that
actually
looked
a
sampling
upon
connections,
I'm
thinking
of
some
of
the
the
architectures
that
we've
seen
in
the
past
that
were
performant
for
certain
applications
like
u-nets
and
things
like
that?
Where
you
skip
over
some
layers
and
connect
some
things
I
mean
it
would,
I
would
think
the
learning
rule
for
that
would
be
very
slow
because
there's
a
if
you
start
forging
connections
beyond
your
local
layer,
there's
a
vast
universe
to
explore.
But
I'm
curious.
A
E
I
I
would,
I
would
say
either
in
in
the
sense
that
I
mean
you
kind
of
stayed
formally
in
the
beginning,
that
with
stochastic
gradient
descent.
Typically,
if
the
forward
network
in
the
back
network
had
to
have
symmetric
weights
or
synaptic
connectivity,
but
I'm
just
trying
to
think
if
there
is
a
rationale
for
saying
you
know
taking
samples
outside
of
your
own
layer
and
skipping
layers
and
doing
forging
those
connections,
whether
it's
in
the
forward
or
backwards
network.
Whether
that's
something
that's
explorable
or
someone's.
A
Okay,
I
think
I
see
what
you're
saying
I'll
answer
the
other
question
as
well,
because
I
think
it's
somewhat
important
is
that
with
the
update
network
like
a
lot
of
times
or
from
what
I've
seen
so
far,
these
genetic
algorithms
can't
really
do
a
lot,
because
you
have
to
learn
you're
learning
two
networks
at
once,
and
this
network
is
really
small.
So
this
is
this.
Network
architecture
is
usually
fixed
and
you're.
Just
like
learning.
A
You
know
a
very
small
number
of
weights
at
any
given
time,
but
with
your
yeah
with
your
main
question,
I
think
yeah,
one
thing
that
that
somewhat,
I
think,
relates
to
what
you're
talking
about.
Is
this
the
modifications
to
feedback
alignment?
There
have
been
a
bunch
and
like
with
the
direct
feedback
climate
there's
that
backwards
weights
going
from
the
from
the
last
layer
to
every
individual
layer,
but
then
there's
all
sorts
of
combinations
where
you
have
weights
going
from
like
skipping
layers,
like
you're
saying
you
have
like
this
b
matrix.
A
E
Right
so
so
one
of
the
you
know,
early
examples
of
genetic
networks
is
when
they
basically
hooking
part
of
the
genetic
code
was
not
just
set
of
parameters
but
sub
graphs
that
got
you
know,
kind
of
became
components
and
they
were
kind
of
hooked
together
somewhat
randomly
so
exploring
that
kind
of
space.
I
was
just
trying
to.
I
don't
know
what
the
biological
plausibility
of
this
is,
except
to
maybe
find
something
out,
search
a
larger
space
of
possible
connectivities.
E
You
know,
would
you
you
know
from
one
of
these
algorithms
come
up
with
that
kind
of
connectivity
or
was
it
you
know
we're
trying
to
abstract
away
something?
That's
the
at
least
the
neocortex
found
to
be
essential
in
in
performance,
so
some
sometimes
having
that
kind
of
exploration
structure
allows
you
to
get
outside
of
your
mindset
of
extreme
symmetry
in
these
things,
and
you
know.
G
G
Maybe
you
could,
although
you
know
it's,
how
we
got
here
is
a
very
complex
path
and
we
have
biological
needs
of
survival
and
right
all
kinds
of
things
that
that,
but
I
wouldn't
put
it
out
of
the
question
it
just
hasn't
been
fruitful
yet
and
most
indicate.
You
know
the
whole
thing
about
the
genetic
algorithms.
G
G
It'd
be
sort
of
like
saying
you
know
I
I
need
to
learn
how
to
write
and
I'm
gonna.
I'm
gonna
start
with
the
set
of
words,
but
maybe
that's
not
the
right
thing.
Maybe
you
need
to
start
with
a
set
of
letters
or
you
know
a
set
of
phone
names
or
something
like
that,
and
you
know
that's
a
very
it's
poor
analogy,
but.
E
G
Yeah
and
again
we're
starting
with
this
genetic
code,
which
is
incredibly
powerful
in
the
number
of
manifestations
it
can
make,
but
we
don't
actually
know
enough
about
it
to
know
you
know
what
what
can
it
represent?
What
can
what
kind
of
things
can
the
genetic
code
represent?
Could
it
produce
other
animals
there's
a
huge
gap
there,
which
may
be
necessary?
The
the
the
explanation
between
going
from
genes
to
you
know
phenotype
so
large
that
anyway,
it's,
I
think
xanax
albums
are
cool,
but
so
far
people
have
tried
to
sort
of
evolve
brains.
E
Yeah,
well,
oh
yeah,
I'm
just
you
know
it's
it's!
The
the
the
current
progress
I
I
think
is
is
conceptually
limited,
it's
kind
of
reminds
me
of
the
the
early
days
when
they
were
trying
to
do
voice,
basically
audio
to
to
text
and
kind
of
gave
up,
trying
to
make
the
models
of
the
vocal
cords
and
stuff
like
that.
Just
went
to
pure
machine
learning,
algorithms
and
those
proved
out
to
be
much
much
better
and
at
a
meta
level.
E
I
I
kind
of
feel
we're
at
the
same
space
here
for
machine
learning
where
we're
trying
to
say
well
what
what
would
look
symmetrically
good?
What
would
be
you
know
functionally
good
and
it's
the
in
some
aspects:
some
genetic
algorithms
allow
you
to
break
out
of
that
and
yeah
and
try
things
you
wouldn't
conceive
of.
A
A
C
A
E
G
If
you
look
at
the
brain
yeah,
it
is
fair,
but
if
you
look
at
the
brain
I
would
say:
there's
a
hell
of
a
lot
more
a
design
into
the
architecture
than
into
the
learning
rules.
There'll
be
just
a
handful
of
learning
rules
in
the
brain,
but
there's
a
hell
of
a
lot
of
architecture.
That's
going
on,
and
so
that's
where
most
of
the
evolution
has
been
occurring.
And
again
you
see
that
true.
Just
look
at
the
you
know
our
theory
about
grid
cells
and
place
cells
in
the
cortex
right.
G
So
you
know
the
learning
rule
is
almost
certainly
going
to
be
very
similar
to
what
existed
in
hippocampus
and
toronto
cortex.
But
you
know
the
learning
that
evolved,
that
made
mammals
with
our
neocortex
and
was
you
know
taking
those
arc,
re
copping,
those
architectures
and
acquiring
them
together
differently
and
adding
them
at
different
sort
of
levels?
And,
and
so
it
was,
it
was
all
about.
You
know,
mammals
are
not
new
learning
rules.
G
Mammals
are
all
about
architectural
changes
and
our
intelligence
was
really
evolved
in
a
way
that
popped
up
almost
exclusively
almost
certainly
vast
90
over
99.
If
it's
going
to
be
architectural
changes
versus
changes
in
learning
rules,
so
if
you
think
about
it
that
way,
it's
a
good
way
to
show
the
limits.
A
A
Yeah
yeah,
I
did
see
some
stuff
there
and
specifically,
I
saw
some
papers
on
sparsity
and
feedback
alignment
and
I
was
hoping
to
see
some
results
that
were
like
sparsity
enables
feedback
alignment
to
be
really
good,
but
it
looked
like
it
was
the
the
reason
they
did.
It
was
mainly
for
the
computational
speed
up
and
actually
ended
up
performing
like
slightly
worse,
but
then
you
know
they
could
do
more
iterations
in
the
same
period
of
time,
so
they
were
able
to
justify
that.
B
A
D
C
D
D
D
A
This
this,
like
this
one,
was
trained
on
a
very
simple
task
with
a
very
shallow
network,
and
then
I
think
these
learning
rules
ended
up
being
a
little
bit
more
complicated,
not
anything
like
super
complicated,
but
this
would
have
been.
I
I
forget
the
example,
but
it
was.
It
was
basically
like
you
know
two
variable
edition
or
something
like
just
a
toy
problem
or
like
xor
or
something,
and
then
this
was
actually
a
data
set
of
a
very
simple
yeah.
It
was
a
very
simple
data
set,
but
yeah.