►
Description
The neurons we use in today’s deep learning systems are extremely simple compared to their biological counterparts. In this meeting, guest speaker Toviah Moldwin explored the abilities of a single biological neuron with extended dendrites and synapses and introduced a potential biological model of plasticity that might enable better learning in neural networks.
Work done by: Toldvin Moldwin, Menachem Kalmenson, Li Shay Azran, and Idan Segev from the Edmond and Lily Safra Center for Brain Sciences (ELSC) at the Hebrew University of Jerusalem
Topics covered & related papers:
1/ Biophysical Perceptron
➤ “Perceptron Learning and Classification in a Modeled Cortical Pyramidal Cell”: https://www.frontiersin.org/articles/10.3389/fncom.2020.00033/full
2/ The Gradient Clusteron (G-Clusteron)
➤ “The gradient clusteron: A model neuron that learns to solve classification tasks via dendritic nonlinearities, structural plasticity, and gradient descent”: https://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1009015
3/ Fixed Point - Learning Rate (FPLR) Framework for Calcium
4/ The Calcitron
5/ Hierarchical Heterosynaptic Plasticity
➤ "Asymmetric voltage attenuation in dendrites enables hierarchical heterosynaptic plasticity”: https://www.biorxiv.org/content/10.1101/2022.07.07.499166v1
- - - - -
Numenta has developed breakthrough advances in AI technology that enable customers to achieve 10-100X improvement in performance across broad use cases, such as natural language processing and computer vision. Backed by two decades of neuroscience research, we developed a framework for intelligence called The Thousand Brains Theory. By leveraging these discoveries and applying them to AI systems, we’re able to deliver extreme performance improvements and unlock new capabilities.
Subscribe to our News Digest for the latest news about neuroscience and artificial intelligence:
https://numenta.com/news-digest/
Subscribe to our Newsletter for the latest Numenta updates:
https://tinyurl.com/NumentaNewsletter
Our Social Media:
https://twitter.com/Numenta
https://www.facebook.com/OfficialNumenta
https://www.linkedin.com/company/numenta
Our Open Source Resources:
https://github.com/numenta
https://discourse.numenta.org/
Our Website:
https://numenta.com/
A
When
I
am
a
PhD
student
in
the
lab
of
Edon,
segev
I've
been
here
like
forever
yeah
I'm
at
Hebrew,
u
in
Jerusalem,
and
so
this
is
basically
going
to
be
a
presentation
of
all
of
my
dissertation
work,
which
is
it's
basically
it's
five
papers.
This
is
the
first
time
I've
presented
everything
at
the
same
time,
so
this
is
a
little
a
little
bit
of
a
dry
run
for
my
dissertation
defense,
which
is
not
yet
scheduled.
A
So
it's
a
lot
of
material
feel
free
to
interrupt
me
in
in
the
middle
line,
is
really
I'm
very
used
to
that.
But
with
the
caveat
that
I
recorded
myself
giving
this
talk-
and
it
takes
me
47
minutes
and
10
seconds
without
interruptions.
So
but
with
that
in
mind,
feel
free
to
interrupt
me
if
you
don't
understand
something
right.
So
let's
see
how
I
change!
Oh,
let's
go
here.
A
Okay,
so
I
am
interested
in
a
similar
question
to
what
I
think
you
at
numantha
are
all
interested
in
is
that
we
have
these
linear
neurons,
that
that
people
use
in
machine
learning
and
artificial
neural
networks,
and
you
have
your
inputs.
Those
are
the
X's
each.
You
multiply
each
input
by
a
synaptic
weight.
Those
are
the
W's.
You
sum
up
the
weighted
inputs.
A
You
pass
them
through
a
threshold
function
and
the
the
neuron
returns
either
a
spike
or
no
Spike,
and
you
put
together
a
big
networks
of
these
neurons
and
that's
how
you
get
deep
neural
network
Nets
and
these
neurons
were
inspired
by
real
biological
neurons
in
the
brain.
So
this
these
are
drawings
by
Santiago,
Ramon
neuroanatomist,
one
of
the
fathers
of
Neuroscience.
These
are
pyramidal
neurons,
as
you
can
see
the
basic
structures
here
of
of
the
neurons,
so
you
have
the
dendrites
and
pyramidal
neurons.
A
You
have
two
regions
of
dendrites,
so
dendrites
are
the
input
regions
and
those
are
those
branches
that
you
see
here
so
this
region.
These
are
the
basal
dendrites
up
here,
would
be
the
apical
dendrites,
unless
this
is
hippocampus
in
which
it's
the
opposite
known,
but
these
are
basal
dendrites.
This
is
the
Soma,
that's
the
cell
body
and
I
should
say
also
so
on
the
dendrites.
You
see
these
little
bumps,
which
are
the
spine.
A
So
that's
where
other
neurons
synapse
onto
the
dendrite,
that's
where
other
neurons
make
their
connections,
and
you
have
this
thin
line
here,
marked
with
an
a
those
are
the
axons?
That's
the
output
region
of
the
neuron,
and
so
real
neurons
have
all
these
these
complexities
that
are
not
necessarily
captured
by
this
linear
Neuron
model
of
something
that
just
takes
the
sum
of
the
weighted
inputs
in
terms
of
where
I
fit
into
this
picture
of
the
question
of
neural
computation.
A
A
A
We
use
data
from
electrophysiological
experiments
to
reproduce
the
the
electrical
properties
of
the
neuron
I
personally
am
not
so
involved
in
characterizing,
neurons,
I'm,
more
interested
in
plasticity
and
learning,
and
using
these
kinds
of
neuron
models
to
explore
plasticity
and
learning
and
I
should
say
a
lot
of
the
things
that
I
do
are
not
necessarily
using
these
cable
models
but
using
other
kinds
of
models.
So
I
work
at
a
variety,
various
different
levels
of
abstraction,
so
the
first
oops,
let's
go
back
here,
the
first.
A
The
first
paper
in
in
this
dissertation
is
called
perceptron
learning
and
classification
in
a
modeled
cortical
pyramidal
cell.
The
idea
is
that
well,
first
of
all
so
I'm
sure
you
guys
know
this,
but
so
neurons.
This
linear
Neuron
model
can
be
thought
of
as
a
as
a
classifier.
So
if
you
have
two
classes
of
objects,
say
cats
and
dogs
to
use
the
famous
example
that
I
hate,
but
you
characterize
your
animals,
your
your
objects
by
certain
quantitative
features.
So
in
this
case
it's
domestication
and
size,
and
you
can.
A
You
can
then
plot
those
objects
in
an
on
an
X,
Y
plane
or
usually
it's
much
higher
dimensional
and
what
the
perceptron
does
and
what
what
a
linear
neuron
does.
We
also
call
it
a
perceptron.
Is
it
basically
draws
a
line
in
between
these
two
classes
of
objects,
because
the
equation
for
the
linear
neuron
is
actually
basically
the
equation
for
a
line
in
high
Dimensions.
It's
a
hyperplane
and
there's
something
called
The
perceptron
Learning
algorithm,
which
basically
tells
the
neuron.
A
B
A
Okay,
so
the
idea
of
the
biophysical
perceptron
is
that
just
we're
going
to
take
this
this
algorithm,
as
is
so,
we
modify
the
the
synaptic
weights
proportional
to
the
inputs
and
we're
just
going
to
put
it
on
one
of
these
detailed
biological
models
of
a
neuron
no
I
did
not
I
did
not
create
this
neuron.
This
is
a
neuron
that
was
created
by
a
previous
PhD
student
in
the
lab,
itai
High,
and
he
he
was
really
just
interested
in
having
a
good
model
of
a
neuron.
So
he
wasn't.
A
This
was
not
developed
to
do
a
machine
learning
task.
I
just
took
his
neuron
and
was
like:
let's
see
if
we
can,
we
can
run
the
perceptron
learning
algorithm
on
it.
So
what
we
did
was
we
presented
these
binary
patterns
to
the
neurons,
so
there
were,
we
had
each
pattern.
Each
each
input,
Vector
is,
is
a
thousand
synapses
large
and
and
each
of
those
input
vectors
you
randomly
have
200
synapses
that
are
that
are
active,
that
are
one
and
the
rest.
A
The
other
800
would
be
zeros,
and
this
is
we're
interested
in
characterizing.
The
capacity
of
the
neurons
we're
not
doing
we're
not
doing
like
a
a
benchmark
like
mnist
or
something
like
that.
We're
interested
in
is
classification
capacity.
Classification
capacity
is
the
idea
that,
if
you,
if
you
just
randomly
present
input
vectors
to
to
a
neuron,
so
if
you,
if
it's
a
small
number
of
input
vectors,
then
you
just
randomly
assign
half
of
them,
then
you're
on
to
spike
and
half
of
them
in
neuron,
not
to
spike.
A
So
if
it's
a
small
number
of
patterns
and
neuron
won't
have
a
problem,
if
you
start
trying
to
load
a
large
number
of
patterns
on
the
neuron
to
be
able
to
discriminate
between
the
two
classes,
the
neuron
will
tend
to
have
a
a
difficult
time
and
there's
a
lot
of
theoretical
literature
about
this.
This
is
sort
of
an
old
school
way
of
evaluating
classifiers.
A
A
Yeah,
no
so
so
this
was
I
believe
that
I
tried
to
distribute
the
synapses
uniformly,
so
I
I
should
say
also
that
that
we're
interested
in
in
different
regions
of
the
dendrite,
because
people
think
that
I
mean
in
in
cortex
and
really
everywhere
on
the
brain,
so
the
apical
region
and
the
basal
region
of
dendrites
really
they're
getting
inputs
from
different
brain
regions.
So
it
was
kind
of
an
interesting
question
like
if
we
can
put
these
store
these
patterns
on
on
these
different
parts.
A
A
So
I'll
I'll
get
it's
not
not
on
a
biological
neuron.
Also
there
are
some
other
people
in
the
lab
who
are
working
on
clustering
of
biological
neurons
right.
So
Bartlett
Mel
had
a
paper.
It
wasn't
the
not
the
cluster
on
paper.
He
Bartlett
Mel
had
a
paper
where
he
did
supervised
learning
using
just
using
clustering
on
a
detailed
biophysical
model.
Also
so
so
this
is
like
yeah
the
same
idea
except
this
is
just
the
perceptron
learning
algorithm.
So
the
long
story
short
here
is
that
it
it
works.
A
A
So
you
can,
you
can
store
100
patterns,
no
problem,
a
thousand
patterns,
no
problem,
with
the
caveat
that,
if
you
put
all
the
synapses
on
the
apical
tuft,
then
you
have
reduced
classification
capacity.
So
you
do
worse.
When
you
try
to
store
a
thousand
pounds,
I
don't
really
want
to
get
into
it.
It
has
to
do
with
a
lot
of
like
subtle,
biophysics
things.
A
This
is
a
published
paper,
so
you
can
look
at
it.
The
the
main
takeaway
message
from
from
this
paper
is
that
I
would
say
that
yeah,
if
it
wants
to
a
biological
neuron,
can
basically
implement
the
perceptron
learning
algorithm,
as
is
no
problem,
and
it
can
act
as
a
linear
classifier
and
you
can
store
a
lot
of
patterns
on
it.
We
also
did
like
a
noise
robustness
task
which
is
sort
of
like
a
generalization
task.
I,
don't
really
want
to
get
into
it.
It
did
fine
on
that
also.
A
A
So
there
is
there's
something
called
the
nmda
receptor,
so
the
nmd
receptor
is
both
voltage-gated
and
ligand-gated.
So
that
means
that
well,
one
of
the
consequences
of
this
is
that
if
you
have
two
synapses
that
are
next
to
each
other
on
the
dendritic
branch,
but
they
will
interact
with
each
other
synergistically.
A
So
if
you
just
activate
synapse
a
you
might
get
this
this
depolarization.
If
you
activate
synapse
you'll,
get
that
depolarization.
The
linear
sum
of
them
will
be
the
blue
trace
and
if
you,
if
you
have
both
of
them
at
the
same
time,
you'll
have
this
much
larger
voltage
deflection
and
then
it
has
to
do
with
the
with
the
with
the
super
linearity.
This
voltage
dependence
of
the
nmda
receptor-
and
this
is
this-
is
distance
dependent.
A
A
The
idea
is,
and
instead
of
having
your
neuron
learned
by
adjusting
synaptic
weights
instead
of
what
you
do
is
you
have
you,
you
use
a
non-linear
interaction
between
synapses,
so
you
have
this
model
that
has
a
single
long
dendrite
and
each
synapse
has
something
called
an
inactivation,
and
the
activation
is
that
synapses
in
weighted
input
times
the
sum
of
the
weighted
inputs
of
all
the
other
synapses
within
a
certain
region.
So
that's
determined
by
this
di
here
and
the
the
way
that
you
you
learn
using
this
this
model.
A
So,
first
of
all,
I
should
say
this.
This
model
is
sensitive
to
correlations,
so
in
other
words,
if
you
have
two,
if
you
have
two
synapses
that
are
highly
correlated
and
they
sit
next
to
each
other,
they'll
they'll
interact
with
each
other
synergistically
and
will
have
something
larger
than
if
you
just
had
the
linear
sum
of
the
two
of
them
and
it's
a
multiplicative
non-linearity.
A
So
this
the
clusteron
is
basically
a
a
neuron
that
that
learns
correlations
and
the
the
learning
algorithm
for
the
cluster
without
getting
too
deeply
into
it.
Is
that
you
try
to
so
if
you
want
the
neuron
to
fire
a
spike
for
a
given
input.
What
you
want
to
do
is
you
want
to
for
that
input
for
that
input
class
you
want
to
put
correlated
inputs
next
to
each
other
on
the
dendrite.
So
the
way
that
you
do
that
is
you
feed
the
neuron
examples
of
your
positive
pattern.
A
So
in
this
case
that
would
be,
if
we're
doing
mnest,
and
we
want
to
train
this
neuron
to
fire
in
response
to
to
zero.
So
so
we
present
a
bunch
of
zeros.
We
just
present
the
positive
class
and
we
look
at
all
the
synapses
on
the
dendrite
and
the
synapses
that
are
that
are
weak,
so
the
synapses,
whose
activation
is
less
than
the
average
activation
of
the
other
synapses
we
randomly
swap
them
with
each
other.
So
I
always
compare
this
to
to
to
musical
chairs
right.
A
So
it's
basically
all
the
synapses
get
up
and
they
swap
with
each
other
and
eventually
over
time,
because
you're
swapping
the
weak
synapses.
Eventually
they
find
locations
that,
where
they're
activated
more
strongly
so
one
eye
together
with
Menachem
Commons,
and
he
was
an
undergraduate
student.
What
we
did
is
we
came
up
with
this
thing
called
the
gradient
plastron.
A
The
equation
is
not
so
important.
The
important
thing
is
the
understanding
the
rules
qualitatively
and
what
you
want
to
do
is
because
these
interactions
are
are
multiplicative.
So
if
you
want
the
neuron
to
fire
a
spike,
that
means
that
that
so
it's
a
positive
class
pattern.
You
want
same
sign
synapses
to
be
close
to
each
other
because
positive
times,
a
positive
is
a
positive
and
a
negative
times.
A
negative
is
a
positive.
A
A
A
A
stronger
weighted
inputs
means
stronger
retraction
and
repulsion,
and
the
the
attraction
and
repulsion
is
also
distance
dependent.
So
it's
a
it's
a
stronger
force,
close
by
sorry,
a
weaker
Force,
very
close
by
stronger
Force,
medium
and
and
as
you
get
further
away,
the
the
pull
of
the
different
synapses
gets
gets
again.
C
C
A
A
A
A
So
that's
that's
the
idea
now.
I
just
want
to
show
it
to
you
in
action.
So
this
is.
This
is
a
toy
example.
We
just
want
to
show
the
G
Plus
front,
increasing
its
overall
Activation,
so
it's
going
to
want
to
try
to
put
same
time
synapses
next
to
each
other.
So
what
we
do
is
we
just
put
a
bunch
of
random
randomly
randomly
initialized
synapses
along
the
dendrite
and
then
over
time.
A
This
is
what
happens
so
you
create
clusters
of
excitatory
and
inhibitory
synapses,
not
especially
useful.
It's
visually
nice.
So
toy
example.
Number
two:
is
we
want
to
show
that
you
that
you
can
learn
correlations
with
this,
so
what
we
did
is
we
created
this
multivariate
gaussian
data
set
where
each
group,
each
group
of
five
consecutive
input
Dimensions,
were
correlated
with
each
other.
A
Now
each
of
these
shapes
here
is
going
to
be
one
of
these
groups
of
the
five
consecutive
inputs.
So
one
to
six
will
be
a
triangle:
6
to
11
will
be
a
square
and
so
forth,
and
so
over
the
course
of
learning.
What
will
happen
is
that
it
happens
very
quickly.
You
can
see
that
it
segregates
out
these
groups
of
correlated
inputs
along
the
dendrite.
A
So
that's
okay,
so
we
can
learn
correlations
so
now
we'll
try
it
on
a
real
data
set.
Well,
semi-real,
so
we're
going
to
do
mnis
and
the
the
the
idea
is
essentially
like
I
said:
you're
learning
correlations
within
the
correlational
structure
of
each
of
each
digit,
and
you
can
see
that
before
learning
and
after
learning,
so
you
get
a
nice
separation
of
the
distributions
of
the
positive
patterns
and
the
negative
pattern.
A
So
this
might
be
the
number
two
and
this
might
be
I
think
we
did
like
one
versus
rest,
so
we
might
have
had
like
a
thousand
twos
and
a
thousand
of
everything
else,
and
this
is
what
the
distribution
looked
like
this.
This
corresponds
to
something
like
80
or
90
accuracy.
I
forget
on
the
on
the
one
versus
rest.
A
We'll
get
to
I'll
get
to
the
all
versus
all
and
and
like
the
exact
accuracies
later.
This
is
just
like
an
illustration,
and-
and
you
can
also
see
that
before
learning
and
after
so
before,
learning
there's
basically
no
structure
to
what
happens
to
the
synapse
I
should
say
so
in
this.
This
top
plot
here
there's
red
and
blue,
so
red
and
blue
corresponds
to
the
activations
of
it's,
like
they
synapses
but
histogrammed
by
by
location,
so
and
for
the
positive
and
negative
pattern.
A
A
So
you
see
this
is
this
is
a
a
histogram
of
the
densities
along
the
location
along
the
dendrite,
and
you
can
see
that
before
learning
there
was
like
no
structure
and
after
learning
you
have
these
regions
that
are
very
dense
that
have
a
lot
of
synapses
on
them.
So
you
really
have
these
clustered
synaptic
inputs
and
that's
what's
driving
the
learning.
At
least
we
think
okay,
so
in
terms
of
performance,
oh
I
should
say
also
we
we.
So
that
was
all
of
that.
A
That
separation
that
we
saw
before
that
was
just
by
changing
synaptic
locations.
No
synaptic
weights
were
used,
but
here
I
mean
there
were
synaptic
weights
in
the
model,
but
they
were
all
set
to
one
I
think
we
actually
accidentally
set
them
to
negative
one,
but
it's
the
same
because
it's
multiplication
doesn't
matter
so,
but
anyway
we
also.
A
We
also
have
have
a
rule
for
updating
the
weights
and
updating
the
this.
This
rule
I
don't
want
to
get.
Basically
you
just
multiply
it
by
the
activation
divided
by
the
weight.
So
we
have
both
a
rule
for
changing
the
synaptic
locations
and
the
synaptic
weights,
and
we
did.
We
did
an
all
versus
all
Paradigm.
So
so
we
had,
we
had
a
not
a
network,
there's
a
single
layer
with
10
neurons,
and
so
we
had
that
that
do
the
all
versus
all
classification,
we
had
two
different
para.
A
A
Here
we
were
just
the
the
point
we
were
trying
to
make
is
that,
with
the
with
the
original
clusteron,
you
can't
use
a
soft
Max,
because
it's
not
a
gradient
descent
algorithm,
that's
probably
too
technical
bottom
line
is
that,
with
with
the
Jeep
faster
on
you
can
get
like.
You
know:
85
90,
the
accuracy
on
all
versus
all
MNS,
which
is
basically
the
same
that
you
get
with
logistic
regression
and
you
get
you
can
get
this
accuracy
only
by
using
synaptical
locations
and
not
with
synaptic
weights
right.
A
So
this
is.
This
is
just
with
locations,
and
this
is
with
weights,
and
this
is
with
locations
and
weights.
So
you
might
do
a
little
bit
better
with
having
both
locations
and
weights,
but
really
even
if
you
can
just
modify
the
synaptic
locations-
and
you
have
a
soft
map,
you
can
get
like
85
percent
accuracy
on
all
versus
all
heaviness,
which
I
think
is
reasonably
impressive
for
this
algorithm
that
we
just
like
made
up
and
doesn't
have
synaptic
weights
and
is
just
using
the
correlations.
C
C
A
A
C
A
Well,
so
it's
it's
a
bit
solar
than
that,
because,
okay,
so
in
in
theory,
yes,
but
so
the
best
way
to
think
of
it
might
be
in
terms
of
the
number
of
parameters
right.
So
the
number
of
parameters
here
is
so
if
each
synapse
has
just
a
location
right.
So
that's
that's
the
same
number
of
parameters.
They
have
each
synapse
having
just
weights,
so
the
operations
of
boundaries
that
you
draw
will
be
different,
they'll
be
more
curvy,
but
that
is
not
neces.
Being
more.
A
Curvy
is
not
necessarily
better
with
the
same
number
of
parameters,
so
so
I
I
think
I
think
it's
best
to
think
of
it
in
terms
of
that.
No
it's
true!
So
when
we
have
both
locations
and
weights,
you
do
have
two
times
in
the
number
of
parameters.
So
you'd
think
that
you
might
get
a
bit
of
an
increase
by
doing
that,
but
that's
not
necessarily
guaranteed.
B
A
B
Yeah
so
Tavia
did
you
do
any
theoretical
analysis
of
this
algorithm,
because
two
things
are
coming
to
mind
one
would
be:
is
it
actually
guaranteed
to
converge
so
there's
the
optimal
behavior
that
this
neuron
could
exhibit
theoretically,
but
then
there's
the
Practical
issue
of
actually
finding
the
parameters
and
moving
things
around
so
that
you
get
that
optimal
Behavior.
So
the
first
question
is:
do
you
have
any
guarantees
about
convergence?
The
second
question
is:
did
you
do
any
theoretical
analysis
for
something
like
the
VC
dimension
of
this
model.
A
So
the
answer
to
that
is
is
no
it
like
it's
it's
a
it's
a
good
question
and
I
have
thought
about
it.
The
problem,
so
it's
it's
complicated,
because
so,
if
it
were
just
a
simple
quadratic
classifier
and
you
had
all
of
like
the
N
squared
terms
that
that
might
be
easier,
this
is
this
is
sort
of
a
constrained,
quadratic
classifier.
So
so,
okay,
maybe
I'll,
take
a
step
back
so
Bartlett
Mel.
A
He
did
a
theoretical
analysis
with
perazi
on
the
original
closteron,
and
so
so
that
so
the
original
clusteron
is
called
a
sub-sampled
quadratic
classifier,
because
because
the
original
clusteron
you
have
this
fixed
regions
of
of
interaction.
So
basically
there's
a
fixed
number
of
mixed
terms,
and
you
know
what
they
are.
A
A
Yeah
exactly
so
so
the
the
G
plus
one
is
a
bit
more
confusing
because
you
actually
have
all
of
the
mixed
terms,
but
the
mixed
terms
are
kind
of
tied
together.
Why?
The
reason
why
they're
tied
together
is
because
the
this
facial
constraints
of
being
along
a
single
dendrite
means
that
there
are
certain.
So
if
you
imagine
you
have
you
have
a
matrix
of
the
coefficients
of
all
these,
these
mixed
terms,
so
you
can't
change
those
coefficients
independently
because
they're
all
sitting
on
a
dendrite.
A
So
if
you
change
one
of
them,
you'll
have
to
change
the
other
ones,
because
they're
they're,
a
constraints
right
so
like.
If
you
have
three
points
and
a
is
to
the
left
of
B
and
B,
is
to
the
left
of
C.
So
AF
has
to
be
to
the
left
of
C
right.
So
so
it's
definitely
it's
an
interesting
problem.
Theoretically,
but
yeah
we
didn't.
We
didn't
do
that
analysis.
A
One
thing
that
we
did
do
is
we
showed
that
as
long
as
you
have
a
single
bias,
synapse
so
by
biasing,
apps
I
mean
just
a
a
synapse,
a
g
cluster
on
synapse
that
works,
like
all
the
other
synapses
that
has
a
constant
input,
so
the
G
clusteron
is
able
to
can
can
basically
Implement
any
linear
separation
boundary
right.
So
so
the
G,
the
G
Class
one,
is
capable
of
doing
all
of
the
things
that
a
linear
classifier
is
capable
of
doing
in
theory,
and
then
we
have
one
more
thing.
A
So
this
is.
This
is
just
we
wanted
to
show
that
the
G
colostron
can
be
a
bit
better.
So
the
simplest
way
to
do
that
is
to
show
that
it
can
solve
the
exclusive
word
problem,
so
exclusive
War
problem.
You
have
two
two
binary
bits
of
input
and
it
returns
a
one.
If
the
bits
are
different,
then
it
returns
a
zero.
A
A
So
we
show
we
show
analytically,
that
a
solution
for
exclusive
or
does
exist
in
in
the
G
Plus
run.
So
if
you
you
can
add,
you
can
show
that
if
you
satisfy
certain
inequalities
with
the
weights
and
the
locations,
so
you
you
have
a
solution
in
this
in
this
region.
What
this
region
basically
is.
It
means
that
the
the
synapses
are
like
opposite
signs
and
are
next
to
each
other.
So
so
this
this
configuration
I,
say
opposite
signs.
A
I
mean
the
right,
the
the
signs
of
the
synaptic
weights.
So
this
configuration
where
one
of
the
weights
is
negative,
one
of
the
weights
is
positive
and
the
synapses
are
next
to
each
other.
On
on
the
dendrite,
you
can
solve
the
exclusive
word
problem
with
that
configuration,
so
you
can,
if
you
have
both
the
weight.
Rule
and
the
location
rule
you
can
solve
the
exclusive
or
problem
from
arbitrary
initial
conditions.
C
A
So
one
of
the
things
that
we
that
we
say
in
the
paper
is
that
you
can
actually
one
way
to
think
about
the
update
rule
is
as
though
you
have
like
almost
a
force
field
along
a
dendrite.
So
if
you
have
a
particular
configuration
of
synapses
along
a
denverite
and
you
have
a
unit
synapse
with
the
weight
weighted
and
input
value
one,
so
you
can
show
along
the
dendrite
where
that
unit
synapse
would
want
to
move
to.
A
So
it's
like
like
putting
a
charge
in
an
electrical
field
or
something
so
if
you
imagine
that
you
have
chemical
gradients
along
along
the
dendrite
that
can
tell
synapses
synapses
where
to
move
or
maybe
where
to
create
new
synapses
based
on
the
activity
of
other
synapses
I.
Think
that's
that's
an
interesting
idea.
C
Another
way
this
could
possibly
happen.
It's
not
exactly
it's
not
this
learning
rule,
but
if
you
think
about
structural
plasticity,
we
can
add
and
drop
synapses.
If,
if
a
particular
input
really
wants
to
be
connected
to
a
different
physical
location
along
the
dendrite,
you
could
imagine
learning
rules
that
you
know:
drop
drop,
the
synapse
in
one
location
and
and
create
a
synapse
at
the
other
location.
C
A
So
that's
that's.
That
is
the
way
to
think
about
it.
Yeah
I've
gone
through
groups
in
my
head,
trying
to
figure
out
exactly
how
it
would
how
it
would
look
biologically
there.
There
are
some
nice
like
molecular
biology
papers,
forgetting
I,
think
there's
people
with
like
Klein
dice
or
whatever
they're
so
so
there.
There
is
definitely
a
lot
of
chemical
signaling
for
structural
plasticity,
so
they're
stuck
with
like
bdnf
and
like
trpk
or
something
they're
about.
A
C
B
A
And,
and
also
I
think
that
you
might,
you
might
have,
let's
say,
variation
in
the
size
of
these
clusters,
because
this
is
also
you
know
when,
when
you
produce
these
things,
it's
also
Verity
contingent
on
like
bin
size
and
stuff.
You
know
so
it
looks
like
these
highly
dense.
Oh
sorry,
also
these,
because
of
the
way
that
the
algorithm
works.
A
So
if
you
let
the
algorithm
run
for
a
long
time,
well,
I
have
to
yeah
I
have
to
think
about
it,
but
but
clustered
this
sort
of
clustered
structure
is
sort
of
in
a
tractor.
If,
if
you
think
of
the
examples
that
I
gave
before
with
the
with
the
correlated
inputs
and
the
different
shapes,
so
in
in
an
idealized
in
an
idealized
situation,
they
all
eventually
Coral
they
converge
to
like
a
single
point,
almost.
C
Right
right,
yeah
I
was
also
thinking
just
in
terms
of
you
know:
NBA,
spikes
and
and
MDA
spiking
thresholds.
You
know
the
I,
don't
again,
I
don't
mean
to
put
too
much
into
it,
but
the
spiking
thresholds
are
sort
of
roughly
in
this
area
right.
You
know
10
to
15
synapses
again,
I,
don't
know
how
biologically
closely
parameters
are
at
all.
But
it's
kind
of
interesting
that
these
spikes
are
well
above
the
spiking
threshold
and
the
other.
The
non-clustered
area
seem
to
be
below
that
threshold.
C
C
But
yeah,
but
these
thresholds
are
seem
to
be
very
Universal
as
long
as
you
have
sparsity
and
reasonable
dimensionality,
you
get
some
reasonable
threshold
above
which
you
can
get.
You
can
do
pretty
accurate
classification.
So
true,
this
is
just
mnist,
but
there
is
something
somewhat
Universal
about
those
thresholds
according
to
our
theoretical
analysis.
Yesterday,
as
long
anyway,
as
long
as
you
have
sufficiently
high
dimensionality
and
sufficient
sparsity.
A
A
Right
yeah,
we
were
actually
playing
around
with
that
a
lot.
Yes,
that
that
is
true,
whether
that's
some
sort
of
Universal
thing,
that's
good
for
computation,
or
it
might
just
be
a
thing
that
biology
has
some
sort
of
mechanism
to
simultaneously
activate
right.
So
so
there
might
be
a
thing
called
clustered
activation,
which
is
a
separate
thing
than
like
individually,
randomly
activating
them,
but.
C
C
You
know
that
there
is
some
analytical
reason
to
to
believe
in
this
sort
of
number
at
threshold
around
10.
You
know
somewhere
or
maybe
a
little
bit
higher.
Sometimes
you
know
it
does
vary
depending
on
the
exact
conditions.
But
again,
if
you
have
sufficiently
high
dimensionality
and
reasonably
sparse
inputs,
you
can
you
can
get
by
with
sort
of
absolute
thresholds
across
a
wide
range
of
tasks.
C
You
know
if
you
have
extremely
noisy
inputs
coming
in
and
you
have
you
know
thousands
of
inputs
on
this
neuron.
How
could
you
possibly
do
anything
reasonable
with
the
threshold
of
eight
or
nine
and
you
can
actually
show
analytically?
That's?
Actually
it's
a
pretty
decent
threshold,
Sonic
CS
varsity.
A
So
I
I
have
three
more
three
more
papers
together.
So
well,
let's,
let's
just
see
what
we
can
do
so
the
next
thing
I
wanted
to
do
is
I.
Are
you
interested
in
the
actual
actual
biological
mechanisms
of
plasticity,
so
these
perceptron
algorithm,
gradient
cluster
on
algorithm
and
so
forth
are
really
like.
Really
we
just
made
them
up.
There's
you
know
me,
you
can
sort
of
hand
wave
about
how
they
happen
biologically,
but
but
really
they
were
designed
to
to
solve
classification
tasks.
A
So
we
wanted
to
get
into
the
biology
of
of
of
real
plasticity.
So,
typically,
when
people,
when
biologists
talk
about
plasticity,
there
are
two
experimental
protocols
that
you
can
get,
what
they
call
long-term
plasticity,
it's
a
little
bit
of
a
misnomer,
because
long-term
plasticity
may
only
last
10
minutes,
but
we'll
ignore
that
for
the
moment,
the
right
so
there's
something
there's
frequency
dependent
plasticity,
which
is
a
presynaptic
only
protocol.
A
So
if
you
have
very
slow
frequency
inputs,
so,
for
example,
you're
you're,
stimulating
presynaptic,
axons,
very
slow
inputs
will
not
really
will
not
really
induce
plasticity.
If
you
get
to
sort
of
a
medium
level
of
of
the
of
the
input
frequency,
you
get
depression
and
if
you
have
a
high
frequency
of
stimulation,
you
have
potentiation.
So
that's
frequency,
dependent,
plasticity
and
there's
another
kind
of
plasticity
called
Spike
timing,
dependent
plasticity
where
you
have.
A
She
she
worked
with
with
me
on
this,
and
we
want
to
sort
of
understand
the
the
biological
mechanisms
under
underlying
plasticity
and
what
people
think
in
terms
of
the
the
the
molecular
stuff
that's
going
on
is
that
when
you're
doing
the
synaptic
potentiation
calcium
entered
the
cell
either
through
the
nmda
receptor
or
through
voltage,
getting
calcium
channels
or
they're
not
shown
here
so
calcium
enters
the
cell.
It
interacts
with
chem,
K2
and
ampa.
Receptors
are
are
shuttled
to
the
synaptic
spine
so
that
so
you
have
increased
conductance.
A
We
have
an
increased
synaptic
strength
and
with
synaptic
depression,
calcium
comes
into
the
cell.
It
interacts
with
calcinarin
and
ampereceptors
are
removed
from
the
spine,
so
both
of
these
processes
start
with
calcium
entry.
So
the
question
is:
how
does
the
synapse
know
whether
to
potentiate
or
depress?
So
this
leads
us
to
the
calcium
control
hypothesis,
calcium
control
hypothesis
says
that
if
you
have
that
everything
depends
on
the
concentration
of
calcium
in
the
postsynaptic
spine.
So
if
you
have
not
very
much
calcium,
nothing
happens.
A
If
you
have
a
medium
amount
of
calcium,
you
have
depression,
and
if
you
have
a
lot
of
calcium,
you
have
potentiation,
and
so
there
are.
There
are
models
that
have
that
have
shown
that,
from
this
from
this
calcium
controller
process
from
this
from
this
concentration
based
tool,
you
can
get
those
effects
of
frequency,
dependent,
plasticity
and
Spike
timing,
dependent,
plasticity,
I,
don't
want
to
get
into
that
so
much.
We
also
did
some
modeling
of
that.
A
There
are
a
few
other
important
things
about
biological
plasticity,
so
biological
plasticity,
saturates.
So,
instead
of
you
know,
if
you
have
like
a
normal
learning
algorithm
like
the
perceptron,
you
can
just
keep
going
up
forever
and
keep
going
down
forever
in
biology.
So
synaptic
strength
will
will
saturate.
Eventually
you
just
potentiate
the
synapse
so
much
it
won't
get
any
stronger
and
you
can
weaken
it
so
much
and
it
won't
get
any
weaker
and
it
and
the
the
bottom
is
not
necessarily
zero.
A
A
A
So
if
you
block
protein
synthesis
in
in
the
cell,
plasticity
is
really
temporary.
This
this
calcium-based
stuff,
so
the
calcium
happens
within
let's
say
an
order
of
hundreds
of
milliseconds,
maybe
seconds,
but
without
without
protein.
It
just
goes
back
to
Baseline.
If
you
have
protein
synthesis,
this,
the
synaptic
weight
is
stabilized
and
the
same
thing
happens
for
for
depression.
A
A
A
A
So
we'll
give
this
calcium
step
stimulus,
a
lot
of
calcium,
so
potentiative,
calcium,
depressive,
calcium,
and
if
you
do
that,
so
so
you
do
that
and
then
then
you
turn
off
the
calcium
and
then
you
sort
of
see
what
happens
so
to
illustrate
this,
these
Dynamics.
So
if
you
have
this
potentiative
level
of
calcium,
so
the
fixed
Point
learning
rate
says:
okay,
potentiate,
a
level
of
calcium
you
go
up
to
this.
This
is
the
upper
fixed
Point,
that's
the
fixed
Point
specified
over
here.
A
This
is
a
potentiative
region
of
calcium.
If
you
have
a
depressive
amount
of
calcium,
you
go
to
the
fixed
points
specified
over
here.
So
that's
this
lower,
fixed
Point
here
and
then.
Finally,
when
you
turn
off
the
calcium,
you
go
back
to
this
to
the
middle
fixed
point.
So
this
is
going
back
to
Baseline.
That's
that
that's
that
neat
phase
drift
back
to
Baseline
that
you
have
without
protein
synthesis.
We
also
have
a
model
for
when
you
do
have
protein
synthesis.
A
B
A
So
the
overall
context
is
that
this
is
supposed
to
approximate
what
happens
in
biology
now
in
terms
of
in
terms
of
the
consequence
of
it
for
sort
of
learning
rates,
so
I
a
little
bit.
Think
of
it
as
like.
You
know
like
an
L1
or
L2
loss,
so
so
so
with
right.
So
so,
if
you,
if
you
have
weight
panels,
it's
what
I
want
to
know
to
loss
penalizations
do
is
that
they
say
well.
A
If
the
sum
of
the
weights
is
too
big,
then
you
know,
then
then
we
basically
penalize
that.
So
this
is
a
little
this
you
can
think
of
it
as
a
different
way
of
penalizing
ways
where,
instead
of
putting
it
into
loss
function,
we
just
say
that
the
way
that
synapses
adjust
their
weights
is
that
sort
of
you
have
a
maximum
and
minimum
weight
and
the
closer
that
you
get
to
the
maximum
or
the
closer
that
you
get
to
the
minimum,
the
less
they're
going
to
change.
A
A
Yeah
I,
don't
even
know
if
I
would
I
would
talk
about
it
in
terms
of
the
objective
function,
because
really
this
is
at
the
level
of
the
learning
algorithm
right,
because
this
is
you
know,
I,
don't
I,
don't
know
if
this
is
optimal
for
something
I.
My
intuition
is
that
it
actually
probably
is
computational
useful
for
something,
but
this
is
this
has
really
designed
to
imitate
just
what
we
observe
in
biology,
but
yeah,
like
you,
can
definitely
think
of
scenarios
when
this
would
be
helpful.
A
B
A
B
C
A
A
So
to
do
that
we
develop
a
a
model
neuron
that
basically
has
calcium,
so
it's
basically
a
linear
neuron,
but
each
and
so
so
yeah.
So
it's
a
linear,
neuron
and
it
has
calcium
sources.
So
the
calcium
sources
are
so
you
can
have
like
calcium
from
local
sources.
That's
that's
at
the
synapse,
that's
from
the
input
itself.
You
can
have
heterosynaptic
calcium,
which,
which
comes
from
other
synapses
I.
A
So
it's
very
simple
example:
heavy
and
learning
you
set
the
coefficient,
such
that
the
local
calcium
alone
or
the
post-synaptic
spike
alone.
They
give
you
calcium,
which
is
below
the
depression
threshold,
but
if
you
have
both
of
them
together,
they
sum
up
and
are
above
the
potentiation
thresholds
right.
So
this
gives
you
fire
together
wired
together,
you
can
also
get
out
of
sync
loser
link
just
by
raising
up
the
individual
ones
to
be
above
the
depression
threshold.
A
There
are
a
lot
of
other
things
we
did
with
this
there
are
you
can
do
a
lot
of
learning
rules.
You
can
do
the
perceptron
with
this.
If
you
have
a
supervisor,
we
did
homeostatic
plasticity,
that's
the
calcitron
and
then
finally,
there
is
this:
this
product
that
I
did
with
Menachem
commonsen.
That
I
think
will
interest
you
that
this
has
to
do
with
heterosynaptic
elasticity.
So
sometimes
you
can
induce
plasticity
at
that.
One
synapse
say
the
middle
synapse.
A
You
only
you'll
get
potentiation
and
then,
if
you
look
at
other
synapses
you'll
get
depression,
so
there's
a
John
lisman
who
basically
invented
calcium-based
plasticity.
He
had
this
theory
that
the
at
the
synapse
that
you
induce
plasticity
at
that's
called
the
homosynaptic
synapse
and
they're.
The
way
that
you
get
calcium
is
through
the
nmda
receptor
that
and
that's
just
directly
from
from
the
presynaptic
input.
A
Also
you
have
the
voltage-gated
calcium
channel,
because
you're
depolarizing
the
spine
now
since
you're
depolarizing
the
dendrite,
with
this
input,
some
of
that
some
of
that
current
also
travels
and
depolarizes
others
funds,
so
that
depolarization
it
doesn't.
It
won't
open
up
the
nmda
receptor,
because
nmd
receptor
requires
neurotransmitter,
but
it
will
open
up
the
voltage-gated
calcium
channels.
A
So
this
is
one
theory
of
how
you
get
heterosynaptic
plasticity.
Now,
if
you
do
this
in
a
cable
model
of
a
dendrite,
there
is
a
thing
to
know
about
dendrites,
which
is
that
the
voltage
attenuates
in
dendrites.
So
if
you
put
a
synapse
you
have,
this
is
a
ball
in
Stick
Cable
model
of
the
dendrite
and
if
you
put
a
synapse
somewhere
in
the
middle
of
the
dendrite,
so
the
volt
the
voltage
will
attenuate
towards
the
Soma.
A
So
if
you
see,
if
you
look
at
this
image
here,
so
this
direction
is
is
towards
the
distal
tip
of
the
dendrite
in
this
direction
is
towards
the
Soma.
This
axis
is
time,
and
so,
if
you
stimulate
the
synapse,
you
see
that
distally,
you
depolarize
a
lot
and
approximately
approximately
you
get
much.
You
get
a
lot
of
attenuation
and
if
you
do
an
nmda
spikes,
this
is
what
the
nmda
spike
looks
and
then
right
so
again,
depolarize
a
lot
distally
and
then
our
attenuates
proximally.
A
And
if
you
have
a
branching
dendrite,
you
get
this
kind
of
structure.
You
put
the
synapse
over
here,
you
depolarize
this
dendritic
Branch.
So
these
are.
This
is
a
branching
structure
of
a
dendrite
right.
This
is
for
Antoine.
Two
Etc,
so
you
depolarize
everything,
that's
distal
to
you
and
then
you
attenuate
towards
the
Soma
and
yeah
here
here.
You
also
you
depolarize,
a
sister
Branch,
because
the
sister
branches
basically
gets
the
depolarization
from
the
proximal
Branch
here.
A
So
this
is
just
a
property
of
the
biophysics
from
the
fact
that
the
Soma
is
is
acts
as
an
electrical
sink,
because
it's
attached
to
a
bunch
of
other
dendrites
okay.
So
this
is
this
is
nice,
but
we've
known
this
since
since
raw
since
the
1960s,
but
the
cool
thing
is
that
now,
if
you
think
about
heterosynaptic
plasticity
and
you
think
about
this
asymmetric
attenuation
of
voltage,
so
maybe
what
you
can
do
is
you
can,
if
you
have
a
strong
input
to
to
somewhere
on
a
done
right.
A
A
So
what
if
this
becomes
cool
is,
if
you
have
a
branching
done
right
and
you
have
a
strong
input,
let's
say
here
at
this
Branch,
so
you
can
depolarize
the
entire
Branch
distally
to
you
or
if
you
here,
this
is
a
slightly
more
proximal
Branch,
so
you
need
a
more
inputs
because
lower
input
resistance,
but
again
you
can.
You
can
depolarize
and
induce
plasticity
at
the
at
that
entire
distal
sub-free.
So
this
is
I.
A
This
can
be
thought
of
as
a
hierarchical
plasticity,
supervision,
rule
hierarchical
in
the
sense
that
if
you
have
a
more
proximal
input
that
proximal
input
is
a
supervisor
to
everyone,
who's
distal
to
it.
So
so
this
this
location
is
in
control
of
all
these
locations
and
say
this
location
is
in
control
of
all
of
those
locations.
A
A
So,
to
summarize
everything,
so
you
have
the
biophysical
perceptron,
which
says
that
the
real,
biologically
detailed
neurons
can
implement
the
perceptron
learning
algorithm.
We
have
the
gradient
plus
run
with
the
traction
and
repulsion
of
synapses,
the
fixed
Point
learning
rate
for
calcium
I,
didn't
say
it
this
way,
but
I'll
say
it
now:
calcium
tells
the
synaptic
weights
where
to
go
and
how
fast
they
get
there.
That's
that
that
saturating
plasticity
we
have
the
calcitron,
which
is
that
you
can
have
different
learning
rules
that
you
can
create
using
calcium-based
plasticity.
A
D
A
No
because
because
yeah
so
so
we're
not
so
I'm
so
because
I'm
I'm
working
on
the
single
neuron
my
perspective
is
that
I
care
I
care
about
everything
that
happens
up
until
the
Soma
of
the
post-synaptic
neuron
and
I.
Also
am
not
thinking
about
the
presynaptic
right,
I'm
thinking
about
the
presynaptic
neurons,
just
as
like
neurotransmitter
release,
so
axonal
delays
in.
C
A
So
so
he
has
I
think
it's
still
a
pre-print
out
now
I
forgot
exactly
what
the
title
is,
but
it's
called
the
filter
and
fire
neuron,
so
so
that
so
that
work
is
entirely
about
delays
along
the
the
dendritic
tree,
not
along
the
axon
and
the
claims
that
he
makes.
Is
that
because
you
have
those
Den
brights
are
you
can
think
of
sort
of
as
as
delay
line?
A
A
B
More
general
question:
I,
don't
remember
the
details
of
this,
but
I
kind
of
recall
some
data
saying
that
the
amount
of
neurotransmitter
released
can
actually
vary
to
a
surprising
degree.
So
like
from
one
Spike
to
the
next,
you
can
have
like
a
lot
of
neurotransmitter
released
or
a
really
small
amount
released.
Did
you
ever
look
into
like
robustness
or
tolerances
to
like
stochastic
amounts
of
neurotransmitters.
B
A
Yeah,
yes,
yes,
so
so,
if,
if
you
so
in
the
context
of
of
the
biophysical
perception,
okay,
so
right
so
I
should
say
generally
yeah.
That
neurotransmitter
release
is
just
one
of
many
many
forms
of
noise
that
can
exist
so,
but
one
of
the
things
that
we
did
with
the
biologic
with
a
biophysical
perceptron
is
that
we
did
an
experiment
with
where
we
basically
did
bit
flips
right.
A
So
we
just
turned
the
ones
into
zeros
or
vice
versa,
randomly
and
we
did
different
proportions
of
these
bit
flips
and
we
wanted
to
see
how
how
robust
it
is.
You
know
if
we,
you
know
flip
10
of
the
synapses
2050
and
I
have
the
results
in
the
paper.
I
don't
have
them
here,
but
basically
you
have
what
you
expect,
which
is
that
you
know
the
more
bits
that
you
flip,
the
worse
performance
becomes,
but
you
know
if
you,
if
you
flip
10
synapses
you
can
it
will.
A
It
will
still.
You
know,
do
reasonably
well
so
yeah
so
in
in
the
in
the
blue
brain
when
they
do
these
cable
models
and
they
have
these
synapses.
They
actually
have
in
the
model
synapse.
They
have
the
the
synaptic
release,
is
probabilistic
and,
and
they
hold
they
have.
They
also
have
like
synaptic
depression
facilitation,
because
you
have
what's
it
called.
A
You
have
vesicles
that
are
that
are
like
used
up
and
then
that
are
docked
and
there's
a
whole
process
of
getting
them
to
so
so
these
these
are
things
that
exist
in
in
more
detailed
models.
I
didn't
address
it
so
much,
except
in
in
the
case
of
the
biophysical,
perceptron
yeah.
We
did
do
that
that
noise
robustness
check
yeah.