►
From YouTube: Numenta Research Meeting - October 14, 2020
Description
Subutai Ahmad and Jeff Hawkins clarify aspects of active dendrites, and some of the history behind them, for other members of our research group. Using two key papers as a backdrop, they highlight issues such as basic dendritic integration, technological advances for stimulation of synapses, the controversy around and meaning of dendritic spikes, and the importance of understanding temporal dynamics. The team then discusses how to better understand dendrites from a machine learning perspective.
Papers mentioned:
'The Decade of the Dendritic NMDA Spike': https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5643072/
'Active Properties of Neocortical Pyramidal Neuron Dendrites': https://pubmed.ncbi.nlm.nih.gov/23841837/
- - - - -
Numenta is leading the new era of machine intelligence. Our deep experience in theoretical neuroscience research has led to tremendous discoveries on how the brain works. We have developed a cohesive theory, core software technology, and numerous software applications all based on principles of the neocortex. Our innovative work delivers breakthrough capabilities and demonstrates that a computing approach based on biological learning principles can do things that today’s programmed computers cannot do.
Subscribe to our News Digest for the latest news about neuroscience and artificial intelligence:
https://tinyurl.com/NumentaNewsDigest
Subscribe to our Newsletter for the latest Numenta updates:
https://tinyurl.com/NumentaNewsletter
Our Social Media:
https://twitter.com/Numenta
https://www.facebook.com/OfficialNumenta
https://www.linkedin.com/company/numenta
Our Open Source Resources:
https://github.com/numenta
https://discourse.numenta.org/
Our Website:
https://numenta.com/
A
A
I
would
say
the
dendrite
world
is
hard
for
neuroscientists
to
understand
in
general
because
they
have
their
own
within
neuroscience.
It's
a
it's
a
particular
subfield
and
they
have
again.
They
look
at
it
from
a
very
from
a
particular
perspective
and
there's
a
lot
of
history
behind
what
they're
talking
about
and
a
lot
of
that's
kind
of
implicit
in
the
language,
but
it's
not
explicit
in
there
and
so,
and
they
they're
very
much
into
like
these
detailed
mechanisms
and
the
dynamics
of
what's
going
on
inside
the
cellular
membrane.
A
And
there's
chemical
stuff
and
there's
electro
electrical
stuff,
that's
going
on
and
it's
I
find
it.
You
know
quite
hard
to
understand
sometimes
as
well.
So
you
know
I
thought
I'd.
I
just
give
a
pick
up
a
few
different
things
from
a
couple
of
papers
and
then
I
can
give
some
color
and
jeff
can
give
some
color
on
it
and
how
to
think
about.
A
It
might
be
helpful,
so
the
two
papers
that
we
often
refer
to
one
is
this
review
paper
by
antic,
which
reviews
a
decade
of
dendritic
research
and
focused
on
the
nmda
spike,
and
then
there's
this
one
by
major
larkam
and
chiller,
which
reviews
kind
of
similar
things
essentially
the
same
thing.
But
again
you
look
at
the
two
papers
and
sometimes
it's
hard
to
even
correlate
the
two
papers
together.
A
So
what
did
I
do
here?
I
picked.
I
highlighted
a
few
phrases
here,
so
you
know
a
lot
of
what
they're
talking
about.
Is
this
thing
called
dendritic
integration,
which
is
on
the
dendrites
of
cells
when
you
get
inputs
coming
in
there's
all
of
these,
the
the
inputs
and
there's
this
sort
of
electrochemical
reaction.
That's
going
on
and
the
dendritic
inputs
influence
the
the
cell
in
some
way.
A
Right
and
here
you
know,
they're,
saying:
okay,
how
in
dendritic
inputs
combine
to
influence
the
membrane
potential
and
there's
a
potential,
the
electrical
potential
right
inside
the
dendrite
and
then
there's
also
the
electrical
potential
at
the
cell
body
or
the
soma.
Those
two
are
very
different
and
you
have
to
keep
that
in
mind
when
you're,
when
you're
reading
these
papers
as
well.
A
A
A
So
here
they
say
you
know
in
visual
recordings
and
computational
studies,
modeling
studies
shows
that
dendrites
can
switch
to
a
highly
supralinear
regime
which
in
principle,
can
lead
to
so-called
dendritic
spikes
right.
So
that
means
that
in
the
dendrite,
if
you
have
inputs,
you
can
get
this
non-linear
response
and
they
have
some
pictures
of
this
I'll
show
you
in
a
second
which
can
lead
to
dendritic
spikes,
but
back
then
you
know,
then
they
ask
do
pyramidal
cell
dendrites
actually
fire
spikes.
A
A
B
It's
funny
from
a
from
a
theorist
point
of
view.
It
was
it's
almost
certain
that
this
has
to
happen,
because
if
it
doesn't
happen,
then
90
of
the
synapses
on
a
neuron
don't
seem
to
have
an
effect.
So
it's
it's
like
you
know
and
experimentalists,
don't
like
it.
If
something
like
myself
says
that
like
well,
how
do
you
know
like
well?
You'd
be
pretty
certain.
A
Yeah-
and
this
is
the
language
they
used
to-
I
think
they
used
to
describe
that.
So
here,
if
you
look
here,
it
says
you
know
the
apical
distal,
basal,
apical
and
tuft
epsp
peak
voltages,
attenuate
by
30
fold
or
more
as
they
spread
from
the
synapse
down
down
to
the
soma.
What
this
means
is,
if
you
look
at
the
potential
right
at
the
dendrite
versus
the
soma,
it
just
decreases
by
a
factor
of
30
or
more
so
that
means
by
the
time
it
gets
to
the
actual
cell
body.
It's
it's
hardly
measurable.
C
A
B
They
would
eject
current
right
at
the
you
know,
some
synapses
on
the
end
of
the
dendrite.
They
know
what
they
can.
They
know
there's
various
ways
of
doing
that
or
they
can
activate
synapses,
optically
and
so
on.
So
they
know
that
there's
a
a
pulse
of
voltage
transition
right
at
the
synapse
and
then
they
could
just
measure
the
voltage
at
the
soma
and
they
say
it
was
hardly
anything
it's
barely
visible
at
all.
So
that's
why,
for
so
many
years,
people
just
ignored
them
because
they
say
well.
They
can't
seem
to
do
anything.
C
C
B
C
B
A
Well,
yeah
I'll
explain
this
in
a.
I
think
I
I
don't.
I
don't
understand
all
the
details
of
the
exact
technique,
but
what
happened
is
sometime
in
the
90s
there
was
this
technology
called
glutamate,
uncaging
that
that
came
into
play
and
they
basically
do
four
things.
They
can
control
four
different
things
at
a
time-
and
this
was
a
dramatic
breakthrough
in
kind
of
technology
to
allow
you
to
do
this
stuff,
so
they
can
pick
a
specific
location
in
a
dendrite.
A
D
A
Yeah
130
would
be
huge
here.
It's
about
10,
10,
yeah,
10,
microns,
here's
the
scale
bar,
so
they
can
precisely
control
how
many
synapses
are
being
kind
of
activated
at
the
same
time-
and
this
is
all
within
a
very
short
window
of
time-
a
few
milliseconds
basically
and
you
can
see,
and
then
they
can
measure
the
the
epsp
here
and
what
you
can
see
is
if
they,
if
they
stimulate
seven
synapses,
you
get
this.
You
know
smaller
response,
but
at
eight
synapses.
A
Suddenly
you
get
this
much
bigger
response,
and
so
this
is
this
kind
of
non-linear
response
you
get,
and
this
is
how,
in
our
temporal
memory,
we
we
have
this
kind
of
threshold
where
we
say
if
we
get
more
than
a
certain
number
of
active
synapses
in
a
small
in
a
dendritic
segment,
you
get
a
kind
of
a
non-linear
response.
You
have
a
dendritic
spike
there
and
what's.
B
It
also
invokes
various
mechanisms
locally
in
the
dendrite,
so
that
dendrite
is
now
in
a
different
chemical
biochemical
state
than
it
would
have
been
if
it
only
had
one
or
two
signatures
active,
and
so
this
is
an
important
part
of
the
learning
rules,
because
we
need
to
have
those
synapses
trained
and,
and
so
they're
they're,
all
sort
of
in
a
different
state
at
this
time.
So
it's
it.
It
has
a
local
effect
that
lasts
through
the
important
important
part
of
the
learning
room.
A
So
here,
in
the
terms
of
this
paper,
they
kind
of
did
make
that
distinction
there's
the
spike,
which
is
basically
a
very,
very
sharp
and
quick,
very
quick
response,
usually
about
a
millisecond
or
two,
where
you
get
this
big
response
and
it
goes
down
again.
So
this
is
what
an
action
potential
looks
like.
This
is
what
a
typical
neuron
spike
looks
like
these
regenerative
events
that
jeff
was
talking
about.
A
There's
some
internal
positive
feedback,
loop,
that's
going
on,
and
it
raises
the
epsp
for
a
sustained
period
of
time,
there's
a
there's
a
and
in
the
antique
paper.
They
have
a
really
nice
figure
on
this
here,
and
sometimes
they
call
this
a
plateau
potential,
which
means
that
the
the
epsp
or
the
the
impact
on
the
on
the
cell
is
sustained
for
a
period
of
time,
something
like
50
milliseconds
to
75
milliseconds,
and
you
can
see
here
this.
A
A
So
this
is
the
gen.
This
is
kind
of
the
bio
physical
kind
of
substrate
for
our
prediction
mechanism
and
in
our
in
the
temporal
memory,
we're
saying
if
there's
if
the
number
of
synapses
crosses
this
threshold,
like
I
showed
before
you
have
the
sustained
effect
of
the
cell
body,
which
lasts
for
a
long
period
of
time
and
during
this
period
of
time
the
cell
is
in
this
upstate
right.
It's
it's
not
at
the
point
where
afar
is
an
action
potential,
but
it's
for
a
sustained
period
of
time.
A
A
D
A
Let
me
see
if
I
can
find
a
yeah
here's
a
good
picture
of
that
on
the
so,
if
you
have,
if
you
have
a
neuron
here's,
the
cell
body,
you
have
all
of
these
dendritic
segments
and
everything
I've
been
talking
about
these
dendritic
spikes
with
these
local,
like
small
section
of
the
dendrite,
if
you
get
enough
synapses,
you
get
the
nmda
spike
right,
and
this
happens
here,
and
this
happens
up
in
these
apical
dendritic
segments
as
well.
The
calcium
spike
is
something
that
is
a
signal
that
goes
from
the
apical,
tuft
or
apical.
B
A
Just
that,
where
the
tufts
just
yeah,
I
think,
there's
a
there's
a
name
for
this
thing,
which
I
forgot
now
but
there's
a
separate
integration
zone
here
called
the
calcium
integration
zone
and
and
and
a
calcium
spike
is
a
strong
signal
from
here
down
to
the
cell
body,
and
so
this
is
a
separate
signal
completely.
So
when
we,
when
I
spoke
with
matthew
larkham
a
while
ago
when
he
visited,
he
said
you
can
almost
think
of
the
neuron
as
being
two
neurons
connected
together.
A
If
you
ignore
the
top
part,
the
apical
part,
if
you
just
look
at
the
bottom
part,
you
have
these
nmda
spikes
and
then
you
have
an
action
potential.
You
can
think
of
this
top
part
as
another
neuron,
that's
just
connected
only
to
the
cell
body
and
if
an
mba
spikes
any
of
these
calcium
spikes
that
send
a
strong
signal
down,
but.
B
So
it's
believed
to
be
there's
a
lot
of
hypothesis
about
what
the
calcium
spike
could
do
right,
but
it's
all
involved
in
learning,
and
so
it
it's
not
like
it's
just
another
way
of
transmitting
information
to
the
soma.
It's
like
saying
no,
I
I
have
detected
certain
patterns
in
the
in
the
top,
the
apical
top
and
we're
gonna.
That
means
one
thing
and
we're
gonna
start
certain
patterns
in
the
proximal
synapses,
and
that
means
another
thing,
so
you
know
so
the
the
top
gets
different
inputs
than
the
proximal
synapses.
B
A
There
and
one
of
the
mechanisms
they
found
is
that
if,
if
you
have
an
nmda
spike
here
right
that
this,
this
area
is
kind
of
depolarized
and
if
you
get
input
and
there's
an
action
potential,
there's
something
called
a
back
action
potential
is
the
signal
when
there's
a
spike
here,
there's
a
signal
that
goes
back
throughout
the
cell.
So
it
goes
back
this
way
and
you
can
see
this
arrow.
It
kind
of
interacts
with
this
calcium
zone
and
then
there's
a
this
very
strong,
sustained.
A
Signal,
if
you
will
that
got
sent
from
here
to
the
cell
body
and
that
causes
the
cell
to
fire,
multiple
spikes
in
a
row
so
functionally
we
can
think
of
it.
As
you
know,
if
the
neuron
has
detected
some
contextual
pattern
up
here
and
then
there's
an
action
potential,
then
there's
going
to
be
a
very
a
burst
of
spikes
that
happen
afterwards.
So
it's
a
coincidence
of
some
context
detected
in
this
zone,
plus
the
cell
actually
detecting
a
pattern,
causes
this
burst
of
activity.
B
A
E
A
B
A
So
when
you
look
at
michelangelo,
if
you
think
about
kind
of
what
we're
trying
to
do
with
dendrites,
this
is,
I
don't
know
any
of
what
we
said
made
sense
to
you.
But
what's
happening
is
you
know
this?
This
there
are
all
of
these
patterns
detected
independently
on
these
dendritic
segments
right
and
they
by
themselves
don't
have
a
direct
effect,
but
they
bias
the
cell
body
to
win
in
a
subsequent
winner-take-all
network.
F
E
F
A
Yeah,
so
let
me
let
me
talk
about
that
point.
So
that's
a
fallacy
that
thought
the
way
they're
talking
about
is
different
because
they're
ignoring
the
temporal
dynamics
of
what's
going
on.
So
here's.
If
you
look
at
d
panel
d,
this
is
the
typical
thinking
of
what's
happening
with
these
dendrites.
You
have
kind
of
these
threshold
things
that
are
going
on
in
all
of
the
dendrites
and
they
get
summed
together
and
then
you
do
a
normal
linear,
weighted
sum
and
you're
completely
ignoring
the
temporal
part
of
it
here.
A
This
looks
just
like
a
two-layer
network
right
and
if
you
model
things
this
way,
it's
going
to
be
no
different
than
a
two-layer
network,
and
if
it's
just
a
two-layer
network,
there's
no
reason
to
think
about
dendrites.
You
can
just
have
a
two-layer
network.
You
don't
have
to
worry
about
dendrites
yeah.
A
Exactly
but
this
ignores
a
bunch
of
critical
facts,
so,
first
of
all,
it
ignores
completely
this.
This
sustained
depolarization
thing
right,
yeah
it
the
the
fact
that
when
you
get
this
at
mda
spike,
you
get
a
sustained
upstate,
it's
going
to
affect
the
subsequent
inputs.
It's
not
about
the
current
input.
That's
coming
in
it's
the
next
time
step
in
the!
A
If
you
will,
I
mean
it's
all
analog
here,
but
it's
it's
future
inputs
that
are
going
to
get
affected
by
this
plateau
potential
right.
It's
not
about
not
about
thinking
about
a
single
feed
forward
system,
where
you
only
have
a
single
input
coming
in
and
you're
doing
this,
and
then
you
do
this,
and
then
you
get
the
output,
that's
not
what's
going
on
here!
You
have
to
look
at
the
temporal
dynamics
right
and
the
other
part
of
it
is
that
these
events
are
extremely
rare
and
they
don't
generally
cause
the
neuron
to
spike.
E
A
B
There's
another
big
thing
too,
which
I
think
I
think
you
forgot
to
mention
this
is
so
what
they're
proposing
is
each
of
those
orange
blocks
that
are
labeled
nmda
are
just
like
a
point
neuron
and
but
something
that
happens
here
is
when
the,
when
the
body
of
this
neuron
generates
an
action
potential.
So
there's
a
spike
that
goes
down
with
the
axon.
B
A
B
A
B
F
B
B
A
C
In
terms
of
the
strikes
me
about
that
you're
saying
that
the
current
machine
learning
doesn't
capture
the
temporal
dynamics,
machine
learning
basically
depends
upon
synchronous
logic.
Everything
happened
at
the
same
time,
but
there's
a
whole
branch
of
asynchronous
logic
that
could
capture
some
of
this
stuff.
I'm
curious
if
any
of
the
neuromorphic
guys
are
actually
trying
to
do
that
where,
basically,
you
have
self-time
circuits
and
the
clock
is
not
distributed
globally.
It's
it's
going
from
point
a
to
point
b
and
then
in
any
kind
of
complicated
pattern
back.
C
So
when
you
talked
about
that
back
propagating
signal
going
backwards.
Basically,
what
what
happens
is
that
you
try
to
avoid
it
in
in
logic
design,
but
you
get
these
things
called
like
rut
pulses
and
various
other
things
where
they
you
have
race
conditions,
and
in
some
of
these
cases
here
those
would
actually
be
meaningful
because
the
temporal
dynamics
becomes
important
rather
than
something
to
be
avoided.
B
I
think
neuroscience
researchers
wouldn't
be
looking
at
that
at
all
correct
me
if
you
think
I'm
wrong,
sometimes
yeah.
These
are,
like
you
know,
computational
ideas
from
computer
scientists
that
you
know
the
realm
of
all
possible
things
we
might
build.
But
neuroscientists
don't
look
about.
Oh
here's!
What
these
neurons
do
they
build
these
models
of
these
neurons
using
very
detailed
biological
simulations.
B
You
know
they
literally
place
every
ion
channel
on
a
cell
and
they
model
the
effects
of
all
the
ion
channels
and
so
on.
So
they
don't.
I
don't
think
they
think
along
those
lines
to
them.
It's
like
it's.
It's,
like
you
know,
they're,
not
looking
for
ways
a
neuron
under
some
strange
set
of
circumstances
could
do
something
they're,
saying
here's
what
it
looks
like.
B
What
does
it
do,
and
so
I
think
that
question
maybe-
and
I
think,
when
you
talk
about
neuromorphic
people,
mostly
mostly
neuromorphic
researchers,
are
trying
to
capture
what
is
actually
happening
in
the
biology
of
the
neuron,
and
so
I
don't
think
they'd
be
thinking
along
those
lines
that
you
mentioned,
that
that
stuff
doesn't
come
out
of
just
like
modeling
minerals.
It
would
come
out.
It's
more
of
a
conceptual
computer
scientist.
C
A
D
B
At
like
the
chip
that
ibm
and
other
places,
but
they
they
the
first
thing
they're
trying
to
model
neuromorphically
is
the
action
potential
as
if
yeah
that's
the
key
component
right
away,
our
cells
actually
spike.
So,
let's
add
spiking
to
our
neurons.
I
think
it's
totally
off
base.
That's
not
the
critical
thing,
but
that's
the
way
of
approaching
it
and
subverting
right
now.
They
don't
have
anything
like
dendrites
or
anything
like
them.
That.
A
B
B
A
A
B
E
A
So
it's
like
decades
and
decades
and
decades
of
work-
you
know,
didn't
have
any
of
this
stuff
and
it's
just
even
even
now
in
the
neuroscience.
I
think
this
has
become
more
accepted
over
the
last
few
years,
but
it
certainly
hasn't
come
into
the
machine
learning
world
at
all
into
theoretical
world.
Well,.
F
A
Exactly
and
and
still
the
prevailing
and
even
within,
even
within
the
dendrite
community,
this
d
is
still
the
prevailing
thinking.
You
know
so
this
the
stuff
I
mentioned
about
temporal
dynamics.
Everyone
is
speaking
in
the
dendritic
world
will
say:
yes,
that's
there,
the
sustained
depolarization
is
there,
but
their
mental
model
still
reverts
back
to
this.
Oh,
it's
just
a
three-layer
model
and
so
there's
a
distant
disconnect
there
with
when.
So
they
think
this
is
really
exciting.
A
It's
a
three-layer
model
that
means
that
neuron
is
more
powerful
than
you
used
to
think.
It
was
the
point
neuron
to
a
machine
learning
person.
If
it's
a
three-layer
model,
it's
completely
uninteresting
right
because
then,
okay,
you
just
add
more
layers
in
your
model.
You
already
have
hundreds
of
layers.
It's
just.
It
is
completely
uninteresting
to
a
machine
learning
person
that
this
is
three
equal
to
a
three-layer
model.
A
A
View
this
is
a
purely
a
capacity
issue
and
it's
a
small
capacity
issue
and
it
has
no
impact
when
you're
dealing
with
when
you're
creating
models
with
hundreds
of
millions
of
weights
in
it.
It's
zero
impact.
Basically,
so
there's
a
big
disconnect
between
how
the
dendritic
world
talks
about
this
and
they're
they're
to
them.
A
This
is
very
important
because
they're
trying
to
understand
how
a
neuron
works-
and
this
means
a
neuron-
can
do
more
than
they
thought
it
could
and
that
a
neuron
might
be
able
to
do
an
xor
in
the
dendrites
or
you
know
things
that
previously
used
to
require
two
layers
and,
and
so
on
could
be
done
in
a
single
neuron.
That's
really
exciting
gets
tons
of
twitter
mentions
and
all
of
this
stuff
to
machine
learning,
people,
that's
totally
boring.
You
know,
xor
is
like
40
years
old.
Who
cares
about
xor?
B
E
B
A
E
F
A
A
You
know
it
would
have
to
go
back
and
update
the
segment
that
was
responsible.
Just
for
that
particular
thing
again.
Some
of
this
might
come
out
automatically
with
backdrop
through
time.
Some
of
it
might
not.
I
think
you
have
to
be
just
really
careful
about
the
details,
but
yeah.
There
definitely
is
a
way
to
to
create
the
circuit
for
doing
it.
C
One
of
the
things
that
I
saw
at
these
harvard
conferences
is
a
convergence
to
analog
computation,
because
when
you
look
at
memories
and
and
and
static,
non-volatile
memories
there's
a
huge
analog
component
that
they
try
to
abstract
away.
But
by
relaxing
some
of
those
conditions
they
actually
talk
about
doing
computation
and
memory.
C
You
know
doing
summing
along
these
buses
and
along
those
lines,
but
very
regular
structure,
of
course,
because
that's
what's
easy
to
lay
out,
but
I
think
they're
opening
their
minds
the
possibilities
now
of
analog
computation
and
when
you
have
that,
there's
going
to
be
inherently
a
temporal
component
to
that
and
some
of
these
models
that
you're
showing
here
could
find
a
home
in
some
of
those
concepts.
The
fact
that
they're
developing
processes
and
putting
them
on
ships
now
means
and
putting
mem
wristers
in
there
as
in
in
the
mix
as
well.
B
It's
going
to
be
a
proof.
I
have
a
you
know,
I'm
a
you
know,
I'm
a
commercial
about
these
things
kevin.
You
know.
The
problem
I
see
is,
you
know,
there's
a
whole
bunch
of
people
who
love.
Mrs
ambrose's
are
cool
devices,
and
so
they
want
to
make
them
work.
They
want
to
put
them
into
the
systems
and
but
it's
very
misleading
from
a
from
a
computational
point
of
view.
I'd
rather
not
have
any
analog
processes
going
on
and
I'm
not
sure
we
need
them.
B
We
can
model
digitic
spikes
as
a
almost
like
a
binary
thing.
It's
a
spikes!
Therefore,
it's
a
it's
a
non-linear
system
and
you
know
we
model
synapses.
We
what
we've
learned
and
other
people's
proposes
too,
that
synapses
are
really
best
viewed
as
binary
as
well.
You
can,
you
can
think
of
them
as
non-binary,
but
but
really
when
you
understand
how
the
whole
thing
works,
they
don't
they're
not
being
the
system
doesn't
treat
them.
That
way,
it's
more
of
a
coincidence
detection.
B
So
we
have
to
be
very
skeptical
when,
when
the
device
groups
get
together
and
they
propose
ways
of
doing
things
that
they're
coming
in
with
their
own
bias
they're
coming
with
what
they
want
to
build,
you
know
if
you're,
an
amerister
researcher
for
the
last
20
years.
You
definitely
want
to
figure
out
how
to
put
those
into
these
systems,
but
and
they'll
look
away
for
doing
that
stuff.
I
I'm
just
not
you
know
until
they
can
prove
that
we
need
these
analog
processes.
Theoretically,
I'm
very
I'm
very
nervous
or
hesitant
about
adopting
this
stuff.
A
It
could
be,
I
think
the
burden
of
proof
would
be
on
them
is,
is
you
know
we
can
come
up?
We
have
a
functional
model
of
how
these
things
work,
including
these
time,
delays
and
and
learning
and
all
of
this
stuff,
and
so
I
guess
it
would
be
up
to
them
to
say.
Okay.
This
is
a
more
efficient
way
to
implement
it.
It's
not
like
it's
going
to
be
it's
not
like
you
have
to
have
that
implemented.
B
Yeah,
but
just
take,
let's
look
at
amristers
as
a
specific
example,
one
of
the
things
that
the
people
of
study
imresponses
are
trying
to
do
is
they're
trying
to
get
a
greater
range
of
memory
or
activation
scale
right,
they're
trying
to
say.
Oh
we're
trying
to
model
these
kind
of
synapses
that
are
being
used
in
neural
networks.
They
have
three
forward
reducer
precision
the
emirates
only
have
you
know
how
much
precision
so
they're
trying
to
get
to
a
more
analog
representation
of
weight.
B
If
you
will,
where
I
think,
that's
actually
a
completely
wrong
direction
to
go
that
that
they
shouldn't
be
trying
to
do
that,
they're
trying
to
match
what's
going
on
neural
networks,
but
that
doesn't
really
match.
What's
going
on
in
the
brain
where
the
individual
synapses
are
very
stochastic
and
they
look
more
like
binary
things
than
anything
else
again.
C
B
C
That
well
yeah,
but
if,
if
if
you,
if
you
look
at
the
fact
that
they've
only
looked
at
you
know
these
point,
neurons
yeah,
of
course,
they're
going
to
be
off
the
wrong
track.
But
I'm
I'm!
What
I'm
saying
is
that
the
infrastructure
is
becoming
available
now,
because
that's
being
driven
by
these
other
forces
that
it's
you
know
finding
the
right
person
to
talk
to.
They
could
probably.
A
A
C
C
A
C
B
Yeah,
the
problem
is,
they
don't
have
those
insights
and
and
without
them
they
go
off
in
crazy
directions
and
another
direction
that
has
been
very
prominent
over
the
years
is
the
this
idea
that
inter-spike
timing
intervals
are
encoding
information
that,
from
an
efficiency
point
of
view,
you
can
get
a
hell
of
a
lot
more
information
across
from
a
neuron
if
you're
timing,
the
individual,
the
timing
between
individual
sites
that
those
the
timing
between
the
spikes
matter
and
there's
a
whole
little
subfield
of
people
who
focus
on
that.
It's
from
an
efficiency
point
of
view.
B
It's
true,
but
I
don't
think,
has
anything
to
do
with
the
way
brains
work.
So
if
you
you,
you
you,
you
just
got
to
make
sure
you're
pairing
that
you
know
what
you're
working
on
with
what
we
know
theoretically
has
to
happen,
and
so
I
think,
almost
all
that
effort
that's
been
put
into
those
things
is
useless.
Well,.
B
Yeah
but
that's
a
very
different
type
of
timing:
that's
not
the
interest!
Timing
interval.
That
is
the
coincidence
of
multiple
spikes
at
different
at
different
frequencies.
That
is
essential,
so
we
can
say
that's
important,
but
when
I
look
at
the
output
of
an
individual
neuron,
I'm
not
encoding
mooring,
you
could
say
okay,
I
have
four
spikes.
B
If
I
by
timing
and
putting
the
relative
timing
between
the
spikes
at
different
sizes,
I
can
decode
a
hell
of
a
lot
of
information,
and
so
that's
what
people
are
arguing,
but
that's
that's
not
what's
going
on
in
the
brain,
whereas
the
actual
relative
phase
of
two
spikes,
as
we
see
going
on
in
the
in
this
gamma
system,
we're
talking
about
well,
that
is
true.
So
again
it's
just.
We
have
to
prepare
what
we
know
what's
actually
happening
with
the
brain,
with
what's
theoretically
required,
that
we've
figured
out
that
this
is
important.
C
B
Yeah
again,
we
have
to
go
back
to
what
do
we
know
is
actually
going.
What
is
the
process
that
actually
makes
the
difference
in
the
brain
and
in
intelligent
machines?
And
if
you
just
take
it
from
a
pure
shan
information
theory-
or
you
know,
some
sort
of
mathematical
construct,
you'll
end
up
with
completely
different
constructs,
completely
different
ideas,
and-
and
I
think
this
has
been
sort
of
the
this
has
been
the
failure
of
brain
theory
in
some
sense
or
the
failure
of
ai.
B
There
hasn't
been
this
sort
of
merging
of
like
what
do
we
know.
What's
actually
going
on
that
matters
which
of
these
things
don't
matter,
and
then
you
know
it
just
hasn't
been
that
before
you
know,
when
we
put
out
our
neuron
paper,
which,
for
the
first
time
proposed
this
this
this
timing
difference
between
the
initial
spike.
B
A
B
Yeah
when
this
guy's,
like
a
dendrite
researcher-
and
he
like
he,
was
so
ecstatic-
like
my
god
for
the
first
time,
someone
can
explain
how
our
work
matters
and
why
it's
important-
and
you
know
it
was
the
first
time
that
we
sort
of
provided
this
theoretical
basis
for
what
they
were
doing
up
to
that
point.
No
one
had
any
idea
what
these
nmda
spikes
are
good
for.
E
A
That
neuroscientists,
you
know,
would
also
be
able
to
do
some
of
that,
but
you
can
see
here.
You
know
the
thing
that
I've
highlighted.
You
know
these
nmda
plateaus
could
help
sustain
these
network
upstates.
That's
what
that's!
What
we've
been
saying
is
that
you
know
in
the
machine
learning
thing
context
that
would
be
an
nmda
plateau
would
be
detecting
a
contextual
pattern
and
then
upstate
is
biasing
it
to
win
or
to
fire
or
whatever.
B
A
And
yeah,
the
other
thing
you
know
in
our
theory
when
these
things
fire,
when,
if
they
have
an
nba
spike,
they
will
fire
sooner
and
therefore
they
will
win
in
the
winner.
Take
all
competition
afterwards
and
what
ends
up
happening
is
you
have
a
very
you,
have
a
very
sparse
subset
of
the
network?
That's
actually
active
and
the
exact
subset.
That's
for
any
given
input
you
keep
the
input
constant.
The
exact
subset
that
becomes
active
is
completely
dependent
on
the
context
right.
So
this
is
what
I
was
saying
a
few
weeks
ago.
A
Is
you
know?
One
way
to
think
about
this
is
that
dendrites
are
activating
very
sparse
sub
networks
within
this
much
larger
networks
and
depending
on
the
context,
you
invoke
a
very
different
sub
network,
and
so
that
ties
into
some
of
the
super
masks
papers
and
potentially
some
of
the
attention
effects
and
transformers
and
all
of
that
stuff,
and
it's
just
another
way
to
explain
what
what
our
temporal
memory
does.
E
B
Let
me
just
say:
like
take
our
take
our
sequence
number
right.
One
of
the
things
our
sequence
memory
doesn't
do
at
all.
Is
timing,
there's
no
concept
of
the
timing
between
individual
individual
events
and
sickness.
So
we
know
that
has
to
be
there
and
I
am
virtually
certain
that's
happening
through
the
apical
density.
B
So
so
we
know
there's
a
reason
for
doing
that.
We
know
that
we
we
have
a
place
for
where
that's
going
to
occur.
So
it's
not
like
we
just
totally
ignore
it.
It
just
we
didn't
deal
with
that.
Yet
I
just
I
just
want
to
point
that
out
that
you
know
just
because
we
haven't
incorporated,
it
doesn't
mean
we
haven't
thought
about
it,
that
there
are
things
like
there.
We
know
that
have
to
happen
right.
So
I'm
sorry,
I've
been
yeah.
E
A
E
A
A
That's
at
least
some
of
what
we
know
so
far,
so
you
can
think
of
you
know.
Maybe
you
know
we
could
model
it
potentially
by
having
a
second
set
of
dendrites.
You
know
here
and
we'd
have
to
think
about
what
the
learning
implications
are
as
well,
but
that's
one
way
we
could.
We
talked
about
this
in
our
temple
in
our
sequence,
memory
paper
as
well.
E
Okay,
all
right,
that's
something
to
think
about.
Second
question
is
in
this
latest
model
and
and
again
not
about
hdm,
but
the
model
we're
proposing.
Now
we
don't
sustain
activations
from
the
dendritic
branches
like
crowd
times
that
I
mean
yeah.
That's
that's
another
thing,
that's
different
from
what
you're
showing
right.
A
B
So
so
today,
yeah
we're
just
done
with
you
know,
backprop
or
air
propagation
or
whatever
that's
great,
but
ultimately
we
have
to
keep
in
mind
where
our
ultimate
destination
is,
and
so
many
of
these
dynamic
properties
are
going
to
be
essential
in
a
movement
situation
where
you
know,
patterns
are
changing
over
time
rapidly.
All
the
time.
E
All
right
well,
on
monday,
we're
going
to
have
vivian
presenting
and
what
the
framework
is
working
on.
It
involves
a
sensor
motor
behavior
like
an
agent
learning
environment.
Maybe
we
can
think
about
applying
this
direct
networks
there,
and
then
we
can
have
some
actual
data
of
what
we're
doing
right,
what
we're
not
doing
and
then
we're
going
to
need
to
implement
those
things
in
order
to
get
it
working
right,
yeah.
It's.
A
I
think
it's
a
pretty
important
point,
but
it's
it's
a
it's
a
you
know
a
more
subtle
point
than
what
we've
been
talking
about,
which
is
you
know,
there's
a
bunch
of
work
on.
You
know
how
many
synapses
do
you
need
to
initiate
an
nmda
spike
and
the
basic
thinking
is
somewhere
around
10
inputs
coming
in
into
the
same
segment
or
dendritic
branch
can
actually
cause
this
dendritic
spike,
and
I
think
they
actually
even
talk
about
it
a
little
bit
here,
but
this
is
very.
This
is
very
weird
like.
A
How
can
you
know
you
have
thousands
of
inputs
coming
in?
You
know
what?
How
can
10
inputs
possibly
be
something
meaningful,
and
I
forget,
if
the
the
language
in
here
it's
again,
it's
it's
it's
kind
of
hard
to
I'm.
Sorry,
I'm
not
sure
if
I'm
making
much
sense,
but
the
the
idea
is
here
is
that
you
know
if
you
have
a
yeah
if
a
thousand
inputs
coming
into
the
the
neuron,
you
know
what
is
the
information
contained
in
10
synapses?
A
You
know
how
could
that
possibly
be
important,
and
what
we
showed
is
that
if
you
have
very
sparse
activations,
10
inputs
can
be
extremely
meaningful
and,
and
so
the
threshold
of
10
can
actually
convey
a
huge
amount
of
information
and
be
very,
very
robust
things.
You
could
actually
do
a
lot
of
computation
just
with
10
to
20
inputs.
As
long
as
you
have
a
very
sparse
activations
in
a
dense
kind
of
setup,
you
don't
you
don't
get
those
benefits.
B
The
question
then
becomes
well
how
many
of
those
neurons
do.
I
have
to
detect
being
active
to
know
that
the
pattern
exists.
What
if
I
only
connect
to
10
of
them,
so
I'm
only
looking
at
10
of
them
and
not
400
of
them.
Well,
I
have
a
lot
of
possibilities
for
error.
There
right,
there's
a
huge
number
of
patterns
that
could
match
the
10.
B
That's
not
the
400.,
but
when
you
do
the
math
and
you
just
think
about
it
logically,
you
realize
that
the
chance
of
getting
it
wrong,
even
though
you're
only
looking
at
10
of
the
400,
is
extremely
low.
Yes,
there's
a
huge
number
of
patterns
that
could
match,
but
if
you
look
at
all
the
patterns
out
there,
it's
almost
certain
that
you're
not
going
to
match
incorrectly.
B
So
it's
just
this
idea
that
yeah,
if
it's
sparse,
then
I
only
have
to
look
at
find
a
few
that
are
coincident
and
you
can
do
this
calculation
in
the
back
of
an
envelope.
It's
not
hard
to
do
to
show
that
it's,
it's
extremely
unlikely
that
you'll
get
a
a
a
false
positive
and
so
10
is
good
enough
or
usually,
if
you
say
20,
but
so
that's,
maybe
it's
counterintuitive
to
people
a
certain
one.
Obviously,
when
you
start
thinking
about
this
and
then
you're
like.
E
C
B
Yeah
yeah
yeah
totally
so
you
have
to
like.
I
we
our
assumption.
Is
you
don't
by
random
chance
having
to
have
the
right
number
right?
You
know
it's
like.
Oh,
I
already
have
a
dendrite
branch
that
has
the
right,
20
synapses,
that's
very
unlikely
to
occur,
but
it's
very
likely
to
occur
that
you
might
have
a
dendritic.
Dendritic
branch
has
five
of
the
right
synapses,
that's
not
so
unlikely,
and
so
there's
a
balance
there,
and
so
I
don't
know
if
we've
walked
through
this
in
great
detail.
B
But
the
basic
theory
we
have
is
that
some
dendritic
branch
is
going
to
be
more
matching
than
other
vendors.
Just
by
randomness,
you
could
say
all
right:
some
are
going
to
have
more
synapses
that
just
by
chance
have
the
and
so
we're
going
to
build
on
those
right.
Why
don't
we
we'll
start
with
that?
One,
and,
and
so
you
can
imagine
a
cell
fires
that
generates
an
action
potential.
The
action
potential
sends
a
back
ax,
the
juncture
back
to
the
to
the
branches
of
the
synapse.
B
Even
if
not
none
of
those
branches
actually
generated
an
nmda
spike
either
none
of
them
actually
had
enough
synapses.
They
will
have
a
local
depolarization
event
that
was
significant
and,
and
you
would
try
to
know,
then
they
would
sort
of
try
to
learn.
I
said:
hey,
I
was
close
to
generating
a
dendritic
spike,
but
I
didn't
those
synapses
would
be
enforced
and
they
would
grow
new
ones
and
say:
okay,
let's
start
growing
new
synapses
or
turning
off
some
ones
that
are
silent
or
something
like
that.
B
There's
about
40
synapses.
You
can
get
in
40
microns,
so
it's
unlikely
that
dendrite
will
recruit
more
than
40..
It's
just
like
that
gets
too
big,
so
you
kind
of
have
an
upper
limit
of
about
40
synapses
and
you
know
about
20
is
plenty
good
and
some
such
as
10
is
okay,
but
wanted
to
be
better,
and-
and
so
you
can
imagine
that
you
know
some
branches
will
say:
okay,
I'm
going
to
start
learning,
but
it's
not
going
to
keep
adding
more
and
more
synapses
over
time
it
sort
of
maxes
out.
If
you
will.
A
Yes,
the
the
slide,
I'm
showing
here
talks
about
the
a
couple
of
some
of
the
papers
that
talk
talk
about
that
phenomenon.
It's
sometimes
called
sub
threshold
ltp
in
this
in
this
particular
sub
community.
So
basically,
there
was
no
dendritic
spike,
but
you
still
want
learning
to
occur,
and
this
often
requires
this.
You
know
small
number
of
synapses
to
be
clustered
together.
A
Just
like
jeff
would
say,
you
know,
be
unlikely
to
get
10
by
random,
but
you'll
get
a
small
number
of
them
and
if
you
get
specific,
you
know
if
you
get
a
small
cluster
learning
can
start
happening
and
you
can
start
recruiting
more
and
more
synapses
onto
that
segment
over
time.
So
there's
some
there's
not
a
lot
of
papers
on
this,
but
there's
I
think
you
hear
a
couple
of
papers
and
then
there
are
a
few,
a
couple
more
that
are
more
recent
than
this
as
well.
C
B
C
A
B
Yeah
but
kevin,
I
think
I
have
a
good
insight
there.
First
of
all,
it's
super
change.
There's
nothing,
no
one's
more
significant
than
another
one.
The
question
is:
can
you
get
enough?
Can
I
form
enough
synapses
that
I
have
20
and
what,
if
I
just
don't
what,
if
the
axons,
I
need
to
connect
to
don't
go
close
enough
to
me
right.
So
this
is
a
another
little
subfield
of
research
called
potential
synapses,
meaning
what
how
many
it's
not
like.
B
This
is
a
whole
another
little
minor
subfield
of
neuroscience
where,
when
you
look
at
there
are
when,
when,
if
you
look
at
hepatic
learning,
you've
got
a
dendrite
and
an
axon,
and
you
can
say
well
they're
really
close
to
each
other
on
my
chronic
heart.
Well,
they
can
easily
form
a
thin
ass,
but
what,
if
they're
further
away?
B
You
know
you
can
say:
okay,
I
know,
there's
an
axon
over
here,
there's
a
dendrite
over
there.
You
guys
aren't
close
enough
to
form
a
synapse.
I
can
detect
that
I'm
going
to
actually
make
you
grow
towards
each
other,
and
so
you
can
essentially
the
growth
of
the
dendrites
and
the
growth
of
the
ends
of
the
axons
are
not
random.
They
are
directed
to
a
large
extent
to
try
to
move
towards.
B
Yes,
so
so
that's
there
aren't
many
people
who
study
that,
but
there's
this
lot
of
evidence
that
this
is
going
on
and
they
and
and
we
know
that
the
the
ends
of
the
axons
and
the
dendrites
are
not
static.
They're,
constantly
moving
about
they're,
going
sideways
and
so
on.
This
whole
stuff
is
like
squirming
worms
in
there
right,
they're,
trying
they're
trying
to
find
the
right
connections.
So
all
this
all
these
things
were
necessary.
B
When
we
came
up
with
the
sequence
memory
theory,
I
wrote
in
the
book
you
may
not
remember,
but
I
wrote
the
book
is
that
when
we
came
up
with
the
series
a
whole
bunch
of
things
we
needed
dendrites
to
do
and
we're
talking
about
a
lot
of
them
here
and
we
didn't
know
they
could
do
these
things,
and
so
we
had
to
go
back
and
look
through
all
this
literature
to
find
out.
Is
this
really
possible?
You
know:
can
they
form
these
synapses?
B
Can
you
know
the
action
potentials
reach
them?
Can
you
know
can?
Can
we
get
enough?
What's
the
minimum
number
of
synapses?
We
need
all
these
kind
of
things,
and
so,
but
theoretically
it
was
almost
certain,
it
had
to
happen,
and
so
it
was
like
it's
so
theoretically
powerful
that
this
is
almost
certainly
going
on.
Let's
find
the
evidence
for
it
and
we
have
to
really
dig
and
that's
what
some
of
these
papers
are.
We
have
to
dig
deep
to
say:
is
this
really
possible
yeah?
It
is
possible.
A
Yeah,
I
think
it's
it's
it.
You
know
you
could
see
how
you
could
quickly
get
dragged
into
like
so
much
detail.
You
know
each
of
these
things
are
like
little
subfields,
and
so
I
think,
from
a
machine
learning
standpoint,
it's
almost
like
a
never-ending
set
of
things
and
then,
at
the
end
of
it,
you're
left
completely
confused
as
to
what
does
it
actually
mean
once
you
start
going
down
these
things,
and
so
I
think
it's
important
to
sort
of
step
back
and
and
just
think
about
functionally.
A
A
What
you
know
what
the
meaning
of
it
is
and
what's
important
about
it,
and
it's
just
really.
I
think
it's.
I
think
it's
actually.
If
you
want
to
understand
it
from
a
machine
learning
standpoint,
you
have
to
really
the
trick
is
to
know
what
details
are
important
and
what
is
not
yeah.
You
have
to
be
direct,
that's
really
hard
to
get
from
these
papers.
I
have
a.
I
have
a
book.
B
C
C
B
So
what
are
the
advantage,
of
course
in
in
our
in
our
machine
intelligence
effort?
We
don't
have
to
do
any
of
that.
You
know
we
can.
Just
I
mean
we
have
restrictions
on
what
we
can
connect
to
what,
but
those
are
the
restrictions
imposed
by
like
fpga
or
memory
architecture,
or
something
like
that,
but
in
general,
if
you
think
about
my
software
point
of
view,
we
can
make
any
number
of
connections
we
need.
B
So
in
our
simulations
of
the
temple
memory,
we
just
randomly
choose
from
the
from
the
from
the
active
active
20
synapses.
Let's
just
pick
20
active
cells
when
we
make
connections
to
them
where
brains
are
going
to
be
slower
and
brains
are
going
to
have
to
grow.
These
connections
and
brains
are
going
to
be
struggling
to
do
this,
where
we
can
just
do
it
instantly
from
a
theory
point
of
view.
B
A
E
A
A
F
I
definitely
found
it
helpful
because
I
thought
the
presentation
by
larkum
the
other
day
was
or
a
few
weeks
ago
was
yeah
really
interesting,
but
that
was
kind
of
more
focused
on
the
biology
and
was,
I
think,
here,
kind
of
using
the
language
of
machine
learning
and
yeah.
That
kind
of,
like
I
think,
you're
hoping
to
do
kind
of
distills
away,
what's
actually
relevant,
to
designing
systems
to
perform
tasks.
B
This
is
the
I
think,
the
classic
problem
of
of
discovery
and
science,
where,
when
you
don't
understand
something,
you
have
a
huge
amount
of
detail,
a
huge
amount
of
facts
and
no
one
knows
which
of
those
facts
are
important,
which
ones
was
actually
not
important
and
which
of
the
details
are
going
to
matter
which
detail
is
not
going
to
matter
this.
Is
you
pick
any
endeavor
in
scientific
inquiry
before
people
understood
something?
That's
what
it
looks
like.
You
have
huge.
B
Data,
no
one
can
agree
about
what
the
data
means
and
which
is
important,
and
when
you
have
a
theoretical
framework
that
comes
along
that
explains
like
okay.
We
now
know
basically,
what's
going
on,
that
framework,
tells
you
which
these
details
matter,
which
of
these
details,
don't
and
until
you
have
that
framework.
It's
all
yeah
everyone
for
themselves.
You
know
so,
and
I
think
that's
our
role.
Our
role
is
to
say
we
have
a
framework.
We
understand
more
better
than
pretty
much
anybody.
B
What
the
functional
components
are
and
why
they're
important
and
said
we,
our
job,
is
to
sort
of
try
to
get
people
to
think
about
it
our
way
and
then
then
they
can
on
their
own
decide
like
hey.
I
guess
some
of
these
details
are
really
important.
Some
of
them
aren't,
but
until
you
do
that
it's
you
know,
everybody
has
their
own
opinion.
C
A
A
B
C
B
B
What's
the
representational
capacity
of
64
binary
digits
right,
it's
that
level,
but
this
is
what
it
is
in
the
brain.
The
brain
is
no,
you
need
to
understand,
sparse,
distributed
representations
and
what
happens
when
you
have
a
sparse
representation
for
a
while?
I
was
giving
in
this
talk.
I
had
to
talk
about
sparse,
distributed
representations
and
all
I
would
the
whole
talk
was
on
sparse,
distributed
representations,
and
I
would
argue
about
this
is
the
future
of
a.I.
B
E
B
C
Seriously,
though,
like
I
could
think
of
some
animations,
that
would
help
illustrate
that
you
know.
Basically,
you
know
you
show,
you
know
what
the
possibilities
are
for.
You
know
like
you,
know,
20
connections
and
you
go
blah
blah
blah
blah
blah
and
you
just
keep
hitting
the
people.
The
fact
all
these
things
are
not
interfering
with
each
other
and
they
start
getting
an
idea
that
you
know.
How
can
that
be
true
and
then
you've
then
you've
peaked
your
curiosity,
because
you
you've
handed
them
a
powerful
concept
that
they
say.
C
Theory
here
yeah,
but
I
could
I
can
visual,
I
mean
taking
that
paper
and
taking
that
math,
and
just
I
mean
I
I
agree
with
what
matt
did
was-
was
a
very
a
very
powerful
thing.
But
each
of
these
things,
I
think,
has
has
an
aspect
that
that
could
lend
itself
to
an
animation
yeah
and
that
that's
communication.