►
Description
Our research intern Alex Cuozzo discusses the book Sparse Distributed Memory by Pentti Kanerva. He first explores a few concepts related to high dimensional vectors mentioned in the book such as rotational symmetry, distribution of distances etc. He then talks about the key properties of the Sparse Distributed Memory model and how it relates to a biological one. Lastly, he gives his thoughts and explores some follow up work that aims to convert dense factors to sparse distributed activations.
Sources:
➤ “Sparse Distributed Memory” by Pentti Kanerva: https://mitpress.mit.edu/books/sparse-distributed-memory
➤ “An Alternative Design for a Sparse Distributed Memory” by Louis Jaeckel: https://ntrs.nasa.gov/citations/19920001073
➤ “A Class of Designs for a Sparse Distributed Memory” by Louis Jaeckel: https://ntrs.nasa.gov/api/citations/19920002426/downloads/19920002426.pdf
➤ "Comparison between Kanerva's SDM and Hopfield-type neural networks" by James Keeler: https://www.sciencedirect.com/science/article/abs/pii/0364021388900262
➤ "Notes on implementation of sparsely distributed memory" by James Keeler et al: https://www.semanticscholar.org/paper/Notes-on-implementation-of-sparsely-distributed-Keeler-Denning/a818801315dbeaf892197c5f08c8c8779871fd82
- - - - -
Numenta is leading the new era of machine intelligence. Our deep experience in theoretical neuroscience research has led to tremendous discoveries on how the brain works. We have developed a framework called the Thousand Brains Theory of Intelligence that will be fundamental to advancing the state of artificial intelligence and machine learning. By applying this theory to existing deep learning systems, we are addressing today’s bottlenecks while enabling tomorrow’s applications.
Subscribe to our News Digest for the latest news about neuroscience and artificial intelligence:
https://tinyurl.com/NumentaNewsDigest
Subscribe to our Newsletter for the latest Numenta updates:
https://tinyurl.com/NumentaNewsletter
Our Social Media:
https://twitter.com/Numenta
https://www.facebook.com/OfficialNumenta
https://www.linkedin.com/company/numenta
Our Open Source Resources:
https://github.com/numenta
https://discourse.numenta.org/
Our Website:
https://numenta.com/
A
Well,
today,
I'm
going
to
be
talking
about
something
that
I
hope
a
decent
amount
of
you
are
familiar
with
and
that's
sparse
distributed
memory.
I
have
my
copy
right
next
to
me,
but
this
was
a
book
that
I
was
actually
pointed
to
by
the
like
one
of
the
original
numenta
papers.
I
forgot
which
one,
but
it
was
a
while
ago
it
was,
I
think,
one
of
the
first
references
listed.
A
A
Right
so
I'm
going
to
talk
about
a
few
things.
So
first
off
are
some
concepts
related
to
high
dimensional
vectors.
Again,
I
think
a
lot
of
us
should
be
familiar
with
these,
but
I
think
it's
important
to
keep
revisiting
these
sort
of
high
dimensional
vector
concepts
because
they're
just
very
important
and
just
very
unintuitive,
because
we
we
live
in
a
three-dimensional
world
and
we
just
have
a
really
hard
time.
A
A
I
think
1
000
and
so
each
you
know,
binary
vector
will
look
like
just
a
string
of
ones
and
zeros
and
of
course,
there's
going
to
be
n
capital
n
equals
2
to
the
power
of
lowercase
m,
which
is
the
number
of
you
know
bits
in
the
vector,
and
so
this
is
a
very
large
space.
It
gets
very
large,
very
quickly.
A
It's
exponential
on
the
size
of
the
address
and
we
can
define
an
important
point
as
the
origin,
and
I
denote
that
as
o
throughout
the
rest
of
the
presentation
and
that's
just
the
zero
vector
it's
just
in
this
case,
a
thousand
zeros
and
one
thing:
that's
really
nice-
that
space
has
rotational
symmetry.
So
if
we're
comparing
two
different
points,
we
can
just
run
rotate
one
point
back
to
the
origin.
A
One
way
that
you
can
do
that
is
with
the
x
or
operation,
so
just
review
of
what
the
xor
operation
is.
This
is
when,
if
you're
xoring,
x
and
y
so
x
would
be
this
0
1,
0,
1
and
y
would
be
this
bottom
0
0,
1
1,
then
the
x
or
operation
of
x
and
y
would
be
ones
at
the
bits
at
which
x
and
y
differ,
and
so,
when
you
xor
one
vector
with
another
vector,
it
tells
you
how
to
rotate
one
vector
to
get
to
the
other.
B
Before
before
we
go
on
alex,
so
I'm
very
familiar
with
this
book,
I've
read
it
multiple
times,
but
not
a
long
time,
and
I
don't
understand
the
slide.
So
I
just
I
don't
know
if
it's
important
for
me
to
understand
it
or
not
so
rotational
symmetry
because
familiar
with
concept
of
symmetry,
different
types
of
symmetry,
I'm
not
sure
exactly
what
you
mean
by
rotational
symmetry
in
this
case
right.
What
does
that
mean.
A
B
B
B
Oh
you're,
saying
you're
saying
maybe
make
that
make
a
point
the
origin
and
then
I
can
I
can.
I
can
rotate
a
cube.
I
can
imagine
the
cube.
I
can
take
any
point
point
on
a
three-dimensional
cube
and
I
can
and
sort
of
rotate
the
cube
around
that
point
and
it'll
be
it'll,
become
lined
up
again
in
a
little
bit.
Is
that
what
is.
A
That
what
you
mean,
that's
good,
the
idea
yeah.
So
the
idea
of
rotation
really
is
in
handy
when
you're
talking
about
multiple
points.
So
two
points
so
or
even
you
know
three
or
four
points.
If
you
rotate
the
entire
cube,
nothing
is
going
to
happen
that
changes
the
actual
relationship
between
those
points.
Yes,
so
that
that's
like
the
idea
here
is
that
so.
C
C
B
B
Okay,
I
got
that
that's
simple
all
right,
so
basically
we
just
saying
hey
the
concept
here.
I
know
you're
going
with
the
concept
about
the
distance,
these
points
and
how
interesting
that
is,
but
but
you're
saying
here.
If
I
have
two
points,
if
I
apply
x
or
operation
to
one
of
them,
that
gives
me
all
zeros,
then
the
distance
is
is
just
the
number
of
ones
and
the
other
one
something
yeah.
A
C
A
B
C
A
Right
so
then,
you
can
ask
how
many
points
are
of
distance
d
away
from
a
point
from
from
the
origin
right
and
again,
since
any
point
can
be
rotated
to
the
origin.
This
this
applies
to
all
points
and
that's
just
a
binomial
coefficient.
So
if
you
have
all
the
points
that
are
exactly
one
bit
away,
it's
all
the
places
where
you
have
it's
all
the
vectors
on
which
one
bit
is
on
compared
to
the
origin.
A
So
it's
a
thousand
and
when
this
is
equal
to
a
thousand
and
then
for
two,
it's
one
thousand
choose
two
and
so
on,
and
then
the
distribution
for
the
distance
between
random
points
and
the
origin.
So
you
have
your
whole
space.
You
start
at
your
origin
and
you
just
pick
a
random
point
and
what
is
the
distribution
of
the
distance
between
that
random
point
and
the
origin,
and
that
is
equal.
A
It's
basically
equal
to
the
binomial
coefficient
of
of
the
the
distance
over
two
to
the
n,
because
all
of
these
points
are
equally
weighted
right.
So
the
probability
in
in
the
book,
the
probability
of
ones
and
zeros
in
this
space
are
exactly
equal.
So
there's
fifty
percent
ones,
two
percent
zeros
this
term
just
collapses
down
to
0.5
to
the
end,
okay,
and
so
then
yeah.
This
is
the
again.
A
Okay.
So
then,
now
we
can
start
looking
at
this
graphically.
This
is
the
distribution
of
distances
for
high
dimensional
vectors
you
can
imagine
instead
of
talking
about
you,
know
random
points
and
distances
from
the
origin.
It's
just
flipping
coins
right.
It's
the
same
sort
of
it's,
this
binomial
distribution,
so
you
can
say
well.
The
number
of
points
that
are
exactly
you
know,
one
bit
away
from
the
origin
is
the
same
as
the
number
of
times
that
you'll
get
exactly
one
head
in
a
thousand
coin
flips.
A
So
that's
just
why
I
put
that
there
and
the
important
thing
here
is
that,
as
you
increase
your
dimensionality,
the
sort
of
proportion
or
the
the
scent,
the
sort
of
meat
of
the
distribution
gets
more
and
more
centered
around
the
mean
just
to
explain
this
x-axis
here.
This
is
just
normalized
so
that
they
subtract
off
the
average
and
divide
by
the
average
that
you
can
compare
all
the
different
sort
of
dimensionalities
of
the
binomial
distribution.
Otherwise,
they'd
be
centered
at
like
1.
E
A
A
There
distance
two
away
distance
three
away
up
to
distance
r
away,
and
that
is
the
same
as
taking
an
integral
under
this
probability,
distribution
going
from
the
origin
here
all
the
way
out
to
this
point
over
here,
or
this
is
all
points
that
our
distance
r
away,
not
just
point
p,
and
so
that's
that's
that
integral
and
so
for
something
like
n
equals.
Ten
thousand.
If
you
have,
this
r
is
equal
to
forty.
Seven
hundred
bit
says
forty
percent
forty
seven
percent
of
the
addresses
is.
A
You
can
imagine
that
that's
sort
of
corresponding
to
this
black
curve,
but
even
more
steep,
where
all
this
probability,
distribution
down
here
is
basically
negligible
and
then
most
of
the
points
are
just
exactly
or
not
exactly,
but
very
close
to
the
mean
away,
so
that
this
is
the
the
key
property
that
sparks
distributed.
Memory
relies
on
is
that
these
random
vectors
are
going
to
be
just
about
orthogonal
or
exactly
sort
of
the
mean
distance
expected
distance
away
yeah
and
going
back
to
this
this
ball
here.
A
One
way
to
think
about
it
is
that
if
you
pick
your
point,
you
know
the
origin
point
on
top,
then
the
distribution
of
these
sort
of
rubber
spiky
things
is
not
going
to
be
equally
distributed
around
the
ball.
Most
of
them
are
going
to
be
on
the
equator
right.
So
you
have
a
very
sparse
amount
of
points
that
are
close
to
your
point.
On
the
top
of
the
ball
and
then,
as
you
get
closer
and
closer
to
the
equator,
it
just
goes
exponentially.
A
B
D
B
Yeah,
I
I
think
about
a
totally
different
way,
and
so
I
I
don't
want
to
burden
people
with
my
way
of
thinking
about
it.
If
everyone
understands
it
this
way,
so
I
mean
when
I
read
penny's
book
first
time
I
had
to
go
through
all
this
math
and
I
it
was,
I
found
it
very
confusing
and
in
the
end
it
seemed
very,
very
simple,
and
it
goes
back
to
your
your
coin.
Toss
example.
C
B
Basically,
that's
what
this
whole
thing
is
about.
I
mean
basically,
that's
what
it
is
and
and
and
the
number
of
20
to
get
the
closer
and
closer
it
gets
to
being.
You
know
very
close
to
that
mean
number
flips.
I
resemble
flips,
so
it's
if
it's
close,
it's
obviously
everyone
else.
I
find
the
math
non-intuitive
to
think
about
it.
These
ways,
even
the
circle
thing
is
not
intuitive
to
me,
but
it's
good,
okay.
Well,
maybe
it's
obvious
to
everyone
else.
A
B
C
B
C
B
That's
not
a
sparse
representation,
so
he's
he's
we're
talking
about
these
high
dimensional
vectors
and
the
properties
they
have,
but
we're
not
talking
about
sparse
high
dimensional
vectors,
which
is
what
the
brain
uses
and
he
uses
the
word
sparse
here.
I
assume
you
get
to
this
to
talk
about
the
memory
allocation,
but.
E
This
is
worth
pointing
out
yeah.
It's
also
worth
pointing
out
that
some
of
the
operations
that
we
used
are
quite
different
and
so
hamming
distance,
for
example,
is
not
a
good
distance
metric.
I
think
for
these
scenarios,
and
particularly
at
least
the
way
we
did
in
htm-
and
so
you
know,
dot
products
or
overlap
is
a
much
better
distance
metric
and
it
has
somewhat
it's
somewhat
different.
You
get
some
of
these
kind
of
same
intuition
show
up,
but
it
is
a
very
different
set
of
operations.
It's
worth
kind
of
keeping
that
in
mind
as
well.
A
Yeah
definitely,
I
was
gonna
yeah
at
the
end
of
the
presentation.
After
I
talk
about
sdm
I'll
talk
about
some
improvements
that
other
people
have
made
and
proposals
that
work
on
sparse
representations
as
well.
So
you
can
have
a
spark
yeah,
there's
like
several
different
kinds
of
sparsity
going
on
here,
but
yeah.
What
I
I
knew
sort
of
the
the
numenta
sdr
framework
does
not
map
onto
this
dense
distance
factor.
C
A
Is
sort
of
a
very
basic
overview
of
the
the
best
match
problem
is,
if
you
have
a
bunch
of
things
to
store
in
your
memory,
we'll
say
those
are
called
these.
These
words
g
right
so
say
you.
You
store
this
handwritten
digit
one
in
your
memory
and
then
two
and
then
three
you
wanna
be
able
to
recover
these.
Given
something,
that's
you
know
somewhat
similar.
So
if
you
have
a
corrupted
version
or
a
poorly
drawn
version
of
a
one,
then
hypothetically
you
should
be
able
to
recover
the
original
stored
memory.
A
A
A
So
as
if
it's
over
a
certain
threshold,
you
can
imagine
the
neuron
firing
if
it's
below
it
doesn't
fire
and
that's
what
the
activation
is.
And
so,
if
you
think
about,
the
weights
actually
correspond
to
a
specific
address
in
the
input
space.
So
if
you
have
an
input,
there's
going
to
be
one
exactly
one
input
that
maximally
excites
this
neuron
given
its
weights,
so
the
weights
actually
parametrize
and
address
for
the
neuron
and
that
you
can
see
here.
A
A
So
if
you're
given
an
address
here,
your
neuron
will
fire
it
if
the
yeah,
if
the
radius
is
large
enough
right,
let's
see
so
here,
is
where
the
concept
of
sparsity
in
sparse
distributed
memory
comes
up,
and
that's
that
if
you
have
a
thousand
dimensional
space,
that
means
there's
two
to
the
one
thousand
possible
addresses
that
is
just
astronomically
large,
much
larger
than
the
number
of
atoms
in
the
universe
and
whatnot,
and
you
only
have
a
million
neurons
and
those
neurons
are
randomly
distributed.
Threats
distributed
throughout
the
space,
and
this
is
actually
not
even.
A
This
is
again
an
impoverished
view
of
a
thousand
dimensional
space
because
they'd
be
it's
really
hard
to
visualize
something.
That's
all
of
these
are
kind
of
equally
equidistant
apart,
but
the
idea
is
that
it's
sparse
in
the
number
of
neurons
compared
to
the
size
of
the
space,
not
necessarily
the
the
address
space.
The
address
space
is
still
dense
in
terms
of
the
ones
and
zeros
that
make
up
each
address,
but
we'll
continue
continuing
on
from
here
a
quick
little
quick
question
alex
yeah
yeah.
A
A
B
C
A
A
A
A
Okay,
so
what
is
writing
entail?
So
let's
say
you
want
to
write
a
piece
of
data
to
an
address.
So
what
you
do
is
you
take
your
data?
Let's
say
it's
a
one
right
this,
this
handwritten
digit
of
a
one
and
wt.
What's
that
wt
correspond
to
the
address
of
of
this
of
this
one
right.
So,
let's
imagine
that
this
one
was
represented
as
a
string
of
a
thousand
ones
and
zeros
that's
what
wt
is
right
here
now.
Wt
does
not
correspond
to
any
sort
of
neuron.
A
So
there
can
be
repeats
of
ones.
There
can
be
other
pieces
of
data
written
to
the
neurons
this.
This
too
is
not
in
this
access
set.
So
it's
not
getting
written
to,
but
yeah
you're
going
to
be
writing
to
each
of
these
sort
of
each
of
the
neurons
that
you're
within
a
certain
radius
of
and
those
are
actually
called
hard
locations
in
the
paper
or
not.
In
the
paper
in
the
in
the
book.
A
A
Okay.
So
then
what
does
reading
entail?
How
do
you
read
from
your
memory
once
you've
written
to
it?
So
again
you
use
your
data
as
your
address
and
remember
this
time.
It's
gonna
be
different,
because
it's
not
the
the
one
that
we
store.
It's
a
new
one
that
should
hopefully
get
us
back
to
this
original
memory.
Point
that
we
stored.
So
we
have
this
new
corrupted
version
of
one
and
we
get
the
access
overlap,
which
is
just
this.
A
So
then,
when
you
pool
a
thousand
a
thousand
neurons
times,
10
words,
that's
10
000
entries
and
then
for
all
these
entries,
you
take
the
bitwise
average,
so
that
would
be
taking
in
this
case.
It
would
be
like
overlaying
this
one
and
this
one
and
this
three
and
putting
them
all
on
top
of
each
other
and
taking
the
average,
and
then
you
can
round
up
or
down
to
like
a
zero
or
a
one,
and
that
gets
you.
A
A
But
I
should
highlight
one
of
the
intuitive
reasons
why
this
should
work.
Is
that
if
you're
reading
from
here
from
this
location
of
the
corrupted
one,
then
you're
going
to
get
some
overlap
with
these
neurons?
That
also
have
the
actual
data.
You
want
to
recover.
You're
going
to
have
a
bunch
of
other
hard
locations
that
are
giving
you
a
bunch
of
sort
of
random
noise
and
these
hard
location
neurons
that
you're
reading
from
are
sort
of
sufficiently
far
apart,
that
all
of
their
sort
of
data
is
going
to
be
extremely
decorated.
A
Such
that
you're,
basically
getting
the
average
of
a
bunch
of
random
noise,
so
you're
not
going
to
be
getting.
You
know
three
different
copies
of
this
three
in
here.
Otherwise,
you'd
expect
that
this
address
that
you're
at
would
be
closer
to
a
three
than
to
an
actual
one,
so
yeah
you're
reading
from
a
bunch
of
d
d,
correlated
multisets
and
then
taking
the
bitwise
average,
and
so
the
signal
coming
from
here
should
overpower
the
noise
coming
from
the
decorated
neurons.
D
So
I
I
have
a
question
on
on
this.
So,
let's
suppose
your
space
is
a
thousand
bit
vector
and
you
say:
okay,
I've
got
this
address.
I
want
a
query
to
see
to
kind
of
collect
all
the
guys
who
are
within
distance
one
of
me,
so
that
means
you've
got
to
query
a
thousand
locations.
You
you
take
each
of
the
address
bits
flip
it
by
one
and
test
to
see.
If
something
is
actually
there,
is
that
how
and
then
then
you
just
for
those
that
are
there,
then
you
collect
their
inputs.
C
A
Well,
yeah,
I
mean
the
idea:
is
that
it's
a
lot?
It's
a
lot
of
operations,
so
yeah,
so
yeah
I'll
go
into
the
next
slide.
It's
a
lot
of
operations,
but
so
part
of
it
can
be
sped
up
by
the
fact
that
you're
storing
a
running
average
instead
of
a
bunch
of
actual
multi-sets
and
then
the
other
part
of
it
is
that
it's
supposed
to
be
done
in
parallel.
D
C
D
E
E
E
A
C
B
B
D
Well,
yeah,
I
mean
I
mean
hilbert
spaces
are
great
theoretical
concepts,
but
you
don't
want
to
implement
anything
in
them.
So
that's
that's.
Why
I'm
just
kind
of
looking
to
see
this
is
like
a
discreet
version
of
a
hilbert
space,
so,
okay
I'll
I'll
I'll
back
off,
it's
I'm
going
down
here
and
just
kind
of
trying
to
figure
out.
You
know
how
I
would
actually
you're.
B
So
all
right
I
mean
the
intuitions
you
get
from
this
are
very,
very
important,
and
the
intuitions
from
that
earlier
chart
about
the
the
distance
of
points
in
a
high
dimensional
space
are
fundamental
to
the
work
we
did
and
we're
doing
I
mean
this
is
the
you
know.
This
is
the
first.
For
me,
this
is
the
first
interesting
thing
I
read
about
brains,
it
wasn't
even
a
book
about
brains,
but
it
was
talking
about
talking
about
properties
of
spaces
which
were
very
useful.
C
D
B
E
B
I
mean
I
don't
take
away
from
this.
It's
a
very
influential
book.
For
me,
too,
the
point
was
in
the
end.
The
thing
I
walked
away
with
was
that
one
intuition
about
the
distances
in
high
dimensional
spaces
and
that's
the
only
part
we
actually
needed
to
really
understand
the
sdrs
and
all
that
stuff,
so
the
rest
of
it
doesn't
really
pertain
too
well
to
brains
or
artwork,
but
it's
still
interesting.
So
I
think
it's
interesting
that
alex
is
going
through.
It.
B
D
B
Even
think
that's
right
to
me,
and
maybe
maybe
someone
else
will
disagree
with
me.
The
whole
concept
of
memory
in
this
space
is
different
than
the
memory
concepts
I
think
about
in
the
brain
and
str's
and
so
on.
It's
really
just
the
mathematical
properties
of
high
dimensional
spaces,
which
are
the
the
key
item,
but
the
way
these
memories
work,
I
don't.
B
I
don't
that
I've
never
never
thought
about
the
brain
and
sdr's
any
of
our
work
like
that
at
all.
Maybe
maybe
there
is
a
correlation,
but
but
I
don't
think
that's
useful
to
think
of
it.
That
way.
It's
really
just
a
useful
thing.
Like
oh
shoot,
these
high
dimensional
spaces
are
very
unintuitive
and-
and
you
can,
you
can
basically
just
just
walk
away,
saying
I
have
a
high
ventral
space
and
and
and
bits
get
corrupted,
I'm
you
know
I'm
still
going
to
be.
I'm
still
going
to
get
the
right
answer.
E
Are
what's
really
important
here
and
just
as
jeff
mentioned,
that
the
distances
in
high
dimensional
spaces
can
be
very,
very
have
very
non-intuitive
properties
and
those
properties
can
be
extremely
powerful.
That's
really
the
takeaway
from
this
book,
and
now
you
can
apply
that
intuition.
Okay,
given
that
thing,
how
do
we
now
compute
deal
with
dimensionality
in
other
contexts,
and
you
can
look
at
sdrs
and
try
to
do
it,
but
I
don't
think
you
can
it's
easy
to
take
the
literally
what
he's
doing
and
kind
of
try
to
convert
it
to
what
I
don't
think.
E
A
Yeah,
actually,
if
I
might
interject
here,
I
think
the
so
I
mean
yeah.
I
agree.
One
of
the
my
biggest
gripe
was
this
whole
thing
was
the
dense
addresses,
but
I'll
talk
about
later.
That's
actually
kind
of
completely
separable
from
the
idea
of
the
sparse
distributed
memory
and
you
can
use
sparse
addresses.
You
can
also
use
sdrs,
I'm
pretty
sure
so.
A
A
A
No
less
than
that.
Basically,
this
is
a
very
large
radius.
That's
what
I'm
trying
to
say
here,
so
the
hamming
distances
are
distributed
very
much
around
500
and
as
soon
as
you
strike,
you
know
more
than
50
something
away
from
that
median
handling
distance.
Then
you
activate
these
neurons.
So
that's
what
this
y
vector
is.
A
So
these
are
all
the
activated
hard
locations
and
all
these
activated
hard
locations
then
are
connected
to
sort
of
another
sort
of
set
of
synapses
that
centers
onto
another
population
of
neurons,
which
in
this
is
which
is
represented
by
a
matrix
and
each
hard
location,
contributes
the
average
of
its
multi-set
right
and
the
word
in
register
you'll
notice
that
the
addresses
are
that
the
word
in
is
different
from
the
address.
That's
for
a
heteroassociative
memory
which
I'll
talk
about
in
just
a
second.
A
A
Instead,
it's
a
dynamical
system
in
which
you
continuously
keep
reading
and
iterating
on
your
memory,
so
notice
that
the
output
space
is
the
same
thousand
bits
as
the
address
register,
which
is
a
thousand
bits.
So
you
put
something
in
goes
through.
This
process
comes
out
on
the
other
side,
and
then
you
plug
that
back
in.
So
what
will
that
do
for
you
there's?
A
There,
then,
is
this
concept
of
a
critical
distance,
so
this
critical
distance
is
given
by
this
circle
here
and,
if
you're,
outside
the
critical
distance,
in
which
case
the
in
the
numbers
in
the
book
it
comes
out
to
209
bits.
You
do
not
converge
if
you're
outside
this
critical
distance.
So
you
you
do
one
hop
and
you're
farther
away.
A
So
if
you
start
out
with
something
with
noise,
maybe
like
a
zero
here
with
a
lot
of
noise,
one
iteration
will
get
you
down
to
like
six
percent
noise
and
then
another
iteration
will
get
you
down
to
two
percent
noise
and
then
once
you
get
to
the
center,
you
will
just
continue
to
just
loop
back
to
the
center
and
that's
how
you
know:
you've
converged
just
because
you're
you're
getting
the
same
thing
over
and
over
again
now.
This
was
sort
of
the
big
surprising
sort
of
one
I'd
say.
A
One
of
the
biggest
and
most
surprising
sort
of
concepts
that
was
talked
about
in
the
book
is
the
rates
of
convergence
and
divergence,
and
so
convergence
here
happens
in
something
like
five
six
iterations,
but
just
about
as
a
rule.
It
converges
in
like
less
than
ten
iterations,
for
you
know
reasonable
choices
of
of
numbers
in
terms
of
the
card
locations
and
address
space,
and
so
it's
extremely
fast
versus
divergence.
A
A
So
the
the
thing
that
I
liked
about
this
was
that
it
kind
of
related
back
to
the
natural
phenomenon
of
knowing
how
you
know
something.
So
it's
yeah
he
maps
that
on
to
starting
within
the
critical
distance.
So
if
you
start
within
the
critical
distance,
then
you'll
know
that
you
know
by
iterating
to
convergence
and
getting
there
quickly.
But
if
you
iterate
from
a
random
location
or
you're,
not
converging,
you
will
know
that
you're
not
converging
after
you
know
only
10
iterations.
So
that's
why
he
claims.
B
B
I
mean
that
he's
relating
this
to
to
you,
know
psychological
phenomena
right
and
the
question
is.
I
don't
think
this
is
how
the
these
type
of
memories
she's
creating
are
not
the
kind
of
memories
in
our
brain,
but
are
these
properties
that
properties
would
apply
to
the
type
of
memories
that
we
do
think
are
in
the
brain
than
the
kind
of
memories
that
we're
modeling?
I
have
zero
intuition
on
that,
so.
C
E
B
B
A
Okay,
so
I'll
briefly
touch
on
prediction.
This
is
one
of
those
things
that
he
goes
on
in
much
more
detail
than
I'm
gonna
give
here.
But
the
idea
is
that,
instead
of
storing
the
data
at
its
own
address,
you
store
data,
that's
different
from
the
address.
So
at
a
certain
address,
you
write
another
piece
of
data,
so
maybe,
if
you
wanted
to
store,
you
know
the
sequence
of
natural
numbers,
you
take
your
image
of
a
one
and
at
that
address
corresponding
to
that
image.
A
A
Store
a
a
b
and
c
and
then
have
that
be
followed
by
a
d
so
yeah.
The
idea
is
that
you
can
store,
you
know
various
types
of
transitions
and
then,
once
you,
when
you
want
to
use
your
sequence
memory,
it
operates
in
a
very
similar
way.
So
this
took
me
a
little
bit
to
understand
like
why
this
was
the
case,
but
I
promise
this
works.
A
A
Then,
if
you
start
out
within
the
critical
distance
you're
not
going
to
converge
within
this
circle,
because
it's
pointing
somewhere
else
but
as
a
rule,
if
you're
within
the
critical
distance,
the
new
address
that
you
go
to
will
be
more
similar
to
the
data
written
at
that
address
than
the
original
address
was
similar
to
its
corresponding
piece
of
data.
So
that
was
kind
of
a
mouthful.
A
But
it's
best
demonstrated
by
this
picture
that,
as
you
iterate,
you
get
closer
and
closer
to
the
actual
sequence
for
the
actual
data
and
address
pointers
at
each
part
of
the
sequence
over
time,
and
so
the
sort
of
convergence
rule
still
still
stands.
If
you're
doing
this,
you
don't
have
the
same
sort
of
tip
of
the
tongue,
phenomena
and
and
the
or
knowing
how
you
know,
because
that
would
require
sort
of
the
you
have
to
know
that
you're
not
changing,
but
if
you're
operating
well
then
you're.
A
If
this
chain
is
operating
correctly,
it's
going
to
be
changing
continuously.
So
you
have
to
have
another
sort
of
attractor
for
each
one
of
these
to
say.
Not
only
do
I
know
where
I'm
going
in
the
next
part
of
the
sequence,
but
I
recognize
this
part
of
the
sequence,
something
like
that.
So
you'd
have
to
store
both
like
the
part
of
the
sequence
in
your
memory
and
the
next
step.
B
A
Yeah,
I
will
say
I
am
leaving
out
a
lot,
so
I
mentioned
that
there
were
other.
You
can
go
to
higher
orders.
I
mean
I'm
not
gonna
lie
I
like
juventus
approach
better.
I'm
not
I'm
not
gonna
argue
for
this
over,
like
the
html
ideas,
but
there
are
sort
of
arguments
proposed
that
you
can
store
all
the
transitions.
You
need
to
be
able
to
tackle
the
complex
problems
of
like
seeing
the
abc
versus
like
error
like
abcd
versus.
A
C
A
Have
like
one
neuron
and
it's
axon
extends,
and
you
have
another
set
of
fibers
that
exactly
wraps
one
to
one
on
this.
On
these
larger
neurons,
I
think
they're
purkinje
cells.
I
have
some
notes
here
coincidence
the
climbing
fibers,
with
the
dendrites
of
the
purkinje
cells
yeah.
So
that's
that's
why
he
thinks
that
this
this
model
would
be
realistic,
because
you
have
to
have
that
exact
structure
where
you
have
an
input
line
and
an
output
line
sort
of
connected,
because
you
can't
use
the
same
line
for
both.
C
A
At
for
the
hard
location
contents
matrix,
so
this
is
describing
this.
This
picture
is
like
here's,
the
matrix
representation,
here's
the
representation
as
counters
and
sums
and
then
here's
the
representation
as
a
biological
neuron,
where
you
have
to
have
synapses
that
are
tightly
tied
for
each
of
the
contact
matrix
locations.
B
You
know
you
have
certain
deficits,
but
they're,
not
they're,
not
really
cognitive
deficits.
There
are
behavioral
deficits
like
you,
it's
hard.
If
you,
you
have
to
concentrate
really
hard
on
things,
you're
doing
as
opposed
to
doing
them
fluidly,
so
so
some
of
these
principles
that
might
actually
apply
in
the
cerebellum,
but
they
they
you
know
we're
biased
and
and
they
don't
really
necessarily
apply
the
same
way
in
the
cortex.
So
I
just,
I
think,
that's
worth
just
pointing
that
out.
B
Or
okay,
it's
it's!
What
happens
is
the
following:
like
the
kind
of
memories
we
think
about
everyday
memories
are
not
stored
in
the
cerebellum,
the
cerebellum
is
is
sort
of
an
intermediary
to
fine
motor
control,
so
people
with
cerebellar
deficits
have
they
may,
for
example,
if
they're
going
to
pick
up
a
coffee
cup,
you
and
I
could
be
looking
at
the
screen
and
just
move
our
hand
over
and
grab
the
coffee
cup
and
not
have
to
think
about
it.
B
B
It's
that
kind
of
deficit,
there's
lots
of
subtle
deficits,
but
that
it's
that
kind
of
deficit.
So
it's
not
like.
I
don't
know
if
the
cortex
can
pick
it
up.
It's
just
that
people
learn
to
compensate
so
I've
mentioned
this
many
times.
I
have
a
friend
who
does
not
have
most
of
her
cell
problems.
She
lost
it
because
it
was
removed
because
she
had
tumor
in
it
and
she's
pretty
normal
and
but
she'll
tell
you
oh
yeah.
I
have
to
think
about
everything
a
little
bit.
Why
can't
she
say
her
example?
B
Is
I
can't
multitask
and
but
she's
learned
to
compensate
it.
So
if
you
met
her,
you
wouldn't
know
there's
anything
wrong
with
her
she's
a
little
bit,
maybe
slower
to
talk
about
things
or
a
little
bit.
You
know
not
slow
and
in
a
deficit
just
physically
slower
to
choose
a
little
bit
of
things
about
more
things
anyway.
C
B
B
B
These
are
these,
are
things
related
to
reflex,
behaviors
and
fine
motor
control,
and
you
can
think
of
it,
sort
of
like
a
I
don't
want
to
oversimplify,
but
you
can
think
of
it's
like
a
smoothing
function
for
highly
learned,
behaviors
and
that
that
are
controlled
and
some
reflux
behaviors,
like
rabbits
when
you
puff
an
air
you
puff
air
air
puff
into
their
eyes
their
their
eyes
blink
and
that
blink
reflexes
slower
slowed.
If
you
damage
their
cell
phone,
it's
it
doesn't
work
as
well.
That
kind
of.
A
Okay,
moving
on,
let's
see
oh
yeah,
here's
my
slide
on
our
our
main
points
of
concern
with
this
yeah.
Basically,
I'm
just
saying
we
want
to
go
from
the
dense
vectors
to
a
sparse
distributed
representation
which
is
much
more
realistic,
so
there
are
some
proposals
that
sort
of
try
to
alleviate
this
slightly.
This
isn't
exactly
that,
but
this
is
a
step
in
the
right
direction.
This
is
the
selected
coordinate
design.
A
Yeah
so
then,
instead
of
having
yeah
each
neuron
pay
attention
to
all
the
bits
and
the
address
it
just
checks
for
equality
and
says
is
this:
is
the
bid
at
this
stress
equal
to
zero?
Is
it
bet
at
this
address
equal
to
one?
Is
it
equal
to
zero
and
you
sum
and
then
you
say,
is
it
equal
to
three
right?
So
it
is
exactly
it.
It
checks
for
exact
sort
of
sparse.
A
E
A
So
yeah,
as
I
said,
it's
not
it's
not
fully
there,
but
it's
a
step
two
towards
more
sparsity,
and
so
there's
also
some
some
interesting
there's
a
follow-up
paper
on
this
paper
by
the
same
guy
and
he
points
out
you
can
interpolate
between
the
two
designs.
So
you
can
have
this
this
regime,
where
the
neuron
pays
attention
to
all
of
them.
It
has
a
certain
threshold
which
corresponds
to
a
radius
or
you
can.
You
can
have
this.
A
This
is
the
other
side
of
the
spectrum
or
you
have
a
neuron
that
pays
attention
to
just
a
very
sparse
subset
and
it
has
to
have
all
of
them
activated.
You
can
go
anywhere
in
between
and
sort
of
dynamically
adapt
your
model
to
have
address
neurons
that
are
activated
by
you
know
various
various
size
subsets
of
the
address
space
and
then
also
those
can
have
thresholds
in
and
of
themselves.
A
A
Yes,
this
is
my.
My
proposal
is
that
you
actually
use
the
sparse
addresses
so
instead
of
having
dense
dresses.
This
is
this
is
using
sparse,
addresses,
use
a
selected,
coordinate
design
where
you're
only
paying
attention
to
specific
sort
of
addresses
and
you're
only
testing
for
once
right,
you're
only
testing
for
activations
and
then,
instead
of
looking
at
inequality,
you
also
have
a
threshold.
So
this
is
the
idea
of
sort
of
the
sparse
distributed
representation
and
overlap.
A
C
A
Have
a
sparse
address
and
the
neuron
pays
attention
to
separate
subsets.
Each
neuron
has
multiple
subsets,
with
thresholds
in
and
of
themselves,
and
then
you
check
if
any
of
the
dendrites
are
active.
This
is
just
this
is
just
a
random
proposal,
but
I
believe
that
this
would
this
would
still
work
as
long
as
you
have
sort
of
the
right
proportion
of
hard
locations
being
activated.
A
A
The
reason
why
this
design
was
supposedly
better
than
the
original
design
was
that
the
overlap
of
hard
locations
is
closer
for
sort
of
close
address
is
higher
for
close
addresses
and
lower
for
far
addresses.
So
what
that
means
is
that
when
you're
reading
you're
going
to
get
more
again
more
signal
because
you're
reading
from
more
relevant
hard
locations
and
less
noise,
because
you're
reading
from
less
farther
away
locations,
so
I
don't
know
how
this
idea
would
affect
that.
D
C
C
D
D
B
D
D
B
Is
are
there
some
of
the
accents?
Well,
I'm
confused
about
this
is
in
in
our
models
and
in
the
brain.
The
way
this
works
is
we
don't
form
synapses
to
things
that
as
randomly
and
then
pick
out,
which
ones
are
useful.
There's
some
exceptions
to
this,
but
but
the
general
idea
is
you're
going
to
connect
to
the
you,
have
a
pattern
in
memory
that
you're
trying
to
recognize
right
and
that
pattern's
existing
over
maybe
millions
of
neurons,
but
it's
a
sparse
pattern
of
activation,
and
so
there
may
be.
You
know.
B
A
C
D
C
E
E
D
B
As
being
it's
still
something
that's
being
dense
right,
so
I'm
not
sure
whether
we're
talking
about
the
sparsity
here
when
you
have
a
dense
address
like
this,
almost
almost
all
the
desirable
properties
we
get
from
sdr,
aren't
there
I
I
wish
we
could
try
to
walk
through
them
again,
but
but
in
some
sense
in
the
debts
address,
every
bid
is
equally
valid
as
every
other
bit,
and
you
know
the
zeros
are
as
important
as
the
ones
in
some
sense.
So
I
just
it's
fundamentally.
D
E
The
information
what
this
figure
is
showing,
if
I
understand
correctly,
is
there's
really
nothing
about
neurons.
Here
I
mean
I
know
the
word.
Neuron
is
up
there,
but
this
is
just
a
substance
operation.
You
have
an
original
address,
that's
dense
and,
if
he's
saying
an
efficient
way
to
match
that
dense
address,
rather
than
storing,
all
thousand
of
them
is
to
store,
let's
say
three
of
them
randomly
subsample
and
to
match
it.
You
have
to
say:
okay,
if
I
happen
to
connect.
E
E
D
B
B
If
I
connect
to
20,
I
have
activated
a
pattern
in
those
in
those
10
000
neurons
and
if
I
connect
to
20
synapse
20
of
those
active
20
of
those
active
neurons
in
a
sparse
activation,
those
20
active
neurons
will
be
almost
absolutely
guaranteed.
I
recognize
this
pattern.
It'll
be
highly
noise.
Immunity,
high
numerator
noise-
if
I
connect
to
20
active,
even
active
neurons
in
a
dense
population,
there's
very
little
information
in
that
and
I
can't
recognize
the
pattern
reliably.
B
C
A
Yeah
I
mean
I,
I
certainly
agree
I
think
yeah.
I
I
again
I'll
reiterate
my
my
main
concern
with
the
paper
or
the
original
book
was
the
dense
addresses.
Is
that
immediate?
Because
I
had
already
read
the
sort
of
juventus
stuff-
and
I
was
like
oh
yeah-
well,
sdr
seems
to
be
the
way
that
things
work,
but
so
yeah
the
one
of
the
reasons
I'm
highlighting
this
is
because
yeah
basically
yeah.
A
This
is
just
to
introduce
the
sub
sampling
operation
and
to
say
that
you
don't
need
dense
connectivity
between
all
the
neurons,
all
the
address,
neurons
and
all
the
yeah
between
the
address
and
all
the
address
neurons
you
you
can
have
weights
that
only
connect
to
a
subset
of
the
of
the
address.
That's
what
this
is
supposed
to
do,
but
yeah
the
I
I
haven't
seen.
I
can't
say
that
there
isn't
something
that
already
does
this,
but
this
this
is
more
in
line
with
the
sdr
proposal.
E
A
A
Maybe
change
up
the
operations
happening
here
and
then
out
comes
another
sdr
but,
like
I
actually
haven't
given
too
much
thought
in
mapping
them
together,
but
I
believe
it's
possible
to
replace
all
the
dense
parts
of
this
with
sparse
distributor
representations.
So
the
activations
of
these
hard
locations
are
going
to
be
sparse.
Actually
they
are
they're
going
to
be
even
more
sparse.
I
think
yeah
yeah
that
that's
that's
kind
of
where
I
was
going
with
this.
So
this
I
suppose
you.
B
Could
think
about
sparse
our
kind
of
sparsity?
You
can
say
okay.
Well,
instead
of
representing
an
address
base
of
two
to
the
n,
we
have.
You
know
we
have
a
binomial
like
you
know,
you're
choosing,
you
know
20
bits
out
of
you
know
a
thousand
choose
20
type
of
thing,
which
is
still
extremely
large
space,
but
it's
much
much
smaller
than
two
to
the
n
right.
B
I
guess
you
could
call
it
that
way
and
then
you
could
say
well
we're
not
going
to
populate
all
those
spaces,
so
I'm
mentally
trying
to
map
the
two
together
right,
you
could
say
yeah
well,
the
sparse
distributed
address
is,
is
a
smaller
adjust
space
than
two
to
the
end,
but
and
then
they'll
have
different
properties,
and
you
know
I
guess
you
could
try
to
make
that
connection.
I'm
not
sure
the
advantages
of
doing
that,
but
but
one
could
think
of
it.
That
way
again.
B
E
B
B
B
A
I
think
this
was
like
this
was
the
last
slide,
yeah
I'll
just
quickly
mention
this
yeah,
the
other.
The
other
problem
that
I
had
was
that
these
the
neurons
are
chosen
to
be
randomly
distributed
throughout
the
space.
So
you
have
your
population
around
and
you
say:
okay
go
they're
going
to
be
random,
randomly
distributed
and
they
don't
change
their
addresses
right
once
you
have
a
neuron,
that's
fixed
weights,
it's
going
to
be
fixed
at
that
address,
but
there
is
some
additional
follow-up
work
in
which
I
think
the
neurons
can.
A
I
think
in
the
sparse
distributed
addresses
scenario
then
yeah
these
neurons
cluster,
where
they're
supposed
to
cluster
to
represent
that
part
of
the
space
better.
So
I
like
this
because
it's
it's
the
idea
of
kind
of
learning
a
basis.
It's
it's
very
similar
to
learning
a
basis
for
every
single
space.
A
A
The
same
thing
can
be
applied
to
sdn,
yeah
and
then
yeah
I'll.
Just
briefly
mention
this.
I
have.
I
have
a
friend
who's
working
on
a
a
very
interesting
thesis
that
relates
sdm
to
modern
machine
learning,
but
I
can't
really
talk
about
it
because
it's
not
published
yet,
but
I
would
like
to
talk
about
that
in
the
future.