►
From YouTube: Numenta On Intelligence Podcast, Episode #13: Subutai Ahmad on Applying HTM Ideas to Deep Learning
Description
After a summer hiatus, we are back with a new episode of the Numenta On Intelligence Podcast.
Host Matt Taylor talks to Numenta VP of Research Subutai Ahmad about the effort he has been leading in applying Numenta research and HTM principles to deep learning systems.
Paper link: https://arxiv.org/abs/1903.11257
0:51 Why is Numenta looking at deep learning?
2:43 What are the inherent problems in deep learning and how can neuroscience help?
3:06 Continuous learning
3:48 Catastrophic forgetting - “If you have really sparse representations and a better neuron model and a predictive learning system you can learn continuously without forgetting the old stuff.”
5:11 What does high dimensionality mean in deep learning and neural networks?
6:34 Why does sparsity help?
11:23 Other types of sparsity: dendrites are tiny, sparse computing devices
14:47 Another type of sparsity: Neurons are independent sparse computing devices
15:34 Numenta’s paper on sparsity: How Can We Be So Dense? The Benefits of Using Highly Sparse Representations
19:34 The surprising benefit of sparse activations AND sparse weights, a rarity in the machine learning community
20:52 Benchmarks that we’re working with: MNIST, CIFAR10, VGG19
24:22 What does the future of research look like at Numenta?
A
Welcome
to
the
new
Mendte
on
intelligence,
podcast,
I'm,
Matt,
Taylor,
community
manager
and
today
I'm
talking
to
suit
I,
am
on
our
VP
of
research.
I've
worked
with
sue
battalion
dementia
for
almost
eight
years
now
and
I
have
the
utmost
respect
and
admiration
for
his
drive,
his
thoughtfulness
and
he
calm,
studious
atmosphere.
He
brings
into
every
situation.
It
was
a
pleasure
to
talk
to
sue
batai
about
his
most
recent
work,
applying
the
ideas
of
HTM
to
constructing
deep
learning
networks,
I
hope,
you'll
enjoy.
B
A
B
A
A
C
This
was
so.
This
was
interesting.
It's
a
it's
a
change
for
us
this
year
compared
to
previous
years,
so
we've
been
very
focused
on
the
neuroscience
and
understanding,
neuroscience
principles
and
creating
theoretical
models
of
experimental
data,
and
we
published
a
lot
there
sort
of
early.
This
year
we
published
the
frameworks
paper
which
put
a
lot
of
that
together
into
a
consistent
framework
and
scaffolding.
That
ties
a
lot
of
our
findings
into
one
place
right
and
I
felt.
At
that
time
it
was
had
the
right
time
to
start
thinking
about.
C
Ok,
we've
done
all
of
this
neuroscience
research.
Can
it
actually
apply
to
practical
machine,
intelligent
systems
and
practical
systems
right
yeah?
Initially,
the
answer
wasn't
really
clear,
and
but,
as
we
started
looking
into
it
more
and
more,
it
looked
like
we
can
actually,
rather
than
starting
from
scratch,
we
can
do
a
more
incremental
approach
of
looking
at
existing
machine
learning
and
machine.
Intelligent
systems
start
incorporating
neuroscience
principles
and
keep
expanding
in
proving
on
the
on
the
functionality
there
and
with
deep
learning.
In
particular,
there
are
some
really
big
fundamental
problems
with
deep
learning.
C
D
D
A
C
And-
and
these
are
everything
I'm
going
to
say-
is
sort
of
generally
acknowledged
within
the
field
as
well,
but
there
are
sort
of
deep
learning
has
been
incredibly
successful.
But
it's
nowhere
near
the
capability
of
human
intelligence,
which
we
think
of
as
being
extremely
flexible
and
dynamic
voice.
C
And
so
some
of
these
things
would
be,
and
we've
talked
about
these
for
a
long
time,
but
the
idea
that
systems
should
be
continuously
learning.
Today's
deep
learning
systems
are
very
static
and
rigid,
you
train
them
and
then
that's
it.
They
actually
not
really
learning
systems
they're,
more
they're,
trained
and
then
they're
static.
We.
A
Use
them
online
they're,
not
online
ewis
I
mean
people,
try
hacks
or
ways
to
batch,
train
them
or
get
them
to
train
at
certain
intervals,
but
that
doesn't
address
the
underlying
problem,
which
is
it
doesn't
it
doesn't
like
in
our
brains.
Our
synapses
is
constantly
updating,
as
we
learn
with
every
time
step
right,
yeah,.
C
Exactly
and
and
in
in
the
deep
learning
context,
this
is
an
active
area
of
research
in
the
deep
learning
world.
You
know
they
have.
This
thing
called
catastrophic,
forgetting
where,
as
you
learn
new
things,
it's
very
easy
to
forget
the
old
stuff
unless
you
really
pay
special
attention
to
it,
but
from
the
neuroscience
that
we've
done,
continuous
learning
is
relatively
straightforward
and
we've
shown
this
in
different
isolated
settings
in
the
past.
C
C
Attacks
is
sort
of
the
extreme
example
of
that
yeah,
but
we
know
the
human
brain
is
not
as
sensitive
as
these.
These
networks
are
right
and
what
we
can
show
is
that,
if,
if
you
have
really
high
dimensional
sparse
representations,
those
networks
can
also
be
a
lot
more
stable
against
noise.
Then
then
systems,
let's.
A
D
A
Then
the
sparseness
with
in
there,
it's
I
think
it's
hard
for
some
people
to
think
about
dimensionality
from
a
mathematical
perspective,
because
we're
always
living
and
breathing
in
three
dimensions.
You
know
any
time,
but
when
we
talk
about
high
dimensionality
and
deep
learning
or
in
sort
of
neural
networks,
what
does
that
really
sort
of
meaning
like.
D
C
B
C
A
C
C
C
D
A
C
D
C
That
layer
projects
to
another
layer
and
if
you
look
at
one
of
the
out
one
of
the
layers
in
the
destiny,
one
of
the
units
in
the
destination
layer,
there's
another
vector
of
weights.
That
indicates
what
the
connection
is
to
each
of
these
input
layers
right,
and
so,
when
you
have
an
input-
and
you
want
to
find
an
output
of
one
of
these
units,
you
do
a
vector
multiplication
or
a
dot
product
between
the
weights
and
the
activations,
and
that
gives
you
the
output.
So
those
so
there's
two
little
places
you
can
be
sparse.
C
C
C
A
C
So
it's
in
the
neocortex,
where
it's
known
for
a
long
time
that
the
neocortex
is
extremely
sparse
in
just
about
every
way
you
can
imagine
yeah.
There
are
more
ways
than
these
two
actually
yeah,
so
the
neocortex
to
act.
The
activity
of
neurons
at
any
point
in
time
is
extremely
sparse.
So,
as
as
I'm
talking
to
you-
and
you
are
talking
to
me-
you
know
the
neurons
in
my
neural
cortex
are
firing,
but
typically
it'll
be
less
than
2%
of
the
neurons
at
any
point
in
time
are
firing
and.
A
C
D
C
C
D
C
Be
observing
right,
yeah,
yeah
and,
and
we
should
get
to
the
neuron
units
and
the
dendrites
themselves,
because
that
there
you
have
other
types
of
sparsity
that
come
into
play.
Okay,
so
in
the
neocortex,
the
that
neurons
are
a
lot
more
complex
than
in
in
deep
learning.
You
alluded
to
the
you
know:
dendritic
tree
the
complex
set
of
dendrites
and
that's
where
all
the
inputs
to
a
neuron
come
into
the
onto
the
dendrites
root.
It
turns
out.
C
Those
dendrites
are
very
nonlinear
and
complex
and
at
any
point
in
time,
particularly,
we
get
further
out
from
the
neuron.
The
dendrites
themselves
are
tiny,
sparse
computing
devices,
so
isolated
segments
of
the
dendrites
are
recognizing
sparse
patterns
and
acting
on
their
own
independent
of
the
other
parts
of
the
dendrite
right,
so
that
neuron
itself
has
lots
of
these
tiny,
sparse
computers,
if
you
know
spread
throughout
the
dendrites.
This
is
something
called
active,
dendrites,
neuroscience
literature
and
it's
that's
pretty
interesting
again.
It's
very
different
from
how
deep
learning
systems
work
and.
C
C
C
Unlikely
to
think
you
know,
and
so
so
you
have
that
so
sparse
patterns
don't
interfere
with
one
another
and
then
you
get
to
the
these
dendritic
trees
and
each
one
is
independently
computing.
These
sparse
patterns,
because
the
neuron
is
continuously
learning
it
can
learn
new
patterns
dynamically
in
different
parts
of
the
dendritic
tree
without
affecting
the
other
sparse
patterns
right.
D
C
They're,
independent
and
mathematically
they're
unlikely
to
interfere
with
one
another,
very
highly
unlikely,
and
so
you
can
have
a
continuously
learning
system
that
avoids
this
catastrophic,
forgetting
problems
right.
You
just
be
adding
new
synapses
and
new
things
to
one
part
of
the
neuron
without
affecting
the
other
parts
of
the
north.
D
C
A
D
B
C
Learning
on
a
neuron
is
also
very
sparse,
so
in
a
deep
learning
system,
when
you
learn
every
weight
gets
updated
pretty
much,
but
in
a
neuron
only
this
the
little
segment
that
detected
the
pattern
actually
gets
updated
right.
So
the
learning
happens
a
very
sparse
way
on
a
on
a
neuron
yeah
and
that's
critical.
C
Different
yeah,
there
are
a
lot
of
different
listening
lines.
It's
very
localized
and
this
is
again
helps
continuous
learning
because
you,
when
you're
learning
a
new
pattern,
you
only
update
one
part
of
the
weights
and
you
leave
everything
else
untouched
right
and
it's
a
tiny
percentage
of
the
tired,
neurons
right
set
of
synapses.
So
yeah.
A
C
A
C
It's
actually
turns
out
to
be
very
straightforward,
so
in
the
paper,
what
we
do
is
we
go
through
the
mathematics,
so
sparse
representations
and
show
how
it's
very
stable
right
and
then
the
second
part
of
the
paper.
We
show
how
it
can
be
applied
to
deep
learning
systems.
So
in
a
convolutional
network
you
have
two
types
of
layers:
you
have
the
convolutional
layers
and
linear
layers.
B
C
A
C
C
It's
it's
closer
to
this.
It's
almost
identical
to
the
spatial
Pooler
where
we
have
a
local
inhibition,
and
you
know
if
a
unit
is
really
strongly
active,
it's
going
to
inhibit
its
neighbors
right
and
in
the
in
the
neuroscience
in
the
neocortex.
You
have
these
inhibitory
cells
that
that
form
sort
of
local
inhibitory
networks,
and
we
think
that's
how
the
sparsity
is
raised.
C
C
D
C
C
So
surprisingly,
so
here
we
have
sparse
activations
and
sparse
weights,
which
is
extremely
rare
in
the
machine
learning
community
right
and
when
I
mentioned
this
to
deep
learning
people
they're
out
there
kind
of
surprised
like
how
could
that
possibly
work
yeah,
it
turns
out
works
really
well.
So
we've
shown
that
for
many
data
sets,
the
three
data
sets
now
that
the
accuracy
level
doesn't
change.
C
C
Yeah
and
if
you
know
the
math,
it's
not
at
all
a
surprise
yeah,
it
has
to
be
that
way,
yeah,
but
it
was
kind
of
nice
to
see
that
even
in
a
deep
learning
scenario,
you
can
maintain
that
property
right,
and
so
this
just
shows
that
through
sparsity,
if
both
of
these
things
are
sparse,
so
you
get
representations
that
are
inherently
more
stable
than
dense
representations
in
a
deep
learning
system.
Right.
C
C
And
this
is
kind
of
a
basic
one
that
you
start
with
whenever
you're
doing
something
new,
so
it
works
really
well
with
M
nest.
Then
we
tried
it
with
sigh,
fart,
n
and
so
and
then
there's
a
we've,
also
tested
with
audio
with
the
Google
speech.
Commands
data
set
is
a
sort
of
a
data
set
of
one
word
spoken
commands
and
if
the
results
hold
in
all
three
of
those
things
sanera
for
all
three
of
those
benchmarks
and
the
other
nice
thing
was,
we
tried
different
networks
architectures
too.
C
C
C
Yeah
yeah,
so
you
you
know
going
back
to
the
question
of
what
is
the
benefit
of
sparsity
yeah.
Another
big
benefit
is
computational
things,
because
if
there's
a
zero,
you
don't
can
ignore
that
piece,
and
traditionally
it's
been
very
hard
to
exploit
that
in
machine
learning,
because
of
GPUs
are
inherently
not
as
good
at
handling
sparse
structures,
but
the
brain
is
extremely
sparse
and
because
of
that,
it's
extremely
efficient.
B
D
C
B
C
A
C
I
think
so,
I
think
the
there's
there's
a
bunch
of
people
looking
into
this
already
so
I
think
that's
nice
to
see.
Is
that,
particularly
over
the
last
year,
I
think
there's
been
a
resurgence
of
interest
in
sparsity
yeah,
so
I
think
our
timing
is
really
good.
Great.
No
one
has
really,
as
far
as
I
know,
really
looked
into
robustness
with
sparsity
and
doing
both
sparse
activations
and
sparse
weights
seems
to
be
really
rare,
but
yeah.
There
are
definitely
a
bunch
of
other
labs.
Looking
into
this
as
well
good.
D
D
C
So
I
would
say
we're
not
so
much
focused
on
deep
learning
as
we're
focused
on
practical
machine,
intelligent
systems
and
currently
deep
learning
is
the
best
example
of
that.
So
in
terms
of
our
research
art
map,
so
sparsity
is
you
want
to
start
with
sparsity
so
incorporating
sparsity
everywhere
and
showing
it's
robust
and
showing
it's
fast.
Those
are
the
first
steps
and
then
continue.
C
C
Yeah
and
the
way
we
did
in
the
HTM
I
think
could
really
apply
to
deep
learning
systems
as
well,
which
is
the
system,
is
constantly
making
predictions
about.
What's
about
to
happen,
and
then,
when
you
get
the
actual
data
about
what
happened
that
serves
as
an
error
signal,
and
you
can
immediately
do
unsupervised,
learning
on
that
and
that
can
be
done
in
a
continuous
learning
setup
because
you're
using
sparse
representations
of
these
active
dendrites.
So
now,
all
of
a
sudden,
you
have
something
that
doesn't
require
as
much
label.
C
C
That's
a
great
question:
you
know
in
the
brain:
we
don't
have
a
strict
back
propagation,
like
structure
or
a
learning
system
right
so
over
time,
as
our
systems
become
more
and
more.
You
know,
cortical
I
think
the
the
need
for
back
propagation
will
lessen
so
to
do
this
predictive
learning.
If
it's
happening
at
every
layer
independently,
you
don't
need
to
do
back
propagation
as
much
there
yeah.
D
C
We
might
still
for
a
while
have
gradients
flowing
through
just
because
that
currently
there's
no
better
way
to
create
really
scalable
systems
right,
but
over
time
as
we
learn,
because
we
incorporate
more
and
more
principles
from
the
brain
in
there.
Hopefully
we
can.
We
can
get
rid
of
that
too.
That
sounds
exciting.
That's
when
it's
not
deep
learning
anymore.
C
C
D
D
A
C
D
C
We
feel
very
happy
about
where
we
are
in
the
neuroscience
and
now
we're
charting
going
and
picking
off
all
of
those
pieces.
Everything
we've
done
there
and
start
to
implement
them
in
practical
systems
right
and
I.
Think
the
timing
is
right
for
that
and
it's
I'm
really
excited
about
it.
There's
a
lot
of
potential
there.
How
do
you
do
that.
C
A
C
A
Thanks
for
listening
to
the
the
mid-to
on
intelligence
podcast,
my
name
is
Matt
Taylor
I'm,
the
community
manager
and
engineer
for
the
Mentos.
My
guest
today
was
our
vp
research,
sukhothai
Ahmad.
If
you
like
this
content,
you
should
also
check
out
our
YouTube
channel
I've
been
live
streaming,
our
research
meetings
and
journal
clubs
every.