►
Description
Numenta’s intern Akash Velu discusses the project he has done over the course of his time at Numenta. Building on the key component of the Thousand Brains Theory of Intelligence, his work focuses on multi-task reinforcement learning using dendritic networks to achieve strong performance. In this presentation, he talks about the challenges, codebase, and results obtained through various environments. He then explores the next steps for the project such as experimenting with different dendrite configurations and incorporating more elements of sparsity.
- - - - -
Numenta is leading the new era of machine intelligence. Our deep experience in theoretical neuroscience research has led to tremendous discoveries on how the brain works. We have developed a framework called the Thousand Brains Theory of Intelligence that will be fundamental to advancing the state of artificial intelligence and machine learning. By applying this theory to existing deep learning systems, we are addressing today’s bottlenecks while enabling tomorrow’s applications.
Subscribe to our News Digest for the latest news about neuroscience and artificial intelligence:
https://tinyurl.com/NumentaNewsDigest
Subscribe to our Newsletter for the latest Numenta updates:
https://tinyurl.com/NumentaNewsletter
Our Social Media:
https://twitter.com/Numenta
https://www.facebook.com/OfficialNumenta
https://www.linkedin.com/company/numenta
Our Open Source Resources:
https://github.com/numenta
https://discourse.numenta.org/
Our Website:
https://numenta.com/
B
Yeah
so,
like
I
mentioned,
this
presentation
is
going
to
be
an
overview
of
the
internship
over
the
last
four
or
five
months,
and
I
just
wanted
to
say:
I've
had
a
great
time.
I've
learned
a
lot
met.
All
of
you
guys,
which
was
really
awesome,
and
I
really
wanted
to
thank
everyone
for
making
me
feel
super.
Welcome
and
helping
me
out
throughout
the
process
and
helping
me
have
a
great
time
here.
B
B
Having
an
agent
learn
like
higher
level
skills
in
an
unsupervised
way
using
dendrites,
this
sounds
vague
and
it
like,
after
reflecting
on
it
for
a
bit,
it
was
kind
of
vague,
like
I
think
I
wasn't
really
clear
on
what
exactly
some
of
the
aspects
of
this
were,
and
there
were
a
lot
of
open
questions
on
the
rl
side.
So
I
tried
to
simplify
the
problem
a
little
bit
and
I
thought
continual
reinforcement.
B
So
this
last
one
is
like
kind
of
what
I
landed
on
so
an
overview
of
the
problem
statement
of
multitask
rl.
So
in
normal
reinforcement
learning
you
have
an
agent.
The
agent
interacts
with
the
world
the
world
and,
in
this
case,
with
a
single
task
such
as
like
playing
chess
or
opening
a
window,
for
example,
is
represented
by
something
called
a
markov
decision
process
which
basically
encodes
the
rewards.
The
agent
receives,
and
also
also
the
transition
dynamics
of
the
world.
B
There's
like
a
fixed
set
of
tasks
or
a
task
task
distribution.
So
and
during
the
learning
process,
the
agent
kind
of
gets
to
see
all
of
the
tasks
it
will
encounter.
So
this
is
different
from
like
continual
learning
settings
where
the
tasks
are
kind
of
streamed
in
sequentially,
and
you
don't
come
back
to
certain
tasks
and
there's
usually
a
shared
structure
between
the
different
tasks.
B
So
this
allows
if,
if
there
wasn't
a
shared
structure,
then
the
agent
would
just
have
to
kind
of
memorize
each
one
or
learn
each
one
separately,
but
because
now
there's
a
shared
structure.
This
allows
the
agent
to
take
advantage
of
this
to
perhaps
like
use
one
task
to
aid
in
the
learning
of
another
task,
and
the
goal
here
is
to
learn
a
single
policy
to
solve
all
the
tasks.
B
B
So
again
it's
a
it's
a
step
towards
continual
learning,
which
I
guess
is
like
the
ultimate
goal,
but
it's
an
easier
problem
to
solve
for
sure,
and
this
diagram
shows
the
interference
like
you
have
one
loss
for
task,
one
and
another
loss
for
task
two.
If
you
combine
them
by
adding
them,
you
get
this
combined
loss.
But
if
you
apply
like
a
gradient
descent
algorithm
to
this
naively,
then
you
get
all
these
different
gradient
directions.
B
So
the
the
environment
in
which
I
decided
to
test
the
project
out
was
something
called
metaworld
and
this
is
a
robotic
manipulation
environment
where
there's
50
different
tasks
that
a
robot
arm
has
to
do
and
all
these
50
tasks
are
shown
here.
So,
for
example,
like
turn
on
the
faucet
or
push
some
to
push
an
object
to
a
location
and
what
the
agent
observes
is
not
the
it's
not
like
an
image.
B
So
this
makes
it
like
a
lower
dimensional
input,
which
makes
the
learning
a
little
easier
and
then
the
actions
that
the
asian
outputs
are
the
joint
torques.
So
each
there's,
I
think,
four
joints
here.
So
it's
a
four
degree
of
freedom,
so
the
torque
for
each
of
those
joints
and
the
metal
world
environment
contains
two
multi-task
benchmarks,
one
with
10
tasks
and
one
with
50
tasks.
B
B
B
We
did
a
lot
of
profiling
and
realized
that
there
are
also
several
expensive
operations,
specifically
an
absolute
value
and
max
operation
which
chooses
the
winning
segment
and
because
reinforcement
learning
has
a
lot
of
forward
passes
to
collect
data
and
also
a
lot
of
backwards
passes
to
learn
because
it
needs
a
lot
of
gradient
updates.
The
efficiency
of
the
network
can
be
a
bottleneck
so
profiling.
The
dendrites
revealed
some
of
these
slowdowns
and
several
possible
optimizations.
B
So
a
kind
of
temporary
solution
which
doesn't
solve
the
whole
issue
that
we
landed
on
was
using
an
optimized
one-segment
digitic
mlp.
So
this
does
kind
of
reduce
the
capabilities
of
the
dendrites,
because
it's
one
segment,
but
it
does.
The
trade-off
is
that
it's
a
lot
more
computationally
efficient
because
it
it
can
avoid
this
absolute
value
and
max
operation
and
also
scale
down
the
number
of
parameters.
B
B
B
We
also
encountered
some
challenges
with
garage,
so
we
made
some
modifications
to
that
that
were
made
to
like
suit
our
needs
and
to
speed
up
experiments
which
we're
taking
like
many
days
in
the
50
test
scenario,
and
we
recently
have
been
get
able
to
like
reduce
that
to
at
most
like
one
or
two
days
to
get
reasonable
results
and
in
most
cases
like
a
few
hours.
So
I
think
yeah.
B
B
A
B
Yeah
good
question,
so
even
with
the
optimize
like
one
one
segment,
then
right
so
though
those
would
take
around
a
little
less
than
like
twice
as
long
as
the
mlp
networks,
so
for
like
yeah.
So
if
the
mlp
network
would
take
one
day,
then
an
a
like
a
similar
dendritic
network
with
indirect
segments
would
take
like
two
days.
A
A
B
A
So
I
was
trying
to
understand.
You
know
that
this
sounds
maybe
a
little
weird,
but
you
know
why
isn't
it
100
and
because
the
general
idea
of
dendrites
should
really
separate
spaces
but
it?
But
if
only
if
the
activation
is
varsity
activation
supports
too
so
items.
I
think
I
can
understand
why
yeah,
if
you
had
activation
varsity,
you
might
get
100
on
this.
Yes,
so.
A
A
A
Mean
humans,
humans
can
do
many,
many
tasks
very
reliable
right.
We
don't
get
confused
and
so
ultimately,
I
think
you
know
a
network
with
activation
sparsity
and
a
sufficient
number
of
dendrites
should
also
have
achieved
very,
very
high
performance,
and
but
I
think
I
got
it
so
it's
anyway.
So
far
so
good.
It
looks
good.
C
A
B
A
And
again,
these
are
very
nice
results,
although
you
know
properly
done
not
not
that
you
didn't
do
it
properly,
but
in
a
full
implementation
of
the
theories,
and
we
should.
We
should
really
have
almost
100
success
in
many
tests
yeah
so,
but
this
is
giving
a
this
is
sort
of
an
approximation
of
what
the
following
network
would
be.
That's
pretty
good.
B
So
some
of
the
challenges
with
the
v1
meta
world,
which
is
the
initial
environment
design
from
the
developers,
was
that
the
reward
design
was
really
unstable,
meaning
there
were
the
rewards
could
get
on
the
scale
of
like
many
thousands,
which
kind
of
causes
experiments
to
crash
like
this.
Where
you
have
increasing
performance,
then
suddenly
it
just
drops
because
the
the
network
is
dealing
with
huge
losses
and
huge,
just
high
magnitude
numbers,
and
this
can
cause
like
bad
grading
steps
and
with
this
code
base
it
also
when
we
transitioned
to
the
50
task
environment.
B
A
B
Yeah,
so
I
think
the
hypothesis
is
that
the
reward
from
the
environment
basically
was
designed
so
that
the
the
numbers
the
scalar
numbers
returned
were
really
high,
and
this
because
the
the
rewards
are
used.
A
lot
in
the
loss.
Functions
for
the
actual
neural
networks,
then,
like
the
losses,
are
also
super
high.
A
A
Yeah
and-
and
the
reason
it
gets
very
high
is:
why
is
it
is
it
like?
Is
it
sort
of
going
to
some
sort
of
unstable
oscillatory?
You
know
regime
or
thrashing,
or
something
like
that?
I
don't
understand.
Why
that
I
don't
have
an
intuitive
sense
or
reinforcement
money
network.
They
say
why
that
loss
function
would
get
so
high.
B
Right,
well,
I
think
the
reward
can
get
high,
because
well,
the
reward
should
get
high,
as
you
do
well
on
the
task,
but
like
a
big
part
of
designing
an
environment
is
deciding
what
kind
of
reward
to
give
and
it
can
be
bad
if,
like
the
scale
of
the
reward,
is
super
super
high.
So
I
guess,
like
a
really
simple
example,
would
be
like
if
you're
playing
a
chess
game
and
the
reward
is
one.
A
A
Why
would
that
reward
function
again
by
ignorance?
Excuse
me,
yeah,
reinforcement
linearity,
but
why
would
that
work
function?
Change
that
you
learn,
I
mean,
isn't
winning
a
game,
one
always
one.
Why
would
it
also
start
going
up
to
like
a
million?
You
know
what
would
cause
it.
I
I
I
have
a
fundamental
lack
of
understanding
of
how
reinforcement
learning
works
where
that
reward
function
would
change.
I
don't
understand
that.
A
D
In
this
case,
correct
me
from
wrong
cash,
but
it's
not
that
we
have
sparse
rewards
like
if
you
win,
you
get
one.
If
you
lose,
you
got
zero,
but
we
have
rewards
across
the
environ
across
the
past.
So
if
you
get
closer
to
the
goal,
you
get
a
reward
yeah.
This
is
like
an
aid
to
the
agent
you
can
think
of
like
yeah.
A
B
B
A
D
Dense
rewards
is
sort
of
a
hack.
At
least
I
mean
that
the
traditional
way
would
be
to
use
parts
rewards
right.
So
if
you
win,
you
get
a
point
to
lose.
You
don't
get
a
point,
but
this
the
state
space
is
huge,
so
the
agent
can
never
find
a
solution.
So
what
people
do
in
this
sort
of
environments
like
metalworks?
They
create
these
dense
rewards.
So
you
give
like
this
intermediate
breadcrumbs
to
guide
the
agent
towards
the
goal.
A
Yeah,
but
you
could
say
you
could
say:
okay,
your
reward
for
getting
you
know.
Solving
this
problem
is
ten
and
then
you
could
have
anywhere
between
zero
and
ten
along
your
way
right,
you
think
the
upper
limit
would
be
fixed
as
opposed
to
oh,
you
get
100
or
a
million
or
something
like
that,
or
maybe
this
sticks.
It's
just
a
big
range.
Is
that
what
you're
saying
I.
A
I
mean
this
is
this
is
a
surprising
thing
to
me.
It's
like!
Oh
gosh,
you
see
something
like
this
and
it's
like.
Oh
damn
what
happened
you
know
so
I'm
trying
to
understand
if
this
is
a
fundamental
issue
with
dendritic
theory
or
if
it's
really
it's
something
else
and
I'm
hoping
it's
something
else
and
what
I'm
hearing,
maybe
that's.
It
is
maybe
something
else.
B
B
Did
you
get
there?
Yet?
Sorry,
no
worries-
and
this
also
happens
with
like
this
kind
of
crash
after
a
lot
of
training
steps
is
fairly
common
in
reinforcement
learning,
regardless
of
what
kind
of
network
you
use
just
because,
like
there's
the
way
that
algorithms
work
there's
like
the
training
data
kind
of
changes
over
time,
and
when
that
happens,
it
can
happen.
Well,
that's
a.
B
Yeah,
so
this,
so
these
kind
of
challenges
motivated
us
to
switch
to
a
version
two
meta
world
which
had
been
recently
released
and
also
switched
back
to
the
garage
environment
which
had
a
lot
of
lot
more
support
and
seemed
like.
It
would
just
be
a
good
idea
to
integrate
our
code
base
with
garage,
because
it's
much
it's
better
built
and
we
could
like
build
on
top
of
that
for
our
needs
and
be
in.
B
So
these
are
results
with
the
v2
meta
world
environment
and
the
garage
code
base.
V2.
The
difference
is
the
rewards
are
on
a
better
scale,
like
jeff
suggested,
and
also
like
some
some
smaller
changes,
and
these
changes
as
a
whole
are
meant
to
make
the
training
process
more
stable
across
like
across
different
experiments.
B
So
it's
it's
supposed
to
be
a
better
environment,
still
capturing
the
multitask
nature
of
it.
This
is
on
the
ten
task
environment.
So
I
trained
a
mlp
network
with
hyper
parameters
given
by
the
developers,
and
surprisingly,
it
got
80
percent
success
rate
on
time
tasks
and
just
to
kind
of
validate
that
the
dendrites
could
also
do
pretty
well.
B
I
didn't
really
tune
this
at
all
and
didn't
have
activations
varsity.
This
was
just
kind
of
a
sanity
check.
The
dendrites
got
70
success
rate
and
the
kind
of
challenge
here
was
that
the
baseline
was
already
getting
80,
so
the
room
for
improvement
is
much
less
compared
to
what
we
had
before
and
because,
like
the
success
rate,
is
a
kind
of
noisy
measure
of
how
good
the
network,
how
good
the
agent
is
doing
it
was
hard
to.
B
A
By
the
way
before
you
do
that,
it's
not
clear
looking
at
these
two
charts
that
actually
the
the
the
tuned
bass
line
was
actually
better
than
the
untuned
dendrites.
I
mean
if
you
look
at
the
scales
and
these
charts,
I'm
not
sure.
But
did
you
numerically
verify
that?
Because
if
I'm
looking
at
the
chart
on
the
left,
you
know
you'd
say:
what's
the
average
performance
there,
it
says
around
70
december
75.,
I'm
just
curious,
but
it
wasn't
actually
substantially
less.
B
B
B
One
thing
I
noticed
one
thing
I
was
curious
about
was
like
what,
if
we
just
scale
up,
the
bass
lines
is:
if
we
just
naively
increase
the
capacity
of
the
networks,
does
performance
increase
by
a
lot
it
turns
out.
The
answer
was
yes,
so,
just
by
increasing
the
width
of
the
hidden
layers
of
the
mlp,
we
were
able
to
get
70
to
75
percent
success
rate
on
50
tasks,
and
this
is
with
like
no
dendrites.
B
So
this
this
was
just
an
interesting
observation
that
I
made
kind
of
after
this
next
experiment
that
I
ran,
which
again
I
kind
of
did
the
same
thing
where
I
had
took
a
narrow,
narrow,
meaning,
500,
12,
dimensional,
hidden,
hidden
layers,
dendrites
this
one
I
didn't,
have
k
winners
just
because
this
was
like
the
the
first
experiment.
I
ran
with
kind
of
the
most
basic
addition
to
the
mlp
and
did
no
tuning
on
this
one,
whereas
this
one
is
more
tuned,
so
this
matched
the
performance
of
the
mlp.
B
So
this
was
more
of
like
again
a
validation
on
the
50
50
tasks
and
then
on
the
right
side.
I
included.
I
I
this
is
after
like
this
is
the
best
result
after
several
runs.
So
this
is
a
much
wider
network
with
2048
units
per
hidden
layer
and
okay
winners
of
around
0.2,
and
so
20
of
the
units
are
on
at
every
hidden
layer.
And
here
the
the
success
rate
was
around
80
percent
and
occasionally
got
up
to
like
85,
closer
to
90
percent.
B
A
If
you
give
this
presentation
again
like
gosh,
it
would
be
useful
to
plot
these
on
the
same
graph
use
different
colors,
or
something
like
that,
because
I'm
constantly
looking
at
these
trying
to
figure
out,
you
know
because
they're
basically
plotting
the
same
stuff.
But
you
know
your
charts
are
different
sizes
and
on
the
trend
line.
C
B
B
A
E
B
Yeah,
so
the
that's
approximately
correct,
so
how,
in
this
algorithm
it
works,
is
you
have
a
a
buffer
of
a
fixed,
fixed
size?
So
in
this
case
one
million
and
you
collect,
I
can
keep
collecting
data
and
adding
it
to
the
buffer,
and
once
it
gets
like
over
a
million,
then
the
oldest
oldest
samples
are
kicked
out
and
the
newest
ones
are
put
in.
So
as
as
you
progress
through
this,
the
the
data
collected
within
that
buffer
changes
and
then
okay.
E
So
there's
there's
a
window
there
there's
a
window
there
that
you're
moving
across
but
you're
looking
at
continuously
changing
inputs,
and
then,
when
you
said
you
got
a
bad
batch,
it
meant
something
that's
really
off
kilter
as
far
as
the
expectations
yeah.
What
you've
been
training
up
to
that
point
would
be.
C
C
F
B
E
F
E
B
D
B
D
Yeah
that
makes
sense
yeah.
You
just
want
to
make
a
remark
here
that
this
wide
networks
we
are
not
controlling
for
number
of
params.
So
when
we
say
why
2048
that's
actually
a
network
for
a
time
speaker
than
the
one
on
the
left,
so
that
also
might
be
why
we're
having
better
results?
It's
not
just
because
it's
right.
C
D
B
D
E
C
That's
the
way
to
get
yeah.
Sorry,
sorry
go
ahead,
yeah
a
couple
of
things.
Is
there
a
way
to
get
like
a
summary
number
for
each
of
these
charts?
I
know
it's
very
noisy
and
stuff.
Could
we
say
that?
Is
there
a
way
to
extract,
like
the
mean
of
the
last
few
million
iterations
or
something
like
that
to
say?
Okay,
the
network
on
the
right
was
at
about
eighty-one
percent
and
I
went.
B
B
C
B
A
A
Useful,
unfortunately,
I
have
to
step
out
now
that
was
I'm
not
sure
if
your
own
was
done,
but
it
was
very
nice
and
I
think
it's
russia
I
mean
again.
I
really
don't
think
I'd
like
to
emphasize
here
is
that
we
should
be
able
to
get
100
of
these
tasks
eventually,
but
you
have
to
you
have
to
implement
all
these
things,
and
you
know
I
thought
it
was.
I
thought
that
was
pretty
cool
yeah.
I
have
to
say
goodbye.
So.
B
C
B
I
think
it's
definitely
promising,
because
I
think
it's
a
different
testbed
for
the
dendrites.
I
think
so
I
think
yeah
like
getting
like
jeff
mentioned
as
close
to
100
as
possible
is
possible
with
the
dendrites
and
I
think
showing
that,
like
the
way
dendrites
add
parameters
to
networks
is
a
effective
way
of
doing.
B
That
can
be
done
in
this
project
and
I
think,
like
we're
getting
there
and
I
think
the
product,
the
the
environment
can
be
extended
to
more
difficult
benchmarks
pretty
readily.
So
like
I
meant
I
shared
on
the
slack
recently
that
some
people
have
created
a
continual
learning
version
of
meta
world,
so
it
uses
the
same
base
environment.
It
just
changes
the
way
the
agent
sees
the
tasks
and
changes
adapts
it
to
a
continual
learning
setting.
C
Yeah
yeah:
well,
that's
good
yeah,
the
multitask
one
is
also
you
know.
Just
like
you.
Your
motivation
earlier
is
also
a
good
testbed
for
dendrites
same
underlying
intuitions
apply,
but
a
very
different
kind
of
formalism
than
continual
learning.
But
the
you
know,
concepts
of
avoiding
gradient
interference
and
so
on
are
still
apply.
Yeah.
Do
you
think
these
are
publishable
results
now
that
you've
got
50
task
results.
B
So
I
think
yeah,
like
it's
probably
publishable
now
and
definitely
will
be
soon
with
like
incorporating
these
and
properly
tuning.
I
think,
like
one
thing
I
noticed
in
this
project
is
like.
I
think
the
tuning
is
a
little
bit
tricky,
especially
with
things
with
k.
Winner
activation
sparsity,
like
it's
a
little
bit
finicky
so
like
finding
the
right
configuration
can
really
like
once
you
find
that
right
pattern,
it
can
really
get
good
great
performance.
C
C
F
B
D
D
Well,
but
the
max
is
not
coming
from
the
killing
max
coming
from
the
dendritic
activations
yeah.
It
could
so
yeah
the
max
taken
a
big
though
so.
The
fact
that
I'm
not
sure
the
exact
number
is
now,
but
I
think
it
was
like
3
or
4x
lower,
and
it's
not
just
about
being
a
max
any
operation
that
you
do
to
consolidate
the
dendrites
and
we
tried
like
a
few.
There
is
an
impact
and
yeah.
That's
not
like.
D
Yeah,
which,
if,
if
you
sum
it
afterwards,
it's
similar
yeah
any
operation,
you
do
it.
If
you
break
it
down
into
two
operations,
then
you
have
that
slow
down.
We've
tried
different
versions
that
didn't
break
it
down
into
two
operations,
but
they're,
not
true
to
what
we
want
to
do
with
dendrite
right.
So
we
want
that
max.
We
want
that.
You
know
winner,
take
all
operation
there.
F
F
C
D
C
C
B
F
B
B
C
B
C
B
C
D
Yeah
but
hey
for
this
project,
we
are
sticking
with
garage
and
the
long
run.
I
believe
there
are
better
alternatives.
Facebook
just
released
a
reinforcement,
learning
library,
that's
fully
compatible
with
pytorch.
So
that's
I!
I
don't
want
to
contribute
back
to
garage,
though
we're
with
subclass
when
we
need
it.
We're
like
recreating
some
of
the
things
we
needed,
but
in
the
long
run
I
don't
think
we're
gonna
stick
with
garage
sale.