►
From YouTube: "Towards Continual Reinforcement Learning: A Review and Perspectives” Paper Review - Feb 22, 2021
Description
Our research intern Akash Velu gives an overview of continual reinforcement learning, following the ideas from the paper “Towards Continual Reinforcement Learning: A Review and Perspectives” by Kheterpal et al. He first goes over the basics of reinforcement learning (RL), and discusses why RL is a good setting to study continual learning. He then covers the different aspects of continual RL, the various approaches to solving continual RL problems, and touches upon the potential for neuroscience to impact the development of continual RL algorithms.
Paper: https://arxiv.org/abs/2012.13490
- - - - -
Numenta is leading the new era of machine intelligence. Our deep experience in theoretical neuroscience research has led to tremendous discoveries on how the brain works. We have developed a framework called the Thousand Brains Theory of Intelligence that will be fundamental to advancing the state of artificial intelligence and machine learning. By applying this theory to existing deep learning systems, we are addressing today’s bottlenecks while enabling tomorrow’s applications.
Subscribe to our News Digest for the latest news about neuroscience and artificial intelligence:
https://tinyurl.com/NumentaNewsDigest
Subscribe to our Newsletter for the latest Numenta updates:
https://tinyurl.com/NumentaNewsletter
Our Social Media:
https://twitter.com/Numenta
https://www.facebook.com/OfficialNumenta
https://www.linkedin.com/company/numenta
Our Open Source Resources:
https://github.com/numenta
https://discourse.numenta.org/
Our Website:
https://numenta.com/
A
A
The
paper
is
called
towards
continual
reinforcement,
learning,
a
review
and
perspectives,
and
it's
by
these
authors
and
the
link
is
here
and
it's
it's
fairly
high
level.
It
doesn't
go
into
like
the
technical
details
of
like
each
algorithm,
but
it
just
kind
of
distills
the
main
ideas
of
each
one.
So
I
think
it's
it's
a
good
read.
A
It's
it's
just
an
archive
for
now
it
only
came
out
in
like
december,
so
journal
version
coming
so
before
I
get
started
with
like
the
actual
content,
I
thought
it
would
be
good
to
just
go
over
the
basics.
Basic
overview
of
reinforcement
learning,
so
the
key
structure
in
reinforcement
learning
is
something
called
a
markov
decision
process
called
an
mvp,
so
this
kind
of
summarizes
the
the
world
the
agent
lives
in
so
there's
a
state
space
s,
an
action
space,
a
that
determines
what
kind
of
actions
the
agents
can
take.
A
There's
the
environment
dynamics
which
are
typically
stochastic,
and
it's
like
it's
a
markovian
system.
So
the
probability
of
the
next
state
is
defined,
conditioned
on
the
current
state
and
the
action
that
the
agent
takes
at
the
current
state
and
then
there's
a
reward
function
that
is
based
on
the
state,
the
action
and
what
state
is
transitioned
to,
and
it's
a
scalar
number
that
the
agent
receives
and
people
like
to
call
reinforcement,
learning,
learning
via
trial
and
error,
because
this
is
like
the
classic
diagram
where
there's
an
agent.
A
A
B
A
A
A
So
in
terms
of
reinforcement
learning
the
objective
is
to
learn
a
policy
which
is
basically
like
a
controller
if
you're
familiar
with
a
control
theory
where
you're
trying
to
choose
an
action
based
on
the
current
state
and
a
lot
of
times,
this
is
stochastic,
so
it
will
like
determine
a
probability
distribution
of
your
actions
and
then
you
sample
from
that
distribution
and
the
objective
function.
Is
this
expectation,
which
is
the
expectation
of
the
sum
of
rewards
that
you
see
throughout
the
lifetime
as
an
agent?
A
So
this
is
usually
sum
from
0
to
infinity,
but
there's
this
discount
factor
gamma
and
this
basically
modulates
how
much
you
care
about
the
future
versus
the
present.
So
typically,
this
is
like
0.99
or
0.999,
so
it
weights
the
present
a
little
bit
more
than
the
future,
but
not
too
much,
but
you
could
imagine
if
it
was
like
0.1.
That
means
you're
like
super
greedy,
and
you
only
care
about
the
president
and
not
really
the
future.
A
So
it's
the
expectation
of
the
sum
of
rewards
with
this
discount
factor
gamma
but
you're,
conditioning
that
on
the
current
state.
So
given
that
you're
in
the
current
state,
what
kind
of
what
is
the
expected
reward
in
the
future,
given
also
that
you
act
using
pi
as
your
policy
and
then
q
function,
which
is
very
similar
except
now,
you're
conditioning
both
on
the
state
and
the
action?
So
given
that
I'm
in
state
s-
and
I
take
action
a
what
is
the
expected
reward
that
I
receive
when
when
I'm
acting?
A
So
there's
two
classes
of
reinforcement,
learning,
algorithms,
broadly,
that
that
people
come
up
with
some
model
based
algorithms,
explicitly
try
to
learn
the
mdp
structure,
so
they
learn
the
transition
function
or
the
reward
function
or
both
through
trial
and
error,
and
then
once
they
have
this
model.
There
are
a
bunch
of
algorithms
that,
using
this
model,
you
figure
out
how
to
learn
an
optimal
policy.
A
So
once
you
have
the
knowledge
about
the
mdp
structure,
then
you
can
plan
on
this
mdp
approximation
that
you've
learned
and
then
you
figure
out
what's
the
best
way
to
act
and
there's
model
free
algorithms,
which
don't
learn
the
mdp
structure,
explicitly
at
all,
they
directly
optimize
the
policy
with
that
objective
function.
So
things
like
the
some
algorithms
take
the
gradient
and
of
the
of
the
objective
and
just
kind
of
directly
optimize
the
policy
and
there's
also
another
branch
of
algorithms
called
q
learning
so
yeah.
A
So
this
is
like
another
example
where
model
based
is
like.
You
have
a
model
of
the
world.
So,
like
this
map
and
you're
trying
to
plan
on
it
and
model,
three
is
you've
kind
of
associated
certain
states,
with
whatever
the
optimal
action
is,
and
then
you
just
take
it
without
doing
explicit
planning
on
the
mdp.
A
Okay.
So
now
that
we
kind
of
have
the
bare
bone
basics
of
rl,
we
could
talk
about
continual
rl
in
in
normal
rl.
I
should
say
usually
there's
like
a
very
specific
task
that
you're
training
the
agent,
so
the
famous
examples
are
like
atari
games.
So
if
you're,
if
you're
training
the
agent
to
really
get
good
at
pong,
you'll,
just
let
the
agent
learn
on
that
pong
environment
for
a
bunch
of
time
steps
and
eventually
it
kind
of
figures
out
what
the
optimal
actions
are.
A
But
if
you
stick
this
agent
into
something
some
other
random
environment
or
another
atari
game,
it's
going
to
crash
and
burn
because
it's
never
seen
that
before
so
typically,
rl
is
like
within
a
very
specific
task,
but
in
continual
rl
to
kind
of
model.
What
actual
agents
would
see
you
need
to
generalize
across
tasks?
So
first
you
need
a
notion
of
what
a
task
is.
A
So
when
the
task
changes,
the
actual
mdp
changes
for
the
agent,
the
other
perspective,
which
is
probably
what
we
experience
as
humans,
is
that
the
agent
lives
in
a
huge
but
stationary
mdp,
meaning
its
dynamics
are
fixed.
So
the
physics
are
the
same,
but
the
agent
doesn't
observe
most
of
the
environment,
and
this
is
pretty
natural
for
us
to
think
about.
A
A
So
there's
like
a
wide
spectrum
of
continual
rl
that
kind
of
increases
in
difficulty,
and
this
is
the
way
the
authors
categorize
that
so
the
simplest,
so
it
goes
from
like
simplest
to
hardest.
So
first,
you
have
something
they
call
domain
adaptation,
where
your
agent
is
trained
on
a
very
specific
skill,
for
example
like
self-driving,
an
autonomous
car
in
simulation,
and
then
now
you
need
to
adapt
that
very
specific
skill
to
a
new
environment.
A
So
a
really
good
example
of
this
is
sim
to
real
transfer,
where
you
have
an
agent
that
is
trained
in
a
simulator
and
then
now
you
need
to
put
it
in
the
real
world,
but
like
it's
applying
that
same
skill
in
the
real
world,
so
you
just
need
to
know
how
to
adapt
that
the
next
is
transfer
learning,
so
transfer
learning.
What
what
happens
is
there
are
multiple
domains
of
deployment
and
the
agent
needs
to
learn
different
types
of
skills
and
it
also
experiences
non-stationary
evolution.
A
However,
the
the
key
difference
now
is
that
there's
a
bunch
of
tasks,
but
each
task
has
each
task
has
its
own
policy,
so
this
is
kind
of
simplifying
the
problem.
A
bit
by
allowing
the
agent
to
kind
of
have
a
different
policy
for
each
task.
It
sees
another
class
that
the
authors
talk
about
is
along
the
lines
of
meta
learning,
so
there's
like
a
meta
training
and
I'm
at
a
testing
phase
where
the
agent
is
basically
learning
how
to
learn.
A
Multitask
learning
is
when
the
agent
is
basically
in
different
domains
with
multiple
skills
and
has
a
single
kind
of
policy
that
controls
it
in
all
of
these
settings
except
the
tasks
it
sees
comes
from
an
underlying
stationary
distribution.
So
there
could
be
like
a
distribution
that
defines
what
tasks
it
sees.
This
distribution
is
fixed
and
the
agent
has
to
learn
a
policy
to
kind
of
solve
all
these
and
continual
learning
where
all
of
these
all
of
these
different
options
are
kind
of
turned
on
where
you
have
different
domains.
C
A
B
That
there's
such
variety-
you
know
it
seems
like
I
I
it
just
seems
to
me
somehow
you
would
have
expected
it
to
settle
on
a
you
know,
settle
on
a
single
direction
or
a
couple
directions,
or
you
know,
as
opposed
to
having
some
you
know
here.
You've
got
five
different,
overlapping,
some
ways.
Methodologies,
I
mean
I
just
just
I'm
just
surprised
by
that.
I
just
I
don't
know
if
there's
anything
anything
else
to
say
about
it,.
A
B
All
right,
because
you
know
in
general,
in
any
field
of
study,
we
all
want
to
go
to
some
sort
of
paradigm
or
a
very
limited
number
of
them.
You
know
you
want
to
end
up
with
a
common
solution
to
problems,
and
so,
if
you
said
oh
yeah,
continual
life,
only
learning
is
the
goal,
and
these
are
stages
that
people
have
gotten
along
the
way
to
get
there.
That
makes
sense
to
me
right
well,.
D
But
but
it's
not
it's
not
just
that
right,
like
I
said,
like
each
intermediate
state,
has
its
own
application,
like
especially
in
robotics.
So
even
if
I
just
have
domain
adaptation,
some
people
just
want
to
have
domain
adaptation,
because
that
works
for
their
business
case.
I
don't
think
every
lab
is
trying
to
get
to
continue
lifelong
learning,
yeah.
B
D
B
Well,
that's
that's.
My
first
part
of
my
question
is
this:
it's
just
a
well
agreed
upon
everyone
understands
this
taxonomy
or
is
it
like?
Well,
here's
how
we're
thinking
about
it
right
now
and
the
car
said
no.
This
is
really
well
understood.
Everyone
understands
these
things,
but
or
maybe
it's
not
maybe
someone
else
should
have
a
different
taxonomy,
which
suggests
that
it's
more
fluid.
D
B
It
still,
it
doesn't
seem
that
this
is
a
it.
Just
doesn't
strike
me
a
chart
like
this,
that
this
is
a
long-term
taxonomy
that'll
stick
around,
it
seems
like
somehow
people
are
going
to
figure
out
how
to
get
to
a
single
method
of
doing
all
this
stuff,
maybe
two
or
something:
okay,
just
I'll
just
leave
it
dude.
D
I
mean
yeah
so
perspective.
Two
for
me
sounds
a
bit
strange
because
you,
you
were
you're
talking
about
the
idea
of
you
having
a
large
mdp,
that's
stationary,
because
the
laws
of
physics
don't
change,
but
from
a
single
agent
perspective.
If
you
consider
there
are
other
agents
in
the
environment
and
this
those
agents
have
free,
will
the
environment's
always
going
to
be
done
stationary,
because
I
can't
I
can
never
predict
the
actions
of
all
other
agents
so
from
the
perspective
of
single
agents
always
going
to
be
non-stationary.
A
Well,
yeah
you're
right.
The
authors
also
point
this
out
so
like
from
from
the
perspective
of
a
single
agent,
the
mdp
is
always
non-stationary,
mostly
because
of
the
presence
of
other
agents
and
also
lack
of
of
observation,
but
if
you
consider
the
entire
mdp,
including
all
the
agents
and
everything,
then
it's
stationary
in
that
regard.
Well,.
D
Okay,
so
if
you,
if
you
assume
there
is
no
free
will
then
yes,
we
can
have
a
non-stationary
mdp,
but
otherwise
I
mean,
if
you
consider
they
can
learn.
That
means
their
policy
is
going
to
change.
If
their
policy
is
going
to
change,
then
it's
never
going
to
be
stationary,
even
if
it's
not
just
from
the
perspective
of
a
singularity.
A
Okay.
How
can
I
use
my
past
experience
to
to
leverage?
How
can
I
leverage
that
past
experience
to
act
optimally
in
the
future
to
make
predictions
about
the
future
and
then
also?
How
much
should
I
just
prioritize
updating
myself
to
act
optimally
on
what
I'm
seeing
right
now
so
because
of
this
kind
of
trade-off
that
the
agent
has
to
make
it's
a
good
setting
to
to
study
the
kind
of
classic
problems
that
you
you
come
across
and
continue
learning,
so
the
big
one
being
stability
plasticity.
A
Another
big
thing
is
that
in
supervised,
continual
learning
there's
no
time
dependence,
so
the
agent
has
to
like
see
it's
sees
new
types
of
data,
but
there's
no
notion
of
time
it
just
kind
of
randomly
the
the
types
of
data
it
sees
just
randomly
switches
without
any
sort
of
lead
up
to
it.
Whereas
here
the
observed
data
has
an
explicit
time
dependence,
so
the
agent
is
kind
of
naturally
evolving,
and
then
it
can
transition
itself
into
a
new
kind
of
task
or
a
new
sort
of
goal
that
it
has
to
solve.
A
But
it's
it's
not
going
to
be
as
abrupt
as
it
would
be
in
supervised,
continual
learning,
because
this
is
like
a
real
world
kind
of
evolving
setting.
So
there's
it's
a
more
natural
setting
in
which
to
study
these
problems
and
there's
also
like
more
modes
of
relationships
between
the
data
that
the
agent
can
see.
A
The
notion
of
time
dependence
is
also
a
challenge
in
reinforcement,
learning,
especially
deep
reinforcement,
learning,
because
a
lot
of
times,
the
kind
of
gradient
descent
assumes
that
you
have
iid
data.
But
now
you
have
like
time
correlated
data.
So
there's
a
lot
of
research
on
how
how
to
deal
with
that
as
well.
A
A
So
one
way
of
doing
this
is
shared
parameters
where
the
actual
policy
parameters
are
shared
across
tasks,
so,
like
the
very
naive
way
to
do
this
is
to
have
like
one
policy,
doing
everything
without
any
sort
of
modifications
or
having
different
having
one
policy.
That's
composed
of
separate
policies
for
each
task.
A
Another
really
common
and
kind
of
promising
approach
that
the
authors
point
out
is
learning
a
good
shared
representation
of
the
environment
that
can
be
used
across
tasks.
So
this
is
like
a
very
good,
very
kind
of
fruitful
direction
of
research
and
then
even
having
like
one
single
representation
that
the
agent
develops
throughout
its
lifetime.
A
Another
way
that
reaches
researchers
have
tried
to
do.
This
is
to
learn
a
prior
on
how
much
each
parameter
is
used
in
the
past
for
certain
types
of
settings
and
then
using
this
prior
to
update
those
parameters
in
a
corresponding
way.
So
this
explicitly
kind
of
controls,
the
plasticity
of
that
parameter.
A
Okay,
thanks
so
distillation,
based
I'm
not
too
familiar
with
this.
But
it's
kind
of
the
notion
is
that
you
use
one
kind
of
neural
network
as
a
target
for
another.
So
this
this,
so
you
could
have
one
neural
network.
That's
already
been
trained
on
a
task,
and
then
you
use
that
one
as
like
the
update
target
for
another,
and
this
kind
of
keeps
the
updates.
Conservative
is
the
way
the
authors
put
it.
A
I
haven't
read
too
many
papers
in
this
kind
of
regime,
so
I'm
not
very
familiar
with
it,
but
it's
kind
of
a
direction
that
people
have
tried
in
the
past,
so
it
encourages
stability,
because
your
updates
are
very
conservative,
so
you're
kind
of
very
basing
it
off
of
some
sort
of
past
network
structure
that
you've
learned
rehearsal
based.
This
is
what
lucas
kind
of
touched
upon.
So
a
big
concept
in
reinforcement
learning
is
experience
replay.
A
So
you
store
a
bunch
of
past
experiences
in
some
sort
of
buffer
or
database,
and
you
use
that
throughout
your
learning
process,
to
update
yourself
so
rehearsal,
based
methods,
kind
of
explicitly
make
modifications
on
experience
replay
so,
for
example,
figuring
out
like
which
experiences
are
the
most
important
or
like
figuring
out
when
to
kind
of
forget
or
drop
certain
pieces
of
data,
because
if
you're
a
lifelong
learning
agent,
you
can't
just
keep
adding
data
to
your
to
your
database.
You
have
to
you,
have
to
truncate
it
at
some
point.
A
B
D
D
C
D
A
A
So
there's
like
a
really
big
challenge
here,
which
is
like
they
call
it
like
a
chicken
and
egg
problem
where
you
need
there's
like
a
simultaneous
learning
of
both
the
actual
modules
and
how
to
combine
them.
So
when
you're
learning
to
combine
them
you,
if
you
learn
to
combine
them,
then
the
modules
will
change
at
the
next
step,
so
you're
kind
of
chasing
something
that's
already
moving.
A
But
this
is
like
an
active
area
of
research.
State
state
abstraction
is
another
other
area
where
you
have
you.
Instead
of
using
like
the
default
state
that
the
that
you
observe
you
map
it
into
a
different
state
representation
that
is
common
across
that
that
kind
of
captures
things
that
are
common
across
tasks
and
across
settings
that
the
agent
has
been
in,
so
these
states
should
encapsulate
like
task
agnostic,
shared
state
information,
that's
important
for
the
agent.
A
Another
another
area
that
I
particularly
like
is
skill
focused,
so
this
is
kind
of
taking
a
hierarchical
view
on
actions
where,
if
you're,
for
example,
you're
learning
macro
actions
which
control
lower
level
actions.
So
if
I'm
like
driving
a
car,
I
could
take
an
action
that
says
turn
right.
I'm
not
really
thinking
about
the
actual
motor
commands
that
my
arm
does.
A
In
order
to
turn
right,
I
just
kind
of
think
at
a
higher
level
of
turning
right,
so
learning
to
break
break
apart
decision
making
points
into
these
kind
of
higher
level
and
lower
level
actions
and
then
using
higher
level.
Skills
that
are
common
across
tasks
is
like
another
approach
that
the
authors
point
out.
A
A
A
kind
of
related
but
different
perspective
is
goal
focused.
So
you,
the
agent,
determines
like
what
its
goal
is
at
a
current
point
in
time
and
once
it
determines
this,
it
takes.
It
can
take
a
sequence
of
actions
to
to
achieve
this
goal,
and
the
authors
do
a
good
job
of
kind
of
talking
about
how
goal
is
like
a
kind
of
vague
notion,
so
that's
kind
of
a
benefit,
because
you
can
define
goals
in
a
many
way.
A
So
the
policy
is
conditioned
on
the
goal
and
kind
of
exciting
direction
is
learning
goals
in
an
unsupervised
manner.
So,
like
the
agent
kind
of
through
its
interactions
with
the
world,
learns
a
representation
of
like
okay.
What
are
goals
that
are
useful
for
me,
and
once
I
determine
those
goals,
how
can
I
take
actions
to
achieve
them?
And
this
is
done
without,
like
any
sort
of
external
external
thing
specifying
that
goal
to
the
agent.
A
And
the
last
kind
of
thing
here
is
auxiliary,
auxiliary
tasks,
where
this
is
something
that
jeff
mentioned
at
the
beginning
kind
of
augmenting.
The
typical
reward
training
signal
with
additional
denser
ones,
denser,
meaning
it.
It's
observed
at
like
more
frequent
time
steps
to
encourage
the
agent
to
learn
about
task
agnostic
dynamics
of
the
world
in
the
past.
This
has
been
like
very
much
hand
designed
so
you'd
have
like
a
domain
expert
like
a
human
being
kind
of
engineering.
The
reward
signal
to
encourage
certain
types
of
behaviors
that
we
know
to
be
beneficial.
E
A
Yeah,
so
you
need
like
the
as
like
an
algorithm
designer.
I
guess
you
need
to
come
up
with
that
notion
of
an
intrinsic
measure
of
how
good
a
goal
is,
but
it's
usually
related
to
some
something
along
the
lines
of
like
information
theoretics,
like
kind
of
how
much
information
do
you
gain
about
the
world
using
that
goal
or
reward
associated
okay.
A
A
Well
like
so,
for
example,
if
the
notion
of
a
goal
is
like
a
desired
state,
you
yourself
kind
of
you
determine
the
agent
has
to
determine
what
are
good
states
to
get
to
in
an
unsupervised
way
and
achieve
and
get
to
those
states,
and
then
once
it
does,
that
it
can
like
when,
when
you
actually
give
it
like
a
reward
or
when
you
put
it
in
a
setting
where
there
is
a
reward,
it
can
determine
okay.
What
goal
should
I
set
myself
that
will
maximize
my
reward?
A
A
D
A
A
F
Hey
gosh
one
quick
question:
could
you
talk
a
little
bit
more
about
the
modularity
and
composition,
because
when
you
said
that
my
mind
immediately
thought
of
the
like
skill,
focused
bullet
points
so
like?
Can
you
just
talk
about
what
the
like
what
these
neural
network
modules
are?
Are
they,
like
you,
know,
estimating
the
value
function
or
is
it
like
the
policy
network
or
like
what?
What
exactly
is
being
composed
here.
A
Yeah,
that's
a
good
question,
so
I
think
usually
based
on
what
I
what
I
read
each
module
like
the
most
explicit
ways
like.
If
you
have
a
series
of
tasks
you
have
like
a
module
for
each
and
then
you,
when
you
see
a
new
task,
you
can
learn
how
to
you
can
figure
out
how
to
kind
of
compose
these
to
solve
the
new
tasks.
That's
like
kind
of
a
very
explicit
way
of
doing
it,
and
usually
that's
done
on
the
policy
network,
not
like
the
value
network,
it
okay.
A
Another
another
thing
is
like
I.
I
read
a
recent
paper
where
it's
like
soft
modular
modularization,
where
you
don't
have
like
explicit
modules,
but
you
just
have
like
different
sub
networks
that
are
activated
for
certain
settings
and
then
you
need
to
figure
out
if
you're,
in
a
new
setting
like
what
subnetworks
are
most
relevant
to
solve
that.
So
I
thought
that
was
like
pretty
interesting
as
well.
D
A
A
Okay-
and
I
can
go
back
to
that-
if,
if
there's
any
additional
questions,
the
last
last
point
the
the
last
kind
of
umbrella
they
they
mention
is
meta
learning
approach.
So
the
first
is
context
detection,
which
is
very
specific
to
reinforcement
learning.
I
think
so.
This
kind
of
goes
back
to
the
view
that
the
mdp
stationary,
so
it's
like
the
physics,
are
fixed,
but
the
agent
can't
fully
observe
the
true
state.
A
A
So
there's
a
lot
of
like
theoretical
work
on
like
how
you
can
reformulate,
like
the
actual
mdp
into
an
mdp
that
explicitly
models
the
task
state
and
then
the
agent
has
to
kind
of
learn
to
develop
beliefs
about
this
and
the
word
context.
Detection
is
focused
on
okay,
discovering
these
test
states
and
once
you've
discovered
them
kind
of
acting
optimally,
and
I
think
a
lot
of
work
has
focused
on
like
kind
of
a
bayesian
perspective
on
this
and
like
developing
a
belief
on
what
the
task
state
could
be.
A
Another
kind
of
related
view
is
like
okay,
how
to
detect
points
or
boundaries
where
the
task
or
distribution
that
the
agent
is
observing
kind
of
switches.
So
the
the
author
is
called
this
change
point
detection,
so
kind
of
automatically
figuring
out.
Okay.
This
is
a
point
in
which
the
task
has
switched
or,
like
the
the
environment
dynamics,
have
kind
of
moved
from
one
regime
into
another.
A
So
this
this
other
bullet
point
learning
to
adapt
is
more
of
like
a
classic
meta
learning
perspective,
where
the
app
the
agent
has
to
actually
learn
the
optimization
process
to
adapt
quickly
to
a
new
setting.
So
if
it
transitions
into
a
new
setting,
how
does
how
should
the
agent
update
itself
in
order
to
better
adapt?
A
So
this
the
classic
method,
learning
approach
where
there's
like
two
loops,
an
inner
loop
for
like
fast
adaptation
in
a
new
setting
and
an
outer
loop
that
is
determining
okay?
How
do
I
learn
to
learn
so
like?
How
do
I
actually
update
this
inner
policy
to
better
adapt
and
do
something
and
the
last
one,
which
I
think
is
really
cool,
is
like
learning
to
explore.
A
B
B
It's
it's
again
surprising
and-
and
it
just
tells
me
this
is-
it
seems
like
it's
so
far
away
from
like
sort
of
reaching
a
consensus
on
what
I
mean,
you
must
have
gone
through
20,
different
methods
that
people
are
using
for
this,
and
it
just
seems
like
a
that's,
a
temporary
state
of
confusion.
It
feels
like
to
me.
A
B
I
mean,
if
you
think,
like
the
brain
right,
you
know
the
brain.
Those
will
sometimes
solve
all
these
problems
and
I
can't
be
certain
about
every
one
of
them,
but
and
yet
there's
a
common
methodology.
We
don't
have
one
part
of
the
brain
doing
this
another
part
of
the
brain.
Doing
that
you
know,
there's
a
there's,
a
core
algorithm,
which
is
very,
very
flexible.
B
That
does
almost
all
these
things,
and
but
here
I
I
don't
get
in
yeah
yeah,
you
could
say
well,
they're,
not
completely
separable.
You
could
use
a
curiosity
thing
for
something
else,
but
it
just
it
just
again.
I
could
be
wrong.
Tell
me
if
I'm
wrong,
but
it
feels
like
this
has
not
coalesced.
Yet
you
know
this.
It's
a
it's
a
hundred
different
approaches
to
solve
100
things,
but
if
we're
missing
the
big
picture
it
feels
like
these
have
to
consolidate
into
some
sort
of
uber
policy
or
uber
methodology
of
learning.
D
A
Right-
and
I
think
like
it's
true,
that's
true
and
like
a
lot
of
machine
learning,
so
like
computer
vision,
right
like
there's
like
people
that
focus
on
segmentation
and
like
edge
detection
and
classification
and
like
object
tracking
so
like,
but
and
they
they
kind
of
tackle
each
of
these
independently.
Whereas
the
brain
kind
of
does
this
all.
B
Yeah
yeah
and
that
to
me
is
always
a
sign
that
well
it's
not
right.
You
know
you
can
attack.
If
you
have
to
attack
each
little
key
separately,
then
you
don't
have
the
right
approach
that
somehow
this
they've
got
to
get
united
and
anyway
it's
I'm
not
being
critical.
I'm
just
pointing
out
that's
wow!
I
was
impressed
amazed
that
there's
so
many
different
types.
I
think
way
people
are
going
about
this.
D
Yeah,
but
this
variety
is
not
just
in
terms
of
methods,
but
also
in
terms
of
how
you
define
the
problem,
and
this
connects
back
to
the
presentation
that
michelangelo
gave
last
week
that
we're
discussing
you
know
there
is
a
lot
of
confusion
right
now
in
how
do
you
define
the
continual
learning
problem
and
that's
the
same
here?
Reinforcement
learning,
so
we
are
talking
about
different
algorithms,
but
each
one
of
those
algorithms
solves
a
slightly
different,
continuous
learning
problem.
We
don't
have
consensus
on
that.
Yet.
A
A
How
do
tasks
evolve?
So
it's
kind
of
hard
to
come
up
with
simulation
environments
that
that
touch
upon
this
so
like
what
people
have
done
now
is
like
have
like
video
game
settings
where
there's
like
different
levels,
so
the
agent
might
have
to
like
progressively
update
itself
in
changing
scenarios
where
the
scenarios
are
like
what
level
it
sees
kind
of
like
a
super
mario
bros.
B
A
B
B
B
You
can
say
well,
the
ones
in
the
retina
are
moving
independently,
but
they
could
in
theory,
in
this
case
they're
not
but
touch
they
are
moving
independently
and
and
so
there
you
know,
even
vision
is
not
one
in
one
sense,
you
could
call
it
one
modality,
but
from
the
brain's
perspective,
it's
just
thousands
of
sensors
that
are
moving
and
and
each
one
is
building
its
own
model.
That's
a
thousand
brain
theory.
So
I
just
think
this
way
of
thinking
about
it
is
incorrect
and
I
don't
you
know.
A
And
then
like
metrics,
so
like
measuring
how
how
good
or
bad
like
the
level
of
catastrophe
you're
getting
is
so
this
this
could
be
something
like
kind
of
training
having
the
asian
go
back
to
a
task
that
hasn't
seen
in
a
while
and
then
seeing
how
it
performs,
or
it's
still
a
problem
of
like
how
do
you
actually
explicitly
model
this?
Maybe
skill
reusability
is
another.
Another
idea
that
the
authors
touch
upon
is
like.
A
If
you
take
this
skill
based
or
like
macro
action
versus
micro
action
perspective,
how
well
do
these
macro
actions?
How
often
are
they
reused
across
tasks
and
how?
How
adaptable
are
they
and
can
they
be
composed
into
and
to
solve,
like
a
more
complex,
complex
task
and
also
like
how
does
the
agent
plan
in
the
environment
that
it's
in
so
these
these
last
three?
D
One
one
metric,
which
is
not
there,
we've
seen
in
supervised
learning,
is
a
forward
transfer
and
backward
transfer
and
that
might
fit
into
his
queue.
We
use
a
bit
it
that
just
means
whatever
I
learned
now
can
help
in
a
future
task
or
I'll
ever
learn
in
future
tests
and
actually
help
the
tasks
that
I
learned
before.
So
it's
not
just
about
catastrophic,
forgetting.
A
How
does
the
brain
structurally
address
this?
So
this
is
not.
This
point
is
not
specific
to
like
reinforcement
learning
this
one
is
so
like
what
is
the
reward
signal
that
humans
optimize,
for
this
is
kind
of,
I
think,
there's
some
research
here
that
I'm
not
super
familiar
with,
but
still
like
an
open
question.
A
How
can
we
build
algorithms
that
better
incorporate
like
better
modeled
how
human
memory
and
human
replay
happen?
So
the
authors
point
some
papers
that
talk
about
how
like
dreaming
during
sleep
for
humans
is
like
super
important
for
kind
of
consolidating
experiences
and
consolidating
like
certain
actions
as
like,
more
as
more
like
ingrained
within
the
human's
like
way
of
acting.
A
How
does
the
brain
combine,
model-free
and
model-based
learning
so
there's
like
evidence
that
both
is
occurring
in
the
human
brain,
but
it's
not
yet
clear
how
it's
combined
so
like
figuring
out
how
the
brain
does.
This
can
be
helpful
in
building
better
algorithms
and
does
the
brain
exhibit
modular
structure
where
there's
different
modules
that
do
different
things
and
there's
like
a
different
aspect
of
the
brain
that
determines
how
to
compose
these
to
solve
certain
problems.
A
B
B
Bad
things
I'm
not
saying
but
but
they're
very
fuzzy.
How
can
corporate
better
human
memory
replay
well
to
understand
human
memory
is
not
you
have
to
really
get
down
in
the
details,
exactly
the
kind
of
stuff
we
do
at
nomenta,
but
if
you
focus
on
these
sort
of
very
sort
of
you
know,
there's
studies
about
sort
of
that
are
not
detailed
studies,
they're,
sort
of
testing
animals
in
certain
vitamins
touching
humans
inside
you're,
just
not
going
to
get
there
anyway.
So
you
know
it's
just.
B
B
It's
like
you
know
you,
you
gotta!
You
have
to
spend
months
reading
papers.
I
understand
anyway,
again
not
trying
to
be
critical,
but
just
point
critical,
I'm
just
pointing
out
that
I
think
this
level
of
acknowledgement
in
the
earth
science
is
very
shallow
in
some
ways,
but
that's
not
surprising,
because
this
is
what
we
do
and
very
few
other
people
do
it.
F
D
B
That
was
an
interesting
hypothesis,
although
the
way
one
way
to
look
at
dreaming
is
just
take
dreaming
out
of
the
picture
say
you
have
a
human
or
an
agent,
an
animal
and
say:
what
can
it
do
during
the
day?
It's
not
sleeping
at
all.
It
can
form
new
models
of
the
world.
It
can
act
on
everything
it
can
do
all
these
things.
Absolutely
everything
we're
talking
about
here
and-
and
you
don't
need
the
dream
right.
So
you
from
that
point,
you
can
say
well
something's
going
on
here
where
I
can
learn
all
these
things.
B
I
can
learn
new
skills.
I
can
do
this
now,
maybe
somehow
the
consolidation
over
long
term
has
to
occur
during
the
dream
or
something
like
that,
but
the
mechanisms
that
for
all
the
things
we
can
do,
are
there
without
dreaming
without
sleep
and
so
and
most
of
the
theories
of
sleep
and
I'm
not
an
expert
in
this
at
all.
But
most
sisters
of
sleep
have
to
do
with
other
things.
B
You
know
cleaning
up
junk
in
the
brain,
re-establishing
connections,
you
know
chemical
processes
and
who
knows
but
but
you
can
just
say
yourself
well,
I
can
you
know
over
over
a
20-hour
period.
I
can
do
all
these
things
in
a
house
of
sleep,
so
sleep
is
not
an
absolute
requirement
to
perform
these
functions.
It's
it's
got
to
be
a
supplementary
to
it.
So
that's
why
we've
never
really
focused
on
sleep
when
I'm
working,
because
it
doesn't
seem
to
be
necessary
since
it's
necessary
for
an
animal,
and
it's
still
somewhat
of
a
mystery.
B
B
G
F
D
B
B
And
that
and
that's
how
they
show
pretty
much,
every
animal,
the
brain
sleeps,
you
know,
and
even
it
comes
you're
right.
It's
it's
amazing
how
preserved
that
is,
so
it
must
be
really
really
fundamental,
but
it
doesn't
again
that
could
be
very
fundamental
to
the
biology
of
the
brain.
Just
like
oxygen
is
in
you
know,
glutamate
or
you
know
these
are
fundamental
things
brains
have
to
have.
B
B
I
I
don't
know,
but
I
think
any
animal
with
brain
sleep.
I
think
that's
that's
the
rule.
I
think
I've
read
so
you
know
I
don't
know,
I
mean
fish
sleep
and
what,
if
you're
a
fish
down
in
the
dark
depths
of
the
ocean,
they
sleep
so
fish
fish
that
swim
continuously,
birds
that
birds
that
fly
continuously
sleep,
but
they
only
sleep
half
the
brain
at
a
time.