►
From YouTube: Task-Agnostic Continual Reinforcement Learning: In Praise of a Simple Baseline - June 10, 2022
Description
Guest speaker Massimo Caccia introduces a simple baseline for task-agnostic continual reinforcement learning (TACRL). He first gives an overview of continual learning, reinforcement learning, and TACRL. He then goes through empirical findings that show how different TACRL methods can be just as performant as common task-aware and multi-task methods.
Papers:
“Task-Agnostic Continual Reinforcement Learning: In Praise of a Simple Baseline”: https://arxiv.org/abs/2205.14495
"Avoiding Catastrophe: Active Dendrites Enable Multi-Task Learning in Dynamic Environments": https://www.frontiersin.org/articles/10.3389/fnbot.2022.846219/full
- - - - -
Numenta is leading the new era of machine intelligence. Our deep experience in theoretical neuroscience research has led to tremendous discoveries on how the brain works. We have developed a framework called the Thousand Brains Theory of Intelligence that will be fundamental to advancing the state of artificial intelligence and machine learning. By applying this theory to existing deep learning systems, we are addressing today’s bottlenecks while enabling tomorrow’s applications.
Subscribe to our News Digest for the latest news about neuroscience and artificial intelligence:
https://numenta.com/news-digest/
Subscribe to our Newsletter for the latest Numenta updates:
https://tinyurl.com/NumentaNewsletter
Our Social Media:
https://twitter.com/Numenta
https://www.facebook.com/OfficialNumenta
https://www.linkedin.com/company/numenta
Our Open Source Resources:
https://github.com/numenta
https://discourse.numenta.org/
Our Website:
https://numenta.com/
A
So
task
agnostic,
continual
rl,
is
the
subject
of
the
day
and
basically
today,
I'm
going
to
talk
about
a
really
simple
baseline,
intel's
agnostic,
continual
rl,
and
how
awesome
it
is
okay.
So
I'm
not
here
to
sell
a
fancy
new
method,
so
here's
the
tldr,
if
you're
like
me,
you've,
been
interested
in
continual
learning
for
a
while
and
you've
been
mainly
working
on
this,
like
incremental
image
classification
problem.
A
A
So
that's
kind
of
my
motivation
for
doing
continual
learning
and
then
today
what
I'm
going
to
talk
about
is,
if
you
align
with
this
use
case
of
deploying
these
agents
in
the
world
that
learn
autonomously,
then
the
the
the
the
the
upper
bounds
that
we
use
in
continual
learning.
So
this
idea
of
like
using
the
training,
your
algorithm
on
stationary
data
and
seeing
and
comparing
it
to
your
considerable
learning
method,
to
understand
how
much
forgetting
there
is
in
a
benchmark.
A
So
if
you
come
from
the
the
field
of
continual
supervised
learning,
then
I
I
think
the
conclusions,
or
actually
or
at
least
the
observations
that
we're
gonna
make,
can
be
quite
surprising.
Okay,
so
hopefully
that's
clearer
at
the
end
of
the
talk.
But
basically
what
I
will
do
is
I'll
do
a
quick
background
and
continual
learning,
then
I'll
jump
I'll.
Do
a
quick
review
of
reinforcement.
Learning
then
I'll.
Try
to
explain
the
setting
of
task
agnostic,
continual
rl,
then
we'll
talk
about
the
bounce.
A
A
Else,
but
I
think
most
of
us
yeah,
okay,
okay,
so
basically,
consumer
learning
is
about
accumulating
knowledge
on
non-stationary
data
distributions
and,
sadly,
our
deep
machine
learning
methods
or
continual
learning
methods.
They
tend
to
forget
when
we
train
them
on
non-stationary
data
distribution,
as
shown
in
this
little
example.
A
We
call
this
big
drop
in
performance
when
the
task
changes
or
the
data
distribution
changes,
catastrophic,
forgetting
and
even
though,
at
least
for
me,
the
the
motivation
for
doing
continual
learning
is
this
idea
of
learning
new
tasks
faster
and
better
right
now,
the
field
mostly
studies,
the
phenomenon
of
catastrophic
forgetting
and
trying
to
solve
it.
So
how
can
we
make
our
algorithms
remember?
B
It's
you
know
in
many
cases,
if
you
think
about
lifelong
learning
of
an
agent
in
the
world
or
something
it's
not
necessarily
that
the
the
the
statistics
are
non-stationary,
it's
more
like
you've,
only
sampled,
a
small
part
of
the
world,
and
then
you
can
continually
get
exposed
to
more
and
more
of
the
world.
But
then,
when
you
go
back
to
where
you
had
sampled
before
the
distribution
hasn't
changed,
it's
still
the
same.
You
just
don't
want
to
forget
it
right.
A
B
B
A
A
B
A
B
A
So
an
mdp
is
this
here,
it's
unrolled,
so
it's
this
cute
little
thing
where
there
is
a
state
variable
of
the
world
and
then
an
agent
here
in
blue
is
going
to
take
some
actions
which
will
throw
him
or
it
in
a
new
state
and
will
deliver
a
reward
okay.
So
the
name
of
the
game
in
rl
is
to
find
a
policy
pie
or
that
maximizes.
The
cumulative
rewards
that
the
agent
accumulates
in
his
life
or
in
an
episode
you
know.
A
So
what
is
really
important
about
mdps
is
that
one
the
state
of
the
world
is
fully
observable
by
the
agent
and
two.
The
state
of
the
world
is
also
markovian.
Okay,
so
basically
we're
assuming
that
the
agent
at
every
time
step
its
tensors
is
giving
it
all
of
the
information
it
needs
to
take
optimal
actions.
A
A
You
need
something
that's
much
more
general,
which
brings
us
to
the
partially
observable
mdp
okay.
So
now
here
the
states
of
the
world
in
gray
is
not
observed
by
the
agent.
The
agent
only
observes
observations
that
are
called
xt
here
and
yeah.
So
now
we're
not
assuming
that
at
any
time
step
the
agent
with
its
sensors
can
get
all
of
the
information
that
is
relevant
to
it
to
take
optimal
action.
Okay,
so
we're
assuming
we're
taking
into
account
that
much
more
is
happening
beyond
our
our
sensors.
A
Okay,
and
I
think,
if
we're
serious
about
you
know
eventually
deploying
agents
in
the
world
that
could
learn
and
accumulate
knowledge,
we
need
to
frame
our
thinking
in
palm
dps
and
not
mvps
right
like
right
now.
I
I'm
talking
to
you
guys,
but
so
much
is
happening
beyond
my
sensors.
The
stock
market
is
changing
and
this
has
an
effect
on
my
portfolio.
A
Someone
is
reviewing
my
paper
and
I'll
be
sad
if
it's
rejected
later
on.
So
a
lot
is
happening
beyond
our
sensors,
so
I
think
pump
dp's
is
important.
Now,
where
does
task
agnostic
continual
rl?
Coming?
Oh
yeah,
sorry,
I
skipped
something.
If
you
want
to
act
optimally
optimally
in
a
palm
dp,
you
need
to
take
into
account
all
of
the
previous
observations
and
rewards
okay.
So
this
scales
really
poorly
obviously,
but
we
will
shut
this
under
the
rug
for
now
and
come
back
to
it
later.
Okay.
A
So
now
oh
yeah,
with
the
pump
dps
now
you
can
do
more
interesting.
Things
like
maybe
play
poker
where
the
hidden
state
would
be
the
cards
that
your
opponents
have,
or
you
can
think
about
like
more
interesting
robotics
example.
Maybe
I
want
my
robot
to
help
me
build
a
house
where
so
in
this
case
my
robot
probably
needs
to
keep
inferring
what
the
state
of
the
house
is
and
what
other
people
are
doing
in
order
to
in
order
to
take
optimal
actions
so
yeah.
A
Now
we
can
do
much
cooler
things
than
before.
Okay,
so
now
I'll
try
to
explain
what
task
economic
continual
rl
is
as
a
palm
dp
special
case
okay.
So
it's
going
to
look
bad
at
first,
but
you'll
understand
later,
where
I'm
trying
to
get
to
okay.
So
the
first
thing
we're
gonna
do
is
instead
of
having
a
state
of
the
world
that
is
unobserved
and
an
observation
that
is
sampled
from
that
state.
A
Instead
we're
going
to
reap
our
reparament
re-parametrize,
the
mdp,
the
palmdp,
such
that
the
state
of
the
world
is
separated
in
two
parts,
an
unobserved
part.
So
here
it's
sh
and
an
observed
part
okay.
So
I'm
just
re-parameterizing
things
now
the
graph
looks
more
complicated,
but
essentially
it's
the
same
thing.
Okay,
and
now
we
can
do
things
like
point
out
what
certain
mechanisms
are
doing.
For
example,
here
the
purple
arrows
are
basically
representing
the
mechanism
through
which
the
hidden
state
is
changing.
A
The
dynamics
of
the
world,
okay,
whereas
the
orange
line
represents
how
the
hidden
state
changes
the
reward
function
of
the
agent
okay.
So
now
we
can
talk
about
those
and
yeah
now
to
ex
to
further
explain
tactics
to
continual
url.
I
like
to
come
back
to
this
stupid
running
example,
which
is
a
robot
learning
to
do.
A
Massages,
okay
and
you'll
understand
later
why
it
makes
sense,
so
the
robot
basically
has
a
distribution
of
two
clients,
one
likes
rough
massages
or
like
therapeutic
massages,
and
one
likes
relaxing
massages
or
soft
massages.
Okay
and
the
agent
needs
to
enter
what
the
client
likes.
Okay,
so,
basically
here
the
hidden
state
would
be
the
client's
preferences.
A
A
The
hidden
state
is
either
I
like
it
relaxing
or
I
like
it,
therapeutic
okay.
So,
basically,
now
I'm
trying
to
make
some
assumptions
to
make
the
palm
dp
more
easily
manageable,
such
that
I
can
study
this
idea
of
like
continually
accumulating
knowledge
or
continually
accumulating
skills,
which
is
what
we're
ultimately
interested
in
in
continual
learning.
A
So,
okay,
so
now
the
hidden
state
is
simpler.
It's
just
a
single
categorical
variable,
and
now
I'm
going
to
make
another
assumption,
I'm
going
to
assume
that
the
hidden
states
or
the
task
variable
it's
non-stationary
and
more
specific,
specifically
it's
locally
stationary,
okay.
So,
basically
in
the
palm
dp
in
theory,
my
the
client's
preferences,
they
could
change
at
each
time
step
randomly,
but
in
task
agnostic,
continual
rl,
I'm
going
to
assume
some.
You
know
continuity
in
that
task.
A
A
So
this
will
help
me
study
what
we
are
interested
in
next,
I'm
going
to
assume
that
the
agent
cannot
change
the
or
has
no
effect
on
the
hidden
state
of
the
world
or
the
task
variable,
because
this
is
fundament
fundamentally
not
important
for
the
problem.
So
we're
going
to
remove
that
link
to
make
things
simpler.
A
A
I
think
it's
much
more
important
to
the
the
fundamental
problem
we
want
to
study
so
basically
in
this
work,
we're
just
going
to
remove
the
the
link
where
the
agent
can
change
the
physics
of
the
world
or
change
the
dynamics
of
the
world
and
we're
just
going
to
focus
on
a
task
variable
that
can
change
the
reward
function.
Okay,
so
I
hope
that
this
was
somewhat
clear
and
so
yeah.
A
A
Oh
wow,
thank
you
very
much
so
yeah
just
a
super
quick
summary.
We
started
with
the
pundi
p
whoops.
We
assumed
non-stationary
hidden
states,
we
assume
a
passive
non-stationarity,
so
the
agent
cannot
change
the
hidden
state
of
the
world.
We
also
assume
that
the
hidden
state
is
just
a
single
variable,
which
we
often
call
task
id
for
better
or
for
worse.
A
Okay,
so
in
task
agnostic,
continual
rl,
just
like
in
task
agnostic,
supervised
learning,
you
can
compute
a
task
awareness,
soft
upper
bound
by
providing
a
task
label
to
an
agent
okay.
So
in
the
if
we
go
back
here,
if
I
provide
the
the
task
variable
to
the
agent,
my
problem
just
collapse,
collapses
back
to
an
mdp
okay.
A
I
can
do
a
similar
thing
if
I
want
to
understand
how
much
is
lost
due
to
forgetting
okay.
So
if
I
have
an
algorithm
in
mind,
I
can
always
run
it
on
a
stationary
data
distribution
and
then
by
comparing
its
performance
with
the
performance
of
the
continual
learning
algorithm,
I
can
understand
how
much
is
lost
in
terms
of
forgetting
okay,
so
these
two
bounds
are
well
established
in
continuous
supervised
learning
and
I
think
also
in
conditional
reinforcement
learning
and
they
will
be
really
important
for
the
experimental
part
of
this-
the
presentation,
okay
and
yeah.
A
A
A
A
What
I
can
do
is
I
can
condition
on
all
the
previous
time
steps
of
my
life,
but
that
that
scales
really
poorly
the
one
kind
of
simple
trick
I
can
do
is
I
can
learn
a
variable
that
is
going
to
represent
the
past
time
steps.
Okay,
so
here
z,
t
would
represent
whatever
happened
between
zero
and
t
in
terms
of.
A
Observation
reward
in
action:
I
guess
I'm
missing
a
little
arrow
here,
and
so
basically
you
can
think
about
zt
as
a
memory
that
the
agent
is
going
to
use
to
take
its
actions
and
one
simple
way
to
instantiate.
This
memory
is
with
a
recurrent
neural
net
or
an
rnn
okay.
So,
basically
to
end
all
the
partial
observability
of
my
problem,
I'm
just
gonna
mount
an
rnn
to
my
favorite
rl
algorithm.
Okay,
so
that's
that's
one
part
of
the
simple
baseline
that
we're
building
and
then
the
other
part
is
super
simple.
A
I
need.
I
need
something
to
end
all
forgetting
because
I'm
in
a
continual
learning
scenario,
so
I'm
going
to
do
the
simplest,
continual
running
method,
I'm
going
to
use
experience
replay
okay
to
to
alleviate
my
forgetting.
So
in
case
you
are
not
aware
of
that
method.
Experience
replace
super
simple,
I'm
keeping
I'm
gathering
data
in
the
buffer
and
then
when
I
learn
a
new
task.
I
augment
my
mini
batch
with
old
data.
A
Just
to
be
sure,
I
don't
forget
it
okay,
so
this
takes
care
of
or
forgetting,
but
I'm
paying
a
huge
compute
price
right.
If
I'm,
if
I'm
learning
100
tasks-
and
I
want
to
keep
my-
I
want
to
keep
approximating
a
stationary
data
distribution
in
my
mini
batch.
Well,
I'm
going
to
be
spending
like
99
of
my
compute
replaying,
all
data
and
just
one
percent
learning
the
new
task.
So
the
experience
replay
works,
but
naively
it
doesn't
scale
very
well.
A
A
I
have
experience
replay
that
I
just
explained,
and
then
I
have
the
multitask
strategy.
This
one
gets
to
train
on
all
tasks
simultaneously,
and
so
it
doesn't
incur
forgetting
so
that's
my
three
strategies
and
then
for
my
task
aware
baselines.
So
the
baseline
that
can
observe
the
task
ib,
basically
we'll
we
can
create
two
more
baseline
with
the
task
id.
We
can
either
use
it
to
create
a
multi-ed
algorithm.
A
So
this
algorithm
is
super
simple:
it
has
the
same
representation,
it
learns
the
same
representation
for
all
tasks
and
then
it
enjoys
one
output
head
per
task
and
this
output
that
is
basically
going
to
choose
the
proper
action
for
its
task.
Okay.
So
it's
a
really
really
common
baseline
to
create
task,
aware
method.
A
A
C
A
Okay,
now
there
is
a
backbone
algorithm.
We
use
throughout
all
the
experiments.
It's
called
soft
actor
critic.
I'm
gonna
try
to
explain
it
in
like
30
seconds.
Okay,
so
basically
you
start
from
q
learning
where
you
are
learning,
basically
the
the
value
of
state
action
pairs
in
a
table,
and
then
the
policy
of
the
agent
is
implicit
at
any
state,
you're
just
going
to
choose
the
action
that
is
best
or
that
has
the
most
value
in
a
particular
state.
A
Okay,
so
that's
super
simple,
but
it
doesn't
scale
very
well
if
you
have
a
large
state
spaces
right.
So
one
thing
we
can
do
is
we
can
approximate
the
values
of
the
state
or
the
q
value
the
q
value.
We
can
approximate
it
and
what
better
way
to
approximate
it
is
yeah.
We
can
approximate
it
with
a
neural
net
and
that's
what
we
call
dq
n
or
dq
learning.
Okay
and
then
again
the
policy
is
implicit.
I'm
just
going
to
choose
the
best
action.
A
So
that's
where
that's,
where
dot
factor
credit
comes
in
in
soft
actor
critic.
Now
my
policy
won't
be
implicit.
I
will
actually
parametrize
my
policy
and
I
will
train
it
to
maximize
the
values
of
my
q
function.
Okay,
so
basically
the
policy
is
just
learning
about
the
world
through
the
q
value.
It's
just
learning
everything.
A
It
is
all
of
the
information.
It
knows
bleeds
in
through
the
q
value,
so
it's
kind
of
like
a
it
looks
a
bit
like
a
generative
adversarial
network
where
there's
two
neural
nets
and
the
generator
learns
everything
through
the
discriminator.
So
that's
how
I
like
to
think
about
sat
but
anyways.
It's
not
super
important
for
the
conclusion
of
the
paper.
Just
just
yeah.
Just
oh,
you
know,
okay,
so
something
that
is
super
important
for
the
experiment
is
the
benchmark.
A
A
So
a
lot
of
these
continual
rl
papers,
what
they
will
do
is
use
these
little
mutual
tasks
to
create
continual
reinforcement,
learning
environments.
So,
for
example,
this
could
be
like
an
ant
that
needs
to
move
in
different
directions
and
then
the
task
would
be
the
particular
direction
that
the
ant
needs
to
go
to.
Okay.
A
On
the
other
end,
a
lot
of
work
in
continual
rna
has
been
on
atari,
so
these
little
games,
but
now,
in
my
opinion,
the
tasks
are
two
dissimilar,
so
I
don't
think
you
can
get
much
transfer
by
learning
pong
and
then
learning
how
to
play
mario
right,
I
think
the
task
or
to
the
simulator.
So
it's
not
really
interesting
to
study
this
idea
of
like
forward,
transfer
or
accumulating
knowledge
that
can
be.
A
You
know,
reused
for
faster
learning
later,
so
I
think
we
need
to
study
a
continual
rl
somewhere
in
between
those
things,
and
this
is
where
metalworld
comes
in
or
it's
continual
version
which
they
call
continual
world.
So
basically
in
meta
world
it
meta
world
is
a
suite
of
50
manipulation
tasks
and
even
though
the
tasks
are
different,
they
all
share
a
common
reward
structure.
Okay,
so
all
tasks
are
combinations
of
pushing
grasping,
placing
reaching
components
combined
with
the
objects
of
different
sizes
and
connectivity.
A
So
some
skill
that
is
learned
in
task
3
can
somehow
be
reused
in
task
5,
you
know,
and
and
so
on,
and
so
on.
So
for
me,
this
is
where
it
makes
the
most
sense
to
study
continual
rl
and
all
of
the
experiments
will
be
done
in
metal
world
and
the
disadvantage
of
metal
world
is
it's
super
hard
and
it's
super
long
to
run
but
yeah.
Now
the
experiments
are
done
so
so
we're
good.
A
A
Yeah
yeah,
so
that
that's,
why
that
will
we'll
try
to
get
to
the
bottom
of
it,
but
I'm
glad
that
you're
surprised
but
yeah.
This
is
really
surprising
because
you
would
think
that
the
rnn,
the
best
thing
it
can
do
is
infer
the
task
which
would
bring
it
to
the
level
of
the
task
aware
baseline.
But
it
will
be
that
maybe
the
rnn
is
doing
something
more.
A
B
A
So
here
in
the
dotted
lines
you
see
the
analog
versions
of
the
baseline
in
the
multitask
regime
and
now
for
me
that
was
super
surprising
because
that's
like
that's
the
first
thing
you
hear
in
continual
learning:
it's
like
you
get
into
considerable
learning,
because
you're
excited
about
forward,
transfer
and
blah
blah
blah.
But
then
everyone
tells
you
oh
no
focus
on
catastrophic,
forgetting
it's
the
problem.
So
I
was
kind
of
super
surprised
when
I
saw
this
it.
B
Is
a
little
different
in
the
reinforcement
learning
scenario
yeah
in
the
in
the
supervised
learning
like
every
sample
can
be
from
a
different
task,
and
so
in
the
you
know,
in
the
in
the
non-continued
learning
scenario,
every
sample
can
be
from
a
different
task,
and
so
it's
seeing
everything
in
the
continual
learning
scenario.
You
still
have
to
stay
within
a
task
for
a
little
while
and-
and
you
know,
there's
still
potentially
some
task
interference
going
on.
A
A
A
So
what
we
did
is
we
trained
an
agent
with
and
without
an
rnn
in
the
single
task
regime?
So
basically
before
we
were
doing
10
or
20
tasks
now
we're
continually
now
we're
doing
them
independently,
and
it
turns
out
that
adding
another
end
was
not
improving
the
performance.
It
was
even
hurting.
So
we're
kind
of
invalidating
the
hypothesis
that
the
rnn
is
just
better
at
rl.
Okay,
something
else
is
happening.
A
A
So
a
lot
of
a
successful
continual
learning
strategy.
They
will
increase
the
parameter
stability
either
in
a
soft
way.
So
maybe,
like
ewc,
keeps
you
from
moving
the
parameters
too
much
or
you
can
also
also
do
this
in
the
hard
way.
Like
maybe
pacnet,
that's
going
to
completely
freeze
parts
of
the
net,
so
yeah
you
can
successful.
Continual
learning
methods
increase,
parameter
stability,
so
maybe
the
rnn
somehow
increases
the
stability
of
the
thermometer.
A
So
we
tested
this
hypothesis
and
basically,
what
we
did
is
we
reported
how
much
the
parameters
are
moving
through
an
epoch
of
learning
and
somehow
all
of
the
methods
as
training,
so
there's
a
dip
at
first
in
the
movement
of
the
parameters
and
then
somehow
it
just
constantly
increases
throughout
the
training,
and
that's
even
true
for
the
multitask
method
in
in
a
dotted
lines
here
so
yeah.
This
invalidates
the
the
hypothesis
that
the
rna
increases
parameter,
study.
B
A
Yeah,
so,
okay,
so
if
I
didn't
want
to,
I
wanted
to
take
the
effect
of
adam
out
of
this
analysis.
So
basically,
what
I'm
doing
is
I'm
accumulating
the
I'm
looking
at
all
the
gradients,
I'm
getting
through
an
epoch
and
then
I'm
computing.
How
much
each
parameter
would
have
moved
given
those
gradients
and
then
once
I
have
the
distribution
of
how
each
parameters
would
have
moved.
I
compute
the
entropy
of
that
vector
basically
so.
B
B
A
B
A
A
So
this
leads
us
to
the
last
hypothesis
yeah,
but
this
is
I
like
the
most,
which
is
that
the
rnn
correctly
places
the
new
task
in
the
context
of
the
previous
task.
Okay,
so
I
think
the
simplest
way
to
explain
this
is
with
a
simple
example.
Let's
say
so.
You
know
in
this
paper
we're
doing
manipulation,
tasks,
so
the
task
one
could
be
reaching
a
doorknob,
whereas
task
two
could
be
open
a
door
okay.
A
So
if
the
rnn
understands
well,
if
the
rnn
learns
a
proper
task
representation
or
a
proper
representation
of
each
time
set
in
its
world,
maybe
while
it's
learning
the
second
task
through
the
trajectories
of
observations
and
rewards,
it
will
understand
that
it's
in
the
in
the
context.
So
if
you
want
to
open
a
door,
you
need
to
first
reach
the
door,
which
is
fact
zero.
A
B
A
B
A
I
guess
it's
just
yeah,
maybe
I
should
be
more
clear
about
that,
but
it's
kind
of
simple
the
rnn
takes
in
the
previous
trajectory
up
to
time
tips
t
for
example,
and
then
it's
going
to
encode
it
encode
it
in
zt.
So
that's
like
its
output,
the
output
of
the
rnm
and
then
we're
just
going
to
feed
this
to
the
classic
mlp,
the
classic.
A
A
This
is
the
simple
baseline
great,
so
maybe
the
rnn
recontextualizes
the
new
task
in
the
in
the
previous
ones
correctly.
So
it's
hard
to
test
for
that
hypothesis
directly,
but
maybe
there's
some
things.
We
can
there's
some
effects
of
that
hypothesis
that
we
could
observe
if
it's
true,
okay,
so
one
would
be
that
if
this,
if
this
is
true,
the
rnn
could
potentially
learn
faster
than
the
other
methods.
A
So
the
first
thing
we're
going
to
do
is
like
it's
going
to
so
we're
going
to
look
at
the
current
performance
of
the
method.
Okay,
before
were
looking,
I
didn't
even
say
so,
but
we
were
looking
at
the
global
performance
of
the
methods
on
all
tasks,
but
here
we're
looking
at
their
current
performance.
So
this
is
why
we're
seeing
these
spikes
and
then,
if
you
zoom
in
on
so
here,
we
have
like
the
metals
that
are
doing
continual
learning.
These
ones
will
struggle
on
a
lot
of
tasks.
A
So
I
don't
know
it's
kind
of
not
super
clear,
but
they
will
only
learn
task,
0,
whoops
task,
0,
2,
5
and
then
8.,
and
then
we
have
like
these
independent
methods.
Those
are
like
always
trained
from
scratch,
and
these
are
a
bit.
Sometimes
they
can
learn
a
bit
more,
but
anyway,
that's
not
super
important.
What
I
want
to
zoom
in
is
the
top
part
of
the
performance
graph,
and
here
you
can
see
that
3rl
not
only
learns
faster
than
all
of
the
previous,
all
of
the
other
continual
reading
methods.
A
It
also
learns
faster
than
the
model
trains
from
scratch,
so
there
is
definitely
some
weight
to
the
hypothesis
that
the
rnn
is
kind
of
reusing
the
past
to
learn
faster
here.
I
think,
okay,
so
not
sure
if
that
was
super
clear,
but
that's
kind
of
like
the
first
observation
we
can
use
if
we
want
to
put
some
weight
on
this
hypothesis.
A
B
A
Yeah,
so
my
last
slide
is
actually
the
related
representations
of
the
task.
It's
just.
It's
really
really
hard
to
quantify.
Yeah
meta
world
is
awesome,
but
it's
it's
so
convoluted.
The
reward
functions
that
it's
hard
to
quantify.
All
tasks
relate
to
each
other.
Like
you
know,
they
all
share
a
comment
structure
but
yeah
pinpointing.
It
is
hard
okay.
So
that's
observation,
number
one.
A
The
second
one
is
okay,
let's
go.
Let's
have
a
new
running
example.
Now,
let's
say
task:
zero
is
for
task.
One
is
to
push
pencil
and
task.
Two
is
to
pull
a
pencil
okay.
So
for
both
tasks
reaching
the
pencil.
This
is
the
we're
gonna,
it's
the
same
policy,
but
when
you
get
to
the
pencil
now,
one
task
is
is
going
to
tell
you
to
go
forward.
The
other
is
going
to
tell
you
to
go
backwards,
so
you're
going
to
get
conflicting
gradients
right,
as
you
mentioned
earlier
there.
A
A
So
if
your
task
aware
so,
if
you
know
that
you
need
to
push
something
or
pull
something,
then
it's
fine,
you
shouldn't
get
much
grading
conflict,
but
if
the
rnn
correctly
places
the
new
tax,
the
new
task
in
the
context
of
previous
one,
then
we
can
expect.
We
might
expect
that
the
gradient
conflict
is
reduced.
A
Okay,
so
my
the
next
observation,
what
we're
doing
is
we're
running
we're
looking
at
the
variance
in
the
mini
batch
as
a
proxy
for
gradient
conflict,
a
lot
of
previous
work
instead
of
looking
at
the
variance
of
the
gradient,
they
look
at
the
the
angle
between
the
gradient.
I
think
that's
a,
I
think,
that's
not
as
good
as
looking
at
the
the
variance
of
the
gradient.
I
have
an
argument
for
that.
A
A
So
now
so
here
we're
looking
at
the
representation
that
the
rnn
is
learning
throughout
its
life.
Okay,
so
basically
I'm
focusing
on
four
tasks
and
what
you
can
see
at
the
bottom
is
like
pca
representation
of
an
episode
in
each
task.
Okay,
so
there's
a
couple
of
things
I
want
to
point
out
here,
so
this
is
before
training.
This
is
after
one
task,
and
this
is
at
the
end.
Okay,
so
before
training,
the
initialization
is
the
kind
of
for
each
task.
A
A
D
D
A
A
You
want
to
learn
an
invariant
initial
representation
and
two
now
the
representation
they're
like
constantly
evolving
throughout
the
episode-
and
you
know,
they're
also
intersecting
in
interesting
ways
like
at
different
stages
of
the
episodes
of
each
task.
So
you
know
it's
only
qualitative,
but
this
is
kind
of
aligned
with
the
hypothesis
that
you
know
the
the
rnn
is
doing
more
than
just
task
inference.
It's
decomposing
each
task
and
it's
kind
of
overlapping
them
when
necessary,
yeah.
So
that's
that's
for
the
qualitative
analysis
and
yeah
mostly
done.
A
A
That
was
my
main
motivation,
so
for
me,
for
me
at
least
it's
unclear
if
I
should
be
working
on
catastrophic,
forgetting
or
task
inference
anymore.
Maybe
it's
now
time
to
focus
on
forward
transfer
if
this
use
case
is
in
your
motivation
and
yeah
like
it's
also
interesting
to
discuss
like
why
is
this
not
happening
in
continual
supervised
learning
like,
what's
fundamentally
different
in
continual
reinforcement,
learning
versus
continual
supervised
learning?
A
One
thing
I
can
see
that
the
most
obvious
is
that
in
continual
reinforcement,
learning
or
at
least
the
way
we
study
it,
the
action
space
or
the
prediction
space
of
the
agent-
it's
always
fixed.
It's
not
like
art,
it's
not
increasing
at
each
task,
contrarily
to
continual
supervised
learning,
or
you
know
incremental
classification
where
at
each
task,
I'm
just
increasing
my
output
space
and
only
operating
in
these
new
actions
or
new
predictions.
A
Now,
why
am
I
saying
this?
Well,
if
you
keep
increasing
your
your
action
space
with,
for
example,
new
classes,
then-
and
if
you
also
apply
a
soft
max,
then
if
you're
not
doing
replay
you
or
if
you,
if
you're,
not
smart
about
it,
your
soft
max
will
be
badly
calibrated
and
you
what
we
call
forgetting
might
just
be
or
a
part
of
it
could
be
explained
by
a
badly
calibrated
output
head
because
of
this
increasing
in
increasing
output
space
with
the
softmax.
A
So
that
could
be
one
difference
between
continual
reinforcement,
learning
and
continual
supervised
learning,
and
also,
I
think,
there's
this
idea
of
like
in
continual
reinforcement,
learning
the
ortiz
in
the
scenarios
we
studied
today.
You
know
that
the
task
we
kind
of
know
how
they
relate
to
each
other.
So
we
kind
of
know
that
some
power
transfer
is
achievable.
B
B
A
Exactly
yeah
yeah
yeah
yeah.
I
think
it
actually
would
make
more
sense
to
study
like
if
we
want
to
stay
in
continuous,
supervised
learning.
What
you
refer
to,
I
think
it's
called
domain
incremental
learning
exactly
it
makes
more
sense
and
you
can
actually
understand
better
with
like
now
you're
forgetting
is
the
forgetting
in
the
representation,
and,
I
think
that's
what's
important.
Yeah.
B
A
B
D
C
B
A
A
A
B
A
B
So
you
talked
about
what
is
a
good
benchmark
and
I
thought
you
know
it
seemed
like
you're
really
proposing
a
benchmark,
but
then
this
rnn
method,
your
baseline,
does
really
well
and
I
think
another
property
of
a
good
benchmark
is
that
there's
still
room
to
improve
and
other
people
can
take
that
benchmark
and
try
to
beat
the
baselines
and
improve.
So
do
you
think
it's
your
benchmark
is
solved
or
is
there
significant
room
for
improvement
still
over
over
the
rnn.
A
B
A
B
A
A
Yeah
exactly
so
so,
even
in
multitask
learning
like
oh,
I
don't
have
the
reference.
I
think
it's
from
the
previous
slide.
Okay,
so
if
you
train
all
of
the
method
independently
with
one
million
step
per
task,
which
is
like
the
same
computer
allowed
for
the
in
continual
learning
and
with
the
learning
you'll
reach
this
performance,
so,
okay,
okay,
there's
still
a
lot
of
room
just
just
to
just
to
get
to
the
model.
That
has
absolutely
no
transfer
between
tasks,
because
it's
just
learning
one
model
per
task
independently.
A
Yeah,
so
definitely
more
compute
would
help
so
like
like
the
independent
model.
Let's
take
the
purple
guy
like
here
it's
starting
to
learn
here,
it's
starting
to
learn,
but
it
it
doesn't
have
enough
compute
to
learn
all
tasks,
but
so
to
give
you
an
example
like
now,
I'm
running
10
million
time
steps,
but
the
meta
world
guys
to
to
learn
all
tasks:
they're
like
running
100
or
200,
or
500
million
time
steps
okay.
So
it's
like
we're
operating
in
a
really
really
data
and
compute
constrained
regime
compared
to
them.
A
So
the
tasks
are
learnable,
but
there's
a
lot
room,
there's
a
lot
of
improvement
on
the
data
efficiency
side
and
then
what
I
also
want
to
point
out
is
like
there,
it
you're
losing
a
lot
like
sometimes
the
initialization
you're.
Getting
from
these
continual
learning
models
is
actually
hurting
you
in
learning
the
new
task,
so
that's
kind
of
like
something.
I
think
we
should
try
fixing
right
it
really
sucks
you're,
not
you're.
A
C
A
D
A
Pairwise
distances:
well,
I
I've
looked
at
a
three-dimensional
representation,
but
the
I
think
the
first
two
observations
were
explaining
like
around
70
of
the
variants.
So
you
know
I
think,
most
of
what's
happening.
What
yeah
most
of
the
interesting
stuff
can
be
seen
in
those
two
first
dimensions
but
yeah,
I
guess
with
the
pairwise
distances
you
could
understand,
really
what's
happening
under
the
hood.
It's
just
that
without
a
notion
of
task
similarity.
A
A
C
Yeah
one
thing
that
was
kind
of
interesting
to
me
about
your
method
is
that
there
are
other
methods
that
try
and
explicitly
facilitate
forward,
transfer
or
even
backward
transfer
from
current
tasks
back
to
previous
ones.
By
mixing
and
matching
the
outputs
of
different
classification
heads
or
things
like
that,
and
your
method
is
kind
of
interesting
in
that
it
it's
sort
of
less
explicit
in
how
it
does
that.
A
B
C
A
Well,
that's
a
very
open-ended
question
yeah,
but
I'm
curious
it's
a
good
question,
so
yeah.
So
maybe
I
can
tell
you
a
bit
about
the
method
that
I
was
trying
to
develop
and
that
somehow
it
was
never
beating
the
rnn,
but
I
think
I
know
why.
But
essentially
what
I
was
trying
to
do
is
learn
these
masks
on
top
of
the
weights,
so
per
task
mask
to
do
multitasking.
A
So
this
is
kind
of
like
a
trick
that
is
really
used
right
now
in
continual
learning
literature.
But
then
I
wanted
in
the
task
agnostic
setting.
I
wanted
the
rnn
to
output
the
mask
that
I
was
applying
on
the
actor
and
critic
okay,
so
I
was
hoping
that
the
rnn
would
infer
the
task
and
somehow
segment
into
different
parts,
the
actor
and
critic
so
that
they
could
perform
multitask
learning.
But
somehow
this
was
never
better
than
just
speeding
the
rnn
representation
to
the
actor
and
critic.
A
C
C
A
Yeah,
so
it's
actually
created
for
that.
So
if
you
go
to
the
continual
wallpaper
like
it's
mostly
about
forward
transfer
and
they
have
a
reference
reference
point
for
like
how
much
forward
transfer
you
should
expect-
and
in
my
case,
because
we
were
tack,
agnostic
and
always
doing
replay,
we
were
not
getting
as
much
power
transfer
as
you
could
get
with
their
like
well
calibrated
pacnet,
which
is
like
the
method
that
works
the
best,
at
least
for
them
in
the
task
aware
setting
but
yeah.
D
C
A
Yeah,
so
if
by
backward
transfer,
you
mean
like,
if
negative
transfer,
if
negative
backward
transfer
is
forgetting
yeah
trial,
the
baseline
was
getting
was
able
to
remember
everything
when
the
scene
was
successful
because,
like
a
lot
of
variance
in
this
benchmark
comes
from
like
something
going
wrong
at
some
point
and
then
your
performance
on
all
the
tasks
might
go
down
but
yeah.
When
everything
was
going
correctly,
the
rnn
was
able
to
remember
everything
and
which
was
not
the
case
for
all
baselines.