►
Description
In this meeting Subutai discusses three recent papers and models (OML, ANML, and Supermasks) on continuous learning. The models exploit sparsity, gating, and sparse sub-networks to achieve impressive results on some standard benchmarks. We discuss some of the relationships to HTM theory and neuroscience.
Papers discussed:
1. Meta-Learning Representations for Continual Learning (http://arxiv.org/abs/1905.12588)
2. Learning to Continually Learn (http://arxiv.org/abs/2002.09571)
3. Supermasks in Superposition (http://arxiv.org/abs/2006.14769)
C
B
So
what
I
thought
I'd
talk
about
today
is
these
three
topics
is
sort
of
the
focus
is
on
continuous
learning,
but
and
we'll
look
at
kind
of
several
papers,
so
we're
not
going
to
go
through
each
one
in
any
detail,
but
there's
some
underlying
concepts
in
each
one
which
are
which
are
quite
interesting
and
relevant.
So
I'll
sort
of
spend
a
couple
of
slides
on
each
one,
but
the
papers
would
be
sort
of
mammal,
which
we
already
talked
about.
B
B
So
we'll
talk
about
a
bunch
of
those.
So
before
we
do
that,
I
thought
I'd
just
sort
of
review
a
little
bit
about
continuous
learning
in
htms
and
temporal
memory.
I'm
not
going
to
go
into
this
in
detail,
but
the
way
we
often
talk
about
how
inference
works
in
htns
and
how
continuous
learning
works
is
using
a
particular
type
of
language.
B
These
then
cause
predictions
and
specific
cells
and
other
mini
columns
and
as
kind
of
shown
on
the
right
finger
here.
The
way
this
prediction
happens
is
that
you
have
connections
from
some
of
these
cells
in
these
active
mini
columns
onto
one
dendrite
on
on
this
red
cell
and
another
dendrite
on
this
red
cell
and
so
on,
and
so
when
this
red
cell
detects
a
pattern
on
one
of
its
dendrites,
it
becomes
depolarized
or
predicted
state.
That's
what
showed
in
these
kind
of
red
dots,
and
so
this.
B
If
these
correspond
to
the
b
mini
columns,
then
we
say
kind
of
b
is
predicted
and
then,
when
b
actually
comes
in
this
cell
because
it
was
depolarized
will
win
and
the
the
rest
of
the
cells
in
this
mini
column
won't
become
active.
Let
me
just
turn
on
the
here:
okay
and
if
there
was
a
mistake,
if
if
there
wasn't
a
correct
prediction
here,
so
if
these
cells
were
not
depolarized,
then
all
of
these
cells
would
become
active.
B
We'd
pick
a
winter
cell
and
then
we'd
learn
the
previous
pattern
on
a
specific
dendrite
on
those
cells.
Okay,
so
these
dendrites
bias
these
cells
to
win,
and
if
there's
no
winner,
then
we
pick
a
random
one
and
learn
patterns
on
that.
So
this
is
kind
of
the
language
we
typically
use
to
describe
it.
I'm
gonna,
say
short
slightly
different
way
of
showing
the
exact
same
thing.
So
one
thing
to
note
is
that,
after
learning,
in
reality,
this
network
is
very
densely
recurrently
connected
and
we
never
really
show
this
in
our
pictures.
B
I
realized,
but
there's
a
you
know,
once
we've
learned
a
whole
bunch
of
sequences,
there's
tons
of
connections
going
back
and
forth
between
these
cells.
So
it's
a
very
densely
connected
recurrent
network
right,
what's
kind
of
going
on
is
that
when
you're,
showing
a
particular
task
or
a
sequence,
you're
instantiating,
a
very,
very
sparse
sub
network,
that's
embedded
in
this
densely
recurrent
network
right,
that's
the
connected
recurrent
network!
So
again
this
we
don't
typically
talk
about
it.
B
This
way,
but
what's
happening,
is
that
these
dendrites
are
actually
choosing
which
sub
network
to
instantiate
at
any
given
point
right.
So
if
you
think
about
this
being
a
full
really
densely
connected
network
for
a
particular
sequence,
you
you
know
you
instantiate
a
particular
subset
which
is
like
this
guy
this
guy,
this
guy
connected
to
a
previous
set
of
cells.
There
that's
for
this
one
transition
in
this
one
sequence
and
the
specific
dendrites
that
become
active
by
biasing,
the
cell
and
determining
who
wins:
they're
actually
instantiating,
this
particular
sub-network.
B
Okay.
So
that's
in
some
ways
what
a
dendrite
is
doing.
It's
it's
choosing
a
sub-network.
Of
course
we
have
very
sparse
distributed
representations
so
that
avoids
significant
overlap
between
the
tasks.
So,
even
if
a
few
of
these
cells
might
overlap
with
other
other
tasks
or
a
few
of
these,
you
know
portions
of
the
sub
network,
the
network
subnetwork
as
a
whole,
represented
by
an
sdr,
because
it's
super
sparse.
B
So
this
is
kind
of
background.
I'm
just
saying
the
exact
same.
This
is
no
change
in
temporal
memory.
I'm
just
explaining
it
in
a
slightly
different
language
and
you'll
see
the
point
of
this.
As
I
describe
some
of
these
papers,
any
questions
on
that
is
that
was
that
is
this
already
all
obvious
to
people,
or
is
this
anything
new.
D
B
It's
it's
in
our
algorithm:
it's
it's
pretty
random
in
in
in
the
neocortex.
It
might
not
be
truly
random.
There
might
be
still
a
little
bit
of
each
cell
is
not
going
to
be
exactly
have
the
same
amount
of
depolarization.
There
will
be
some
randomness
in
there.
You
know
something
we'll
choose
it,
but
in
our
algorithm
it's
it's
completely
random
and
it's
actually
good
that
it's
random,
that
that
gives
you
kind
of
the
big,
the
full.
You
can
use
the
full
representational
space
that
way.
B
And
what
I'll
do
is
I'll
go
through
it
in
this
sequence?
So
I'll
first
talk
about
this
one,
which
is
the
oml
paper
and
then
I'll
do
the
animal
paper
which
is
down
here
then,
as
a
third
one,
I'll
talk
about
the
super
masks
and
superposition
mammal
I'll.
Just
briefly
show
my
drawings
from
when
we
talked
about
a
couple
of
weeks
ago,
but
mammal
is
used
in
oml
and
animals,
so
it
might
be
useful
just
to
kind
of
remember
what
what
it
is,
but
the
core
is
kind
of
these
three
papers.
B
Okay
in
in
the
original
mammal
paper,
they
described
good
as
being
how
quickly
you
can
learn
each
task.
But
the
idea
is
that,
through
this
meta
learning
process,
you're
gonna
find
some
optimal
set
of
weights,
which
is
basically
like
an
initialization
for
the
network
and
from
that
set
of
weights.
It's
going
to
be
primed
to
be
really
good
at
this.
B
This
big
task
that
you've
defined
and
the
basic
you
know
terminology
and
loop
was
there's
a
network
with
weights
theta
and
that's
the
kind
of
network
you're
trying
to
optimize
you
initialize
with
some
random
values
for
theta,
and
then
you
go
through
this
loop,
where
you
pick
a
task
out
of
your
collection
of
of
tasks
and
then
for
each
task.
You
perform
the
task.
B
In
her
case,
it
was
training
on
on
k,
samples
and
then
you
what
you
do
is
for
each
task:
you're
going
to
get
a
different
weight,
a
different
network,
which
is
this
theta,
updated
by
the
the
gradient
steps
for
that
task,
and
then
what
you
do.
Is
you
update
this
original
theta
to
minimize
the
loss
on
each
of
these
individual
networks?
Here,
I'm
using
a
test
set?
B
They
test
this
on
a
completely
new
set
of
tasks
that
have
not
been
seen
during
training
as
well,
but
they're
all
kind
of
part
of
the
same
thing.
Okay,
so
you
can
imagine
you
can
so
what
the
oml
paper
does
is
apply
this
basic
framework
to
continuous
learning,
instead
of
looking
at
how
fast
it
learns
they're
going
to
look
at
how
well
it
remembers
previous
categories.
B
The
output
of
that
is
then
fed
to
another
network
called
the
prediction
learning
network,
and
what
happens
is
that
during
the
meta
learning
phase
you
up
this?
This
is
the
theta
you're
going
to
update
you're,
going
to
learn
the
weights
for
this
network
for
the
representation
learning
network
and
during
the
continuous
learning
portion
you're
just
going
to
adapt
the
weights
on
the
prediction
learning
network
to
see
how
well
it
for
each
successive
task?
B
B
So
here
they
look
at
mainly
omni
glot,
that's
their
their
main
result,
which
is
this
data
set.
I
forget
how
many
I
think
it's
like
1600
different
categories
or
something
like
that.
So
they
choose
randomly
choose,
so
they
first
have
a
subset
of
all
the
categories
as
their
training
subset
and
then
there's
another
subset,
which
is
the
test
subset,
which
is
not
seen
during
training
at
all.
So
what
happens
during
training?
Is
they
choose
randomly
some
list
of
200
of
those
training
classes?
Okay,
so
these
are
200
omniglad
classes.
B
B
Okay,
so
this
inner
loop
is
that
the
standard,
continuous
learning
thing
that
that
that's
big,
where
you
have
a
set
of
tasks-
and
you
learn
each
task
in
a
row
and
then
you
want
to
see
how
well
you've
learned
all
of
the
classes
at
the
end
of
the
day,
and
then
that
becomes
the
inner
loop
for
this
whole
meta
learning
set
in
reality.
If
they
were
to
actually
do
this,
this
would
be
a
thousand
steps
and
you'd
have
to
accumulate
these
gradients
over
a
thousand
steps.
B
So
what
they
do
is
they
only
update
five
of
these
at
a
time?
So
they
pick.
You
know
some
random
point
in
the
sequence
train,
I'm
sorry
they
did
five
classes
at
a
time.
So
they
pick
a
random
point.
Within
this
list
of
200
classes,
they
train
for
the
next
five
classes.
Then
they
update
the
weights
and
they
they
iterate.
So
they
don't
do
a
full
thousand
in
here.
B
B
B
B
And
the
one
big
point
they
make
in
this
in
this
paper
is
that
by
doing
this,
the
representation
that
they
learn
is
super
sparse,
so
that
that's
kind
of
nice
that,
if
you
look
at
you,
know
any
particular
output
at
any
particular
point
in
the
scheme
after
the
whole
meta
learning
phases
is
learned.
If
you
look
at
the
output
of
the
representation
network,
it's
extremely
sparse
and
the
other
thing
they
mention
is.
B
B
E
A
B
B
It
could
have
just
learned
weights
such
that
they're
still
using
relative
they're,
not
using
k,
win
or
take
all
or
anything
like
that
right,
so
they
might
have
learned
weights
such
that
the
average
dot
product
is
very
close
to
zero,
and
you
know
for
specific
types
of
inputs.
Only
some
of
them
actually
get
above
zero
and
some
don't
it's
not
clear
to
me
exactly
what
what
they're
not
explicit
like.
I
said
it's
not
explicitly
designed
to
be
a
sparse
network,
but
the
meta
learning
process
learned
that
sparsity
is
best
for
this
continuous
learning.
C
C
B
C
B
And
so
this
modulatory
network
is
sort
of
gating
the
outputs
to
the
classifier,
creating
the
activity
that
eventually
gets
to
the
classifier.
So
the
way
I
kind
of
think
of
it
is
this
network
is
kind
of
learning
which
activations
are
important
for
a
given
input.
It's
trying
to
pick
some
characteristics
of
this
input
and
saying,
oh
based
on
this
characteristics.
B
Specific
output
units
are
going
to
be
more
important
than
others,
okay,
so
it's
it's
kind
of
like
an
attention
system.
It's
it's
gating
the
outputs
here,
so
so
during
meta
learning,
they
learn
all
of
these
weights.
The
weird
thing
is
that
during
continuous
learning,
only
these
only
the
classifier
weights
are
updated.
B
B
I
was
not
very
satisfied
with
this.
It's
like
I
thought,
is
this
really
continuous
learning
in
that
case,
is
it
really
well?
It
is
continuous
learning,
but
is
it
really
solving
the
catastrophic
forgetting
problem,
because
during
continuous
learning,
all
of
these
weights
are
fixed,
so
I
was
kind
of
left
slightly
unsatisfied
with
that.
I
would
have
been
happier
if
at
least
some
part
of
this
basic
network
was
also
updated.
B
So
so
what
do
you
think
is
happening
in
the
last
layer?
It's
just
memorizing
everything
yeah.
So
you
know
when
it's
learning
class
k,
only
the
head
for
class
k
is
being
updated.
Maybe
I
don't
remember
if
they're
learning
a
couple
of
categories
at
a
time
or
they're
just
learning
one
category
at
a
time-
and
I
think
it's
just
essentially
like
like
you're
saying
memorizing,
which
which
activations
it
should
it
should
attach
to
that
head
right.
B
B
So
it
was,
I
thought
this
was
a
little
odd
and
not
fully
satisf,
satisfying
as
a
continuous
learning
system
or
solving
the
catastrophic
problem,
forgetting
problem
they
get,
but
they
get
really
good
results.
So
here's
they
go
up
to
600
categories
in
a
row
instead
of
200,
and
you
can
see
that
the
animal
during
testing
time
it
retains
something
like
63
accuracy
on
all
600
classes
that
were
that
it
was
trained
on.
B
So
just
as
a
reminder
again
during
the
meta
training,
it's
trained
on,
you
know
hundreds
of
classes
where
each
time
it
does
continuous
learning
on
some
subset
of
these
classes.
Once
meta-learning
is
finished,
they
don't
change
those
weights
at
all.
They
get
a
completely
new
set
of
classes,
and
now
you
learn
600
of
them
in
a
row.
Okay,
so
it
says
never
seen
those
classes
during
training.
B
Okay,
so
it
is,
you
know
it's
a
completely
new
set
of
classes,
but
and
it's
able
to
learn
600
of
them
in
a
row
where
learning
involves
just
learning
the
output
weights
so
one.
So
this
is
quite
impressive
that
they
were
able
to
learn.
600
classes,
one
kind
of
really
interesting
twist
to
all
this
that
I
just
learned
recently,
I
was
in
touch
with
kuram
javed
who
did
the
oml,
so
they
he
just
found
a
pie,
torch
bug
in
the
scripts.
B
As
we
know,
writing
these
continuous
learning
things
in
pi
torch
is
really
tricky.
I
think
he
didn't
accumulate
the
gradients
quite
correctly,
and
so
he
just
found
a
budget
bug
in
the
scripts
and
after
he
fixed
the
bug.
If
he
puts
a
single
output
layer,
sort
of
mimics,
the
oml
and
the
animal
setup,
he
gets
the
same
results
as
animal.
B
B
B
At
the
outputs,
after
the
gating
here
before
and
after
the
gating
and
what
they
see
is
that,
after
the
the
gating,
the
outputs
are
super
sparse
again.
So
in
their
case
also,
the
classifier
is
seeing
extremely
sparse
activations
coming
in.
So
this
is
like
the
output
of
the
of
the
basic
network.
This
is
like
the
output
of
the
gating
signal
and
wherever
there's
a
coincidence
between
those
two,
that's
presumably
where
there's
a
activation
coming
in.
F
F
B
F
F
B
B
This
one
right,
yeah
line
four
yeah,
yeah
yeah,
exactly
yeah,
so
so
the
training
on
this
new
trajectory,
as
well
as
some
subset
of
the
previous
ones,
yeah
yeah.
So
that's
that's
a
little
weird!
That's
interesting!
You
know
that
that
means
you
know
they
must
have
put
that
in
because
they
needed
it
right
exactly.
F
B
B
C
C
B
B
F
F
B
F
B
B
B
B
Yeah,
it's
a
so
in
what
you're
saying
is
in
computing.
The
meta
learning
loss
function,
they're
incorporating
a
few
random
examples
from
so
they
might
learn
on
sort
of
five
classes
in
sequence,
but
they're
also
including
some
so
of
course,
they're
going
to
test
on
those.
But
I'm
sorry
they're,
going
to
compute
the
meta
loss
on
those
but
they're,
also
using
some
other
previously
learned
classes
from
the
full
set
of
200
or
600.
B
B
C
F
F
The
convolutional
layers
are
also
frozen
there
during
the
inner
loop
and
they're
updated
in
the
outer
loop
and
what
they
did
in
animal
in
order
to
keep
the
network
the
same
size,
they
had
the
four
convolution
and
two
fc
and
they
broke
into
two
networks
so
that
the
the
nm
network
has
two
calm
and
one
fully
connected,
and
the
prediction
network
has
two
convent
fully
connected.
So
what
they
do
is
they
freeze
the
comp
and
only
train
the
fully
connected?
But
that's
the
same
thing
they
do
in
oman.
F
B
B
Yes,
it's
a
it's
a
it's
a
it's
a
linear,
connected
network,
but
this
all
you
know,
you're
just
training,
a
subset
of
the
classifier
here
you
here
you're
doing
multi-head
trading
right
right,
and
so
your
pic,
you
know
what
task
it
is.
You're
only
updating
those
weights
and
the
hard
work
is
all
done
during
meta
learning.
B
B
B
F
B
B
B
B
This
is
slightly
different.
There's
no
meta
learning
here.
In
fact,
the
learning
is
super
simple,
and
so
here
you
know,
you
know
you
have
a
standard
case
where
you're
outputting
a
probability
over
all
the
classes
given
a
set
of
weights
and
an
input.
So
this
is
your
the
output
of
your
neural
network
and
it's
trained
by
cross-entropy
loss.
So
this
is
just
the
standard
setup
and
in
continuous
learning
you
have
a
bunch
of
different
l-way
classification
tasks.
B
B
B
B
Okay.
So
a
super
mask
is
a
sparse
sub
network
with
essentially
binary
weights
that
are
never
changed.
Each
super
mask
is
specific
to
each
task
and
they
make
the
point
that
this
is
really
efficient
to
store.
You
don't
need
to
store
all
of
the
weights.
You
just
need
to
store
the
random
seat
that
generated
the
weights,
and
then
you
have
to
store
each
of
these
masks
which
are
binary
okay.
So
the
only
thing
now
is:
okay.
How
do
you
learn
these
super
mass
for
each
particular
task?
B
It's
got
a
weight,
but
it's
also
got
this
score
suv
and
then,
in
the
forward
pass.
They
only
look
at
the
top
k
percent,
the
weights
with
which
have
the
top
k
percent
of
the
scores,
so
they
pick
some
subset
of
it.
You
know
where
it's
determined
by
k.
What
that,
how
large
that
subset
is,
but
then
in
the
backward
pass
they
update
all
the
scores
with
a
straight
through
estimator,
so
they
you
know,
update
this.
They
update
the
scores,
only
not
the
weight
values,
so
they
update
the
score.
B
So,
during
the
backward
pass,
this
score,
which
was
not
in
the
top
k
percent,
could
suddenly
become
in
the
top
k
percent,
and
this
guy
could
now
drop.
So
you
can
actually
you
can
change
the
connectivity
dynamically
through
learning
and
to
me
this
is
very
analogous
to
what
marcus
has
showed
last
year
as
well
in
terms
of
learning
sparsity
through
the
variational
techniques.
B
I
don't
know
if
it's
exactly
equal
or
not
marcus,
maybe
you
can
you
can
open
on
that,
but
it's
at
least
conceptually
it's
very
similar,
so
you
can
think
of
this
score
as
almost
being
like
a
permanence,
except
here
he's
taking
the
top
k
percent
of
the
permanences.
Instead
of
setting
a
threshold
on
the
on
the.
C
A
B
E
B
You
need
all
the
they
need,
all
the
scores
in
the
forward
pass
during
training
after
training.
They
can
just
store
the
mask
for
each
once.
Training
is
completely
finished
and
now
you're
adjusting
inference
and
you
don't
care
about
the
training
anymore.
At
that
point,
they
can
just
store
the
mask
for
each
task.
B
B
So
I
think
this
is
like
uniformly
set
in
the
beginning,
and
then
they
run
a
couple
of
steps
of
gradient
descent
to
minimize
the
entropy
of
the
output,
so
they're
trying
to
find
basically
a
set
of
coefficients
so
that
the
output
is
very
certain.
So,
if
you
think
about
in
the
beginning,
the
output
may
be
maybe
like
this
on
the
left.
B
B
So
here
what
they're
doing
is,
if
they'd
run
a
couple
of
steps
of
this
gradient
and
the
network
is
still
uncertain,
then
let's
say
you're
training
on
a
new
example,
and
even
after
doing
this,
your
out
network
is
still
somewhat
uncertain.
You
don't
get
low
enough
entropy,
then
they
just
instantiate
a
new
mass
and
train
it
on
that
task.
B
B
B
B
D
B
D
E
B
B
Yeah,
I
think
their
upper
bound
is
probably,
if
you
train,
normally
on
all
the
classes
in
a
typical
setting.
What
is
the
maximum
accuracy
to
get
so
that
would
be
the
best
that
this
particular
network
could
do.
Assuming
you
know
full-on
training,
no
continuous
learning,
I'm
assuming
that's
what
that
means.
G
G
G
B
C
F
B
B
B
C
B
B
I
like
this
chart
here.
What
they
show
is
that,
as
you
make
the
networks
wider
and
wider
and
wider,
by
doing
these
sparse
subsets,
you
can
get
closer
and
closer
to
the
accuracy
of
the
dense
version
of
the
model,
and
so
this
is
again
a
case
where
kind
of
dimensionality
matters,
and
here
also
the
way
they're
doing
it.
You
have
many
combinations
that
are
possible
with
a
wider
network.
B
C
F
B
C
F
B
But
I
I
thought
I'd
just
close
with
the
same
thing
again,
I
thought
there
were
a
lot
of
connections
to
atm
temporal
memory.
You
know
here
as
you're,
seeing
with
the
super
mass
one
as
well
as
the
the
neuromodulatory
neuromodulatory
network.
Is
the
instantiate
very
sparse?
Well,
they
instantiate
sub
network
specific
to
each
task.
In
the
case
of
the
supermass,
it's
not
very
sparse,
but
in
the
case
of
animal
it's
at
least
the
output
is
that's
sent
to
the
classifier
is
very
sparse.
B
B
I
think
I
think
the
dendrites
option
could
be
much
more
flexible
and
much
more
powerful
here
and
if
we
can
get
to
extremely
sparse
sub
networks,
I
think
that
could
be
super
powerful
as
well
in.
In
almost
all
of
these
cases,
I
think
they
have
sparse
representations
to
avoid
significant
overlap,
and
in
the
supermax
case
it
looks
like
the
weights
are
close
to
binary.
It's
just
two
different
values:
they
don't
and
in
their
case
instead
of
permanences,
they
have
this
pop-up
score,
which
chooses
which
way
it's
to
make
activity
a
learning.
B
So
I
thought
these
were
kind
of
interesting
correlations
between
that.
I
think
they're
sort
of
very
much
in
the
same
spirit
as
what
was
in
htms,
but
hopefully
we
can
do
things
in
a
much
more
flexible
way.
Each
of
these
seems
still
a
very
kind
of
rigid
in
some
particular
way,
and
it's
not
fully
satisfying
yet
and
hopefully,
if
we,
if
we
can
incorporate
a
lot
of
these
principles,
we
can
get
to
something:
that's
really
nice
and
flexible
and
and
powerful.
B
B
C
G
B
B
Yeah
I
feel
like
here:
if
it
was
that
low,
it
would
have
mentioned
it
yeah.
So
I'm
guessing
it's
just
how
much
they
change
every
well.
How
much
is
that
changed
every
iteration
is
is
based
on
the
back
props.
So
I
don't
know
if
they
have
a
budget.
So
I
I'm
confused.
I
don't
know
what
this
layer
wise
budget
is.
E
E
B
E
F
F
And
I
think
the
reason
quran
is
that
if
you
look
at
the
original
super
mask
paper,
they
started
they
started
with
random
weight
and
they
tried
to
make
it
simpler
and
simpler
and
simpler
to
see
how
far
they
could
get
by
just
learning
the
mask,
and
they
got
to
the
point
that
all
you
all
you
needed
is
supposed
to
set
a
constant
and
all
that
matter
if
they
were
positive
or
negative.
So
that
was
kind
of
the
end
of
the
super
market,
the
original
one.
F
E
So
in
supermass
I
think
they
cited
two
papers
for
the
way
they
actually
learned
the
supermass.
So
one
was
the
ramanujan
paper
that
subita,
I
think
you
also
talked
about
that.
That's
one
that
uses
a
straight
through
estimator,
right,
yeah,
yeah
and
then,
and
then
this
one
that
michelangelo
just
sent
is
this.
Is
this
the
other
method
of.
F
Yes,
so
so
just
to
give
an
overall
view.
So
the
idea
here
of
this
original
supermass
paper
was
following
the
lottery
ticket
paper
and
what
they
were
trying
to
show
here
is
that
you
didn't
even
have
to
reinitialize
the
weight
so
in
the
lottery
ticket,
they
learn
the
mask
and
they
reinitialize
their
weight
to
the
original
value,
and
then
they
retrain
from
there
and
what
they're
showing
here
is
that
you
don't
even
have
to
initialize
the
weights.
F
You
just
learn
the
masks
and
if
you
re-initialize,
all
the
weights
just
to
a
constant
value,
that's
the
same
sign
of
the
original
weight.
So
if
it
was
originally
positive,
it's
a
plus
one.
If
it's
originally
negative,
it's
a
minus
one,
then
you
get
the
same
results.
So
this
was
following
the
lottery
ticket.
So
that's
that's
at
the
end
here
this
mask
the
large
final
same
sign
mass
criteria
so
yeah,
just
if
you
go
through
the
blog
paper
later.
It's
it's
actually
very
interesting,
heidi
actually
presented
to
us
here.
F
B
I
think
the
one
another
nice
thing
with
this
particular
paper
is:
they
were
actually
able
to
get.
You
know,
competitive
or
better
than
you
know,
state-of-the-art
results
on
this
really
long,
continuous
learning
task,
whereas
I
think
some
of
the
other
ones,
the
results,
weren't
quite
state-of-the-art,
but
they
were
getting
close
to.
They
were
just
doing
better
than
well.
B
You
know
a
lot
better
than
chance
and
better
than
what
you
might
expect,
whereas
here
I
think
I
don't
think
there's
any
technique
so
far
that
comes
close
to
our
2500
categories
in
a
row
and
and
the
memory
usage
and
speed
is
really
fast.
It's
such
a
simple
technique,
so
they
make
a
point
of
explaining
how
that
they
can
do
this
really
fast
in
pytorch
and
on
gpus.