►
From YouTube: Viviane Clay on Sparse and Meaningful Representations Through Embodiment - October 19th, 2020
Description
We invited guest speaker Viviane Clay from the University of Osnabrück to talk about her research on learning sparse and meaningful representations through embodiment. In the first part, she explores how these types of representations of the world are learned in an embodied setting by training a deep reinforcement learning agent on a 3D navigation task with RGB images as main sensory inputs. She then discusses how the model learns sparse encoding of high dimensional visual inputs without explicitly enforcing sparsity, and what the possible hypothesis for this phenomena are.
In the second part, she covers her undergoing work on extracting concepts by identifying a minimal set of co-occurring activations that represents an object in a curiosity-driven learning setting. These concepts can be used to improve sample efficiency and performance in downstream tasks, such as object classification or the full reinforcement learning task.
- - - - -
Numenta is leading the new era of machine intelligence. Our deep experience in theoretical neuroscience research has led to tremendous discoveries on how the brain works. We have developed a cohesive theory, core software technology, and numerous software applications all based on principles of the neocortex. Our innovative work delivers breakthrough capabilities and demonstrates that a computing approach based on biological learning principles can do things that today’s programmed computers cannot do.
Subscribe to our News Digest for the latest news about neuroscience and artificial intelligence:
https://tinyurl.com/NumentaNewsDigest
Subscribe to our Newsletter for the latest Numenta updates:
https://tinyurl.com/NumentaNewsletter
Our Social Media:
https://twitter.com/Numenta
https://www.facebook.com/OfficialNumenta
https://www.linkedin.com/company/numenta
Our Open Source Resources:
https://github.com/numenta
https://discourse.numenta.org/
Our Website:
https://numenta.com/
A
All
right,
yeah
hi,
I'm
viviane
clay.
I
work
at
the
university
of
network
and
study
for
a
phd
there
and
I
almost
visited
you
guys
last
at
the
beginning
of
this
year,
but
didn't
work
out
with
the
visa.
So
now
I
have
to
talk
to
you
remotely,
but
I
still
hope
that
maybe
we
can
work
together
on
something.
A
Yeah
so
I'll
just
give
a
quick
overview
of
my
work
and
my
overarching
goal,
and
it
will
be
very
general
in
the
beginning
and
then
I
hope
that
I
stay
on
the
best
level
of
detail
for
you
guys.
So
if
I
am
too
general
or
too
specific
or
too
fast
or
too
slow,
just
let
me
know
and
yeah
feel
free
to
interrupt
me
all
right.
So
I'm
looking
at
learning
through
embodiment,
so
learning
through
interacting
with
the
world
and
how
I
got
to
this
is
previously.
A
C
A
C
A
All
right,
so
conventional
neural
networks
are
really
good,
often
better
than
human
performance
by
now,
but
they
have
some
drawbacks
and
some
weaknesses.
I
don't
know
how
many
of
you
have
heard
of
adversarial
attacks.
It's
basically,
when
you
add
some
specific
noise
to
an
image,
really
weak
noise.
So
here
in
this
example,
it's
multiplied
by
0.007.
A
You
can't
even
really
see
the
difference
between
the
panda
on
the
left
and
on
the
right,
but
it
makes
a
huge
difference
to
the
network,
because
without
the
noise
it
classifies
the
image
as
a
panda
and
with
the
noise.
It
classifies
it
with
very
high
confidence
as
a
given,
which
is
a
mistake
that
a
human
would
never
make
and
the
same
similar
effect.
A
You
can
also
see
with
natural
images
that
you
didn't
add
any
noise
to,
but
they
just
have
some
low
level
properties
that
trick
the
network
like,
for
example,
the
image
of
the
squirrel,
that's
misclassified
as
a
sea
lion,
because
you
can
kind
of
see
the
texture
of
this
water.
Fountain
looks
a
bit
like
the
texture
of
a
sea
lion's
skin,
but
the
shape
is
totally
different.
A
And
to
this
I
think
it's
always
a
nice
idea
to
look
for
some
inspiration
in
nature,
because
nature
has
spent
millions
of
years.
Evolving
and
sometimes
different
species
have
evolved.
The
same
solution
to
the
same
problem
independently,
like
using
sonar
to
see
in
the
dark
bats,
have
evolved
it
and
dolphins,
for
example,
and
then
taking
it
as
an
inspiration
it
can
be
used
by
humans
in
submarines,
for
example
same
with
wings,
the
structure
of
wings
being
developed
in
bats
and
birds,
or
just
a
general
body,
shape
evolving
in
the
same
direction.
A
So
since
I'm
looking
at
learning,
I
took
a
look
at
some
theories
of
how
learning
in
humans
works
and
here
are
prg's
stages
of
cognitive
development
in
children.
So
you
can
already
see
from
the
number
of
years
at
the
bottom
that
it
takes
a
long
time
for
a
human
child
to
really
develop
to
its
full
potential
and
in
the
beginning,
there's
just
some
very
weakly
supervised.
Interaction
with
the
world.
A
I
was
actually
surprised
that
this
is
only
understood
at
the
age
of
seven
starting
at
the
age
of
seven
and
yeah
then
later
on
in
development,
children
learn
things
like
deductive
and
inductive
reasoning,
hypothetical
thinking
and
so
on,
and
how
we
train
a
lot
of
machines
currently
is
very
different
to
that.
So,
first
of
all,
most
of
the
artificial
neural
networks
don't
have
a
body,
so
they
don't
interact
with
the
world.
A
Often
they
get
trained
on
the
final
task
from
the
beginning
on
so
there's
no
gradual
easing
into
the
learning
and
often
they
are
trained
fully
supervised
so
at
every
instance.
There's
a
supervision
signal
telling
the
network.
What
is
the
correct
or
the
wrong
answer,
and
I
will
mainly
focus
on
supervision
and
embodiment
in
learning
and
look
at
how
how
the
learning
changes
if
we
make
them
more
similar
to
how
humans
learn.
A
A
And
in
my
case,
I
use
a
3d
navigation
task
where
the
main
observations
are
rgb.
Camera
images
collected
in
a
simulated,
3d
maze
world
and
additionally,
a
small
vector
observation
telling
the
agent
some
things
about
the
environment,
like
how
many
key
stages
is
holding,
how
much
time
is
left
and
the
level
and
the
actions
are
walking
forwards
and
backwards,
turning
left
and
right,
jumping
or
not,
jumping
and
turning
the
camera
left
or
right,
and
just
to
give
you
a
better
impression
of
this
here's,
a
short
video
of
the
agent
running
through
this
tower.
A
There
are
several
rooms
on
each
floor.
You
have
to
go
through
the
green
doors
to
get
to
the
next
room.
The
yellow
doors
lead
to
the
next
level
blue
spheres
give
more
time
once
the
time
is
up.
It's
game
over
and
the
agent
has
to
restart.
Sometimes
agent
needs
to
pick
up
a
key
to
open
a
specific
door
and
later
on,
some
puzzles
are
introduced
and
even
later
on
enemies
and
more
complex
tasks
and
the
environment
is
randomly
generated,
which
means
that
the
agent
can't
just
memorize
the
path.
C
I
have
a
question
about
that
vision.
Why?
Why
did
you
I'm
sorry?
I
was
just
surprised.
You
didn't
have
a
first
person
camera
view,
which
is
what
I
would
have
expected
because
that's
the
agent
and
that's
what
the
agent
be
seeing.
So
it
was
the
level
of
complexity.
You
were
adding
this
this.
You
know,
I
don't
know
third-person
can't
review
it.
I'm
just
surprised
by
that
surprise
that
worked
and
and
why
you
chose
that.
A
Yeah,
that's
not
a
very
good
answer.
I
have
for
that,
because
this
environment,
I
started
using
it
as
part
of
her
challenge.
So
it
was
a
challenge
posed
by
unity,
and
I
took
part
in
that
challenge
and
trained
that
agent
for
this
basically
and
since
the
training
takes
like
about
30
days
and
it's
difficult
for
the
agent
to
even
learn
this,
I
didn't
do
it
again
with
the
first
person
camera
view,
but
this
is
definitely
on
the
list
of
next
things
to
change
to
use
the
first
person
view.
C
It
definitely
made
it
harder
for
me
to
understand
how
the
whole
system
works
because
like
well,
tim
is
changing
the
agents
changing.
How
would
it
you
know?
It's
certainly
not
going
to
learn
what
we
would
typically
consider
a
sensory
motor
or
an
embodiment
which
is
like.
Oh,
I
move
my
head
left,
that's
what
I
change
do
I
see
here
you
know
here
you
have
the
camera
moving
and
the
body
moving,
and
it's
like
okay,
but
I
got
it
you're
saying
this
was
a.
This
was
a
pre-existing
challenge
you
just
took.
A
B
Yeah,
this
is
a
good.
This
image
is
a
good
example
like
we
right
now.
We
don't
actually
see
what
the
agent
is
seeing.
Yeah
like
if
there's
a
door,
because
there's
some
pillars
that
are
kind
of
obscuring.
If
there's
a
green
door
or
a
yellow
door
or
whatever
beyond
it,
we
don't
actually
see
that
right
now,
yeah.
C
A
C
A
C
Know
it
it
struck
me
maybe
what's
going
on
here,
is
you
know
the
the
the
woman
the
person
the
agent
whatever
she
is
she's,
always
in
the
center
right?
The
video
shows
you
she's
in
center.
I
wonder
if
the
system
would
work
just
as
well
without
her
there
at
all,
I
mean
basically
just
ignore
her
yeah
whatever.
Basically,
then,
the
camera
becomes
your
first
person
view
and
that's
what
you're
solving
for.
C
A
Yeah,
so
it
solves
it
pretty
well,
it
can
reach
up
to
level
10
at
and
at
that
point
some
spatial
tasks
are
introduced
which
need
some
memory
to
solve
them,
and
the
network
I'm
using,
is
really
simplistic
and
doesn't
have
any
memory
at
all.
So
this
agent
right
now
reaches
up
until
that
task
shows
up.
E
E
So
you
have
a
body
part
if
you
wish
that
you
know
what
the
appropriate
sensitive
position
of
it
is,
which
is
the
agent
sitting
right
there,
and
then
it's
exploring
the
world
in
in
you
know,
there's
the
where
the
agent
touches
something
like
a
key.
It's
almost
like
a
tangibility,
so
I
I
think
it
is
verging
on
solving
a
a
it's
very
abstract,
but
solving
a
problem
of
where
am
I
relative
to
the
world
and
where
is
my
body
part
relative
to
the
world
type
of
thing?
E
So
I
I
don't
necessarily
see
it
as
being
totally
tangential
to
you
know
the
goal
of
basically
moving
my
whole
body
through
the
thing
it's
like
I've
got
these
two
pieces
and
I'm
trying
to
guide
my
interaction,
and
you
said
it
develops
a
policy
with
how
it
treats
you
know
where
to
put
the
camera
relative
to
its
body.
Part,
if
you
wish
that's
part
of
the
scene,
so
I
still
think
it's
an
interesting
problem.
C
C
Does
give
it
some
information?
I
I
think
this
network
would
solve
it
perfectly
the
same.
If
it
didn't
have
that
picture
of
the
woman
there,
it's
probably
just
ignoring
it,
because,
although
it
basically
is
trying
to
get
like
the
key
into
that
point
and-
and
so
I
I
don't
think,
there's
going
to
be
any
representation
of
the
body-
the
camera,
to
the
object,
everything
it
just
seems
like
it
would
solve
the
problem
completely
without
her
there,
which
just.
C
A
B
A
So,
as
I
said,
there's
a
closed
loop
between
action
perception,
so
we
have
the
observations
going
in.
They
are
being
encoded
by
the
neural
network
into
an
encoded
state.
This
is
what
I'm
mainly
looking
at
what
what
information
is
in
the
encoded
state
and
then
this
encoded
state
is
used
to
decide
on
the
next
action
with
this,
which
is
then
executed
and
determines
the
next
observation,
and
the
observations
have
a
dimensionality
of
over
84
000
and
the
encoded
state
then
contains
256
neurons.
A
F
A
E
A
Okay,
so
now
this
encoded
state
has
basically
256
activations
with
every
input
image,
and
I
reduce
this
dimensionality
to
two
using
tsne
just
to
visualize
it,
and
even
after
reducing
the
dimensions
down
to
two,
you
can
still
see
a
lot
of
a
lot
of
structure
just
from
the
action
that
was
performed,
so
the
encoding
that
the
agent
learns
is
very
action
oriented,
but
also
action.
Relevant
objects
are
encoded
in
the
network.
A
For
example,
if
you
look
at
the
circles
that
are
circled
red,
they
all
showed
an
image
of
a
door
that
leads
to
the
next
level,
and
you
can
see
that
they
form
kind
of
clusters
within
the
action
clusters.
So
they
don't
form
a
single
cluster,
but
within
the
action
clusters,
which
makes
sense
because
the
door
on
the
left
side
side
is
something
different
than
a
door
on
the
right
side
and
needs
to
be
encoded
distinctively
for
the
agent
to
make
the
right
move
to
walk
through
the
door.
A
But,
for
example,
things
like
illumination
that
is
not
actually
relevant
at
all
seems
to
be
discarded.
So
if
you,
for
example,
look
at
the
images
on
the
top
left,
there's
one
with
a
very
yellowish
light,
one
with
not
really
much
light
at
all
and
one
with
more
greenish
light,
and
they
are
all
encoded
similarly
to
each
other,
because
they
all
contain
the
same
type
of
door.
In
a
simple
location
where
the
agent
needs
to
walk
towards
the
right
side
to
go
through
the
door.
A
A
They
basically
have
the
same
encoding
so
also
two
convolutional
layers,
two
dense
layers
up
to
the
encoded
state.
They
are
all
three
exactly
the
same,
but
then
they
have
different
tasks.
So
the
autoencoder
then
decodes
this
encoded
state
and
is
supposed
to
reconstruct
the
input
image
and
the
classifier
is
supposed
to
say
which
objects
can
be
seen
in
this
image
and
receives
labeled
data
that
was
annotated
by
me.
B
B
A
B
A
So
here
we
have
one
run
of
the
eighteen
through
the
tower.
It's
four
thousand
images,
and
I
give
the
same
four
thousand
images
to
the
embodied
agent,
the
autoencoder
and
the
classifier,
and
look
at
the
activation
in
the
hidden
layer-
and
this
is
just
one
neuron
neuron
number
one
in
this
layer,
and
you
can
see
that
autoencoder
and
classifier
neurons
are
mostly
active
all
the
time
but
modulate
their
strength.
A
The
strength
of
the
activity
and
the
agent
seems
to
have
more
spike
like
activation
pattern.
So
most
of
the
time
it's
just
at
zero
or
slightly
below
zero,
and
then
sometimes
it
is
has
like
a
spike
like
activation
and
just
for
understanding.
I
use
a
swish
activation
function
which
is
shown
on
the
right,
so
the
activation
can
range
from
about
minus
three
minus
0.3
to
infinity.
A
C
A
C
And
if
you,
if
I'm
just
curious,
how
robust
is
it
to
certain
changes?
For
example,
if
I
drop
the
agent
in
a
spot
in
the
room
it's
never
been
before,
will
it
figure
out
where
it
is
and
somehow
get
back
on
track,
or
does
it
wander
around
aimlessly
until
it
recognizes
something,
and
also
is
it
robust
adversarial
changes
like
you
know
what,
if
I
change
the
the
texture
of
the
walls,
yeah
and
so
on,
something
like
that.
A
Yeah,
oh
also,
the
texture
of
the
walls
can
can
can
be
different
in
different
runs,
so
it's
consistent
throughout
one
level,
but
it
can
be
a
different
texture.
The
next
level-
and
I
also
after
training
the
agent
tested
it
with
different
lighting
conditions,
so,
for
example,
turning
the
light
red
or
purple
or
yellow
which
it
has
never
trained
on
before,
and
then
the
performance
definitely
drops
drops
a
bit,
but
it
can
still
navigate
a
little
bit.
D
A
B
A
B
D
A
D
B
A
A
B
Yeah,
that's
what
I
was
wondering
is
like
is
is
for
to
do
the
full
navigation
task
is
this
is,
do
you
actually
need
the
sparsity
or
not?
I
guess
we
don't
know
because
you
could
represent,
you
could
just
they're
all
exactly
identical.
You
could
just
replace
the
representation.
That's
learned
through
embodiment
with
the
one.
That's
represented,
learn
through
autoencoder
or
learn
through
the
classifier,
and
you
can
still
back
propagate
through
the
whole
thing.
You
can
just
leave
those
weights
fixed.
A
Right
yeah:
that's
a
good
idea
to
test
I've
read
a
paper.
They
showed
that
it
actually
helps
to
enforce
sparsity
in
reinforcement
learning
agents
so
that
they
are
better.
If
you
have
some
kind
of
sparsity
loss
added
into
the
optimization
but
yeah
I
haven't
tried
that
yet
and
how
how
easily
they
can
learn
a
policy
on,
for
example,
the
autoencoders.
A
A
It's
shown
in
how
many
percent
of
the
frames
each
of
the
neurons
are
active,
x
and
y
axis
means
nothing
it's
just
because
you
can't
really
plot
a
256
long
vector,
so
I
just
reshaped
it
in
a
square
format.
Just
hope
it
doesn't
confuse
and
yeah
in
the
embodied
agent.
Most
neurons
are
active
less
than
forty
percent
of
the
time
more
around
ten
to
twenty
or
thirty
percent
of
the
frames
in
the
outer
encoder.
Basically,
all
neurons
that
are
not
dead
are
active.
A
All
the
time
and
in
the
classifier,
the
neurons
active
around
like
30
to
70
percent
of
the
frames
just
for
quantifying
it
a
bit,
so
the
embodied
agent
has,
on
average
12
active
neurons
in
each
frame.
That's
about
5
well,
outer
encoder
has
about
93
percent
active
at
each
frame
and
the
classifier
68
active
and
in
the
agent
there
are
only
three
neurons
that
are
active
in
more
than
40
percent
of
the
frames,
and
the
most
active
neuron
is
active
in
68
of
the
frames.
So
basically
it's
like
yeah.
A
If
you
look
at
all
these
numbers
together,
it's
like
in
every
frame.
There
are
very
few
neurons
active,
but
overall,
almost
like
a
lot
of
the
neurons
are
used
in
general,
but
yeah
with
distributed
unique
patterns
for
the
different
visual
inputs
and
yeah.
Similar
things
have
been
observed
in
in
nature
and
yeah.
I
think
you
guys
have
shown
also
that
it
is
very
efficient
and
robust
way
to
encode
information.
C
And
that
plus
brains
tend
to
they
have
mechanisms
to
make
things
sparse.
So
it's
it's
enforced.
It's
not
just
learned
in
your
case,
you're,
not
enforcing
it.
You're
just
saying
it's
learned
right
and
the
brains
also
force
it
so
yeah,
which
is
an
interesting
question.
I
think
this
is
your
question
like:
why
did
they
come
out
sparse.
A
B
C
I
guess
I
don't
understand
these
these
neural
networks
well
enough
to
to
know
how
it
is
even
possible
that
you
train
a
convolutional
neural
network
and
one
comes
out
really
sparse
and
one
doesn't
I
mean
that's
beyond
sort
of
my
understanding
of
these
convolutional
neural
networks.
It's
just
sort
of
so
I'm
kind
of
confused
by.
D
A
C
B
Yeah
everyone
one
other.
There
are
a
couple
of
other
papers
that
have
found
something
similar
in
the
context
of
continual
learning.
So
there's
a
paper
I
think
came
out
last
year,
where
they're
learning,
whether
they
use
meta
learning
to
learn
a
representation
such
that
continual
learning
works
well
and
they've
also
found
to
their
surprise
that
the
representations
were
sparse.
B
You
know
they
didn't
build
in
any
sparsity
in
the
beginning.
So
that's
another
case
where,
for
whatever
reason
you
know
representations
came
out
as
sparse
there,
so
it
might
be
an
interesting
correlation.
You
know
analogous
work
to
what
you're
doing
here.
It's
very
interesting.
It's
like
why.
Why
does
it
come
out
that
way?
Yeah.
A
F
Yeah,
so
sorry,
if
anyone's
asked
this
already
and
my
internet's
occasionally
getting
up
but
yeah,
so
you
mentioned
this
paper
that
suggested
sparsity
improves,
reinforcement
or
can
potentially
improve
agents
and
reinforcement
learning
tests.
Have
you
had
a
chance
to
see
if,
if
you
do
explicitly
regularize
for
sparsity,
if
enforcing
that
improves
your
agent's
performance
or
even
further.
A
C
A
Yeah,
so
in
this
case
they
are
yeah
39
that
are
not
active
in
any
of
the
test
frames,
but
once
if
I
add
more
test
frames,
it
also,
this
number
increases
of
the
number
of
neurons
used.
So
it
might
be
that
the
neurons
that
were
not
active
in
these
8
000
frames
that
they
are
just
not
receptive
to
the
objects
that
were
or
the
textures
that
were
in
this
test
set.
So
I'm
sure
some
of
the
neurons
are
just
dead
and
not
used
at
all,
but.
C
C
C
A
C
C
You
know
well,
if
I
understand
this
correct,
that,
like
the
classifier
you're
just
saying
you're
saying
oh
there's
a
door
right,
but
what
you
really
need
to
know
is
that
this
is
the
door
in
this
particular
room
on
this
level
or
something
like
that.
And
so,
if
you
just
classify
doors
you
would
you
would
not
have
that
additional
context,
if
I
understand
it
correctly,
but
if
you're
trying
to
solve
the
problem,
you
need
that
additional
context.
So
at
least
you
would
some
places.
C
I
assume
I
don't
know
well
like
I
see
this
room
and
it's
like
well.
It
looks
like
a
room
on
the
second
floor,
but
it
could
be
the
same
room
on
the
fourth
board,
but
I
have
different
actions
based
on
those
rooms,
and
so
just
from
that
point
of
view,
when,
whenever
you
add
context
to
a
representation,
you
have
the
ability
you
have
the
option
of
including
you
know,
a
a
more
unique
representation.
C
A
Else,
yeah,
that's
actually
a
really
really
interesting
explanation
for
it.
I
haven't
thought
of
it
this
way
specifically.
Yet
I
have
a
plot
here
about
how
the
sparsity
develops
over
time
and
the
blue
line
is
the
embodied
agent.
So
you
can
see
it
already
gets
sparse
pretty
much
from
the
beginning
on.
F
A
C
A
D
C
A
I
think
it
depends
on
the
task
like
if
there's
some
kind
of,
if
it's
like
a
continually
difficult
task
or
if
there's
like
a
threshold
like
in
this
task.
If
you
once
learned
that
it's
good
to
walk
through
doors
and
what
the
door
is,
then
you
can
already
kind
of
have
like
a
performance
and
if
you
never
figure
that
out,
then
you're
never.
D
Yeah
one
one
thing
is
that
the
main
difference
between
reinforcement,
learning,
supervised
learning
is
that
in
supervised,
the
projector
is
fixed,
you're,
always
going
to
see
those
images,
whereas
in
reinforcement
learning
the
trajectory
changes.
So
if
you
take
like
a
wrong
turn,
let's
say
you
have
a
two
doors
as
your
environment
and
if
you
take
a
wrong
turn
in
the
beginning,
you
cannot
end
up
with
a
completely
wrong
trajectory
and
you're
not
going
to
see
anything
useful.
C
It's
it's
surprising
still,
but
in
sense
like
you
would
think
that
the
learning
rate
would
be
slow
enough
that
that,
even
if
you
made
the
wrong
turn
in
the
beginning,
you'd
eventually
get
back
on
track
and
it
would
figure
out
what
was
the
right
turn.
I
mean
but
you're
saying
no
once
you
once
you
started
on
the
wrong
path:
you're
dead.
D
Yeah,
you
knew
some
environments,
that's
what
happened.
You
take
wrong
path
in
your
dad
or
the
time
man.
So
it's
usually
like.
So
you
have
a
pressure
to
to
end
the
problem,
so
that
pressure
might
be
time
or
might
be
like
traps
or
something
like
that.
So
you,
if
you
take
a
wrong
turn,
you
may
just
end
up
that
and
then
you
have
to
go
for
the
next
episode.
I
never
reduction.
A
Yeah
and
it's
like
the
the
policy
that
is
trained,
basically
determines
the
quality
of
the
data
that
it
can
use
to
train
itself.
So,
in
the
beginning,
your
policy
slips
into
a
bad
policy
you're,
not
getting
good
observations
to
learn
from,
but
it
can
recover.
So
I've
had
trainings
where
I
didn't
learn
anything
for
three
or
four
days,
and
I
suddenly
learned
something
so
yeah.
C
C
F
Jeff
there's
some
good
animations
of
like
this
is
task
called
half
cheetah.
I
think
where
you
have
to
like
get
this
cheetah
model
to
like
move
forward
and,
for
example,
it's
supposed
to
like
use
its
legs
to
run.
Obviously,
but
if,
at
the
start
it
learns
to
flip
on
its
back,
then,
while
it's
on
its
back,
it
can
kind
of
like
jitter
on
the
ground
and.
C
F
C
C
Yeah
so
so
you
know
it
happens
with
cooking
a
lot
like
oh
shoot.
You
know
I
just
couldn't
save
myself
a
lot
of
time,
but
but
in
this
case
I
think
vivian's
example,
it's
not
even
it's
not
even
solving
the
problem,
it's
not
solving
it
poorly.
It's
not
like
you
know
the
agent's
just
jumping
up
and
down
all
the
time,
but
it's
just
not
even
solving
it.
Interesting.
C
A
So
it's
called
fast
mapping
in
children
that
you
that
they
can
remember
a
new
word
or
concept
just
with
one
example,
and
then
that's
what
I
was
going
to
look
at
next
and
just
as
a
little
extra
I
took
out
all
external
rewards
to
be
able
to
say
that
the
agent
really
learns
self-supervised
and
doesn't
have
any
external
supervision,
and
instead
let
it
learn
with
curiosity
which
basically
yeah
it's
a
little
long
to
explain
I'll
skip
over
it.
For
now.
A
You
know
in
order
to
see
unpredictable
observations,
you
have
to
take
actions
that
take
you,
for
example,
to
new
rooms.
You
can't
just
be
staring
at
the
wall
all
the
time,
because
that's
perfectly
predictable,
and
just
by
using
curiosity
as
a
reward,
the
agent
can
actually
learn
to
navigate
in
this
tower,
as
well
without
external
supervision.
A
So
that's
the
only
change
in
these
results
now
and
I
want
to
look
at
so
we
have
an
image
we
give
it
to
the
network.
We
look
at
how
this
is
represented,
and
then
I
want
to
associate
a
semantic
label
with
it
and
the
first
two
steps
we
already
have.
We
have
a
trained
network
and
can
get
the
representation
of
the
image,
and
now
how
can
it
be
associated
with
a
semantic
label?
A
And
I
I
call
this
fast
concept:
mapping
inspired
by
fast
mapping
of
children
and
you
take
n
examples
of
the
concept.
For
example,
here
I
take
five
examples
of
the
concept
level
door,
so
a
door
that
leads
to
the
next
level.
It's
marked
by
a
yellow
arrow,
pointing
upwards
and
the
examples
shouldn't
be
all
the
same
image.
Of
course,
they
should
have
a
little.
They
should
represent
this
concept,
but
they
can
be
also
other
objects
in
the
image.
The
object
can
be
at
different
positions:
different
illumination
everything.
A
So
then
you
take
the
m
strongest
connections
to
define
the
concept.
Here.
I
take
the
10
strongest
connections
and
I
define
the
concept
with
the
additional
information
of
weights
for
each
connection
and
the
way
it
is
determined
by
in
how
many
of
the
five
example
images
this
connection
is
present
or
active,
and
then
you
can
do
this
for
as
many
concepts
as
you
like
to
extract,
for
example.
A
A
Just
as
a
comparison,
the
purple
line
shows
a
fully
supervised
network
that
was
trained
with
about
6
million
labeled.
Examples
so
normally
object.
Detection
networks
need,
like
tens
of
thousands
of
labeled
examples
in
order
to
learn
this
task,
while
the
agent
already
has
above
chance
performance
with
just
one
example
and
the
performance
increases
more
and
more,
the
more
examples
you
use
to
extract
the
concept,
but
from
the
beginning
on
it's
a
lot
better
than
the
autoencoder
or
a
random
representation.
A
A
On
the
left
side
is
the
connectivity
graph
for
five
examples
of
a
level
door
and
on
the
right
side,
250
examples
of
the
level
door
and
in
the
250
examples
there.
Of
course,
a
lot
more
different
connections
present
at
some
point,
but
the
ones
that
are
consistently
present
in
the
250
examples
are
the
same
as
the
ones
that
are
consistently
present
in
the
five
examples,
which
should
be
a
property
of
a
good
encoding
that
encodes
this
concept.
A
And
here
are
just
examples
of
the
different
concepts
extracted
and
you
can
see
that
they
are
quite
different
from
each
other.
There's
concept
overlap
in
some
of
them,
which
is
not
necessarily
a
bad
thing
because,
for
example,
the
yellow
door
and
the
blue
time
warp
both
give
additional
time
to
the
agent.
So
it
might
be
that
the
overlapping,
the
the
connections
that
are
the
same
encode
that
this
object
should
be
approached
and
gives
additional
time
and
yeah.
You
could
extract
any
kind
of
concept.
B
A
Okay,
all
right
so
what's
also
cool
about
this
here
I
added
the
classifier
performance,
so
not
the
classifier
output
layer,
but
just
performing
performing
the
same
concept.
Mapping
on
the
classifier
encoding
with
also
1
250
examples
and
the
classifier
encoding
was
trained
specifically
to
encode
these
objects,
but
still
the
agent
is
almost
as
good
as
the
classifier
to
to
encode
these
objects
and
for
them
to
be
extracted
with
few
examples.
B
A
And
then
I
look
at
the
hidden
layer
and
the
classifier
and
try
to
extract
these
objects
with
the
same
procedure
as
in
the
agent,
where
I
just
take
like
five
examples
and
look
which
neurons
are
active
together
and
then
see
how
good
this
can
already
be
extracted
from
few
examples.
Just
from
the
hidden
representation.
A
You
can
see
that
the
agent
performs
better
on
objects
that
are
well
integrated
in
its
policy,
so,
for
example,
the
level
door
and
the
orb
work
really
well
to
extract
by,
for
example,
the
concept
no
door
can't
be
extracted
at
all,
it's
even
worse
than
chance
performance,
because
no
door
is
just
not
really
a
concept
you
use
for
an
action
policy,
so
it
doesn't
seem
to
be
encoded
in
the
representation.
So
it
also
can't
be
extracted.
D
Can
I
interpret
real
quick
so
so
last
slide,
I
asked
we
trained
the
new
classifier
for
the
embodied
agent,
but
my
understanding
is
that
you
didn't
train
a
new
classifier.
You
just
got
those
few
samples,
you
label
them,
and
then
you
learn
just
through
like
clustering.
So
you
know
that
example
is
close
to
this
example,
so
it
belongs
to
to
this
class,
but
there
is
no
actual
learning
process
going
on
after
right.
F
A
A
D
A
Yeah
so
in
in
accuracy
the
on
average,
the
agent
actually
outperforms
all
other
conditions,
except
for
the
the
classifier
readout
layer,
which
was
trained
fully
supervised,
like
with
weights
going
from
each
neuron
to
the
output
classification
and
when
taking
into
account
the
precision
and
recall
as
well.
The
classifier
is
a
little
bit
better
than
the
agent
overall
in
extracting
the
different
concepts
out
of
the
representation,
and
this
is
all
for
five
examples.
For
each
concept.
A
So
if
the
concept
is
encoded
in
the
representation,
it
can
be
extracted
without
retraining
or
huge
amounts
of
labeled
data,
so
in
a
fully
supervised
object
recognition
network,
if
you
train
it
and
then
you
notice,
you
also
want
to
detect
keys.
You
usually
have
to
retrain
the
whole
network
again
with
this
additional
object
class.
While
here,
if
the
embodied
agent
encodes
keys,
because
they
are
action
relevant,
then
you
can
also
extract
it.
A
One
can
add
some
concept,
logic
like
if
there's
a
door,
it's
just
one
type
of
door,
not
multiple
types
of
doors
and
that
can
either
be
explicitly
defined
to
improve
performance
or
extract
from
concept
overlap.
Somehow,
implicitly,
and
one
of
my
outlook
next
ideas
is
to
bring
these
concepts
these
extracted
concepts
back
into
the
world
and
use
them
for
decision
making.
A
B
B
Have
you
tried
that,
at
all
with
and
compared
that
against
the
classifier
version,
because
when
you're
learning
the
same
concepts,
it
looked
like
the
classifier
and
the
fast
concept
mapping
they
both
worked
equally
well,
which
was
great.
But
I
wonder
if
you
try
on
new
objects
that
the
classifier
was
not
trained
on
at
all,
whether
the
whether
the
classifier
network
would
still
work
reasonably
well
or
not.
D
B
A
So
the
observation
is
again
encoded,
similar
to
the
green
part
into
an
encoded
state,
and
then
it
has
two
different
tasks.
One
task
is
to
use
the
current
encoded
current
state
and
the
encoded
next
state
to
say
which
action
was
performed
in
between
these
two
states,
and
that
is
then
compared
to
the
extra
action
that
was
performed
and
is
called
the
inverse
loss,
and
the
task
is
to
use
the
encoded
state
plus
the
action
that
is
performed
to
predict
what
will
be
the
next
state.
So
if
I
am
here,
I
perform
this
action.
B
Okay,
that's
interesting!
That's
a
little
different
than
so
some
of
the
stuff
we've
done
and
also
in
the
biology
there's
like
there's
something
called
a
reference
copy
where
the
the
action
you're
about
to
perform
is
also
fed
into
the
system,
and
so
you
can
use
the
the
action
you're
about
to
do
to
predict
the
next
state.
B
A
B
A
F
That
was
a
super
interesting
talk.
One
thing
I
was
wondering
and
again
sorry
if
you
mentioned
this
and
it
just
cut
out
when
you
did,
but
could
you
do
anything
almost
like
a
concept
arithmetic?
Do
you
think
where,
like
because
you
mentioned
that,
for
example
like
blue,
the
blue
ball
and
the
yellow
door
seem
to
share
some
representational
overlap
because
they
kind
of
give
time,
but
then,
presumably
maybe
the
doors
share
some
representational
overlap
as
well.
A
Yeah,
so
I'm
thinking
about
that
and
I
haven't
really
formally
tested
it
yet
that
would
basically
be
my
step
when
I
try
and
move
on
to
the
concrete
operational
stage.
I
guess:
okay,
currently,
I'm
still
in
the
pre-operational
stage
very
early
in
the
pre-operational
stage,
so
I
haven't
been
into
more
complex
concept
logic.
Yet.
E
E
E
A
Sense,
yes,
I
I
only
use
this
very,
very
simplistically
and
in
this
video
that
I
said
that
for
a
key
and
orb,
it
has
to
be
above
a
certain
probability
to
be
detected
and
then
for
the
doors
it
takes
the
most
probable
door
and
also
no
door.
It
can't
say
no
door
and
another
door
so
that
it
just
looks,
which
is
most
probable
and
above
the
probability
threshold,
and
that's
the
only
place
where
I
use
this
pretty.
E
A
E
So
it
would
take
for
when
you
move
to
the
the
kind
of
conceptual
thing
of
the
notion
of
saying
what
the
com,
what
the
commonality
of
network
is
like,
if
I
look
you
know,
do
ands
across
all
the
door
representations
does.
Is
there
a
sub
network?
That
kind
of
you
know
abstracts
out,
you
haven't
reached
that
stage
yet.
A
A
Yeah
here
this
is
things
that
I'm
looking
for
next,
so
one
is
a
big
question
responsibly.
How,
where
does
it
come
from
and
does
it
help
with
concept
extraction
and
all
this
then
adding
memory
to
the
agent
and
what
effect
that
would
have
and
then
also
concept,
relations
and
logical
reasoning
and
the
concept
overlap
and
so
on?
D
B
D
D
A
Yeah
so
basically,
for
example,
the
first
image
is
given
to
the
network
and
then
I
look
at
the.
So
it
goes
through
the
convolutional
layers
to
through
two
dense
layers,
and
then
the
activations
of
the
second
dense
layer
are
the
ones
that
I
used
to
decide
the
actions
on.
So
I
look
at
this
those
activations
and,
for
example,
for
this
first
image.
There
are
one
two
three
four
five
six
active
neurons
when
this
image
is
shown,
and
then
I
just
draw
connections
from
each
of
the
six
neurons
to
each
of
the
other
files.
B
E
B
E
A
A
E
E
D
A
A
B
D
B
A
D
D
B
A
E
A
Yeah,
should
I
go
over
a
few
of
the
sparsity
plots,
sure
yeah,
okay,
so
most
of
them
just
to
preface
it.
I
only
run
once-
and
I
know
that,
usually
you
should
run
experiments
multiple
times
to
really
know,
know
it,
but
it's
just
there
were
so
many
things
to
run
and
each
of
them
take
a
couple
of
days
to
run
so
didn't
have
time
to
really
do
multiple
runs
but
yeah.
A
So
the
question
is:
where
does
the
sparsity
come
from
and
first
I
tested
if
it
is
just
because
of
the
task
so
because
I'm
predicting
actions,
maybe
predicting
actions,
is
a
certain
kind
of
task
that
enforces
this
so
to
test
it.
I
trained
a
supervised
network
to
predict
the
agent's
actions
and
it
gets
yeah.
It
wants
to
predict
the
actions,
but
the
the
genie
index
which
I
used
to
measure
sparsity
doesn't
increase
so
yeah.
A
A
A
Then
I
wondered
maybe
it's
the
shift
and
input
distribution
that
in
the
beginning,
it
just
sees
one
room
and
then
the
input
distribution
changes
with
the
policy
changing
and
to
check
it.
I
once
trained
an
agent
that
only
has
the
first
floor
available
to
it
so
very
small
environment
and
then
one
that
just
use
a
random
policy
to
collect
its
observations
and
both
of
them
increase
in
sparsity
from
the
beginning
on.
A
Then
I
thought
maybe
it's,
because
I'm
optimizing
two
objectives,
the
actions
and
the
value
estimate
on
one
representation,
but
also
when
using
two
separate
representations
for
those
two
outputs.
It
still
gets
about
equally
sparse
as
when
using
one
representation,
and
the
last
idea
I
tested
was
if
it's
maybe
some
hyper
parameter.
So,
for
example,
there's
the
entropy
over
the
action.
C
A
A
B
B
A
A
E
E
So
you
can
maybe
tease
that
apart
and
just
get
a
sparsity
index
for
each
of
the
layers,
but
I'm
also
just
wondering
if
there's
anything
structurally,
which
is
going
on
on
the
weights,
whether
they're,
just
you
know
totally
random.
You
know
if
the
sparsity
appears,
if
it's
random
or
if
there
is
some
kind
of
structural
aspect
to
it
as
well.
E
A
F
F
F
A
C
A
A
Yeah
and
then
also
the
agent,
has
a
very
sparse
and
uninformative
reward
signal
to
learn
from
so
it
gets
a
in
the
in
the
supervised
condition.
It
gets
a
reward
in
less
than
two
percent
of
the
frames,
and
even
there
just
gets
told
this
is
the
last
few
actions
that
you
took
seem
to
be
kind
of
good,
but
not
really
like.
In
this
case,
you
should
take
this
action
like
a
supervised
network
would
get
as
to
learn
from.
E
A
B
Might
be
a
if
you're
using
unity,
there
might
be
other
parameters
in
unity
which
could
suddenly
make
the
rendering
much
faster,
like
maybe
the
fidelity
of
the
textures
doesn't
have
to
be
that
high
or
the
quality
of
the
lighting
doesn't
have
to
be
quite
as
high,
and
it
could
be
things
like
that
which
could
really
speed
up.
Though.
D
A
D
So
I
have
a
hypothesis
so
on
the
with
paris
representation,
so
it
seems,
like
you
concluded,
that
the
difference
that
the
weak
signal
is
induced
in
sparse
representations.
That's
kind
of
my
take
away
from
that,
and
I
was
wondering-
and
maybe
you
have
tried
that
I
think
you
you
talk
about
entropy
at
some
point.
If
we
have
supervised
learning
but
with
a
weak
signal,
you
say,
like
you
just
move
out
the
distribution,
you
add
some
noise
there
and
you
have
a
very
weak
learning
signal.
D
B
D
Like
the
other
way
around
so
instead
of
you
can
have
the
reward
per
frame,
so
that
would
help,
but
you
also
treat
supervised
learning
in
such
a
way
that
the
signal
is
weaker.
So
if
you
can
smooth
the
distribution
or
you
can
inject
some
noise
there,
so
you
still
have
some
learning
signal,
but
not
as
strong
as
traditional,
supervised
learning
and.
A
Tried
adding
noise
in
the
classifier
labels
or
just
saying
it
was
correct
or
wrong,
not
saying
what
would
have
been
the
right
label
and
it
didn't
increase
the
sparsity.
But
I
wouldn't
exclude
this
explanation
because
I
it
could
be
the
type
of
noise
that
I
added
but
yeah.
I
actually
tested
it
a
little
bit
in
this.