►
Description
In this Brains@Bay meetup, we focused on how sensorimotor learning can lead to more flexible and robust machine learning systems. To learn more about Brains@Bay, visit our Meetup page: https://www.meetup.com/Brains-Bay/
Richard Sutton (DeepMind and University of Alberta) first talked about the increasing role of sensorimotor experience in AI and proposed a minimal architecture for an intelligent agent that is entirely grounded in experience.
Next, Clément Moulin-Frier (Flowers Laboratory) presented an evolutionary and developmental perspective on open-ended skill acquisition in humans and machines, highlighting the key role of intrinsically motivated exploration in the generation of behavioral regularity and diversity.
Finally, Viviane Clay (Numenta and University of Osnabrück), talked about the capabilities that emerge from letting deep neural networks learn more like biological brains - through sensorimotor interaction with the world.
Follow-up Q&A and slides will be posted soon - check the meetup page for updates.
Link to meetup and talk abstracts: https://www.meetup.com/Brains-Bay/events/282182527/
0:00 Introduction
2:50 Richard Sutton: The Increasing Role of Sensorimotor Experience in Artificial Intelligence
40:59 Clément Moulin-Frier: Open-ended Skill Acquisition in Humans and Machines: An Evolutionary and Developmental Perspective
1:13:05 Viviane Clay: The Effect of Sensorimotor Learning on the Learned Representations in Deep Neural Networks
1:31:33 Discussion + Q&A
- - - - -
Numenta is leading the new era of machine intelligence. Our deep experience in theoretical neuroscience research has led to tremendous discoveries on how the brain works. We have developed a framework called the Thousand Brains Theory of Intelligence that will be fundamental to advancing the state of artificial intelligence and machine learning. By applying this theory to existing deep learning systems, we are addressing today’s bottlenecks while enabling tomorrow’s applications.
Subscribe to our News Digest for the latest news about neuroscience and artificial intelligence:
https://numenta.com/news-digest/
Subscribe to our Newsletter for the latest Numenta updates:
https://tinyurl.com/NumentaNewsletter
Our Social Media:
https://twitter.com/Numenta
https://www.facebook.com/OfficialNumenta
https://www.linkedin.com/company/numenta
Our Open Source Resources:
https://github.com/numenta
https://discourse.numenta.org/
Our Website:
https://numenta.com/
A
Give
a
short
introduction
of
what
brains
at
bay
is
and
what
this
event
is
about,
so
yeah
hi
everyone
we're
still
waiting
for
you
to
pile
in.
So
I
think
we're
gonna,
give
it
around
five
minutes
left,
but
just
welcome
to
another
brains
at
bay.
Today,
we're
going
to
be
talking
about
sensory
motor
learning
and
how
that
can
lead
to
more
flexible
and
robust
machine
learning
systems.
A
So
for
our
newer
members
we're
an
online
brain
inspired
machine
learning,
meetup
and
this
is
hosted
by
dementa.
Our
goal
here
is
to
foster
the
study
and
development
of
machine
learning
algorithms
heavily
inspired
by
newer
science
research.
So
if
you're
interested
in
watching
one
of
our
previous
meetups
or
you
have
to
leave
early
today,
we
record
all
our
sessions,
and
this
will
be
posted
on
nomento's
youtube,
channel
and
I'll,
also
post.
A
Today's
meetup
recording
to
the
meetup
page
afterwards
for
any
feedback
and
suggestions
for
the
next
topic
and
speakers,
please
feel
free
to
just
email.
Me
shoot
me
an
email
in
this
in
the
email
on
the
slide
below
and
also
or
you
can
also
send
me
an
email
through
the
meetup
page
yeah
and
I
think
super
tight
is
joined
who's.
Also,
one
of
the
panelists
hi
supertai,
hey.
B
A
So
we're
very
thrilled
to
have
three
speakers
with
us
today:
there's
richard
sutton
from
deepmind
and
university
albert
alberta,
clermont
moulin,
fried
from
flowers,
laboratory
and
vivian
clay
from
dementa.
If
you
have
any
questions
for
the
speakers,
please
feel
free
to
ask
them
with
the
q
a
feature
on
zoom.
It's
right
below
you'll,
see
a
button
and
there's
an
option
to
ask
a
question
and
you
can
also
vote
on
existing
ones.
A
We'll
also
try
to
address
a
few
more
questions
towards
the
end
of
each
talk,
as
most
of
them
will
be
discussed
during
the
discussion
panel
at
the
end
and
I'll
post
this
blurb
again,
once
more
people
join
and
I'll
just
post
it
on
the
zoom
chat,
so
keep
an
eye
out
for
that
so
yeah.
Let's
give
it
a
few
more
minutes
and
with
that
being
said,
we'll
have
jeff
introduce
our
first
speaker
at
around
1005..
C
Yeah
well,
yeah
give
it
one
more
minute,
but
I
think
I'll
just
say
a
few
words
while
we're
waiting,
it's
it's.
The
brains
at
bay
series
has
been
quite
successful
for
us.
We
we've
had
a
lot
of
people
either
participate,
live
or
online
later
to
review
the
recordings,
and
it's
been
a
great
forum
for
getting
discussions
related
to
machine
intelligence
and
ai
related
to
brains,
which
course
is
a
large
part
of
what
an
event
is
about.
C
C
Charmaine
has
been
introducing
and
running
the
session.
I
just
thought
I
would
introduce
our
first
speaker.
Rich
sutton,
rich,
as
I
say,
is
one
of
those
people
who's
quite
famous
in
the
field
of
ai
and
machine
learning,
and
from
some
people
won't
need
an
introduction.
But,
as
charmaine
said,
he
has
two
positions
on
it:
the
university
of
alberta
and
also
one
of
deep
mines,
alberta,
which
I
think
they
created
just
for
you
rich,
I'm
not
sure.
C
He
also
said,
he'd
been
thinking
about
this
topic
a
lot
and
he
has
a
talk
he
wanted
to
give
us,
which
is
the
title
of
itself.
That's
the
increasing
role
of
central
motor
learning
experience
in
ai,
so
I'm
rich,
I'm
thrilled
that
you
accept
our
invitation
and
that
we
get
to
hear
your
latest
thoughts
on
this
topic,
which
I'm
sure
are
going
to
be
amazing.
So
I
will
turn
it
over
to
you.
D
D
D
Let
me
just
say,
though,
before
we
start
is
the
general
idea
of
the
interaction
between
ai
and
in
between
engineering
and
science
and
natural
science.
It's
a
really
fruitful
interaction,
I'm
one
of
the
many
people
who
tends
to
think
that
that
these
two
have
the
same
goal:
they're
heading
for
the
same
place
in
principle.
They
could
be
different.
D
D
This
is
true
because
the
things
are
diverged
and
they're
treated
in
different
ways,
but
this
talk
today
is
about
how
really
they
keep
being
drawn
back
into
the
same.
The
same
questions,
the
same
line
of
thinking
and
the
line
of
thinking
is
to
think
about
sensory
motor
experience,
and
so
I'm
going
to
talk
about
over
the
history
of
the
70-year
history
of
ai,
how
we
keep
coming
back
to
sensory
motor
experience
and
and
how
it's
somewhat
begrudging
and
reluctant.
D
D
So
we
have
a
picture
of
the
agent
interacting
with
the
environment
with
the
world
and
it
sends
actions
and
gets
back
sensations
and
those
sensations
and
actions
are
what
we
mean
by
experience,
sensory
motor
experience,
and
so
this
these
things
are
obviously
involved
in
reinforcement,
learning
and
predictive
learning,
but
they're
not
involved
really
in
in
supervised
learning,
which
dominates
sort
of
modern
machine
learning,
because
that
those
systems
don't
learn
from
ordinary
experience.
They
learn
from
special
training
experience
that
doesn't
occur
when
the
system
is
put
in
operation
and
during
its
normal
life.
D
Okay,
so
then
the
key
thing
about
experience
experience
is
the
agent's,
only
interaction
with
the
world,
its
only
source
of
information
about
the
world,
and
it
has
meaning
only
with
respect
to
other
data,
except
for
one
one
case.
One
instance
is
that
the
reward
signal,
the
reward
signal.
Well,
I
think
one
over
yeah,
except
for
reward,
which
has
a
special
it's
a
special
scalar
sensation,
which
has
the
meaning
of
being
good
okay.
So
the
the
the
scientific
question
I
want
you
all
to
focus
on
today.
D
First,
step
is
just
the
idea
of
an
agent
and
the
idea
of
having
experience
second,
because
the
idea
is
reward
where
your
goals
of
your
agent
are
in
terms
of
experience,
and
the
third
is
experiential
state.
What
your
state
is
in
terms
of
experience
and
then
finally,
your
knowledge
in
terms
of
experience
and
you're
gonna
you're
gonna,
see
this
slide
many
times.
Don't
worry
about
it
too
much
memorizing
these.
These
four
steps,
I'm
gonna
kind
of
use
this
as
my
outline
and
come
back
to
it
over
and
over
again.
D
The
message
in
in
the
four
steps
is
that
for
each
of
them,
ai
has
has
reluctantly
allowed
experience
to
enter
into
their
thought
patterns.
It's
it's
something
they
they
do.
They
don't
do
first,
first,
they
they
think
in
terms
of
the
external
world
and
then
later,
when
they're
forced
to
or
to
get
the
advantages
of
being
more
learnable,
more
grounded
and
more
scalable
that
they
bring
in
experience.
D
D
So
if
you
look
at
the
first,
the
first
30
years
or
so,
ai
systems
were
mostly
problem,
solvers
and
question
answers.
They
had
no
sensations
and
no
actions.
There
was
always
some
robotics,
but
that
was
an
exception.
A
typical
problem
was
a
or
given
a
start
state
in
the
goal.
State
and-
and
there
were
operators,
state
transformation
steps,
but
they
were
not
expressed
terms
of
actions
and
there
were
certainly
there
was
no
perception
or
sensation.
D
The
solution
was
a
sequence
of
operators
and
in
fact
you
you,
you
never
actually
executed
your
solution
because
your
operatives
were
deterministic,
and
so
you
knew
would
be
the
outcome
each
one,
and
so
once
you've
constructed
the
sequence
of
operators.
You
knew
you
had
a
solution,
you
didn't
actually
have
to
do
it,
so
there's
no
doing
and
there's
no
perceiving
in
in
classic
ai.
D
D
This
is
the
first
edition
of
what
is
now
the
standard,
ai
textbook,
russell
norvig,
and
they
based
their
book
on
the
idea
of
agents
and
at
the
time
that
was
controversial,
not
controversial,
to
talk
about
agents
but
controversial
the
base,
the
field
of
ai
on
an
agent's
perspective,
and
so
these
are
some
quotes.
They
say
the
underlying
theme
of
the
book
is
the
concept
of
an
intelligent
agent,
and
in
this
view
the
problem
of
ai
is
to
describe
and
build
agents
that
receive
percepts
from
the
environment
and
perform
actions.
D
D
D
Now,
with
the
rise
of
machine
learning
and
ai,
the
reward
formulation
is
just
becoming
standard.
For
example,
a
mark
of
decision
processes
are
are
thought
of
as
a
standard
as
one
standard
way
of
doing
planning,
and
even
critics
like
jan
lacoon
would
admit
that
reward
is
the
cherry
on
the
top
of
the
cake
of
intelligence.
D
D
D
Okay,
maybe
I
want
to
say
more
about
that
yeah.
I
do
want
to
say
sorry
about
that,
because
for
many
many
many
people
reward
doesn't
seem
like
it's
enough.
Okay,
the
proposal
is
that
it's
enough,
but
it
doesn't
seem
enough
enough,
maybe
for
animals
or
for
engineering,
but
it's
not
enough
as
an
understanding
of
what
people
are.
What
people
do?
D
A
single
number
from
outside
the
mind
just
seems
too
small
and
too
reductive
and
too
demeaning
that
people's
goals
seem
grander,
raise
a
family,
family-saved
plan,
contribute
to
human
understanding
or
make
the
world
a
better
place.
Surely
our
goals
are
more
grander
than
just
maximizing
our
pleasure
and
comfort.
So
all
these
things
are,
I
can
empathize
with
all
these
things.
I
think
we
can
all
empathize
and
that's
what
I'm
trying
to
do
in
this
talk.
I
want
you
to
empathize
with
both
sides.
You
know
there
there
is.
There
is
a
way
that
we
want.
D
We
want.
We
don't
want
to
think
about
our
goals
as
being
a
number
coming
into
our
brain.
We
want
to
think
about
our
goals
as
doing
something
out
in
the
world.
These
are
the
two
different
perspectives,
thinking
about
what's
going
on
in
the
world
and
achieving
it
and
thinking
about
the
number
coming
into
your
your
mind
and
these
two
ways
of
thinking
we
have,
we
always
start
with
one
and
then
we're
driven
back
towards
an
experiential
thing.
D
Thinking
about
these,
these
numbers
going
into
our
head,
these
signals
going
into
our
head,
and
you
know
there
are
reasons
we
like
that
reason.
We
don't
like
that.
I
want
you
to
empathize
with
both
of
them
and
just
sort
of
understand
this
tension
that's
going
on
and
how
historically
nai,
specifically,
we
reach
first
for
the
external
terms,
and
then
we
are
gradually
driven
to
experience.
D
These
are
good
past
examples
of
this
begrudging
acceptance
and
embracing
of
experience,
and
I
the
next
two
steps
are
going
to
be
more
about
the
future,
although
they
are
in
progress,
but
I
want
to
take
a
little
interlude
and
just
talk
a
little
bit
more
concretely
about
experience.
You
know
what
I'm
trying
to
say.
So,
let's
just
look
at
experience
and
here's
some
some
imaginary
experience
where
we
have
time
steps
clicking
along
here
and
and
like
maybe
right
now
we're
at
the
most
recent
time
step.
D
Seven,
and
we
see
we
see
some
sensations
with
some
numbers,
some
symbols
and
we
generate
some
output
bits.
Okay,
and
so
these
output
bits
are
are
binary,
zeros
and
ones,
and
the
sensory
bits
are
are
not
bad.
These
seem
to
be
bits,
but
then
these
other
signals
seem
to
have
zero
to
three
four
values,
and
this
other
signal
it
looks
like
it's
continuous
could
be
positive.
D
It
can
be
negative
okay,
so
that
would
be
what
experience
might
look
to
coming
into
your
into
your
mind
at
some
past
time
scale
like
a
tenth
of
a
second
okay.
Now
now
that
you
understand
what's
going
on,
I
know
it
looks
kind
of
this.
It's
kind
of
busy
and
kind
of
difficult
to
have
intuitions
about
so
I'm
going
to
convert
it
to
colors
and
and
make
it
more
visual,
I'm
going
to
make
the
zeros
and
ones
will
be
white
and
gray,
and
these
four
valued
signals
will
be
four
different
colors.
D
These
all
are
also
bits
but
they're
action
bits
so
I'll
make
them
circle,
grays
and
whites
and
and
the
positive
things
are
going
to
be
green
bars
where
the
length
of
the
bar
indicates
how
big
the
number
is
and
the
negative
ones
will
be
pink,
and
now
they
understand
exactly
what
this
means.
I
can
take
away
the
numbers
and
make
it
less
cluttered.
D
So
here
I
have
a
look
at
the
data
that
might
be
coming
into
a
mind,
and
now
we
can
see
take
on
the
activity
of
the
mind,
which
is
to
observe
its
data
and
try
to
look
for
patterns
and
try
to
predict
it
and
understand
it
and
generally
trying
to
understand
it,
maybe
control
it.
So
if
you
look
at
this
data,
you
might
see
some
patterns
there
are
patterns
in
this
data.
D
This
is
data
that
that's,
you
know
made
up
it's
generated
by
a
program,
but
it's
not
it's
not
it's
not
random,
not
entirely
random,
and
so
I'm
just
using
it
to
help
you
think
about.
You
know
what
is
the
job
the
mind
that
looks
for
patterns
and
data?
So
if
you
looked
at
this
for
a
while,
you
might
see
some
patterns,
here's
a
here's,
something
you
might
notice
it.
D
D
That's
an
example
of
a
pattern
in
the
data.
There
are
others
this.
This
is
one.
If
there's
a
red
I'll
call
it
a
pixel,
it's
not
really
a
pixel,
but
it's
a
component
of
the
data
stream.
If
it's
red
on
one
time
step,
then
the
the
sixth
sixth
component
is
on
the
next
time.
Step
is
green,
and
that
happens
once
it
happens
twice.
We
don't
really
know
if
it's
a
pattern,
it
could
just
be
chance.
We
really
need
more
data
to
establish
that.
D
So
so,
let's
get
some
more
data.
Let's
just
carry
this
out
further
and
for
a
couple
of
columns
and
then
we
can
see
first,
here's
the
that
pattern
you
saw
before
and
yes
and
b
that
is
carried
out
throughout
the
data
many
many
times.
So
that
seems
to
be
an
effect,
and
if
you
look
for
additional
things
you
could,
if
you
studied
for
a
long
time,
you
would
find
that
that
after
this
little
pattern,
two
steps
later,
you
always
get
a
blue
signal.
D
So
this
carries
throughout
the
data
as
well.
So
this
is
this:
is
the
idea
of
a
major
activity
of
the
mind
is
looking
for
these
patterns
trying
to
predict
successfully
the
future
sensations
and
particularly
trying
to
predict
reward
and
actually
there's
one
more
pattern
in
this
data.
I
wonder
if
you
could
see
it,
you
might
be
able
to
see
it.
D
B
D
D
And
that's
that's
that's
the
way
it's
been
set
up,
and
so
now
we
can
see
ways
in
which
the
the
sensations
can
be
qualitatively
different.
This
is
what
these
tend
to
be
sticky
right.
These
last
three
of
ten
be
sticky.
These
first
four
only
have
two
values,
and
then
there
are
these
temporal
small
relationships
between
the
the
data,
so
there's
all
kinds
of
way,
even
though
it's
just
you
know
bits
or
elementary
signals,
they
can
be
qualitatively
different
from
each
other.
D
I'm
going
to
remind
those
who
who
already
know
about
it
that
you
know
the
main
thing
we
do
in
reinforcement
learning
is
we
predict
the
discounted
sum
of
the
future
reward.
We
predict
future
reward,
so
I
think
I
have
an
example
of
that
here
yeah,
so
this
is
the
return,
is
what
we're
predicting
we're,
trying
to
predict
time
steps
where
there's
a
lot
of
reward
coming
up
so
like
here
this
this,
let's
take
this
green
bar.
This
green
bar
is
big
means,
there's
a
lot
of
reward
after
it.
D
D
This
is
big
because
the
the
waiting
there's
a
lot
of
green
bars
coming
up
imminently
after
this
one,
okay
in
the
reward-
and
so
this
this
kind
of
slides
around
along
and
we
get
each
prediction
here-
we're
predicting
negative
because
there's
mostly
negative
things
coming
up
under
this
weighting,
okay,
and
so
so
here
when
we
predict
the
return,
the
return
is,
is
not
an
actual
number
that
you
see
it's
not
in
your
sensation.
This
this
column
is
unlike
the
others.
It's
not
a
sensory
signal.
It's
a
it's!
The
correct
prediction
of
the
future.
D
D
What
the
sum
of
the
in
the
next
100
time
steps
I'll,
have
you
know
more
ones
than
I
will
have
zeros
or
I
might
predict
you
know
twice
as
many
ones
as
there
are
zeros
that
that's
not
a
signal
that
we
we
measured
later,
it's
a
it's
a
function
of
the
future,
so
these
are
the
most
important
things.
They're
they're,
they're
functions
of
the
future
and
prediction
of
value
is
one
example
of
that,
and
this
is
generally
generalized
to
general
value
functions
which
predict
any
signal
not
just
reward.
D
D
So
now
now
that
we've
said
the
predictions
can
be
anything
we
don't
want
to
have
arbitrary
predictions,
because
many
predictions
are
hard
to
compute
efficiently.
We
want
things
that
could
that
we
can
predict
efficiently.
So
this
is
becomes
the
bread
butter,
as
I
see
it,
of
an
experiential
approach
to
ai
okay.
Now
that
was
a
bit
of
this.
That
ends
my
interlude.
So
now,
maybe
we
know
more
concretely
what
experience
means
and
what
it's
like
to
deal
with
it.
D
Let's,
let's
go
on
to
deal
with
state,
okay,
so
as
as,
in
all
the
other
cases
conventionally
in
ai,
we
start
out
talking
about
objective
state
external
world
state.
This
block
is
on
the
other
block.
John
loves
mary.
These
are
all
statements
about
absolutely
external
things,
and
even
in
more
modern
things
like
probabilistic
graphical
models,
the
state
is
still
probably
distribution
of
the
world
states.
Here's
a
probabilistic
graphical
model.
You
know
the
states
are
like
it's
cloudy
outside
it's
raining,
the
grass
is
wet.
These
are
states
of
the
world.
D
The
sprinkler
is
on
and
another
canonical
case
is
palmdp's
partially
observable
marked
decision
processes,
in
which
case
the
state
is
a
probability
distribution
over
underlying
world
states.
It's
explicitly
in
terms
of
underlying
world
states,
and
so
I'm
going
to
call
these
things.
Objective.
State
representations,
they're
all
separated
from
the
individual
they're
about
the
world
and
that's
the
way
ai
has
started
so
the
alternative
called
experiential
state.
Here
we
define
the
state
of
the
world
entirely
in
terms
of
experience.
D
D
Summary
of
past
experience
it's
good
for
predicting
future,
so
there's
no
mention
of
external
entities
out
there
in
the
world.
There's
just
experience,
experience
summarizing
experience
to
predict
experience,
and
this
is
the
way
we
talk
exponentially
about
state,
and
this
is
this
is
done
in
some
of
the
sum
of
the
fields
like
the
simplest
cases
where
you
just
build
it
in
in
atari.
We
remember
the
last
four
frames
of
video
and
we
and
that
becomes
the
state
or
maybe
you
also
need
to
know
the
recent
actions
when
they
do
compression.
D
You
have
to
summarize
something
about
what
you've
seen
to
help
you
predict
and
compress
the
data
as
you
move
along
and
lstms.
If
you
know
what
that
that
means
are
the
way
in
deep
learning
where
they
they
summarize
the
past,
so
that
something
doesn't
have
to
be
in
the
current
observation
and
all
these
methods
have
and
they
learn
and
discover
exponential
state
so
exponential
state.
If
you
think
about
it,
it
should
be
done
recursively.
D
If
you
just
take
it
naively
like
summarizing
the
these
are
supposed
to
mirror
what
we
saw
earlier,
there's
a
sensation,
action
sensation,
accidentation
action.
This
is
all
the
past
and
you
might
want
to
summarize
all
the
past
in
the
current
state
before
you
pick
your
action,
okay,
but
this
is
this
would
be
a
poor
way
to
do
it
because
think
what
it
means
as
you,
as
you
really
need
to
stay
at
all
these
earlier
times
and
you
and
it's
just
awkward
to
to
bring
in
all
the
past
at
each
point.
D
This
is
a
better
approach,
because
the
last
stage
should
summarize
everything
before
so
like
this
state
really
should
only
need
to
know
the
previous
state,
because
the
previous
state
should
summarize
everything
before
that,
and
it
will
use
the
state
before
that
and
so
on.
So
you
have
this
sort
of
lattice
structure
and
that's
a
useful
where
you're
doing
it
recursively,
and
that
leads
to
a
picture
which
will,
I
think,
make
sense
that
you
have
some
sort
of
perceptual
process.
D
That's
producing
this
state
and
it
has
to
use
the
most
recent
state
as
well
as
the
last
action.
The
last
sensation-
and
then
this
experiential
stage,
is
what
you
that
would
use
to
as
input
to
your
policy
to
pick
an
action,
and
maybe
you'd
use
it
as
input
to
your
predictions
and
right
here.
I
think
I
want
to
just
stick
with
this
for
a
moment
and
and
think
about
what
the
whole
picture
would
look
like.
This
is
just
perception
like
what
we're
talking
about
so
far.
It's
like
the
the
axis
of
of
the
agent.
D
D
So
this
is
the
full
picture.
The
full
basic
picture
is
that
we
have
the
central
axis
and
then
we
also
have
predictions
that
we
use
by
learning
processes
to
adjust
the
policy,
and
then
we
also
have
this
model
of
the
world.
Really
it's
a
transition
model,
because
so
much
of
the
model
is
also
here,
but
the
transition
part
is,
after
the
state
and
its
state
to
state
transitions,
and
that
transition
model
is
also
used
through
processes
to
adjust
the
policy
in
those
processes.
D
We
call
planning
or
reasoning
those
that's
how
we
bring
together
all
four
of
the
steps
agent
head,
reward
state
and
knowledge.
I
really
wanna
yeah.
So
let's
go
on
then
to
knowledge.
The
last
one,
the
biggest
one
knowledge
where
we
say
that
to
know
is
to
predict
experience
the
function
or
functions
of
experience.
D
D
The
eiffel
towers
in
paris,
birds
have
wings.
Oregon
is
north
of
california.
The
car
is
10
meters
ahead.
Fire
engines
are
red.
These
things
are
seem
to
be
about
the
external
world.
This
is
our
normal
way
of
talking
about
the
world,
but
there's
other
knowledge
which
is
more
experiential.
It
feels
more
about
experience.
You
know
it's
a
long
walk
to
the
city
center.
I
can.
I
can
lift
200
pounds
it's
cold
outside
today.
D
D
And
now,
if
we
look
at
the
the
history
of
ai,
the
classical
ai
systems,
which
are
called
good
old-fashioned,
ai
systems,
that's
what
gofi
means
good
old-fashioned,
ai.
It's
been
embraced
by
the
good
old-fashioned,
ai
types
as
as
as
yeah.
It's
not
really
a
bad
thing.
They
accept
that,
but
those
original
systems-
you
know
they
didn't
have
experience.
They
couldn't
really
talk
about
prediction
and
much.
Modern
ai
still
treats
knowledge
as
like
a
database.
So
like
this
is
what
a
database
would
look
like.
D
You
know
the
president
of
the
us
equal
to
joe
biden.
Capital
of
france
is
paris
not
predictive
at
all,
and
then
we
go
back
to
those
problems
to
graphical
models.
Those
are
not
predictive
either
because
it's
all
about
simultaneous
events.
You
know
they'd
have
things
like
well.
If
it's
cloudy,
then
there's
a
higher
probability,
that's
going
to
rain,
but
it's
not
it's
viewed
as
statements
about
simultaneous
events.
D
D
World
knowledge
proper
can
be
divided
into
two
types:
knowledge
about
the
state
which
we've
already
talked
about
step.
Three,
now
we're
talking
about
knowledge
about
state
transitions,
that
kind
of
predictive
knowledge
of
the
world
so
and
then
a
state
to
state
predictive
model.
It
doesn't
have
to
be
low
level.
It
doesn't
have
to
be
like
differential
equations.
It
doesn't
have
to
be
like
a
markov
decision
process.
The
model
can
be
abstracted
state
if
it's
using
experiential
state.
Well,
I
actually
guess
generally
it
can
be
abstracted
state
and
it
can.
D
D
That
will
take
many
many
tenths
of
a
second
time
steps,
and
so
there's
a
well-developed
way
of
dealing
with
this,
and
so,
if
we've
conditioned
the
prediction
on
the
entire
waves
behaving,
these
ways
of
behaving
are
called
options
and
an
option
is
just
a
policy
the
way
of
behaving
plus
a
way
for
it
to
terminate,
and
then
you
can
make
a
transition
model
for
the
option.
It's
well
understood,
we
know
how
to
learn
them
and
then
the
question
is.
D
D
So
that's
the
fourth
step
and
last
step
and
remind
you
that
the
the
the
pattern
is
that
for
each
step,
ai
has
started
out
by
doing
non-experiential
things
and
has
gradually
moved
towards
an
experiential
point
of
view
in
order
to
be
grounded
connected
to
to
data
and
to
be
learnable
and
to
scale
with
computer
computer
power.
So
as
you
get
more
computer
power,
you
want
you
want
your
systems
become
more
powerful
and
that's
harder
to
do
if
you're
reliant
on
human
understanding
than
if
you
can
process
the
data.
D
D
Yeah
and
I
guess
I
took
out
the
slides,
in
the
sake
of
time,
to
reminding
us
why
why
we
think
in
experiential
terms?
Let
me
just
point
out
that
why
we
like
to
think
in
both
terms
first,
why
we
like
objective
terms,
objective
terms,
I
think,
are
the
right
terms
for
almost
everything
that
humans
do
like
humans
talk
to
each
other
humans,
humans
work
together.
D
We
both
understand
the
words,
and
so
we
should
talk
about
the
external
world,
but
the
exception,
maybe
is
understanding
the
mind.
The
exception
is
building
robots,
the
robots
don't
have
that
understanding
a
priori.
We
have
to,
they
have
to
develop
it
and
and
relate
it
to
their
sensations
and
their
actions.
So,
for
that
we
need.
We
need
to
talk
about
experience
now
these
trends,
which
we
see
historically,
you
know,
there's
no
reason
why,
just
because
something's
happened
before
it'll
happen,
more
and
it'll
happen.
C
D
But
I
think
it
could
they
could
go
much
further.
There
are
research
opportunities
and,
ultimately,
perhaps
the
story
of
intelligence
may
be
told
sensory
motor
terms.
Now
I
I'm
going
to
show
one
more.
Sometimes
I
try
to
make
my
message
really
simple,
maybe
maybe
too
simple.
But
if
I
try
to
really
express
this
in
a
single
slogan,
it
would
look
something
like
this
data
drives.
Ai
and
experience
is
the
ultimate
data.
A
A
C
E
G
G
And
so
I
have
we
joined
the
followers
team.
A
couple
of
years
ago
and
and
recently
I
have
been
starting
a
new
research
direction
in
this
group
which
attempts
at
integrating
our
original
developmental
perspective,
with
an
evolutionary
and
ecological
perspective
and
opponent
school
acquisition
in
humans
and
machines.
G
G
Typically
innate
behaviors
genetically
passed
from
generation
to
generation
can
do
the
job.
So
the
interesting
case
when
it
comes
to
learning
seems
to
be
when
between
generation
creativity
is
low
and
within
generation.
Predictability
is
high,
and
so
this
simple
framework
raises
a
first
interesting
question,
which
is:
what
are
the?
G
Then,
from
a
developmental
perspective,
it
has
been
proposed
that
biological,
motor
and
cognitive
development
can
be
considered
as
a
complex
dynamical
system
and
a
typical
example
comes
from
the
wellington,
epigenetic
landscape
metaphor,
which
is
illustrated
here
in
the
figure
so
on
this
video.
Basically,
we
can
consider
the
shape
of
the
landscape
as
a
set
of
environmental
and
organismal
influences,
and
the
agent
development
and
the
motion
of
this
metaphoric
ball
here
on
the
landscape
reflects
the
developing
phenotype
that
changes
through
time
as
the
ball
is
rolling.
G
Which
is
induced
by
a
church
landscape,
okay,
so
this
leaves
us
with
two
main
questions
which,
from
the
outline
of
the
of
the
talk,
and
so
I
will
start
with
an
evolutionary
ecological
perspective
where
I
will
present
a
preliminary
proposition
of
a
framework
which
aims
at
understanding.
What
are
the
processes
generating
experience
and
predictability
and
how
these
processes
interact
at
multiple
specialty
scales.
G
G
G
And
this
is
based
on
recent
hypotheses
in
bilochae
mythology
and
human
behavioral
ecology,
which
actually
point
at
a
very
particular
climate
dynamics
in
the
red
valley
at
east
africa,
approximately
7
million
years
ago,
with
complex
patterns
in
climate
viability.
That
could
have
had
a
strong
influence
on
early
human
evolution
and,
in
particular,
regarding
our
ability
to
quickly
generalize
and
to
extend
our
behavioral
repertoire
to
changing
environments,
and
so,
whereas
there
is
an
analogous
tendency
in
ai
towards
increased
complexity
in
submission
developments.
G
So,
first
it
feeds
back
to
environmental
complexity
through
the
modification
of
resource
availability
and
relation
pressures
and
in
human
behavioral
ecology.
This
positive
feedback
loop
is
often
referred
as
an
evolutionary
arms
race,
while
in
ai
it
has
sometimes
been
called
autocurricular.
This
is
when
an
agent
adaptability
modifies
the
fitness
landscape
of
the
environment.
G
G
F
G
So,
let's
imagine
you
are
living
in
a
relatively
unpredictable
development.
For
example,
you
are
an
early
homo
living
in
east.
Africa
should
be
a
few
million
years
ago,
where
chaotic
climate
dynamics
modulate
resource
sociability
and
exposition
to
creators
across
generations,
or
maybe
easier
to
imagine.
You
are
just
a
young
baby
in
a
room
full
of
toys
and
other
objects
how
to
efficiently
collect
informative
training
data
to
acquire
interesting
skills
in
just
complex,
unknown
arguments.
G
So
this
is
a
video
of
a
baby
which
is
alone
in
a
room.
It's
not
hungry,
not
thirsty.
There
are
no
other
humans
interacting
here
and
so
from
a
racial
point
of
view.
One
may
think
that
if
the
evolutionary
fitness
of
this
guy
is
just
to
grow
up
and
to
reproduce,
then
he
should
probably
keep
safe.
He
should
avoid
putting
the
fingers
in
the
blood
which
basically
should
do
nothing
and
just
wait
his
parents
to
provide
him
food
when
he
will
become
hungry
and
yet,
as
we
see
here,
is
not
at
all
what
he
does.
G
He
goes
for
the
fingers
in
the
plug.
He
explores
his
body
he
tries
to
grasp.
He
throws
all
object
around
and
he's
not
doing
this
because
they
are
externally
imposed
tasks
or
rewards,
but
rather
it
is
driven
by
different
forms
of
what
psychologists
call
motivation
and
what
we
may
call
curiosity
in
everyday
language.
G
G
Third,
the
agents
roll
out
an
action
policy
conditioned
by
the
sample
goal,
for
example,
moving
its
body
to
this
body
to
displace
the
objects
on
the
table
and,
fourth,
it
observes
it
observes
the
outcome
and
updating
certain
internal
models.
Accordingly,
in
particular,
refining
the
learned
action
policies
to
better
achieve
those
targeted
goals,
and
then
these
four
steps
will
beat
indefinitely.
G
They
have
been
applied
to
supervised
learning
problems,
for
example,
learning
forward
or
inverse
models
in
robotics
also
to
model
track
development,
for
example,
vocal
control
and,
more
recently
to
gold
condition,
encounter
learning
agents
in
ai,
and
so
in
the
following.
I
will
focus
on
two
computational
experiments,
which
both
rely
on
this
framework.
G
So
learning
to
control
such
an
articulatory
synthesizer
or
more
generally,
a
vocal
tract,
is
actually
a
classical
robotic
control
problem.
How
to
reach
certain
auditory
targets
through
the
execution
of
certain
documents,
and
this
one
recommends
here
are
articulatory
trajectories,
and
this
can
be
achieved
by
learning
an
inverse
model
which
maps
targeting
effects
in
the
auditory
trajectory
space,
which
is
hero6d
space.
Two
motor
commands
in
the
arctic
in
an
articulatory
trajectory
space,
which
is
here
18d
space,
but
to
learn
an
adequate
inverse
model.
G
So
instead
a
solution
to
efficiently
explore
such
systems
is
the
one
proposed
by
the
igep
framework
I
previously
described
and
in
this
particular
experiment.
We
consider
that
the
agent
is
sampling,
auditory
goals.
Here.
Sorry
and
that's
the
goal
condition
policy
is
implemented
by
the
invest
model,
which
is
iteratively,
learned
and
refined
through
experience.
G
G
G
So
what
does
this
mean?
Typically,
for
example,
in
the
previous
experiment,
I
have
shown
the
space
in
which
the
agent
can
sample
the
goals
was
fixed
and
the
agent
could
only
sample
goals
within
that
predefined
space,
but
to
power
a
creative
exploration,
however,
agents
will
need
to
generate
novel,
creative
and
abstract
goals,
goals
that
are
out
of
distribution
of
the
things
already
seen,
and
in
this
process
children
use
language
as
a
creative
tool.
G
Is
actually
going
grounded
in
key
ideas
from
the
development
of
psychology
and
linguistic
literature,
so
psja
first
discovered
that
children
used
egocentric
speech
to
narrate
their
ongoing
activities
later.
The
gods
key
showed
that
they
used
it
to
generate
goals
and
plan
to
solve
them,
and
language
is
indeed
compositional
and
creative
by
nature,
as
chumsky
illustrated
with
this
extravagant
construction.
Colorless
green
ideas,
sleep
seriously,
and
so
the
compositionality
of
language
can
be
used
to
generate
a
lot
of
distribution
goals.
That
is
to
imagine
new
goals
from
not
one
from
known
one.
G
G
But,
most
importantly,
the
agent
can
now
invent
new
out
of
distribution
goals
by
overcoming
recombining
words
to
perform
new
sentences.
Modeling,
the
meaning
using
the
learn,
goal,
achievement
function
and
using
this
internal
reward
function
to
train
the
policy
to
achieve
this
out
of
distribution
growth.
G
G
G
To
conclude,
the
need
show
how
language
once
invented
provides
a
powerful
cognitive
tool
for
imagining
out
of
distribution
goals
and
enabling
open-ended
creative
innovation.
So
in
a
sense
imagine
provides
a
some
sort
of
computational
evidence
and
a
statement
from
daniel
dorr,
which
summarizes
where
the
main
idea.
First,
we
invented
language,
then
language
changes,
and
so
it
changes
in
many
aspects.
G
This
challenge,
but
then,
as
we
have
seen
in
the
second
experiment,
once
language
has
emerged
one
way
or
another,
it
can
then
have
strong
feedback
effects
and
an
adjacent
cognitive
abilities
and
in
particular
it
enables
open-ended
creative
innovation,
and
so
this
in
turn
opens
the
way
towards
the
formation
of
a
cryptogram
repertoire,
creating
a
positive
feedback,
loop
to
environmental
viability,
potentially
driving
the
ever-expanding
complexity
of
human
skill
acquisition,
and
so
this
will
be.
My
last
word
like
thank
you
for
your
attention
and
to
all
my
collaborators
and
these
topics.
Thank
you.
C
A
A
Vivian
recently
completed
her
phd
set,
sent
recently
completed
her
phd
studies
as
a
student
at
university
of
osnabruck
in
germany
on
computational
cognition
and
I'm
very
happy
to
say
she
decided
to
join
nementa
as
a
research
scientist.
So
her
research
focuses
on
working
on
neural
networks
to
learn
meaningful
world
representations
in
3d
virtual
environments,
and
she
has
written
extensively
about
using
the
brain
to
inform
ai
algorithms
to
learn
as
an
active
sensory
motor
process.
So
thank
you
vivian
for
presenting
today
and
whenever
you're
ready,
take
it
away.
E
E
Okay,
just
to
motivate
this
a
bit
current
ai
systems
have
a
lot
of
flaws
and
points
where
you
notice
that
they
are
not
really
doing
the
same
thing
that
our
brain
is
doing.
I'm
sure
this
picture
on
the
left
is
familiar
to
people
who
are
in
the
ai
or
computer
vision
field.
So,
for
example,
if
you
take
an
image
and
add
some
very
minor
noise
to
it
for
a
human,
it
still
looks
exactly
the
same
as
before,
but
for
an
artificial
neural
network
that
was
trained
to
recognize
objects
and
images.
E
E
So
here
are
the
stages
of
development
he
proposes
and
I'm
not
going
to
go
into
the
details,
but
you
can
see
that
the
human
mind
develops
over
many
many
years
and
gradually
so
you
start
out
simply
interacting
with
the
world
in
the
beginning:
very
weak
supervision,
learning,
sensory
motor
contingencies
and
so
on.
And
then
you
start
attaching
words
to
objects.
You
learn
language.
E
Inductive
and
deductive
reasoning
later
on
abstract
thinking,
thinking
about
philosophy,
math
and
so
on,
but
it
gradually
builds
up
over
time
and
compared
to
how
humans
learn.
There
is
a
body
there
is
active
interaction
with
the
world
and
most
of
the
learning
actually
happens
without
supervision
or
self-supervised.
E
And
those
are
the
factors
that
I
will
be
looking
at
and
this
quote
from
alan
turing.
Instead
of
trying
to
produce
a
program
to
simulate
the
adult
mind,
why
not
rather
try
to
produce
one
which
simulate
the
child's?
If
this
were
then
subjected
to
an
appropriate
course
of
education,
one
would
obtain
the
adult
brain.
E
So
I'm
approaching
it
from
this
angle,
basically
taking
an
artificial
agent,
letting
the
agent
simply
interact
with
the
world
and
learn
a
model
of
the
world
through
interaction
and
then
seeing
what
is
actually
being
learned.
And
what
can
we
do
with
this,
and
I
work
with
this
3d
maze
environment.
It's
from
the
obstacle
tower
challenge,
so
you
have
this
agent
that
walks.
Through
this
maze
there
are
multiple
rooms
and
levels
and
doors
different
types
of
doors.
E
E
All
right,
so
here's
the
conditions
we
have
one
embodied
agent
with
an
active
loop
between
action
and
perception,
meaning
the
agent
receives
an
image
of
the
world,
how
he
sees
it
right
now.
This
gets
processed
by
a
neural
network
to
produce
an
action,
and
this
action
then
leads
to
the
next
perception
and
the
two
control
conditions
that
are
not
embodied.
E
All
right,
here's
what
the
sense
remote
learning
agent
actually
learned.
So
this
are
the
activations
inside
the
neural
network,
corresponding
to
different
images,
and
they
are
quite
high
dimensional.
So
here
they
are
projected
into
two-dimensional
space
using
tsne,
and
you
can
see
even
in
two-dimensional
space.
E
E
And
another
interesting
thing
is
that
the
activations
in
the
network
of
the
embodied
agent
are
much
much
much
sparser
than
the
activations
in
the
autoencoder
or
the
classifier,
which
means
it
only
uses.
A
few
select
neurons
to
represent
each
input
image
just
to
quantify
it
a
bit.
But
I'm
not
going
to
go
into
detail
for
time.
E
There
are
only
about
less
than
five
percent
of
neurons
used
to
encode,
an
image
which
has
over
84
000
pixels,
pixel
values
and
there's
still
a
lot
of
the
neurons
are
used.
So
it's
not
that
it's
only
the
same
five
percent
all
the
time,
but
a
variety
of
different
sparse
patterns
is
being
used
to
encode
the
input
and
all
three
conditions.
E
So,
in
summary,
the
representations
learned
through
weekly,
supervised
or
self-supervised
exploration
are
structured
and
meaningful.
They
encode
action-oriented
information
in
various
activation
patterns,
and
then
coatings
learned
through
interaction
differ
significantly
from
the
encodings
learned
without
interaction.
E
E
And
now
the
idea
is
letting
the
agent
learn
purely
through
curious
interaction
with
the
world.
There
are
no
external
rewards
just
trying
to
explore
a
lot
and
interacting
with
the
world
and
then
taking
the
learned
representations
of
the
world
and
giving
a
few
labeled
examples
and
see
how
this
works
out
same
three
conditions
again
and
then
the
idea
is
to
take
an
input
image,
see
what
activations
are
active
in
the
network
and
detect
an
object
in
this
image.
E
If
we
have
a
new
image,
we
just
put
this
image
into
the
brain
of
the
agent
we
look
at
which
neurons
are
active
and
compare
it
to
the
learned
concept
of
the
yellow
door,
and
if
enough,
if
there's
enough
overlap
in
these
activities,
if
there's
enough
evidence,
we
say:
okay,
there's
a
level
or
you
can
do
this
for
all
of
the
concepts
that
have
been
learned
and
the
nice
thing
about
this.
Is
it
works?
E
E
E
E
Yeah
and
another
interesting
thing
is,
you
can
in
reverse
also
use
this
method
to
look
at
what
the
agent
actually
learned
during
the
unsupervised
exploration,
for
instance.
Here
you
can
see
it
struggles
with
the
key
door
and
the
key,
and
if
you
look
at
the
performance
of
the
agent
you
can
see,
it
hasn't
really
learned
yet
how
to
pick
up
the
key
and
unlock
the
door,
and
this
is
reflected
in
how
well
you
can
actually
extract
these
concepts
from
the
representations
that
were
learned.
E
E
So,
in
summary,
the
representations
learned
encode
information
about
concepts
which
were
never
explicitly
taught
so
the
agent
learned
just
through
curiosity,
it
was
never
told
this
is
a
door.
This
is
a
time
orb.
This
is
a
key
or
anything
like
that.
The
concepts
can
be
learned
with
very
few
examples.
E
It
has
no
problem
with
catastrophic
forgetting
because,
if
you
add
another
label,
you
don't
retrain,
actually
the
whole
network.
The
network
has
already
been
trained
through
sensory
motor
interaction
and
therefore
there
is
also
no
need
for
a
big
label.
Data
set
and
overall
learning,
more
natural
seems
to
lead
to
some
abilities
that
we
also
find
in
humans,
such
as
fast.
E
E
What
a
model
learns
is
a
reflection
of
its
environment
and
task.
So
here
in
these
experiments,
all
the
models
and
learning
methods
were
exactly
the
same:
deep
neural
networks,
same
structure,
same
back,
propagation
and
so
on.
The
difference
was
really
in
the
in
the
task
and
the
way
learning
was
happening
and,
third,
if
we
want
to
understand
cognition
and
our
brains,
we
need
to
understand
the
tasks
that
need
to
be
solved,
so
the
cognitive
aspect
of
it
computational
problems
that
need
to
be
solved
and
for
artificial
models.
E
A
All
right,
let's
move
on
to
the
discussion
and
q
a
then,
since
we
have
30
minutes
left.
So
thank
you,
everyone
for
still
being
here
and
for
our
attendees.
If
you
have
any
questions,
feel
free
to
just
add
it
in
the
q
a
function
below,
and
you
can
comment
and
also
vote
on
the
questions
you'd
like
asked
so
yeah.
Let
me
first
open
it
to
the
panelists
first
and
do
you
guys
have
any
questions.
B
Yeah
I
have
a,
I
can
ask
sort
of
a
high
level
question
for
to
all
three
speakers.
As
I
was
listening
to
the
talks,
this
is
sort
of
motivated
by
really
clement's
video
of
the
baby
exploring
the
world,
and
this
has
to
do
with
kind
of
the
nature
of
interesting
exploration
and
vivian.
You
know
you
showed
how
important
that
sort
of
sensory
motor
stage
of
prj's
development
is
to
to
you
know,
being
able
to
do
quick.
You
know
one
shot,
learning
or
quit
few
shot
learning
later
and
so
on.
B
So
I
guess
the
sort
of
high
level
question.
Maybe
it's
not
clear
that
the
role
that
reward
plays
in
that
initial
stage
and
it
you
know
if
you
think
about
the
baby,
there's
no
clear
external
reward
from
the
world
and
in
with
babies
it
may
be
a
really
long
period,
maybe
several
years
potentially
where
the
nature
of
external
reward
is
really
unclear,
and
it's
really
about
intrinsic
exploration
and
learning
about
the
structure
of
the
world
so
that
you
can
later
use
it.
B
And
so
it's
almost
you
know
one
extreme
view
point
maybe
having
no
reward
might
be
critical,
and
so
I'm
being
intentionally
controversial
here
but
wondering
rich
and
then
the
others.
What
you
feel
you
know
is
to
what:
how
does
the
reward
framework
fit
into
that
stage?
And
what
do
you
think
about
you
know
that
stage
in
general.
D
Well,
let
me
be
clear
that
I
really
agree
that
creation
of
play
play
is
really
super.
Important
and
play
is
creation
of
sub
goals,
and
you
know
goal-oriented
behavior,
that
is
oriented
towards
goals
that
are
not
the
not
reward,
and
this
is
really
really
important.
This
is
the
perhaps
the
most
important
way
we
formulate
structures
in
our
mind
is
we
propose
sub
problems
and
then
we
work
on
them,
and
then
we
learn
skills
to
achieve
those
sub
problems
and.
B
D
Yeah,
so
I
guess
this
is
something
that
that
jeff
has
has
thought
about
as
well.
You
know
what
is
the,
what
is
the
the
the
interplay
between
the
understanding
the
world
and
creating
subproblems
that
are
that
are
different
than
than
reward,
and
then
there
is,
there
is
the
goal.
There
is
the
thing
that
you're
trying
to
achieve,
and
I
do
feel
that
the
the
sub
problems
are
some
subsidiary.
C
It
seems
that
this
is
a
personal
observation
that
we
you
know.
I
get
a
lot
of
satisfaction
out
of
solving
problems
that
I
pose
to
myself.
There's
a
there's,
definitely
a
reward
that
goes
on
it's
the
aha
moment.
It's
the
damn.
I
figured
that
out
type
of
thing.
You
know
whether
you're
solving
a
crossword
puzzle,
while
you're
solving
a
brain
problem.
C
Look,
maybe
it's
not
maybe
it's
the
same
thing
I
don't
know,
but
but
there
seems
to
be
this
sort
of
tremendous
reward
that
some
of
us
get
at
least
from
just
whatever
problem.
Even
a
made-up
problem
has
no
real
life
world
consequences,
just
solving
it.
It
seems
to
be
a
very
pleasurable
thing
to
do.
I
don't
know
how
anyone
feels
about
that
that
maybe
what
the
child
is
doing
in
those.
G
A
reward
in
this
case
of
like
self-goal
generation,
as
you
said,
and
also
in
the
like,
the
very
young
infant
playing
in
the
room,
and
actually
these
three
words
are
likely
the
same
nature
that
he
was
we
what
we
call
external
rewards
in
enforcement
learning
like
getting
food
or
whatever,
and
in
any
case
there
is
actually
no
reward
in
the
environment.
The
rewards
are
internal,
they
are
in
the
brains,
so
environment
don't
doesn't
contain
any
reward
in
itself,
and
so
this
is
maybe
one
of
of
let's
say
assumption
of
different
point.
G
C
Yeah,
I
think,
there's
a
clear
advantage
to
having
an
organism
want
to
discover
the
structure
of
the
world
and
figure
it
out
right.
So
I
guess
I
was
a
little
curious
for
rich.
There
is
that
does
that
fit
in
the
reward?
Is
enough,
or
do
you
view
that
as
a
different
type
of
reward?
I
I'm
a
little
confused
by
that,
but
you
know
is
that
is
that,
like
a
different
type
of
reward
than
you
were
talking
about,
or
do
you
say?
No,
that's
just
the
rewards.
D
Yeah
well,
so
I
always
have
a
little
trouble
seeing
why
there's
confusion
here?
We
we
know
we
we,
the
observation
is
animals
are
goal
directed,
but
they
also
if
their
if
their
goals
are
not
threatened.
They
like
to
learn
about
the
world,
and
you
know,
they're,
undoubtedly
reusing
the
mechanisms
to
learn
about
the
world
that
they
also
learn
to
use,
to
survive
or
to
get
their
ultimate
goal
and.
D
The
mystery
is
that
you
know
why
do
we
play
because
by
definition,
play
is
is
not
if,
if
if
by
play,
we
mean
things
that
we
do
like
we
play
soccer
or
we
play
chess
or
you
play
part
cheesy.
We
learn
and
we
learn
to
do
math
things
even
if
they're
not
relevant,
we
learn
all
these
things
and
we
enjoy
them.
And
it's
it's
it's
it's.
D
And,
and
and
yet
we
know
that
we
do
do
basic
research,
we
justify
basic
research
because
we
think
that
oh,
it
may
not
be
obvious,
but
someday
it
will
be
useful.
And
similarly,
you
know
when
we
see
a
child
play.
We
think
well
yeah
he's
learning
how
to
do
the
blocks
and
he's
learning
critical,
hacks
and
someday.
D
That
will
be
useful
and
then
maybe
evolution
has
built
in
something
that
makes
him
realize
that
and
realize
that
it's
that
it's
it's
pleasurable
and
you
know
evolution
has
encoded
the
the
idea
that
there's
this
thing,
that
that
predicts
future
fitness,
their
future
ability.
I
think,
that's
all.
That's
all
fine.
C
Maybe,
as
an
adult,
sometimes
this
mechanism
has
continued
on
beyond
its
utility
potential
utility.
You
know
I
spend
if
I
spend
hours
doing,
crossfit
puzzles,
I'm
not
sure
I'm
learning
anything
new,
but
maybe
I
am
assuming
I
had
a
question
for
clement.
If
that's,
if
it's
all
right,
just
to
throw
that
out
right
now,
you
talked
about
language
as
being
used
for
out
of
distribution.
Setting
out
of
distribution
goals
is
language
necessary.
For
that
I
mean.
Could
I
do
that
without
language?
C
F
G
G
Computer
experiments
I
have
shown
would
probably
work
as
well
with
any
compositional
mechanism
that
allowed
to
combine
different
goals
together,
but
then
what
the
particular
function
of
language
here
is
also
to
like
to
be
able
to
learn
the
relationship
between
like
experience
in
the
environment
and
goals
and
language
through
interaction
with
an
ideal
social
player
yeah,
and
so
then
this
can
actually
bootstrap.
Basically,
this
compositional
aspect
that
then
leads
to
compositional
good
imagination,
yeah.
C
But
it's
not
it's
essentially
you're
not
arguing
it's
absolutely
requirement
for
an
individual
to
have
language
to
do
those
things
I
mean
there's
a
lot
of
smart
mammals
and
I
I
would
like
to
believe
that
they're
also
able
to
think
about
things
that
you
know
don't
exist
and
try
to
imagine
and
from
a
neural
mechanism
point
of
view.
I
prefer
that
as
well,
so
it
sounds
like
you're
open
to
that
idea.
Then.
G
E
I
had
more
of
some
interesting
evidence
of
how
strong
curio
the
curious
drive
is
in
humans.
I
recently
read
a
paper
that
was,
they
gave
some
people
pens
and
told
them.
Some
of
them
are
electrified
if
you
click
them
and
the
people
had
no
incentive
to
actually
click
them,
but
they
would
still
do
it
to
figure
out
which
ones
are
electrified
and
also
with
rats
already
in
the
1950s.
I
think
they
showed
that
rats
would
walk
over
an
electrified
ground
in
order
to
explore
new
parts
of
the
cage.
E
B
C
E
F
B
D
D
But
but
what
all
policy
means
is
that
you
can
learn
about
skill
even
when
you're
not
working
on
it
like
you
can
learn
how
the
world
works,
even
if
you're,
not
even
if
you're
trying
to
I
don't
know,
make
tea
and
you
still
learn
how
the
world
works,
because
you
see
the
way
that
it
works
and
so
there's
a
lot.
There's
lots
of
things
to
be
learned
in
parallel.
You
don't
have
to
commit.
D
You
can
learn
about
many
ways
because,
because
what
you're
actually
doing
will
overlap
with
many
different
things,
and
so
you
so
this
is
off
policy
learning
and
and
there's
a
whole
whole
field
of
it
really
about
how
to
learn
off
policy.
Is
it
possible
in
all
policy
what
things
can
be
when
off
policy?
What
things
can't
can
you
learn
all
policy
about
value
functions?
Can
you
learn
off
policy
about
about
new
policies,
yeah,
so
off
policy
learning
is
like
a
big
thing
that
I
think,
is
part
of
learning
knowledge.
D
Another
thing,
let
me
let
me
talk
about
something
new
that
I'm
working
on.
I
don't
know.
I
think
I
think
it
would
be
fun
and
I'll
teach
you
a
new
fun
word.
The
word
is
stomp
stomp
s-t-o-m-p
and
it
stands
for
the
progression.
So
it's
it's.
It's
progression
for
creating
a
cognitive
structure
about
about
things
you
might
do
so.
Stop
means
you
start
with
a
subtask
and
you
convert
it
to
an
option
which
solves
the
subtask
achieves
the
subtask.
Once
you
have
the
option
s
t
subtask,
o
stow
sto,
the
o.
D
D
C
I
think
in
some
sense
the
talks
here
today
and
then
this
whole
theme
today
is
like
that,
because
everyone
knows
about
sensory
motor,
it's
so
obvious,
you
know,
but
that,
as
you
pointed
out
eloquently
rich-
and
I
just
mentioned
too-
that
it
hasn't
really
been
part
of
it's
sort
of
been
an
elephant
in
the
room.
In
some
sense,
no
one
talks
about
it
for
so
long-
and
you
know,
I'm
always
wondering
you
know
is
is:
is
it
going
to
be
like
a
big
reckoning,
one
day
where
we
say?
C
Oh,
my
god,
we've
been
doing
this
all
wrong
or
just
going
to
be
sort
of
gradually
sliding
into
you
know
yeah.
We
just
we
knew
that
all
along.
We
were
getting
towards
that
central
motor
goal
all
along.
You
know
it's
not
clear
to
me
actually,
because
there's
so
much
in
the
machine
learning
our
world
today
is
very
oblivious
to
the
whole
idea
of
essentially
modern
learning,
but,
as
you
point
out,
there's
a
lot
of
people
picking
it
up
too.
So.
D
Well,
just
to
focus
on
that
history,
you
know
like
we
could
see
similar
transformations
like
the
old
ai
was
all
about
expert
systems
and
symbols,
and
now
the
modern
ai
is
all
about
about
deep
learning
and
and
neuron.
Like
things
and
you
know,
that's
happened
fairly
gracefully
I
mean
there
are
still
old
guys
like
myself.
D
G
This
gap
you
are
mentioning
is
basically
about
how
to
bridge
this,
like
data
centric
approach
to
ai,
with,
like
the
oldest
symbolic
approach
to
it
right.
This
is
what
this
guy
precise,
basically,
and
so,
if
we
yeah,
the
challenge
is
basically
to
be
able
to
to
create
these
symbols
from
the
center
of
the
stream.
D
F
C
That's
it's
pretty
close
to
the
research
we're
doing
at
nomento
right
now,
so
I
think
you're
right
about
it.
It's
there's
a
there's,
a
middle
ground
where
you
assume
there's
some
structure
to
the
world
that
could
be
represented
by
graphs
in
some
sense
and
but
but
it
has
to
be
learned,
experientially
and
so
there's
a
there's,
a
foundation
that
assumes
that
there's
a
structure,
graphical
structure
but
but
the
details
are
not
known
and
and
that's
that
seems
to
be
the
middle
ground
and
we're
shooting
for
that
here.
B
Yeah
there's
a
question
in
the
q
a
that
sort
of
relates
to
this.
Let
me
read
it
out
and
I'll
explain
why
it's
curious.
You
know,
since
machines,
unlike
the
brain,
can
be
connected
to
the
cloud
internet
sensors
in
spatially
distant
locations.
Could
this
accelerate
sensory
motor
learning
due
to
more
availability
of
data?
B
C
B
C
F
E
But
then,
once
you
get
to
points
where
action
is
actually
very
costly
or
difficult
like,
for
example,
in
a
hospital
deciding
the
treatment
of
patients,
you
can't
just
experience
and
play
around
and
see
if
the
patient
dies
or
not.
You
kind
of
have
to
build
on
this
model
of
the
world
that
you
learned
before
and
also
be
incorporated
and
be
able
to
incorporate
knowledge
about
the
model
that
other
people
have
accumulated
through
research.
C
It's
an
interesting
question:
whether
language
fills
that
role,
because,
typically
like
we're
doing
right
now,
language
involves
there's
an
exploration
to
it
as
well.
You'll,
say
something
vivian
and
I
won't
understand
you
and
I'll
say
well.
What
did
you
mean
by
that
and
that's
sort
of
me
acting
on
your
knowledge
base?
In
some
sense,
I'm
I'm
I'm
performing
an
action
and
I'm
going
to
see
what
happens
what
you
say
when
I
say
something.
C
So
if
that's
an
interesting
way,
just
I'm
just
saying
it's
in
this
way
to
think
about
expanding
beyond
what
we
can.
Personally,
it's
not
quite
related
to
your
your
hospital
example,
but
the
idea
that
there's
things
that
you
know
that's
you
you're.
I
know
that
you're
living
in
europe
right
now
and
I
could
ask
you
about
where
you're
living
and
I
don't
have
to
go
there,
and
I
can
probe
you
and
you
can
answer
it
almost
as
if
I
was
there
and
I
could
extend
my
knowledge
base.
That's
you
know.
C
We've
talked
a
lot
of
people.
We
talked
about
language
and
comet
talked
about
it
a
bit,
but
language
could
be
viewed.
That
way.
I
thought
language
could
be
used
as
a
sensory
motive
experience
as
long
as
I
get
to
interact
with
you
as
long
as
I
get
to
probe
you
and
say
you
know
what
happened
to
people.
If
I
say
this,
what
are
you
going
to
say?
I
say
this:
what
are
you
going
to
say.
B
G
C
A
C
B
Yeah
I
mean,
since
there
is
a
neuroscience
flare
to
brains
and
bay,
I'll
ask
one
of
the
neuroscience
connection
questions
you
know.
If
I
recall
correctly,
there
is
stimulation
of
dopaminergic
pathways
in
recognition
of
novel
stimuli.
Could
this
be
what
it
serves?
Could
this
be
the
substrate
for
goal
formation
and
play
and
exploration
as
well?
B
B
D
D
D
Of
course
is
not
reward,
dopamine
is
the
td
error
and
it's
easily
confused
with
reward.
But
it's
it's
really
like
it's
the
moment
by
moment
of
things
getting
better,
which
is
different
from
awards.
Maybe
you
expected
the
reward
and
it
wasn't
as
much
as
you
were
expecting
and
so
you're
feeling
bad,
and
so
when,
like
when
jeff
solves
a
a
a
problem
he's
working
on,
he
feels
satisfaction.
D
We
tend
to
say
he
feels
rewarded
by
that,
but
maybe
he
only
he
didn't
get
reward,
but
he
just
got
and
improved
in
his
value
function,
which
would
feel
like
reward,
because
it
would
be
the
dopamine
and
everything
but
and
and
and
the
tear
has
assumed
you
know-
reward
to
the
td.
The
td
error,
the
temporal
difference
error
that
you're
changing
moment
by
moment
change
in
how
well
things
are
going
as
a
component,
that's
reward,
and
also
as
a
component
and
change
in
value,
so
change
in
value
and
reward.
D
They
add
together
to
be
the
td
error,
which
is
what
don't
mean
can
be
interpreted
as
being
okay
and
so
reward
and
change
in
value.
They
feel
the
same.
They
both
feel
like
dopamine
and
the
vast
mass
of
your
brain
only
gets
dopamine,
it
doesn't
get
the
reward
and
we
don't
see
the
reward,
because
the
reward
is
not
important.
Only
the
combined
combination
of
reward
with
the
change
in
value,
that's
important
that
that
combined
thing
is
important
and
needs
to
be
spread
everywhere,
not
the
individual
pieces.
D
D
D
We
know
you
know,
I
think,
there's
evidence
that
both
of
these
things
happen
or
could
happen,
and
we
can
think
about
them
and
then
there's
the
question
about
multiple
rewards
and
again
it's
it's
a
question:
how
you
want
to
think
about
things
and
and
how
you
want
to
use
the
words.
I
think
I
think
you
know
to
think
about
a
goal
seeking
system
you
have
to
reduce
it
to
one
goal.
D
It's
a
big
area
in
which
you
extend
reinforce
something
reinforcing.
These
limited
only
has
one
goal,
but
if
we
could
have
six
goals,
that
would
be
better
and
I
want
to
say
I
just
always
want
to
think.
Well,
that's
that's
somewhat
somewhat
nonsense.
You
can
only
have
one
goal
and
but
you
mean
sometimes
it's
useful
to
think
in
all
these
different
ways
and
it's
all
a
different
way
of
thinking
about
things.
C
B
C
C
Lost
your
specificity
then
so
then
it
could
be.
You
know
they
could
be
acting
in
different
places.
You
know,
I
don't
know
an
interesting
question,
but
I
think
rich's
comment
is
really
interesting
when
it's
it's
almost
an
oxymoron
if
they
have
multiple
goals
because
you
got
to
pick
one
and
and
if
they
can
all
be
solved
at
the
same
time-
and
I
guess
that
was
one
goal,
so
it's
like
it
wasn't
multiple
goals
to
begin
with
interesting
observation,
yeah
yeah,.