►
From YouTube: Lucas Souza Reviews “Rearrangement: A Challenge for Embodied AI” Paper - January 6, 2021
Description
Lucas Souza continues his discussion on machine learning benchmarks and environments. In this meeting, he reviews the paper “Rearrangement: A Challenge for Embodied AI”. The paper proposes a set of benchmarks that captures many of the challenges the AI community needs to overcome to move towards human level sensorimotor intelligence. He discusses how goals can be specified, a taxonomy to categorize different types of agents and environments and some examples of benchmarks that follow the proposed structure. The team then discusses how they can translate the machine learning benchmark / environment to Numenta's work.
“Rearrangement: A Challenge for Embodied AI” by Dhruv Batra, et al.: https://arxiv.org/abs/2011.01975
Lucas Souza on iGibson Environment and Benchmark - December 14, 2020: https://youtu.be/feteCs80bIQ?t=4170
- - - - -
Numenta is leading the new era of machine intelligence. Our deep experience in theoretical neuroscience research has led to tremendous discoveries on how the brain works. We have developed a framework called the Thousand Brains Theory of Intelligence that will be fundamental to advancing the state of artificial intelligence and machine learning. By applying this theory to existing deep learning systems, we are addressing today’s bottlenecks while enabling tomorrow’s applications.
Subscribe to our News Digest for the latest news about neuroscience and artificial intelligence:
https://tinyurl.com/NumentaNewsDigest
Subscribe to our Newsletter for the latest Numenta updates:
https://tinyurl.com/NumentaNewsletter
Our Social Media:
https://twitter.com/Numenta
https://www.facebook.com/OfficialNumenta
https://www.linkedin.com/company/numenta
Our Open Source Resources:
https://github.com/numenta
https://discourse.numenta.org/
Our Website:
https://numenta.com/
A
So
today
I
have
a
continuation
of
the
discussions.
We've
been
having
on
machine
learning
benchmarks.
This
presentation
is
maybe
a
bit
long,
I'd
say
20
minutes.
If
I
just
deliver
it,
but
I
don't
think
delivery
is
to
go.
We
want
to
discuss
so
probably
an
hour,
so
yeah
bear
with
me
the
there
are
some
interesting
aspects,
I'll
move
fast,
but
I
can
stop
if
someone
is
interested
in
any
specific
aspect.
A
So
last
time
we
thought
I
don't
think
marcus.
Were
you
here
we're
going
to
talk
about
in
character,
gibson
yeah.
I
think
you
were
just
last
week,
you're
out
right.
So,
let's
last
time
you
talk
about
environment
that
we
could
use
to
run
experiments,
but
I
did
mention
a
benchmark
there,
but
the
benchmark
there,
I
didn't
think,
was
very
relevant.
Neither
the
authors
were
pushing
for
that
benchmark.
So
the
benchmark
there
was
just
navigation
in
a
cluttered
environment
and
today
I'm
actually
going
to
do
the
opposite.
A
So,
instead
of
talking
in
environment,
I'm
going
to
talk
about
a
benchmark
that
could
be
applied
in
different
environments.
Even
on
that
one
we
discussed.
So
the
paper
name
is
a
rearrangement.
The
challenge
for
embedded
ai
and
I
came
into
contact
with
it
at
jitendra
malik
stock
at
the
self-supervised
workshop
and
since
we're
being
recorded,
are
we
recording,
yeah
right?
Okay,
since
we're
recording?
I
just
should
have
mentioned
it's
a
paper
review
and
image
and
text
are
all
extract
from
the
paper,
so
full
credit
to
the
authors.
A
There
is
not
no
new
content
here,
and
these
are
the
authors
by
the
way.
So
it's
a
it's
a
group
of
people
from
from
many
universes,
a
lot
of
people
working
in
machine
learning
for
robotics
or
sometimes
are
known
here
like
jet
banks
or
the,
and
I
think
it
was
released
last
month
so
found
the
street
say
talking
about
the
process
so
saying
this
reports
a
result
of
13
to
our
discussion
over
five
months
with
this
group
of
people.
A
A
So
the
goal
of
the
benchmark
is
to
transform
an
initial
environment
from
a
state
as
0
to
a
go
state
as
star
that
belongs
to
a
set
of
states.
S
so
as
star
the
the
goal.
State
may
not
be
unique,
but
it
can
be
an
element
from
a
set
of
acceptable
states.
So
a
star
belongs
to
a
set,
a
big
ass
star,
which
is
a
subset
of
all
possible
states.
So
here's
an
example:
we
have
so
the
current
state
is
this:
the
kitchen
is
kind
of
a
mess
lucas.
B
A
A
A
A
The
idea
is
that
to
define
a
threshold
for
each
object
and
then
we
make
a
binary
test
for
each
one
to
see
if
the,
if
we
are
able
to
rearrange
that
object
or
not,
and
we
can
report
the
overall
completion
percentage
as
a
fraction
of
objects
which
pass
that
threshold
test.
So
there
might
be
multiple
objects
that
need
rearrangement
and
in
that
metric,
in
that
threshold
we
can
take
into
account
sensible
tolerances.
So
the
book
doesn't
have
to
be
in
a
specific
place.
A
It
can
be
you
know
in
that
region
and
we
can
also
account
for
desirable
robustness
properties.
So
let's
say
we
want
to
rearrange
the
rune
and
we
wouldn't
mind
if
one
or
two
objects
in
the
room
are
still
out
of
place
as
long
as
the
overall
rune
is,
is
needing
a
nice
condition
and
as
secondary
metrics.
A
There
are
several
things
we
can
add,
but
one
that
comes
to
mind
or
more
objects
is
to
add
a
do,
no
harm
test,
so
we
can
avoid
agents
making
a
mess
on
other
objects,
no
related
to
the
goal
to
achieve
the
goal
right.
So
if
the
goal
is
just
to
get
a
book
across
the
room
and
put
it
in
the
bookshelf,
you
don't
want
the
agent
to
just
mess
up
the
rune
along
the
way,
just
move
the
sofa
and
break
a
lamp.
A
So
we
can
add
this,
do
no
harm
test
to
check
if
other
objects
have
been
moved
or
not,
and
we
can.
We
can
even
test
go
ahead
and
not
only
test
the
pose
the
final
pose
of
the
object,
but
we
can
also
test
things
like
maximum
acceleration
or
forces
experienced
by
objects
within
the
simulator
so
that
we
don't
wanna
shake
something
very
hard
because
it's
gonna,
it's
gonna
break
or
it's
gonna
affect
the.
C
A
That
that's
right,
that
could
be
a
secondary
metric,
but
it's
by
default.
It's
not
a
metric,
but
I'm
going
to
show
in
the
end
an
example
implementation
from
inhab.ai
from
facebook,
where
they
define
the
metric
as
test
completion
divided
by
by
length
the
absolute
length
how
many
steps
it
took
to
complete
so
with
taking
into
account
this
speed
is
actually
what
you're
saying,
but
by
default
the
paper
doesn't
define.
Speed
is
a
metric.
A
So
there
are
several
level
of
abstractions
we
can
think
of
so
in
terms
of
actuators
and
sensors.
So
what
we
want
is
to
simulate
something
that
could
happen
in
the
physical
world
in
the
simulated
world.
So
in
the
actual
actuator
site,
the
lowest
level
of
abstraction
would
just
be
like
a
magic
pointer
like
you
have
in
vr,
so
you
just
have
a
mouse
and
the
mouse
manipulates
objects
which
is
very
far
away
from
reality.
A
The
second
one
would
be
you
have
an
articulated
arm.
For
example,
you
have
the
robot,
but
you
abstract,
some
of
its.
You
attract
some
of
its
movement
like
abstract,
grasping.
So
if
the
hand
moves
close
enough
to
the
object,
then
it's
going
to
grasp
kind
of
like
we
have
in
vr
today,
but
you're
not
actually
expecting
the
robot
to
go
and
grasp
the
objective
strength
and
the
third
level
would
be
a
full
physical
simulation
and
that's
the
one.
A
We
are
interesting
that
we
are
actually
going
to
take
into
account
the
strength
that
the
agent
applies
to
its
graph
in
order
to
to
get
the
object
or
not.
So
if
it's
heavy
and
it
doesn't
apply
enough
strength
it's
going
to
and
on
the
sensor,
we
have
the
same
three
levels
of
abstraction,
so
the
first
one
will
be.
A
You
have
ground
tooth
positions,
you
you
know
where
every
object
is
so
you
have
this
complete
information
beforehand
and
the
second
one
is
the
one
that's
more
using
robotics
today,
which
you
have
access
to
intermediate
representations.
So
the
traditional
pipeline
in
robotics
today
is
you
start
from
a
semantic
map.
So
today
it's
very
easy
to
get
neural
networks
which
do
semantic
mapping
and
can
actually
classify
every
pixel
in
in
the
in
an.
A
In
a
3d
mesh
or
in
an
image,
so
you
start
from
that
representation,
and
then
you
learn
how
to
do
actions
and
how
to
play.
That's
kind
of
the
scenario
today
and
the
the
highest
level
of
abstraction
is
a
full
simulated
perception.
So
you
don't
have
access
to
these
intermediate
representations.
You
only
have
whatever
the
robot
would
have
access
to
like
a
rgb
camera,
and
you
also,
you
know,
even
add
some
realistic
noise
and
uncertainty
and
perfect
calibration
to
reflect
what
actual
sensors
are
going
to
look
like
in
real
life.
A
It
doesn't
have
to
be
so
like.
I
said
this
is
more
of
a
framework
and
I'm
going
to
defy
multiple
ways
of
defining
sensors.
But
what
that
what
the
authors
are
proposing
is
that
we
move
towards
that.
We
move
towards
sensors
that
are
realistic
and
since
they're
interested
in
training
robots,
it
doesn't
only
have
to
be
camera,
I
mean,
but
if
the
final
robot
only
has
a
camera,
then
it's
going
to
be
a
camera,
but
if
they
find
a
robot
also
has
access
to
a
lighter,
then
you
could
have
access
to
lighter
inside
the
simulation.
A
B
I
think
the
interesting
thing
is
that
when
you
say
full
simulated
perception
that
to
me
means,
I
think,
maybe
you're
going
this
way
lewis,
but
it
means
that
you
have
to
have
the
somatosensory
perception
too.
I
mean
you
can't
possibly
manipulate
objects
and
pick
them
up
just
by
vision.
I
mean
it'd
be
almost
impossible
you'd
have
to.
B
A
B
Yeah
but
but
when
you
say
then
then
you'd
have
an
intermediate
representation
right
then
you
say:
oh,
I
might
have
a
whole
visual
system,
but
I
I
I'm
going
to
assume
that
I
have
some
sort
of
grasping
feature
or
you
know
some
I
mean,
but
but
humans,
we,
you
know
we
have
to
feel
these
objects.
We
have
to
touch
them
every
muscles
that
are
moving.
B
D
B
A
D
B
But
just
physically
being
able
to
move
something
means
you
have
it.
It
just
seems
like
that
at
the
task
you
defined
you,
it
can't
be
a
purely
visual
test
to
pick
something
up
and
move
it
requires.
If
you're
going
to
do
full
simulated
perception,
it
requires
some
hapticness.
I
don't
know
how
you
could
get
around
that.
So,
what's
going
to
what's
going
to
pick
it
up,
you
know
magic
laser
beam.
I
don't
know.
D
B
Yeah
and
not
just
that,
but
just
you
know,
as
we've
talked
about
here
so
many
times
just
when
you
pick
up
a
book,
you
often
don't
even
look
at
the
book
as
you're
picking
it
up.
You
just
use
your
hands
to
feel
where
you
know
you
can
reach
out
and
you
oh,
this
is
the
spine.
That's
ed!
You
know
you
do
so
much
of
this
stuff
without
any
visual
feedback.
It's
if
you
watch
yourself
do
this.
It's
amazing
how
much
you
do
in
close
manipulation.
Just
through
touch
sensors.
B
A
Yeah,
I
agree
so
some
of
the
guidelines,
the
the
authors
propose
is
one
is
focus
on
strong
generalization
versus
weak
generalization
and
and
the
way
they
define
it
is
the
weak.
One
is
a
system
that
you
evaluate
with
non-objects
in
non-environment,
so
both
environments
and
objects
first
seen
during
training,
even
if
sometimes
the
arrangement
is
novel.
So
if
you,
you
you've,
seen
a
book
and
you've
seen
that
one
during
training,
but
you
just
put
the
book
in
a
different
room:
that's
that's
still
weak
generalization
and
what
they
call
strong
generalization.
A
This
is
the
agent's
task
with
rearranging
new
objects
in
new
environments.
So
you
train
it
to,
I
don't
know
close
the
fridge,
and
now
you
have
to
move
a
chair
in
a
completely
different
room.
So
it's
it's
a
new
tacit
it
hasn't
seen
before
and
they
recommend
this
should
be
the
default
mode
of
operation.
A
A
second
guideline
is
track
based
on
realistic,
perceptor
input.
So
in
order
to
do
that,
we
should
add
some
noise
and
incomplete
perception,
because
that's
how
it
is
in
real
life
and
like
jeff
was
just
mentioned.
They
recommend
sensing
environment,
realistics
and
server
of
realistic
means
for
the
rob
robot,
rather
than
just
planning
in
an
idealized
kind
of
map
where
you
already
have
the
waypoint.
So
you
have
some
sort
of
map
beforehand.
B
I
have
a
question
about
this.
You
know
if
you
have
a
realistic
sensor,
which
should
be
an
array
of
sensors.
Like
you
know,
a
vision
system
is
an
array
of
visual
inputs
and
on.
Why
do
you
have
to
add
noise?
I
mean
they
may
be
inherently
noisy,
but
this
you're
suggesting
that
you
have
to
add
noise
to
the
system.
A
Yeah
no,
but
not
add
as
in
more
than
realistic,
but
they
are
inherently
noisy,
like
at
actual
sensors.
If
you
have,
I
don't
feel
like
a
camera,
the
camera
one
in
in
a
thousand
times,
it's
gonna,
show
something
noisy
like
different
than
it
should
be,
and
sometimes
this
is
not
reflected
in
simulation,
and
it's
a
big
issue
that
if
you
take
something
from
simulation,
that's
perfect,
just
something
in
real.
B
C
Yeah,
let
me
this
has
been
a
big
change
in
robotics
over
the
last
couple
of
years.
Traditionally,
it's
been
really
really
hard
to
take
something
that's
learned
in
simulation
and
transferred
to
the
real
world
and
what
they
found.
Basically,
one
of
the
big
tricks
is
to
you
just
during
simulation.
You
add
noise
everywhere
throughout
the
system,
even
though
you
might
not
actually
get
it
even
in
the
real
world.
You
just
want
your
whatever
you
learned
in
the
simulated
world
to
be
much
more
robust
than
what
the
simulation
itself
would
would
require.
C
A
Are
nice
and
the
sensors
are
perfect
and
that's
independent
of
the
algorithm,
that's
from
the
benchmark
and
and
and
this
these
in
particular,
the
arguing
that
the
sensor
should
never
be
non-noisy
I'll
always
have
to
have
just
a
little
noise
to
reflect
what
real
world
should
be,
but
whether
the
algorithm
is
going
to
inject
more
knives
or
no,
not
injecting.
Then
that's
an
algorithmic.
A
And
the
last
one
that
ties
into
this
really
well
is
that
they
should
be
transferable
to
physical
words,
so
one
thing,
for
example,
subject
should
be
removed
by
forces
when
possible,
most
commonly
applied
via
contact.
So,
like
I
said,
we
are
looking
for
that
last
scenario
where
we
we
actually
want
to
have
like
a
full
full
physical.
A
Okay:
these
are
the
characteristics.
They
are
good,
that
this
problem
helps
that
can
emerge
from
solving
this
problem
right.
So
we
are.
The
agent
needs
to
have
perception
learned
how
to
navigate
environment.
Manipulate
objects,
have
some
memory
to
where
objects
were,
or
has
seen
an
object
before
or
not
do
planning
and
do
communication
in
the
case
you're
using
a
natural
language
as
well,
which
is
just
one
specific,
so
communication
just
for
a
specific
case,
all
the
others
would
fit
most
of
the
cases.
B
A
B
That
seems
like
a
really
huge
burden
to
put
on
this
network.
I
mean
it'd
be
great
to
do
it,
but
you
one
way
you
could
do
is
you
could
have
a
system,
learning
environment
and
then
come
back
later
and
say:
oh,
it's
changed.
Let
me
put
things
back
the
way
they
were
before,
which
wouldn't
require
any
communications.
B
C
A
A
So
there
are
several
ways
to
specify
the
goal,
so
one
the
most
traditional
one
is
what
they
call
geometric
and
you
do
a
geometric
specification
of
the
transformation
that
the
object
should
go
from
to
go
from
state
at
zero
to
state
that
star.
So
one
example-
and
I
think
that's
the
one
I've
seen
mostly
using
the
examples-
is
the
center
of
mass
of
the
object
in
the
go
state
relative
to
start
state.
A
A
That's
a
great
question,
so
I
didn't
put
everything
in
this
presentation.
That's
one
thing
I
had.
They
highlight
the
difference
between
how
you
evaluate
and
how
you
communicate
that
to
data,
because
how
you
communicate
also
hides
the
agent's.
Not
gonna
have
access
to
the
same
way.
You
will
you
actually
evaluate
so
when
the
agent
goes
and
calculates
his
own
reward,
if
no
one
is
closer
or
farther
from
the
goal.
He's
not
have
access
to
the
to
the
same
information
that
the
experiment
designer
is
going
to
have.
B
All
right,
so
that
makes
sense
to
me
I
mean
because
that's
a
way
of
saying
objectively
saying:
okay
did
the
agent
achieve
what
we
wanted
to
do,
but
it
seems
to
me
this
would
be
a
very
odd
way
of
specifying
what
the
agent's
supposed
to
do.
I
mean
you
know
again
the
example
of
the
little
messy
room.
You
might
say
your
goal
is
to
rearrange
the
room
as
it
was
before
and
that's
the
goal,
the
specified
goal,
but
then
you
would
evaluate
it
using
these
metrics.
A
A
Performance,
yeah
yeah,
but
in
this
case
it's
a
way
of
evaluating
performance
yeah,
but
also
you'd
have
to
communicate
to
the
agent
in
some
way,
which
is
not
very
different
than
this.
Maybe
you're
not
going
to
use
the
full
data
that
you
have
access
to
travailic,
but
the
geometric
way
you're
still
going
to
communicate
kind
of
in
that
in
this
language,
which
is
not
a
natural
way
or
not,
that's
very
unnatural,
so
so
the
second
one
is
a
bit
more
natural,
that's
image!
A
So
you
give
a
visual
rendering
or
a
presentation
of
a
star
like
from
a
third
person.
Point
of
view
could
be
like
overhead
perspective
or
isometric
image
of
the
environment
in
the
ghost
state.
So
let's
give
it
a
picture.
Here's
how
the
room
should
look
like
the
issue
is
that
the
go
image
may
be
underspecified,
so
certain
objects
cannot
may
not
be
visible
in
the
image
or
multiple
possible.
A
Valid
goal
placements
can
be
out
of
you
in
that
picture,
especially
if
it's
a
truly
picture.
It
also
limit
our
ability
to
specify
that
certain
objects
should
be
placed
or
contained
inside
other
objects.
So
if
you,
if
you're
going
to
say
that
you
know
I
want
you
to
put
this
book
inside
the
box,
maybe
it's
going
to
be
really
hard
to
show
it
with
a
picture
and
a
way
to
go
around
is
that
it
can
also
be
a
video.
A
A
A
C
A
This
back
and
forth
recognized
clarification
in
this
case.
It's
just
stating
what
needs
to
be
done
and
there
might
actually
several
go
states
that
can
fulfill
one
of
these
language
posts.
Maybe
there
are
more
than
one
blue
block,
you
know.
Maybe
there
is
more
than
one
chair,
so
it's
it's
hard
to
tell,
but
still
it's
it's,
it's
more
natural,
even
more
natural
than
the
image.
A
One
so
the
fourth
one
is
experience
and
that's
the
one
jeff
was
referring
to.
So
you
just
immerse
the
agent
in
the
environment
in
the
goal,
condition
and
you
give
it
like
a
time
or
interaction
budget.
So
you
can
see
and
you
let
the
agent
build
whatever
representation
the
teams
appropriate
to
bring
that
environment
back
to
the
state.
A
So
the
agent's
gonna
build
his
own
representation
and
the
goal
is
to
make
it
as
it
was
before
so,
like
you
have
a
rom
robot,
it
maps
the
kitchen
before
so
it
knows
how
it
should
be
when
it's
clean
and
after
you
have
a
party
at
home,
it's
gonna
know
where
it
should
go
back
to
that.
They
say
that
it
should
go
back
to
and.
C
A
Environment,
yeah
and
and
the
last
one
it's
from
symbolic,
ai,
it's
to
define
in
terms
of
predicates.
So
the
goal
is
a
set
of
predicates
that
should
be
satisfied
in
the
ghost
state
and
on
plate.
One
on
table
one
and
the
bread
gates
need
to
be
evaluated
somehow,
and
we
can
do
that
through
geometric,
showing
of
positions.
A
You
can
use
a
neural
network
classifier
to
evaluate
for
predicate
it's
true
or
not,
or
we
can
programmatically
check
the
logical
state
of
all
called
objects.
So,
let's
see
if
the
microwave
is
on
or
off-
and
it's
not
a
natural
interface
for
humans
for
sure,
but
it
could
be
an
useful
interface
for
systems.
C
C
A
A
C
A
B
Well,
the
question
might
be,
and
I
have
a
similar
question
is:
is
you
know,
is
there
when
you
read
this
proposal?
Is
it
like?
Well,
here's
the
end
goal.
We
want
to
be
able
to
either
do
this
by
language
or
experience
and
here's
some
intermediate
goals
that
we
could
use,
or
is
it
like?
No,
we
think
in
the
end.
Predicates
are
a
good
way
of
specifying
goals
in
a
robot
factory.
In
fact
we're
run
by
robots,
and
so
we
think
that's
a
desirable
end
goal
as
opposed
to
that's
an
intermediate
step
to
a.
A
B
B
I
got
it,
I
got
it
yeah
yeah,
I
think
that's
it's
interesting,
I'm
not
being
critical
of
it.
It's
just
there's
two
choices
here,
one
is
to
say
it's
the
potpourri
choice
it's
like
well,
we
don't
really
know
what
the
ultimate
end
goal
should
look
like.
So
here's
a
bunch
of
different
ways
and
this
could
create
all
kinds
of
research
or
it
could
be
like
here's,
an
end
goal
and
here's
some
intermediate
steps
to
get
there.
B
But
the
end
goal
is
to
be
human-like,
then
the
experience
and
language
ones
would
be
the
only
one
that
really
makes
sense
mostly.
But
if
your
end
goal
is
like
it's
just
a
platform
for
all
different
types
of
research.
Well,
then,
that's
that's
okay,
but
it's
a
different
goal.
So
it
sounds
like
it's
that
it's
like
you
know.
Maybe
the
people
who
love
predicates
and
they
just
want
to
make
sure
there's
a
predicate-based
system.
A
B
Yeah,
I
think
that
gets
down
to
this
big.
You
know
lifelong
issue
that
I've
been
struggling
with
and
we've
been
struggling,
which
is
you
know,
do
we
can
we
create
the
true
ai
and
so
on
and
ignore
you
know
then
human-like
systems
and
and
it's
or
not,
so
I
guess,
but
I
I've
enough
sad
about
it.
I
just
wanted
to
know
what
their
attitude
was.
A
A
So
mobility
is
how
the
agent
moves
and
that
can
restrict
the
rearrangement
scenarios
that
can
be
performed
like
if
it's
a
fixed,
robot
arm.
It
cannot
move
long
distance
and
also
the
terms
the
available
action
space
which
can
it
can
range
from
like
discrete
actions.
The
movement
like
just
turn
left
and
right,
like
clock
forward
one
meter
to
parameter
transactions.
You
fully
continuous
control
for
each
motor.
Let's.
A
A
Second,
one
is
manipulation
how
how
the
agent
manipulates
environment
so
simplest,
one.
You
have
the
magic
pointer
obstruction.
You
have
section
based
grippers
and
you
know
I
can
have
a
hand
or
neither
stick
force.
Applicators
are
very
common
in
robotics.
You
know
just
that.
You
know
that
pushes
things
or
all
the
way
to
five
finger,
humanoid
grabbers,
sorry,
I
believe-
and
this
defines
the
spectrum,
the
spectrum
of
manipulation
capabilities.
They
form
some
natural
abstraction
for
research
and
we're
going
to
make
that
higher
levels
versus
lower
levels.
A
So,
if
you
actually
have
like
a
hand,
then
you
have
to
be
worried
about
like
fully
continuous
control
in
the
on
the
hand
how
to
move
each
part
of
the
finger.
But
if
you
only
have
like
a
neurostick
force,
then
your
the
problem
is
going
to
be
at
a
higher
level.
It's
going
to
be
more
about
planning
than
how
try
to
you
know
grasp
you
can
move
the
object.
B
Yeah
I
find
I
find
the
distinction
between
mobility
and
manipulation
is,
might
be
misleading
and
false
that
you
know
from
the
brain's
point
of
view,
I'm
not
sure,
there's
a
difference.
They're,
all
sensory
motor
movements
and
sensory.
You
know,
you're
moving
you're
sensing
you're
even
walking
is
not
like
walking,
is
in
some
sense,
touching
the
floor
and
touching
steps
and
and
and
manipulating
rough
surfaces.
It's
it's
it's
it's
it's
a
sort
of
a
false
distinction.
In
my
mind,
which
can
again
I'm
looking,
you
know
it
could
lead
to.
A
One
thing:
that's
really
common
in
robotics
research
is
when
we're
more
concerned
about
manipulation.
We
abstract
the
mobility
part
just
using
like
a
fixed
space.
We
just
we
use
those
products
which
you
know
have
that
base
that
moves
like
matte
robots
that
are
at
meantime,
and
then
you
can
focus
on
on
grapping
and
other
things.
And
if
we're
worried
about
navigation
and
then
we
might
add
mobility,
but
I
have
seen
very
few
works
that
have
both
that
actually
are
concerned
about
mobility.
A
B
I
mean
if
you
look
at
the
brain
right.
You
know
common
cortical
algorithm,
the
the
parts
in
your
cortex
that
you
know
move
the
limbs
of
your
body,
including
your
feet
or
in
your
fingers.
They're
all
it's
all
one
system.
It's
not
like.
There's
a
separate
mobility
system
in
a
separate.
You
know
manipulation
system,
it's
it's
so
from
a
biological
point
of
view.
We
can
almost
say
there
may
be
some
differences
and
details,
but
conceptually
they're
the
same
problem,
you're
moving
your
fingers
is
it's
no
different
than
moving
your
body
right
and
you're.
B
Moving
around
your
fingers
in
the
space
of
the
coffee
cup
is
the
same
as
moving
your
body
in
the
space
of
a
room.
So
it's
just
it's
just
when
you
break
it
apart
this
way-
and
I
understand
that
this
has
been
historically
correct
and
unfortunately,
where
they've
done
it,
and
this
may
be
where
they
keep
going,
but
I'm
looking
for
things
here
where,
where
you
know
overall
I'm
listening
to
this
and
thinking
like.
Oh
you
don't
want
this
to
be
another
blocked
world.
B
A
Sorry
about
the
typos,
this
lights
up
or
shutter
hang
optics,
and
so,
if
it's
object,
grasping
task,
you
need
some
sort
of
haptic
sensing.
If
it's
a
navigation,
have
it
test,
you
need
some
sort
of
long-range
visual
sensing
and,
as
as
I
mentioned
a
few
times
before,
the
recommendation
is
to
simulate
noisy
sensors.
So
if,
if
cameras
in
the
real
world
are
going
to
be
noisy,
cameras
in
the
simulated
world
have
to
be
nice
as.
A
A
You
want
to
put
the
vase
on
top
of
of
the
of
the
chair,
but
it
has
to
be
on
the
center
or
is
it
okay
if
it's
a
little
bit
to
the
side
and
whether
ordering
is
important
or
not?
So
if
it's
a
stacking
task,
you
want
to
stack
something
like
when
you
put
the
books
on
the
shelf.
Is
the
order
of
the
books
relevant
or
not,
and
the
structure
of
the
environment
also
defines
the
complexity?
A
Is
it
an
open
space
or
it's
highly
cluttered,
so
david's
gonna
have
a
hard
time
moving
around
how
many
destructor
objects
are
there
and
if
you're
looking
for
a
specific
book
in
a
table
that
is
full
of
books,
it's
gonna
be
a
lot
harder
and
the
last
one
is
dynamicity,
which
is
whether
objects
and
environment
are.
They
are
subject
to
changes
or
perturbations,
which
are
only
related
to
agent
actions,
whether
the
environment's
dynamic
or
not.
A
So,
for
example,
can
have
the
oscillating
fan
or
you
can
have
wing
or
vibrations
which
are
changing
things
in
the
environment,
independence
of
the
action,
so
the
agent
has
to
learn
how
to
deal
with
that
and
whether
or
not
there
are
recoverable
states.
Let's
say
if
you
you
drop
the
plate
and
it
breaks
and
and
then
that's
a
change
that
you
cannot
never
recover
from
the
changes
you
that
they
states
that
you
can
go
from
there.
A
And
just
give
you
some
examples,
so
the
authors
give
a
few
examples
on
the
paper
and
I
think
five
of
them,
I'm
gonna,
go
super
fast,
so
one
very
simple
one
so
buy
manual
sweeping
you
control
two
robot
arms
two
collaboratively
pick
up
box
with
your
randomly
placed
on
the
table
and
they
have
to
place
them
in
this
target
pin
and
then
in
the
paper
they
all
that
specification.
I've
been
talking
about
like
in
terms
of
task
and
agent.
That's
for
each
of
these
tasks.
A
It's
defined,
how
sorry,
how
it's
a
complexity
and
interruptability
dynamicity,
but
in
short,
in
just
a
few
words
this
is
the
problem
and
inside
parenthesis.
Here
is
the
environment,
so
these
were
implemented
in
different
computational
environments.
So
sapien
is
one
of
the
like
interactive
gibson.
I
discussed
last
time
so
it
paints
like
a
similar
one.
B
A
B
A
Right
exactly
yeah,
it's
just
like
the
interactive
gibson
like
the
last
time.
So
like
another
one,
so
this
one
was
a
challenge
in
iros
2020.
I
think
it
was
september,
I'm
not
mistaken.
So
this
is
a
cloud
robot
table
organization
and
the
goal
was
to
do
it
in
a
you.
Could
train
it
in
a
virtual
environment,
but
you
actually
test
it
in
an
actual
robot
and
yeah.
It's
just
table
organization
and
you
have
to
move
things
from
one
side
to
another
and
that
this
also
use
the
geometric
goal.
A
A
B
B
A
Another
one
is
room
rearrangement,
that's
in
ai
to
tor,
that's
another
environment,
and
so
the
agent.
This
is
more
in
line
with
the
what
we
discussed
from
the
beginning.
So
agent
has
to
make
changes
to
the
scene
by
placing
an
object
at
different
locations
or
change
their
state,
somehow
like
close
or
up
in
the
computer.
In
this
case
they
only
have
close
and
oops.
They
only
have
oops.
A
A
A
A
C
A
Remember
last
time
we
talked
about
interact
gibson
that
uses
these
photorealistic
scans
as
well,
so
they
use
the
same
scans
and
what
they
did
is
they
didn't
get
cluttered
environments.
They
got
empty
apartments
and
houses
which
are
realistic
and
they
added
scanned
object,
meshes
from
another
object
data
set
and
they
added
to
the
rooms
like
programmatically.
A
So
every
scenario
is
a
combination
of
empty
houses
and
inserted
objects.
You
just
add
a
bunch
of
objects.
Some
of
some
of
those
objects
are
going
to
be
pertaining
to
the
goal
and
some
are
not,
and
this
this
kind
of
allows
for
some
controlled
generation
of
training.
Testing
right
since
you
can
add-
or
you
can
make
subjects
and
runes,
then
it
gets
easier
to
generate
environments
on
the
fly,
and
the
good
thing
is
that
all
these
objects
added
since
they're,
not
from
the
scan
they're
from
these
object
meshes.
They
are
also
interactive.
A
A
So
so
that
episode
is
defined
by
the
the
house
name:
the
environment,
name,
oops,
the
alien
spawn
location
rotation
so
where
the
agent
is
born
and
where
he's
looking
at
and
the
initial
object,
location
and
rotation,
that's
how
you
define
an
epsilon
for
gold.
They
use
the
geometric
go
specification,
so
they
have
the
initial
and
the
desired
position
of
the
center
of
mass
of
each
object.
A
A
This
this
problem
doesn't
have
like
open,
close
book
kind
of
problem,
and
the
agent
is
this
little
guy
here
that,
maybe
I'm
sure
you
know
it's
that's
called
locobot
and
the
agent
here
is
a
virtual
local
box.
So
this
local
bot
is
a
very
common
agent.
Using
research
like
a
low
cost,
open
source,
robot
low
cost.
I
mean
low
cost
costs
like
six
thousand
dollars
and.
C
A
C
A
And
the
metric
finally,
the
metric
is:
is
this
a
success
like
it
was
defined
before
so
zero
to
one
but
their
weight
by
path
length,
so
they
calculate
the
optimal
path
like
using
like
a
traveling
salesman,
kind
of
algorithm
and
the
weight
that
the
successful
platform.
So
if
the
agent
takes
a
long
time
it's,
but
it
solves
it's
going
to
be
worse
than
the
other
one
that
solves,
but
you
can
do
it
faster.
D
A
B
A
B
A
D
A
C
A
B
A
B
B
C
A
A
We
definitely
could
yeah
that's
definitely
an
option,
although
looking
at
the
environments,
I
look
at
all
the
ones
that
are
mentioned
here.
I
I
really
like
hype
that,
and
maybe
you
can
speak,
that
at
some
point
and
and
I
like
it
just
because
how
it's
integrated
by
torch,
I
mean
it's
within
the
facebook
suite
of
machine
learning.
A
Yeah
yeah,
we
have
to
see
which
one
is
more
useful.
They
are
very
similar
like
they
both
use
the
same
realistic
rune
from
gibson,
and
they
both
insert
objects
which
are
manipulated.
It
can
be
manipulated
so
one
they
do
it
in
different
ways
right,
but
it's
the
end
goal
is
the
same.
They
look
very
similar
to
me.
So
it's
more
of
a
matter
and
which
one
is
easier
to
use
and
they're
both
from
big
groups
right,
one
is
from
faithfully
stanford
and
the
other
is
from
facebook.
So
that
definitely
is.
A
Yeah,
that's
that's
what
I
had
maybe
next
time
I
can
bring
a
proposal,
an
idea
of
something
that
can
be
more
interesting
to
us.
I
know
you're
not
looking
at
goal
specific
problems,
but
we
can
certainly
define
something
in
terms
of
the
agent
can
start
in
a
self-supervised
way,
so
the
agent
is
allowed
to
explore
the
region
for
a
while
and
then
it
has
to
solve
a
test.
So
it
has
to
go,
explore
view
the
model
and
then
it
can
use
that
model
to
do
some
sort
of
rearrangement.
B
I
I
have
a
you
know,
a
bias
or
an
opinion
about
how
to
think
about
the
different
problems
you
might
implement.
You
know
our
goal
is
to
to
be.
You
know
everything
we
do
is
on
the
path
to
you
know
real
machine
intelligence
and
under
the
belief
that
the
thousand
brains
concept
of
you
know
many
sentry
motor
models.
Work
together
is
the
answer
to
that,
and
so,
when
we
think
about
these
problems,
I'm
trying
I'm
trying
to
imagine
like.
B
Oh,
what
is
the
biological
equivalent
that
I'd
be
implementing
to
solve
one
of
these
problems
and
you
could
start
off
like
oh
well,
I
might
start
off
with
a
single
cortical
column.
What
can
it
do?
Can
I
define
a
problem,
a
single
sensory
motor
problem
with
a
very
simple
sensor
and
a
simple
actuator
or
then
the
next
step.
I
would
go.
Okay,
let's
create
a
set
of
those
you
might
try.
Let's
create
a
set
that
represents
a
multi,
you
know
two-fingered
hand
or
three-fingered
hand
and
they
they'll.
B
I
would
say
like
okay,
here's
a
solution
that
involves
one
for
column.
Here's
the
solution
involves.
You
know
10
cortical
columns,
I
know,
are
those
10
quarter,
columns
representing
things
that
are
adjacent
to
each
other,
like
the
muscles
in
your
hand
or
the
representing
different
parts
of
a
robot,
but
I
would
think
about
it
along
those
lines.
That
would
be
my
suggestion,
but
I
don't
that
was
clear.
B
A
B
Of
course,
of
course,
I
mean
you
could
start
as
simple
as
a
single
quarter
column.
In
some
sense,
I
think
I
just
think
that
the
problem
we
pick
is
going
to
very
much
in
some
sense
dictate
the
kind
of
solutions
we
could
try
to
solve
and
solutions.
We
would
pick
to
solve
the
problem
and-
and
some
of
them
may
be
too
ambitious.
B
You
know
if,
if
the
solution,
we
said,
oh
rearranging
the
room,
the
elements
in
the
room.
Well,
that
means
I
have
to
have
you
know
both
in
my
mind.
I'd
have
to
have
a
visual
system
and
the
tactical
system.
I'd
have
to
have
articulated
arms
and
articulated
a
body
to
move
in
the
world,
and
that's
a
it's
a
lot
to
bite
off.
B
It
might
be
better
to,
or
we
might
be
able
to
do
that
or
we
might
be
better
off
saying,
let's
start
with
a
two-fingered
hand,
and
can
it
you
know,
can
it
pick
up
an
object
and
using
your
fingers
just
rotate
it
and
place
it
back
down?
You
know
in
the
right
orientation,
which
seems
less
in
some
sense
might
seem
less
ambitious,
but
you're
actually
maybe
working
closer
to
the
final
solution
that
we
would
like
to
get.
B
So
I
think
it
requires
some
more
thought,
but
when
you
propose
some
ideas,
I'll
be
looking
at
it
that
way,
you
know
all
right.
Can
we,
let's
imagine
what
the
solution
would
be
look
like?
Can
we
imagine
solving
that
in
the
norm
near
term
using
the
principles
of
the
thousand
brain
theory
or
not
too
ambitious,
you
know
or
are
we?
Are
we
trying
to
manipulate
objects
without
having
a
tactile
system?
Therefore,
you're
going
to
end
up
doing
all
kinds
of
wonky
stuff?
You
know
something
like
that.
A
Oh
one
thing
that,
should
I
added
yesterday-
and
maybe
you
can
further
talk
about
it.
Sometimes
we
should
be
thinking
about
continual
learning
as
well,
and
I
think
most
of
the
problems
I've
seen
so
far.
They
still
have
a
distinction
between
training
and
testing
and
that's
a
distinction
we
want
to
break
through
break
from.
So
we
want
that
idea
that
you
can
continuously
learn
in
each
new
environment
without
forgetting
about
the
previous
ones.
So
we
wouldn't
have
that
trained
test
definition,
as
it's
done,
for
example,
in
this
problem
here
that.
C
Yeah,
I
was
just
saying
you
know,
since
we're
looking
spending
a
lot
of
time
on
continued
learning
as
well.
I
mean
the
existing
machine.
Learning
benchmarks
are
fine,
but
they're
all
you
know
really
constrained
and
limited.
I
was
just
wondering:
okay,
could
we
actually
look
at
this
as
a
way
to
explore
continue
learning
in
addition
to
some
of
the
sensory
motor
stuff,
you
know
it'd
be
nice
to
have
something
that
could
really
scale
to
handle
everything
that
we're
looking
at.
Instead
of
always
trying
this,
you
know
one
at
a
time
individual
benchmarks.
B
Yeah,
I
think
that's
consistent
what
I
was
recommending
in
some
sense.
You
know
picking
things
along
the
way
we
know
we
have
to
do
and
using
this
environment
to
implement
those
things.
The
way
we
think
we
should
be
implementing
continuous
learning
is
one
of
them,
but
then
again
you
wouldn't
want
to
pick
a
task
with
continuous
learning,
but
we
spent
90
of
our
time
solving
some
robotic
problems
in
a
way.
That's
not.
You
know.
C
B
Consistent
with
brain
cells,
so
you
know
even
even
a
single
cortical
column
with
continuous
learning
would
be
desirable,
even
if
it
just
means.
Oh,
I
have
one
finger.
I
can
you
know,
or
maybe
you
know
a
thousand.
I
don't
know
what
it
is
I'm
making
this
up,
but
you
know,
but
just
planning
out
which
principles
are
we
implementing
right
now
and
are
they
all
along
the
path
that
we
want
to
go
along
so
continuously.
C
Do
any
of
these
yeah
I
mean,
do
any
of
these
systems
come
with
sort
of
some
set
of
working
models,
or
is
there
something
you
can
or
that
may
have
I'm
just
saying
about
what
jeff
was
saying?
You
know
we
don't
want
to
spend
all
our
time,
looking
figuring
out
all
the
details
of
exactly
how
to
move
each
actuator
and
how
to
do,
but
if
there's
their
existing
models
that
do
that
we
could
kind
of
just
take
that
body
and
plug
in
our
brain.
If
you
will,
I
don't
know
if
there's
anything
like
that.
D
A
C
A
Are
available
and
and
it's
kind
of
the
way
we
choose
to
define
the
problem
we
can,
we
can
decide
to
do
it
a
full
end
to
end
or
we
can
decide.
You
know
I
just
want
to
tell
it.
No
just
go
there
or
just
pick
it
up.
I
don't
want
to
actually
tell
the
forces
it
should
apply
to
it
joint
of
the
r
right,
so
we
can
define
the
problem
on
that
level.
Definitely,
but
that's
that's
our
decision
right.
B
A
Yeah
yeah
yeah,
but
definitely
are
I
mean
there
are
agents,
we've
all
capabilities
and
we
can
just
import
and
use
those.
I
I
don't
know
specifically
like
how
it
works
in
habitat,
but
I
know
out
there
there
yeah,
we
don't
have
to
implement
these
things
ourselves,
but
we
should
think
about
it
because
I
don't
know
like
robotics
is
moving
to
end
to
end.
So
we
should
think
about
what
level
of
abstraction
we
want
for
account.
And
that's
that
comes
to
the
question
we
talked
about
yesterday.
C
A
C
Well,
you
know
he
asked
me
about
segmentation
maps.
I
said,
there's
no
really,
there's
no
such
thing
as
a
segmentation
map
in
the
braid.
Really,
you
know
we
have
individual
cortical
columns
that
are
making
models
of
the
sensory
data
coming
in,
but
there's
no
there's
not
a
pre-processing
step
where
you,
you
know,
create
a
detailed
segmentation
map
and
operate
on
that.
So
I
don't
think
we
want
that
for
sure.
B
B
B
If
your
goal
is
to
eventually
implement
true
intelligence
using
the
thousand
brain
theory,
you
could,
I
wouldn't
call
it
cheating,
but
you
could
rely
on
other
sensors
as
long
as
you
don't
violate
the
basic
principle,
which
is
the
learning
algorithm
learning
system
doesn't
have
sort
of
super
built-in
knowledge
about
what
kind
of
what's
being
represented
out
there.
You
know
it's
sort
of
some
it's
a
basic
sensor,
but
that
center
could
be
richer
than
the
one
we
you
know
than
a
retina
or
it
could
be
impoverished.
B
It
could
say:
oh
it's,
it's
not
in
color,
and
it
doesn't
have
this.
It
doesn't
have
that,
but
it
has
distance.
You
know,
that's
fine,
so
I
I
don't.
I
don't
have
any
other
specific
examples,
but
I'm
just
suggesting
that
we
can
develop
the
algorithmic
principles
and
choose
very
novel
sensors
as
long
as
we
don't
violate
the
biological
principles.
So
so
we
could,
you
know
we
don't
have
to
you
know.
The
visual
system
is
extremely
complex
and
they
go
from
pixel
level
representation
to
something
useful.
B
We
take,
you
know
a
quarter
of
our
near
cortex
or
you
know
20
or
25
percent.
So
you
know,
that's
a
you
know,
that's
a
lot
to
bite
off.
You
know
our
auditory
system
is
much
much
smaller,
so
you
know
anyway.
That's
that
would
be
my
answer.
That
question
is:
don't
try
to
try
to
come
up
with
a
problem
with
a
set
of
sensors
that
we
can
design.
As
long
as
we
don't
violate
the
principles
of
the
system,
we
want
to
create
meaning
the
algorithmic
principle.
B
Pre-Processing
stuff
in
the
in
the
sensor
right
in
the
sensor
again,
if
I
had
a
lighter
eyeball
that
wouldn't
be
a
neocortical
pre-processing
stuff,
it
would
be
a
sensor.
I
mean
the
retina
is
a
huge
preprocessor
right.
That's
what
it
is.
It's
a
you
know
it
does
all
these
calculations
and
so
on.
So
I'm
saying
you
could
you
could
you
could
heavy
and
load
the
sensor,
but
the
sensor
can't
be
like.
Oh,
I
detect
books
and
and
cars.
You
know
that
that
would
be
that's
not
adhering
to
the
principle.
B
The
sensor
could
have
basic
information
about
distance
and
maybe
even
texture
or
you
know
something
whatever
you
want.
You
think
you
could
you
know
physically
or
even
imagine
a
sensor
could
do,
but
it
would
be
at
a
very
sort
of
autumnal
or
you
know,
granular
level.
Not
as
oh.
This
thing
recognizes
books,
you
know.
Probably
that
would
be
the
wrong
thing.
B
You
know
again,
I
sort
of
hinted
at
this
earlier.
When
I
talked
about
you
could
have
multiple
cortical
columns,
let's
say,
and
they
could
all
be
co-adjacent
representing
you
know,
successive
patches
of
the
skin
or
you
could
have
multiple
cortical
columns,
each
one
representing
a
different
sensor
modality
and
in
some
sense
that's
how
the
brain
is
described.
You
know
each
part
of
a
column
doesn't
know
what
the
modality
is.
It's
just
got
some
inputs
and
it
tries
to
figure
out
what's
going
on
and
it
uses
other
particle
columns
to
help
it
figure
it
out.
B
B
You
sense
something
you
move,
something
that
what
you
sense
changes
and
that's
pretty
much
it
so
we
could
decide.
For
example,
you
could
have
a
multimodal
system,
so
I'm
guessing
anything
with
multiple
coordinates.
If
I
think
from
a
brain,
anything
with
multiple
quadruple
columns
is,
in
a
sense,
a
multi-modality
system.
It's
not
the
way.
Most
people
think
about
modality,
but
you
know
the
tip
of
my
finger
is
not
the
same
as
the
the
pad
of
my
hand.
Those
are
in
sentence.
Two
sensors
two
modalities.
B
If
you
want
to
call
it
that
from
an
algorithmic
point
of
view,
the
brain
doesn't
know
this
stuff
right.
It
just
doesn't
know
it
co-locates
things
and
says:
well,
I'm
assuming
I
can
communicate
with
you,
so
you
could
do
a
multimodal
system.
You
could
do
a
a
three-column
multi-modal
system,
one
column,
doing
something
about
remote
sensing.
That
would
be
something
like
vision,
another
one
doing
something
about
local
tactile
sensation.
B
I
mean
I
don't.
I
know
how
complex
of
a
problem
you
could
solve
with
single
columns
on
these
different
different
sensory
modalities.
So
again,
I'm
trying
to
if
you
think
about
the
problem.
Broadly
then
multi,
you
know
if
you
talk
about
multiple
modality
yeah
but
think
about
it
as
like,
I
have
multiple
columns.
What
are
they?
What
is
each
one
going
to
be
sensing.
D
B
A
B
Again,
I'm
suggesting
think
about
the
problem
you
might
solve
at
the
same
time
think
about
what
sensors
we
might
want
to
use
to
solve
that
problem
and
again
the
sensors
may
not
be
the
kind
of
sensors
we
have
in
humans.
So
audio
may
make
no
sense
at
all.
In
terms
of
some
of
these
problems
right,
then,
your
environment
has
to
have
some
sort
of
audio.
B
B
You
know
so,
but
you
know
I
don't.
I'm
not
gonna
call
lidar
audio
right.
So
I
I
think,
thinking
about
like
in
terms
of
hearing
and
vision
and
touch
is,
is
the
way
we
tip.
You
know
typically
roboticists
think
about
it,
but
we
have
a
little
bit
of
luxury
to
say.
No,
that's
not
really
what's
important.
What's
really
important
is
we
have
a
sensor
that
sends
assumptions
or
attribute
of
the
world
and
as
we
move
that
sensor,
where
that
sensation
is
coming
from
changes,
you
know
that's
the
key
item
there
right.
B
B
So
I
think
multi-modal.
I
think
they
look.
If,
when
we
succeed
at
this,
we
have
to
educate
the
world
about
this.
What
I
think
about,
I
think,
it'll
be
brilliant.
When
we
do,
I
mean
in
some
sense
we
can
say,
like
you
know,
think
about
this
problem
very
broadly
and
what
it
means,
and
so
the
names
we
put
on
these
sensors
is
kind
of
not
really
that
important.
It's
what
the
attributes
of
the
sensors
have
and
we
can
mix
and
match
them
all
different
ways,
and
it's
going
to
work,
no
matter
what
right.
B
So
that's
the
ultimate
goal
of
this.
So
I
think
it's
a
very
interesting
question.
We
want
to
limit
ourselves
to
sensors
that
might
be
very
co-located
like
multiple
patches
on
the
retina
or
all
the
patches
on
your
skin.
Or
would
we
want
to
take
multiple
sensors
that
are
in
some
sense
what
we
traditionally
call
different
modalities
and
they're
less
related?
Therefore
they're
harder
to
vote?
You
know
between
them,
but
they
can
vote.
You
know
what
I'm
saying.