►
Description
Heiko Hoffmann gives an overview of the “Neural Descriptor Fields” paper. He first goes over how the Neural Descriptor Fields (NDFs) function represents key points on a 3D object relative to its position and pose, and how NDFs can be used to recover an object’s position and pose. He then discusses the paper’s simulation and robot-experiment results and highlights the useful concepts and limits of the paper.
In the second half of the meeting, Karan Grewal presents the “Vector Neurons” paper. He first gives a quick review of the core concepts and terminology of the paper. Then he looks into the structure of the paper’s SO(3)-equivariant neural networks in detail and how the networks represent object pose and rotation. Lastly, Karan goes over the results of object classification and image reconstruction and points out a few shortcomings.
“Neural Descriptor Fields: SE(3)-Equivariant Object Representations for Manipulation” by Anthony Simeonov et al.: https://arxiv.org/abs/2112.05124
“Vector Neurons: A General Framework for SO(3)-Equivariant Networks” by Congyue Deng et al. https://arxiv.org/abs/2104.12229
Datasets mentioned:
Shapenet: https://shapenet.org/taxonomy-viewer
ModelNet40: https://3dshapenets.cs.princeton.edu/
- - - - -
Numenta is leading the new era of machine intelligence. Our deep experience in theoretical neuroscience research has led to tremendous discoveries on how the brain works. We have developed a framework called the Thousand Brains Theory of Intelligence that will be fundamental to advancing the state of artificial intelligence and machine learning. By applying this theory to existing deep learning systems, we are addressing today’s bottlenecks while enabling tomorrow’s applications.
Subscribe to our News Digest for the latest news about neuroscience and artificial intelligence:
https://numenta.com/news-digest/
Subscribe to our Newsletter for the latest Numenta updates:
https://tinyurl.com/NumentaNewsletter
Our Social Media:
https://twitter.com/Numenta
https://www.facebook.com/OfficialNumenta
https://www.linkedin.com/company/numenta
Our Open Source Resources:
https://github.com/numenta
https://discourse.numenta.org/
Our Website:
https://numenta.com/
A
A
C
Suppose
a
robot
is
provided
with
a
demonstration
of
grasping
an
upright
mug
and
hanging
it
on
a
rack.
We
propose
a
system
that
can
use
just
a
small
handful
of
these
demonstrations
to
repeat
the
task
on
a
new
mug
which
has
a
different
shape
and
is
placed
in
a
never-before-seen
pose
more
generally,
our
goal.
Okay,.
D
A
A
A
A
That
part
is
not
the
original
book
and-
and
this
is
also
a
figure
from
one
of
the
previous
works
that
they
cite
in
the
paper
and
so
for
example,
once
you
get
this
mapping,
you
can
get
a
really
continuous
function
of
the
surface
of
the
object
to
kill
a
bunny
and
here
so
you
so
you
learn
a
continuous
representation
of
what's
inside.
What's
outside,
you
could
also
interpret
this
as
a
probability
of
being
inside
or
outside,
because
but
the
function
you
learn
will
be
continuous.
A
A
So
so
they
use
a
function.
The
phi
is
the
function
of
the
x.
So
x
is
the
point
in
the
space
3d
point,
and
then
they
use
a
latent
representation
of
the
point
cloud
and
and
this
this
they
use
the
leverage
prior
work,
which
is
called
the
pointnet
and
the
pointnet
maps.
Those
point
cloud
onto
some
classes
and
in
between
you
have
some
latent
representation,
which
is
for
a
specific
pose
so
think
about
it
as
a
sort
of
like
a
course
trained,
co-specific
representation
of
the
point
cloud-
and
this
is
in
some
k-dimensional
space.
F
A
F
A
F
A
What
they
do
is
they
combine
those
two
right:
the
latent
representation
and
the
x
in
another
network.
So
this
is
the
capital
phi
and
this
network
is
structured
according
to
the
vector,
neurons,
okay,
and
so
so
imagine
this
produces
like
a
set
of
vectors
in
3d
space
or
some
concatenated
version
of
those
vectors
and
and
then,
and
then
you
combine
those
with
with
these
3d
vectors
in
this
vector,
neurons,
victor
neural
networks
and
then
so.
So
karen
will
talk
about
that.
A
A
D
A
D
A
D
F
F
You
know
you
have
a
set
of
points
and,
as
the
pose
of
the
rabbit
changes
in
world
coordinates,
the
actual
x
y
z
points
are
going
to
change
right
as
the
rabbit
rotates
right
and
this
this
this
big
epsilon.
I
don't
know
what
the
greek
letter
is
there.
I
guess
it's
absolute
yeah.
It
looks
like
a
big
big
ups,
curly
epsilon.
F
A
F
What
you're,
currently
seeing
exactly
so
this
this
big
epsilon
thing
is,
is
some
latent
representation?
That's
for
all
object
categories
and
all
poses
and
partial
point
clouds,
and
this
big
psi
tells
you
you
know,
given
that
latent
representation
will
tell
you
whether
the
point
is
inside
or
outside
that
full
object.
Yeah.
D
F
D
A
A
F
A
So
let's
say
they
have
learned
this,
and
so
so
the
capital
phi
max
maps
onto
this
occupancy,
but
then
they
again
want
to
have
some
latent
representation
of
the
capital
phi,
and
so
they
look
at
the
capital
phi
for
different
layers
in
there.
So
not
just
the
end
output
and
and
then
they
concatenate
those
values
and
and
that's
what
they
do,
what
they
call
the
the
this
this,
what
they
call
the
neural,
descriptor
or
neural
descriptor
field
field,
because
you
get
that
value
for
each
point
in
space.
A
So
if
you
rotate
your
point
cloud,
so
you
look
rotate
your
p
and
you
rotate
all
the
your
x,
so
they
should
rotate
in
exactly
the
same
way,
which
is
achieved
through
a
vector,
neural
vector,
a
vector,
neuron
network,
so
as,
for
example,
to
imagine
how
this
could
work
with
a
vector,
neural
network
or
a
very
simple
operation
in
there.
It's
like
the
vector
product
right,
so
you
get
the
cosine
between
two
vectors
and
and
that
thing
would
that
value
would
rotate
with
its
two
vectors
right.
A
So
they
achieve
sort
of
like
displacement,
vectors
right.
They
achieve
translation,
variance
by
subtracting
a
common
center
so
effectively
of
a
difference
between
points
and
then
achieve
the
rotational
variance
through
this
vector
neurons
so
effectively
deal
with
a
whole
bunch
of
cosine
angle
values.
So.
B
I
have
a
quite
I
have
a
question
about
that.
So,
if,
if
you're
viewing
this
point
cloud
from
one
side,
we're
only
seeing
one
half
of
the
cup,
how
are
you
getting
a
common
center
there,
because
the
the
centroid
of
that
is
actually
not
the
center
of
the
object?
It's
the
center
of
the
partial
point
clouds
or
there's
going
to
be
a
displacement
there.
A
Yeah
yeah
yeah
that
that's
actually
a
very
good
issue.
Yeah,
I
didn't
see
it
at
rest,
but
I'm
not
sure
if
I
missed
it,
I
didn't
see
it
at
rest
and
yeah
it's.
I
guess
they
assume.
You
see
enough
of
the
point
cloud
that
is
sort
of
close
right
center
is
so
close,
but
yeah
it's
if
you
see
just
let's
say
you
just
see
the
handle
right.
Everything
else
is
covered
I'll
screw
it
up
for
sure
and.
F
A
What
my
feel
is
why
it's
sort
of
sort
of
robust
to
that
thing,
it's
because
at
the
end
of
the
day
they
map,
let's
say
they're,
interested
in
the
handle
here
right
and
then
so
they
map
a
set
of
point
clouds
like
around
the
handle
they
rotate
it
and
translated
it.
Then
it
matches
here
this
descriptor
field
right.
This
imagine
this
is
the
color
here
is
like
a
landscape,
this
local
landscape
around
the
cup.
That's
essentially
what
you
want
to
match
and
then
it
does.
F
A
F
I
think
another
issue,
another
thing:
that's
helping
them
is
because
they're
training
with
partial
point
clouds
they
will
have
trained
with
so,
let's
say
kevin
in
your
example.
If
they're
only
seeing
half
of
the
cup,
they
will
have
also
trained
with
a
view.
That's
probably
just
you
know
a
similar
half
of
the
cup,
and
so
they
will
have
trained
with
that
particular
translation
or
mean
point.
F
F
A
Yeah,
oh
okay,
yeah,
which
is
different
from
the
object
yeah.
This
is
a
good
point.
It's
with
just
with
respect
to
a
specific
p
and
then
formally
it's
it's
strictly
a
translation
variant
right.
You
subtract
the
center
yeah
so
and,
and
then
so
so
this
is
how
they
get,
how
the
follow
a
point
on
the
object
right.
A
A
A
You
wanna
minimize
that
for
a
whole
set
of
points-
and
this
way
you
can
imagine
like
visually
right
for
this
one
here-
that
there's
only
a
certain
rotation
where
those
colors
across
multiple
points
will
match
and
and
and
they
show
they
so
this.
This
doesn't
show
the
rotation
here,
but
it's
just
for
they
show
based
on
this
demonstration.
F
F
A
A
And
then
so
yeah,
so
so
this
relates
here
to
what
I
mentioned
with
how
how
do
they
get
this
set
of
points,
and
they
essentially
do
it
based
on
the
on
the
bounding
box
around
the
so
you've
got
your
gripper.
So
this
this,
this
grey
thing
is
the
robotic
it's
like
a
linear,
gripper
opens
and
closes,
and
then
and
when
they
grab
at
a
certain
location,
they
see.
What
is
the
bounding
box
of
the
the
gripper
piece
here,
which
is
just
below
the
gripper
and
based
on
that
they
sample
a
bunch
of
points
like
this.
A
One
here
looks
like
some
random:
don't
remember
exactly
what
they
did.
It
looks
like
some
gaussian
random
sampling
here
and,
and
then
they
represent
those
two
points
in
terms
of
the
descriptor
value,
and
you
can
for
this
location
you
can
assign
each
according
to
the
function,
f
a
value,
and
then
you
just
rotate
and
what
you
do
in
the
optimization.
A
You
rotate
that
coordinate
system
on
the
bottom
row
that
just
visualize
yeah.
Why
you
want
to
do
it
relative
to
this
local
area?
Because
if
you
don't
know
the
shape
of
the
cup
this
bottom
area,
you
would
try
to
grasp
it.
Let's
say
your
coordinate
system
rotates
relative
to
the
bottom
right
and
you
couldn't
grasp
it
up
there.
A
So
this
is
an
example
of
this
optimization,
so
they
do
an
optimization
of
of
the
rotation
and
translation.
So
this
is
the
full
four
by
four
rotation
translation
matrix
applied
to
a
to
each
point
and
then
applied
to
the
reference
frame
of
this
end
effector
and
then,
as
I
mentioned,
it
is
rotated
until
they
find
the
best
match
regarding
the
individual,
neural,
descriptive
field
of
these
individual
points,
and
these
this
shows
part
of
the
iteration
of
getting
there
to
that
point.
So
it's
sort
of
like
the
mental
rotation
right
it
takes.
A
Function
I
showed
before
yeah
this.
This
doesn't
show
the
whole
thing
just
for
one
point,
but
but
you
have
this
for
each
point:
yeah
and
then
you
let's
say
you
rotate
you.
Let's
say
this
is
fixed
and
you're.
You
want
to
adjust
this
one.
Then
you
rotate
all
the
x
you
put
in
here.
Your
rotation
or
translation
function
in
here
matrix,
and
this
is
all
a
linear
operation
right
and
then
it
goes
to
the
whole
network
and.
G
A
F
A
A
A
B
I
I
guess
if,
if
I
were,
if
I,
if
I
were
trying
to
abstract
the
problem,
saying
I
want
to
find
where
the
grip
points
are
on
this
thing
and
it's
a
cup
and
it's
you
know,
concave
inside
you
know
anywhere
along
the
rim,
would
be
a
grip
point.
The
handle
is
a
grip
point,
so
I
mean
I'm
seeing
in
this
in
this
potential
function
or
this
energy
function
or
whatever
they,
they
were
calling
it
they've
kind
of
highlighted.
B
You
know
the
the
handle,
but
there
are
other
grip
points
on
there
that
when
they,
when
they
illustrate
that
when
they
they
take
the
when
they
try
to
train
the
robot
through
that,
like
I'm,
not
seeing
you
know
either
the
top
or
bottom,
I'm
not
even
sure,
which
is
the
top,
which
is
well
okay.
Obviously
it's
on
on
the
left
on
the
main
one
you
can
see
where
the
top
is,
but
it
wouldn't
even
it
looks
like
it,
wouldn't
even
attempt
to
go
for
gripping
it
by
the
by
the
top
rim.
A
Yeah,
in
their
case,
it's
really
the
grass
point
is
very
specific
to
a
demonstration
and
it's
specific
to
how
how
that
location
of
the
grass
point
is
sort
of
in
in
the
object.
Centric
coordinate
system
by
having
a
relationship
to
the
the
features
of
the
whole
object
right,
which
includes
the
handle
everything
so
whenever
they
demonstrate
some
arbitrary,
grasping,
let's
say
on
on
the
cup
here,
everything
is
this:
this
would
be
in
relationship
to
the
sort
of
overall
shape
of
the
whole
thing.
A
A
You
know
no
well,
if
you
demonstrate
it
first,
the
right
way
up
and
say
this
is
your
grip
point
and
then
you
turn
it
around
as
long
as
they
didn't
get
confused.
There's,
as
you
see
here
right,
if
there's
enough
symmetry,
then
you
might
not
know
which
way
is
the
right
way.
So
I
could
imagine
error
like
that
right,
but
then
it's
if
it's
sufficiently
anti-symmetric
regarding
up
and
down,
then
it
should
still
pick
it
up
the
right
way
around
and
turn
it.
B
B
Notion
of
of
you
know
where
what's
concave
and
what's
convex
on
the
object
and
as
you
say,
it
might
not
have
sufficient.
You
know
wherewithal
to
to
know
the
exact
I
mean
it
could
ram
the
bottom
through
the
through
the
anyway.
I
I
I
I
I
I
get
the
point
that
this
demonstrating
a
specific
right,
pose
translation
and
variant
way
of
finding
a
particular
feature
robustly
on
on
the
object,
I
mean,
I
have
to
assume
that
you
know
whenever
they
there's
other
stuff
going
on.
B
In
the
background
for
the
for
the
robot
arm,
planning
and
execution
of
move
like
it,
it
knows
not
to
try
to
run
the
gripper
through
the
point
cloud
and
it
tries
to
orient
itself
so
that
the
gripper,
you
know
you
know,
comes
orthogonally
into
you
know.
Whatever
that
I
mean
I
I'm
guessing,
there's
a
lot
more
planning.
That's
kind
of
you
know.
A
Yeah,
well
they
say
regarding
the
planning
and
like
robot
movement
execution.
They
use
something
off
the
shelf.
They
didn't
specify
exactly
what
but
yeah
that
that's
sort
of
a
solved
problem
and
you
can
include
collision
avoidance
in
that
too,
and
so
essentially
what
they
recover.
They
recover,
not
just
the
grass
point,
but
since
they
recover
the
whole
six
degrees
of
freedom,
they
require
the
orientation
how
the
gripper
would
hold
it.
Let's
say
in
this
particular
orientation
and
then
the
planner
solves
how
to
get
from
any
starting
point.
To
that
point,.
D
A
A
A
So
they
essentially
show
that
their
approach
here
is
works
much
better
because
of
all
the
rotational
variants
built
in
works
much
better.
If
you
change
to
an
arbitrary
pose-
and
you
see
here-
first,
there
may
be
sort
of
maybe
yeah.
The
other
approach
cannot
handle
the
role,
but
in
some
cases
this
sort
of
works
similar
and
but
then
their
their
approach
is
vastly
superior.
If
you
put
in
an
arbitrary,
pose
yeah
another
comment
about
the
results
here.
A
They
also
demonstrate
this
in
the
robot,
as
you
see
in
the
video
at
the
beginning,
but
they
didn't
mention
the
quantitative
results
of
the
robotic
system
set
up,
and
so
so
I
suspect,
if
it
would
be
anywhere
close
to
that
they
probably
would
have
mentioned
it,
and
and
so
it's
probably
very
low
and
and
then
from
experience.
I
know
it's
it's
easy
to
much
easier
to
get
a
video,
a
nice
video
going
and
getting
the
system
really
reliable,
and
so
it's
maybe
at
the
10
20
it
works.
A
A
A
Flexibility
regarding
the
kind
of
deformations
you
would
still
recognize,
though
it
would
be
interesting
to
see
what
is
really
the
limits
there.
Regarding
that
information,
I
I
suspect,
with
this
relatively
small
amount
of
deformation,
it's
okay
like
different
cups,
they'll,
sort
of
still
look
very
similar,
but,
like
some
of
the
things
we
discussed
like
a
t-shirt,
can
look
totally
different
or
like
the
dolly
clocks,
it
will
probably
fail
in
such
a
case
also.
I
think
it's
interesting
in
terms
of
that's
a
fairly
interesting
demonstration
of
the
robot.
A
Although
the
I'm
not
100
sure
about
it.
If
the
vector
network
yeah,
I
need
to
check
how
they
actually
train
the
night
vector
network
in
there,
because
because
they
they
leverage
this.
This
other
network
in
there
which
maps
onto
the
latent
space-
and
this
is
was
trained
through
across
multiple
objects.
A
Now,
I'm
not
sure
if
they
trained
on
a
per
object,
specific
because
case
this,
this
mapping
on
the
capital.
Five
remember
they
have
this
occupancy
map,
they
need
to
train
with
a
net
vector
network,
and
that
depends
on
a
specific
post
or
specific
posts.
You
train
to
recover,
and
so
was
this
done
in
an
object-specific
case.
It's
not
clear
to
me
and
then
yeah.
As
I
mentioned,
the
success
rate
on
real
robots
is
probably
still
low.
A
So
I'm
sorry
for
going
overtime,
10
44.,
maybe
any
any
follow-up
questions
they
can
discuss
offline.
Just
send
me
a
question
and
let's
now
jump
into
the
vector
neurons,
which
is
a
that's
a
very
neat
concept
to
get
working
with
the
point
clouds.
How
do
we
represent
the
point
cloud
in
some
rotational
variant.
E
E
So
what
is
the
paper
about
so
they're
constructing
representations
from
their
inputs
that
that
reflect
the
pose
of
the
object
and
also
satisfy
some
interesting
interesting
properties
such
as
mainly
equivariance,
which
they
talk
about
which
I'll
get
into
so
and
just
just
like
in,
and
then
the
paper
hike
we
talked
about.
They
have
they
have
these
point
clouds
and
that's
what
they're
they're
working
with
and
constructing
these
representations
that
you
know,
as
you
rotate
the
object
in
a
certain
way.
E
Okay,
so
first,
I
think
it's
it's
worth
going
over
some
of
these
concepts
that
which
are
very
crucial
to
the
paper.
The
first
is
invariance,
so
function
f
is
in
variance
to
transformation
r.
If
you
know,
if
you
apply
that
trans,
if
you
apply
that
transformation
prior
to
applying
f
or
after
it
doesn't
matter,
you
still
get
the
same
result
or
sorry
wait.
E
Think
I
have
them
flipped
here,
yeah,
okay,
so
I
have
them
flipped
at
the
yeah.
The
first
number
doing
that
so
invariant.
So
so
these
definitions
should
actually
be
flipped.
Otherwise,
I
think
this
idea
of
the
cat
being
translated
and
you're
still
getting
a
cat
that
I
that
that's
still
that's
still
true
for
invariance.
So
I
have
that
flip
that
that's
my
bad
okay,
so
so
in
invariance.
E
Okay,
so
we're
used
to
working
with
scalar
neurons
in
neural
networks
today,
so
a
a
hidden
layer
would
look
like
you
know.
You
have
these,
these
z
values
and
you
just
have
a
vector
of
them
and
so
they're
all
they're.
All
there's
just
a
list
of
scalars
that
you
have,
whereas
vector
neurons,
what
the
others
propose
is
instead
of
having
a
single
scalar
for
each
for
each
unit,
you
replace
it
with
a
3d
vector.
E
So
now
we
go
from
z,
which
is
just
a
real
number
to
a
to
a
three-dimensional
real
number
through
a
three-dimensional
vector
v.
So
this
is
diagrammatically
what
it
would
look
like,
and
so
you
get
this
you
get
this
now.
You
have
your
your
latent
state.
Is
this
matrix
of
the
number
of
the
number
of
entries
that
you
have
there,
which
is
c
times
three,
because
each
of
them
are
three
dimensional.
E
I
should
note,
though,
that
just
because
it's
3d
doesn't
mean
that
it's
representing
an
xyz
location,
it's
3d
because
we
want
we
want
these
to
be
in
ver
equivariant
to
3d
rotations
in
a
in
a
in
a
latent
space.
So
it's
so
so
so
nowhere
do.
The
authors
mention
that
that
the
three
dimensions
here
are
representing
xyz,
although
that
that's
the
case
with
the
input.
E
So
so
now
the
author's
constructed
a
bunch
of
different
type
of
layers
in
the
network.
So
you
know
you
have
you
have
linear
layers?
You
have
non-linearities
batch
normalization,
pooling
all
that
and
they
talk
about
how
they,
how
they
are
constructing
these
layers,
specifically
for
vector
neurons
such
that
it
satisfies
rotation
equivariance
if
you
rotate
it
around
the
origin
in
that
space.
So
first
is
a
linear
layer.
E
E
Next,
they
have
a
non-linearity,
so
you
want,
in
addition
to
having
just
linear
linear
layers.
You
also
want
to
be
able
to
apply
a
non-linearity
and
they
specifically
focus
on
a
relu
type
of
non-linearity
here.
So
for
so
so
remember.
V
is
this:
is
this
c
by
three
matrix
and
so
each
of
so
v
you
can
think
of
it
as
cont,
basically
being
a
list
of
these
three-dimensional
vectors
which
is
given
by
small
v
here
now,
how
do
you
apply
a
non-linearity
to
that
which
also
satisfies
rotation
equivalence
so
how
they
do?
E
E
So,
each
for
each
three
dimensional
vector
they're
computing,
they're
computing,
a
separate
these
two,
these
two
vectors
which
they
call
a
direction
vector,
k
and
a
feature
vector
q,
and
they
both-
and
these
are
both
3d
vectors
in
a
3d
space
and
basically,
what
they
do
is
they
apply
their
their
their?
I
guess
higher
dimensional
version
of
relu
is
taking
q
to
be
the
output
if
it's
in
the
same
half
space
as
k.
So
that
means
like
the
angle
between
q
and
k
is
less
than
90
degrees.
E
You
would
take
that
to
be
the
output
of
the
non-linearity
here,
so
so
this
is
the
equivalent
of
you
know
of
the
of
the
input
to
the
relu
being
less
than
zero.
You
just
you
just
ignore
the
actual
value
and
you
you
you
take
zero
here.
You're
just
project
projecting
it
onto
this
plane.
Now
it
is.
I
did
find
it
a
bit
arbitrary
that
they're
they're
doing
it.
E
This
way
that
they're
picking
this
k
and
the
q,
but
they
did
mention
in
the
appendix
that
sometimes
instead
of
having
this
q,
which,
which
is
a
linear
transformation
of
the
input,
they
can
just
use
the
input
small
v
itself
so
and
and
the
reason
why
they
have
this
direction
k
as
opposed
to
just
picking
as
opposed
to
picking
the
the
unit
vector
along
the
z
axis
as
their
as
their
direction
vector
is
because
they
want
it.
They
want
this
k
to
change
as
the
pose
of
the
input
changes.
E
A
D
A
B
E
So
this
is
also
rotation
equivalent.
I
then
I
didn't
bother
to
show
the
the
proof
of
it
here.
I
took
their
word
for
it
and
finally,
we
have
an
invariant
layer,
so
this
would
be
good,
for
you
know,
towards
the
end
of
the
network.
If
you
have
a
classifier
layer,
you'd
want
it
to
be
invariant
to
whatever
the
whatever
pose
you
have,
and
so
here
how
they
achieve
invariance
in
a
layer.
I
found
this
to
be
a
bit
strange.
E
How
they
did
it
was
they
have
so
so
so,
basically
they
define
their
invariant
function
like
this.
So
again,
it's
just
it's.
Just
a
single
matrix
multiplication
where
your
matrix
v,
which
is
the
input
to
the
layer,
is
multiplied
by
a
matrix
t,
which
is
a
three
by
three
matrix,
but
you
wanted
to
in
order
to
satisfy
invariance
they
actually,
the
way
they
can
do.
E
This
is
that
they
make
the
matrix
they
get
the
matrix
t
they
derive
this
as
the
output
of
an
of
an
mlp,
and
that
mlp
is
is
just
a
linear
layer
which
I
talked
about
two
slides
earlier,
so
that
is
equivariant,
which
means
that
t
is
equivariant
here
and
and
the
input
to
that
mlp
is
they're
regressing
v,
along
with
something
else
on.
Actually
I
don't
even
know
what
I
don't
recall
exactly.
What
the
target
here
is.
E
This
will
also
be
rotated,
because
this
side,
this
guy
satisfies
the
equivariance
property.
Then
you
just
get
the
rotation
matrix
multiplied
by
its
transpose
and
since
the
rotation
matrix,
the
3d
rotation
matrix
is
worth
an
orthonormal.
Multiplying
it
by
its
transpose,
will
just
give
you
the
identity,
so
you
end
up
getting
just
the
same
output
as
you
had
before.
So
this
is.
This
is
just
showing
that
it's
rotation
in
variant
so.
F
F
F
Yeah,
because
it's
trying
to
do
our
trying
to
learn
this
r
transpose
or
no,
no
sorry
it's
if
you,
if
you
could
without
that
non-linearity
in
between
this,
is
very
easy
to
do
right
as
you
just
it's
a
linear
mapping
and
you
do
the
inverse
mapping
and
you're
back
to
the
original
vector
right.
So
the
only
reason
this
is
difficult
is
because
they
have
that
non-linearity
in
the
middle.
E
F
A
F
A
E
I
am
not
sure
if
that's
how
they're
doing
it
I'll,
I
all
I
cite
as
some
way
to
obtain
a
matrix
t
that
is
rotation
equivalent,
and
so
I
thought
this
is
just
this
is
possibly
just
one
way
to
do
it.
You
just
need
this
guy
to
be
rotation
equivariant.
Once
you
have
that,
then
this
layer,
the
way
they've
set
it
up,
will
be
rotation
invariant.
B
Could
it
be
that
you're,
starting
out
with
this
one-dimensional
of
this
particularly
dimension
space,
and
are
you
maybe
projecting
to
a
lower
dimensional
space
and
in
that
space
the
rotations
are
kind
of
removed
and
you
so
you
still
get
the
same
output
but
you're.
It's
not
necessarily
you're,
not
necessarily
recovering
the
same
dimensionality
as
you
started
with,
but
you're.
Getting
a
result
that
is,
is
invariant.
B
Yeah
well
this
this
operation
that
that
we're
that
you're
trying
to
do
when
you're
saying
you
know
I
I'm
trying
to
I'm
trying
to
understand
what
could
be
preserved
past
a
rally
that
relates
to
invariance
and
it's
I'm
having
a
hard
time.
You
know
grasp
I
mean
I'm
I'm
I
I
don't
want
to.
I
don't
want
to
go
down
the
same
path.
B
Subut,
I
didn't
say
we're
trying
to
exactly
undo
it
because
I
don't
think
that's
possible
because
you're
losing
information
through
the
irelia,
but
I'm
just
wondering
if
it's
looking
for
some
kind
of
output
that
is
invariant
to
the
rotation.
That
is
useful,
but
it's
not
necessarily
as
dimensioned
as
the
original
input
is.
What
I'm
trying
to
what
I'm
trying
to
understand,
whether.
B
F
B
A
E
Okay,
I
think,
should
move
on.
So
there
are
a
couple
of
data
sets
that
that
they've
that
they
worked
with,
I
thought
it
would
be
worth
looking
at
them,
so
one
is
shapenet
these.
These
were
new
to
me.
I
didn't
know
about
these
data
sets
before,
but
here
you
know
you
have
these.
I
guess
point
clouds
are
different
objects,
and
you
can
see
here
on
this
thing
that
drops
down
that
they're,
giving
you
all.
E
A
I
think
these
are
good
ones.
Yeah
also,
I
think
they're
nice
from
the
angle
of
having
some
kind
of
comparison,
for
example,
on
the
model
net
40,
there's
a
lot
of
published
work
showing
some
recognition
performance,
and
so,
if
you
want
to
convince
ourselves
that
ours
is
better
or
it
does
work,
that's
one
thing
to
test
it
on
and
compare
against
others.
E
This
one
set
up
b
here.
I
thought
this
was
the
more
interesting
one
where,
when
you're
training
it,
it's
only
rotated
around
the
z
axis,
but
when
you're
testing
it
it's
any
3d
rotation
applied
to
it.
So
that's
significantly
harder
right
now,
because
now
you're
actually
viewing
it
from
from
from
viewpoints
that
you
did
not
observe
during
training
and
then
the
last
one
is
you've,
seen
a
bunch
of
rotations
during
training
and
during
testing.
E
Here
I
highlighted
in
red
they
actually
did
the
best
on
the
scenario
b,
which
is
you're
you're,
viewing
it
from
significantly
fewer
viewpoints
during
training
than
testing.
They
seem
to
have
gotten
the
best
score,
although
it
seems
like
these
models
down
here
got
pretty
close,
so
they
were
able
to
do
pretty
well
on
that
and
then
there's
reconstruction.
E
I
think
there
are
a
lot
of
details
here,
which
I
didn't
fully
grasp
because
I
just
glossed
over
it,
but
here
what
they
did
is
again.
They
had
three
different
scenarios,
where
they're
training,
with
training
and
testing
where
there
are
no
rotations
applied
and
then
b
is
the
harder
scenario
where
you're
training
with
with
no
rotations
but
then
you're
supposed
to
identify
it
from
any
any
3d
rotation
applied
to
the
object
and
then
the
last
one
is
same
as
before.
E
So
I
suspect
how
they're
doing
this
is
that
you
know
the
the
encoder
part
so
that
they
have
an
encoder
and
a
and
a
decoder,
and
that's
what
they're
using
for
reconstruction
and
these
are
using
their
vector
neurons.
So
the
encoder
is
entirely
rotation
equivalent
and
that's
just
probably
because
they're
they're
using
those
eq
rotation
equivalent
layers
throughout,
and
so
once
you
get
a
latent
code.
E
You
can
actually,
you
can
actually
apply
the
transformation
to
that
and
the
decoder
should
be
able
to
reconstruct
something
in
the
pose
that
you
want,
and
it
should
reflect
that
rotation
that
you've
applied
to
the
latent
code.
So,
but
I
think
there
might
be
some
details
here
that
I'm
missing
so
so
I
don't
want
to
talk
about
this
too
much,
but
they
did
show
they
did
show
visual
results
here.
D
D
A
Yeah,
I
also
find
that
the
table
three
on
the
same
page
as
this
one
nice,
because
just
comparing
the
two
different
networks
right,
making
it
with
the
vector,
neurons
and
and
the
other
corresponding
network
from
the
literature.
And
it
totally
drops
in
terms
of
performance
of
classification
accuracy
in
the
b
case
here.
F
G
Here
with
the
table
yeah
by
the
way-
oh
yeah
yeah,
I
was
just
wondering
it's
interesting-
that
from
a
to
b,
they
have
zero
performance
change,
so
it
is
exactly
the
same
performance
between
condition
a
and
b.
But
then,
if
you
train
on
3d
rotation,
they
get
better.
I
guess
this
is
just
strange
that
there's
no
change
in
performance
between
a
and
b
for
them.
A
E
So,
but
you
know,
but
I
found
that,
like
you
know,
if
you
can
look
at
it
from
different
rotations,
that's
that's
still
pretty
good,
so
I
guess
what
they're?
What
they're?
When
they
talk
about
affine
transformations,
it
might
be,
you
know
zooming
out
versus
zooming
in
or
can
you
can
you
still?
Can
you
identify
the
object,
even
regardless
of
rotation?
Also,
if
you're
translating
it
around
and
then
the
second
thing
is
like
mentioned
here
right,
the
reconstructions
aren't
as
good
as
the
baseline
reconstructions.
E
In
this
case
I
mean
it's
a
little.
It's
a
little
bit
the
baseline's
a
little
bit
better.
So
I
wouldn't
I
didn't
take
that.
I
didn't
worry
too
much
about
that
one,
but
I
guess
I
guess
this
airplane
here
is
a
bit
better
than
this
one,
but
but-
but
I
guess,
since
since
they
are
able
to
deal
with
these
other
scenarios
b
and
c,
it's
still
pretty
good.