►
Description
Modern machine learning has origins in human learning, taking cues from human perception to build, train and evaluate machine learning models. As machine learning (ML) has begun to outperform humans in many challenging tasks, the focus has shifted from modeling humans to simply improving the performance of these ML models. We focus instead on what can be learned from human perception to improve these models and make them more transparent and understandable. With many applications of machine learning having real-world impacts on humans, we consider explainability essential for these models. In this talk, I will detail our approaches to explainable attribute recognition, prominent feature recognition, and face recognition. With each problem, we will highlight our influences from human perception.
B
Today's
talk
is
co-sponsored
by
IEEE
young
professionals,
affinity,
group,
High,
Tripoli
women
in
engineering
affinity
group
and
the
IEEE
control
system,
Society
chapter
of
San
Diego
section.
It's
my
great
pleasure
to
introduce
Professor
Emily
Han
from
the
University
of
Nevada
Reno
Emily,
received
her
PhD
degree
from
the
University
of
Maryland
College
Park
in
2018
and
joined
UNR.
Her
primary
research
interest
lies
in
the
intersection
of
computer
vision
and
machine
learning,
with
an
emphasis
on
understanding
human
perception
she
has
performed
on
several
IR
projects
related
to
face
recognition
and
action,
recognition
from
images
and
video.
B
Our
research
focuses
on
Bridging
the
Gap
between
human
and
computer
vision,
using
research
in
human
perception
and
machine
learning.
Emily
is
the
director
of
the
machine
perception
Lab
at
UNR.
The
MPL
has
active
projects
in
computer
vision,
natural
language
processing
and
machine
learning,
and
this
research
is
funded
by
several
active
grants
through
the
National
Science
Foundation
she's,
very
passionate
about
teaching
and
is
a
part
of
several
NSF
reu
and
ret
programs
at
UNR
Emily
thanks
a
lot
for
being
with
us
today.
The
floor
is
yours:.
C
Thank
you.
What
a
nice
intro,
okay,
so
I'm,
just
gonna
minimize
that
and
hopefully
everybody
can
see
my
screen
and
I'm
just
gonna
get
started
so
for
this
talk,
I
I
gave
opala
title
of
using
human
perception
to
inform
machine
perception.
Usually
my
talk.
Titles
are
are
making
life
accessible
because
it's
kind
of
what
we
try
to
do
in
my
lab.
C
And
so
this
is
our
sort
of
lab
logo
and
I
like
to
take
a
second
to
talk
about
it.
Because,
typically,
when
you
talk
about
machine
learning
or
or
machine
perception,
you
see
like
brain
logos
and
and
artificial
brains,
and
things
like
that
and
we've
just
got
a
paper
airplane
and
the
reason
for
that
is:
we're
really
trying
to
build
wearable
systems
for
individuals
with
social
skills
deficits
to
improve
their
social
interactions
in
their
lives,
and
we
do
that
through
machine
perception,
whether
that's
visual
or
audio
or
otherwise.
C
And
so
the
idea
is
that
we
all
kind
of
start
with
the
same
piece
of
paper,
and
we
can
all
fold
it
into
a
paper
airplane
that
can
fly
right
it
can.
We
can
repurpose
our
starting
point
to
make
something
that
allows
people
Liberty
and
freedom
in
their
lives,
and
so,
like
I,
said,
our
long-term
goal
really
is
to
develop
a
wearable
device
for
people
with
social
skills
deficits.
And
what
does
that
look
like?
So
we
generally
speak
about
people
that
are
on
the
autism
spectrum.
C
C
Right
is
really
important,
but
most
of
our
life
right
isn't
about
getting
from
point
A
to
point
B.
It's
really
about
the
social
interactions
that
we
have
and
so
to
have
a
really
rich
life.
It's
important
to
be
able
to
have
realistic
social
interactions,
and
so
that's
our
long-term
goal.
And
how
do
we
address
that?
While
we're
working
on
the
back
end
pieces
of
that
and
that
involves
computer
vision,
natural
language
processing
and
machine
learning?
C
But
in
this
talk,
I'm
going
to
focus
pretty
much
just
on
the
computer
vision
aspect
with
some
of
the
machine
learning
I
left
some
of
the
natural
language
processing
stuff
that
we're
doing
out
of
this
talk.
So
for
our
research
directions,
most
everything
covers
these
four
pillars.
The
first
one
is
adding
common
sense
to
machine
learning,
learning
from
noisy
data,
explainable
machine
learning
and
bias,
and
fairness
and
machine
learning
and
I'm.
C
C
We
really
try
in
our
work
to
add
common
sense
to
all
of
our
machine
learning
applications,
and
so,
rather
than
you
know,
just
using
machine
learning
for
what
it
is
right
and
and
using
statistics.
We
also
try
to
add
World,
Knowledge
and
common
sense
to
our
models.
The
way
that
they're
trained
the
way
that
they're
evaluated
and
so
I'll
talk
about
that
as
I
talk
about
our
projects,
the
second
one
is
learning
from
noisy
data.
C
When
you
think
about
noisy
data
a
lot
of
times,
you
think
about
like
a
noisy
signal
like
this
and
trying
to
figure
out,
you
know
what
the
actual
signal
is
right.
Maybe
this
blue
signal
is
the
Real.
The
true
signal
and
we've
got
this
sort
of
red
noisy
signal,
as
well
as
this
pink
noisy
signal.
But
realistically
what
we're
dealing
with
right
is
yeah
a
lot
of
the
time
and
so
we're
trying
to
describe
people.
C
So
one
example
might
be
that
we
search
we
do
a
Google
search
for
a
blonde
woman
and
the
first
thing
you
might
notice
that
there's
a
couple
of
brunettes
in
here,
and
so,
if
you
were
to
just
scrape
this
data,
it
wouldn't
be
perfectly
clean
right.
So
we
wouldn't
be
guaranteed
that
every
woman
that
we
scraped
was
blonde.
In
addition
to
that,
you
might
notice
that
almost
all
of
these
women
are
white,
and
so
that's
another
issue
right
of
dealing
with
this
kind
of
noise.
C
And
so
we
try
to
be
able
to
take
this
information
which
is
noisy
but
there's
valuable
signal
in
it
and
learn
from
that,
but
not
learn
everything
from
it
and
then
realistically,
when
we're
talking
about
building
wearable
devices
that
can
provide
feedback
to
people,
it's
we
need
to
be
able
to
handle
the
noisy
world
right.
The
world
is
noisy,
there's
a
lot
going
on,
and
so
we
focus
a
lot
on
being
able
to
handle
noisy
data
in
these
systems.
C
C
In
a
number
of
ways,
one
is
write,
post-hoc,
processing,
being
able
to
really
explain
when
and
how
the
method
works
and
the
other
is
incorporating
explainability
into
the
training
of
the
model
right
so
making
sure
that
the
model
itself
is
explainable
rather
than
just
trying
to
pull
back
the
curtain
afterwards
and
then.
Finally,
we
try
to
address
bias
and
fairness
and
machine
learning
as
much
as
possible
and
basically
in
every
project.
C
And
so
we
want
to
make
sure
you
know
we're
dealing
with
bias,
but
we're
also
being
fair
to
all
of
these
users
that
might
be
using
the
data
so
before
I
dive
into
my
projects,
I'm
going
to
give
a
little
bit
of
background,
that's
kind
of
important
to
understand
what
exactly
we're
doing
and
what
kind
of
data
we're
working
with.
So
a
lot
of
what
we
do
is
working
with
describable
features,
so
we're
interested
a
little
bit
less
in
recognizing
people
and
a
little
bit
more
in
recognizing
what
people
look
like.
C
There's:
a
sort
of
a
subtle
distinction
there.
So
we're
not
really
that
interested
in
verification
or
face
recognition,
we're
much
more
interested
in
being
able
to
take
a
face
and
say
what
that
person
looks
like
right,
so
actually
having
this
sort
of
describable
features
is
really
what
we're
focused
on,
and
we
do
that
through.
Basically
all
of
our
work
and
so
attributes
are
just
human
describable
visual
features
and
we
mostly
work
with
facial
attributes.
C
So
these
are
all
pictures
of
the
same
woman,
and
so
you
might
think
that
you
know
she
looks
pretty
similar
in
all
of
these
images,
but
you
could
separate
her
by
her
different
attributes
right.
So,
if
you
use
wavy
hair,
you
would
have
the
negative
instances
on
the
left
and
the
positive
instances
on
the
right.
If
you
had
smiling,
you'd
have
negative
on
the
left
and
positive
on
the
right
and
we'd
have
a
completely
different
pairing
than
we
did
before.
C
Similarly,
with
bangs
right,
it
separates
the
data
like
this,
and
so
we
we
love
this
describability,
because
it
allows
us
to
describe
a
person
but
also
to
describe
an
image,
and
so
we'll
talk
a
little
bit
more
about
that
later.
When
we
talk
about
using
these
types
of
attributes
for
face
verification,
but
before
we
do
that,
we'll
talk
a
little
bit
about
the
data
set
that
we
use
for
most
of
our
work.
C
We
use
this
data
set
called
celeb
a
it
has
200
000
images
about
160
000
for
training
and
twenty
thousand
each
for
validation
and
testing.
It
came
out
in
like
2015
2014
2015,
something
like
that
and
and
it
was
the
largest
scale
data
set
out
there
for
facial,
attribute
recognition,
and
it
still
is,
and
so
each
image
is
labeled
with
40
binary
attributes.
C
What
that
means
is
either
you're
blonde
or
you're,
not
either
you
have
high
cheekbones
or
you
don't
either
you're
smiling
or
you're,
not,
and
so
that
binary
label
causes
some
problems
later
on,
because
there's
really
a
gradient
of
these
things,
but
we'll
talk
about
that
as
we
get
into
it.
So
to
date,
there's
still
no
data
set,
that's
larger,
and
so
many
people
still
use
this
data
set.
C
So
if
I
were
to
ask
you
to
picture
this
right,
a
chubby
male
with
straight
hair,
five
o'clock,
shadow
and
arched
eyebrows-
you
might
quickly
come
with
something
in
your
mind
and
although
it
might
not
be
this
exactly,
this
would
maybe
fit
your
description
right.
So
you
wouldn't
be
surprised
by
seeing
this
so
I
apologize
for
the
color
contrast
on
this
image,
but
when
I
show
it
on
an
actual
presentation
like
on
a
in
a
in
a
room,
it
for
some
reason
appears
much
darker.
C
So
it
looks
better
like
this,
but
on
the
computer
it
actually
looks
pretty
strange,
but
anyway
you
probably
wouldn't
picture
something
like
this
right.
This
probably
didn't
come
to
anybody's
mind
and
there's
a
couple
of
reasons
for
that.
Right,
the
first
one
is
that
we
use
the
word
mail
and
a
lot
of
times.
We
don't
use
that
word
when
we're
talking
about
babies.
We
talk
about
babies
by
using
the
word
baby
or
child.
C
The
other
thing
is
that
they
have
a
five
o'clock
shadow,
which
is
another
indication
that
they're
not
a
baby,
and
so
we
kind
of
use
this
common
sense
to
understand
what
types
of
images
and
what
types
of
people
would
fit
this
description.
So
we
decided
to
incorporate
this,
and
this
is
actually
work.
I
did
a
very
long
time
ago
we
decided
to
incorporate
this
sort
of
Common
Sense
knowledge
of
faces
into
a
model
that
was
learning
attribute
relationships.
So
we
did
two
things.
C
The
first
thing
is
we
actually
grouped
attributes
by
location,
and
so
we
combined
pointy
nose
and
wide
nodes.
We
combined
mouth
open,
smiling
and
big
lips,
which
are
all
Mouse
related
attributes.
We
combined
all
of
our
facial
hair
attributes,
all
of
our
eye,
related
attributes,
hair
related
attributes
and
cheek
related
attributes
together,
and
then
we
trained
a
model
that
used
this
physical
representation
to
improve
predictions,
and
then
we
added
in
addition
to
that
in
explicit
and
implicit
attribute
relationship
Network.
C
So
at
first
right
we
started
with
our
Network,
where
we
had
shared
representations
at
the
lower
layers,
and
then
we
separated
out
into
different
physical
locations.
That
would
recognize
certain
attributes
at
those
locations,
and
so
this
was
a
type
of
explicit,
attribute
relationship
learning
where
we
explicitly
modeled
the
physical
relationships
between
attributes.
C
And
then
we
introduce
this
implicit
attribute
relationship
learning
which
we
called
our
auxiliary
Network,
and
the
idea
here
is
is
if
we
know
one
thing
about
a
face:
it
can
help
us
to
better
learn
something
else.
So
if
we
know
that
an
individual
is
wearing
lipstick,
for
example-
and
we
have
a
really
high
confidence
of
that-
but
for
some
reason
we're
wavering
on
whether
or
not
they
have
a
mustache
right,
and
so
the
model
is
sort
of
sitting
around
50
on
mustache.
C
We
can
use
the
fact
that
we're
very
confident
about
lipstick
to
maybe
predict
that
this
person
likely
doesn't
have
a
mustache,
and
so
that's
what
this
auxiliary
network
is
doing,
and
so
we
have
our
explicit
model
being
fed
into
our
implicit
model
and
being
all
trained
together
to
understand
these
explicit
physical
relationships
and
these
implicit
relationships
between
attributes
and
then
what
did
we
learn?
Well,
we
looked
at
the
weights
for
this
auxiliary
Network
and
we
found
a
lot
of
really
interesting
relationships.
C
So
the
first
thing
we
found
was
that
receding
hairline
in
bald
had
a
positive
relationship.
So
on
the
surface,
this
seems
like
a
good
thing,
because
receding,
hairline
and
bald
do
seem
like
they're
related,
but
because
these
attributes
are
binary,
we
actually
run
into
the
problem
where
you
either
have
a
receding.
Hairline
or
you
don't
right,
you
either
have
hair
or
you
don't,
and
so
they
really
shouldn't
have
a
positive
relationship.
They
should
have
an
inverse
relationship.
So
if
someone
has
a
receding
hairline,
they
should
not
have
the
label
of
bald.
C
But
what
we
found
here
is
that
this
data
set
actually
had
some
noise
in
it.
Where
people
were
labeling
both
bald
and
receding,
hairline
for
an
image
we
found
chubby
and
double
chin
had
a
positive
relationship
which
makes
sense
heavy
makeup
and
lipstick.
Have
a
positive
relationship
also
makes
sense.
Necklace
and
necktie
have
an
inverse
relationship.
So
in
this
data
set,
that
means
that
no
one
is
wearing
a
necklace
and
a
necktie
at
the
same
time.
C
Does
this
mean
it's
impossible?
No,
does
it
mean
our
model
probably
can't
handle
it?
Yes,
and
then
we
also
find
that
straight
hair,
wavy,
hair
and
bald
have
an
inverse
relationship
which
we're
happy
with,
because
you
should
only
have
one
of
those
things
given
the
binary
requirement.
Another
thing
that
we
found
is
that
smiling
and
high
cheekbones
had
a
positive
relationship,
so
this
was
actually
really
interesting
and
it
helped
us
realize
that
nobody
really
knows
what
it
means
to
have
high
cheekbones
and
we'll
talk
about
that
in
a
minute.
C
So
our
next
project,
I'm,
going
to
talk
about
addresses
actually
all
four
of
our
pillars,
including
the
bias
and
fairness
in
machine
learning
and
this
project
is,
is
called
selective
learning,
and
this
is
really
inspired
right
again
by
human
perception
and
by
how
babies
learn
right.
So
babies
there's
been
many
studies,
but
one
in
particular
studied
how
babies
learn
based
on
their
base,
basically
on
their
their
visual
interest
in
something,
and
so
when
they
understand
something
they
sort
of
stop
paying
attention
to
it
and
move
on
to
something
else.
C
But
if
they
feel
like
they
need
to
learn
something
they
look
at
it
a
lot,
and
so
there's
this
idea
of
like
once
you've
figured
something
out.
You
don't
have
to
pay
any
more
attention
to
it.
So
it's
very
much
like
us
right.
We
have
selective
hearing
right
where
a
lot
of
us
right
just
hear
the
things
that
we
think
we
need
to
hear,
and
then
we
move
on
right
and
that
never
really
goes
away.
C
It's
all
just
about
efficiency
right,
and
so
we
tried
to
incorporate
this
idea
of
efficiency
in
the
way
that
babies
learn
in
to
a
machine
learning
model.
Now,
without
going
into
all
the
details,
we
basically
did
this
selective
learning
approach,
where
we
would
take
the
data
that
we
had
from
celeb
a
where
a
lot
of
things
were
really
imbalanced.
In
particular,
the
label
for
bald
was
only
positive.
C
Two
percent
of
the
time
in
the
data
set
and
the
label
for
young
was
positive
over
80
of
the
time,
and
so
we
took
this
approach
of
selective
learning
where
we
would
grab
many
many
batches
from
within
our
batches
that
were
balanced
in
some
way
and
through
that
process
we
were
able
to
build
a
model
and
evaluate
it
on
a
new
data
set
that
we
had
collected.
This
was
back
from
my
PhD,
where
we
called
this
data
set.
C
We
were
very
happy
with
that,
but
we
really
were
concerned
about
this
area
here,
where
both
the
state-of-the-art
model
and
our
model
were
getting
around
50
accuracy
and
if
we're
getting
50
accuracy
on
a
binary
problem,
we're
not
really
learning
anything.
So
we
decided
to
investigate
some
of
these
attributes
and
we
found
some
really
interesting
things.
So,
first
one
is
oval
face.
Now
it's
anyone's
guess
which
half
is
positive
and
which
half
is
negative.
C
It
turns
out
that
the
ones
on
the
left
are
labeled
as
positive,
and
the
ones
on
the
right
are
labeled
as
negative,
but
an
oval
face
is
actually
a
very
specific
measurement
and
it's
only
something
that
can
be
measured
when
someone
is
frontal
facing.
So,
in
addition
to
being
basically
a
frontal
face,
detector,
no
one
was
taking
these
types
of
measurements
when
they
were
labeling
the
data,
and
so
it's
really
a
useless
attribute
now
attractive
right.
C
Another
thing,
I,
don't
think
I
mentioned
about
this
data
set-
is
that
all
of
these
individuals
are
are
celebrities
and
for
the
most
part
they're
on
the
red
carpet
or
some
version
of
that
right,
and
so
what
it
means
to
be
unattractive
is
very
unclear
because
they're
all
very
good
looking
people,
and
so
the
ones
on
the
left
happen
to
be
labeled
as
positive
and
the
ones
on
the
right
are
labeled
as
negative.
It's
anybody's
guess
why
that
is
again.
Attractiveness
is
something
that's
very
subjective.
C
C
We
learned
that
high
cheekbones
actually
has
to
do
with
basically
this
this
fat
in
your
cheek
kind
of
being
missing,
and
so
this
sort
of
divot
down
here
and
less
to
do
with
the
fact
that
your
cheekbone
is
actually
physically
high
on
your
face.
But
what
happens
when
people
smile
is
that
their
cheekbones
raise
up
right,
and
so,
when
you
take
a
smiling
photo
it
for
the
most
part,
looks
like
somebody
has
high
cheekbones,
and
so
again
this
is
much
more
of
just
another
label
for
smiling
arched.
C
Eyebrows
is
another
example
of
something
that's
just
poorly
labeled,
because
there's
a
lot
of
different
art,
eyebrow
shapes,
but
this
certainly
wasn't
any
measurement
going
on
and
I
doubt
that
there
was
enough
information
provided
to
the
labelers
for
them
to
really
do
a
a
solid
job.
On
this
and
last
but
not
least,
we
have
lipstick
and
actually
it
turns
out.
The
ones
on
the
right
were
labeled
as
positive
and
the
ones
on
the
left
were
labeled
as
negative.
C
Several
of
the
people
on
the
left
are
wearing
lipstick
and
several
of
the
people
on
the
right
are
not
wearing
lipstick.
So
what
ended
up
happening
when
we
reviewed
this
data
is
that
we
realized
that
a
bunch
of
male
labelers
were
asked
to
label
whether
or
not
women
were
wearing
lipstick
and
they
had
no
idea.
So
this
is
a
very
challenging
problem
works
very
well
for,
like
red,
lipstick
right,
but
any
kind
of
neutral,
colored
lipstick.
It's
not
that
great.
C
We
focused
on
these
two
aspects,
specifically
learning
from
noisy
data
and
explainable
machine
learning,
and
so
we
wanted
to
be
able
to
clean
up
the
data
set
a
little
bit
because
even
for
these
more
objective
attributes,
some
of
the
data
was
still
very
poorly
labeled
and
there's
not
a
lot
of
information
and
by
that
I
mean
there's
like
no
information
in
the
original
paper
for
sale
eBay
to
indicate
how
they
did
the
labeling.
It
just
says
that
they
hired
a
professional
labeling
company
and
then
that's
it.
C
C
So
here's
just
an
example
of
bald
right.
This
is
just
10
random
samples,
but
we
removed
something
like
4
000
images
and
only
one
person
is
actually
labeled
as
bald.
So
if
you
think
about
removing
all
of
these
people
from
the
bald
category
and
this
one,
the
amount
of
noise
actually
still
goes
down,
and
then
we
were
able
to
remove
wearing
eyeglasses
where
a
bunch
of
these
people
write.
These
people
were
all
labeled
as
negative,
and
these
people
were
labeled
as
positive.
C
Now
the
question
becomes:
is
this
person
wearing
sunglasses
or
are
they
wearing
a
hat
and
I
really
think
that's
more
of
a
philosophical
question
that
I
can't
answer,
and
then
we
even
male
versus
female
was
labeled
incorrectly
for
some
images,
and
so
we
were
able
to
automatically
prune
that
as
well,
and
that
was
over
a
thousand
images
and
so
for
our
next
project.
We
actually
focused
on
the
ability
of
these
attributes
to
be
used
for
face
verification
because
we
found,
in
addition
to
all
of
the
noise
associated
with
this
data.
C
We
also
found
that
these
attributes,
these
particular
40
attributes,
were
not
descriptive
enough.
So
we
were
able
to
use
them
to
describe
images
right,
but
they
weren't
really
describing
people,
and
so
they
weren't
really
descriptive
enough
to
do
verification
with
them,
which
is
part
of
the
reason
why
they've
kind
of
fallen
out
of
popularity
in
recent
years,
when
they
first
came
on,
you
know
into
the
the
research
scene
in
like
2008
2009.
C
Facial
attributes
were
touted
as
this
new
thing
that
we
were
going
to
use
to
describe
faces
and
we
were
going
to
be
able
to
fix
all
the
problems
with
face
verification
by
using
these
attributes,
but
they're
not
granular
enough,
and
so
we
started
working
on
a
separate
project,
separate
related
project
where
we're
focusing
specifically
on
caricatures.
So
I,
don't
know
about
you,
but
when
I
was
a
kid
and
went
to
this
like
a
boardwalk,
so
I
grew
up
in
the
Northern
Nevada
area
and
I.
C
C
It
feels
a
little
bit
like
a
personal
attack
because
they
typically
highlight
the
things
about
you
that
you
don't
love,
but
what
they're
really
doing
is
highlighting
the
things
about
you
that
make
you
unique,
which
also
sometimes
happen
to
be
the
things
about
you
that
you
don't
love,
and
so
a
caricature
is
said
to
be
a
good
caricature.
If
it
looks
more
like
you
than
you
do,
and
so
there's
this
sense
of
it
capturing
your
likeness.
C
So
here's
a
couple
of
examples
right
so
first
this
is
Justin
Trudeau
right,
so
there's
a
caricature
of
him
and
a
real
image
of
him.
Now,
if
somebody
walked
in
and
looked
like
this
caricature
of
Justin
Trudeau,
you
would
be
concerned
for
their
health.
Maybe
they
had
been
stung
by
a
bee
or
something
because
humans
don't
look
like
this
right,
and
so
you
would
be
really
worried.
But
at
the
same
time,
when
you
see
it,
you
can
see
him
in
it
right.
C
It
doesn't
look
like
what
a
human
looks
like,
but
it
captures
his
Essence
right.
It
captures
the
things
about
him
that
make
him
him.
It
sort
of
captures
his
unique
eyebrows.
His
sort
of
long
nose,
his
very
flat
ears,
his
floppy-ish
hair
and
his
big
lower
lip
right.
So
it
captures
those
things
about
him
that
are
much
more
granular
than
attributes
are
similarly
with
Barack
Obama
right.
That's
not
what
a
human
looks
like,
and
yet
it
captures
his
Essence
right.
It's
got
his
teeth,
it's
got
his
nose.
C
It's
got
his
ears
right
and
his
thick
eyebrows
and
then
that
chin
right,
so
it
captures
all
of
those
things,
and
you
know
it's
him
very
quickly
from
looking
at
the
caricature.
Similarly,
with
Morgan
Freeman
right
again,
that's
not
what
a
person
looks
like,
but
it
also
somehow
looks
just
like
him
and
then
Mark
Zuckerberg.
C
Then
we're
really
just
storing
sort
of
caricatured
versions
of
people,
and
so,
even
though
we
have
this
caricatured
version,
it
still
captures
the
person's
likeness
and
now
there's
still
room
to
improve
a
lot
of
these
face.
Verification
systems
on
familiar
faces,
humans
are
still
better
at
recognizing
familiar
faces
than
machines
are,
and
so
there's
something
there
right
and
we're
interested
in
bridging
that
Gap
and
so
here,
for
example,
Emma
Watson
right.
C
So
these
kind
of
describable
features
could
be
used
in
addition
to
attributes
to
make
these
more
verification
friendly,
and
so
how
are
people
actually
using
caricatures
right
now
in
practice?
Well,
people
are
generally
trying
to
take
an
image
and
generate
a
caricature,
but
they
don't
really
have
any
means
of
doing
that.
C
C
Now
the
problem
is
the
more
I
I
would
say
realistic,
but
what
I
mean
is
like
the
more
fine-grained
right
and
and
better
looking
the
caricatures
get
the
less.
They
actually
represent
the
original
identity
right,
so
these
ones
on
the
right
hand,
side
are
very
good.
They
look
like
good
drawings,
right
or
good
renderings,
but
they
don't
look
anything
like
the
person
they're
supposed
to
be
representing.
C
So
in
this
process
a
lot
of
people
are
really
losing
the
point
of
the
caricature,
which
is
capturing
that
Essence
and
that
likeness,
and
so
people
are
using
data
sets
and
I'll
talk
about
the
data
sets
in
a
second
but
they're
using
data
sets
where
they
have
like
one
realistic
image
and
then
a
bunch
of
caricatures
of
that
person
and
then
they're
just
trying
to
generate
caricatures
of
that
person.
But
they're
not
really
doing
a
couple
of
things
that
are
really
important.
One
is
making
sure
that
the
image
actually
captures
that
person's
likeness
right.
C
That
makes
Anne
Hathaway
Anne
Hathaway,
and
this
isn't
even
really
a
complete
drawing.
So
we
have
issues
with
the
data
that's
being
used,
mostly
because
people
that
are
working
on
this
problem
are
really
just
trying
to
generate
caricatures
for
the
sake
of
generating
caricatures
and
they
don't
really
care
about
what
they're
being
used
for,
on
the
other
hand,
we're
really
interested
in
generating
caricatures.
So
we
can
better
understand
what
people
look
like.
C
So
this
is
Anne
Hathaway,
but
it
really
isn't
a
caricature.
Her
lips
are
just
big.
This
is
Bono,
but
it's
not
a
caricature.
It's
just
abstract.
Similarly,
here
with
Bono,
this
is
just
a
drawing
of
Bono.
This
is
just
a
painting
of
Anne,
Hathaway
and
so
on
and
so
forth.
So
there's
some
quality
control
issues,
but
this
is
one
of
the
bigger
data
sets
available
for
this
problem.
There's
252
identities.
In
this
data
set
with
roughly
6
000
caricatures
and
6
000,
realistic
or
veritical
face
images.
C
We
did
some
cleaning
on
this
data
right,
removing
any
identities
that
didn't
have
we
removed
identities
after
we
removed
caricatures
and
vertical
faces.
We
found
that
you
know
if
we
we
looked
at
each
individual
caricature
and
we
had
several
people
look
at
each
individual
caricature
and
if
they
couldn't
figure
out
who
it
was
with
any
amount
of
confidence,
we
removed
it.
So
we
almost
removed
half
of
the
caricatures
and
then
we
removed
about
200
of
the
vertical
face
images.
C
Indian
people,
Asian
people
and
so
on
and
so
forth,
so
that
we
didn't
just
have
a
bunch
of
white
caricatures
and
our
data
set
has
229
identities
with
1600
characters
and
3
100
vertical
faces,
so
part
of
our
data
set.
That's
very
unique,
as
opposed
to
the
web.
Caricature
data
set
is
for
every
identity
because
we're
interested
in
using
these
caricatures
to
better
understand
what
people
look
like
and
to
use
that
for
discriminate.
Discrimination
purposes
so
basically
to
discriminate
two
identities.
C
We
needed
to
understand
what
makes
them
unique
so,
rather
than
just
collecting
images
and
caricatures
and
guessing,
we
actually
had
three
labelers
label
every
identity
with
prominent
features
in
the
data
set,
and
so,
for
example,
we
have
two
identities
here
on
the
left.
The
labels,
for
what
makes
them
unique
is
they
have
a
well-defined
tip
of
their
nose.
Their
ears
stick
out.
They
have
pierced
ears,
they
have
arched
eyebrows,
puffy
eyelids
bags
under
their
eyes
a
mustache
and
a
goatee,
so
those
are
things
that
are
sort
of
continuous
throughout
their
identity.
C
So
we
would
look
at
at
least
five
images
per
identity
to
make
these
predictions
right,
because
it's
every
image
can
be
very
different.
But
if
you
see
many
images
of
an
individual,
you
can
get
a
sense
of
what
they
generally
look
like
and
then,
on
the
right
hand,
side.
The
labels,
for
this
were
wide
eyes.
Long,
eyelashes
hooked,
nose,
thin
nose,
small
head
full
lips
and
arched
eyebrows.
C
C
And
so
we
actually
were
able
to
calculate
the
distance
between
the
eyes
distance
between
distance,
sorry
between
the
eyebrows,
the
nose
width
and
Heights,
the
height
and
width
of
the
eyebrows
and
so
on
and
so
forth.
And
then
we
were
able
to
use
those
measurements
to
predict
different
things
like
whether
or
not
the
eyes
were
widely
set
on
the
head,
whether
or
not
they
were
narrow
set,
whether
or
not
people
had
wide
nostrils
or
their
nose
was
wide
or
thin,
long
or
short,
and
so
on
and
so
forth.
C
This
person
has
a
wide
nose.
This
person
has
very
wide
set
eyes
and
so
on
and
so
forth,
so
we
were
able
to
actually
identify
quite
a
few
people
with
very
distinctive
attributes,
and
so
what
we
were
able
to
do
with
this
is
take
this
and
add
this
to
our
set
of
attributes
from
celeb
a
and
actually
use
it
for
face
recognition.
C
So,
instead
of
just
doing
plain
face
recognition,
we
were
kind
of
interested
in
being
able
to
search
for
images
using
text
descriptions,
and
so
this
one
in
particular,
is
a
man
with
a
long
nose,
narrow
set
eyes
and
gray
hair,
and
it
pulls
back
these
images
and
we
were
able
to
find
faces
where
the
eyes
are
very
close,
together,
gray,
hair
and
then
long
nose
and
so
on
so
forth.
So
we
use
this
more
for
search
than
we
have
really
for
verification,
but
they're
very
similar
problems.
B
C
Yeah,
so
it
is
really
important.
We
haven't
gotten
to
the
point
where
we
have
enough
data
that
we
can
pray
those
two
things.
It
is
very
important
and-
and
we
know
that
it's
going
to
be
an
issue
as
we
move
forward,
because
there
are
some
caricatures
that
are
just
plain
drawings
and
then
there's
some
that
are
renderings
and
the
renderings
obviously
are
going
to
be
much
more
useful
because
they're
more
detailed
and
they're
they're
more
in
the
same
domain
as
the
realistic
images.
C
But
right
now
we're
working
on
trying
to
see
if
we
can
predict
the
prominent
features
from
the
caricatures
themselves
and
then
we're
going
to
try
to
combine
the
two
methods
and
then
from
there
we're
actually
interested
in
generating
caricatures,
which
will
look
much
more
like
rendered
ones.
And
so
that'll
actually
tell
us
through
that
process.
We'll
definitely
understand
more
the
limitations
of
the
drawn
caricatures.
D
Hi
hi,
thank
you
so
much
Emily.
This
is
really
helpful.
One
first
tradition:
this
is
by
the
way
one
additional
thought.
Have
you
looked
into
hint
modeling,
where
you
know
you,
if
you
give
a
hint
or
an
abstract
element
of
let's
say:
Anne,
Hathaway
and
and
the
model
is
learning
on
on
the
the
aspects
of
the
hint
to
caricature,
for
it
to
learn
in
in
context.
C
So
I
haven't
I,
haven't
looked
into
that.
That
sounds
really
interesting,
but
it
is
something
that
we
want
to
I
mean
along
those
lines.
We
do
want
to
have
like
effective,
like
human
in
the
loop
correction
type
thing,
and
so
that
sounds
very
much
like
something
that
we
can
look
into
and
I
think
would
be
really
useful.
C
You
so
for
our
next
project,
it's
related,
but
we
we
found
that
a
lot
of
what
makes
the
caricatures
interesting
is
their
their
ability
to
capture
someone's
likeness,
and
so
our
next
project
really
is
focused
on
the
first
three
pillars
where
we're
focused
specifically
on
likeness,
and
we,
we
Define
likeness
in
this
aspect
by
doppelgangers.
So
that
is
people
that
really
just
look
very
similar,
even
though
they
have
no
business
looking
similar
because
they're
not
related
necessarily
so
here's
a
perfectly
good
example.
C
That
Celebi
has
we're
kind
of
creating
our
own
and
we're
doing
this
following
kind
of
a
protocol
from
like
Tom
versus
Pete.
If
you've
read
that
paper
or
heard
of
that
work,
which
was
a
long
time
ago
when
they
were
doing
sort
of
data-driven
attributes,
so
we're
starting
with
that,
but
we're
using
deep
learning
as
the
main
mechanism
for
that
and
then
we're
going
to
move
forward
with
describing
faces
in
terms
of
other
phases.
C
C
And
we
actually
found
just
with
this
very
simplistic
method
that
this
way
outperforms
facial
attribute
as
facial
attributes
as
features
for
face
verification.
And
that's
just
with
this
very
simple
approach.
And
so
we've
actually
been
able
to
use
this
to
create
a
system
that
can
identify
your
doppelganger.
And
so
we
had
a
summer
program
where
we
brought
in
teachers
and
they
were
working
on.
C
Cyber
security
problems,
and
so
we
did
like
a
face
verification
problem,
and
this
is
one
of
our
teachers
Kevin,
and
he
created
this
demo,
which
uses
this
system
and
found
that
he
looks
like
William
Baldwin,
which
I
don't
think
is
too
far
off
so
I
thought
that
was
cool.
We're
still
working
on
this,
we're
finalizing
the
data
and
making
sure
it's
clean
and
ready
to
go
and
we're
submitting
a
paper
on
that
very
soon,
and
so
the
data
set
will
be
available
and
we're
still
working
on
different
approaches
for
these
likeness-based
attributes.
C
But
basically
that's
just
a
couple
of
the
projects
that
we
have
going
in
the
lab.
We
have
quite
a
few
other
ones
that
I
didn't
have
quite
a
lot
of
time
to
talk
about,
including
really
dissecting.
These
models
to
understand
if
they're
learning
the
same
things
that
humans
are
learning
in
certain
scenarios
which
we
found
with
one
of
our
works,
but
didn't
have
time
to
include
it
here.
So
basically,
we
talked
about
describing
faces
right.
C
We
talked
about
facial
attributes,
caricatures,
prominent
features
and
likeness-based
attributes
all
as
means
of
describing
faces
in
the
way
that
humans
do.
Taking
cues
from
human
perception
and
using
those
to
improve
the
explainability
and
the
actual
performance
of
these
verification
models,
so
I
wouldn't
be
anywhere
without
my
team.
C
B
B
C
B
C
It
is
unfortunately,
I
mean
part
of
the
problem.
Is
that
there's
a
lot
of
problems
since
11
is
still
the
data
set,
because
it's
the
biggest
one?
Some
people
have
done
some
work
on
cleaning
it.
Some
people
have
done
some
work
on
very
specific
aspects
of
it
like,
like
fixing
just
the
facial
hair
attributes
and
and
adding
additional
ones,
to
really
account
for
different
things.
C
But
that's
the
problem.
Is
it's
so
tedious
to
label
that
many
things
for
each
image
that
no
one's
really
come
out
with
a
larger
data
set
or
a
cleaner
version,
or
anything
like
that
and
so
I
think.
What
really
has
happened
is
that
people
are
just
kind
of
working
on
different
aspects
of
it
and
not
necessarily
just
trying
to
predict
attributes
anymore
and
we're
not
really
trying
to
predict
attributes
anymore
either,
because
it's
just
not
given
that
data.
C
It's
not
helpful
and
and
we've
realized
the
issues
with
it
in
addition
to
like
they're,
just
not
discriminative
enough,
and
so
that's
why
we
use
the
data,
but
for
other
things
now,
and
so
we
we've
kind
of
expanded.
The
definition
of
attributes
to
include
all
these
other.
More
discriminative
things,
but
yeah,
it's
definitely
still.
The
data
set
and
people
use
it
more
for
like
generating
images.
Now,
because
there's
such
a
variety
of
of
different
places
and
things
like.
C
C
Celebi
has
when
we
do
that,
it's
really
challenging,
because
if
you
think
about
it
right,
when
you
do
data
collection
like
this
right,
you
you
do
just
like
a
lot
of
Googling
and
scraping,
and
it's
like
we're
Googling
in
the
United
States.
So
it's
really
hard.
You
have
to
do
very
specific
type
of
searches
to
make
sure
that
you're
getting
diverse
data,
but
we
I
mean
we
do
it.
It's
just
it's.
It
would
be
nicer
if
we
were
collecting
data
from
all
over
the
world,
but
yeah.
It's
it's
a
challenge
for
sure.
C
Yes,
so
in
in
our
caricature
work,
specifically
in
The
prominent
feature
work
because
we're
taking
multiple
images
of
faces
to
predict
prominent
features
and
most
of
the
time,
it's
challenging
to
predict
prominent
features
from
an
off
frontal
view.
So
a
lot
of
what
we
did,
especially
with
the
distance
based
learning.
We
used
only
frontal
images,
so
we
kind
of
we
didn't
limit
ourselves
in
the
data
collection
process
to
only
frontal
images,
but
in
the
evaluation
for
that
unsupervised
method.
C
We
did
right
now
we're
also
building
we're
building
a
supervised
method,
both
on
just
the
regular
images
and
the
caricatures
and
we're
we're
exploring
both
involving
the
the
Angles
and
and
trying
to
ignore
them,
and
so
because
there's
so
little
data
we're
having
trouble
with
with
both
right,
because
we
don't
want
to
get
rid
of
any
additional
data.
But
then
the
angles
pose
a
significant
problem,
and
so
we're
we're
working
on
on
approaches
to
that
that'll
be
robust
to
pose.
B
A
B
C
So
messy
there's
just
no
way,
yeah
and
so
I
think
that's.
The
thing
is
like
a
lot
of
people
that
are
a
lot
of
systems
that
are
working
really
well
are
using
some
other
data
that,
like
we're
just
not
privy
to
you,
know
or
they're,
like
leveraging
external
data
and
then
also
maybe
celebrate
at
some
point,
but
certainly
not
relying
entirely
on
that
data
set.
B
B
C
Know
a
lot
of
people
that
are
still
working
on
attributes
right,
that
I
see
them
at
conferences,
and
they
tell
me
that
they
have
a
personal
cleaned
version
of
Celebi
and
I'm
like
why
haven't
you
released
like
I
saw
mine
yeah,
so
I
know
that
some
people
do
have
done
the
work
of
cleaning
it
in
some
aspects,
and
so
there
there's.
Certainly
some
people
are
working
with
cleaner
data
than
we
are.
B
B
C
B
C
C
And
we
also
want
to
be
able
to
take
a
real
image
and
kind
of
generate
a
character
based
on
what
we
can
identify
as
prominent
about
that
person.
So
going
the
going.
The
first
way
right,
taking
a
caricature
and
making
it
more
realistic,
is
a
much
more
challenging
problem.
I
would
imagine
than
the
other
way
around
and.
C
C
So
if
we
have
sort
of
a
caricatured
version
of
them
and
so
that'd
be
like
you
know,
for
police
sketches
and
things
like
that,
but
going
the
other
way
where
I
have
a
student
right
now,
I
have
a
a
new
student
kind
of
trying
to
test
that
out
and
see
if
they
can
identify
the
prominent
things.
But
the
problem
with
that,
which
is
probably
obvious,
is
the
artistic
interpretation
of
these
images
right.
B
B
C
And
that's,
the
thing
too
is
like
it's
sort
of
was
just
my
my
fascination
with
noisy
data
has
been
like
for
so
long
because
it's
just
sort
of
like
that's
just
what
we
have
to
work
with
like
there's,
no
reality
in
which
we're
going
to
be
dealing
with
perfect
data
or
perfect
test
situations.
You
know
so
it's
like.
We
always
start
with
noisy
data
and
just
try
to
figure
out
what
we
can
get
from
there.
Yeah.
B
Of
course,
yeah
yeah
sounds
good,
so
thank
you,
Emily
I
think.
If
anyone
has
any
question,
this
is
the
kind
of
last
call
probably
the
as
I
mentioned
earlier.
The
talk
was
sponsored
by
a
Tripoli
young
professionals,
women
in
engineering
and
the
competition
intelligence
affinity
group.
So
so
the
chairs
I
do
see
them
they're
present
here
at
the
attend
at
the
top.
It
was
very
interesting.
Thank
you
so
much
yeah.