►
Description
In this presentation, Eric Levine will demonstrate how Airbnb’s Trust and Safety team has built machine learning infrastructure from the ground up to catch and defuse fraudsters on the platform. His talk will briefly go through the history of the machine learning platform from its very basic roots, to the advanced system that is in place today.
About CodeConf
CodeConf improves the software community by providing a forum for thought-provoking talks and forging social connections. The third installment of the CodeConf series took place in Nashville in 2015. Attendees came together to discuss open source, best practices, documentation, and community.
For more information on this year's CodeConf, go to:
https://codeconf.com/
A
Thanks,
hello,
everyone
I
hope
you're
enjoying
code
conf
so
far
today,
as
you
mentioned,
you're
going
to
be
learning
about
how
we
detect
and
mitigate
fraud
on
Airbnb,
so
Who
am
I.
My
name
is
Eric
Levine
I'm,
the
engineering
manager
for
the
trust
and
safety
team
at
Airbnb
I've
been
with
the
company
for
a
little
over
three
years
and
prior
to
this
I
was
at
Google
for
a
couple
of
years.
A
So
for
those
of
you
who
don't
know
what
is
Airbnb,
Airbnb
is
a
platform
that
connects
guests
and
hosts
all
around
the
world
where,
if
you
want
to
travel
somewhere
and
stay
in
someone's
home
instead
of
in
a
hotel,
for
example,
Airbnb
will
connect
you
or
if
you
have
a
spare
bedroom
in
your
house,
you
can
also
list
your
space
on
the
platform
and
we
will
help
find
you
guests.
So
what
is
trust
and
safety?
Well
Airbnb?
A
So
our
job
is
to
stop
bad
actors
before
they're
able
to
affect
us
or
our
community,
we're
comprised
of
two
organizations,
a
product
organization
and
an
Operations
organization,
and
we
work
together
to
create
a
safer,
more
trusted
community.
So
today
we're
going
to
learn
about
what
fraud
actually
exists
on
Airbnb
we're
going
to
talk
about
some
machine
learning,
basics,
how
we
apply
those
basics
to
Airbnb,
some
of
the
philosophies
that
we've
developed
over
the
years
and
finally,
what's
next
for
the
trust
and
safety
team.
So
what
does
fraud
on
Airbnb?
Actually
look
like?
A
It
turns
out
that
it's
a
pretty
standard
smorgasbord
of
fraud
that
we
see
on
our
platform.
We
see
payment
fraud
where
someone
uses
something
like
a
stolen
credit
card
to
make
a
transaction.
We
see
fake
listings,
where
someone
will
put
a
room
on
the
platform
that
doesn't
actually
exist
in
the
real
world.
We
see
spam
messages
when
someone
is
trying
to
solicit
for
some
unaffiliated
third
party.
We
also
see
account
hijackings
when
someone
gets
their
credentials.
Fish
they'll
try
to
go
on
to
their
account
and
cause
all
sorts
of
havoc.
A
So
now
these
are
the
types
of
fraud
that
we
see
on
the
platform.
However,
we
can't
talk
a
lot
about
exactly
how
we
stop
those
things,
of
course,
because
this
is
being
live-streamed,
and
so
we
don't
want
those
bad
actors
who
are
trying
to
defraud
us
to
know
all
the
secrets
that
we
have
so
instead
we're
going
to
talk
about
a
slightly
more
contrived
example
of
a
issue
that
we
had
in
the
past,
where
people
could
time
travel
on
Airbnb
and
how
we
can
detect
and
mitigate
people
who
are
taking
advantage
of
the
system.
A
So
we
had
this
bug
on
April,
1st
2015
April
1st
in
case
that
wasn't
clear.
We
revert
it
in
24
hours
but
in
the
meantime,
there's
this
particular
pair
that
was
creating
all
sorts
of
bookings
on
the
site
and
traveling
all
throughout
history
collecting
all
people
through
through
these
things
and
well,
there
were
nice
enough
guys.
They
didn't
really
have
a
lot
of
respect
for
the
space
and
they
left
things
in
a
bit
of
a
mess.
So
we're
going
to
talk
about
how
we
could
potentially
stop
this
this
duo.
A
So
let's
talk
about
machine
learning,
basics.
So
what
is
machine
learning
machine
learning?
Is
the
science
of
getting
computers
to
act
without
being
explicitly
programmed
courtesy
of
Wikipedia?
What
does
that
mean
in
practice?
That
means
we
take
things
that
have
happened
in
the
past
and
use
them
to
predict
things
that
happen
in
the
future.
So
let's
go
through
a
basic
example
of
that
when
you're
trying
to
build
a
machine
learning
system
first,
you
need
to
come
up
with.
What
are
you
actually
trying
to
predict,
and
in
this
case,
is
it
going
to
rain
tomorrow?
A
So
features
are
what
you're
going
to
use
to
decide
one
way
or
another
what
your
prediction
is
going
to
be
in
this
case.
Is
it
cloudy?
The
prediction
is
what
you
think
is
going
to
happen,
and
then
you
have
ground
truth
which
at
the
bottom,
you
can
see
the
features
and
the
labels
from
past
data.
So
these
are
the
values
for
the
features
from
the
past
and
what
actually
happened
so
those
two
things
together
make
ground
truth
and
those
are
the
critical
aspect
for
building
a
machine
learning
system.
A
So
when
you're
building
out
the
system,
you
also
need
to
build
out
infrastructure,
and
you
need
to
have
a
couple
of
things
that
you
need
to
have
figured
out.
One
is:
what's
your
delay,
tolerance.
Is
it
okay?
If
you
detect
something
72
hours
after
it
happens,
or
do
you
need
to
be
able
to
detect
things
in
real-time?
What
models
do
you
want
to
run?
A
There
are
many
different
machine
learning
models
from
support,
vector
machines
to
naive,
Bayesian
classifiers,
which
one
is
going
to
work
well
for
your
scenario
and
then,
when
you
detect
something
that
is
fraudulent,
what
do
you
actually
do
about
that?
Do
you
send
it
for
manual
review
so
that
a
human
makes
the
decision
of
whether
it's
fraudulent
or
not?
Do
you
introduce
some
friction
like
CAPTCHA
to
stop
BOTS
from
you
know,
wreaking
havoc
on
your
system,
or
do
you
require
additional
data
so
that
you
can
be
more
confident
in
the
decisions
that
you're
making?
A
So
how
does
this
all
fit
together?
So
again
you
have
the
ground
truth:
the
labels
and
the
features
plug
into
a
model
training
system
that
will
produce
a
model
and
that
will
go
into
your
model
infrastructure
that
will
then
allow
you
to
score
things
in
the
future.
So
how
does
this
apply
at
Airbnb,
so
we
built
the
system
called
the
fraud
prediction
service,
which
is
comprised
of
three
separate
components:
the
signal
service,
the
model
service
and
the
model
builder
and
we'll
go
through
each
of
these.
A
In
turn,
the
signal
service
is
how
we
collect
the
features
that
we
use
to
make
the
predictions
in
real
time.
So
that's
one
of
the
things
that
we've
determined
is
important
to
us
is
that
we
need
to
be
able
to
make
these
decisions
when
they
happen,
so
that
we
can
stop
the
bad
actors
before
they're
able
to
have
any
effect
on
our
community.
So
we
need
this
to
be
really
fast.
A
It
needs
to
be
configuration
driven
so
that
people
who
might
not
know
how
to
actually
write
code
can
still
contribute
to
the
feature
set
that
we're
able
to
use
to
predict
fraudulent
activity.
We
collect
data
uniformly,
so
it
doesn't
matter
whether
this
data
is
coming
from
an
external
database
or
a
separate
service,
or
even
like
a
third
party
service.
It
all
comes
in
the
exact
same
way
and
it
resolves
based
on
graph
resolution
we'll
go
through
an
example
of
what
I
mean
by
that
in
just
a
moment.
A
But
let's
get
back
to
that
example
of
how
we're
going
to
catch
those
time
travelers.
So
we
need
to
have
discriminatory
facts
that
will
make
up
the
features
that
we're
going
to
use
in
our
system,
so
in
this
case
the
guests
being
located
in
San
Dimas.
The
use
of
the
word
dude
in
the
profile,
description
and
a
reservation
date
in
the
past
are
the
sort
of
contrived
examples
of
things
that
we're
going
to
use
to
predict
this
particular
attack
vector.
A
So
the
API
looks
something
like
this,
where
you
provide
inputs
in
this
case,
it's
the
reservation.
Id.
You
want
to
know
if
this
particular
reservation
is
with
these
two
actors
and
then
the
signals
that
you
want
to
get
out
the
features
that
you
want
to
get
out
so
again,
it's
whether
the
guest
is
located
in
San
Dimas,
whether
the
profile
contains
the
word
dude
and
whether
the
reservation
date
is
in
the
past.
So
let's
get
back
to
that
graph
resolution
thing
I
was
talking
about
so,
as
you
can
see
here
we
have
a
graph.
A
A
Neither
of
those
are
actually
required
to
figure
out
the
features
that
we're
trying
to
get
so
we
can
actually
just
scratch
them
out
of
the
graph
all
together,
they're
gone
for
the
purposes
of
this
particular
call
they
might
as
well
not
exist.
So
next
we're
going
to
mark
things
as
synchronous
versus
asynchronous.
So
synchronous
in
this
case
is
just
a
little
bit
of
code.
That's
going
to
run,
you
know.
A
Do
some
comparison,
do
a
subtraction,
some
something
very
quick,
we're
talking
about
a
couple
microseconds
here,
whereas
the
asynchronous
ones
our
database
fetches
service,
calls
things
that
are
going
to
actually
require
a
network
round-trip.
So
we're
looking
at
several
milliseconds,
so
this
is
multiple
orders
of
magnitude
difference
between
one
versus
the
other,
which
is
important
for
how
we
actually
are
going
to
resolve
this
graph.
A
So
the
way
we're
going
to
do
it
is
we're
going
to
greedily
process
the
ones
that
are
synchronous
so
that
we
can
batch
up
as
many
of
the
asynchronous
ones
as
possible
together,
so
that
we
have
as
few
asynchronous
batches
as
possible.
So
what
does
this
look
like?
First,
we
start
with
the
reservation
ID.
This
is
again
the
input
that
was
provided
to
us
and
in
this
case
it
was
given
to
us.
So
there's
no
work
involved
in
actually
resolving
this.
A
There's
no
synchronous
signals
for
us
to
so
we're
going
to
do
a
batch
of
asynchronous
signals
all
at
the
same
time,
then
we're
going
to
process
the
synchronous
signal
that
we
have
there
at
the
top,
the
pass
reservation
and
then
another
batch
of
asynchronous
signals
and
finally,
a
batch
of
synchronous
signals,
and
so
now
we've
managed
to
process
all
the
signals
that
we
need
for
for
the
features
that
we've
requested
and
we're
able
to
make
the
prediction
against
that.
So
that's
the
signal
service.
A
Next,
we'll
talk
about
the
model
service,
so
the
model
service
is
a
PMML.
It
deploys.
Pmml
based
random
forests
and
PMML
stands
for
the
predictive
modeling
markup
language,
and
what
that
means
is
it's
an
XML
format
that
specifies
predictive
models.
There
are
many
different
systems
that
will
produce
PMML
and
many
different
systems
that
will
consume
it.
A
So
it's
trying
to
be
like
a
lingua
franca
for
these
predictive
models
so
that
they're
very
portable,
it
uses
an
HTTP
interface,
so
any
of
the
systems
in
our
infrastructure
will
be
able
to
figure
out
whether
some
something
is
fraudulent.
We
score
multiple
models
for
a
given
event,
so
we
want
to
minimize
the
number
of
again
network
round
trips
that
we
have
to
do,
and
so,
if
we
have
some
particular
activity
that
we
want
to
predict
multiple
types
of
fraud
against,
we
don't
want
to
make
multiple
calls.
A
We
want
to
make
one
call
and
have
them
all
score
and
we'll
go
through
what
exactly
I
mean
by
that.
It
actually
looks
like
this.
So
at
the
top
we
have
the
service.
The
second
layer
are
the
actions
on
the
site
that
we
might
want
to
predict
against,
and
the
third
layer
are
the
models
that
we
have
behind
those.
So
again,
we
call
the
model
service
which
we
want
to
figure
out
whether
a
reservation
is
a
time-traveler
or
not.
A
So
we
go
to
the
reservation
requested
action
and
then
the
time
traveller
and
chargeback
models
are
behind
that
chargebacks,
referring
to
when
someone
uses
a
stolen
credit
card
on
our
platform,
so
now
we'll
be
able
to
score
both
of
those
without
having
to
call
the
service
multiple
times.
So
next
we'll
talk
about
how
we
actually
train
models.
A
So
when
we
gather
all
these
features
from
the
signal
service,
we
log
all
of
the
things
that
we
saw
in
production
to
HDFS
the
Hadoop
file
system.
We
also
import
the
labels
from
manual
review
into
HDFS
so
that
we
all
have
we
have
all
of
it
in
one
place
and
we'll
go
through
that
really
quick.
So
a
request
comes
in
to
the
fraud
prediction
service
on
the
left.
A
The
first
thing
it
needs
to
do
is
to
collect
the
features
it
needs
in
order
to
make
the
prediction,
so
it
will
call
the
signal
service.
The
signal
service
will
gather
the
features
from
the
databases
or
services
and
when
it
returns
to
the
fraud
prediction
service,
it
will
also
log
them
to
HDFS.
Like
I
mentioned
just
a
moment
ago,
then
the
features
are
sent
to
the
model
service
to
produce
a
score.
A
A
The
answers
for
those
manual
reviews
go
into
the
production,
databases,
labels
and
those
labels
are
then
exported
into
HDFS.
Now
we
have
all
the
ground
truth.
We
need
in
order
to
build
a
model
living
in
one
place,
and
so
we
can
export
those
together
to
the
training
system
that
will
produce
a
model
that
we
can
then
upload
onto
the
model
service.
So
that's
how
the
training
system
works,
and
you
can
see
that
there's
a
lot
of
arrows
pointing
at
a
lot
of
other
arrows
but
I
promise.
It
all
works.
A
So
I
mentioned
that
we
use
random
forests.
So
I'll
talk
a
little
bit
about
what
that
actually
is.
So
a
random
forest
is
made
up
of
a
number
of
different
decision
trees.
So
a
decision
tree
might
look
something
like
this,
where
each
node
in
this
tree
represents
a
feature
and
each
edge
represents
a
value
that
that
feature
might
represent.
So
in
this
case,
we
can
follow
the
arrows
down.
Just
you
know,
making
choices
as
we
go
so
does
do
they
use
the
word
dude
in
their
profile.
Yes,
they
do.
Is
the
reservation
in
the
past.
A
Yes,
it
is
so
we
can
conclude
that
this
is
indeed
the
the
actors
that
we're
looking
for.
So
the
random
forest,
as
I
said,
is
made
up
of
a
number
of
random
trees.
Are
these
decision
trees,
and
so
the
score
that
you
produce
is
the
percent
of
the
trees
that
vote
it
as
bad.
So
in
this
case,
seven
percent
of
the
trees
vote
bad
and
so
your
score
for
how
fraudulent
you
think
something
is,
is
seventy-five
percent
so
now
we've
determined
that
some
activity
is
indeed
these
time,
travelers
we're
looking
for.
A
Let's
switch
gears
a
little
bit
and
talk
about
the
machine.
Learning
philosophies
we've
developed
over
the
years,
so
we
view
the
machine
learning
system
as
a
collaboration
between
many
different
teams,
the
operations
team,
the
data
science
team
and
the
engineering
team,
and
it
looks
something
like
this.
A
Where
the
operations
team
there
are
the
boots
on
the
ground,
they're
the
ones
who
are
actually
looking
at
the
fraudulent
patterns
that
are
happening
day
in
and
day
out,
helping
people
when
they,
when
you
know
things
fall
through
the
cracks,
so
it
starts
with
understanding
and
they
are
really
the
ones
who
are
best
suited
to
really
understand
the
fraudulent
activity.
That's
happening
on
our
platform
they'll
hand.
These
insights
off
to
our
data
science
team
to
design
features
that
we
can
use
to
discriminate
between
fraudulent
and
non
fraudulent
activities.
A
They'll
also
train
models
that
we
can
then
use
to
score
activities
that
are
happening.
The
engineering
team
is
responsible
for
actually
using
those
models
and
scoring
them
in
production.
We're
also
responsible
for
mitigating
things.
When
we
detect
that
something
is
or
is
not
fraudulent.
In
certain
cases,
we
will
ask
the
operations
team
to
review
the
decisions
we've
made
and
then,
as
they're
going
through
and
reviewing
all
these
patterns,
they'll
gain
a
better
understanding
of
the
fraud.
A
That's
happening
on
the
site,
so
it's
a
virtuous
cycle,
and
hopefully
we
get
better
and
better
at
it
as
time
goes
on.
So
a
couple
other
things
you
require
trustworthy
ground
truth.
If
your
trust
excuse
me,
if
your
ground
truth
is
not
reliable,
your
models
will
never
be
effective
and
it's
just
it
won't
happen.
A
So
there's
a
lot
of
human
involvement
in
every
step
along
the
way
of
the
process
that
I
just
described,
but
a
lot
of
it
could
be
automated,
like
the
model
building
is
currently
done
by
a
human
there's.
No,
no
need
for
that,
and
then
gathering
more
data
sources
as
I
said
feature.
Engineering
is
the
primary
way
that
we're
going
to
gain
make
gains
in
fighting
this
fraudulent
activity.
So
how
can
we
get
more
and
more
data
to
be
more
and
more
confident
in
the
decisions
we're
making?
So
that's
it.