►
Description
Presented by Jens Vanderhaeghe, Software Engineer (Airbnb).
Airbnb has grown exponentially over the last few years—and so has their codebase. Most of Airbnb's code lives in a monolithic Rails application called Monorail. Despite having hundreds of contributors, the team is still able to ship fast and frequently. Hear some of the unique challenges Airbnb faces as their engineering team grows and see how they plan to transition toward a service-oriented architecture—a multi-year effort involving hundreds of engineers across the company.
About GitHub Universe:
GitHub Universe is a two-day conference dedicated to the creativity and curiosity of the largest software community in the world. Sessions cover topics from team culture to open source software across industries and technologies.
For more information on GitHub Universe, check the website:
https://githubuniverse.com
A
Hey
everyone
welcome
I,
see
a
lot
of
new
faces
here
today.
I
will
be
sharing
kind
of
a
journey
with
you
guys.
I
will
be
sharing
airbnb
journey
from
mole
Arel
into
a
mono
repo,
and
this
is
not
just
my
journey
as
a
developer.
This
is
kind
of
the
journey
of
all
of
the
software
engineers
at
Airbnb
and
I'm
very
excited
to
kind
of
share
with
you
guys
our
findings
and
kind
of
our
progress
in
migrating
from
a
really
big
monolithic
application
into
like
a
service-oriented
architecture.
A
So
before
we
get
started,
my
name
is
Yance.
Vander
I
could
probably
give
a
different
talk
on
like
the
intricacies
of
having
a
Dutch
last
name
in
the
US,
but
you
can
see
the
spelling
up
there.
I'm
Belgian
I
moved
to
San
Francisco
in
2013
and
I
actually
was
originally
a
front-end
engineer
and
then
I
somehow
ended
up
on
an
infrastructure
team
at
Airbnb.
If
you
want
to
follow
me
on
all
the
relevant
social
media,
especially
github,
you
can
kind
of
find
my
tags
down
there
on
the
slide.
A
We
don't
develop
like
Airbnb
products,
I,
don't
work
on
Airbnb
comm
I
support
other
software
engineers
my
build
to
so
that
other
software
engineers
can
feel
productive
and
my
team
actually
used
to
be
known
as
developer
happiness,
and
that
name
really
says
it
all.
My
day-to-day
job
is
really
making
other
engineers
in
the
office
as
happy
as
possible
as
productive
as
possible.
So
my
team's
customers
are
engineers,
even
though
I'm
on
deploy
infra
I,
don't
actually
deploy
code
into
production.
I
merely
provide
tools
for
engine
to
be
able
to
ship
the
code
into
production.
A
A
So
let's
start
this
journey
with
kind
of
like
humble
beginnings,
maybe
you're
looking
at
Nate
Nate
as
one
of
the
three
founders
of
Airbnb,
so
nate
is
kind
of
the
more
technical
founder
of
Airbnb.
He
was
kind
of
the
original
software
engineer
at
Airbnb.
You
look
you're
looking
at
the
very
first
commit
in
our
code
base.
This
is
like
history
right
there.
It
actually
wasn't
good
back.
Then
it
was
still
SVN.
A
So
what
does
it
look
like?
So
it's
a
Ruby
on
Rails
codebase,
the
front-end
in
the
early
days,
was
backbone,
I'm
sure,
there's
still
quite
a
bit
of
backbone
code
out
there
in
monorail,
but
in
2015
we
kind
of
jumped
on
the
new
hotness.
We
moved
to
react,
Redux
and
so
the
backends.
We
have
some
Java
services
that
don't
live
in
monorail,
but
most
of
our
back-end
actually
also
lives
in
monorail.
So
there's
a
lot
of
like
Ruby
logic
that
does
the
Airbnb
backend
all
the
way
from
payments
reservations.
A
All
of
that
and
that's
that
was
actually
all
done
in
this
small
rail
application
and
we
did
a
lot
of
stuff
in
like
asynchronous
jobs.
Let's
say
sending
an
email
confirming
a
reservation,
and
all
of
that,
so
we
used
rescue
rescue
is
kind
of
like
Ruby
on
Rails
is
messaging
queue
where
you
can
kind
of
in
queue,
jobs,
it's
kind
of
a
job
worker,
and
so
you
might
wonder
like
how
do
you
deploy
this?
Like
really
big
baller
l
application?
A
It's
actually
quite
a
quite
an
interesting
story
on
how
we
deploy
that
so
you're,
looking
at
the
life
of
a
poor
request
at
Airbnb
or
at
least
up
until
maybe
two
years
ago-
and
this
is
a
monorail-
pour
request.
So
we
kind
of
have
these
steps
that
a
developer
goes
through
on
their
day
to
day
life.
So
it
kind
of
starts
with
this
CI
and
code
review.
So
this
is
actually
before
you
merge
your
guest
FPR
into
master.
We
kind
of
call
this.
A
The
inner
loop,
like
developers,
are
like
making
changes
that
are
committing
the
code,
but
they're
not
merging
it
into
production,
yet
they're
not
ready
to
ship
it.
This
is
like
the
inner
development
loop
and
I
would
say
this
is
about
80%
of
our
developers.
Day-To-Day
life
is
like
kind
of
spent
in
this
like
inner
loop,
so
we
do
a
bunch
of
status
checks
before
you
merge
your
poor
request.
We
do
linting,
one
of
our
linters
is
actually
really
well
known
or
javascript
es
lint
configuration.
A
We
run
unit
tests,
we
check
for
code
coverage,
you
know
all
the
good
stuff
to
make
sure
that
whatever
you
merge
into
master,
isn't
like
completely
broken
code
and
once
you
kind
of
get
through
that
phase,
your
PO
request
gets
approved.
We
have
a
code
review
process
as
well.
Your
pull
request
is
actually
emergent
to
master,
and
then
we
rerun
CI
again.
Why?
Don't
we
rerun
CI
again
so
mo
Lorella
is
a
really
huge
repo.
A
So,
while
you
were
running
CI
on
your
pull
request,
other
people
might
have
first
code
in
so
we
want
to
make
sure
that
whatever
you
merge
is
actually
tested
again.
So
we
run
the
whole
test
suite
again.
We
run
all
of
the
linkage
again,
so
we
just
want
to
make
sure
that
whatever
you
did
on
your
PR,
other
pull
requests
haven't
kind
of
affected
the
changes
that
you
made
in
in
master,
and
so
eventually
we
deploy
those
changes
to
next.
So
next
is
kind
of
our
internal
staging
environments.
A
It's
basically
a
complete
copy
of
Airbnb,
but
it
doesn't
take
any
production
traffic,
and
so
that's,
where
developers
will
kind
of
go
in
and
kind
of
test
at
changes,
make
sure
that
whatever
they
did
is
actually
good
to
go
into
production
and
then
eventually,
changes
go
into
production.
The
live
and
visible
to
all
users.
A
With
a
caveat,
a
lot
of
our
developers
actually
put
the
changes
behind
a
feature
flag
and
so
they'll
actually
launch
it
to
maybe
a
subset
of
10%
of
all
users
or
20%,
or
something
like
that
and
they'll
eventually
roll
it
out
to
all
users.
So
you
kind
of
see
this
pattern
of
most
developers
putting
their
changes
behind
a
feature
flag
and
so
at
Airbnb
we
have
this
kind
of
unique
system
in
the
industry
of
democratic
deploys.
So
what
does
it
mean?
A
As
I
mentioned
before,
deploy
infra
just
provides
the
tools
we
don't
actually
go
and
sit
there
and
kind
of
deploy
code
for
engineers
we're
pretty
good
at
like
continuous
delivery.
So
that
means
every
engineer
that
comes
in
to
Airbnb
can
deploy
into
production
on
day
one
even
an
intern
can
deploy
two
airbnb.com
as
soon
as
they
get
their
laptop.
Basically,
that
also
means
that
every
engineer
has
great
power
and
great
responsibility.
So
that
means
we
don't
really
have
a
dedicated
QA
team
for
the
airbnb.com
website.
A
So
every
engineer
kind
of
goes
in
tested
their
own
changes
and
then
we
roll
it
out
into
production.
So
we
didn't
have
a
release
management
team.
Every
engineer
is
kind
of
their
own
release.
Manager.
Every
engineer
is
kind
of
their
own
QA
team
and
so
you'd.
The
delay
between
like
merging
your
PR
and
two-masted
and
shipping
your
code
into
production
is
usually
only
about
an
hour
and
people
ship
multiple
changes
a
day
we
ship
about
200
changes
a
day.
So
we're
like
we
have
this
really
high
velocity
of
changes
going
on
to
the
airbnb.com
website.
A
And
how
do
we
do
that?
Well,
my
team
built
this
internal
tool
called
deploy
board
you're,
looking
at
a
screen,
shot
of
deploy
board
right
here
and
it's
kind
of
our
own
internal
web
app.
It's
also
building
Ruby
on
Rails
and
the
deploy
pod
UI
shows
you
all
of
the
commits
that
are
coming
into
the
master
branch
and
it
shows
you
CI
as
well.
So
you
can
kind
of
see
all
of
the
green
checks,
which
means
CI
has
passed,
and
if
you
look
good,
you
can
actually
see
the
end,
the
yellow
and
green
pea.
A
So
the
yellow
n
is
next,
so
you
can
see
which
code
is
running
on
the
next
environment
at
that
point
in
time,
and
you
can
also
see
the
P.
So
what
we
do
is
we
deploy
a
batch
of
changes
to
nexts
and
then
we
kind
of
allow
people
to
test
their
changes
and
then
we
deploy
production
to
whatever
next.
Is
that
and
we
kind
of
do
this
cycle
all
day
long?
We
recently
also
added
a
canary
target
to
that,
but
it's
similar.
We
basically
do
next
and
then
production
next
production.
A
A
It
even
has
some
monitoring
built
in
to
make
sure
no
disaster
is
happening
when
you're
doing
a
deploy
and
that
worked
well
for
a
while
and
around
the
time
I
joined.
We
were
starting
to
see
like
a
pretty
rapid
growth
of
our
code
base.
So,
as
you
all
might
know,
Airbnb
has
been
around
for
about
a
decade
now,
and
so
in
the
early
days
when
they
spun
up
this
Ruby
on
Rails
application.
Had
scaled
pretty
well,
we
had
a
couple
of
like
listings
in
the
United
States.
Everything
was
fine.
A
We
could,
like
just
horizontally
scale,
a
Ruby
on
Rails
application
and
we
were
fine
for
for
a
couple
years
and
then
eventually
you
know
like
2017
a
year
ago.
Airbnb
is
this:
like
global
brand,
everyone
has
heard
about
Airbnb.
You
can
use
it
all
across
the
world.
It's
really
big!
There's
four
million
total
homes
that
you
can
book
on,
Airbnb,
there's
about
65,000
cities,
one
191
countries,
there's
only
maybe
one
or
two
countries
where
you
cannot
book
an
Airbnb,
so
we've
kind
of
become
this
global
brand.
A
But
we're
still
on
this,
like
small,
on
this,
like
Ruby
on
Rails
application.
So
it
became
problematic.
Just
to
give
you
some
context.
We
have
about
twelve
hundred
and
fifty
four
unique
contributors
to
monorail
last
time.
I
checked.
So
if
you
were
to
take
everyone
in
this
room,
multiply
by
five
and
then
put
everyone
on
the
same
Ruby
on
Rails
application
and
commit
every
day
and
that's
kind
of
the
situation
that
we're
dealing
with.
So
it
was
pretty
crazy.
So
we
shipped
220
PRS
a
day
in
master.
A
We
have
about
thirty
thousand
database
columns
that
active
record
kind
of
manages
to
support
a
Ruby
on
Rails
application
and
up
until
like
maybe
a
couple
weeks
ago,
we
had
merged
a
hundred
and
fifty
five
thousand
PRS
just
in
monorail.
So
it's
it's
a
beast.
It's
such
a
beast
that
one
I
wanted
to
get
bitter
information
and
github
I
was
looking
at
this
for
about
20
minutes,
but
eventually
it
showed
up.
It
just
took
a
while.
A
Nobody
was
brave
enough
to
actually
look
at
that,
so
it
probably
had
to
do
some
caching,
so
you
can
actually
see
the
growth
so
you're,
looking
at
like
number
of
contributions
a
day.
So
at
some
point
we
were
hitting
eight
hundreds
and
you
can
actually
see
it
going
back
down
and
I'm
here
to
explain
you
why
we're
how
we
got
there
so
that
we
actually
have
less
changes
to
mole
arel
nowadays,
so
you're
looking
at
some
pretty
big
numbers,
three
hundred
and
seventy
thousand
commits
it's
probably
the
biggest
Ruby
on
Rails
application
out
there.
A
By
far
and
at
first
we
were
like
you
know,
maybe
we
don't
need
this
microservices
thing.
Maybe
we
can
just
kind
of
build
tools
around
molla
rail
and
we
can
cut.
We
can
keep
scaling
that
way.
It's
a
huge
investment
to
move
into
micro
services.
We
were
growing
the
company
at
the
time.
We
didn't
really
have
the
manpower
to
kind
of
shift
into
micro
services,
so
we
thought
you
know
why?
Don't
we
build
some
tools
around
this?
We
can
do.
We
can
make
it
happen.
A
So
the
first
two
we
built
a
scram
subscribe
is
kind
of
an
automated
automated
rollback
incident
prevention
tool
and
it's
kind
of
an
internal
tool
to
Airbnb
it's
kind
of
like
a
canary
tool.
So
every
time
a
production
deploy
happens,
we
consume
the
errors
that
are
coming
in,
so
we're
using
a
tool
called
sentry
if
you're
not
familiar
with
sentry,
it
catches
an
exception
and
then
kind
of
bubbles
it
up
to
your
back-end.
And
then
you
get
all
the
information
about
the
exception.
So
wasn't
as
a
deploy
is
going
out.
A
It
catches
most
of
the
errors
before
a
human
eye
can
even
see
them
on
a
graph
or
something
with
that
like
scram,
will
immediately
roll
back.
It's
it's
really
good.
So
this
is
what
scram
looks
like.
So,
as
you
can
see
the
white
as
a
char
that
we
were
running
and
at
some
point
we
initiated
the
deploy,
and
so
the
light
gray
is
a
new
deploy
going
out,
and
so
you
can
see
all
of
the
airs
coming
in
on
the
left
side.
A
A
It's
called
merge
queue,
so
you
see
this
picture
with
zombies
in
the
backgrounds,
so
what
developers
are
kind
of
like
zombies,
they
kind
of
want
to
ship
their
code
as
as
fast
as
possible,
but
everyone
wants
to
ship
their
code
before
they
go
home.
So
we
kind
of
saw
this
pattern
where
everyone
trying
to
ship
their
code
at
4
p.m.
and
so
people
merge,
6070
PRS
at
4:00
p.m.
A
You
can
also
write
the
NQ
keyword,
and
so
what
happens
you
can
see
it
here
in
this
animated,
gif,
I'm,
basically
typing
rocket,
and
then
my
PR
gets
and
queued
I
get
a
message
in
slack,
giving
me
an
ETA
of
about
when
my
mind
might
change
will
be
merged
and
then,
once
my
change
gets
merged
into
production,
I
get
a
ping
back
on
slack,
and
that
says
like
hey,
you
should
test
your
changes
on
next
and
then
I
could
test
my
changes.
So
this
is
great
because
it
used
to
be
kind
of
a
whack-a-mole.
A
Everyone
was
just
trying
to
hit
that
merge
button.
50
changes
might
not
at
once,
so
we
can
kind
of
use,
merge
queue
to
enforce
batches,
so
I
showed
you
the
life
of
a
pull
request
earlier.
So
we
added
this
extra
step
where
you're
kind
of
sitting
in
the
merge
queue
before
your
PR
gets
changed.
The
great
thing
about
merge
queue
is
also
it's
non-destructive.
If
you
are
in
the
merge
queue
and
you
need
to
run
into
a
meeting,
you
can
just
DQ
again
come
back
later.
A
I'm--You're
for
request
hasn't
been
merged
yet,
and
we
even
have
this
UI
and
deploy
board.
That
shows
you
this
date
of
the
queue,
so
you
can
kind
of
dig
in
all
and
into
all
the
PRS
that
are
currently
in
the
queue.
We
even
have
this
mechanism
that
allows
you
to
flag.
Your
PRS
ready
did
basically
telling
the
telling
the
merge
queue
that
you're
at
your
computer.
You
know
afk,
because
we
don't
want
to
merge
changes
where
the
developer
is
another
round.
A
So
if
you're
not
around,
we
kind
of
recycle
you
at
the
top
of
the
queue
and
then
eventually
DQ
you
if
you're
afk
for
too
long,
and
so
we
built
this
whole
queuing
system
just
so,
we
could
deal
with
monorail
and
things
were
okay
with
merge
queue
and
scram.
We
definitely
got
like
maybe
one
or
two
years
out
of
molar
l,
but
things
weren't
okay
for
too
long,
and
then
we
started
seeing
all
of
these
graphs.
We
started
seeing
people
in
slack
panicking.
A
We
saw
a
lot
of
incidents,
we
were
not
in
a
good
place,
and
so
we
had
to
make
a
more
more
drastic
change.
Meanwhile,
this
is
kind
of
my
team.
We
were
just
like
trying
to
keep
things
alive.
Like
alerts
were
going
off
left
and
right,
everyone
was
kind
of
panicking.
We
did
an
internal
survey
to
ask
people
like
hey.
How
satisfied
are
you
with
like
the
deploy
process
at
Airbnb
and
only
maybe
a
very
small,
maybe
10,
ish
percent
of
people
were
actually
happy
with
the
deploy
process.
Everyone
was
like
extremely
dissatisfied
or
dissatisfied.
A
So
those
are
not
the
kind
of
numbers
you
want
to
see
from
a
team
called
the
developer
happiness.
The
developers
were
clearly
clearly
not
happy,
so
we
had
to
do
something
about
that.
So
we
did
it.
We
actually
did
move
into
the
release
management.
I,
know,
I,
know,
I
said
you
know
democratic
deploys.
We
still
do
democratic
deploys,
but
it
didn't
work
for
Moeller
l.
It
was
just
too
big.
We
couldn't,
we
couldn't
do
it.
A
So
we
introduced
release
management's,
so
release
management
is
kind
of
a
team
that
kind
of
keeps
this
monorail
on
track
and
they
basically
go
into
the
merge
queue.
They
handpick
a
batch
of
20
changes.
They
merge
those
in
and
they
kind
of,
Shepherd
the
deploy.
So
we
actually
ended
up
building
some
tooling
in
to
deploy
board
to
kind
of
make
sure
that
the
release
managers
could
kind
of
Shepherd
the
deploy
themselves,
and
this
was
actually
a
pretty
dramatic
cultural
shift
for
Airbnb.
A
We
had
some
like
please
senior
software
engineers
who
really
liked
this,
like
continuous
delivery.
Palen.
They
really
liked
shipping
their
own
changes
and
moving
into
a
release.
Management
system
definitely
was
kind
of
a
kind
of
a
big
change
for
for
a
company,
but
we
had
to
do
it.
That
was,
we
had
to
do
it
kind
of
to
keep
the
lights
on.
We
only
do
the
lease
management
during
business
hours.
So
if
someone
was
to
ship
it
change
at
1:00
a.m.
A
they're
on
their
own,
it's
back
to
democratic
deploys,
so
there's
only
a
release
manager
on
call
during
business
hours.
It's
a
volunteer
rotation.
We
didn't
want
to
like
have
someone
sit
there
at
2:00
a.m.
deploying
changes
that
doesn't
sound
like
a
great
experience
either.
So
we
kind
of
found
a
middle
ground
there
and
the
results
were
promising.
We
were
able
to
merge
more
changes.
A
Our
velocity
went
up,
we
merged
more
pr's
a
day
and,
as
you
can
see
from
the
second
part
chart
people
were
much
more
satisfied,
even
though
people
said
like
you
know,
democratically
deploys
is
all
the
hotness
and
the
greatness
they
like
the
idea
of
it,
but
it
didn't
scale
well
for
a
monolithic
application
like
monorail,
but
obviously
we
needed
to
do
something
else.
Just
like
our
company
is
kind
of
moving
into
like
different
businesses.
For
example,
trips.
A
Our
infrastructure
also
needed
a
more
long-term
vision,
a
more
sustainable
plan
like
what
do
we
do
with
this
monorail
situation?
We
double
the
engineering
team
every
year.
How
do
we
not
run
into
this
huge
fire
where
nobody
can
deploy
changes
anymore?
So
we
came
up
with
this
long
term.
Vision
called
monorail.
A
We
actually
will
not
allow
developers
to
contribute
codes
into
mall
RL
anymore,
unless
it's
like
maintenance
code,
you
will
not
be
allowed
to
add
new
features
into
monorail,
and
this
is
really
to
kind
of
force,
people
to
adopt
this
micro
services
palette
and
so
we're
investing
very
heavily
and
services
oriented
architecture.
But
we
have
to
make
sure
that
we
do
this
carefully.
We
have
to
make
sure
that
we
don't
impact
company
growth
in
the
meantime,
you
know
the
company
is
growing,
we're
adding
more
engineers,
we're
spinning
out
new,
your
businesses
left
and
right.
A
A
So
let's
talk
about
these
two,
these
two
different
multiples.
One
might
ask:
why
do
you
have
two
mono
repos?
We
decided
to
kind
of
split
hermano
repos
based
on
language,
so
we
have
a
front
end
mono
repo
Hyperloop,
and
then
we
have
a
back-end
mono
repo
in
Java,
which
is
three
house,
so
we
can
kind
of
build
tooling
around
these
two
different
mono
repos.
We
found
that
that
scale
pretty
well
for
us.
So
what
is
Hyperloop
so
Hyperloop
is
our
front
at
mono,
repo
and
all
of
our
front
end
codes.
A
All
of
our
ER
be
Ruby
on
Rails
code
is
actually
moving
in
into
no
Jess.
So,
as
you
may
or
may
not
know
know
we
have
this
kind
of
server-side
react
strain
work
called
hypernova
that
we
open-source,
and
so
it's
based
on
hypernova
and
so
Hyperloop
basically
does
json
over
HTTP,
and
so
it
basically
allows
you
to
package
a
smaller
front,
end
components
with
an
OG
I
surrender
and
kind
of
like
host
it
by
itself,
and
so
you
can
do
independent
deploys.
So
if
you
look
at
Airbnb
dot-com,
you
can
think
about
the
different
pages.
A
A
So
this
is
what
IBM,
B's
front-ends
kind
of
looks
like
nowadays,
so
you
can
see
that
the
blue
part
is
just
one
of
the
many
hyper
loops,
and
so
you
can
put.
We
can
create
more
loops
based
on
businesses.
So
if
a
B&B
ever
wants
to
dive
into
a
new
business,
we
can
just
spin
up
more
Hyperloop
applications
and
then
we
kind
of
have
this
layer
of
like
shared
services,
what
we
call
like
kind
of
a
middleware
layer.
A
So
we
have
some
of
those
services,
for
example,
hundred
two
as
our
authentication
service
momento
those
sessions
Krakken,
which
is
our
API
gateway.
So
we
kind
of
have
a
few
dozen
services
that
kind
of
live
in
this
network,
middleware
layer,
so
basically
from
that
developers,
don't
have
to
worry
about
those
kind
of
things
like
authentication
and
sessions
and
stuff
like
that.
So
that's
our
front
end
mono
repo.
Obviously,
a
lot
of
our
code
also
lived
on
the
Ruby
side
of
a
Ruby
on
Rails
application.
We
had
a
lot
of
back-end
code
in
there.
A
So
what
do
we
do
with
that?
We
moved
it
into
another
mono
repo
called
tree
house,
and
so
why
tree
house
so
story
goes
that
our
most
popular
Airbnb
listing
is
a
treehouse
that
Airbnb
listing
is
on
the
island
of
Java
or
mono.
Repo
holds
Java
code.
So
that's
why
we
call
it
tree
house,
that's
what
the
legend
goes.
So
why
don't?
We
choose
Java
debatable,
but
it's
pretty
close
to
C++
and
performance.
It's
definitely
faster
than
you
know,
Ruby
and
also
static
typing.
A
We
found
that
we
really
got
a
lot
of
benefits
from
having
like
static
type
checking
analysis
on
the
back
end
and
a
lot
of
our
treehouse
applications
are
based
on
this
open
source
framework
called
drop,
wizards,
so
drop.
Is
it
this
kind
of
a
library
that
allows
you
to
make
these
like
micro,
restful
java
applications,
and
so
it's
not
the
same
as
a
hyper
loop,
so
hyper
loop
communicates
JSON
over
HTTP,
but
treehouse
can't
kind
of
holds
our
internal
back-end
services,
so
we
don't
communicate
with
JSON.
A
A
Well,
so
we
changed
our
data
format
for
enter
service
communication
into
drifts,
and
so
drift
is
basically
this
binary
format
that
allows
you
to
do
schema
validation,
it's
standardized,
it's
open
source
standard,
an
apache
license
and
we
actually
built
our
own
compiler
on
top
of
drifts.
So
internally
we
have
this
project
called
sparse,
so
sparse
as
a
C++
compiler
that
can
compile
drift
to
make
it
even
more
efficient
and
even
faster,
and
so
how
do
these
services
talk
to
each
other?
A
They
talk
to
each
other
by
doing
RPC
calls
over
TCP,
so
there's
no
JSON
over
HTTP
between
services,
and
so
basically
this
also
allows
us
to
do
cross
language
communications.
If
we
ever
wanted
to
introduce
a
cutland
service
or
a
go
service,
we
can
just
kind
of
talk
over
our
PC
using
drifts,
which
is
kind
of
this
language
that
all
of
these
programming
languages
can
share.
A
A
I
can
just
say
like
send
an
email,
but
I
don't
expect
the
backend
service
to
send
this
email
immediately,
so
we
use
Kafka's
or
a
message
queue,
so
you
would
basically
just
in
queue
a
message
just
as
like
send
an
email,
and
then
we
have
all
these
workers
that
are
kind
of
like
consuming
these
email
jobs
and
sending
them
out
so
you're
not
kind
of
waiting
around
for
that
job
to
complete.
So
anything,
that's
asynchronous
anything
that
takes
more
than
a
few
milliseconds
kind
of
goes
into
our
message
queue.
A
A
So
the
first
thing
we
did
is
we
build
a
tool
called
cube
gen,
so
we're
actually
moving
all
of
our
back-end
services
into
kubernetes.
So,
basically,
if
you
want
to
make
a
new
micro
service,
you
just
do
cube
gen
Ruby
and
it
spits
out
a
Ruby
on
Rails
application
integrated
with
all
of
our
services
out
of
the
box.
A
It
has
jitney,
it
has
all
of
the
gems,
so
you
can
make
a
service
within
seconds
and
it
comes
with
all
of
the
tools
inside
we
found
that
it
was
really
good
and
it
gives
a
lot
of
people
like
good
velocity
to
spin
up
new
services.
We
wanted
to
make
sure
that,
if
we're
telling
people
to
make
micro
services
that
we
had
good
tools
to
actually
create
those
services,
another
part
is
like
service
configuration.
A
It
used
to
be
where,
if
you
wanted
to
spin
up
a
service,
you
have
to
go
into
the
AWS
console
you
have
to
database,
you
have
to
create
an
s3
bucket.
You
have
to
set
up
a
github
repo,
so
you're
kind
of
all
over
the
place,
setting
up
dashboards
and
all
of
that,
so
we
created
this
project
internally
called
OneTouch,
and
so
basically,
with
a
single
touch,
you
should
be
able
to
change
not
only
your
applications
code,
but
also
your
infrastructure.
A
So
any
of
our
internal
applications
has
an
underscore
info
folder,
and
so
you
can
put
a
llamo
file
in
there,
for
example,
deploy
Bordeaux
and
it
will
kind
of
describe
what
the
deploy
process
for
your
application
looks
like
you
can
put
data
doc
in
there
anything
that's
kind
of
infrastructure
lives
in
this
folder.
If
you
want
to
add
an
s3
bucket,
you
would
configure
it
in
that
infraorder.
So,
once
you
check
in
your
changes,
your
infrastructure
will
kind
of
grow
together
with
your
codebase
and
it's
like
just
all
uniform.
It's
all
in
just
one
application.
A
It's
all
easy
in
one
spot
and
we
actually
allow
developers
to
kind
of
extends
this
underscore
infra
concept.
So
if
someone
wanted
to
like
add
Redis
support
to
it,
they
can
kind
of
extend
this
concept.
So
one
touch
is
really
good
because
it
allows
you
to
kind
of
like
put
all
your
configuration
in
the
same
repo
as
your
code
and
so
we're
moving
to
infrastructure
as
code.
So
basically
developers
no
longer
have
to
care
about
hardware,
so
we're
moving
into
kubernetes
we're
packaging.
A
All
of
these
micro
services
into
docker
containers
we're
scheduling
those
on
a
kubernetes
cluster.
So
if
you're
a
product
developer,
you
never
even
interact
with
any
of
the
instances
and
the
under
that,
and
so
we
put
some
sidecar
containers
next
to
your
base
container
for
some
shared
logic
like
service
discovery
and
stuff.
Like
that,
obviously
we
could
manage
all
of
our
infrastructure
using
kubernetes.
We
really
rely
pretty
heavily
on
AWS.
So,
for
example,
Amazon
s3
I
am
roles
load
balancers.
Those
are
managed
using
terraform,
fun
facts.
A
We
obviously
also
need
like
good
monitoring
tools,
so
we
built
this
new
tool
called
watch
point.
We
realized
we
had
a
lot
of
like
different
monitoring
to.
We
were
using
data
doc.
We
were
using
New
Relic,
we
were
using
Cabana,
but
they
were
kind
of
all
over
the
place,
and
so
we
couldn't
really
find
a
tool
that
kind
of
aggregates
all
of
them.
So
we
decided
to
build
it
in-house,
and
so
what
point
is
kind
of
this
observability
tool
that
allows
you
to
kind
of
observe
the
state
of
the
world
and
an
SOA
world
I'm?
A
So
it's
really
built
for
a
service
art,
oriented
architecture
from
the
ground
up,
and
it
has
this
like
service
graph,
so
you
can
actually
see
which
services
talk
to
other
services,
because,
if
you're
dealing
with
like
thousands
of
micro
services,
they're
all
communicating
with
each
other-
and
you
can
do
like
things
like
cross
service
tracking.
So
if
you're
sending
a
request,
you
can
kind
of
trace
it
all
the
way
from
our
load
bouncer
into
our
front-end
and
then
eventually
into
all
of
the
three
health
services.
And
then
back.
A
So
you
can
kind
of
trace
a
request
through
our
infrastructure.
We
also
kind
of
aggregate,
our
logs
and
Kabana,
and
we
add
links
to
Cabana
in
watch
Point,
and
so
it
a
grits
from
data
dock
a
new
relic
and
also
like
our
in-house
aggregation
service
called
up
shop.
So
we
basically
use
watch
point
as
our
kind
of
visualization
tool
to
see
it's
kind
of
our
pane
of
glass
into
this
micro
services
worlds,
and
you
can
even
add
s
loz,
so
an
SLO
as
a
service
level
objective.
A
So
basically,
we
basically
tell
people
to
set
up
these
sort
of
contracts
if
I'm,
building
a
micro
service
I
need
to
have
a
contract
that
says
like
okay,
I'm
promising
my
customers,
99
percent
uptime.
If
you're
building
an
email
service
and
it's
unreliable,
we
need
to
be
held
accountable,
so
we're
basically
tracking
all
of
these
as
silos
in
watch
point
as
well.
So
just
to
give
me
and
my
team
a
pat
on
the
back,
this
is
the
SLO
for
deploy
board.
A
So
you
can
see
we're
at
100%
uptime,
so
we're
doing
pretty
well
and
so
we're
tracking
all
of
these
SL
O's
in
watch.
Point
and
you'll
get
an
email
if
you're
not
kind
of
breaking
that
SLO.
So
this
basically
allows
us
to
kind
of
enforce
these
contracts
in
this
micro-services
world.
If
we're
talking
about
thousands
of
services,
we
need
a
team
to
be
responsible.
We
need
some
kind
of
like
contract
between
service
owners
and,
last
but
not
least,
we're
also
moving
in
to
deploy
pipelines.
A
So
it
used
to
be
where
anyone
can
kind
of
deploy
any
piece
of
code
to
any
environments
right
away.
So
you
could
basically
grab
a
piece
of
code
and
deploy
it
into
production
without
deploying
it
to
staging
first.
So
we
wanted
to
kind
of
create
a
dependency
graph
that
you
kind
of
have
to
traverse
and
that
dependency
graph,
basically
is
a
story
of
how
your
code
lands
into
production,
and
so
it
consists
of
stages
and
every
stage
consists
of
like
steps
and
then
every
step
has
tasks
and
every
tasks
have
operations.
A
So
you
can
kind
of
describe
your
whole
deploy
process
as
this
kind
of
like
a
dependency
graph.
You
can
even
mass
pipelines
into
other
pipelines,
and
so
this
is
really
useful.
Let's
say,
for
example,
that
you
wanted
to
spin
up
a
micro
servers
and
you
wanted
to
deploy
it
into
staging
down
in
Canary
and
then
only
when
certain
canary
metrics
are
good.
Then
you
want
to
deploy
into
production,
and
then
you
want
to
do
a
database
migration
and
for
a
developer
to
remember.
A
All
of
these
steps
is
just
not
doable
so
just
having
all
of
these
steps
kind
of
written
out
as
a
deploy
pipeline
and
a
llamó
configuration
is
just
a
very
useful
tool,
and
so
to
do
that
we're
actually
moving
into
spinnaker,
as
some
of
you
may
or
may
not
know.
Spinnaker
is
kind
of
this
open
source,
continuous
delivery
tool.
It's
built
in-house
by
Netflix,
they
open
sourced
it
Google
used
it
for
ways,
and
other
companies
are
kind
of
also
jumping
on
that.
A
So
with
Airbnb
we're
also
looking
at
moving
into
spinnaker
in
the
future,
and
so
you
can
see
a
very
basic
deploy
pipeline
right
there.
So
we're
basically
deploying
our
OneTouch
config
we're
waiting
a
little
bit
we're
deploying
to
staging
we're
waiting
a
little
bit
longer
and
we're
deploying
into
production.
That's
just
a
very
basic
pipeline,
but
you
can
create
this
really
intricate
dependencies.
For
example,
if
you
had
three
services-
and
you
wanted
to
wait
for
another
service
to
come
back
online,
you
can
kind
of
link
pipelines
together.
A
So
spinnaker
is
really
the
future
for
us
for
deploys
is
really
this
powerful
tool
for
us
to
kind
of
ship
code
into
production.
So
in
conclusion,
so
lessons
learned
for
us:
it's
been
a
long
journey.
We've
been
dealing
with
this
moral
mono
repo
for
a
very
long
time.
So
we
learned
a
lot
of
lessons
along
the
way.
I
would
still
say
if
you're
a
new
company
don't
bother
too
much
with
coming
up
with
this
fancy.
Service-Oriented
architecture
focus
on
your
business.
First
make
sure
you
have
customers
make
sure
you
can
actually
scale
your
business
first.
A
Are
we
angry
at
Nate
for
spinning
up
this
Ruby
on
Rails
application
and
kind
of
like
giving
us
this
year-long
migration?
No
we're
not!
He
did
what
he
had
to
do.
He
was
able
to
make
a
B&B
into
a
sustainable
business.
He
was
able
to
you,
know,
make
money
and
like
he
was
actually
able
to
make
sure
that
we
are
still
here
10
years
later.
So
you
want
to
think
about
service-oriented
architecture
and
later.
Why
and
my
personal
opinion,
service-oriented
architecture
requires
a
lot
of
expertise.
A
You
have
to
have
a
lot
of
like
really
good
infrastructure
engineers
and,
if
you're,
a
small
start-up,
you
might
not
even
have
like
a
dedicated
infrastructure
engineer
and
so
like
setting
up
all
these
tools,
it'll
probably
be
like
80%
of
your
workday,
so
don't
bother
just
make
a
quick
and
dirty
rails
application
and
then
take
it
apart
later.
I
also
want
to
emphasize
automation.
Anything
that
can
be
automated
should
be
automated.
A
You
can
actually
automate
that
by
using
a
tool
like
spinnaker,
and
you
also
want
to
automate
anything
that
like
interacts
with
your
infrastructure,
so
use
tools
like
terraform
to
spin
up
s3
buckets
rather
than
like
manually
going
into
the
s3
console.
That'll.
Save
you
a
lot
of
time.
That'll
also
give
you
kind
of
like
a
backlog
of
what
happens
so
to
come
back
to
that
point,
immutable
infrastructure.
A
You
really
want
to
make
sure
that
if
a
disaster
happens,
you
can
kind
of
grab
this
bomb
repository
that
holds
all
of
the
configuration
of
an
entire
company,
and
you
can
basically
spin
up
your
entire
data
center
by
just
looking
at
your
terraform
and
kubernetes
config.
So
immutable
infrastructure
is
really
powerful
concept
that
basically
keeps
track
of
all
the
changes
to
your
infrastructure
as
if
they
were
code.
It
basically
allows
you
to
think
of
your
infrastructure
as
source
code
and
it's
a
really
powerful
concept.
A
Let's
say
that
you
had
a
point
to
server
out
there
and
someone
installed
like
an
Apache
on
it
and
it
wasn't
tracked.
You
wouldn't
really
know
how
to
reproduce
that
server.
If
you
have
a
mutable
infrastructure,
you
can
always
reproduce
any
piece
of
infrastructure.
Just
like
you
can
recompile
any
piece
of
code.
A
Another
important
point
is
people
always
talk
about
s
away
from
backends,
but
we
took
it
a
step
further.
We
actually
also
did
s
away
for
front-end,
so
we
came
up
with
this
kind
of
Hyperloop
concept
to
build
like
smaller,
independent,
independently
deployable
front-end
applications
about
50%
of
our
developers
are
like
front-end
developers.
A
So
we
wanted
to
make
sure
that,
even
though
we're
like
splitting
out
all
of
the
Ruby
logic
out
of
monorail,
we
want
to
make
sure
that
they
also
had
a
good
story
in
their
day-to-day
work
and
that
they
could
deploy
changes
just
as
quickly
as
a
back-end
engineer.
And
last
but
not
least,
you
need
really
good
monitoring
tools,
especially
if
you're
moving
from
a
monolith
into
an
SOA
monitoring.
A
monolith
is
like
pretty
straightforward.
A
You
just
have
one
application
and
you
just
make
sure
that
it's
up-
and
you
can
kind
of
build
your
tooling
around
that,
but
suddenly
you're
find
yourself
in
this
like
microservices
world,
and
now
you
have
dozens
hundreds,
maybe
even
thousands
of
services
that
are
communicating
with
each
other
and
something
goes
wrong.
You
don't
know
where
it
went
wrong,
so
having
really
good
tools
that
kind
of
allow
you
to
visualize
I
visualize,
how
a
request
kind
of
travels
through
all
these
dependencies.
You
really
need
good
monitoring
tools
before
you
embark
in
this
journey
of
micro
services.
A
If
you
want
to
ask
me
some
questions
just
feel
free
to
hit
me
up
after
this
talk
and
I'm
just
happy
to
share
this
journey
with
you
guys,
I
hope
that
you've
learned
something
today
and
that
you
can
kind
of
take
what
we
learned
at
Airbnb
back
into
your
own
companies
and
as
you
embark
into
this
journey
in
micro
services,
maybe
learn
a
lesson
or
two
for
us
from
us.
Thank
you.