►
Description
Spinnaker at Airbnb - Jens Vanderhaeghe & Manish Maheshwari, Airbnb
It’s been almost two years since Airbnb started investing into Spinnaker. In the past year we’ve hit some significant milestones in our quest to bring Continuous Delivery to all of Airbnb engineering. In this talk by Jens Vanderhaeghe (Software Engineer) and Manish Maheshwari (Product Manager), you will learn how we’ve made Spinnaker generally available and how we’ve convinced hundreds of teams to adopt Spinnaker for their deployment process. We’ll cover some of the challenges we’ve faced fitting Spinnaker to Airbnb’s specific use cases and how we leverage Spinnaker to reduce Airbnb site downtime and improve deployment velocity.
For more Continuous Delivery Foundation content, check out our blog: https://cd.foundation/blog/
A
A
So
who
are
we?
My
name
is
jens.
I
am
originally
from
belgium.
I
am
a
software
engineer
at
airbnb.
I
work
on
the
continuous
delivery
team,
so
really
my
day-to-day
is
always
involving
continuous
delivery.
I
was
originally
a
front-end
engineer
and
then,
when
I
joined
airbnb
a
couple
years
ago,
I
transitioned
into
infrastructure.
B
A
So,
let's
take
a
little
overview
of
our
presentation
before
we
dive
deeper,
so
we'd
like
to
talk
about
three
different
topics
today.
The
first
topic
is
spinnaker.
General
availability
we'd
like
to
talk
about
how
we
achieved
general
availability.
2021
was
really
a
breakthrough
year
for
us,
spinnaker
is
the
default
deployment
tool
at
airbnb.
Nowadays,
we'd
like
to
talk
about
how
we've
achieved
that
and
the
lessons
that
we've
learned
along
the
way.
We'd
also
like
to
spend
some
time
to
talk
about
ui
and
ux
improvements
that
we've
made
to
spinnaker.
A
A
Last
but
not
least,
we'd
like
to
talk
about
how
we've
extended
spinnaker
with
new
features,
we've
really
chosen
spinnaker,
because
it's
a
flexible
tool
for
continuous
delivery.
I'm
sure
a
lot
of
other
companies
are
looking
at
spinnaker
for
those
exact
same
reasons.
So
we'd
like
to
talk
about
how
we've
extended
spinnaker
on
our
side
to
fit
some
of
airbnb
specific
use
cases
manisha
is
going
to
talk
about
spinnaker's
journey.
B
Great
thanks
and
so
I'll
go
over
the
spinnaker
journey
with
you
all
and
how
spinning
a
journey
at
airbnb
with
you
all
and
how
from
where
we
started
to
where
we
are
today.
I'm
pretty
excited
to
share
this
with
you
all,
so
next
slide
cool,
so
it
will
be
like
most
of
the
startups
started
with
the
monolith
architecture.
B
It
really
worked
well
until
the
engineering
organizations
started
to
grow,
but
then
we
started
to
experience,
challenges
with
reliability
and
and
the
development
velocity
changes
affecting
the
downstream
had
to
be
coordinated
and
many
times
were
missed,
causing
causing
causing
outages
to
to
innovate
and
to
respond
to
changes
faster.
We
we
modernized
our
tech
stack
to
be
modular
and
cloud
native.
B
B
This
move
to
a
modernized
architecture
led
to
several
new
challenges,
and
one
of
them
is
is
outages
due
to
code
changes.
We
were,
we
were
still
using
our
our
in-house
deployment
tool,
which
was
built
for
monolith.
This
particular
tool
did
not
support,
deploy,
pipelines
and
and
an
orchestration
of
complex
workflows.
It
was
not
even
built
for,
like
our
cloud
native
workloads,
any
outages
had
a
significant
impact
on
the
on
our
host
and
guest
experience.
B
Imagine
you
had
to
yeah,
you
have
traveled
with
your
family
and
are
having
issues
trying
to
check
in
into
your
airbnb.
You
open
the
airbnb
app
only
to
find
out
that
you're
unable
to
check
connect
with
your
host.
You
are
now
struck
and
frustrated,
and,
and
we
as
a
company,
don't
want
such
experiences.
We
we
want
our
system
to
have
high
availability
to
provide
magical
trips
to
our
to
our
guests,
and
so
so
that
is
the
impact
of
outages
on
on
outages
on
the
airbnb
product.
B
In
the
when
I
talk
about
the
development
workflow
with
the
old
tool.
Basically,
it
worked
like
a
service
operator,
had
to
remember
to
start
a
release.
They
would
cut
a
release,
branch
run,
some
verification
tests
and,
in
many
cases,
hand
over.
The
changes
to
this
to
the
qa
team.
Qa
team
would
would
run
these
tests
and
hand
it
back.
Then.
The
developer
would
deploy
to
pre-production
fleet
manually,
verify
a
few
metrics
and
then
and
then
promote
the
release
to
production
in
many
cases.
B
Next,
so
I
mentioned
our
our
our
tool
did
not
have
the
concept
of
pipeline,
and
it
was
not
built
for
cloud
native
stack,
which
led
to
a
lot
of
inconsistent
release
patterns
leading
to
outages.
We
made
a
decision
to
transition
to
spinnaker,
which
was
purpose-built
for
for
cloud-native
workloads.
We
we
did
evaluate
a
lot
of
various
options
and
landed
on
spinnaker,
mainly
because
it
was
it
served
our
purposes
itself.
B
It
served
our
needs
and
it
was
an
open
source
tool
and-
and
we
had,
we
also
had
to
customize
a
lot
of
large
spinnakers,
a
lot
of
airbnb
specific
patterns
into
a
tool
and
and
spinnaker
being
an
open
source
tool
provided
and
provided
those
capabilities
with
spinnaker.
We
got
fast,
repeatable
and
safe
pipelines
which
significantly
improved
the
overall
reliability
of
airbnb
product.
B
B
Developers
were
no
longer
babysitting
their
deployments.
They
would
check
in
their
code
and
move
on
to
the
next
task,
increasing
their
productivity.
It
was.
It
was
easy
for
for
service
operators
to
reproduce
an
issue.
An
issue
in
the
release,
as
the
deployment
pipeline
was
then,
was
codified
and
it
was
easy
to
audit
each
step
in
in
the
release
process.
B
B
B
Should
they
be
investing
more
on
the
reliability
improvements
versus
the
the
feature
features
for
for
the
business
during
pilot.
We
we
realized
that
there
was
a
a
steep
learning
curve
in
terms
of
using
a
new
tool
and
also
adopting
a
continuous,
a
new
culture
of
continuous
delivery
within
the
company.
B
B
Next,
so
what
worked
for
us?
How
did
how
did
we
get
the
entire
company
invest
into
a
completely
new
deployment
tool,
at
a
time
when,
when
we
were
focused
on
launching
new
products
for
our
host
and
guest,
such
as
online
experiences
and
like
everyone
else
dealing
with
strategic
changes,
strategic
changes
due
to
the
pandemic?
B
So
mainly,
we
focus
on
three
things.
First,
we
focus
on
customer
benefits.
We
did
not
force
anyone
to
migrate
from
the
old
tool
to
the
new
tool.
We
we
provided
a
new
tool
as
an
alternative
to
the
to
the
existing
tool.
We
help.
Customers
realize
10x
benefits
from
the
from
from
from
moving
on
to
spinnaker,
by
by
enabling
them
to
increase
the
developer
product,
the
product
productivity
of
their
teams
and
also
significantly
increasing
improving
the
reliability
of
the
services
that
they
own.
B
B
B
B
B
Next
so
finally,
I
just
wanted
to
share
this
graph
with
you,
which
is
the
the
uptick,
the
exponential
growth
that
we
saw
in
2020
as
soon
as
we
launched
using
the
as
soon
as
we
launched,
and
we
used
the
three
tactics
that
I
mentioned
and,
and
we
have
more
than
700
services
today-
that
are
using
spinnaker,
spinnaker
and
and
we
are
adding
more
workloads
within
the
company.
So
finally,
using
the
three-pronged
approach
benefit
basically,
first
from
benefits.
B
A
All
right
now
we'll
dive
a
little
deeper
into
some
of
the
things
that
manish
talked
about.
Let's
start
with
the
ui
and
ux
improvements
that
we've
made
to
spinnaker,
as
many
mentioned
before
when
we
started
adopting
spinnaker,
our
customers
had
some
friction
with
spinnaker's
open
source,
ui
and
user
experience.
A
We
did
some
customer
research,
and
here
are
some
of
the
quotes
that
some
of
the
engineers
at
airbnb
gave
us.
The
ui
is
confusing
it's
hard
to
identify,
live,
builds,
it's
unclear
how
ci
and
cd
are
integrated.
This
makes
a
mediating
instance,
error-prone
and
difficult.
This
was
a
very
common
feature
request
or
very
common
piece
of
feedback
when
we
initially
started
rolling
out
spinnakers
the
other
section
of
commons
that
we've
got
revolves
around
the
debugging
experience
engineers
mentioned
to
us
that
it
was
often
very
difficult
to
debug
a
pipeline
failure.
A
Now
this
is
actually
pretty
important
to
us,
because
when
engineers
are
debugging
a
pipeline
failure,
they
may
or
may
not
be
in
an
active
incident,
and
we
want
to
make
sure
that
we
can
resolve
incidents
as
quickly
as
possible
without
the
tool
getting
into
the
way
of
the
engineers
and
the
day-to-day
workflow.
So
what
do
we
do?
The
first
thing
we
did
is:
we've
integrated,
airbnb's
ci
platform.
We
really
wanted
to
make
sure
that
spinnaker
is
not
a
stand-alone
tool,
but
it
integrates
within
the
entire
ci
cd
story
and
doesn't
just
do
cd.
A
A
A
We've
made
some
small
changes
to
make
it
more
easy
to
spot
how
ci
and
cd
are
integrated
and
we've
also
added
some
more
prominent
colors
around
failed
pipelines,
so
that
developers
can
quickly
see
where
a
pipeline
failed
and
why
a
pipeline
fails.
The
last
thing
we
did
is
we
actually
added
a
improved
deployment.
History
view
so
now
that
we
have
this
metadata
around
snapshots,
we
can
actually
correlate
deployments
with
a
certain
ci
bill.
A
The
ci
run
and
we
do
have
the
pipeline
executions
view,
which
will
actually
show
us
which
snapshots
went
through
specific
pipelines,
but
our
developers
actually
wanted
more
information.
They
wanted
a
different
perspective
on
the
same
data.
So
what
they
really
requested
is
show
me
the
last
15
production
builds
now
in
our
use
case.
We
might
have
multiple
pipelines
that
can
all
deploy
to
production,
so
finding
that
information
in
the
pipelines,
you
can
actually
be
pretty
difficult.
A
So
what
we
did
is
we
basically
created
a
second
view
on
the
same
data,
but
we
slice
the
data
by
environments
and
by
snapshot.
So
in
this
page
you
can
actually
filter
it
on
the
specific
environments.
So
here
we
can
do
something
like
I'm
showing
the
last
10
production
deploys
and
then
actually
seeing
the
pipelines
associated
with
that.
So
we're
using.
A
And,
last
but
not
least,
we've
made
some
improvements
to
the
debugging
experience.
We've
added
small
but
meaningful
touches
like,
for
example,
showing
specifically,
which
stages
failed,
which
is
very
useful
and
in
case
of
airbnb
we
have
some
services
with
pipelines
that
can
consist
out
of
50
or
60
stages,
so
getting
to
the
specific
stage
that
failed,
very
quickly
can
save
a
lot
of
time
and
can
potentially
save
save
a
lot
of
time
during
a
active
incident.
A
Let's
also
talk
about
how
we've
extended
spinnaker
with
new
features
new
functionality.
How
did
we
do
that?
Well,
actually,
spinnaker
is
very
flexible,
specifically
spinnaker's
service
or
its
orchestration
service.
What
we
basically
did
is
we
taught
spinnaker's
orca
service
to
speak
airbnb,
service,
idl
or
interface
definition,
language.
So,
what's
idl,
it
is
basically
a
abstraction
layer
on
top
of
http
that
has
is
that
is
strongly
typed
and
that
allows
airbnb
services
to
communicate
with
each
other.
A
And
so
what
we
did
is
we
added
a
new
stage
to
orca
that
allows
people
to
sense,
specific
requests
in
the
airbnb
idl
format,
and
by
doing
that,
we
actually
enable
people
to
interact
through
spinnaker
through
a
single
stage,
with
all
of
the
all
the
services
that
we
have
available
within
our
internal
tooling.
So
this
allows
us
to
do
things
like
communicating
with
the
ci
platform
communicating
with
deploy
boards
our
old
deployment
tool
and
even
do
things
like
triggering
fault
injection
tests,
integration
tests,
you
name
it.
It
is
possible
with
spinnaker.
A
Now,
we've
also
added
a
new
feature,
specifically
after
some
user
feedback.
So
the
feedback
that
we've
got
was
that
in
ci,
when
something
fails,
the
ci
will
fail
and
they
won't
be
able
and
developers
are
not
able
to
deploy
it.
But
when
someone
actually
deploys
a
build
to
a
certain
environment
and
a
failure
gets
caught
in
cd,
nothing
prevents
users
from
actually
redeploying
the
same
chains.
It's
not
obvious
that
a
change
was
previously
deployed
and
had
an
issue
with
it.
A
So
what
we
did
is
we
added
a
little
bit
of
metadata
that
allows
us
to
mark
a
specific
build
as
bad
so
now
when
people
are
actually
running
a
change
to
a
cd
pipeline
and
spinning
it
and
something
gets
caught
in
canary
analysis.
It's
very
easy
to
mark
that
specific,
build
as
bad
when
the
next
engineer
comes
in
and
they
will
be
presented
with
a
warning
not
to
deploy
this
build
and
because
it
actually
has
already
failed
cd
in
the
past.
A
We
also
wanted
to
make
spinnaker's
value
very
clear
to
our
customers,
as
manish
mentioned
before,
we
didn't
want
to
forcefully
adopt
spinnaker.
So
what
we
did
is
we
generated
dashboards
for
each
service
specifically.
So
when
we've
adopted
our
first
few
services,
we
basically
had
the
data
to
prove
that
spinner
could
actually
improve
the
reliability
of
those
services.
This
gave
the
engineers
that
initially
adopted
spinnaker
the
tools
to
go
back
to
their
management
and
say:
hey.
Look
at
this
data.
We've
actually
improved
the
reliability
by
using
spinnerbait.
A
We
also
created
an
in-house
tool
for
declarative
pipelines,
anchored
it's
basically
pipelines
as
code.
You
can
think
of
it
as
you
write
a
yaml
file
and
then
that
spits
out
spinnaker
specific
configuration
anker
is
airbnb
unique.
It's
basically
an
abstraction
layer
that
allows
us
to
generate
all
of
these
airbnb
specific
stages
like
integration
tests
that
we
use
the
service
idl
for
that
I've
talked
about
before.
A
A
First
for
the
cd
team,
we
have
the
daunting
task
of
upgrading
spinnaker.
You
might
have
heard
us
mention
a
few
times.
Oh,
we
should
upstream
some
of
these
changes.
Unfortunately,
while
we
were
adopting
spinnaker,
we've
neglected
upgrading
the
open
source
version
of
spinnaker
that
we
run,
and
we
are
now
several
verses
behind.
A
A
We
obviously
will
continue
to
invest
in
spinnaker's,
scalability
and
uptime
as
we
onboard
more
services
onto
spinnaker.
We
need
to
make
sure
that
we
can
continue
delivering
a
great
experience
as
an
infrastructure
team.
We
hold
ourselves
to
a
high
standard
and
we
want
to
at
least
have
two
nines
in
terms
of
reliability,
for
our
internal
tools,
especially
a
tool
that
is
in
the
critical
path
of
the
deployment
tool
deployment
flow.
A
A
A
A
A
We
are
planning
to
make
some
extensions
to
kayanta
so
that
it
can
work
with
our
tools
for
feature
flags.
That's
something
that
we're
looking
at
in
2021,
but
we
are
currently
just
using
the
native
spinnaker
analysis,
so
yeah,
that's
what
we're
currently
doing.
I
think
nabil
asked
is
anchor
available
as
open
source,
so
it's
sort
of
a
similar
story
to
to
the
rest
of
our
extensions.
A
A
A
Our
first
goal
is
to
get
all
deployments
on
onto
spinnaker.
We
still
have
a
subset
of
services
that
are
not
deploying
using
spinnaker
like
edge
cases.
I
would
say
so.
Those
are
like
gems
npm
packages,
ec2
services,
so
we
want
to
make
sure
we
have.
We
cover
100
of
deploys
and
then
we'll
be
looking
into
managed
delivery
probably
next
year-
and
we
are
pretty
excited
about
about
that.