►
From YouTube: Airbnb on Istio
Description
#IstioCon2021
Presented at IstioCon 2021 by Weibo He & Stephen Chan
In this presentation, we will walk through Airbnb’s Istio Journey - why we needed a modern service mesh, how we vetted Istio as the solution, where we are today, the lessons we learnt along the way, and our future plans. We will cover topics including:
Airbnb’s multicluster/cell setup, problems we ran into/ideas for UX improvements
Airbnb’s upgrade setup for gradual rollout of newer versions of Istio
Airbnb’s test pipeline for vetting features we care about
How we handled k8s & mesh expansion in a consistent philosophy
How we approached migration (zero downtime, no regression)
Airbnb’s learnings/pain points/future expectation with Istio
Current areas of open discussion - come talk to us more about this
A
Welcome
everyone,
stephen
and
I
are
both
engineers
working
on
the
cloud
foundation
team
at
airbnb.
In
today's
case
study,
we
are
excited
to
walk
you
through
airbnb's
issue,
journey
to
understand
why
we
started
the
journey.
Let's
take
a
trip
down
the
memory
lane
since
2013
airbnb
has
been
using
an
service
discovery
system
called
smartstack
that
we
open
sourced
on
a
high
level.
Smartstack
uses
zookeeper
as
the
service
registry
and
relies
on
a
bunch
of
sidecars
to
read
and
write
service
discovery.
A
A
A
In
the
end,
we
ended
up
with
four
init
containers
and
six
sidecar
containers
for
smartstack
to
work
on
kubernetes,
resulting
in
the
complexity
and
maintenance
cost
to
explode,
to
make
matters
worse
as
the
size
of
the
mesh
grew
and
the
rate
of
changes
in
the
mesh
increased.
Both
zookeeper
and
sidecar
started
seeing
scalability
issues
in
2019
we
decided
to
stop
patching
smart
stack
and
search
for
a
modern
service
mesh
solution
with
the
following
requirements
in
mind.
A
First,
the
new
service
mesh
must
be
able
to
propagate
changes
quickly
and
scale
to
tens
of
thousands
of
managed
proxies.
The
new
service
mesh
must
provide
complete
security
features
like
mtls
and
search
rotation
for
data
plane
proxy.
We
preferred
envoy,
as
we
had
in-house
operational
expertise
with
it
as
the
secondary
proxy
for
smartstack
we'd,
also
like
the
new
service
mesh
to
provide
rich
features
in
traffic
management,
resilience
and
observability
and
last
but
not
the
least.
The
new
service
mesh
must
support
kubernetes
as
well
as
ec2.
A
A
A
As
airbnb
has
a
few
dozen
kubernetes
clusters
in
production,
another
concern
we
had
was
around
multi-cluster
support.
We
knew
very
early
on
that.
We
wanted
to
do
an
external
steel
deployment
for
the
following
reasons:
first,
we'd
like
tight
access
control
for
the
control
plane.
Second
we'd
like
to
isolate
control,
plane,
components
from
data
plane,
workload
and
third
we'd
like
ice
we'd,
like
ease
of
of
operation
provided
by
the
central
control
plan.
A
A
A
In
early
2020,
mesh
expansion
on
istio
was
very
manual.
We
evaluated
two
options
to
support
ec2,
including
building
a
zk
plugin
for
istio.
We
didn't
go
this
route
because
we
no
longer
want
to
maintain
zookeeper
second
building
a
custom,
sidecar
and
kubernetes
controller
to
update
service
entries.
We
built
a
prototype
and
originally
planned
to
contribute
to
the
community.
A
A
Stephen
will
talk
more
about
our
setup
in
a
few
slides,
here's
the
final
easter
architecture.
We
landed
on
it's
being
used
active
in
production
today,
as
mentioned
previously,
we
run
ecod
in
a
dedicated
kubernetes
cluster.
Only
control,
plane,
components
run
from
this
cluster
and
only
mesh
administrators
have
permission
to
this
cluster.
A
We
added
a
simple
configuration
file.
Airbnb
engineers
interact
with
the
istio
crds
will
be
generated
from
this
configuration
file
as
part
of
the
ci.
The
generated
crds
will
then
be
deployed
and
managed
by
our
deployment
system.
All
seo
configuration
changes
are
monitored
and
can
be
easily
rolled
back
within
a
single
mesh.
We
run
a
flat
network
kubernetes
parts
across
kubernetes
clusters
assigned
vpc
ips
and
communicated
directly.
A
Today,
airbnb
runs
in
a
single
region.
We
are
working
actively
to
run
airbnb
from
multi-regions.
We
will
be
running
one
or
a
few
meshes
per
region
depending
on
the
size
of
the
region,
and
one
mesh
does
not
know
about
workloads.
In
other
meshes
cross.
Mesh
traffic
will
be
routed
to
the
ingress
gateway
running
in
the
destination
mesh.
A
A
To
securely
handle
multi-region
traffic,
we
need
a
strategy
to
authenticate
between
regions
to
ensure
a
ca
in
one
trust.
Domain
cannot
forge
certificates
from
another
trust
domain.
We
enforce
name
constraints
in
our
ca
hierarchy
and
denying
access
by
default
access
model
for
each
region.
We
assign
independent
root
cas
with
trust
domain,
locked
name
constraints.
A
We'd,
like
the
new
service
mesh
migration
to
be
safe
and
easy.
As
a
result,
we
support
edge
by
edge
migration,
where
an
edge
is
defined
as
a
logical
edge
in
the
service
mesh
from
service.
A
and
server
to
service
b.
A
service
should
not
require
a
leap
of
faith.
When
onboarding
the
new
service
batch,
it
should
be
able
to
migrate
each
of
its
edges,
one
by
one,
starting
from
the
least
important
edges
and
for
critical
edges.
A
We
support
percentage
based
traffic,
shifting
from
smart
stack
to
the
new
service
mesh,
allowing
side-by-side
comparison
of
error
rate
and
latency
in
case
anything
goes
wrong.
Service
owners
can
quickly
roll
back
throughout
the
migration
process.
We
provide
tooling
to
generate
the
migration
configs
automatically
here's
how
the
traffic
shifting
works
on
a
high
level.
A
A
Then
we
added
traffic
shifting
capability
to
airbnb's
standard
http
client
to
shift
traffic
from
smart
stack
to
the
new
service
mesh.
A
service
owner
can
simply
increase
the
traffic
percentage
using
our
dynamic
configuration
system
and
as
traffic
is
being
ramped
up.
The
service
owner
has
access
to
a
migration,
dashboard
tracking
the
changes
in
error
rates
and
latency.
B
Thanks
level
I'll
now
talk
about
upgrades,
istio
publishes
four
minor
releases
a
year
and
so
far
there
have
been
a
lot
of
changes
with
each
release
we
upgrade
soon
after
new
minor
releases
and
as
soon
as
possible
after
security
releases,
we
aim
to
minimize
toil
and
catch
operational
safety
issues
as
early
as
possible
in
the
upgrade
process.
We
break
this
process
down
into
four
main
steps.
B
B
B
B
B
We
use
the
default
profile
and
then
always
pass
in
a
base
operator
spec,
which
has
settings
common
to
all
types
of
kubernetes
clusters.
Additional
operator
specs
get
layered
on
top.
These
are
more
specific
to
the
type
of
mash
and
cluster.
We
are
installing
istio
on
the
top
right
gray
box
shows
our
sandbox
mesh
setup,
which
is
single
cluster
and
ephemeral.
B
In
addition
to
using
operator
specs,
we
directly
concatenate
custom
objects,
such
as
mesh
wide
envoy
filters
to
the
manifests
for
upgrades.
We
download
the
new
version
of
stl,
update
operator
specs
and
generate
manifests
which
get
checked
in
this
lets.
Team
members
perform
code
review
on
changes
compared
to
previous
versions
before
deploying
istu
supports
canary
control,
plane
deployments
since
1.6,
so
most
control,
plane
object.
Names
are
suffixed
with
the
revision,
however,
objects
such
as
rbac,
configs
and
custom
resource
definitions
are
not
suffixed
and
have
a
blast
radius
affecting
all
controller
plane
revisions.
B
So
we
check
changes
to
those
objects,
particularly
closely
after
manifests,
are
checked
in
we
start
control
plane
deploys
first,
we
deploy
to
the
sandbox
mesh
and
run
integration
tests
to
check
for
regressions
in
seo.
Next,
we
deploy
to
the
test
mesh,
which
runs
some
toy
workloads
which
generate
traffic
and
finally,
we
deploy
to
the
production
mesh.
B
In
the
last
slide,
I
talked
about
running
integration
tests
before
deploying
the
control
plane
to
prod.
Let's
talk
about
some
of
the
test
cases
that
were
checked
before
each
control,
plane
and
data
plane
deploy.
The
most
important
features
are
around
the
overhead
of
the
data
plane
in
terms
of
resource
consumption
and
latency.
B
B
B
After
the
control
plane
is
rolled
out,
we
start
upgrading
the
data
plane.
You
can
upgrade
a
kubernetes
deployment
or
other
workload
by
changing
the
revision
label
on
the
pod
template
when
pods
are
created,
the
new
istio
injection
web
hook
gets
called.
However,
we
need
a
safe
and
scalable
way
to
upgrade
thousands
of
workloads.
B
A
B
Sidecar
injector
continues
doing
most
of
the
heavy
lifting
the
rollout
functionality
of
crispr
is
more
important
for
us.
Most
of
our
workloads
use
standardized,
namespace
suffixes,
so
crispr
allows
us
to
select
the
revision
label
based
on
these
suffixes
for
a
given
suffix.
We
can
also
split
revision
labels
by
percentages,
for
example,
pods
in
25
percent
of
namespaces
ending
in
staging,
might
use
revision
1.6.11
and
the
remaining
75
would
use
1.6.9,
namespace,
suffix
matching
and
percentage
rollouts
give
us
fine
grid
control
over
how
proxies
connect
to
the
new
control
plane.
B
In
addition,
because
crispr
runs
as
a
mutating
admission
controller,
we
can
rely
on
regular,
node
and
pod
rotations
to
gradually
update
the
proxy
and
pods.
Without
forcing
all
service
owners
to
redeploy,
you
can
read
more
about
other
features
and
design
decisions
made
by
crispr
in
our
blog
post,
let's
step
away
from
testing
and
upgrades
to
talk
about
our
strategy
around
supporting
vms
on
nistial,
like
many
companies,
airbnb
has
a
number
of
workloads
which
are
hard
to
run
on
kubernetes
for
various
reasons.
So
this
is
important
to
us.
B
Here's
a
high
high-level
feature
comparison
of
running
istio
on
kubernetes
and
istio.
On
vms,
as
a
1.5,
the
two
biggest
gaps
for
us
were
that
there
was
no
automation
for
adding
and
removing
vm
instances
from
a
service
and
that
there
was
no
support
for
checking
whether
instances
were
healthy
and
ready
to
serve
traffic.
B
B
Now
that
we've
covered
the
evolution
of
vm
support
in
upstream
istio,
let's
do
a
quick
rundown
of
how
we
provide
each
of
these
pieces
of
functionality
at
airbnb
for
installing
the
proxy
istio
provides
debian
and
rpm
packages.
We
want
to
use
the
same
custom
envoy
filters
that
we
use
in
kubernetes,
so
we
can't
directly
use
those
packages.
B
Instead,
we
must
we
bundle
up
our
custom
envoy
pilot
agent,
binaries
and
a
few
other
files
into
a
custom
artifact
for
configuring
proxy
seo
offers
a
sidecar.end
environment
variable
script
that
can
change
the
behavior
of
pilot
agent.
You
can
also
run
istioctlx
workload.
Injury
configure
command
to
automatically
set
environment
variables
since
airbnb
is
building
a
custom
proxy
artifact
we
opted
to
instead
create
a
proxyrunscript
template
which
is
rendered
for
individual
workloads.
B
This
also
integrates
slightly
better
with
our
internal
tooling
for
pki
bootstrapping
seoctlx
workload.
Entry
configure
also
provides
a
token
in
a
ca,
but
it's
up
to
users
to
transfer
them
to
the
vm
that
they
will
be
used
on.
We
built
some
internal
tooling,
which
runs
on
each
vm
instance
and
regularly
fetches
the
ca
token
directly
from
kubernetes.
B
B
B
B
Third,
it's
a
good
idea
to
have
one
or
more
additional
pairs
of
eyes
on
manifest
before
they
get
deployed,
even
if
you're,
using
a
canary
control
plane
and
speaking
of
which
always
use
a
canary
control
plane
and
gradually
roll
your
data
plane
onto
the
new
one
fifth
test.
The
features
you
use
in
a
pre-production
environment.
Nobody
cares
more
about
your
services
than
you
and
your
team.