►
From YouTube: Airbnb Service Discovery: Past, Present, Future (Challenges of Change) - Chase Childers, Airbnb
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Airbnb Service Discovery: Past, Present, Future (Challenges of Change) - Chase Childers, Airbnb
In 2013, Airbnb released an open source service discovery solution (SmartStack) and has functioned on the same framework for years. Historically our infrastructure ran on AWS EC2 instances and utilized HAProxy (within Smartstack) for proxying traffic. With a migration to Service Oriented Architecture and Kubernetes, our service discovery must also change. In this presentation we will cover the evolution of our service discovery framework starting with where we started, where we’ve been, where we’ve failed, and where we’re going (hint: Envoy) at Airbnb. This includes both our missteps and our learnings from migrating within a hybrid EC2/Kubernetes world. We’ll dive deep into topics such as challenges of managing and migrating your own service discovery stack, migrating ingress and egress traffic independently, and rolling out infrastructure changes across a massive fleet of services.
https://sched.co/UaVz
A
Hello,
everybody
we'll
be
talking
for
about
service
discovery
today
at
Airbnb,
we'll
be
kind
of
walking
through
the
transition
from
Airbnb
over
the
years
talking
about
where
we
started,
where
we're
at
now
and
like
where
we
plan
to
go
and
a
lot
of
the
challenges
that
have
come
with.
Maintaining
our
own
service
discovery
stack.
A
So
it's
a
little
about
me,
I'm
insight,
little
reliability,
engineer
will
kind
of
talk
about.
Why
I'm
up
here
talking
about
this
and
not
some
other
engineers
throughout
the
presentation,
but
it
has
to
mostly
do
is
just
as
we
grow.
We
know
things
get
reorg
and
things
get
shifted
around
and,
incidentally,
sree
has
contributed
the
most
over
the
years
on
our
service
discovery
stack
also
at
Airbnb.
Whenever
you
joined
the
company,
we
always
do
fun
facts.
So
whenever
you
meet
a
new
person
at
Airbnb,
they're,
always
like.
A
What's
your
fun
fact,
so
my
fun
fact
is
I've
stayed
at
like
12
different
air
bee-bees
I
was
twelve
different
countries
with
Airbnb.
Also,
my
team,
specifically
the
SRE
team,
does
boring
facts.
So
the
boring
facts
are
always
just
something
really
lame,
like
probably
I
drink
coffee.
This
morning,
which
is
actually
true,
I
wrote
made
the
slide
a
while
ago,
so
and
also
since
I
already
once
again,
the
last
incident
that
I
caused
was
in
July
of
this
year,
which
is
actually
also,
incidentally,
because
of
service
discovery.
A
So
what
kind
of
we
might
cover
yeah
we'll
cover
in
the
migration
section,
spoiler
alert,
so
this
is
kind
of
the
agenda
today.
These
are
like
the
four
main
things
I'll
be
talking
about.
We've
talked
about
service
discovery,
we
were
getting
about
scaling
and
why
scaling
is
hard,
also
my
migrating,
and
why
migrating
is
hard,
as
well
as,
like
all
the
engineering
work
that
goes
with
it
and
as
we
all
know,
that
like
engineering
is
hard,
otherwise
we
wouldn't
get
paid
right.
A
So,
let's
kind
of
go
back
to
the
beginning,
the
beginning
of
service
discovery
at
Airbnb.
So
initially
what
we
came
up
with
us
as
we
came
with
a
smart
stack.
So
this
is
an
open
source
project
that
we
created
for
a
distributed
service,
discovery
framework,
and
here
we're
gonna
dive
into
some
of
the
parts.
Just
so
I
Reich
is
like
a
general
idea
of
what
what's
happening
here,
but
it's
made
up
of
a
few
different
parts
here.
Nerve,
so
nerve
is
actually
execute
aunt
on
the
host
of
the
service
that
you
want
to
connect
to.
A
It's
really
just
an
advertiser.
It's
really
broadcasting
where
you're
at
and
it
broadcasts
this
zookeeper
nerve
also
does
a
simple
health
check
to
the
local
service
before
it
broadcasts
this
information?
So
if
you're
running
on
silicon
or
bare
metal
it
will
actually
just
like
you
can
configure
your
health
check
to
see
like
is
the
local
application
running
and
if
it
is
running
it
will
actually
register
this
service
zookeeper
after
we
broadcast
the
information.
Now
a
client
wants
to
talk
to
a
service.
A
So
the
way
we
a
client
can
connect
to
services
via
synapse,
so
soon
UPS
is
actually
a
Zagat.
So
synapse
is
a
service
that
runs
on
the
on
the
client
host
and
it
pulls
the
information
about
all
the
backends
that
it
wants
to
connect
to
from
zookeeper
and
from
there
it
actually
rewrites
each
a
proxy
config.
So
we
actually
use
proxy
a
proxy
for
Howey
load
balance
and
distribute
our
traffic
to
all
the
backend
hosts,
and
once
again
this
is
it.
This
is
run
on
the
client
machine
or
the
client
host
and
zookeeper.
A
A
You
could
also
go
there
as
well
I'm
on
and
on
the
nurse
website.
You
can
actually
configure
how
you're
doing
the
local
health
checks,
because
all
its
really
doing
is
something
kind
of
really
naive.
It's
like
well,
let
me
ping
an
endpoint,
my
local
endpoint,
what
port
do
I
ping
in
my
opinion,
or
am
I
going
to
slash
health,
but
all
these
configurations
are
stored
in
chef
itself,
so
this
is
all
like
framed
in
the
in
the
ec2.
A
Slash
like
bare
metal
world,
so
kind
of
moving
forward,
we
wanted
to
our
infrastructure,
was
migrating
from
easy
to
to
kubernetes.
So
a
lot
of
them.
The
work
kind
of
go
back
to
that
point
of
ownership
is
a
lot
of
the
kubernetes
work
was
actually
not
done
by
SRE
team.
It
was
originally
owned
by
the
surface
orchestration
team.
This
this
also
predates
our
current
traffic
team.
So
a
lot
of
this
work,
the
first
drafts
of
it
were
done
by
people
that
are,
are
not
unnecessary
or
not
even
owned.
A
By
on
the
traffic
team,
which
now
owned
this
so
moving
the
kubernetes
world,
how
do
we
get
all
our
service
discovery
to
work?
We
containerize
everything
right,
we're
just
going
to
throw
all
the
applications
and
all
the
processes
into
different
containers,
so
this
was
kind
of
what
we
expected
to
happen
all
right.
A
We
expected
that,
like
now
that
we're
in
a
containerized
world
we're
going
to
wrap
the
application
in
container
we're
gonna
wrap
nerve
in
the
container,
we're
gonna
wrap
synapse
in
the
container,
we're
gonna
wrap,
H
a
proxy
in
a
container
and
that's
gonna,
and
it's
gonna
like
just
work
right.
Unfortunately,
that's
not
one
enough
ended
up
happening.
This
is
more
of
a
closer
reality
of
to
how
our
current
framework
works
and
we're
gonna
dive
into
each
of
these
individual
parts
here,
so
first
mini
announcer.
This,
actually
is
the
replacement
for
nerve.
A
So
nerve
was
how
we
were
initially
announcing
how
our
services
or
broadcasting
the
availability
and
where
our
services
are
located.
So
now
we
actually
replace
it
with
mini
announcer
mini
announcer.
Does
nerve.
Remember
was
very
naive.
It
basically
like
looked
at
the
local
hosts
and
they
knew
how
to
help
check
and
ping
the
local
hosts
in
the
local
application.
So
now
mini
announcer
as
we
move
to
kubernetes.
We
can't
just
check
that
container
anymore.
We
actually
want
to
make
sure
that
all
the
other
containers
are
up.
A
So
the
way
many
announcer
works
is
that
it
actually
pulls
the
k8
api
and
it
says
it
waits
for
all
the
other
containers
to
be
ready
before
it
actually
broadcasted.
On
top
of
that,
the
way
IP
addresses
and
ports
worker
is
also
different
in
kubernetes,
so
it
actually
gets
that
information
from
the
cube
api
as
well,
and
then
it
also
broadcast
us
back
to
zookeeper.
One
of
the
other
things
to
note.
Is
that
before
hand,
a
lot
of
the
port
configuration
in
nerve
was
like
very
static
and
chef.
A
A
So
actually,
when
we
move
to
the
mini
announcer,
we
also
had
to
update
a
lot
of
this
so
that
it
broadcasts
the
port
and
that
way
synapse
could
also
read
from
the
port,
because
this
assumption
is
like
incorrect
now,
because,
whenever
you're
connecting
to
kubernetes
a
kubernetes
container,
the
port
is
also
can
change
as
well
at
based
on
like
how
your
application
gets
scheduled
on
the
container.
On
top
of
that,
so
then
it
also
publishes
the
IP
and
all
the
port,
but
mini
answer.
A
We
also
use
for
keeping
the
service
discovery
and
like
turning
off
the
lights
correctly
in
our
application,
as
we
get
me
scheduled.
So
if
you
turn
off,
if
you
reschedule
a
part
or
if
a
pod
is
scheduled
for
removal,
sometimes
it
would
actually
just
terminate
the
pod
and
we
get
a
bunch
of
500s,
because
either
the
application
was
still
trying
to
communicate
or
the
the
request
had
not
completed
at
all.
So
we
actually
do
a
pre,
a
pre
stop
hook
in
our
mini
announcer
to
make
sure
that
all
of
our
connections
closed
correctly.
A
A
So
we're
going
to
dive
into
now.
We
have
the
on
way
container,
so
the
on
void
container
was
actually
so.
This
is
done
in
part
with
service
orchestration,
as
well
as
security.
So
one
of
the
things
that
moving
to
scooper
Nettie
is
a
security
wanting
to
get
their
foot
in
the
door
on
getting
em
TLS
to
work,
so
M
TLS
required
us
to
also
have
an
ingress
proxy.
So
we
started
using
envoy
for
our
ingress
proxy
here
and
it
was
out
like
out
of
the
box
ingress
proxy.
A
We
we
were
using
it
for
pulling
configs
and
like
how
are
configuring,
the
local
cluster,
the
local
application
and
a
white
listing
all
traffic
to
the
local
application.
The
way
that
we
set
up
envoy
is
like
slightly
different
than
a
lot
of
other
companies
set
up
envoy,
so
we
actually
use
what
we
call
SDSU,
which
is
actually
gonna,
be
this
next
side
card
that
we're
talking
about
so
SDS
GM
is
essentially
a
shim
container
that
wraps
the
XDS
XDS
interface.
That
envoy
uses
to
poll
for
the
all
the
cluster
information.
In
the
end.
A
A
On
top
of
that,
SDS
JM
is
actually
made
to
merge
multiple
configurations
from
different
sources,
so,
on
top
of
like
being
able
to
pull
information
from
zookeeper,
which
is
where
we're
keeping
all
of
our
data
about
all
of
our
backends
and
all
of
our
listeners
and
as
well
as
all
of
our
configs,
it
can
actually
read
from
another
local
or
other
file
stores
to
actually
merge
these
configurations
before
it
feeds
it
up
to
on
voice.
So
this
is
actually
how
we
get
our
in
TLS
to
work.
A
Is
that
we're
talking
about
the
next
container,
the
MPLS
container,
and
it
merges
that
information
with
the
zookeeper
information?
So
that
way
we
get
the
ingress
information
or
lease
the
ingress
services
that
are
actually
allowed
to
talk
to
the
services
so
like
I
was
talking
about
our
MPLS
sinker.
So
basically,
this
is
just
security
to
develop
this,
and
it's
on
by
all.
A
So
every
view
is
growing
over
the
years
the
number
of
hosts
number
of
guests
are
growing,
the
amount
of
traffic
is
growing,
and
as
we
do
this
like,
we
have
to
scale
up
a
lot
of
our
hosts.
Scalloping
scaling
up
our
back
ends
as
well
as
scaling
out
some
of
our
services
and
as
a
result
of
these
things,
we
ended
up
having
incidents
or
issues.
So
it's
kind
of
go
back
to
the
smart
stack
example
here,
and
we
know
that
the
thing
that's
really
important
to
that
we're
gonna
call
out
here.
A
Oh
sorry,
we're
gonna
run
through
a
quick
demo
here,
where
we're
going
to
kind
of
see
what
the
effect
of
scale
is
as
we
go
forward
and
like
it's
really
easy
to
think
about
these
things
in
retrospect
beforehand.
We
never
actually
thought
we'd
get
to
this
kind
of
scale.
So
more
specifically,
we're
actually
gonna
be
talking
about
the
synapse
and
H
a
proxy
container.
A
A
This
config
is
also
getting
a
larger,
a
thousand
back
ends,
and
even
then,
if
you
have
a
thousand
back
ends,
you
could
also
have
multiple
services
you're
talking
to
so
you
get
actually
so
you're
actually
now
getting
like
an
M
by
M
effect,
because
it's
like
you're
talking
to
in
different
services
and
a
lot
of
those
in
different
services
have
like
in
different
backends,
so
we're
seeing
these
configs
are
getting
really
large,
and
so
what
is
the
effect
of
this?
So
a
lot
of
this?
A
So
what
ends
up
happening
is
that
we
end
up
creating
a
very
large
config
and,
and
we
end
up
hitting
memory
exhaustion.
So
the
memory
exhaustion
actually
comes
from
the
fact
that
H
a
proxy
has
to
take
time
to
now
load.
This
convey
we're
actually
doing
two
things:
we're
writing
the
config
and
we're
loading
the
afaik
and
we're
actually
reloading
the
config
into
memory.
Hu
proxies
loading
it
every
into
memory
every
time.
On
top
of
that,
since
it's
trying
to
persist
and
like
make
sure
it
closed
on
all
the
connections
safely.
A
A
Some
protection
around
like
standard
deployments,
but
at
the
same
time
now
you're
how
long
that
deploy
takes
you're,
essentially
refreshing
your
backends
periodically
and
you're
trying
to
reload
this
file,
you're
trying
to
reload
the
site
multiply,
make
file
lots
of
times
very
quickly
which
ends
up
actually
doing
memory
exhaustion,
because
we're
trying
to
write
the
file,
we're
trying
to
read
the
file
and
then,
at
the
same
time,
we're
not
like
giving
up
all
of
our
threads
80
procs
is
not
giving
all
the
threads
back.
So
now
we
actually
hit
in
memory
exhaustion.
A
So
you
can
so
these
graphs
are
just
examples
of
things
that
we'd
see
during
deploys.
So
during
this
report
we
would
see
essentially
a
spike
in
memory
usage
on
the
in
host
from
and,
additionally,
we
would
just
with
it,
take
two
to
five
milliseconds
or
two
to
five
seconds
in
some
cases,
depending
on
the
size
of
the
backends
and
the
size
of
the
deploys
themselves,
and
then
on
top
of
that,
a
lot
of
these
services
as
they're
restarting
they're.
A
Actually,
we
were
seeing
a
lot
of
network
saturation
here
too
we're
seeing
the
saturation
just
because,
as
things
come
up,
we're
also
health
checking
it.
So
we're
not
we're
not
optimistic.
So
whenever
a
service
gets
registered
in
zookeeper
synapse
before
it's
a
cheap,
proxy
and
synapse
before
they
send
traffic
to
that
service,
they
do
a
quick
health
check.
A
So
what
so
did
we
end
up
doing
so
to
actually
fix
some
of
this
problem?
So
because
we
maintain
this
and
because
we
do
this,
we
own
smart
sack
and
we
own
the
internal
application
of
smart
start.
We
fixed
it
by
to
prompt
like
two
different
ways.
One
thing
that
we
did
was
they
actually
started
moving
things
away
from
smart
stack
or
yeah.
We
started
moving
things
away
from
agent
proxy.
A
A
It
will
reload
the
information
and
just
give
it
to
envoy
periodically,
and
it
has
some
of
these
built-in
protections
to
prevent
us
from
some
of
the
stale
data
and
some
of
these
other
things
via
either
outlier
detection
or
some
of
the
other
real
resiliency
features
that
are
built
into
envoy.
We're
actually
saving
us
from
some
of
these
things.
On
top
of
that
envoy
actually
had
a
smaller
memory
footprint
because
it
wasn't
trying
to
reload
everything
so
so
quickly,
because
we
were
actually
forced
reloading
a
chip
proxy.
A
So
because
envoy,
does
this
periodic
polling
to
get
new
information?
It
wasn't
considered
consuming
nearly
as
much
memory,
and
on
top
of
that
we
ended
up
charting
some
of
the
roles
as
well.
We
ended
up
starting
some
of
the
roles
in
shifts,
so
we
had
a
lot
of
these
large
backends.
A
lot
of
the
services
talked
to
a
lot
of
these
backends,
so
we
ended
up
doing.
Is
we
try
to
figure
out
like
traffic
actually
needed
to
talk
to
what
traffic?
A
So
a
lot
of
these
services
were
actually
very
monolithic,
so
they
did
a
lot
of
different
things
and
they
scaled
very
large.
So
instead
we
started
at
cutting
it
out
and
saying
like:
are
you?
Are
you
like
a
pathway?
So
are
you
like
a
proxy
like
a
DB
proxy
service?
Are
you
just
like
proxying
through
another
service,
like
what
are
you
actually
doing
and
I
cutting
these
a
host
down,
so
that
the
back
number
of
backends
that
these
hosts
have
it's
much
smaller.
A
Okay,
so
like
because
of
this,
we
kind
of
already
hinted
at
that,
like
we
have
SDS
amen.
We
have
envoy
so
kind
of
moving
forward.
That
kind
of
gives
us
to
our
next
phase,
where,
as
we
migrate
into
kubernetes,
we're
also
going
to
start
migrating
things
to
envoy,
and
this
has
to
do
with
the
fact
that
that
memory
issue
it
doesn't
go
away.
If
you
have
synapse
and
H
a
proxy
in
containers
actually
makes
the
problem
worse.
A
Because
now
that
you
it's
really
hard
because
you're
requesting
the
amount
of
resources
in
your
container
and
as
you
request
those
resources
inside
the
container,
what
ends
up
happening
is
that
I?
You
it's
really
hard
to
configure
those
correctly,
because
you
don't
want
to
over
resource
these,
and
you
don't
want
to
under
resource
if
you
under
resource
something
you're
gonna
get
rescheduled
if
you
over
resource
them
now.
A
A
So
this
is
a
quick
reminder
of
like
essentially
breaking
down
the
containers
that
we
have
where
now
we
have
the
Envoy
container,
and
the
big
thing
here
is
that
we
are
going
to
be
adding
our
ingress
having
sorry
egress
traffic
as
well
so
beforehand.
It
was
mostly
set
up
for
ingress
traffic,
but
now
we're
actually
going
to
start
using
a
for
egress
traffic
as
well,
because
we
have
most
of
the
framework
and
most
of
the
bare-bones
set
up
to
actually
use
this
for
the
egress.
A
So
one
thing
that
we
want
to
talk
about
real
quick
touch
on
here
is
that
you
probably
see
this
in
some
of
the
other
Airbnb
presentations,
or
you
heard
a
mention,
but
we
have
a
tool
called
coop,
Jen
I.
Think
a
lot
of
other
companies
have
a
similar
tooling,
where
we
essentially
use
this
on
top
of,
like
coupe
carnal,
where
we
can
generate
our
configs.
It
takes
some
of
our
api
internal
configs
and
it
puts
it
in
and
it
transforms
it
into
like
kubernetes
configs,
which
then
you
can
deploy
straight
to
kubernetes.
A
So
the
idea
is
that,
like
the
developer
itself
doesn't
actually
have
to
know
all
the
kubernetes
configs,
because
they
just
know
some
of
the
basic
configs
and
then
it
will
generate
all
the
containers
and
all
the
configs
and
all
the
stall
you
met.
It
needs
to
push
the
kubernetes,
and
so
this
is
kind
of
important,
because
a
lot
of
this
going
back
to
all
the
containers.
A
lot
of
this
was
done
by
service
orchestration
service.
Orchestration
owns
couvert
kujan,
so
this
work
was
actually
done
by
the
service
orchestration
team.
A
So
now
we
also
have
to
control
plain
containers,
and
so
how
do
we
decide
who's
actually
going
to
whit's
proxy
to
use
on
in
grocery
on
your
egress
traffic?
So
actually
introduced
a
new
container?
Here
we
we
Inc,
we
introduced
the
SD
configurator,
so
this
is
actually
in
a
NIC
container,
so
it
doesn't
actually
run
as
a
sidecar.
It
runs
on
an
it
time
and
it
makes
in
it
time
decisions.
A
It
has
some
deterministic
randomness
in
it
in
that
we
actually
use
the
pods
name
to
to
create
a
hash,
and
that
will
actually
we
can
introduce
probabilities
here.
So
we
actually
introduce
probabilities
of
like
if
you
have
a
50%
chance
of
be
using
Envoy
versus
Envoy
versus
H,
a
proxy.
If
you
have
the
same
pod
name
and
you
get
rescheduled
you're
always
going
to
have
the
same.
That
results
like
if
you
get
rescheduled
under
the
same
name.
It
will
also
have
the
same.
A
It
would
have
the
same
result,
and
this
is
this
was
mostly
in
effect.
So
that
way
we
could
do
for
reproducibility.
So
that
way
we
could
determine
like
how
many
how
many
services
were
actually
using
a
chat
proxy.
How
many
were
using
envoy
and
we
can
good-
we
can
get
a
reproducible
value
under
those
two,
and
also
the
biggest
thing
here.
Is
that
the
reason
we
also
did
it?
A
That
way
is
that
we
were
afraid
that
if
pods
were
to
get
if
for
some
reason
on
well,
he
wasn't
working
or
hte
proxy
wasn't
working,
it
would
get
rescheduled.
You
know
in
some
of
the
pods
can
actually
be
rescheduled
if
we're
not
careful
and
Twitter's
like
all
on
white,
because
H,
a
proxy
is
felling
or
all
of
H
a
proxy,
because
our
envoy
is
felling
because
those
sidecar
chickens
are
felling,
it
will
like
crash
back
off
loop
and
then
it'll
reschedule
and
then
it'll
actually
pick
up
a
different
config.
A
So
we're
trying
to
do
this
in
a
way
that
we
can
actually
reproduce
some
of
this
information.
So,
on
top
of
that
so
now
this
is
all
these
are
kind
of
all
the
containers
that
we
have,
which
is
a
lot
going
forward,
but
it
like
to
actually
take
a
step
back.
Remember
that
in
the
beginning,
we're
talking
about
those
configs
that
how
you
config
synapse
and
how
you
config
nerve,
so
we've
been
putting
those
in
chef
and
so
what
happens?
These
can
fix
now
right.
A
So
the
problem
is
that
we
want
to
keep
these
configs
or
we
customers.
Like
me,
these
configs
or
I'm
gonna,
say
like
quote-unquote,
need
these
configs
and
don't
reason.
I'm
gonna
say
that
is
because
over
time
a
lot
of
the
config
in
chef
were
kind
of
copy,
pasta
and
iooks
lightly
modified
and
people
that
modified
them
may
have
left,
and
they
may
have
modified
them
for
reasons
whether
we
know
that
or
not
or
the
services
service
owners
may
have
modified
it
for
other
reasons,
but
we
need
to
keep
those
configs
around.
A
So
what
do
we
do
with
them?
So
we
actually
modified
synapse,
we
modified
nerve,
we
modified
STS
ship
and
some
of,
and
actually
we
have
our
own
innate
container
as
well.
In
kubernetes
for
services
to
publish
these
configs
so
actually
on
a
chef
converge,
it
will
actually
upload
this
config
to
zookeeper
in
kubernetes
on
a
deploy.
It
will
upload
the
config
zookeeper
as
well,
and
so
then,
on
top
of
that,
we
modified
synapse
and
STS
shim
to
actually
be
able
to
pull
these
configs
from
zookeeper.
So
that
way
we
can
actually
propagate
it.
A
A
So
going
back
to
our
example,
here
we're
going
to
talk
about
same
thing
when
service
lots
of
backends.
This
could
be
now
that
we
have
like
pods.
It
actually
grows
as
well,
but
now
we're
talking
about
how
we're
talking
to
zookeeper
so
we're.
Actually
so
we
talked
to
zookeeper.
We
could
actually
be
pushing
a
lot
of
information
here,
and
so,
if
we
have,
if
you're
pulling
configs
from
one
host
or
from
one
back
end,
each
back
in
is
pulling
it
ten
times
from
zookeeper.
What,
if
you
have
100
backends?
A
These
were
all
requested,
zookeeper,
100,
early
2000
and
then
like
ten
thousand
right.
What
ends
up
happening
is
that
we
have
two
that
we
end
up
like
consuming
a
lot
of
network
bandwidth
here
and
like
whenever
we
see
the
poised.
We
also
see
the
packet
rate
on
ZK.
Our
zookeeper
cluster
actually
go
up,
so
this
is
actually
one
of
the
metrics
that
way.
So
this
is
actually
the
metric
from
that
previous
graph
that
we're
actually
monitoring.
So
we
actually
have
an
alert
on
this.
A
So
what
was
happening
was
that
the
traffic
team
was
getting
this
alert,
but
they
weren't
exactly
sure
what
it
was,
and
this
has
to
go
to
the
fact
that,
like
sre
said
a
zookeeper
up,
sre
like
set
up
this
alert
and
then
we
kind
of
threw
it
over
the
shelf
like
traffic
team
and
they
were
like.
Oh
this
looks
firing.
It
can't
be
that
bad
right,
so
this
is
kind
of
this-
was
the
alert
message
that
we
set
up
with
it,
as
you
can
see
here
that
we're
kind
of
talking
about.
A
What's
going
on
here,
we're
seeing
an
increase
in
packet
rate
zookeeper,
and
we
also
know
that
it
kind
of
tells
us
like
how
many
packets
that
we
think
is
like
okay.
They
saw
it
like
our
own,
like
fun,
research
via
failure.
We
don't
know
that,
like
in
ec2
that
we're
going
to
see
we're
gonna
see
these
these
limits,
and
but
it
once
when
we
do
vbc,
we
actually
see
higher
limits.
The
problem
was:
is
that
we're
actually
starting
to
hit
some
of
these
other
invisible
limits?
A
On
top
of
that
you're
gonna
see
this.
This
gives
you
like
delayed
a
config
updates,
come
basically
all
the
same
problems
that
we
saw
before
we're
like
back
in
hosts,
aren't
gonna,
be
there
you're
gonna,
see
five
hundred
erratically,
you're,
not
really
sure,
and
then
in
the
problem
is
that
service
owners
are
seeing
this
and
it
makes
it
really
hard
to
cool
it
was
for
a
while.
A
It
was
really
hard
to
correlate
it
because
the
traffic
team
was
seeing
this
alert
and
they
weren't
really
sure
what
was
happening,
but
then,
like
certain
services,
would
do
deploys
or
they
or
downstream
would
actually
deploy
it.
It
was
more
of
the
downstream
service
deploys
affecting
the
clients
and
they'd
be
like
I.
Don't
know
why,
but
I
just
got
like
an
increase
in
504
like
no
reason
or
an
increase
in
latency
for
no
reason
and
they'd
be
very
confused.
A
So
eventually
we
had
to
connect
the
dots
and
we
got
back
to
it,
and
then
we
figured
out
that,
like
like
SRS
kind
of
stepped
in,
and
we
were
talking
about
what
is
this?
What
is
this
actually
made
and
was
that
you
just
necessary?
It
was
like
traffic
sre
service
orchestration
because,
like
they're
also
dealing
with
the
kubernetes
side
of
this,
so
digging
to
the
learn
more.
It's
like
easy.
A
Here
it
actually
took
a
couple
months
to
upgrade
mainly
because
we
set
it
up
forever
ago,
and
it's
one
of
those
things
that,
like
nobody
touches
and
because
nobody
touches
it,
nobody
really
knew
how
to
do
a
fool
like
they
knew
how
to
do
like
single
host
replacement,
but
nobody
actually
knew
how
to
do
it
like
an
observer
and
a
leader
or
node
replacement.
So
this
actually
took
a
lot
of
like
two
to
three
months
of
engineering
time
to
do.
A
We
actually
made
a
lot
of
the
different
solutions
as
well
to
essentially
get
us
out
of
this
mess,
so
one
of
those
things
is
actually
to
start
migrating
some
of
these
services
away
from
was--let
list.
So
basically,
we're
also
doing
this
again,
we're
like
figuring
out
what
things
are
still
really
large
and
causing
us
problems.
They
have
a
lot
of
backends
that
are
causing
these
propagation
delays
and
like
storming
our
suit
keeper,
and
we
actually
tried
breaking
them
up
I'm.
A
On
top
of
that,
we
actually
inside
of
nerve
and
synapse,
we
actually
added
ZK
whenever
we
talked
to
CK
we're
actually
doing
some
data
encoding
as
well.
So
those
configs
that
we're
talking
about
those
the
backends
that
were
talking
about
we're
actually
doing
encoding
now.
So
no
we're
not
sitting
like
raw
like
strings
across
anymore.
Now,
we're
actually
sending
out
like
serialize
and
deserialize
data,
to
help
optimize
the
packet
number.
A
On
top
of
that,
we
actually
found
that
there
was
a
slight
bug:
I,
don't
know
if
it's
a
bug
or
how
we
want
to
classify
it,
but
there's
an
D
optimization
in
inside
synapse
as
well,
where
essentially
we
would
be
reading
we'd
be
traversing
the
tree,
which
means
that
we
would
say
like
give
me
this
node
give
me
this.
Then
children
of
this
nodes.
A
Because
now
we're
not
making
accelerate
round
trips
just
to
do
some
of
these
things,
we're
actually
getting
it
all
in
one
roundtrip
and
then
on
top
of
that,
we
we
added
a
mix
of
things
like
jitter
and
retry
mechanics
as
well
as
self
throttling.
So
that
way,
we're
actually
not
just
going
to
start
storming
ourselves
so
storming
our
CK
as
well.
A
A
So
we
have
envoy,
we
have
all
these
containers,
but
this
isn't
like
where
we
want
to
be,
obviously,
because
we
have
seven
containers
up
here
and
that's
like
way
too
many
containers,
as
we
know
that
like
resources
are
precious
and
if
you
have
seven
containers
or
six
sidecars
and
one
1/8
container
on
every
deploy
or
every
application
you're,
just
like
kind
of
throwing
money
out
the
window
right
like
because
you're
Europe.
If
there's
no
way
you
can
optimize
these
for
like
every
service.
A
A
So
one
of
the
things
that
we
wanted
to
do
during
these
migrations
is
that
we
wanted
to
create
transparency
to
and
from
service
discovery
for
service
owners.
So
what
I
mean
is
that,
like
service
owners
can
either
not
be
engaged
or
if
they
want
to
be
engaged,
we
give
them
enough
data
so
that
they
can't
be
engaged
so
one
things
that
we
create
is
we
actually
create
a
be
dashboards,
so
this
is.
This
was
actually
to
solve
the
problem.
A
Where
a
lot
of
times
we
initially
when
he
started
like
tests
migrating
some
services,
a
lot
of
service
owners
were
very
hesitant
and
they
weren't
sure
they
were
afraid.
Like
oh,
is
envoy
gonna,
like
increase
my
p95.
Is
it
gonna,
like
increase
my
error
rate,
like
what
are
all
the
things
that
like
could
go
wrong
a
service
discovery,
and
this
is
mostly
on
like
infrastructure
spot,
mainly
because
we
tried-
and
we
had
all
these
scaling
problems.
So
there
wasn't
a
lot
of
trust
there.
A
So
to
help
rebuild
a
lot
of
the
stress,
we
built
a
lot
of
these
dashboards
so
that
way
that
service
owners
could
actually
see
what
service
discovery
was
working
on.
If
they
did
come
in
because
a
lot
of
times
they
did
come
in
on
and
slack
can
be
like.
Did
you
change
something
of
service
discovery,
because
my
error
rate
is
is
wrong
and
and
like?
A
Okay.
On
top
of
this,
we
wanted
to.
We
still
want
to
take
iterate
because
we're
not
in
our
final
state
here
we
actually
wanted
to
build
in
new
features
or
enable
new
envoy
features,
because
we
own
SDS
Jim,
which
actually
feeds
in
this
is
the
XDS
API
is
for
envoy.
If
we
want
to
support
new
features
or
we
want
to
configure
new
features
or
non-void
like
outlier
detection,
we
actually
have
to
build
that
ourselves
in.
So
we
want
to
so
like
to
enable
some
of
these
new
things.
A
We
wanted
to
make
sure
that
we
can
maintain
iterating
while
doing
this
migration,
and
so
part
of
that
was
that
we
actually
had
to
take.
We
had
to
take
a
step
back
and
build
a
lot
of
CI
and
C
d-four
there
or
in
just
different
ways
that
we
could
verify
a
lot
of
this
communication
when
we
migrated.
So
that
meant
is
that
we
built
in
using
our
SD
configurator.
We
actually
built
in
flags
kill
switches
as
well
as
like
hot
switches.
A
So
basically,
if
you
turn
it
on
or
you
can
turn
it
off,
you
can
see
like
what
the
differences
were
and
those
were
the
playtime
decisions.
So
basically
you
deploy
it
with
the
flag
on
and
it
would
turn
it
on
and
you
could
roll
it
back
and
it
would
turn
it
off
as
well.
On
top
of
that,
we
actually
built
a
verification
fleet,
so
we
actually
have
a
fleet
that
it
employs
to
where
it
will
try
all
of
the
service
discovery,
side,
cars
and
try
to
get
them
to
communicate
to
each
other.
A
On
top
of
that,
we
built
services
that
were
essentially,
we
use
it
on
some
of
our
canary
services
as
well,
so
we'd
actually
deploy
some
of
these
changes
and
the
canary
services,
where
there's
like
lower
traffic
and
it
and
actually
wasn't
even
like
production.
There's
like
some
of
our
staging
environments,
just
to
see
that
the
service
discovery
was
actually
working
and
a
verify
that
some
of
these
things
were
working
and
a
lot
of
times.
It
actually
caught
a
lot
of
the
corner
cases
where
people
I
could
figure
things
differently
or
just
theirs.
A
Their
service
was
a
standard,
it
wasn't
configured
standardly
and
then,
on
top
of
that,
we
actually
went
through
invented
a
lot
of
services,
and
we
we
supported,
like
the
general
case.
What
that
meant
is
that
we
supported
a
lot
of
our
basic
traffic,
but
then
the
if
anything,
was
talking.
That
was
very
specific.
A
So
we're
actually
just
going
to
make
sure
that,
like
none
of
the
none
of
the
DB
proxies
or
anything,
can
actually
enable
it,
and
we
do
that
via
that
SD
configurator
as
well.
So
the
SD
configurator
does
that
runtime
decision,
and
we
just
say
you
like
hard
notes.
A
lot
of
these
things
are
on
the
disallow
list,
and
on
top
of
that,
we
actually
just
migrate.
A
A
But
the
problem
is
that
we
actually
had
a
lot
of
ad
hoc
and
like
went
off
configs
just
kind
of
floating
around
going
back
to
the
point
where
people
would
copy
pasta,
some
configs,
they
change
like
their
keepalive
or
like
how
many
threads
and,
like
all
these
different
things
and
then
they'd,
leave
and
then
we'd
be
like
well.
Your
configures
like
this
is
the
different
configured
like
our
general
config.
A
Do
you
know
if,
like
do
you
know
what
the
effects
of
this
are
and
like
nobody,
anyone
actually
know
they're
like
well,
you
can
just
like
configure
it
and
pray
like
it's
something
breaks
we'll
just
like
let
you
know
which
wasn't
like
a
great
decision.
So,
on
the
same
time,
we
we
actually
tried
to
we
handheld
a
lot
of
these
services
and
we
basically
made
calls
on
like
how
different
the
config
was
from
our
standard
config.
A
So
if
there
were
like
more
than
a
few
settings
that
were
completely
off,
we
actually
put
them
back
on
our
disallow
list
until
they
could
resolve
those
and
they
could
kind
of
move
on
to
our
paved
road.
Essentially,
like
our
standardized
configuration
at
their
own
pace,
and
then
on
top
of
that,
we
actually
had
a
problem
now
where
we
actually
had
multiple
sources
of
truth
for
our
default
config,
because
we
have
the
deep
we
have
config
in
chef.
A
We
have
configs
in
coop
gen,
which
is
essentially
our
kubernetes
wrapper,
and
the
problem
was
that
whenever,
as
we're
in
this
my
end
middle
to
migration
process,
what
would
happen
as
a
service
would
deploy?
And
then
maybe
someone
would
actually
try
to
tune
their
service
discovery
convey,
but
they
go
in
and
be
like.
It's
not
applying
my
service
to
cut
the
discovery.
Config,
so
we'd
actually
have
to
look
at
a
lot
of
different
places
because
we
have
to
be
like.
Is
it
a
zookeeper,
it's
a
zookeeper
configure
right.
Is
it
in
chef?
A
Is
that
configure
right?
Is
it
in
like
kubernetes?
Is
it
in
the
SD
configurator?
Is
that
configure
8?
So
a
lot
of
so
what
we
ended
up
doing
is
that
whenever
we're
reading
or
writing,
they
can
fix
the
zookeeper.
We
actually
say
who
wrote
that
or
who
read
that
convey
or
who
wrote
that
config
so
they're?
Never
you
pull
it
down.
You
can
actually
see
like
where
that
configures
published
from
was
a
Polish
from
chef.
Was
it
published
from
kubernetes?
A
So
that
way
we
naturally
know
who
is
like
causing
the
issue
so
like
someone
to
change
the
chef
but
in
reality,
like
the
source
of
truth,
is
in
kubernetes
and
their
application.
So
we
did
some
of
that
modifications
in
syntax
as
well.
So
that
way
we
could
tell
who
is
actually
reading
and
writing
these
configs
and
then,
on
top
of
that
we
actually
did
service
cover
version
rollout.
So
we
have
SD
configurator.
We
have
Envoy,
we
we
use
envoy
one
tin
right
now,
I
think,
and
then,
on
top
of
that
we
have
SDS
chamber
ship.
A
So
we
actually
have
these
three
different
versions,
so
the
problem
was
is
like:
how
do
we
track
these
different
things?
So
this
is
actually
a
throwback
to
that
first
slide
with
the
dashboards.
So
this
is
actually
just
as
you
can
see
here.
It
lets
you
in
each
of
the
different
colors
is
actually
a
different
service
discovery
package.
So
that
way,
you
could
see
that
if
there
was
an
increase
in
error
rate,
at
the
same
time,
this
color
were
to
change
that
you
would
see
like.
A
Oh
it's
actually,
the
the
service
discovery
package
that
is
breaking
my
service
and
that
made
that
gave
more
visibility
and
more
version
tracking
for
us.
So
that
way
we
could
actually
determine
if
it
was
a
service
discovery
change
or
if
it
was
actually
like
a
service
change,
going
back
to
like
service
owners
not
being
clear
on
whether
it's
their
service
or
service
discovery.
A
We
also
know
that
scaling
is
hard
right,
like
that,
as
everybody
grows,
it's
really
hard
to
think
about
some
of
these
scales
when
you're
a
smaller
company
over
and
you're
a
much
smaller,
less
services,
less
back-end.
All
these
things
like
really.
Why
would
I
ever
think
that
we
get
to
like
10,000
hosts
10,000
back-end,
it's
10,000
different,
like
services
that
we
could
be
and
even
then
like
what
what
happens
tomorrow,
when
it's
like
a
hundred
thousand?
How
does
envoy
handle
this?
We
haven't
thought
about
these
things
because
sometimes
you're
moving
so
fast.