►
Description
Changing Oil for a Fast Running (Side) Car Quickly and Safely - Fuyuan Bie
While we all want our features in production ASAP, safety - on the other hand - is the last thing we should sacrifice. At Pinterest, mesh configuration story looks like this: - thousands clusters - under 25 minutes to fully deploy through all stages and availability zones serially - 0 incidents during xDS v3 migration - every change is validated individually - full visibility into change and client history Thanks to comprehensive pre-deploy validation, holistic health checks and a specially designed feedback channel based on xDS, configuration roll-out is safe and yet very fast. The machinery carries every config change to VMs, dockerized hosts, and k8s automatically. Should an issue happen, it can spot the problem within a minute. In this talk, we will share the architecture, design considerations, good practices, and lessons learned along our path towards configuration nirvana.
A
A
So
today
we
are
going
to
start
with
a
look
at
avoid
pinterest
briefly
and
follow
that
or
look
at
the
configuration
layout
story.
He
add
pinterest
the
challenges
we
faced
and
the
solutions
we
used
and,
at
the
end,
we'll
share
some
best
practices
we
applied
and
the
lessons
we
have
learned
during
this
journey.
A
So
at
pinterest
we
have
been
using
hawaii
started
around
four
years
ago
and
we
use
it
very
widely
and
deeply
when
you,
when
user
traffic
comes
into
pinterest,
it
firstly
hit
he'd,
always
an
agile
boy
from
there
trafficker
that
will
get
routed
to
our
mesh
mesh
envoy
and
starting
from
there.
We,
the
traffic,
that
allowed
routed
to
different
services,
and
it
did
not
stop
there.
A
A
And
this
is
the
architecture
of
our
mesh
at
the
very
top.
Is
the
tower
tower?
Is
the
central
lyso
control
plane
officer
service
mesh
when
a
user
land
a
config
into
the
gate
report
jenkins,
will
upload
the
config
in
into
the
centralized
control
plan
and
it
will
be
processed
in
zookeeper
and
behind
that
the
centralized
control
plan?
We
install
a
agent
on
every
host,
it's
called
beacon,
beacon
and
their
tower.
A
They
are
between
buick
and
harvard
is
a
simple
interrupt,
but
a
very
robust
grpc
streaming
protocol
and
it's
a
generic
config
distribution
protocol
between
them
and
we
will
leave
the
worsening
and
other
complexities
to
the
last
mile.
So
the
beacon
will
be
the
agent
dealing
with
xcs
services
and
they
will
distribute
the
coffee
through
xcs
to
outboy
and
to
build
seating
side
by
side
of
his
armoire.
We
have
sds
sidecar
and
opi
agent
to
the
austin
to
help
our
way
to
do
authentication
under
authorization.
A
A
A
A
That
is
how
we
get
thousands
clusters,
and
some
of
the
clusters
are
small.
Only
one
listeners,
one
or
two,
maybe
three
upstreams
some-
are
huge.
The
I
had
I
just
look
at
the
largest
cluster
we
have.
It
has
about
300
000
lines
of
config
and
the
config
dump
was
more
than
seven
megabytes
and
we
will
on
average
we
go
out
tens
of
changes
every
day
and
we
deploy
it.
We
need
to
deploy
them
first
and
then
you
come
to
the
question.
A
If
our
problem
happen
to
config,
how
do
we
root
cause
the
problem
problem
quickly
and
it
was
after
you
relocate
the
problem.
What
do
you
do?
Should
it
support
your
back
or
should
we
mind
it
forward,
fix
so
think
about.
This
scenario
is
like
you
are
driving
outside
a
car
and
your
customer
is
enjoying
the
ride
and
this
it
it's
perfect
and
but
deep
in
your
heart,
you
know
you
are
due
on
oil
change
and
you
don't
want
your
customer
to
be
a
happy.
How
do
you
do
that?
A
A
A
A
All
the
configurations
are
based
on
a
set
of
agenda
templates
that
we
provide
and
when
the
user
write
their
configuration,
they
need
to
compile
and
configurations
are
materialized
at
compile
time,
and
it's
also
versionized
when
the
configural
the
ground
shoes
is
in
kit.
It
keeps
the
rail
with
the
version
and
the
centralized
when
the
zinc
winching
is
copied
the
configuration
into
the
tower
into
the
centralized
control
plate.
It's
at
add
version
there,
so
the
configuration
stored
in
sync
to
a
centralized
control
plate
is
also
organized
and
is
tested.
A
We
run
the
unit
has
a
comprehensive
set
of
unit
tests
before
and
after
landing.
So
when
we
we,
the
tests
are
comprehensive.
If
we
test
the
listeners,
every
listener
has
must
have
have
valid
route.
Every
route
need
to
have
valid
valid
upstream.
If
a
really,
if
a
in
upstream,
require
tls
the
downstream
must
give
the
correct
configuration
they
must
match.
Also
like
a
poor
conflict.
These
kind
of
things
are
detected
during
build
time
and
also
it
is
deployable.
A
If
we
have
a
pipeline
deploy,
deploy
changes
after
apr
landed
until
it
does
that
for
every
pr.
Every
company
changes
so
infrastructure
governance
from
perform
called
mesh
perspective.
What
is
basically
two
questions?
First,
one:
where
should
I?
Where
should
the
mesh
configurations
go?
Should
it
be
in
a
centralized
ripple?
A
This
of
this,
of
course,
has
the
proton
counts.
Pros
of
a
synchronized
ripple
is
consistent.
You
get
consistency.
You
have
a
quick
turnaround
for
horizontal
changes
or
by
horizontal
changes
I
mean
you
will
make
a
change
which
applies
to
multiple
clusters
and,
of
course,
it
also
have
counts.
User
experience
may
not
be
ideal.
If
you
are
a
service
owner.
You
most
likely
would
like
to
have
your
configurations
in
your
own
repo.
A
Instead
of
making
a
change
to
another,
centralized
ripple
or
should
we
seek
is
through
the
sit
next
to
each
service
pros
users
have
better
control
of
their
their
conflict.
They
just
need
to
modify
that
configuration
in
their
repo.
Is
that
go
to
your
ripple.
However,
this
there's
another
big
conference
limitations.
A
You,
when
you
are
your
configuration
with
a
mesh
configuration
scattered
in
multiple
places,
the
build
cost
think
about
the
build
cost.
When
you
build
the
company,
how
do
you
build?
How
do
you
sure
make
sure
that
everyone,
every
company
needs
to
be
built
is
built,
and
another
question
is
actually
more
severe.
Question
is
if
a
critical
problem
is
spotted
like
a
security
bug
that
needs
to
change
the
config
immediately.
What
do
we
do?
You
have
hundreds
of
maybe
hundreds
per
report.
What
will
you
do?
How
do
you
fix
the
problem?
A
A
Another
question
I
will
with
regard
to
infrared
or
governance
is
who
owns
what
mesh
is
big,
no
matter
how
many
measures
you
have,
you
probably
have
one?
How
many
do
you
sell?
You
always
have
this
required
problem
whole
consequent
at
pinterest.
We
have
three
personas
infrastructure,
engine
security,
engine
and
service
owner,
each
of
them
own
different
things
of
the
mesh
as
you
as
are
listed
here.
A
A
A
A
Think
about
that.
It's
the
right
thing
to
do,
and
it's
it
it's
beyond
those
traditional
definition
of
service
owner
a
of
service
and
service
owner
b
owns
the
things
of
their
service.
Totally,
that's
not
right.
This
is
different.
This
is
how
a
house
mesh
configuration
should
be
like
look
like
and
with
that.
If
you
look
at
service
service
as
an
owner
of
a
service
fee
from
ibra
side,
he
barely
has
have
something
to
configure,
probably
just
some
declaration
of
upstreams
you.
You
need
to
talk
to
that
stage.
A
Drawing
out,
I
think,
is
reduced,
but
my
cell
conflict
got
landed.
It's
as
we
saw.
We
started
to
push
that
config
into
a
pilot
through
our
pipeline,
so
jenkins
dropped.
He
started
running
unit
tests,
so,
although
unit
has
run
before
pr
land,
however,
after
a
pr
land,
we
still
run
it
because
you
could
have
a
merge
problem,
cause
of
configuration,
failure
and
once
that
is
finished,
we
the
difference.
Job
will
literally.
A
Those
stages
still
activate
the
new
configuration
in
the
stage
and
listen
for
several
minutes
and
keep
holding
the
the
service
status.
And
if
that
there
is
no
problem,
it
will
consider
the
canary
stage
got
passed
and
it
was
in
another
notification
for
approval
once
they
are
approved.
It
will
go
to
go
next
or
go
to
product
stage.
A
A
So
we
use
a
rule
out
configuration.
How
do
we
know
there
is
a
problem,
so
we
have
we.
We
detect
problems
through
a
near
real-time
feedback
loop,
and
this
is
the
definition
of
negative
of
negative
feedback
loop
from
wikipedia.
I
don't
want
to
repeat,
but
it's
basically
like
you,
take
the
output
of
your
amplifier,
apply
some
algorithm
and
convert
the
part
of
the
output
and
fit
it
into
the
input
and
through
that
you
keep
tuning
your
your
system
to
get
your
ideal
output.
A
How
do
you
up?
How
do
we
apply
that
to
our
mesh
and
the
feedback?
Part
is
basically
used
to
detect
problem
and
what
problem
do
we
detect?
We
detect
our
avoid
reject
configs.
Basically,
this
is
through
xcs
protocol
and
also
we
have
a
health
check
failures
like
data
plan
for
the
it
detected
failures,
input,
data,
plane
and
control
plane
for
a
data
plane
side
service
may
be
unhealthy
from
the
control
plane
side.
A
So
the
first
thing,
the
first
failure
that
we
did
have
is
the
xds
are
resource
acceptance
or
rejection,
so
with
it
within
xds3
or
protocol.
It's
about
it's
about
a
real
required
discovery,
request
and
response
from
this
blog,
because
I'll
control,
the
config
pipeline,
send
a
new
configuration
of
a
new
version
of
the
of
the
configuration
to
the
tower
and
tower.
A
I
I'm
okay
with
this
conflict,
and
I
accept
it.
Then
it
will
use
the
discovery
request
to
act.
There
is
a
the
the
newers
and
then
this
will
be
reportedly
be
aggregated
in
beacon
and
the
beacon
will
aggregate
the
repository
report
back
to
the
tower.
So
so
the
pipeline
will
keep
current
tower
to
see.
Oh,
is
there
any
fulfilling
there
anything
so
another
case
is,
for
example,
the
new
version
contains
or
require
a
new
secret.
A
The
sacred,
however,
wasn't
granted
to
the
process
on
that
of
our
way,
so
unlock
will
gather
the
new
configuration
and
see.
Oh,
I
need
a
new
new
new
secret.
The
work
ask
sds
for
the
new
secret
and
access
reject
the
sdsu
account
the
request,
and
I
will
report
an
arrow
to
beacon
and
the
beacon
was
was
eventually
send.
This
aggregated
result
to
tower
tower
or
see.
A
So
another
part
is
the
holistic
health
check.
We
household
check
listener
and
we
health
check
the
coffee
of
the
configuration
about
bobby
child
child
called
controversy
for
like
the
host
agent,
sds
or
agent,
and
they
also
check
listeners
at
both
l7
and
l4
and
as
we
implement
that
as
that
script,
so
that
each
platform
can
invoke
it
easily,
as
it
does
not
not
depend
on
any
platform,
so
real-time
feedback.
The
last
thing
is
the
confidence.
A
Oh,
if
I
get
high
confidence,
you
will
see.
Oh
every
four
hour,
four.
If
this
configuration
ruled
out
touch
1000
clusters
and
all
of
them
has
more
than
90
percent
acceptance
rate,
then
we
got
enough
confidence
and
then
let's
exit
early
without
have
to
wait
just
go
to.
Let's
go
just
move
to
our
next
stage
or
if
it
did
not
get
enough
confidence.
Just
keep
waiting.
A
So
with
regards
to
the
visibility,
since
everyone
uses
slack,
we
send
a
select
message
on
every
ad
on
every
deployment
activity
to
our
channel
from
the
right
side.
You
can
see
that
so
the
mesh
configuration
was
ruled
out
and
there
is
notification
got
sent
out
and
after
it
finished
the
latest
stage,
it
sent
another
notification.
See.
A
Oh,
please
approve
this
change
to
develop
and
canary
and,
as
you
can
see,
someone
clicked
up
accept
and
it
moved
to
pro
pro,
and
it
has
been
very
wise,
though
it
takes
several
minutes
before
it's
actually
finished
about
actually
10
minutes,
maybe
the
finished
prod
and
it
sends
cinecom.
Congratulations.
The
app
everything
was
good
and
on
the
left
side,
you
can
see
that
the
configuration
actually
had
a
problem
and
that
prop
pipeline
detected.
A
So
in
summary,
we
have
a
thousand
clusters
and
which
serves
millions
of
rps
at
the
edge
side
and
from
the
change
landing
time
to
its
fully
activity
production.
If
everything
goes
well,
it's
less
than
30
minutes,
and
we
had.
We
just
finished
the
week
two
two
week,
three
api
and
configuration
migration,
and
we
had
zero
incident
thanks
to
the
configuration
pipeline
and
that
for
the
near
real-time
feedback
loop.
A
So
because
we
have
built
the
configuration,
the
control
plane
to
be
a
generic
resource
distribution
service.
Instead
of
just
for
xds
other
teams
and
from
other
operations.
Our
organizations
are
seeing
value
of
this
this
system
and
they
are
moving
they're
moving
their
existing
service,
their
their
configuration
into
this,
the
mesh
control
plan,
so
that
they,
because
they
also
want
their
super
easy
validation
for
easy
protection.
A
So
last
some
best
practice
and
some
lessons
we
learned.
First
infrared
infrastructure
governance
is
crucial.
This
is
very
important
for
our
success.
So,
let's
admit
today,
service
match
is
still
a
quick
evolving
world.
Therefore
your
changes
are
established
most
of
the
time
and
two
years
ago
we
have
been
talking
about
xds
xds
v2.
Last
year
we
talked
about
v3.
We
talked
about
ui
universal
data
plan
api
and
this
year.
A
That's
the
garment,
make
your
mesh
unmanageable
someday
and
that's.
I
cannot
amplify
amorphous
side
this
enough
and
you
will
know,
and
the
next
one
is
mtrs.
It's
your
good
friend.
If
it's
not
the
best,
we
have
99
problem
with
our
control
plan
which
delay
the
eds
update,
make
sure
this
made
certain
endpoint
data
stored
and
because
of
that,
if
we
did
not
have
x
there,
I
did
not
have
mtrs.
A
A
A
A
Every
envoy
of
pinterest
got
this
wrong
value
within
two
seconds
and
sadly
there
is
a
bug
in
hawaii
which
cause
which
caused
a
catastrophic
failure,
cause
our
trafficker
backs
a
backtracking
bug
which
caused
our
out
of
my
memory
within
one
minute,
so
how
my
crap
crashes,
because
of
our
oom
and
they
restarted
and
they'll
start
or
work
for
maybe
30
minutes,
30
seconds
then
crash
again.
So
I
crashed
every
minute
and
it
took
us
a
lot
of
effort
and
almost
30
minutes
to
recover
every
alloy.