►
From YouTube: Scaling Resilient Systems: A Journey into Slack's Database Service - Rafael Chacon & Guido Iaquinti
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Scaling Resilient Systems: A Journey into Slack's Database Service - Rafael Chacon & Guido Iaquinti, Slack
Monitoring and observability are important concepts, especially in complex and distributed systems. Redundancy and defensive programming are important as well, but sometimes they are not enough. Designing systems to minimize the blast radius when the unexpected happens is often the key. In this talk, Rafael and Guido will share an overview about how Slack designed, built, scaled and then iterated to improve its distributed database service based on top of Vitess, now a CNCF project. The Databases team at Slack scaled a Vitess cluster from 0 to spikes of 2.7 Million queries per second. This journey has taught us how to operate a database cluster with more than 2000 nodes and expecting to growth to more than 3500 in the next 12 months.
https://sched.co/UaWu
A
Okay,
so
we
are
going
to
start
welcome
to
the
next
cube
con
session
cube
contact.
My
name
is
P
odd,
I'm,
a
member
of
program
committee
of
this
conference
I'm
here
to
introduce
the
next
speakers
so
Guido
and
Raphael
are
working
for
slack
where
they
designed
and
built
and
scaled
a
distributed
database
service
based
on
with
us.
If
you
are
curious,
how
please
join
me
and
welcome
on
the
stage
Rafael
Chacon
and
Guido
yeah
Quinn
tea.
B
Hello,
everyone
and
thank
you
like
we
are
so
excited
to
be
here
and
sharing
our
experience
with
all
of
you
and
also
for
us
is
super
cool
to
see
so
many
people
interested
about
the
test
and
I
just
want
to
ask
you
a
key
question
who
knew
about
be
test
before
CN
CF?
Could
you
raise
your
hand,
cool,
okay
without
further
ado,
I,
just
I
gained
a
little
bit
a
little
bit
more
about
me.
B
My
name
is
rafael
chicken,
I'm
infrastructure,
lead
engineer
at
slack
and
I've
been
part
of
the
team
that
took
me
tests
from
a
prototype
to
now
B,
nor
main
data
store
in
production,
as
part
of
that,
I
also
made
a
big
amount
of
contributions
to
be
test
open
source
project
and
nama
maintainer
there.
My.
C
C
C
C
So
it's
a
my
sequel
master
master
set
up.
Our
key
is
via
a
synchronous,
replication.
The
charting
is
done
at
the
application
level
and
the
only
shard
in
his
support
is
workspace
or
team
ID.
This
architecture
surveyed
us
very
well
for
several
years
of
hyperbolic
growth,
but
we
knew
that
eventually
we
needed
to
change
it.
C
The
Vitesse's
char
topology
use
a
more
traditional
primary
replica
setup
that
is
fully
managed
by
with
us
here.
You
can
see
few
of
the
statistic
of
our
database
tier
that
can
give
you
a
better
overview
of
the
scale
that
we
are
operating
today.
As
today
we
serve
over
50
billion
queries
with
this
data
flowing
from
thousand
of
terabytes
stored
on
thousand
of
database
server
and
I.
C
Let's
now
discuss
what
running
database
in
a
cloud
would
mean
when
we
start
a
new
project,
we
usually
start
with
a
small
fleet
of
hosts.
Like
the
customization
is
easy.
We
can
treat
our
infrastructure
as
pets.
Each
one
can
have
like
its
own
name.
Each
one
can
even
serve
a
specific
workload
or
even
a
set
of
table
at
this
scale.
When
we
make
a
change,
we
can
fit
that
variation
in
our
head.
This
is
not
idea,
but
probably
still.
Okay,
again,
all
the
change
can
fit
in
our
head.
C
To
avoid
that,
we
should
strive
to
adopt
few
of
the
concept
of
a
beautiful
infrastructure
where,
for
example,
instance
are
untouched.
After
getting
provision,
configuration
change
only
happened
through
reprovision
in
those
hosts
and
no
in
place.
Patching
is
allowed
at
that
stage.
Technically,
you
can
run
with
a
read-only
file
system
and
those
live
operation
like
SSH,
become
useless.
Of
course
you
need
to
invest
in
monitoring
logging
and
troubleshooting
tools
to
make
this
transition
happen,
but
the
benefits
that
are
greater
than
the
app
they
effort
that
you
need
to
input.
C
Let's
now
talk
about
instance
failure.
So
how
do
we
treat
an
instance
failure
event
in
a
cloud
environment?
The
good
thing
is
that
we
have
a
couple
of
click
or
even
automatically.
We
can
reap
revision
a
host.
So
should
we
trust
an
instance
that
fail
us
even
once,
if
you
are
flying
a
plane
and
your
engine
fail,
all
bets
are
off
from
that
moment
on
the
plane
is
trying
to
kill
you
and,
from
that
moment,
on,
the
plane
is
a
problem
of
the
insurance
company.
C
C
Okay,
this
sounds
good,
but
also
challenging
for
few
set
up
like.
If
you
use
an
actual
storage,
you
can
simply
just
stop
and
start
your
instance
as
your
storage
changing
is
the
couple
from
the
compute
engine
if
you
use
it
in
some
store,
that's
unfortunately
different
like
to
do
that.
You
need
to
complete
a
restore
and
so
download,
possibly
terabyte,
of
data
from
your
last
backup,
the
compress
it
on
the
horse.
C
Look
at
this
point.
There
is
also
the
never-ending
question
about
running
a
bigger
number
of
small
shard
versus
running
a
small
number
of
big
shots.
So
my
personal
answer
is
that
it
depends.
It
depends
on
your
setup.
It
depends
on
your
workload
in
depends
on
your
data
set
and
so
on.
So
there
are
few
Pro
about
running
a
higher
number
of
small
shard,
like,
for
example,
having
like
lower
recovery
time
like
reduce
blast
radius
and
those
who
have
like
much
distributed
workload
and
even
at
the
host
level.
C
C
Traditional
data
storage
software
treated
data
as
durable
as
it
soon
as
it
was
flushed
to
disk.
A
new
approach
of
durability
is
achieved
by
copy
the
data
to
multiple
machine
and
even
geographical
location.
In
this
example,
we
have
a
replication
topology
with
a
master
node
and
three
replicas
in
semi
synchronous
configuration
all
deployed
in
different
data
centers.
We
can
fit
over
to
a
new
replica
when
the
master
died,
with
no
data
loss
or
even
a
replica
dies.
C
Let's
simply
reprobation
it
from
scratch,
and
we
can
do
that
because
we
rely
on
semi
synchronous,
replication
and
those
two
settings.
The
first
is
the
value
in
millisecond
that
control
how
long
the
master
will
wait
on
a
commit
for
acknowledgment
from
a
replica.
We
set
it
to
a
very
high
number,
technically
infinite,
as
we
never
want
to
hock
a
transaction
that
hasn't
been
racked
up
by
at
least
one
replica.
The
second
setting
is
the
number
of
replicas
that
need
to
acknowledge
the
transaction.
C
Rely
on
the
replication
stream
also
allow
us
to
loosen
some
of
the
disk
durability.
Setting,
for
example,
we
will
technically
turn
off
sink
bin
log
and
relax
the
in
ODB
flash
log
setting.
In
this
scenario,
we
use
all
buffered
I/o
for
all
the
operation,
which
will
give
us
much
higher
throughput
and
better
latencies
than
the
previous
approach.
An
important
disclaimer
here
like
you,
can
safely
do
this.
Only
if
you
always
throw
away
a
fail
note,
but
that's
exactly
what
we
suggest
so
we're
good.
C
The
Twitter's
you
can
see
on
the
right
is
from
last
year,
and
a
lot
of
things
have
changed
since
then.
We
test
can,
anyway,
fill
the
gaps
there,
but
in
our
setup,
were
not
there
yet
state
waste
components
are
deployed
in
auto
scaling
group
so
that
we
don't
need
to
care
if
an
instance
fail
or
if
it
gets
the
provision
by
Amazon.
Our
engineer
only
need
to
manage
our
stateful
fleet
leveraging
some
custom
tool
that
we
wrote.
B
So
with
that,
I
want
to
dig
a
little
bit
deeper
into
how
we
deploy
with
us
at
slack,
and
what
do
we
learn
along
the
way
when
it
comes
to
fault,
tolerance
and
isolation
at
our
scale?
So
first
I
just
want
to
do
a
very
like
quick
refresher
of
what
the
BTS
architecture
looks
like
I'm
not
going
to
enter
into
many
details,
but
if
you're
interested
about
this
topic
just
reach
out
after
the
talk
and
I'm
happy
to
talk
about
it.
B
So
first
on
the
left
you
have
a
mitigate
beta
guys
are
at
a
stainless
router
that
the
application
uses
to
talk
to
your
Vitas
cluster
from
the
application
perspective,
Abita
gate
looks
just
as
you
are
talking
to
any
Mexico
instance
mitigates,
implement
a
Mexico
protocol,
then,
on
the
top.
You
have
this
very
important
component
for
the
test.
That
is
the
topology
service.
B
The
topology
service
contains
all
the
metadata
about
your
charts
in
the
chart
and
information,
so
when
a
bt
gates
needs
to
find
a
char
in
the
test,
it
does
that
through
information
that
comes
from
the
topology
service
and
then
on
the
right,
you
have
bt
tablets
bit.
It
out,
as
you
can
think,
of
them
as
a
sidecar
that
protects
my
Seco
from
bad
users
in
your
application
and
Bitto
gates,
talk
to
the
charts
through
bt
tablets
and
then
also
an
important
thing
in
the
test.
B
B
So
with
that,
how
do
you
actually
deploy
a
test
in
your
infrastructure?
The
standard
with
test
deployment
looks
something
like
this.
First,
it's
really
important
to
notice
this
distinction
that
the
there
is
global
and
local
topologies.
The
idea
is
that
in
the
global
topology,
you
have
the
minimum
information
that
is
required
to
strut
the
components.
So,
for
example,
when
a
vita
gate
starts,
the
first
thing
that
is
that
it
does
is
that
it
asks
the
global
topology.
What
is
the
local
topology?
B
That
I
should
be
using
moving
forward
and
from
that
moment
on
in
never
talks
to
the
global
topology,
and
then
you
have
this
concept
of
cells.
Cells
are
also
a
first
class
citizen
impetus.
The
idea
is
that
you
deploy
all
components
within
the
cell
and
the
cells
encapsulate
a
failure
domain
and
they
should
map
to
your
data
centers.
So
then,
within
each
cell,
you
deployed
local
topology
with
the
gates
and
bt
tablets.
B
B
However,
it
does
not
allow
you
to
route
replicas
that
are
in
a
different
cell,
and
this
was
a
design
decision
that
made
sense
when
all
the
data
centers
were
physically,
like
separated
geographic
location,
so
how
we
map
this
deployment
to
or
infrastructure
as
slack.
So
as
we
mentioned
earlier,
we
use
AWS
heavily
an
orbit
test.
Cluster
is
deployed
on
top
of
in
situ
in
l,
WS
easy
to
is
hosted
in
multiple
locations
around
the
world
and
those
locations
are
composed
of
regions
with
availability
zones.
B
The
regions
are
separate,
separate
geographic
areas
and,
as
you
can
see
in
that
image,
you
can
have
a
region
in
US,
East
or
North,
Carolina,
Sydney,
Tokyo,
etc,
etc.
But
within
those
regions
you
have
this
availability
songs.
The
availability
songs
are
connected
to
low
latency
links,
and
the
idea
is
that
you
can
think
of
this,
as
you
have
data
centers
that
are
physically
separated,
but
still
close
to
each
other.
B
Normally
they
deploy
in
different
regions,
but
then
within
those
regions
they
deployed
in
different
places
and
that's
kind
of
what
we
did
a
slack
and
something
to
call
out
is
that
at
this
point
we
have
clusters
in
multiple
regions,
but
for
the
sake
of
the
example
I'm
just
going
to
use
one
region
also
we
deploy
in
multiple
agencies,
but
to
make
this
like
simple
I'm,
going
to
show
how
we
use
to
deploy
a
vides
cluster.
In-
let's
say
us
is
one
and
using
two
aces.
B
B
Next,
it's
important
to
notice
that
data
center
that
we
are
having
for
console,
that
is
the
default
data
center.
This
will
be
relevant
later
now.
The
next
important
thing
to
mention
is
that
we
deploy
a
single
cell,
this
cell
Maps
to
the
region,
but
across
multiple
ACS,
and
then
within
that
cell,
we
deployed
the
local
topology
for
the
cell,
but
notice
that
we
use
the
same
data
center
for
us,
the
global
topology,
meaning
that
we
were
sharing
the
same
underlying
infrastructure
for
console
for
both
the
global
topology
and
the
topology.
B
Last
but
not
least,
we
deployed
mitigates
in
with
the
tablets
in
physical
different
locations
because
they
were
spread
across
all
the
ACS,
but
from
the
test
perspective
they
were
part
of
the
same
cell,
so
how
this
compared
to
the
B
test,
standard
deployment
I
think
there
are
like
two
main
differences.
The
first
one
is
that,
instead
of
having
a
cell
per
data
center,
we
were
having
a
cell
for
the
whole
region.
This
does
not
seem
that
bad,
but
it
end
up
being
problematic.
You
may
ask
like
why
do
you
do
that?
B
And
the
reason
is
that
there
was
a
fundamental
limitation,
as
I
mentioned
in
earlier
in
the
test,
the
current
route
between
cells,
like
between
cross
cells.
When
you
were
doing
replica
reads
when
we
were
using
like
ACS
heavily,
we
needed
that
Abita
gate
in
a
given
AC
was
able
to
talk
to
another
tablet
that
was
in
the
same
region,
but
in
a
different
NC.
So
basically
from
the
Vita's
perspective.
At
that
point,
we
were,
we
didn't
have
any
other
option
than
using
a
cell
that
mapped
to
the
whole
region.
B
Second,
we
were
using
the
same
log
server
for
both
the
local
and
the
global
topology.
This
was
just
circumstantial.
A
slag
was
a
completely
different
company,
two
and
a
half
years
ago,
different
scale,
different
tooling.
So,
from
a
practical
perspective,
when
we
were
starting
to
ramp
up
these
inner
productions
services,
it
just
made
sense
to
reuse
the
same
console.
The
set
up
for
the
most
part
worked
like
pretty
well,
but
in
time
it
surface
some
problems
that
I
want
to
discuss
now.
So
what
happened?
We
would?
There
was
an
issue
in
console.
B
Well,
you
have
a
fire
in
the
global
topology
and
you
think.
Oh,
this
is
okay,
because
the
global
topology
is
not
needed
for
serving
queries,
but
because
it
was
the
same
underlying
console,
it
meant
that
you
also
had
a
fire
in
the
local
topology
to
make
your
team
more
complicated,
because
our
cell
span
across
the
whole
region
fired
in
the
local
in
the
console,
created
a
firing,
global
crater,
finding
local
and
also
a
fire
that
spread
across
all
or
a
disease,
and
because
you
know
when
it
rains
it
pours.
B
We
also
discovered
some
bugs
in
the
local
topology
that
affected
or
serving
path,
meaning
that
when
we
were
in
this
situation,
as
some
queries
will
fail
in
in
orbiters
cluster.
In
a
nutshell,
a
failure
in
or
default
console
data
center
put
us
in
a
very
bad
state.
So
I
think
that
mr.
Massey
coal
could
have
its
set
up
better.
If
the
desktop
is
made
of
you,
it
should
be,
and
after
this
it
was
definitely
in
top
of
my
for
us.
So
what
do
we
do
to
fix
this?
Well?
B
For
us,
the
most
important
part
was
to
serve
our
customers.
So
we
need
everything
that
we
could
to
make
sure
that
these
issues
do
not
affect
the
serving
path.
So,
every
time
we
discover
a
console,
a
problem
due
to
scale
and
then
having
another
ability
in
the
local
topology,
we
made
the
local
topology
more
robust,
and
if
you
look
in
the
upstream
meters
project,
you
will
see
many
contribution
from
the
slack
making
this
more
robust
and
more
resilient.
However,
when
you
think
about
it,
what
we
were
doing
is
just
taking
a
defensive
programming
approach.
B
Every
time
you
run
into
a
failure,
you
try
to
make
sure
that
the
failure
doesn't
happen
again
in
the
future.
But
for
us
this
was
not
enough
and
it
was
not
enough,
because
the
combination
of
a
single
cell
that
spread
across
multiple
Isis
plus
the
fact
that
we
were
sharing
the
same
underlying
infrastructure
for
the
topology
surface
as
a
fundamental
problem
in
order
program,
and
the
problem
was
not
that
we
were
having
issues
killing
the
console
cluster
or
that
there
were
bugs
in
the
local
topology.
B
This
should
be
business
as
usual
for
any
company
that
is
growing
a
really
fast,
and
that
is
constantly
pushing
the
limits
of
its
infrastructure.
The
problem
was
that
those
failures
cascaded
to
other
components
without
clear
boundaries,
so
we
step
back
and
kind
of
like
took
like
first
principles,
approach
and
thought.
What
do
we
need
to
make
our
deployment
more?
What
we
wanted
was
that,
when
the
unexpected
happened,
the
blast
radius
was
minimum
and
for
this
isolation
is
key,
and
here
we
mean
real
isolation.
B
For
instance,
in
our
case,
it
did
not
matter
that
we
were
already
using
multiple
ACS
if
they
were
tightly
coupled
together
and
I.
Think
in
that
image,
like
chips
are
like
really
good
at
samples
of
isolation
and
both
heading,
but
the
analogy
will
be
that
we
created
like
some
both
heading
but
then
left
like
a
small
hole
at
the
bottom,
and
then
it
didn't
matter
the
water
it
came
through
all
those
holes.
The
other
thing
that
we
think
is
really
important
is
that
you
think
about
your
dependencies.
B
This
might
sound
trivial,
but
is
not
when
there
is
fundamental
complexity
in
the
systems
that
you
are
operating.
Also,
when
you
take
the
fact
that
things
are
changing
rapidly,
it
is
also
important
to
mention
that
you
cannot
overlook
the
fact
that
sometimes
you
are
basically
learning
how
these
things
all
fit
together
in
the
full
mental
model
of
your
operating.
These
services
and
production
is
not
fully
developed.
But
if
you
don't
invest
in
this,
it
will
bite
you
hard.
B
In
our
case,
it
was
counsel,
but
everyone
should
see
like
what
are
the
dependencies
that
it
is
hard
to
see
what
will
happen
when
failures
in
those
dependencies
cascade
across
your
whole
infrastructure.
So
what
we
wanted
was
kind
of
to
have
the
standard
with
test
deployment,
but
that
was
able
to
map
to
an
AWS
infrastructure.
B
So
after
we
did
these
changes,
we
took
the
time
to
migrate
or
topologies
to
have
dedicated
console
clusters.
You
can
now
see
that
each
topology
has
a
different
console
data
center,
also
orbit,
as
deployment
now
have
cells
that
maps
to
ACS
instead
of
having
a
cell
for
the
whole
region.
This
has
a
pretty
neat
property.
The
failure
domain
and
the
blast
radius
of
orbiters
components
has
a
one-to-one
mapping
to
the
physical
infrastructure.
B
We
are
not
going
to
run
into
this
issue
again,
but
from
our
clients
perspective,
this
was
completely
transparent.
We
didn't
do
this
project
for
performance
reasons,
but
it
also
unlock
some
performance
games.
So,
given
that
now
that
ACS
are
a
first
class
citizen
impetus
as
well,
we
can
have
logic
in
VTA's
that
know
when
they're
routing,
to
a
replica
to
prefer
a
replica
that
is
in
the
same
AC
if
that
is
possible.
B
So
some
tables
at
SLAC
use
replica
reads
heavily
and
for
this
kind
of
table,
this
avoid
an
extra
network
hub
in
the
query
serving
path
which
had
latency
gains
and
also
some
cost
reductions
because
idle
years
charges
you,
when
you
cross
a
C's
in
this
graph,
you
can
guess
when
we
deploy
this
change
and
it
was
a
pretty
neat
performance
win
where
we
got
300
microns
back
in
the
p50
of
all.
Requests
to
that
table,
which
was
significant
for
us
last
I
want
to
start
closing
with
some
off
like
Carol.
B
First
I
think
it
is
important
to
have
in
mind
that
complex
systems
are
simply
dangerous,
and
here
I
mean
like
fundamental
complexity,
when
you
are
running
systems
that
need
to
have
10
million
users
connected
at
the
same
time,
the
infrastructure
to
support
that
has
some
complexity.
That
is
not
incidental
and
then,
when
you
have
the
windows
failure.
When
those
systems
fails,
the
consequences
are
very
high
and
like
basically
it's
like
money
that
is
being
lost
over
time.
B
It
is
natural
that
you
will
construct
multiple
layers
of
the
defense
against
those
kind
of
failures,
including
like
redundancy
backup
systems
and
also
the
human
processes
that
make
life
safer
to
operate
those
systems.
But
what
is
more
interesting
to
us
is
that,
in
these
kind
of
systems,
catastrophe
is
always
just
around
the
corner,
especially
when
there
are
changes
so
rapidly,
and
it's
just
not
feasible
to
protect
from
unexpected
failures.
So
you
kind
of
have
to
embrace
this
fact
is
not
enough
to
defend
against
failures.
B
So
I
found
like
pretty
remarkable
because
to
me:
if,
if
I
didn't
tell
you
I,
think
all
those
bullets
apply
like
1
to
1
to
distribute
complex
system,
but
in
fact
they
come
from
a
different
field
and
then
I
think
this
led
us
to
think
that
our
inner
industry,
sometimes
we
feel
that
we
are
always
on
the
edge
of
everything
and
that
we
are
always
solving
like
super
normal
problems.
But
in
fact
there
are
many
lessons
that
we
can
get
from
other
industries
and
I.
B
Think
to
us
is
really
important
that
you
have
like
humility
towards
this
kind
of
complexity
and
we
reach
out
to
other
fields
and
learn
from
the
experience
and
with
that
I
think
that
wraps
up
what
we
wanted
to
share
today,
also
just
a
quick
theme
if
you
are
more
interested
in
away
with
test.
There
are
many
talks
too
that
are
happening
this
week.
These
are
some
of
them.
Thank
you
very
much,
and
if
you
have
questions
we
are.
C
C
E
C
So
part
of
again
scaling
our
fleet
is
like
doing
capacity
planning
and
again
as
part
of
capacity
planning
front
at
the
time
as
team
we
decide.
Okay,
maybe
we
need
to
recharge
again
a
key
space,
so
a
set
of
table
or
database
and
again
that's
something
that
we
test
is
very
powerful
because
that's
done
and
they
say
that's
something
that
already
provided
out
of
the
box.
So
there
is
very
little
kind
of
operational
overhead
to
do
that.
Can.
F
C
B
We
use
an
open
source
tool
called
Orchestrator,
so
our
B
test,
shards,
are
my
sequel
cluster
and
they
are
watched
by
Orchestrator.
So
when
there
is
a
master
fell
over,
there
is
an
automated
process
that
chooses
another
master
and
promotes
that
a
master
and
notifies
with
tes
backups,
that's
a
good
one,
so
we
are
using
ghost,
but
because
in
B
test
we
have
like
now
thousands
of
charts.
We
build
an
internal
tool
that
orchestrates
is.
C
H
H
B
C
D
C
Replicas,
just
read-only
replicas.
So
again
we
don't
want
to
have
replicas
ahead
of
the
master,
because
that
really
impact
again
then
like
topology
changes,
consistency
and
so
on
yeah.
So
all
replicas
are
technically
the
same,
of
course
we're
running
with
semi
sync,
so
we
might
have
like
a
bit
of
drift
between
all
of
those.
But
again
that's
expected.
Usually
when
we
again,
we
want
to
read
after
write
consistency
like
right
now
in
Vitesse.
C
You
can
just
do
it
by
creating
the
master,
and
you
can
specify
that
we
are
also
evaluating
to
a
standing
Vitesse
to
expose
in
the
global
transaction
ID
that
our
replica
has
and
then
every
time
that
you
do
a
write.
You
get
back
in
the
client,
a
global
transaction
ID
and
on
the
next
you
can
say:
hey
Vitas,
like
a
query
whatever,
but
please
use
a
replicated.
That
is,
at
least
at
this
point
in
the
transaction.
C
I
C
J
B
The
good
thing
is
that,
because
we
treat
like
all
or
instances
kind
of
like
ephemeral
when
all
the
code
changes
were
ready
to
support
that
we
deployed
the
console,
the
new
console
topologies,
we
created
a
new
cell
symbol
test
and
then
it
was
just
a
matter
of
recycling
all
nodes
in
the
fleet
and
then
when
they
come
up,
they
really
register
to
a
new
cell.
And
then
that
was
it
so
actually
like.
The
big
effort
was
just
making
it
as
ready
for
that
but
deploying
it.
C
B
K
K
C
C
L
C
Again,
I
don't
know
if
it's
kind
of
best
practice
but
I
will
suggest
exactly
what
you
said.
So
you
usually
do
something
within
the
same
region
or
maybe
even
like
close
region
and
then
I
don't
know.
If,
for
disaster
recovery
port,
we
do
disasters
cover
in
another
way
but
like
if
you
need
to
again
have
a
synchronous
replicas
in
Sydney.
Your
data
center
is
in
Europe.
You
can
do
that.
You're,
just
gonna
pay,
probably
150
second
round
trip
lettuce,
so.
B
B
Quickly,
yeah,
so
in
the
new
architecture,
this
is
how
it
looks
so
we
deployed
the
global
topology
in
the
region
across
all
ASIS,
and
then
we
create
cells
for
each
AC
and
those
cells
map
to
with
ourselves
and
when,
within
those
cells,
deploy
all
with
as
components.
So
the
cell
is
like
the
whole
failure
domain
for
okay.
M
C
M
B
Top
illegal,
it
does
not
apply,
but
the
good
thing
about
the
global
topology
is
that
it's
only
required
to
bootstrap
the
other
components.
So
our
idea
is
that
if
a
global
topology
is
partially
unavailable
because
there
are
issues
between
ACS
that
only
affects
bringing
up
new
things,
but
not
normal
operation.
Okay,.
G
B
C
Again,
we
have
another
team
that
manager
kind
of
key-value
like
locking
service
and
service
discovery.
Again
they
do
backup.
Is
it
regularly
so
again
for
us,
as
the
global
topology
is
not
on
the
serving
path
like
we
wouldn't
be
impact
yeah
like
we
will
be
impact
like
if
we
need
to
bootstrap
again
and
other
cells
or
like
things
like
that,
Thanks
all.