►
Description
Don’t miss out! Join us at our upcoming hybrid event: KubeCon + CloudNativeCon North America 2022 from October 24-28 in Detroit (and online!). Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Building for the (Inevitable) Next Cloud Outage - Pavel Nikolov, Section
It seems that every few months we hear about the widespread impact of a major cloud outage. Cloud outages are unpredictable and inevitable; this is what keeps SRE teams up at night. Public cloud remains the most popular data center approach among the cloud native community, with multi-cloud growing in adoption. However, adopting a multi-cloud strategy isn’t as simple as hitting the go button. In this session, we will demonstrate how to deploy a Kubernetes application across clusters in multiple clouds and regions with built-in failover to automatically adapt to cloud outages. You will witness how BGP directs traffic across clusters in a healthy state. Then, we will take one of the clusters offline and show how workloads are automatically rescheduled and traffic is rerouted to healthy clusters in real-time. We will dive into the technologies and logic that are driving this engine and discuss how you can build this type of resilience into your own applications.
A
A
A
There
are
external
factors
like
upstream
provider,
internet
connection
issues.
There
is
the
human
factor,
and
sometimes
we
can
make
mistakes
in
our
deployments
or
code
that
we
deploy.
There
are
natural
disasters
that
can
take
regions
down
or
cause
major
trouble
for
all
kinds
of
services
around
the
globe,
and
it
is
prohibitively
expensive,
no
matter
what
type
of
business
you
are
to
deploy
your
applications
everywhere
around
the
world
at
the
same
time.
A
So
it
is
not
a
question
of
if
your
servers
will
go
down
or
services.
It
is
a
question
of
when,
and
usually
these
things
happen
when
you
least
expect
it
or
you're
not
prepared
for
them,
and
somebody
gets
paged
in
your
company
and
they
have
to
figure
out
in
probably
in
the
middle
of
the
night,
if
they're
lucky
during
business
hours,
how
to
solve
a
problem
and
even
major
things
like
your
dns
can
go
down
and
your
team
can
make.
You
know
a
mistake
that
is
difficult
to
roll
back.
A
A
Many
of
us
usually
deal
with
these
things
with
disaster
recovery
and
the
vast
majority
of
people
fall
in
one
of
these
four
categories.
When
it
comes
to
disaster
recovery,
you
either
have
some
form
of
active
active
deployment,
which
means
you
know
if
your
primary
server
goes
down.
You
flip
the
switch
in
your
dns
and
your
request
go
to
the
second
one,
the
which
is
active.
If
you're
in
this
category,
probably
or
one
of
the
lucky
guys,
the
other
one
is
active
passive.
A
It
is
similar
to
the
previous
one,
but
it's
cheaper
because
you
don't
pay
for
the
hosting
for
the
passive
instance
or
cluster
or
whatever
it
might
be,
and
you
have
to
spin
up
the
passive
instance.
Then
you
flip
the
switch
in
your
dns
and
now
you
have
happy
users.
No
more
complaints
in
you
know
social
media
that
your
services
are
down,
or
maybe
your
customers
aren't
happy.
A
Another
option
is
again
to
have
a
periodic
backups
of
your
databases
and
then,
when
things
go
down,
you
spin
up
your
code
somewhere,
you
restore
the
backups,
and
then
you
continue
serving
as
normal
and
the
final
category
is.
You
don't
have
any
disaster
recovery
whatsoever
whatsoever
because
you're
busy
building
features
and
when
things
go
down,
you'll
figure
it
out,
and
unfortunately,
there
are
way
too
many
people
in
that
category.
A
Even
people
who
used
to
be
in
the
active
active
one,
they
tend
to
fall
down
the
the
category
list
because
it
requires
extremely
high
discipline
and
and
human
factor,
the
reason
I've
titled.
This
slide
like
this
is
because
your
entire
team
needs
to
be
responsible
about
what
will
happen
in
case
of
a
disaster,
because
disasters
will
happen.
A
Disaster
recovery
usually
requires
to
be
played.
You
know
you
have
some
kind
of
run
books
or
some
kind
of
script,
something
that
you
have
to
execute
in
a
case
of
a
disaster
or
some
system
of
yours,
that's
the
best
case.
No
people
are
involved.
Some
system
will
execute
it.
When
some
metrics,
you
know
hit.
A
On
reasonable
low
or
high
level,
whatever
the
metric
might
be,
you
still
have
to
replay
your
disaster
recovery
plan
over
and
over
again
because,
as
you
keep
adding
new
features
and
new
components
to
your
system,
your
disaster
recovery
might
change.
So
it's
not
created
once
and
be
comfortable
forever.
You
have
to
replay
your
disaster
recovery
plan.
A
Maybe
every
month,
every
quarter,
or
as
often
as
you
can
not
all
the
companies
can
afford
to
have
a
dedicated
sra
team
that
costs
money.
Not
everybody
can,
you
know,
be
like
the
big
guys
out
there
like
big
companies.
I
won't
mention
names,
but
you
know
they
have
huge
sra
teams,
not
especially
startups.
You
can't
afford
to
have
a
huge
sre
or
devops
team.
That
is
only
responsible
about
this,
and
does
your
disaster
recovery
require
manual
intervention
intervention.
A
Even
if
you
have
you
know
reasonably
good
disaster
recovery
plan
oftentimes
somebody
has
to
wake
up
or
step
in
or
open
their
laptop
in
the
middle
of
a
trip
or
whatever
and
type
in
some
magic
in
the
terminal
and
things
get
restored
if
they
get
notified
on
time
or
and
so
on,
and
so
on.
So
there
is
a
human
factor,
oftentimes
involved
during
an
incident
or
disaster
recovery,
and
it
all
boils
down
to
these
two
terms:
rto
and
rpo.
A
They
are
extremely
important
in
solutions,
architecture,
world
or
even
management
meetings.
The
rto
is
recovery,
time
objective
and
many
times
you
have
a
talk
with
your
service
provider,
or
you
know
some
some
some
form
of
cloud
provider.
What
is
their
rto
like?
What
is
the
recovery
time
objective
in
case
of
a
disaster?
How
soon
can
you
recover
from
the
disaster
guaranteed
like
there
is
nothing
guaranteed
in
this
world,
but
we
have
some
goals.
They
give
you
some
soas
and
the
rpo
is
how
much
data
can
you
afford
to
lose
during
an
incident?
A
So
if
you
talk
to
your
high
management
or
your
customers,
they
will
always
want.
You
know,
recovery
time
objective
immediately.
Actually
they
don't
want
any
downtime
and
recovery
point
objective
in
some
industries
actually
you're
not
allowed
to
lose
data
financial
industry,
for
example,
healthcare,
industry,
you're
not
allowed
to
lose
data
that
that
could
that
could
lead
to
serious
compliance
issues.
If
you
lose
data
in
these
industries
and
during
an
incident,
you
can't
predict
what
will
go
down
so
these
two
terms,
you
can
hear
sre
teams
talking
about
these
things
over
and
over
again.
A
So
I
have
my
opinionated
view
that
this
type
of
problems
can
be
solved
with
bgp,
specifically
any
anycast
ip
addresses,
and
how
many
people
in
the
room
are
familiar
with
bgp
or
any
cast.
Okay,
we
have
half
the
room.
That
is
super
cool,
so
for
those
that
don't
know
what
bgp
is
I'll
mention
it
very
quickly.
A
The
internet
is
like
a
network
of
networks
and
bgp
is
excuse.
My
drawing
skills,
but
bgp
is
like
make
sure
that
these
networks,
which
are
called
autonomous
systems
each
communicate
between
each
other
in
the
most
efficient
way.
So
if
there
is
one
server
that
needs
to
reach
another
server,
bgp
makes
sure
that
these
tcp
packets
from
server
a
to
its
destination,
find
the
most
efficient
route
on
the
internet.
A
Each
of
these
dark
dots
on
the
map.
They
are,
let's
call
them
bgp
actors,
they
announced
some
ip
range
and
each
of
them
tells
to
their
their
peer
peers.
Their
neighbors
hey
this.
This
ip
range
lives
in
my
autonomous
system,
tell
it
to
your
peers
and
then
each
autonomous
system
propagates
that
message
to
everyone
else.
So
when
a
server
announces
an
ip
address,
bgp
makes
sure
that
in
couple
of
seconds,
the
whole
internet
knows
where
this
ip
range
lives.
A
Then,
when
there
is
a
packet
that
needs
to
reach
one
ip
from
dust
range,
everybody
in
the
world
knows
where
to
send
that
ip
to
and
when
it
reaches
this
guy,
then
it
uses
internal
routing
to
find
the
exact
server
with
the
exact
ip
address
and
send
your
packets
there.
If
that
server
exists,
and
it
will
be
the
shortest
path.
A
This
is
important
because
it
is
what
I'm
going
to
be
using
in
my
demo
and
what
is
any
cast
ip.
That's
another
thing
that
you
need
to
understand:
there
are
different
tcp
packets
on
the
network.
There
is
unica
unicast
packets,
and
these
are
one-to-one
packets.
You
have
one
server
that
sends
a
message
over
the
network
and
that
message
is
supposed
to
reach
exactly
one
destination
they
are
both
somewhere
on
the
internet.
Bgp
will
make
sure
that
this
packet
will
reach
the
destination
server
in
the
most
most
efficient
way.
A
There
is
also
multicast.
Multicast
is
different
in
that
again,
there
is
one
source
of
the
iep
packet,
but
there
are
multiple
destinations.
Think
of
it
as
a
chat
room.
You
type
a
message
in
the
chat
room.
You
don't
set,
send
the
same
tcp
packets,
to
let's
say
all
of
the
hundreds
of
members
of
the
chat
room
and
when
you
send
it
these
packets
reach
all
of
the
members
and
they
get
multiplexed,
that's
multicast
and
anycast,
which
is
what
is
what
is
the
most
interesting?
This
is
where
the
magic
happens.
A
A
A
It
has
ttl
and
we've
tried
in
production
with
dns
servers
that
have
extremely
low
ttl,
but
not
all
isp
providers
honor
that
they
don't
respect
it.
We
still
have
had
to
wait
like
five
and
more
minutes
to
recover
from
a
disaster
that
doesn't
depend
on
us.
We've
set
proper
ttl,
we
tried
to
recover
and
it
just
doesn't
happen
because,
somewhere
around
the
world,
there
is
somebody
who
didn't
own
our
tto
and
the
customers
happen
to
be
just
on
the
other
side
and
there
they
say.
A
A
The
downsides
of
bgp:
well,
you
have
to
have
your
own
ip
range,
but
the
good
thing
is
you
buy
it
only
once
and
your
cloud
provider
has
to
support,
bring
your
own
ip
address,
but
most
of
them
do
majority.
The
vast
majority
of
cloud
providers
support
that
and
of
course
there
is
learning
curve
like
if
you
don't
have
the
expertise
in
your
team.
A
A
I've
created
single
nose
clusters
because
I'm
lazy
I've
used
k3s
just
to
deploy
some
bare
metal
instances.
I've
created
the
kubernetes
cluster
there
and
let's
show
that
I've
recorded
my
demos,
because
actually
I
had
to
record
it
11
times,
because
I
wanted
to
show
something
and
it
happens
only
in
certain
circumstances.
A
Can
everyone
see
it
okay?
What
we
are
looking
at
is
my
yaml
file
of
a
simple
deployment.
It
is
a
simple
golang
application
that
has
a
small
graphql
server
and
I've
deployed
that
to
all
three
kubernetes
clusters.
It
is
exactly
the
same
everywhere
with
one
little
difference:
it
has
this
environment
variable
which
is
location,
and
I've
created
that
just
to
know
where
my
responses
are
coming
from.
A
Okay,
I
overwrite
in
my
customization
yaml
I
override
just
the
value
of
that
variable.
Everything
else
is
the
same
in
all
my
clusters,
and
this
simple
demo
will
now
show
you
I'm
accessing
I'm
intentionally
not
using
a
domain
name,
I'm
using
ips
to
show
you
that
I
don't
use
dns.
Okay,
I'm
now
accessing
my
servers
using
the
ip
that
was
provided
to
me
by
the
cloud
provider.
A
I
will
prove
to
you
that
they
all
have
the
same
public
type.
They
actually
have
two
public
ip
addresses,
the
one
that
was
given
to
me
and
another
one
that
I
announced
and
I
added
myself.
So
I
grabbed
the
response
from
ip
address,
and
I
look
for
that
particular
ip
address
and
the
rating
on
this
directing
on
the
screen
shows
you
that
all
three
of
them
they
have
the
same
public
ip
address,
that's
on
any
anycast
ip
address.
A
A
So,
while
true
every
second
send
the
request
to
my
cluster,
my
little
application
and
tell
me
where
the
responses
are
coming
from,
and
I
start
getting
responses
from
amsterdam,
I'm
using
the
bird
demon
2
to
announce
my
ip
range
and
I
stopped
announcing
my
ip
range,
which
simulates
your
region
going
down,
and
then
you
see
a
little
error.
I
want
to
commend
this
error
because
I
recorded
this
demo
11
times
to
show
there,
but
you
see
that
in
less
than
a
second
I
started
getting
responses
from
new
york.
I
did
not
touch
anything.
A
I
didn't
have
to
do
anything.
I
don't
even
get
paid.
If
I
don't
want
to
right,
then
I
took
down
new
york
and
I
started
getting
responses
from
sydney
without
touching
anything.
My
system
is
hundred
percent
uptime,
and
this
is
how
the
internet
works.
I'm
using
the
backbone
of
the
internet,
bgp
is
the
backbone
of
the
internet.
I
didn't
have
to
disaster
disaster
recovery.
Bgp
does
it
for
me
and
the
reason
I
got
this
error,
this
little
error
and
the
reason
I
recorded
my
demo
11
times
is:
I
wanted
to
show
an
error.
A
A
A
So,
if
you're,
using
something
like
grpc,
some
some
other
long,
lift
connections
or
database
connections
or
some
kind
of
streaming,
you
just
have
to
reconnect
and
then
continue
as
normal.
So
that's
why
you
can't
guarantee
hundred
percent
of
time,
but
it's
pretty
close.
It
took
less
than
a
second
for
the
same
continent
is
going
to
be
milliseconds
and
my
system
recovered
like
how
cool
is
that.
A
Just
remember
that
long-running
connections
have
to
be
reconnected.
That
is
important
and
you
can't
have
a
peace
of
mind.
Oh
this
thing
will
save
me
in
all
situations.
This
has
you
know
serious
implications
and
you
have
to
be
prepared
and
you
have
to
recommend
reconnect
and
your
developers
just
need
to
know
about
this.
They
don't
need
to
be
low
level
network
experts.
They
need
to
understand
how
this
thing
works.
A
Okay
and
now
the
elephant
in
the
room,
the
question
that
I
get
after
showing
this
to
someone
always
always
what
about
my
data,
like
you,
just
killed
one
region,
but
you
promise
low
latencies,
but
your
latency
is
as
low
as
the
nearest
data
right.
If
you
know
you're,
adding
and
removing
clusters
on
the
map,
what
do
we
do
it
with
with
data,
especially
data
consistency?
A
I
believe
that
in
the
vast
majority
of
cases,
especially
micro
service
deployment,
I'm
a
developer,
especially
in
microservices
deployment.
The
answer
to
the
data,
consistency,
problem
and
latency
problem,
reliability,
problem
and
disaster
recovery
problem
is
eventual
consistency,
and
there
is
this
cup
theorem
diagram.
How
many
people
in
the
room
are
familiar
with
the
cup
theorem?
A
Okay,
quite
quite
some
people,
okay,
for
the
remaining
half
which
are
not
which
didn't
raise
their
hands
every
database
or
data
store
solution
that
exists
out.
There
can
have
at
most
two
of
these
three
features:
availability
being
always
online
partition,
tolerance,
meaning
your
cl.
Your
nodes
can
go
up
and
down
at
any
time
like
in,
in
our
team,
being
an
edge
compute
platform.
All
clusters
appear
and
disappear
on
the
map
at
any
time,
and
then
the
third
one
is
consistency.
A
A
You
can
use
a
hosted
db
solution
that
promises.
You
know
high
availability,
but
this
is
yet
another
moving
part
in
your
big
picture
right
and
do
they
do
they
provide
100
guarantee
chances?
Are
they
will
until
you
sign
a
contract?
Do
they
have
hundred
percent
consistency?
Some
of
them
do?
How
much
do
they
cost
and
are
they
yet
another
single
point
of
failure
in
your
system?
A
A
It
is
ideal
for
microservices,
because
your
microservice
can
use
first
a
cqrs
pattern,
which
is
you're
you're,
producing
events,
and
somebody
is
consuming
them,
but
the
producers
do
not
need
to
know
who
is
consuming
them
and
the
consumers
do
not
know
where
the
event
comes
from
came
from
and
in
a
micro
service
world
where,
technically
speaking,
in
order
to
keep
up
with
good
practices,
you
have
to
have
at
most
one
service
using
one
database.
So
every
database
has
exactly
one
owner.
A
A
One
benefit
with
that
is
that
your
data
is
immutable.
You
never
have
a
delete
in
your
system
like
you,
keep
appending
data
forever,
your
bugs
and
data
and
requests
are
repeatable.
You
can
replay
the
events
since
beginning
of
time
at
any
at
any
point
in
time,
and
you
reproduce
the
problem
that
you
had
and,
of
course
it
results
in
eventual
consistency
which
fits
exactly
what
I'm
talking
about.
A
Let's
say
git,
you
start
working
in
a
company
and
this
is
your
first
day
and
you
want
to
write
some
code,
but
you
have
to
download
the
project
on
your
computer,
so
you
get
check
out
the
project
you
consume
the
commit
since
beginning
of
time,
which
is
just
events.
Then
git
builds
the
materialized
view,
current
state
of
the
world
current
files
on
your
file
system.
You
start
editing
them
and
when
you're
done
you
start
sending
commits
your
colleagues
start
sending
commits
everybody's
sending
commits
you
don't
know
where
the
commits
came
from.
A
A
Imagine
that
events
are
things
that
happened:
okay,
you're
sitting
in
your
office,
and
somebody
is
doing
something
somebody
turned
on
their
computer.
Somebody
went
for
coffee.
Somebody
is
doing
something
you
can
ignore
all
these
events.
There
are
events
that
happen,
but
you
don't
care
about
them.
Suddenly,
your
boss
enters
the
office
everybody's,
pretending
that
they're
working
that's
a
significant
event,
so
your
app
can
only
react
to
a
significant
event.
The
events
that
it
cares
about
it
can
ignore
all
the
rest,
which
means
that
you
can
always
consume
the
event
since
beginning
of
time
in
seconds.
A
A
So
what
I
do
is
I
check
the
last
event,
the
one
that
is
the
last
in
the
queue
my
readiness
probe
is
false.
I
start
consuming
events
since
beginning
of
time
when
I
reach
that
one
event,
I
turn
my
readiness
probe
to
true.
I
start
serving
requests
and
then
I
continue
consuming
events
from
that
point
on
as
normal,
I
build
my
materialized
view.
A
A
What
we're
looking
at
is
I'm
trying
to
hit
my
data
in
my
new
york
cluster.
There
isn't
anything
there,
because
it's
a
clean
slate
empty
cluster,
empty
application,
nothing
there,
no
events
and
no
articles.
This
is
an
article
application
and
if
I
try
the
same
thing
from
my
anycast
ip
address
again,
I'm
hitting
amsterdam
because
that
it's
the
close
it
is
the
closest
location
to
to
valencia.
A
I
paste
from
my
clipboard
some
articles
that
I
I
want
to
insert
and
that
will
generate
some
events
for
me.
I
will
try
to
probably
update
some
of
them
delete,
some
of
them
just
to
create
some
more
events.
I
want
to
create
events
that
get
propagated
nuts
jet
stream,
make
sure
that
these
events
will
get
propagated
to
all
the
other
clusters
in
the
nuts
mesh.
A
A
I
can
even
delete
that
article.
If
I
wanted
to,
I
can
change
check
what
what
I
have
currently,
I
have
like
five
five
articles
in
my
database
and
my
database
is
a
variable,
a
map,
simple
map
in
my
memory.
When
I
hit
that
request,
I
request
the
data
in
memory.
I
don't
have
to
make
a
database
connections
connection,
let's
delete
some
article
again.
A
I
send
the
request
that
request
goes
to
nuts.
Then
it
the
event
comes
back.
I
consume
the
event
and
I
change
my
map
variable
with
with
my
articles
and
I'm
left
with
only
four
articles.
Now,
let's
kill
this
server.
Okay,
let's
queue,
I'm
stopping
I'm
using
bird,
the
bgp
demon,
which
is
open
source.
Everything
in
this
session
is
open,
source
and
iq.
A
There
was
no
downtime
for
my
users
and
because
browsers
tend
to
retry
until
they
connect,
you
actually
will
almost
never
have
a
downtime
in
your
browser
and
all
the
data
that
I
created
in
my
amsterdam
cluster
is
now
in
my
new
york
cluster
and
in
my
sydney
cluster,
and
I
didn't
have
to
do
anything
about
it.
That's
how
nuts
jet
stream
works
like
there
were
other
talks
in
this
conference.
I
well,
I
linked
one
of
them
from
a
previous
conference.
A
I
can
give
an
another
example.
I
could
build
like
a
micro
service
to
this
publishing
platform,
that
is,
let's
say,
an
rss
service,
rss
service.
Let's
say
it
only.
Has
the
last
20
articles
that
my
team
has
published
every
time.
I
start
this.
I
don't
even
need
to
consume
the
events
since
beginning
of
time,
because
if
the
company
is
publishing,
let's
say
500
articles
a
day,
I
can
only
consume
the
events
since
yesterday.
A
A
In
summary,
outages
will
happen
sooner
or
later.
It
is
expensive
and
non-trivial
to
keep
up-to-date
disaster
recovery
plan
that
will
guarantee
low.
You
know
short
rto
and
zero
lost,
rpo
and
anycast
and
event
sourcing
are
really
good
fit
to
help.
You
solve
these
problems
as
a
given
benefit,
because
any
cast
ip
addresses
have
the
same
address
for
many
servers
around
the
world.
If
you
get
ddosed
guess
what
happen,
you
load
balance
the
request
to
all
your
servers
or
if
the
requests
come.
A
If
it's
a
dos
attack
coming
from
one
location,
you
spin
up
a
cluster
next
to
that
and
all
those
attacks
go
to
that
one
cluster
which
doesn't
even
need
to
have
any
workload
and
the
remaining
servers
are
not
even
not
touched,
so
that
gives
you
even
more
control
of
protecting
of
yet
another
problem.
That
might
happen.
A
All
the
resources
in
this
demo
that
I've
used
are
open
source,
no
proprietary
solution.
I'm
going
to
share
the
the
the
code
that
I've
written
in
in
my
github
profile.
I've
used
bgp
the
border
gateway
protocol,
which
is
the
backbone
of
the
internet.
Not
my
favorite
cncf
project
and
this
video
over
there
is
highly
recommended
bird
is
the
that
I've
used
to
announce
the
ip
range
and
probably
your
favorite
cloud
provider
has
some
documentation
how
to
announce
your
own
ip.
A
Like
it's
outside
of
the
scope
of
this
talk,
I
just
wanted
to
mention
that
yeah,
it's
possible
to
have
same
ip
address
in
multiple
servers.
I
use
k3s,
which
helped
me
build
this
demo
in
in
hours.
Instead
of
you
know
a
lot
more
time
in
the
graph
graphql
library,
I'm
I'm
maintaining
that
library,
so
I'm
a
bit
like
disclaimer,
I
chose
it
because
I
had
an
existing
app
that
I
tweeted
a
little
bit
and
with
all
that
being
said,
I
believe
that
there
is
some
time
for
questions.
If
anyone
wants
to
ask
anything.
B
Thank
you.
I
just
have
one
question
really,
which
is
just
about
the
persistence
behind
the
scenes,
so
you
mentioned
that
about
cqrs
and
you're,
obviously
interrogating
something
which
is
durable,
somewhere
yep,
how
how
in
a
cloud
scenario
when
potentially
you've
got
an
outage,
and
then
the
storage
is
gone
as
well.
How
do
you?
How
do
you
manage
that?
Okay,
that
use
case.
A
So
the
question
is:
what
happens
if
my
durable
queue
goes
down,
which
in
my
case,
is
in
this
demo
is
not
jet
stream?
Not
jet
stream
was
designed
to
have
zero
points
of
failure,
which
means
that
in
all
my
kubernetes
clusters,
I
can
have
a
nuts
cluster
with
three
or
more
instances
and
they
can
use.
You
know
persistent
volume
claims
and
if
that
goes
down,
I
will
have
some
kind
of
health
check.
A
We
turn
off
the
bgp
announcement
in
that
region
or
in
that
cloud
provider,
because
we
have
different
column
providers
in
different
regions
and
we
are
multi-tenancy
systems,
so
things
are
going
down
all
the
time
and
if
we
had
to
fix
those
things
manually,
probably
we
had
to
double
our
team
and
only
have
people
that
are
doing
disaster
recovery,
but
because
we
have
a
health
checker,
we
turn
off
the
bgp
announcement
and
then
that
cluster
is
off
the
network.
It
doesn't
accept
requests
anymore.
A
Our
user
requests
go
to
the
nearest
cluster
to
them,
which
is
ideally
in
the
same
region,
but
if
not
in
a
different
region
or
a
different
cloud
provider,
and
then
we
we
don't
have
to
care
about
this
pro.
It's
a
problem
for
another
day.
So
these
problems
happen.
That's
why
we
have
bgp
to
take
care
of
them
for
us.
B
A
A
C
C
A
It
does
work
and
many
people
use
it.
This
way
you
can,
you
can
have
somebody
else
in
front
of
your
bgp
announcements
or
after
your
bgp
announcements.
There
are
many
many
talks
explaining
different
scenarios
there,
I'm
just
in
in
my
presentation,
I'm
just
saying:
hey
it's
possible
to
have
two
servers
with
the
same
ip
address.
There
are
really
a
lot
of
solutions
that
you
can.
You
can
work
with.
In
our
case
we
have
our
bgp
and
and
after
that
we
have
a
load
balancer.