►
From YouTube: Migrating From Single-Node Kubernetes Control Plane To HA In Product... Cong Yue & David Oppenheimer
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2023 in Amsterdam, The Netherlands from April 17-21. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Migrating From Single-Node Kubernetes Control Plane To HA In Production - Cong Yue & David Oppenheimer, Databricks
Databricks adopted Kubernetes in 2016, before highly available (HA) Kubernetes control plane deployments were common. As a result, we built our self-managed Kubernetes clusters using a single-node control plane in AWS, and then later also on Azure and GCP. Recently we migrated these production clusters to use a multi-node control plane, which provides higher reliability and enables us to upgrade Kubernetes versions more safely and therefore faster across the fleet. In this talk we describe the architecture we chose for our HA control plane, how we safely migrated a fleet of clusters from a single-node control plane to HA without affecting workloads in production, and how we adapted some of our Day 2 operations to accomodate multi-node control plane.
A
B
So
briefly,
the
outline
of
the
talk
is
that
first
I'll
talk
about
how
we
use
kubernetes
at
databricks.
I'll
talk
about
the
non-ha
control,
plane,
architecture
that
we
used
for
many
years.
Then
I'll
discuss
the
control,
plane,
architecture
that
we
moved
to
and
how
it
handles
different
kinds
of
failures.
Then
Kong
is
going
to
talk
about
the
migration
process.
B
So
the
as
a
brief
overview
of
what
databricks
is
the
databricks
product
is
a
SAS
data
platform
that
runs
on
the
public
clouds.
We
call
it
the
databricks
lake
house
platform
and
it's
a
unified
platform
that
serves
many
Enterprise
data
use
cases
such
as
data
warehousing
data
engineering,
data,
science,
streaming
and
machine
learning.
B
B
So
this
slide
shows
the
high-level
architecture
of
the
databricks
platform.
It
consists
of
a
control
plane
that
runs
in
databricks,
owned,
Cloud
accounts
and
a
per
customer
data
plane
that
runs
in
the
customer's
cloud
account.
So
you
can
see
in
the
left
hand,
box
some
of
the
services
that
constitute
the
databricks
control
plane.
B
These
are
multi-tenant
services
that
run
on
kubernetes
clusters
and
then
the
right
hand
box
shows
the
data
plane
for
One
customer,
which
consists
of
cloud
storage
and
the
compute
that
does
all
of
the
data
processing
for
historical
reasons
on
gcp
the
data
plane
runs
on
kubernetes,
but
on
AWS
and
Azure.
It
runs
on
VMS
that
are
not
managed
by
kubernetes,
but
that's
the
data
plane.
B
This
talk
is
about
the
control
plane,
all
of
which
runs
on
kubernetes,
so
the
databricks
product
gives
customers
the
experience
of
a
single
unified
system
spanning
the
three
clouds,
but
under
the
covers,
the
control
plane
is
built
from
per
Cloud
region.
Kubernetes
clusters
databricks
is
currently
available
in
more
than
60
regions
across
three
clouds,
and
we
have
at
least
one
kubernetes
cluster
per
region
to
host
the
databricks
control
plane
services.
B
Now
databricks
was
a
very
early
adopter
of
kubernetes.
The
company
adopted
kubernetes
in
2015,
which
was
before
all
three
Cloud
providers,
had
managed
kubernetes
services.
So
we
built
our
own
tools
to
deploy
and
manage
clusters
directly
on
top
of
cloud
provider.
Vms
we've
recently
also
started
using
cloud
provider,
managed
kubernetes
clusters
for
certain
services,
but
this
talk
is
about
our
self-managed
kubernetes
clusters
in
these
60
regions
and
we're
still
primarily
using
the
the
self-managed
Clusters.
Even
though
we're
starting
to
adopt
a
cloud
provider
manage
kubernetes.
B
So
this
slide
shows
the
non-aha
cluster
architecture
that
we
had
been
using.
It's
a
very
standard
architecture
that
most
people
who
are
running
kubernetes
on
the
cloud
are
using
whether
they're
doing
it
self-managed
or
using
cloud
provider
kubernetes
the
control,
plane,
pods,
run
on
a
single
VM
and
we
use
cloud
provider
block
storage
for
the
etcd
storage,
there's
also
a
boot
disk.
That's
that's
not
shown
here.
B
The
only
part
of
this
architecture-
that's
kind
of
maybe
a
little
different
than
you
normally
see-
is
that
we
have
two
separate
IP
addresses
for
the
VM
and
therefore
for
the
API
server.
There's
a
public
IP
and
a
private
IP,
the
private
IP
for
the
API
server
is
used
for
the
cubelets
and
the
workloads
like
the
pods
that
are
running
on
the
worker
nodes,
to
talk
to
the
API
server
and
the
public
IP
is
used
by
external
clients
like
Cube
control
and
crcicd
system,
to
talk
to
the
API
server.
B
So
the
problem
with
this
architecture
is
that
if
the
control
plane
VM
fails,
then
the
workloads
will
continue
running,
but
the
cluster
essentially
remains
static.
So,
like
the
cluster,
Auto
scaler
can't
scale
nodes
up
and
down,
because
the
cluster
Auto
scalar
runs
on
the
worker
nodes,
but
needs
to
talk
to
the
control
plane
same
thing
with
horizontal
pod,
Auto
scaling,
the
horizontal
pod,
Auto
scaler,
runs
on
the
worker
nodes
and
needs
to
talk
to
the
control
plane.
B
So
if
the
control
plane's
down
it
can't
do
anything
if,
if
a
pod
fails
or
node
fails,
the
Pod
can't
be
rescheduled
to
another
node
and
if
a
pod
fails
or
a
node
fails
and
it's
part
of
a
cluster
IP
service,
then
the
the
endpoints
won't
be
updated
for
any
of
the
services
that
it's
a
member
of.
So
this
is
not
good,
and
so
the
solution
to
that
is
that
use
an
control
plane,
a
highly
available
control
plane
and
the
idea
is
to
replicate
the
control
plane
VMS
across
multiple
cloud
zones.
B
So
the
control
plane
can
continue
functioning
when
there
is
a
VM
or
cloud
provider
Zone
failure.
This
is
a
kind
of
a
picture
of
the
architecture
that
we
use.
It's
similar
to
the
approach
used
by
Cube,
atom,
cops
and
some
other
kubernetes
tools,
except
that
we
have
two
load:
balancers
and
I'm,
going
to
walk
through
each
of
the
components
here
and
then
at
some
point,
we'll
come
to
the
load.
Balancers
and
I'll
describe
what
we're
what
we're
doing
with
those.
B
So
as
I
mentioned,
the
foundation
of
the
control
plane
is
three
replicas
of
the
single
node
control
plane.
So
three
VMS
spread
across
three
Cloud
zones.
Instead
of
just
one
VM
in
one
zone,
the
three
API
servers
run
as
a
stateless
load
balance
service,
where
any
replica
can
handle
any
API
request
and
they're
all
active
at
the
same
time,
and
each
API
server
replica,
reads
and
writes
its
local
at
CD
replica,
which
brings
us
to
etcd.
B
The
scheduler
and
controller
manager
in
this
architecture
are
Master
elected,
so
all
of
the
replicas
are
always
running,
but
only
one
is
doing
work
at
any
given
time.
So
this
is
a
little
different
from
how
the
API
server
worked.
If
you
remember,
I
mentioned
that
the
API
server,
all
three
replicas
are
always
able
to
handle
requests
they're
essentially
like
identical
to
one
another
in
their
behavior,
whereas
the
scheduler
and
controller
manager,
only
one
is
actively
doing
work
at
any.
B
Given
time
and
there's
a
lease
mechanism
that
allows
one
of
the
standby,
schedulers
or
controller
managers
to
take
over
if
the
active
leaseholder
fails
and
then
lastly,
we
replaced
the
public
and
private
IP
interfaces
from
the
single
control
plane.
Vm
in
that
non-ha
setup.
That
I
showed
you
at
the
beginning
with
public
and
internal
multi-zone
load
balancers.
But
the
philosophy
is
still
the
same.
B
So
the
main
failure
modes
that
this
architecture
addresses
is
a
single
VM
failing
or
a
single
Cloud
Zone
failing.
So,
let's
talk
about
what
happens
in
in
those
scenarios,
so
the
load
balancers
will
notice
that
the
VM
has
become
unreachable
and
will
stop
routing
requests
to
that
API
server,
but
they'll
continue
to
Route
requests
to
the
API
servers
in
the
zones
that
are
still
up.
So
those
are
the
green
arrows
showing
the
load,
balancers
routing
requests
to
the
VMS
or
zones
that
are
that
are
still
still
up.
B
Then
one
of
the
other
two
controller
managers
will
become
the
active
controller
manager
and
then,
lastly,
if
the
FCD
leader
was
in
the
failed
Zone,
then
one
of
the
other
two
at
CD
replicas
will
become
the
leader,
and
so
this
setup
brings
the
theoretical
availability
of
the
control
plane
from
Two
and
a
Half
nines
with
a
single
VM
control
plane.
That's
like
the
typical
VM
SLA
from
a
cloud
provider
two
and
a
half
nines
to
four
nines
with
this
control
plane.
B
Now
you
could
tolerate
two
simultaneous
failures
by
running
five
control:
plane
replicas
instead
of
three
but
Cloud
providers
generally
don't
have
five
zones
in
each
region
so
that
wouldn't
help
tolerate
multiple
Zone
failures.
And
so
we
decided
it
really
wasn't
worth
the
the
cost
that
to
have
five
five
replicas
and
the
last
component
that
we
haven't
covered
are
the
load
balancers.
So
if
the
and
what
happens
if
they
fail,
so
if
the
public
load
balancer
fails,
then
external
clients
won't
be
able
to
connect
to
the
API
servers.
B
So,
for
example,
a
client
can't
create
new
workloads
or
roll
out
a
new
version
of
a
workload
or
anything
else
that
external
clients
typically
do
when
they
talk
to
the
API
server,
but
already
running
workloads
will
continue
to
run
and
all
the
dynamic
behaviors
that
I
talked
about
earlier,
like
pod
rescheduling
and
auto
scaling.
Those
will
all
continue
to
work
and
if
the
internal
load
balancer
fails,
then
the
opposite
is
true.
B
In
theory,
you
can
run
the
same
architecture
where
you
spread
the
three
control
plane
nodes
across
regions
and
use
like
a
multi-region
cloud
load
balancer
and
then
tolerate
region
failure,
but
because
our
control
planes
are
per
region.
We
set
this
up
as
replicated
within
a
region
instead
of
across
regions.
B
C
And
then
we
want
to
change
the
architecture
from
Knight
h8
to
the
class.
State
needs
to
be
still
the
original
class
data
in
the
Iha
and
then
the
interface
to
exercise
the
control
plan
should
not
be
changed.
So
basically,
we
want
to
migrate
the
control
plane
and
of
all
the
workload
as
David
mentioned
during
the
whole
migration.
We
wanted
the
workloads
to
keep
at
keep
up
and
running.
We
don't
want
that
workload
get
affected.
In
our
case,
we
have
a
production
cluster
across
60
regions.
C
So
basically
the
required
level
requirements
is
to
migrate
the
control
plane
without
affecting
the
production
workloads.
So
we
defined
the
requirements
as
follows,
so
our
workloads
should
keep
on
running
during
migration
and
no
client
reconfiguration.
Also,
the
micro,
the
migration
across
the
fleet
should
be
automated,
and
this
should
support
both
zero
forward
and
roll
back.
C
C
Also
couplet
from
worker
can
report
no
status
so
that
the
class
date
for
the
node
will
be
mutated.
Meanwhile,
the
port,
the
workload,
can
also
talk.
The
API
server,
like
those
operators,
can
change
the
class
data
as
well,
so
we
can
see
actually
during
the
migration
we
have
control
plan.
We
have
workers
and
then
we
have
workload.
So
basically
we
want
to
protect
the
workloads,
so
that
means
the
class
States
need
to
be
safe.
We
want
to
migrate
that
to
BHA,
but
we
don't
want
to
break
the
workload.
C
So
with
this
keep
in
mind,
we
designed
our
migration
into
three
phases
and
for
each
phase
actually
to
include
multiple
steps
and
I
want
to
Echo
back
a
little
bit.
You
know
what
things
we
want
to
protect
most
again,
so
basically
we
want
to
protect
workload.
That
means
during
the
multiple
step
migration
process,
any
single
step
can
fail
in
that
field.
We
want
that
the
workload
still
be
protected
and
then
it
will
just
roll
back
or
fix
that
and
move
forward
and
do
the
migration
again.
So
then,
here
we
designed
the
three
phases.
C
The
first
phase,
basically
to
get
the
cluster
state
from
the
Niah
control
plan,
in
particular,
is
in
the
ICD.
Meanwhile,
because
we
want
to
protect
workloads,
we
will
shut
down
all
the
traffic
to
the
control
plane.
This
is
similar
as
like
the
downtime
when
we
held
non-ha
control
plan,
the
next
step
we
want
to
migrate.
The
cluster
state
to
the
control
plan
so
basically
like
to
just
we'll
cover
the
details,
so
high
level
ideas,
basically
replica
one
icd23
and
then
somehow
make
it
work
for
the
control
plan.
C
So
here
is
follow
the
same
Principle
as
the
phase
one.
It's
multiple
steps
and
a
single
step
can
fail.
We
want
to
protect
workload,
so
basically
we
will
shut
down
the
traffic
to
the
control
plan
to
make
sure
the
classes
will
not
get
mutated.
So
it's
consistent
workloads
just
keep
on
running.
It's
decorated,
a
little
bit
as
David
mentioned
here.
The
horizontal
scaling
and
the
workout
scaling
cannot
work,
but
the
workload
is
still
keep
on
running
and
the
last
phase.
When
we
confirm
everything
works,
fine,
then
we
will
reopen
the
traffic.
C
C
So
now,
let's
talk
about
each
phase
as
the
first
phase
based
the
high
level
idea.
We
want
to
get
the
cluster
state
from
the
in
a
safe
way,
so
in
the
night,
HX
control
plan
basis.
This
is
a
you
know,
a
diagram
in
a
single
VM.
It
has
ICD,
has
a
couple
control
plan
components
the
first
thing
before.
Actually
we
get
to
the
state,
we
want
to
make
sure
the
state
is
static.
At
that
point,
so
the
first
step
is
we
mute.
C
We
shut
down
all
the
control,
plane,
parts
it
and
then
so
from
the
second
step
for
the
first
phase.
Basically,
we
want
to
build
a
snapshot,
the
input
we
can
use
for
the
second
phase,
because
at
this
moment
workloads
will
not
get
affected
but
just
get
decorative
in
terms
of
the
dynamic
change.
So
here
we
use
one
ICD
utility
tool
to
take
snapshot
from
ICD
data
folder.
While
we
use
the
ICD
control
here,
basically
for
ICD.
Actually
it
has
two
types
of
information.
C
Once
the
data
there
are
actually
the
coral
member
information
when
we
change
from
not
actually
ICD
to
haicd
the
data
is
the
one
we
want
to
keep,
but
the
current
member
information
we
want
to.
You
know
to
change
that
from
one
node
to
multiple
nodes
and
the
next
step
is
we
shut
down
the
the
single
node
control
plane
to
release
IPS,
as
we
mentioned?
Basically
ICD.
The
cluster
state
in
ICT
definitely
is
the
one
we
want
to
keep
and
then
we
use
it
for
the
control
plane.
Also,
the
two
interface,
the
eyepiece.
C
C
C
C
C
So
the
second
phase
is
to
basically
to
bootstrap
the
HF
control
plane,
but
I
want
to
emphasize
again.
Basically
we
will
not
start
the
traffic.
The
high
level
idea
is
to
keep
the
class
state
to
be
consistent
for
the
whole
migration
process,
because
it
will
take,
you
know
from
10
minutes
to
30
minutes
and
every
single
step
can
fail.
C
So
the
first
step
is
to
build
the
three
control
plan
VM
and
then
the
data
disk
is
from
the
snapshot
we
took
before
from
the
you
know:
980
control,
plane
and
then
the
second
step
for
in
this
phase
is
to
build.
You
know
the
too
low
balancer
and
then
the
IP
from
the
for
the
two
low
bouncers
is
the
original
Ip
from
the
nihas
single
node
control
plane.
C
As
a
last
step
now
we
have
the
old
Cloud
resources
right
deployed,
but
actually
ICD,
as
I
mentioned
before.
There
are
two
sides
for
the
ICD.
One
is
the
data,
the
other
is
the
column.
The
data
is
there,
but
even
we
use
the
snapshot
to
create
new
SD
disk.
The
column
cannot
build
yet
so
we
use
the
the
tool
called
LCD
control
restore.
C
So
the
basically
we
start
to
rebuild
the
new
column
with
for
the
three
nodes
till
here
now.
Actually
we
do
have
a
h8
control
plane
keep
up
running,
but
one
thing
is
because
we
didn't
restart
the
Google
controller
manager
and
then
we
didn't
open
the
traffic
from
the
two
load
balancer.
So
basically
the
workload
the
worker
side
can
still
not
talk
to
the
the
control
plan
yet,
but
the
control
plan
itself
is
isolated
up
and
running.
C
So,
as
a
last
phase,
we
will
just
reopen
the
traffic
because
we
confirmed
the
h8
control
plan
is
bring
up
as
we
expected.
So
during
the
whole
migration
process,
we
can
see
in
each
phase.
It
has
multiple
steps
and
the
principle
for
each
phase
basically
is
to
correctly
get
the
snapshot
and
then
to
protect
workloads
by
you
know
only
when
we
are
ready
to
make
sure
the
whole
control
planes
is
operating,
and
then
we
reopen
the
traffic
so
that
this
actually
make
the
low
back
much
easier.
So
before
phase
three,
we
can
just
roll
back.
C
You
know
by
redeploy
the
knowledge
Cloud
resources,
the
IC
disk
SS
class
of
state
is
not
you
know,
muted
at
all,
and
after
phase
three,
let's
say
you
know
we
migrate
to
can
jumpling
after
say
a
couple
days.
The
class
state
will
keep
will
be
mutated.
In
that
case,
we'll
follow
the
same
principle
to
get
the
class
data
from
the
h8
control
plane
and
then
do
a
snapshot
and
then,
from
that
snapshot
to
rebuild
a
new
IC
test
for
the
9h8
control
plane
and
you
know,
build
a
single
node,
ICD
card.
C
So,
lastly,
actually
I
want
to
share
one
lesson.
We
learned
from
one
of
the
outage
we
caused
by
one
inconsistent
class
stated
during
the
migration,
so,
as
I
explained
in
the
previous
migration
process,
we
can
see
to
protect.
The
class
state
is
the
most
important
thing
for
the
whole
migration
process,
especially
keep
being
the
class
that
is
consistent
in
the
phase
three
we
mentioned.
Actually,
we
just
restart
the
controller
manager
and
reopen
the
traffic
actually
during
the
outage.
C
So
in
in
the
last
phase
of
migration,
we
had
one
back
in
the
code,
so
what
we
basically
try
to
restart
code
controller
manager
and
reopen
the
load
balancer
traffic
at
the
same
time,
but
for
some
reason
the
it
took
much
longer
than
we
expected
to
reopen
the
low
low
balancer
traffic
so
that
there
is
one
there
was
one
state.
Coupon
controller
manager
is
up
and
running,
but
the
load
balancer
is
still
closed.
Then
what
will
happen
so
basic
Google
controller
manager
can
talk
to
API
server
to
start
reconcile
resources,
but
for
workers
they
cannot.
C
They
use
the
load
balancer
the
couplets,
using
the
load
balancer
to
talk
to
control
plan
API
server
to
report
its
no
status.
Oh
actually,
something
else
is
not
healthy
because
it
hasn't
reported
for
a
while.
So
the
service
controller
inside
Google
controller
manager
start
get
kick
in
and
then
start
to
remove
some
of
the
nodes
from
the
load
balancer
for
the
services.
C
Actually,
if
we
keep,
we
got
alerted
immediately.
If
we
keep
the
state
longer,
it
will,
like
you
know,
the
some
nodes
will
even
get
removed
from
cluster.
This
is
one
lessons
we
learned
that
the
keep
the
cluster
State
consistent
is
really
important
for
the
migration
home
migration
process,
so
because
the
migraine
process-
you
know
it's
when
we
do
the
migration
across
different
regions,
the
behavior
and
the
cloud
provides
transition.
Failure
could
cause
outage
like
this,
so
from
the
license
we
can
see.
C
Actually
it's
really
important
to
protect
the
class
date
to
be
consistent
during
the
whole
process.
So,
finally,
if
I
use
one
sentence
to
summarize
the
migration
I
would
say
control
plane.
Migration
is
really
about
the
class
State
migration
and
the
keeping
class
State
consistent
is
critical
to
make
sure
workloads
not
get
affected.
B
B
So
if
the
cluster
level
precondition
checks
pass,
then
we
start
the
upgrade
of
the
first
control
plane
node,
we
run
some
precondition
checks
like
making
sure
all
three
control
plane
nodes
are
up.
This
is
important
because
one
of
the
benefits
of
the
control
plane
architecture
is
that
you
can
take
down
one
of
the
nodes
for
a
kubernetes
version
upgrade
and
still
continue
to
serve
user
traffic,
because
you
only
need
two
of
the
nodes
to
be
up
in
order
to
serve
the
the
users.
B
But
if
you
don't
start
off
with
all
three
control
plane
nodes
up,
then,
when
you
take
one
down,
then
you
you
lose
that
because
then
there
will
be
zero
or
one
nodes
up.
So
in
order
to
prevent
users
from
experiencing
downtime,
we
make
sure
all
three
are
up
before
we
do
a
version
upgrade
on
one
of
them.
So
then
we
shut
down
the
VM
that
we're
going
to
upgrade.
We
create
a
new
VM
at
the
new,
with
the
new
kubernetes
control,
plane
version
and
deploy
everything
on
it,
and
then.
B
B
B
So
the
first
approach
we
considered
is
shown
on
this
slide
and
the
idea
was
to
use
the
success
rate
of
actual
API
requests
to
compute
availability.
So
we
have
a
Prometheus
agent
running
in
the
cluster
that
talks
directly
to
the
API
servers
and
scrapes.
The
API
server
request
total
Prometheus
metric
from
the
API
server's
metrics
endpoint.
This
API
server
requests
total
metric
for
those
who
aren't
familiar
with
it.
It
records
the
total
number
of
API
requests
received
by
or
processed
by,
the
API
server
broken
down
by
various
categories,
including
the
HTTP
Response
Code.
B
B
So
the
second
approach
we
considered
was
to
have
the
Prometheus
agent
periodically
just
probe
the
API
server's
metrics
endpoints,
to
see
whether
the
API
servers
are
alive
and
then
we
Define
availability
as
the
number
of
times.
This
probe
gets
a
response
divided
by
the
total
number
of
probes,
and
the
probing
is
done
through
the
load
balancer.
So
it
reflects
the
user
visible
downtime.
But
this
approach
has
a
downside
of
not
reflecting
the
internal
API
server.
Errors
like
the
first
approach
did
so.
The
approach
we
ultimately
chose
is
a
combination
of
the
two.
B
B
If
the
probing
approach
shows
that
the
user
is
not
getting
a
response,
because
two
or
more
API
servers
are
down
or
the
load
balancer
is
down,
and
so
this
way
we
end
up
with
an
availability
number
that
reflects
API
server
errors
and
also
scenarios
where
the
API
servers
are
down
from
the
user
perspective,
so
to
wrap
up
I'll.
Just
Briefly
summarize
how
the
various
aspects
of
our
overall
design
allowed
us
to
meet
the
requirements
that
Kong
mentioned
earlier.
B
So
we
have
the
requirement
that
all
the
workloads
running
on
the
cluster
must
continue
to
run
during
migration
and
the
way
we
accomplished
that
was
to
snapshot
the
single
node.cd
and
clone
the
disk
to
create
the
two
new
replicas
so
that
the
cluster
state
after
the
migration
was
exactly
the
same
as
the
cluster
State.
Before
the
migration,
we
had
the
requirement
that
there
should
be
no
requirement
for
a
client
reconfiguration.
B
We
didn't
want
to
have
to
change
any
of
the
configuration
of
the
cubelets
or
the
workloads
or
external
clients
that
are
talking
to
the
API
server
and
so
to
accomplish
that
we
reuse
the
same
IP
addresses
from
the
VM
in
the
single
node
control,
plane
architecture
as
the
load
balancer
IP
addresses.
So
the
internal
and
external
VM
IP
addresses
became
the
internal
and
external
load,
balancer
IPA
addresses,
and
then
we
had
the
third
requirement
that
the
per
cluster
migration
and
rollback
should
be
automated
and
safe.
B
So
we
accomplished
that
by
having
a
migration
script
that
automatically
rolls
back
upon
failure
by
not
making
any
state
mutations
until
the
control
plane
is
up
and
running.
Kong
talked
a
lot
about
that,
so
that
we
could
do
safe
rollback
and
ensuring
that
the
cubelets
have
heartbeated
before
the
controller
manager
is
re-enabled.
B
Lastly,
we
have
the
requirement
that
the
migration
across
the
fleet
should
be
automated
and
safe,
so
to
accomplish
that,
we
used
a
pipeline
for
migration
and
check
these
preconditions
and
post
conditions
for
each
cluster
before
moving
on
to
to
the
next
cluster
and
I'll
skip
this
since
we're
running
low
on
time.
This
is
just
summarizing.
The
availability
metric
that
we
talked
about
a
minute
ago
and
the
additional
benefit
of
the
h8
control
plane
doesn't
just
give
you
high
availability.
It
also
lets
you
tolerate
sorry.
B
D
Thanks
for
the
talk,
I
have
two
questions.
So
did
you
have
to
over
provision
your
clusters
to
account
for
the
10
to
13
minutes
that
you
guys
were
migrating
and
it
they
were
statically
stable
so
that
you
know
if,
if
more,
if
more
sort
of
traffic
comes
in
you
you
can
handle
that
load.
The
second
one
was:
did
you
change
the
instance
types
as
you
moved
from
single
node
to
cluster.
C
E
C
C
Oh
yeah,
so
for
control
plan.
Basically
it's
a
CPU
Spike
and
then
for
the
Knight
control.
Actually,
it's
a
more
CPU
Spike
than
the
h8
control
plane.
So,
as
the
first
phase,
we
didn't
change
the
instance
Tab
and
then
now
because
the
h8
control
has
been
loaded
for
all
production
clusters,
we
are
looking
at
tuning
the
instant
actually
together
download
a
bit.
E
Both
for
the
nation,
reusing
IP,
address
both
for
the
public
side
and
the
private
side,
and
there
are
load
balances
that
that
have
you
put
three
IP
addresses
instead
of
one,
how
did
you
manage
those
scenarios
where
the
load
balancer
would
require
an
IP
address
per
Zone,
which
means
it's
not
one,
it's
three.
How
did
you
cover
for
such
situation?
Did
you
have
such
situation?
And
if
you
did,
how
did
you
come
up
with
sausage.
C
Yeah
I
think
it's
slightly
different
from
each
Club
Rider
in
terms
of
whether
a
load
balancer
can
should
have
multiple
IP
in
a
single
zone
or
cross.
It's
just
single
IP,
Coral
different
zone
for
AWS
and
Azure
is
a
single
IP,
and
then
you
can
deploy
a
low
balance
across
different
zones
for
AWS.
It
requires
a
different
IP
in
different
Zone,
but
for
the
lb.
Is
it
a
little
bit
implementation
detail?
It
has
a
zoom
known
DS
and
the
original
DS
for
us,
because
the.