►
From YouTube: Multi-Cluster Stateful Set Migration: A Solution To Upgrade Pain - Peter Schuurman & Matt Schallert
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2023 in Amsterdam, The Netherlands from April 17-21. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Multi-Cluster Stateful Set Migration: A Solution To Upgrade Pain - Peter Schuurman, Google & Matt Schallert, Chronosphere
As more stateful workloads like Redis, Kafka, or custom DBs are migrated to Kubernetes, what operational paradigms need to change to support moving state across clusters and maintaining availability during migration? How do admins safely and reliably perform Day 2 operations and maintenance events while protecting the data and state of the app? What visibility is needed? Today, cluster administrators design complex workflows for data replication, pod and persistent volume migration, and state management for Day 2 ops. What if there was a way to seamlessly migrate StatefulSets between node pools or across clusters to simplify problems related to upgrades, workload migration, and stretching clusters? The speakers will demonstrate the complex patterns developed at Chronosphere to safely migrate stateful workloads to coordinate maintenance operations for thousands of pods across multiple zones and regions. They will then discuss a new enhancement to Kubernetes called StatefulSet Partition which is integrated into a multi-cluster deployment like Chronosphere's and how this can dramatically simplify their operations to focus instead on core business logic.
A
A
So
a
little
general
idea
of
what
we're
going
to
be
talking
about
today,
I'm
going
to
give
you
a
bit
of
context
on
how
we
use
kubernetes
at
chronosphere,
some
of
the
use
cases
and
challenges
we
have
with
cross
cluster
stateful
workloads.
Peter
is
going
to
tell
you
about
some
of
the
work
going
on
in
open
source
kubernetes
to
make
these
use
cases
possible
and
we're
going
to
show
you
a
demo
of
how
it
all
ties
together.
A
So,
first,
to
give
you
a
little
bit
of
context
about
what
chronosphere
does
and
how
we
use.
Kubernetes
chronosphere
is
a
software
as
a
service
observability
platform,
particularly
built
for
high
scale,
use
cases
in
cloud-native
environments
and
given
how
Mission
critical
observability
is,
we
have
a
really
high
SLA
and
we
take
reliability
incredibly
seriously
to
meet
that
SLA.
So,
although
we
have
three
nines
in
our
SLA,
we
aim
for
four
nines
and
have
achieved
four
nines
in
production.
A
Part
of
the
way
that
we
achieve
these
reliability
and
fault
tolerance
needs
is
by
running
on
top
of
kubernetes,
so
our
kubernetes
footprint
spans
multiple
regions
with
thousands
of
kubernetes
nodes
in
total
and
over
40
000
pods.
In
total.
Each
of
these
clusters
were
on
a
mix
of
stateless
and
stateful
workloads,
but
the
largest
stateful
workload
is
our
metrics
data
store,
which
I'm
going
to
tell
you
more
about
so
the
architecture
of
our
metrics
data
store
really
heavily
influences.
A
How
we
operate
it,
how
we
deploy
about
it
or
sorry
how
we
operate
it,
how
we
think
about
it,
deploy
it
and
kind
of
architect
our
clusters
for
it.
So
our
metrics
data
store
is
based
on
M3,
which
is
an
open
source.
Metrics
engine
and
Metric
datastore
clusters
are
deployed
as
three
separate
stateful
sets,
each
in
a
separate
Zone
and
each
having
the
full
copy
of
the
data.
So
data
is
sharded
and
started
across
multiple
nodes
in
a
stateful
set,
and
then
each
staple
set
owns
an
entire
copy
of
the
data.
A
All
reads
and
writes
are
done
through
client-side
Quorum.
So
when
you
go
to
write
data
as
long
as
to
the
three
nodes
that
own
that
Shard
acknowledge
the
right
and
persisted
on
disk,
then
we
consider
the
right
successful,
a
similar
pattern
for
reads,
but
because
all
this
Quorum
is
done
client-side,
it
means
that
the
clients
have
to
be
aware
of
each
unique
database.
Node
in
the
cluster
have
to
be
able
to
open
a
connection
to
it
and
have
to
be
able
to
individually
address
each
database
node.
A
A
So
we
have
a
number
of
kind
of
robust
operational,
workflows
and
general
day-to-day
workflows
that
are
uniquely
aware
of
the
databases,
architecture,
its
replication
strategy
and
fault
yeah
and
Quorum
requirements.
So,
for
example,
things
like
moving
a
database
cluster
between
node
pools,
coordinating
resizing
of
the
underlying
storage
or
upgrading
the
database
cluster.
A
Okay,
and
in
terms
of
why
you
might
want
to
do
this,
why
you
might
want
to
move
a
staple
workload
between
clusters?
There
can
be
a
number
of
reasons.
A
lot
of
organizations
will
start
off
with
just
one
or
two
production.
Kubernetes
clusters
and
over
time
have
to
split
up
those
workloads.
So
maybe
your
cluster
has
maybe
your
kubernetes
cluster
has
grown
too
large
and
you're
worried
about
control,
plane,
scalability
or
maybe
you
just
want
to
reduce
the
blast
radius
of
any
one.
A
You
might
also
have
to
move
workloads
from
one
region
to
another,
so
maybe
you
need
some
data
to
be
in
a
specific
region
for
compliance
or
data
severity
purposes,
or
maybe
you
just
want
to
move
some
data
closer
to
your
users.
Maybe
you
have
a
large
user
segment
in
a
new
geography
and
you
want
your
data
to
be
as
close
to
them
as
possible.
A
Changing
your
cni
implementation
and
kind
of
doing
that
on
a
live
cluster
can
be
like
changing
the
engine
of
a
plane,
mid-flight
and
kind
of
terrifying,
and
if
you
find
yourself
in
the
situation
of
wanting
to
move
workloads
between
clusters
for
stateless
workloads,
there
are
a
number
of
solutions
that
the
community
has
come
up
with,
that
make
moving
workloads
between
clusters
easier.
The
most
notable
ones
are
multi-cluster
services
and
multi-cluster
ingresses,
both
of
which
are
implemented
by
multiple
Open
Source
service
meshes
as
well
as
cloud
provider
implementations.
A
So
if
you're
running
a
purely
stateless
workload,
then
you
can
just
bring
up
a
new
kubernetes
cluster,
bring
up
your
workloads
in
that
cluster
and
then
shift
and
then
shift
traffic
over
to
them
using
one
of
these
tools,
but
for
stateful
workloads.
The
story
isn't
really
as
clear
and
for
our
workload
specifically
given
the
requirement
of
the
client
having
to
address
each
database
node.
We
can't
just
put
a
load
balancer
in
front
of
a
staple
set
and
then
move
them
over
and
call
it
a
day.
B
So
I'm
going
to
talk
about
some
of
the
challenges
you
may
face
with
performing
across
cluster
migration.
So
we
have
the
complexity
of
managing
multiple
kubernetes
control
planes.
If
you're
running
in
a
single
kubernetes
cluster,
you
have
a
bunch
of
invariants
that
your
API
server
will
enforce.
You
have
uniqueness
guarantees
per
pod.
You
have
at
most
one
semantics
of
a
staple
set
and
those
are
enforced
through
API
server.
Now,
if
we're
dealing
with
migrating
a
logical
application
across
clusters,
if
you're
dealing
with
two
API
servers,
so
you're
managing
State
across
two
API
servers.
B
B
B
First,
one
is
clients
need
access
to
updated
application
endpoints.
So,
as
Matt
mentioned,
clients
of
m3db
need
to
be
able
to
uniquely
address
the
replicas
that
are
running
and
so
for
this
you
need
to
have
a
way
for
clients
to
discover
replicas,
as
they
turn
up
in
a
new
cluster,
so
usually
clients
will
communicate
through
a
cluster
IP
or
balancer
or
through
a
headless
service
within
the
cluster.
B
B
So
in
a
single
cluster
you
know
there
is
peers,
are
assigned
with
the
same
DNS
endpoint.
The
name
is
consistent,
even
though
an
IP
address
might
change
as
Pond
life
cycle
changes
and
DNS
updates
are
almost
instant
thanks
to
cube
proxy,
but
across
clusters
replicas,
even
though
you're
they're,
uniquely
addressable,
they
may
not
have
consistent
DNS
naming
based
on
your
service
measure
of
choice
and
the
fully
qualified
domain
name
might
not
be
consistent.
B
There
may
be
some
changes
that
we
need
to
do
to
use
a
different
peer,
Discovery
endpoint,
but
we
don't
want
to
actually
have
to
modify
the
application
significantly
before
we
incorporate
this
cross
cluster
move
because
that's
just
a
bunch
of
extra
engineering
that
we
don't
want
to
do
so.
That's
another
constraint
of
the
system.
B
The
other
challenge
here
is
storage,
so
we
have
a
typical
applications,
may
use
I
guess
for
read,
write
many,
that's
fairly
simple,
because
we
have
application
storage
that
may
be
accessible
across
both
clusters,
just
through
a
network
endpoint
that
stays
consistent.
But
if
we're
using
read
write
once
we
need
to
make
sure
that
our
storage
references
are
able
to
be
shared
across
clusters.
B
B
So
we
need
to
have
a
way
to
have
this
data
be
accessible
across
both
clusters,
or
at
least
this
data
reference
and
within
a
single
cluster.
Pvs
are
a
global
resource,
but
now
that
we
are
kind
of
breaking
out
of
the
cluster
and
going
outside
of
the
cluster
boundary,
we
need
to
still
enforce
this
invariant.
So
that's
another
challenge
we
have
to.
We
have
to
handle
so
yeah
API
server
isn't
enforcing
the
PVE
to
PVC
uniqueness.
So
we
need
the
orchestration
layer
to
handle
that
something
that's
outside
the
cluster.
B
B
So
there's
a
number
of
things
we
need
to
think
about
number
one
is
replicas
need
to
follow
storage.
So,
if
we're
using
Dynamic
provisioning,
if
we
simply
scale
down
replicas
in
the
source
cluster
and
bring
them
up
in
a
destination
cluster
and
we're
using
dynamic
revisioning,
we
may
get
volumes
provision
that
we
don't
want
and
really
we
want
to
have
our
data
layer
be
migrated.
B
So
storage
replicas
need
to
follow
storage
and
we
need
to
bring
up
storage
in
the
new
cluster
so
that
our
application
can
access
it
in
the
in
the
destination
cluster.
The
other
challenge
is
POD
disruption
budget.
So
how
do
we
make
sure
that,
during
a
migration,
any
infrastructure
changes?
Maybe
a
cluster
update
or
maintenance
events,
don't
affect
the
application
and
don't
disrupt
the
budget
of
the
global
application?
That's
kind
of
logically
running
across
both
clusters.
B
So
we
need
some
way
to
enforce
a
pod
disruption
budget
or
maintain
a
certain
budget
of
availability,
while
we're
migrating
and
the
other
challenge
is
with
network.
So
we
touched
on
that.
We
need
some
way
to
have
clients
be
able
to
connect
and
pure
is
able
to
connect
to
other
peers,
but
the
mechanism
that
we're
using
as
I
mentioned,
you
know
in
Cube
Cube
proxy,
is
very
quick.
B
You
know
we
can
update
a
DNS
endpoint
obvious
and
pod
IP
will
get
reflected
there,
but
if
we're
using
a
service
mesh
that
spans
across
clusters,
our
DNS
updates
might
not
propagate
as
quickly
as
Q
proxy.
So
we
need
some
way
of
signaling
to
our
orchestration
system
that
maybe
a
DNS
endpoint
isn't
ready
to
be
served
for
clients,
and
we
need
to
wait
for
that
to
happen.
So
that's
another
thing
we
need
to
be
aware
of.
The
other
challenge
is
with.
B
If
we're
using
an
operator,
the
an
operator
typically
will
reconcile
to
a
particular
state.
So
if
we
have
some
orchestration,
that's
sitting
outside
of
a
cluster,
that's
moving
replicas
over
in
a
staple
set
we
need
to
have
the
operator
remain.
In
sync,
with
what
we're
doing,
if
we
simply
you
know,
move
replicas
from
a
stapler
set,
an
operator
will
try
to
reconcile
back
to
its
specified
state.
B
So
I'm
going
to
talk
about
some
of
the
some
of
the
ways
in
open
source.
We
can
maybe
tackle
these
problems,
so
the
first
building
block
I'm
going
to
introduce
is
multi-cluster
services,
and
so
this
is
kep
1645,
it's
introduced
in
2020
and
it
provides
a
specification
for
cross-cluster
domain
naming
so
the
benefit
we
get
here.
B
So
if
we
query
the
discovery
service
similar
to
a
headless
service
within
the
cluster,
we
can
discover
all
of
the
peers
that
are
behind
this
multi-cluster
headless
service
and
then
for
client
networking,
as
Matt
mentioned
m3db,
as
this
constraint
of
clients
needing
access
uniquely
to
replicas,
multi-cluster
Services
provides
us
a
way
to
uniquely
address
these.
These
endpoints
as
long
as
they're
qualified
by
the
the
cluster
ID
that
these
replicas
are
in.
B
The
other
building
block
I
want
to
introduce,
is
staple
set
slices.
So
this
is
a
new
cap
that
we're
working
on
for
126
and
we're
targeting
Alpha,
but
at
its
core
it's
a
way
of
segmenting
a
staple
set
into
a
subset
of
of
replica
IDs.
So
it
enables
you
to
take
a
staple
set
of
say,
like
n,
replicas
and
split
it
into
two
to
have
like
a
a
set
that
you
can
scale
down
in
a
source
cluster
and
scale
up
in
a
destination
cluster
in
a
complementary
fashion.
B
So
one
of
the
challenges
of
the
staple
sets
today
is
you
know
they
they,
if
you're
using,
ordered
ready,
they
scale
from
zero
to
n
and
they
scale
down
from
n
to
zero.
If
we
simply
scale
down
in
cluster
a
and
then
try
to
scale
up
in
cluster
B
we're
going
to
be
scaling
up
from
zero,
and
so
we
have
an
overlap
of
replicas.
But
with
this
cap
we're
able
to
kind
of
provide
more
granular
control
over
those
replica,
ordinals
and
scale
in
from
replica
n
and
down
to
zero
in
the
destination
cluster.
B
So
how
do
we
tie
these
together
to
build
a
solution
from
for
cross-cluster
migration?
So
we
need
to
coordinate
the
building
blocks.
We
need
to
leverage
multi-cluster
endpoints
or
incorporate
a
peer
Discovery
protocol
that
allows
for
communication
across
these
clusters.
We
need
applications
to
provide
signals
that
they're
healthy,
whether
that's
the
application
finding
Quorum
from
its
peers.
B
If
DNS
propagation
is
an
issue
making
sure
that
the
application
is
able
to
notify
us
when
it's
client
endpoints
are
ready,
we
have
to
make
sure
that
operators
can
coexist
with
application
orchestration
and
we
need
to,
if
you're
using
say,
like
a
CI
CD
pipeline
for
managing
your
staple
sets.
You
need
some
way
to
signal
to
that,
so
that
you
don't
have
contention
between
migration
and
in
your
git
Ops
github's
workload,
and
then
we
also
need
to
migrate
our
dependencies
over.
B
B
So
we're
going
to
Showcase
a
demo
here
of
m3db
migration
and
some
context
on
this
demo.
So
we're
running
m3db
across
three
zones,
so
we
have
three
staple
sets
and
we're
going
to
migrate
these
one
by
one.
So
one
percent,
and
in
this
demo
we're
using
multi-cluster
services
on
gke
for
the
orchestration
of
these
staple
sets
we're
using
cap
335.
So
it's
a
modified
kubernetes
cluster
that
has
this
enhancement
and
then
to
move
the
migration
over.
B
Both
of
these
clusters
are
running
in
the
same
zone
so
the
same
or
the
same
region,
so
they
have
the
same
zono
footprint.
So
as
you
as
you
migrate,
our
storage
reference
is
over
we're
still
referencing
the
same
underlying
persistent
disc,
and
so
as
long
as
our
compute
resources
match
the
storage
resources,
our
replicas
are
able
to
come
up,
as
our
storage
gets
attached
to
the
Computer
Resources
in
the
new
cluster.
So
we're
going
to
roll
this
demo
here
and
I'm
just
going
to
through
it
as
we
go.
B
So
we're
just
kicking
off
this
with
a
crd,
so
on
the
left
here
we
have
our
source
cluster
and
that's
running
a
3db
on
the
right.
We
have
a
destination
cluster
that
doesn't
have
anything
in
it
and
so
orchestration
just
kicked
off
and
it
started
scaling
down
the
source,
staple
set
and
bringing
up
the
destination
sample
set
and,
as
I
mentioned,
replicas
need
to
follow
storage.
So
the
first
thing
we
did
was
migrate:
the
storage
references
so
taking
those
PV,
PVC
config
map
dependencies
and
moving
those
over
to
the
destination
cluster.
B
Once
that
happened,
we
were
able
to
create
a
new
staple
set
that
referenced
one
of
these
one
of
one
of
the
replicas
that
we
wanted
to
migrate
over.
So
we
migrated
over
one
of
the
zones.
If
this
staple
set
had
more
replicas,
we
would
scale
in
the
highest
replica
first,
but
we're
just
moving
over
staple
sets
of
replica
size
one
in
order
to
fit
it
on
the
screen
here
and
at
the
top.
We
have
the
commit
log
traffic
for
our
m3db
application.
B
So
we
can
see,
as
replicas
are
scaled
down
our
traffic
scales
down
to
that
particular
replica,
and
we
start
to
see
as
DNS
propagates
we
can
see
there.
Traffic
starts
to
scale
up
on
the
destination
cluster,
and
so
I've
sped
up
the
demo
and
cut
out
kind
of
the
boring
sections
where
we're
waiting
for
our
DNS
names
to
propagate
to
our
clients,
but
in
effect
we're
able
to
Showcase
how
that
scale
is
up
and
scales
down
and
clients
are
able
to
get
updated
endpoints
as
we're
migrating
over.
B
This
is
a
little
glitch
in
Q
proxy
I
just
had
to
restart
it
here,
but
our
applications
back
up
and
we'll
just
wait
for
the
final
instance
to
come
healthy
and
then
that
should
be
the
end
of
the
demo
there.
So
the
other
thing
to
mention
is
for
for
Readiness.
What
we're
using
here
is
POD
Readiness
gates
to
to
note
that
a
pot
is
ready
and
we
can
tie
in
the
network
piece
here
as
well.
B
Okay,
so
what's
next
so
safety,
that's
one
thing:
I
didn't
really
touch
on,
but
I
did
mention
as
a
constraint.
We
need
to
make
manage
budget
across
our
clusters,
so
making
sure
we've
protected
our
application
across
clusters.
We're
modifying
the
Pod
reception
budget
in
a
distributed
way
so
that
both
clusters
have
a
global
view
of
budget
and
we
maintain
budget
if
there's
any
maintenance
events
increasing
the
speed
so
aligning
our
unavailability
budget
with
failure.
Domains
in
this
case
we're
moving
one
zone
at
a
time
because
that's
kind
of
the
constraint
of
m3db.
B
We
need
two
out
of
three
zones
up
in
order
to
handle
rights,
but
we
can
speed
this
up
based
on
failure
domains.
Maybe
we
take
one
zone
down
at
a
time
or
maybe
we
can
handle
a
subset
of
replicas
with
within
a
single
zone
and
we
want
to
have
a
Max
unavailable
per
Zone
that
we're
managing
at
a
time
data
flexibility
so
being
able
to
move
data
across
regions.
B
You
know
this
demo
just
showed
moving
across
a
single
zone,
we're
still
referencing
the
same
underlying
disk,
but
you
can
imagine
you
know
instead
of
using
that
same
disk
reference
rather
than
just
using
it
directly
and
we
if
we
want
to
migrate
across
across
zones,
we
could
replicate
that
underlying
storage
to
another
Zone
and
then
bring
up
a
new
replica
on
the
other
zone.
So
there
is
some
additional
challenges
here
in
terms
of
network
latency
if
we
have
the
other
cluster
in
another
region
during
the
migration.
B
So
you
know
this
would
probably
want
you
probably
want
to
do
this
only
during
a
period
of
lower
traffic
during
a
regular
maintenance
window
and
then
finally
operator
compatibility
so
supporting
General
operators.
So
for
this
demo
we
made
some
changes
to
make
the
m3db
operator
multi-cluster
aware,
but
we
do
want
to
make
some
improvements
in
open
source
to
kind
of
have
a
specification
for
multiple
op,
like
generic
operators
kind
of
tie
into
to
handle
this
operation.
So
how
do
we
signal
between
orchestration
systems
that
are
outside
of
clusters
to
operators
within
clusters?
B
A
C
B
Yeah
so
for
dependencies
like
in
this
example,
m3db
has
a
dependency
on
hcd,
and
so
in
this
scenario,
we
configured
at
CD
to
be
multi-cluster
to
be
behind
a
model.
Cluster
endpoint
and
m3db
was
communicating
with
NCD
through
its
multi-cluster
domain
name,
and
so
as
m3db
migrated
SCD
remained
in
the
source
cluster.
D
Hey
I
have
a
question
here
all
right,
oh
by
the
way,
thanks
for
the
great
talk,
so
one
of
the
things
that
I
wanted
to
understand
is
how
do
we
kind
of?
Are
there
today
Provisions
to
cut
off
traffic
for
stateful
workloads,
in
the
sense
that,
if
I
wanted
to
ensure
that
in-flight
lights
are
persisted
to
the
backing
store,
are
there
hooks
to
ensure
that
those
are
flushed
before
we
kind
of
move.
B
D
A
D
There
a
way
to
ensure
that
rights
on
the
Final
Destination
store
are
complete.
Is
there
a
way?
Is
there
some
mechanism
to
ensure
that
that
is
notified
before
the
orchestrator
kind
of
kicks
off
and
moves
it
over
to
the
other
cluster?
What
kind
of
mechanisms
do
we
have
on
the
on
the
you
know
the
kubernetes
control
plane
too?
B
Yeah,
so
if
we're
confirmative
concerned
about
data
consistency
from
a
single
replica
yeah,
one
thing
we
can
do
is
make
sure
that
our
application
has
the
right,
Primitives
and
built-ins
to
make
sure
we're
flushing
that
data
to
our
rewrite
once
disk
as
it's
being
terminated.
So
that
means
setting
up
appropriate
graceful
termination
Windows
so
that
we
can
gracefully
terminate
and
flush
our
rights
to
that
disk
before
we
actually
migrate.
The
replicas.
A
Also
in
this
example,
the
workload
migration
controller
gives
feedback
to
any
operator,
that's
observing
it
about
where
you
are
in
the
migration.
So
if
you
build
your
application
to
kind
of
be
aware
of
when
it's
being
migrated
or
when
the
thing
that
your
operator
is
controlling
is
being
migrated,
then
it
doesn't
just
have
to
be
this
opaque
migration.
You
can
kind
of
inform
your
application
and
you
know,
like
you,
said,
shut
it
down
to
ensure
that,
like
all
rights
have
been
flushed
or
something
before
it
moves.
B
Yeah,
so
that
that's
one
of
the
challenges
I
mentioned
that
these
invariants
that
are
enforced
and
single
cluster
today
they're
not
enforced
across
clusters.
So
whatever
is
doing
your
migration,
so
whatever
is
orchestrating
your
migration
outside
of
the
Clusters
needs
to
make
sure
that
those
invariants
are
logically
kept,
even
though
there's
no
way
to
enforce
it
through
a
pi
server.
B
So
yeah
we're
kind
of
dealing
with
like
a
split
brain
scenario,
because
there's
two
attached
attached
controllers
running
in
two
separate
control
planes.
So
it's
really
on
the
the
orchestrator
to
make
sure
that
we've
safely
shut
down
a
pod
and
that
disk
is
no
longer
attached
to
that
VM.
So
there
we
do
need
to
make
sure
that,
like
our
signaling
is
appropriate
for
termination
State
when
a
pod
is
actually
brought
down
before
we
bring
it
up
in
a
new
cluster.
A
E
Just
on
the
same
note,
I
have
another
question
and
stateful
set
if
your
application
is
so
close
to
attached
to
this,
in
this
case
is
a
DB
and
we
are
not
going
for
cluster
migration.
We
are
going
from
blue
green
for
that
specific
DB
schema
change.
How
we
will
do
that
because
existing
DB
is
still
supporting
and
we
need
to
spin
up
and
we
need
to
validate
the
new
DB
schema.
Is
that
something
you
have
thought
about?
Yeah.
B
B
So
I
think
that's
it's
not
something
that
we
showcased
in
this
demo
and
I
think
it
can
be
done
under
the
right
safety
protocols,
but
it
certainly
is
a
bit
of
a
challenge
because,
usually
when
you
migrate
a
database
schema
to
an
updated
version,
you
can't
roll
back
say
for
MySQL,
you
update
minor
version.
You
can't
roll
back
to
the
previous
minor
version.
B
It's
just
not
supported
so
I
I
think
that
in
to
ensure
safety
and
kind
of
like
isolate
some
of
the
changes
to
your
system,
it's
better
to
do
these
updates
separately
so
that
you're
kind
of
minimizing
this,
like
the
benefit
of
moving
across
clusters,
is
you
can
you
can
you
can
roll
back
if
we
have
the
same
database
schema
across
clusters?
But
doing
this
doing
this
upgrade
does
kind
of
introduce
risk
and
kind
of
makes
the
the
rollback
option
no
longer
possible.