►
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Handling Risky Business: Cluster Upgrades - Puneet Pruthi, Lyft
Have you ever had to upgrade your Kubernetes clusters to update to a new release version, push new features or patch critical security vulnerabilities? Did it ever feel daunting to live update API masters or etcds? Can you automate such an operation? We hope to share our musings at Lyft in solving the complexity of automating cluster upgrades and how that is incorporated into the design for - k8srotator - a Kubernetes custom controller. As multiple components operating in cohesion make a cluster healthy, there are numerous points of failure that can occur during an upgrade cycle. Although there are varied ways of operating a Kubernetes cluster, the issues encountered during the process are common. Attendees will walk away with knowledge about different cluster upgrade failures scenarios and ways to automate such operations without being in constant fear of losing the cluster state.
https://sched.co/UaYz
A
Good
morning,
everyone
thanks
a
lot
for
joining
in
my
name
is
penny,
and
today
I'll
be
talking
about
how
to
handle
the
risky
business
of
doing
your
cluster.
Your
cluster
upgrades
a
bit
of
introduction
about
myself.
My
name
is
Pune
I'm.
The
engineering
manager
of
ARCA
station
team
at
lift
for
the
last
two
years.
I
have
been
tinkering
with
the
orchestration
and
infrastructure
things
at
lyft.
So
if
you
didn't
get
a
ride
on
time
yesterday,
it
was
probably
me
not
the
rain.
A
So
just
a
structure
around
how
we're
gonna
do
things
today,
you're
gonna
help
define
the
problem
that
we're
trying
to
solve
we're
of
doing
kubernetes
cluster
upgrades
sort
of
putting
the
puzzles
of
the
pieces
of
the
puzzles
together.
How
we
want
to
solve
this
problem
at
lyft
gonna
do
a
deep
dive
on
the
solution
that
we
bought
about
at
left
around
this
and
the
state
of
things
as
they
are
for
this
tool
today,
just
an
overview
of
kubernetes
at
lyft.
We
basically
are
running
our
kubernetes
clusters
from
scratch,
running
them
on
AWS.
A
We
don't
use
eks,
but
are
building
our
own
packer
am
eyes
and
are
a
firm
believer
in
immutable
infrastructure.
These
nodes
have
inbuilt
kubernetes
tooling
in
them
that
way,
when
the
bootstrap
themselves,
they
can
do
self
discovery
for
the
cluster
as
well
as
we
become
a
part
of
the
cluster
themselves.
This
helps
us
get
a
good
amount
of
grasp
over
the
control
and
the
data
plane,
and
we
are
essentially
using
the
AWS
auto
scaling
groups
to
help
us
vertically
scale.
Our
cluster
sizes
in
times
of
need.
A
How
that
paves
in
into
our
predation
model
is
the
fact
that
if
ever
we
have
to
upgrade
our
kubernetes
clusters,
we
basically
do
our
bug,
fixes
our
feature
set
ruling
build
new
patter
a.m.
eyes
and
then
update
the
launch
configuration
that
goes
along
with
the
auto
scaling
groups.
We
then
terminated
the
nodes,
one
by
one,
making
sure
that
the
auto
scaling
group
boots
the
next
set
of
updated
nodes.
They
are
brought
up
with
the
new
and
updated
launch
configuration
they
become
a
part
of
the
cluster.
A
A
The
first
kubernetes
clusters
were
born
and
we
started
playing
around
with
it
and
we
were
essentially
not
serving
any
production
traffic.
We
were
trying
to
understand
the
technology
trying
to
trying
to
develop
tooling
that
could
get
us
at
parity
with
our
legacy
infrastructure
and
so
upgrading
the
cluster
wasn't
really
an
operational
load
on
us.
We
could
upgrade
it
at
our
own
time.
Clustered
health
could
be
bad
and
nobody
cared
at
all
fast
forward
a
year
from
then
on.
A
We
had
built
upon
a
huge
amount
of
instrumentation
to
get
our
clusters
at
parity
with
our
legacy
infrastructure,
and
so
we
are
running
redundant,
multi
AZ
clusters.
We
had
a
ton
of
our
observability
start
running
in,
and
we
wanted
to
now
maintain
both
good
service
health
and,
at
the
same
time,
always
have
a
cluster
health
being
maintained
throughout
the
cluster
upgrade
cycle
and
with
the
work
flow
that
we
were
doing.
A
This
essentially
became
a
huge
amount
of
load
and
burden
on
the
person
who
was
doing
the
operations
for
the
cluster
upgrades,
and
we
started
up
improving
this
by
using,
like
we
began
with
this
doing
just
the
cluster
upgrades
on
an
AWS
console,
but
that
only
takes
you
so
far
as
soon
as
the
cluster
sizes
they
blow
up.
You
have
a
lot
of
things
to
make
sure
that
the
cluster
is
fine.
A
During
the
entire
course
of
an
upgrade
cycle,
you
need
to
make
sure
what
what
defines
your
cluster
health,
what
defines
your
service,
health
and
all
of
those
are
maintained
throughout
the
entire
process.
We
started
working
with
a
Python
tooling
that
helped
us
get
rid
of
the
AWS
console,
but
that
still
involved
a
lot
of
human
interaction
when
you're
doing
the
cluster
upgrades.
Basically,
that
tooling
wasn't
really
intelligent.
A
It
was
just
a
way
for
you
to
circumvent
going
through
the
AWS
console
so
the
beginning
of
this
year,
when
we
had
a
hyper
growth
of
the
migrations
from
our
legacy
stack,
we
was
trying
to
think
of
a
solution
of
upgradation.
Why
do
we
want
to
upgrade
in
the
first
place?
There
are
numerous
reasons
why
one
person
right?
Why
why
one
might
want
to
upgrade
the
cluster's,
the
basic
one?
A
It's
basically
a
way
to
draw
out
your
infrastructure
changes,
any
bug
fixes
and
even
things
around
data
retention,
where
you
don't
want
your
nodes
to
be
lying
around
for
that
long
after
we've
decided
why
we
need
to
upgrade
depends
on
what
you
need
to
upgrade
could
be
the
entire
cluster.
In
the
case
of
like
versioning
updating
your
version
for
Goo
minute
kubernetes.
In
other
cases,
it
could
be
only
related
to
the
nodes
which
are
either
buggy
or
they
are
specific
to
a
particular
AZ,
and
so
it
varies
with
the
use
case.
A
What
you
want
to
upgrade,
but
that
in
mind
we
said
to
define
a
set
of
rules
for
us
to
set
a
golden
standard
for
what
an
ideal
tooling
would
look
like.
We
would
essentially
a
want
always
the
fact
that
we
can
update
our
clusters
in
any
time
you
want
it
could
be
during
business
hours.
It
could
be
otherwise,
but
essentially
we
didn't
want
the
operational
load
to
go
on
someone
to
do
this
at
1:00
a.m.
in
the
night
on
a
Friday
evening
and
whatnot,
we
should
be
able
to
update
our
clusters
whenever
we
want.
A
We
should
always
be
able
to
maintain
the
P,
95
and
99
latencies
for
our
services.
That's
priority,
numero
uno.
We
should
also
be
always
able
to
maintain
the
cluster
health
throughout
the
cluster
grade
process.
We
wanted
the
tooling
to
be
server
less,
because
we
we
didn't
want
a
constant
interaction
with
the
operation
of
with
the
operator,
who
is
trying
to
update
the
cluster
all
the
time
making
sure
either
the
cluster
is
updated
or,
if
he's
going
through
the
right
set
of
nodes.
A
While
that
was
the
case,
we
also
wanted
to
make
sure
that
there
is
a
way
for
the
operator
to
override
that.
So
we
wanted
to
automate
much
of
the
process
that
you
would
want
to
do
during
an
upgrade,
however,
still
have
a
way
to
override
the
entire
functioning
of
the
tooling.
Given
we
are
running
into
an
incident
or
things
are
going
wonky.
A
So
when
we
set
out
to
find
about
what
kind
of
solutions
do
you
want,
we
have
to
look
at
the
landscape
around
the
two
links
that
existed,
because
we
are
not
running
eks
or
gke.
Those
two
links
don't
really
make
cents
and
for
things
like
cube,
ATM
and
cops,
it's
more
so
ingrained
into
how
you're
building
your
clusters
as
well.
So
anyone
who
is
doing
custom
kubernetes
clusters
set
up
either
of
these
tools,
don't
really
work
for
your
use
case
and
so
keeping
the
ideals
of
what
an
ideal.
A
Sorry,
keeping
the
keeping
the
goals
of
the
ideal
tooling
what
it
would
look
like.
We
started
up
with
building
this
tooling
in-house
and
we
wanted
to
follow
the
principles
of
keeping
it
simple
and
kept
keepin
it
hydrated,
because
we
knew
our
clusters
would
grow
in
complexity
over
time,
how
the
tooling,
within
our
infrastructure,
when
what
define
cluster
health
would
always
be
increasing
in
complexity
as
the
time
went
by
so
the
heart
of
it.
What
does
it
look
like?
It's?
A
Basically,
it's
not
as
complex
of
an
algorithm
that
we
wanted
to
implement,
but
the
way
it
is
implemented
gets
a
lot
more
customized
and,
according
to
the
use
case
that
you
want
so
essentially
what
we
want
to
do
is
we
want
to
ensure
that
this
tooling
is
going
to
update
your
cluster
to
the
desired
state
that
you
want
always
making
sure
that
the
cluster
is
healthy
and
we're
going
to
do
it.
One
node
at
a
time
again
focusing
back
on
keeping
the
tool
as
simple
as
possible.
A
What's
interesting
about
the
algorithm
is
the
fact
that
both
of
these
features
are
specific
to
your
use
case
to
your
business
use
case,
how
you
want
your
cluster
workloads
to
be
drained.
What
does
it
mean
for
the
club
for
the
workloads
to
be
drain
at
the
stateless?
Are
they
stateful
and
the
fact
what
is
defining
the
cluster
health?
This
is
just
the
control
plane.
Is
it
just
a
set
of
nodes?
A
So,
with
that
in
mind,
we
started
with
the
iteration
zero
for
our
tooling
and
implemented
the
tooling
what
we
call
gates,
rotator,
it's
implementing
just
as
a
simple
custom
controller
in
golang.
It
runs
as
a
single
pod
deployment
on
your
target
cluster
because
you
wanted
the
tooling
to
be
automated,
but
having
a
way
to
override
its
its
behavior.
We
differ.
We
design
this
as
a
finite
state
machine
and
which
could
act
on
a
couple
of
external
triggers
which
were
essentially
just
spawn
trick.
A
Map
objects
they
weren't
as
complex,
so
we
didn't
really
want
to
go
and
do
the
stretch
of
doing
CR
DS
with
the
version
0.
We
also
want
to
make
sure
that
we
are
maintaining
the
cluster
health
in
the
most
basic
way
possible,
and
so
we
want
to
make
sure
a
the
control.
Plane
is
always
seen
and
happy,
and
the
fact
that
we
can
always
draw
stats
out
of
your
running
cluster,
because
if
you
can't
like
there's
no
way
that
you
can
either
tell
the
cluster
is
healthy
at
all.
A
So
this
is
what
the
finite
state
machine
for
gates
rotator
looks
like.
We
just
have
binary
States
going
through
a
freeze
or
a
rotate
free
state
is
basically
the
dormant
state
of
the
root
of
the
rotator.
It's
the
default
state
that
it
begins
with,
and
it's
not
doing
anything
exciting.
It's
just
waiting
waiting
waiting
for
an
external
trigger
as
soon
as
it
gets
a
rotate
trigger.
It
then
starts.
A
It
then
puts
the
tooling
into
the
rotates,
rotate
state
and
which
is
essentially
now
tooling,
always
constantly
trying
to
reconcile
your
cluster
to
the
state
which
is
defined
by
the
rotate
trigger
we're
going
to
talk
about
that.
A
little
later,
once
when
you're
in
the
rotate
state,
it
is
possible
that
you
get
another
rotate
trigger
based
upon
a
next
iteration
of
aggradation,
or
what
have
you
may
and
that
just
cancels
the
current
rotation
and
gets
the
cluster
to
the
next
state
that
you
want
when
your
cluster
rotation
completes
or
it
undergoes
a
failure
scenario.
A
A
This
is
what
a
freeze
trigger
basically
looks
like
it's
a
simple
config
map
object,
which
doesn't
require
much
of
the
metadata.
However,
there
are
extensions
available
in
case
you
want
to
define
what
a
freed
state
looks
looks
like
you
can
define
that
in
the
metadata,
but
essentially
we
see
with
justifying
their
as
a
command.
We
define
that
as
a
cluster
name.
That
way
the
trigger
goes
to
the
right
cluster
that
you
are
directing
it
to
this
trigger,
basically
pauses
or
halts
and
ongoing
rotation,
and
it's
a
way
for
overriding
the
behavior.
A
The
rotate
trigger
is
a
bit
more
descriptive
than
the
freeze
trigger,
because
it
is
plain
trying
to
define
your
end
goal
or
your
final
state
of
the
cluster
that
you
would
want
it
to
be.
In
this
particular
example,
we're
showing
that
we
are
taking
our
cluster
to
a
defined
ami
version
that
we
wanted
to
be
could
be
a
tag
could
be
could
be.
Could
the
shark
could
be
anything
that
you
want?
Your
cluster
am
I
to
be
running.
A
Your
Maxis
defines
how
many
nodes
you
want
to
delete
at
a
at
a
time
and
being
the
version
zero.
We
wanted
to
just
do
one
node
at
a
time
it's
slow,
but
it's
gonna
help
us
get
there.
The
drain
time
essentially
defines
how
long
would
you
want
your
workloads
to
drain
off
themselves
before
they're
timed
out
and
then
kicked
off
forcefully
and
there's
another
set
of
metadata
that
you
can
define
to
limit
the
rotation
where
it
is
going?
How
many
nodes
is
it
defining
at
the
time?
A
What
nodes
is
it
deleting
so
now,
if
we
define
the
node
lifecycle
that
each
node
goes
through
as
the
rotator
proceeds?
Essentially,
when
we
begin
gates,
rotator
is
in
a
free
state,
it
doesn't
do
anything
and
it
classifies
all
of
the
nodes
as
just
uninitialized
there's
nothing
exciting
going
on.
Let's
say
we
now
kick
the
rotator
into
a
rotate
state
and
your
new
and
particular
node
that
you're
looking
at
is
just
running
the
latest
AMI.
It
goes
into
directly
an
updated
state.
Alright,
not
that
exciting.
A
Now
comes
the
case
when
you
actually
have
an
older
node
that
is
running
in
the
cluster,
and
it
goes
through
three
set
of
states
gates.
Rotator
marks
the
node
as
an
old
node,
which
is
meaning
the
workloads
have
now
got
to
come
off
it.
It
starts
draining
the
node,
so
the
workloads
get
off.
Of
that
node,
a
new
node
boots
up
the
workloads
migrate
and
at
the
the
old
node,
is
then
deemed
as
drain
once
the
entire
set
of
workloads
have
been
drained
off
your
node.
It
then
starts.
A
It
then
marks
the
node
as
ready
for
being
terminated
and
it
kicks
it
off
the
cluster.
These
are
just
some
illustrations
to
show
you
how
the
cluster
upgradation
is
going
to
go.
So
this
looks
like
a
generate
cluster,
for
everyone
has
a
net
CD
as
a
master
node
and
has
a
few
set
of
cubelets
running
in.
Let's
say
we
now
spin
up
a
gauge
rotator
in
this
particular
cluster
as
soon
as
it
spins
up
it's
in
a
default
free
state
and
it
marks
all
the
nodes
as
just
uninitialized.
A
Now,
if
you
want
to
rotate
this
or
if
you
want
to
upgrade
this
particular
cluster,
we
apply
this
particular
trigger,
which
is
defining
the
cluster
name.
As
my
cluster
takes
us
to
a
new
ami
tag
that
we
built
with
our
bug,
fixes
that
went
in
you
want
to
do
one
node
at
a
time,
and
we
want
to
make
sure
that
the
nodes
that
workloads
are
given
at
least
five
minutes
before
they're
being
forcefully
kicked
off
in
this
particular.
A
In
this
particular
case,
we
are
defining
the
node
label
to
be
cubelets,
and
so
we
only
want
to
upgrade
the
two
blade
nodes.
We
don't
care
about
the
control
plane
at
this
particular
time,
because
the
bug
fix
in
our
particular
case
affects
only
the
work
that
the
worker
nodes,
so
once
when
we
apply
the
Kaito
reader
starts.
Excuse
me
checking
for
the
cluster
health.
It
verifies
if
the
control
plane
is
all
happy
if
you
are
still
getting
our
stats.
If
that
is
the
case,
it
picks
up
a
particular
node
marks
it
as
an
old
node.
A
As
soon
as
we
do
that
the
workloads
in
in
the
node
are
beginning
to
be
drained
off,
workloads
go
away.
Master
node
sees
we're
at
loss
of
nodes.
The
cluster
autoscaler
then
proceeds
to
spin
up
a
new
node.
The
new
node
is
connected
to
the
auto
scaling
group,
for
which
the
lankan,
for
which
the
launch
configuration
has
already
been
updated.
So
the
new
node
comes
up.
It
has
a
latest
AMI,
it
has
bootstrapping
becomes
a
part
of
the
cluster
and
it
is
marked
as
updated,
and
it's
ready
to
now
accept
workloads.
A
The
workloads
that
were
drained
off
from
the
older
node
start
appearing
on
this
new
node.
The
rest
of
the
workloads
take
a
little
bit
take
a
little
bit
of
more
time
but
end
up
actually
migrating
on
the
new
node
that
we
just
spun
up
now.
The
old
node
is
ready
to
be
booted
off
the
cluster.
It
is
marked
up
as
drained,
so
we
are
not
running
any
of
the
workloads
on
it.
It
gets
deleted
from
the
cluster
and
the
next
call
we
make
is
to
the
cloud
provider
to
make
the
node
go.
A
Poof
we
have
the
cluster
with
a
new
node
that
has
brought
up
and
as
workloads
running
all
fine
provided.
The
cluster
is
all
still
healthy.
The
control
train
is
still
up
case.
Rotator
now
proceeds
to
terminate
the
next
node
and
so
on
and
so
forth.
So
this
is
how
gates
rotator
is
doing
a
cluster
of
rotation,
one
node
at
a
time.
A
So
now
we
are
trait
to
the
version
1.
The
version
0
worked
really
well
for
us.
We
were
able
to
improve
the
class,
the
productivity
for
an
operator
who
is
upgrading
the
cluster.
We
were
also
able
to
make
the
tooling
server
less
and
as
automated
as
possible.
What
you
wanted
to
do
still
improve
things.
We
wanted
to
make
sure
that
we
can
extend
the
cluster
health
and
have
a
stricter
control
over
what
a
control
plane.
Liveness
means.
A
We
also
wanted
to
add
a
couple
of
more
things
as
our
stack
progressed
and
be
improved
on
it,
to
have
things
like
log
aggregation
and
having
additional
set
of
pods
that
the
that
that
we,
as
a
team,
are
running
in
to
make
sure
that
the
cluster
is
up
and
running.
We
also
wanted
to
ensure
that
we
can
filter
a
specific
type
of
nodes.
What
we
want
to
upgrade
be
it
just
the
control
plane,
the
API
servers,
the
worker
nodes,
even
so
doing,
nodes
selection
by
a
ZZZ
or
their
placement,
or
even
using
node
labels.
A
We
also
came
up
with
situations
where
we
had
unique
site
type
of
clusters
going
in
where
different
type
of
workloads
require
a
different
set
of
restrictions,
around
control,
plane
and
the
type
of
nodes
they
were
running.
However,
our
tooling
wasn't
really
meant
to
do
that,
and
so
then,
the
next
iteration
we
started
improving.
All
of
these
features.
A
Some
of
the
failure
scenarios
that
the
gates
rotator
is
is
bound
to
it's
bound
to
be
self-healing
with,
is
the
fact
that
it's
either
waiting
enough
for
a
cluster
to
be
in
a
sane
state,
or
it
makes
sure
that
if
it's
not,
it
can
completely
halt
the
cluster
upgrade
rather
than
you
lose
this
entire
state.
So
things
like
h-series
losing
quorum
or
if
your
API
server
or
in
gray
is
overloaded,
and
you
can't
make
API
calls
things
around
AWS
losing
its
capacity
limits
in
the
availability
zones.
A
That
you're
running
in
all
of
these
cases
like
there
are
ton
of
situations
where
a
cluster
is
not
able
to
self
heal
itself,
and
the
cluster
rotation
or
upgradation
should
be
stopped
in
its
tracks.
Then
there
are
things
like
having
an
incompatible
in
place
of
version
upgrade
and
things
around
the
API
rate
limits
changing
for
AWS.
In
either
of
these
situations,
we
should
be
able
to
a
make
sure
that
the
cluster
is
never
losing
its
own
state,
and
we
are
also
able
to
report
that
back
to
the
operator.
A
However,
it's
very
important
for
the
operator
to
make
sure
that
we
can
recover
from
our
failures,
and
so
the
categories
of
failures
that
case
rotator
can
it
go
through
in
its
lifecycle,
are
between
a
soft
and
a
hard
limit.
So
the
soft
failures
are
things
around.
If
your
API
call
to
the
to
the
master
or
to
the
ear
bluest
console
ready
field,
you
can
retry
that,
and
it's
not
concerned
it's
not
considered
as
as
a
hard
and
read
failure.
A
While
it's
is
upgrading
your
cluster,
it
makes
sure
that
you
always
have
a
timeout
for
the
cluster
to
be
in
a
healthy
state
and
once
that
timeout
reaches
that's
considered
as
a
heart
failure
and
something
that
the
cluster
cannot
sell
feel
with
once
we
reach
that
state
gates.
Rotator
goes
into
a
damage,
control
mode
and
applies
a
freeze
trigger
implicitly.
That
makes
sure
that
your
cluster
rotation
doesn't
proceed
ahead,
you're
not
losing
any
state,
and
it
also
alerts
the
user,
the
admin
or
the
team
who's
responsible
for
upgrading.
A
Once
you
get
such
a
notification
through
pager
beauty
or
through
a
slack
board,
you
can
now
apply
certain
things
and
see
the
state
of
the
rotation
through
logs
through
stats
that
you've
been
getting,
and
you
can
once
once.
You
know
that
things
have
been
seen
again.
Things
like
AWS
has
got
its
capacity
back
or
things
of
that
nature.
A
You
can
now
remove
that
freeze,
trigger
and
override
the
behavior
once
you
remove
the
trigger
the
cluster
rotation
goes
back
into
a
rotate
state
and
continues
its
process
on
so
where
we
are
at
with
this
tooling
today
is
we
have
standardized
this
tooling
to
upgrade
all
of
our
cluster
set
lifts.
The
operational
load
has
shrunk,
a
ton.
We
are
able
to
do
the
business.
Our
upgrades,
as
we
wanted,
add
as
an
ideal
goal
for
us,
making
sure
that
we
are
not
waking
up
someone
in
the
middle
of
the
night.
A
A
We
can
cater
to
the
long-running
workloads,
so
things
around
machine
learning,
jobs
which
are
running
for
hours
and
days.
The
tooling
is
not
yet
equipped
to
handle
those
type
of
workloads.
We
would
want
to
make
sure
that
we
can
upgrade
to
those
specific
those
specific
type
of
workloads
and
cater
to
upgrading
them.
A
The
tooling
has
minimal
amount
of
interaction
now
required
from
the
operator.
However,
we
would
ideally
want
it
to
be
at
a
place
where
we
can
update
the
state
once
and
then
be
done
with
it.
The
rotation
tool
should
automatically
pick
up
where
we
need
the
cluster
to
be
in
a
state
and
automatically
you
do
so.
Instead
of
applying
the
trigger
manually
as
we
progress
ahead
and
enter
into
a
state
of
having
multi
clusters
and
having
the
services
spread
across
all
of
them,
we
would
want
to
make
sure
that
we
can
maintain
the
service
health.
A
We
can
maintain
the
entire
class.
We
can
maintain
the
cluster
health
aggregated
lis
for
all
of
them
and
also
make
sure
that
we
are
in
sync
with
the
PAS
disruption
budget
and
make
sure
like
we
don't
fall
below
that.
We
would
also
want
to
make
sure
we
can
extend
this
tooling
out
for
others
to
use
and
make
sure
we
can.
We
can
change
our
cluster
health
reporting
that
way
it
becomes
more
sort
of
a
plug
in
and
is
available
for.
A
Anyone
to
use,
and
also
to
do
things
around
like
have
a
specified
way
how
you
want
to
terminate
the
nodes.
Do
you
want
to
do
the
control
plane?
First,
do
you
want
to
do
the
cubelets
first,
or
even
the
increasing
tier
of
your
services,
like
you
want
to
do
a
specific
ordering
of
the
upgrade?
You
should
be
able
to
define
all
of
that.
So
that's
where
the
tooling
is
today
happy
to
chat
more
and
with
that.
Thank
you.
Everyone
up
for
questions.
A
So
the
rotator
doesn't
save
any
of
the
state
on
itself.
It's
basically
applying
the
state
on
the
objects
themselves,
so
the
node
as
it
goes
through
the
transitions
we
apply,
that
specific
labels
on
the
nodes.
So
when
the
node
on
which
the
gates
rotator
itself
is
running
and
that
needs
an
upgrade,
the
workload
gets
thrown
off
the
reader.
The
case
rotator
spins
up
on
a
new
node
and
just
picks
up
from
where
it
left.
A
So
for
masters,
we
basically
just
make
sure
that
our
cluster
state
is
seen.
You've
got
redundant,
masters
master
nodes
running
in,
and
so
we
make
sure
we
always
have
a
CN
number
of
them
up
and
running.
We
define
the
max
search
for
them
to
be
a
limited
number.
That
way,
we
make
sure
that
the
availability
for
the
master
nodes
is
still
up
and
running,
so
we
can
get
rid
of
one
API
masters,
but
we
still
got
a
redundant
set
of
them
still
running
in,
but.
A
A
At
the
moment,
but
as
I
said
like
we,
we
are
still
working
with
this
with
this
project
right
now,
making
sure
we
can
attest
to
all
of
all
types
of
workloads
that
are
possible,
but
that's
one
of
the
things
that
we
have
that
on
our
roadmap,
we'll
make
sure
like
we
can
preserve
the
state
of
stateful
workloads
and
make
sure
they
are
always
in
a
good
health
when
we
are
upgrading.
But
thanks.
Thank
you.
E
Just
curious,
this
is
interesting
because
I
did
a
similar
thing
when
I
was
at
Capital
One.
What
are
you
or
are
you
kind
of
allocating
things
off
to
the
side
and
saying?
Yes,
this
needs
to
be
rotated
or
we
need
to
rotate
our
instances
because
of
this,
but
we're
not
gonna.
Do
it
through
the
Cates
rotator.
For
some
specific
reason.
A
So
it
depends
in
case
you've
got
a
breaking
change
for
the
API
master
node
and
it
cannot
be
just
rotated
one
node
at
a
time.
You
need
to
take
down
everything
and
then
put
them
up.
It's
on
those
cases
that
you
then
thought
none
of
those
nodes
or
basically
don't
add
them
to
the
set
of
nodes
that
you
wanted
to
do
so
like
do
the
work
and
do
the
STDs
to
the
cubelets,
but
don't
do
the
API
service
and
then
you
can
hand
roll
those
in.
F
When
you
were
describing
your
rolling
update
pattern,
you
kind
of
sorry,
you
illustrated
that
pods
would
move
from
one
to
the
other.
Have
you
done
anything
special
with
the
scheduler
to
ensure
that
kind
of
behavior
to
minimize
service
disruption,
or
is
it
more
the
default
behavior
where
the
pods
will
just
reschedule
where
available.
A
So,
in
the
version
zero,
we
started
doing
a
cordon
of
all
of
the
older
nodes
to
help
make
sure
that
we
are
not
trashing
the
Welkin,
the
workloads
from
going
back
to
the
older
nodes.
That
was
one
of
the
problems
that
I
already
forced
so
like
saying
you,
you've
got
ten
words
running
in
you
term,
one
of
the
nodes
and
that
particular
workloads
get
thrown
off
to
the
older
nine
nodes.
A
So
basically
just
doing
five
at
a
time
or
like
X
number
at
a
time
that
way
use
you
ensure
that
the
SSL
is
are
preserved,
but
it
goes
a
little
slower,
but
because
it's
automated
it
kind
of
like
really
works,
you're,
basically
incentivizing.
The
fact
that
you
can
preserve
the
essays
in
in
loss
of
getting
the
cluster
rotation
done
in
the
amount
of
time
that
thank
you.
F
A
Interesting
take
so
the
kind
of
a
kind
of
things
that
are
involved
in
creating
a
new
ASG
become
more
involved.
When
you
are
doing
an
upgrade,
we
tend
to
reduce
the
amount
of
operational
node
that
you
are
trying
to
do
when
you
are
upgrading
and
which
is
why
we
sort
of
like
stuck
to
the
principle
of
just
sitting
with
the
same
ASG
and
then
updating
X
nodes
at
a
time
that
way
like
it.
A
G
H
A
You
generally
tend
to
keep
the
node
that
it
is
running
on
as
the
last
one
in
the
queue
that
way
you
can
run
through
every
one
of
them,
but
it
doesn't
make
a
difference
if
you
terminate
the
node
that
the
rotator
itself
is
running
on
it
just
picks
up
from
where
it
left
goes
to
the
set
of
nodes,
because
the
state
is
nowhere
saved
in
in
a
persistent
manner
or
a
database.
Awesome
thanks.
A
A
So
basically,
you
just
specify
the
ami
version
that
you
want
it
to
be,
so
be
it
like
when
your
version
at
4.1
and
you
are
rolled
up
to
version
or
r2
as
soon
as
you
see
that
there's
a
problem
going
and
you
want
to
roll
back
to
or
dot
one
you
just
circle,
it
back
saying
make
the
state
of
this
state
make
the
state
of
the
cluster
back
to
what
it
was
in.
So
the
order
of
the
operations
doesn't
really
change
in
terms
of
like
III
thinking.
A
I
A
A
We
always
make
sure
that
we
are
always
important
when
we
are
upgrading,
so
we
don't
take
all
of
them
out
and
even
so,
if
we
are
taking
one
at
CD
at
a
time,
we
always
make
sure
that
they
are
in
quorum
and,
like
that's
the
part
that
we
wanted
to
make
more
stricter
way
that
we
can
ensure
the
control
plane
is,
is
happy.
Okay,.
I
A
So
we're
actually
not
using
the
Amazon
CNI,
it's
very
similar
to
what
we
have
in
source.
Sorry
in
house
I
jest
you
do
to
listen
to
conversation
from
Paul.
We
were
basically
talking
about
the
en
I,
see
and
I,
adapted
that
we
worked
on
so
at
the
time
when
we
were
building
our
clusters.
Amazon
hadn't
gotten
that
out-
and
it
just
so
happened
that
the
release
cycle
was
almost
at
the
same
time,
but
we
just
picked
up
things
that
work
really
well
for
us
I
see.
Thank
you
problem.
J
Hi
hi,
when
you
support,
when
you
upgrade
Carini's
master
components,
do
you
support
updating,
API
objects
from
one
version
to
next?
Let's
say
from
v1
beta
2,
V
1
or
you
know,
as
Cooney's
versions
upgrade
Brigade.
Some
API
objects
versions
could
be
more
specific,
for
example
like
deployments
for
a
while.
We
want
better
type
objects.
Now
they
became
v1.
Do
you
handle
transition
from
older
versions
to
new
versions,
as
you
upgrade
kubernetes
versions,
I.
A
J
A
K
So,
at
any
point,
during
your
design
phase
for
case
rotator,
did
you
consider
also
adding
an
option
of
having
blue/green
rollouts
for
clusters,
as
opposed
to
in
place
example,
several
of
people
who
ask
questions
talked
about
incompatible
upgrades
like
at
City
v2
to
HCD
v3.
We
have
gone
through
that
phase.
We
had
something
similar
to
this
and
we
had
blue
green
in
place
so
just
curious
if
you
ever
had
to
consider
that
as
an
option
as
opposed
to
in
place.
That's.
A
L
A
L
It
terminate
the
nodes
after
you've,
made
them
not
ready,
or
do
you
manual
it
you
manually,
terminate
those
Ryan
say
it
again,
please.
So
when
the
nodes
go
to
not
ready,
do
you
wait
for
the
cluster
autoscaler
to
get
rid
of
them,
or
do
you
manual
it
or
you
terminate?
There
was
a
separate
step
in
there
so.