►
Description
Want to view more sessions and keep the conversations going? Join us for KubeCon + CloudNativeCon North America in Seattle, December 11 - 13, 2018 (http://bit.ly/KCCNCNA18) or in Shanghai, November 14-15 (http://bit.ly/kccncchina18)
Continuously Deliver your Kubernetes Infrastructure - Mikkel Larsen, Zalando SE (Advanced Skill Level)
Kubernetes provides the means for making it simple to continuously deliver our apps on top of clusters. But what about the underlying Kubernetes infrastructure? How can we continuously deliver improvements to the infrastructure and be confident that new changes will not have a negative impact on apps running on top? Using the rollout of Kubernetes at Zalando as an example Mikkel will discuss how Zalando manage 60+ production and test clusters, how they went from Kubernetes v1.4 and continuously updated all clusters to the latest stable release, and how they test every change to the cluster configuration. The talk will also take a small dive into the topic of testing clusters and give examples of how you, with little effort, can use and build upon the extensive conformance tests suite from Kubernetes to automatically end to end test changes to your own custom Kubernetes configuration.
About Mikkel
Mikkel is a Software Engineer working at Europe's leading online Fashion platform, Zalando since 2016. He works in the Platform Infrastructure team focusing on Kubernetes and AWS infrastructure and has a big interest in Continuous Delivery and automation.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
B
B
B
B
Our
current
scale
looks
like
this,
so
we
have
366
AWS
accounts
and
we
have
84
kubernetes
process
as
of
this
morning
and
you
might
think
366
a
Tobias
account
seems
like
quite
a
lot.
One
of
the
reasons
is
that
we
come
from
a
model
where
we
gave
one
AWS
account
per
team
or
less.
Maybe
some
teams
even
got
multiple
AWS
accounts
for
staging
and
testing
production
so
on
we
have
this
tool
set
around
a
double.
Yes,
we
call
Stoops,
which
is
what
the
infrastructure
we
are
transitioning
from.
B
It
is
just
some
tools
to
help
making
the
AWS
services
a
bit
easier
to
use,
but
it's
pretty
much
but
delay
AWS.
You
can
think
of
it.
This
way,
and
in
this
model
we
have
like
I,
said
AWS
accounts
per
team.
We
have
some
rules
that
all
instances
must
run
with
the
same
ami
for
compliance
reason,
so
all
teams
must
use
the
same
ami
and
they
their
Falls
I,
have
to
update
this
ami.
B
At
some
point,
everyone
gets
power,
user
access
so
like
not
exactly
root,
but
almost
today
double
use
account,
so
they
can
pretty
much
do
whatever
they
want,
which
gives
them
a
much
power,
but
with
power
comes
responsibility,
of
course,
and
there's
a
lot
of
responsibility.
If
you
have
to
manage
everything
for
every
team
that
every
team
has
to
do
this,
so
basically
the
rule
is
you
build
it
and
you
run
everything
and
everything
means
also
monitoring.
So
we
of
course,
have
a
central
monitoring
system,
a
solando,
but
you
actually
get
monitoring
for
your
services.
B
You
would
have
to
deploy
something
into
your
idea
base
account
like
an
agent
that
can
can
pull
data
from
from
different
services.
That
we
run-
and
this
you
have
to
manage
yourself-
that
this
18
this
running
as
same
with
CI
CT.
We
also
provision
Jenkins's
for
each
team
if
they
wanted
to,
but
they
have
to
kind
of
manage
this
themselves,
so
we're
transitioning
to
a
model
where
we
provide
kubernetes
as
the
base
infrastructure.
And
then
we
pulled
a
bunch
of
stuff
on
top
of
this.
So
we
have
instead
of
having
one
kubernetes
cluster
per
team.
B
Something
like
this.
We
have
now
clusters
per
product
and
product
is
multiple
teams,
so
three
to
six
teams,
depending
on
how
big
the
product
is
and
then
the
reason
why
we
don't
just
put
everything
in
one
cluster
only
like
this
is
that,
do
you
have
an
organization
that
is
structured
in
a
certain
way
and
you
have
to
like,
do
it
gradually
and
of
course,
there's
also
like
reliability
issues
with
putting
everything
in
one
cluster.
B
Of
course,
you
can
request
previous
taxes,
but
by
default
you
don't
get
this
and
then
we
also
provide
a
lot
of
stuff
out
of
the
box.
So,
like
s
was
talking
about
the
money
trying
stuff
this,
we
provide
automatically
in
the
class
when
you
provision
them.
The
C
ICT
solution
is
something
we
have
that
is
integrated
with
our
company's
setup,
which
is
more
or
less
our
box.
B
When
people
create
a
repository
for
a
new
application
and
just
to
be
clear,
what
I'm
talking
about
what
I
want
to
talk
to
about
today
is
not
like
CI
CD
on
to
kubernetes,
but
actually
continuous
delivery
of
the
kubernetes
infrastructure
and
when
we
started
to
set
up
this
whole
community's
infrastructure
about
one
and
a
half
year
ago,
or
so
we
had
some
kind
of
philosophy.
I
think
philosophy
is
maybe
a
true
big
word,
but
just
to
have
some
idea
of
what
do
we
want
with
this
platform
and
how
should
it
kind
of
work.
B
So
one
thing
is
that
we
don't
want
to
have
any
pet
rosters,
meaning
we
don't
to
manage
ad
clusters
individually,
because
some
teams
have
a
very
special
need
in
this
way
and
we
only
deployed
the
money
trying
for
them
and
they
others
have
some
other
monitoring
solution
or
whatever.
It
should
be
fairly
generic
across
to
all
the
clusters,
because
then
it's
much
easier
to
manage
from
us.
Of
course,
there
will
be
differences
in
what
instance
sizes
the
teams
need
for
the
different
services.
B
Another
thing
was
that
we
wanted
to
always
provide
the
latest
stable,
kubernetes
version,
meaning
right
now
we
have
one
nine
instead
of
one
ten,
but
this
was
just
not
there
yet
fast
and
we
actually
is
transitioned
the
older
cluster.
We
created
the
first
one,
which
translated
from
1.4
all
the
way
to
1.9,
so
updating
all
the
major
versions
in
between-
and
this
is
a
bit
different
from,
like
gke
the
schewe
kubernetes
service,
where
you
kind
of
as
a
user.
B
You
can
choose
to
update
to
which
version
when,
when
it
fits
you,
but
in
our
case
we
kind
of
take
the
responsibility
of
updating
this
for
our
users
and
we
make
sure
that
we
test
it
enough
before
we
roll
it
out
and
then
we
kind
of
just
roll
it
out
when
we're
sure
there's
no
disruption.
That
can
happen
almost
you
all
right
and
then
they
don't
have
to
care
about
it.
B
B
B
Whenever
makes
it
what
much
easier,
when
you
really
need
to
do
it,
and
we
also
think
about
like
we
do
cluster
auto
scaling
of
our
clusters
and
if
you
think
about
rolling
all
the
nodes
in
a
cluster
or
doing
scale
up
and
scale
down,
it's
not
so
much
stiff
and
what's
actually
happening
to
the
for
the
things
that
are
running
inside
the
cluster.
So
it
should
should
work
and
then
another
thing
was
to
be
fully
automated
operations
again
as
in
quotes,
because
I
think
everyone
knows
who
does
operation
that
can't
be
fully
automated.
B
This
is
that
the
ideal
situation
is
that
we
can
just
make
a
pull
request
to
our
configuration
repository
with
some
change,
and
then
we
just
click
merge,
and
then
we
just
see
everything
roll
out,
and
this
also
works
for
the
most
time.
But
of
course,
sometimes
you
have
to
do
something
manually,
but
this
is
the
philosophy
anyway,
to
give
you
a
bit
of
context
about
our
setup.
That
I'm
talking
about
doesn't
mean
that
you
can't
take
these
ideas
that
I'm
presenting
today
and
not
use
it
indefinitely.
B
B
B
Then
we
have
H
a
high
available
control
plane,
which
is
like
I,
said
stateless,
because
we
don't
have
third
CD
and
we
have
an
LP
in
front
where
we
terminate
as
a
cell
and
then
we
can
just
yeah
roll
the
roll.
The
master
notes
behind
this.
If
you
do
a
an
upgrade
of
any
kind,
our
cluster
configuration
is
thought
in
kit
and
we
actually
have
it
on
github.com
our
production
configuration
if
you
want
to
check
this
out.
B
I
have
some
links
later
on
and
we
run
instances
to
be
our
jingyan
set
up
in
to
insist
against
the
clusters
that
we
create
and
then
changes
are
rolled
out
via
something
we
call
the
cluster
lifecycle
manager,
which
is
like
a
controller
component.
That
knows
how
to
first
of
all
apply
this
confirmation
in
the
right
way.
Our
configuration
has
some
go
templating
for
a
simple
substitution
of
values,
and
this
it
can.
It
does
this
kind
of
stuff,
and
then
it
also
has.
B
It
knows
how
to
update
the
notes
in
in
a
reliable
way
and
this
class.
The
lifecycle
manager
is
open
source
since
Friday,
so
you
can
go
check
it
up,
check
it
out
on
our
github,
it's
still
in
a
state
where
it's
a
bit
solando
specific,
but
we
want
to
work
on
it
to
make
it
more
generic.
That
is
useful
without
too
many
of
so
we
have
also
something
that
we
call
the
cluster
registry,
which
is
either
a
CP
rest
API
in
in
our
setup
or
just
a
yama
file.
But
basically
the
format
looks
like
this.
B
And
then
we
have
a
list
of
node
pools
as
we
call
them,
which
is
I,
think
they
also
used
the
term
node
group
in
G
key,
for
instance,
but
it's
basically
in
in
a
davia
terms.
It
just
translates
to
an
auto
scaling
group,
in
our
case,
where
you
just
specify
I
want
a
set
of
nodes
that
identical
that
have
this
instance
type
that
have
this
name
for
the
node
pool,
there's
a
min
and
Max
size,
and
then
it
can
also
scale
between
these
sizes,
and
we
can
have
multiple
node
pools
to
find
our
trust.
B
Our
configuration
is.
This
is
the
one
on
get
up,
which
is
roughly
looks
like
this
with
a
cluster
llamó,
just
the
confirmation
spec,
basically
of
the
of
the
cluster
set
up,
and
then
we
have
the
SD
confirmation
stack,
and
then
we
have
a
folder
called
manifest,
which
is
all
the
kubernetes
manifests
that
we
deploy
as
default.
So
all
like
flannel
team
and
set
for
our
overlay
network,
and
our
monitoring
system
is
deployed
just
by
putting
a
Yama
file
company.
B
This
channel
file
in
this
manifest
folder,
and
then
we
have
this
masked
and
work
without
container
Linux
configuration,
which
is
just
the
like.
The
user
data
of
the
of
the
ec2
instances
also
start
Cupid
and
how
to
bootstrap
things
on
the
nodes
is
what
we
configure
in
these
files
yeah
and
then
we
have
there's
like
at
the
lifecycle
manager,
so
how
it
works
is
I,
try
to
pinpoint
it
here,
so
it
has
two
sources
of
configuration.
B
It
has
the
git
repository
where
we
have
our
cluster
configuration
and
then
it
has
the
cluster
registry,
which
has
this
metadata
about
the
clusters
and
it
basically
watches
both
of
these
configuration
sources
and
if
there's
a
change
to
one
of
them,
it
will
react
on
this.
Merge
the
information
tricky
or
not,
depending
on
the
cluster
and
in
the
beginning,
if
it's
a
new
cluster,
it
will
create
it
by
do
a
create,
with
confirmation,
create
a
new
cloud
formation
stack
for
it,
CD
and
one
for
for
the
kubernetes
infrastructure,
and
this
will
basically
set
up
this.
B
This
will
do
every
time
so
to
synchronize.
So,
even
if
you
go
and
delete
something
in
the
cluster
on
the
next
update,
it
will
be
overridden
and
have
the
actual
configuration
that
we
have
stolen
yet,
which
is
tracked
and
proceeded
to
see
what
what
is
actually
running
in
this
cluster
and
the
way
we
kind
of
feed
a
new
configuration
into
the
cluster.
Is
we
have
this
kind
of
upgrade
flow
where
we,
we
will
usually
create
a
branch
with
my
new
feature,
then
we
make
a
PR.
B
We
add
ready
to
test
label
on
the
PR
when
it's
ready
to
be
pre
answer
interested
and
then
we,
when
it
once,
is
ready.
We
either
add
a
label
ready
to
merge
or
we
just
merge
it,
and
then
we
merge
it
into
something.
We
call
the
dev
branch.
So
the
reason
we
have
these
branches
is
that
we
kind
of
created
this
idea
of
channels,
so
our
clusters
live
in
different
channels,
so
we
have
dev
alpha,
beta,
stable
and
so
on.
B
We
can
just
create
a
new
branch
to
create
a
new
like
channel
level
and
then
it
kind
of
flows
from
from
dev
to
alpha
to
beta,
to
stable
and
so
on,
the
changes,
and
ideally
we
wanted
to
have
it
like
you,
just
merge
two
dev
and
then
after
some
time
after
some
monitoring
has
happened
and
there's
no
issues,
it
automatically
can
follow
it
into
elf
and
PSR
we're
not
there.
Yet
we
are
still
doing
it
manually,
so
creating
a
new
PR
for
the
next
branch.
B
We
have
the
alpha
channel,
which,
right
now
we
have
our
main
infrastructure
cluster
there,
which
is
basically
what
my
team
and
surrounding
teams
are
using
to
apply
all
the
infrastructure
that
we
need
to
that.
All
our
teams
need
to
integrate
and
CD
system,
and
all
this
is
running
there
and
the
reason
we
have
it
already
as
the
Alpha
is,
that
is
very
important
to
us.
B
It
could
be
called,
doesn't
really
matter,
it's
just
all
the
remaining
clusters
for
the
products
that
are
there.
So
if
we
see
that
everything
is
working
in
alpha,
then
we
are
fairly
confident
that
the
pea
clusters
will
also
work
when
we
roll
out
and
update-
and
this
is
right
now-
80
clusters,
like
I,
said
we
do
into
interest
on
every
PR.
B
Basically
looks
like
this
from
the
github
point
of
view,
so
we
have
these
three
orange
parts
that
are
running
different
end-to-end
tests
and
the
entrances
we
run
the
conformance
test
suit
from
directly
from
upstream
kubernetes.
So
there's
this
whole
test
/e
to
E
and
in
the
kubernetes
repository
relah
is
like
almost
maybe
thousands
of
different
test
cases.
Internal
test
cases,
and
currently
it
is
144
of
the
of
those
are
conformance
tests
as
of
1/9,
and
these
we
run
to
like
get
a
general
sense
of
is
the
cluster
performing
in
like
and
conformant
way.
B
This
is
also
the
tests
that
are
run
by
this
run.
Oh
boy
for
validating
clusters
that
are
these
certificate
is
certified
clusters.
It's
the
same
test.
Food
I
think
we
even
run
a
bit
more
than
their
next.
We
also
pick
some
stifle
set
test.
These
are
not
part
of
the
conformance
because
they
require
some
default
storage
class
configuration
in
the
cluster
which
is
not
required
by
the
conformant
part.
B
So
we
picked
these
out
individually
and
we
pick
out
two-
and
these
are
really
really
good
because
they
test
stuff,
like
attaching
volumes
to
stateful
sets
or
I.
Think
we
run
the
the
rate
is
yes
like
in
Redis,
deploying
a
registry
note
register,
and
then
it
each
node
needs
volume
and
in
AWS
this
means
an
EBS
volume,
so
we're
actually
testing
the
full
entrant
of
the
controller
manager
being
able
to
schedule
of
EBS
volume,
adjusting
it
to
the
nodes,
cube
loop
being
able
to
mount
the
volume
and
so
on.
This
is
mostly
this.
B
You
just
work
out
from
kubernetes
already,
but
we
might
miss
configure
something
on
our
notes.
That
would
make
it
not
work.
So
that's
why
we
are
testing
this
and,
lastly,
we
have
some
icons
alano
chests,
but
it's
just
our
chests
for
our
own
tooling.
So
we
have
chests
for
our
interests.
We
have
a
English
controller,
that's
also
open
source
that
creates
application
load
balancers
for
ingress
and
this
retesting.
B
We
also
have
this
external
DNS
project,
which
is
community,
singular
project,
and
so
we're
actually
testing
deploying
an
application
with
a
part
with
an
increase
to
we
get
a
TNS
name
to
be
able
to
route
all
the
way
through
the
to
the
part
in
the
class
time,
and
if
we
also
test
some
part
security
policy
stuff,
which
I
think
is
more
or
less
legacy
now
in
in
our
case,
because
it's
also
chest
at
upstream
now.
But
it
wasn't
when
we
introduced
it
last
year
and
the
way
we
sorry.
B
The
day,
the
way
we
run
the
in
twenties
is
that
we
think
that
you
have
to
you
want
to
do
an
upgrade
from
the
dev
channel
to
alpha.
You
want
to
move
all
the
things
that
are
new
in
dev
into
alpha.
Then
we
basically
have
a
Jenkins
pipeline
where
we
first
create
a
cluster
based
on
the
alpha
branch,
because
it's
like
the
base
branch,
then
we
do
a
cluster
update
using
our
cluster
lifecycle
manager
from
the
dev
branch.
B
So
we
take
all
the
changes
from
dev
to
alpha,
but
if
there
and
apply
these-
and
the
reason
we
do
this-
is
that
if
we
just
create
a
cluster
directly
from
the
dev
branch
or
from
the
from
the
merge
between
alpha
and
F,
then
we
just
test
creating
a
new
cluster
from
the
new
configuration,
and
this
might
work
perfectly
fine.
But
what
we
also
want
to
test
is
that
we
are
actually
testing
doing
a
live
migration
from
one.
Maybe
it's
one
kubernetes
version
to
a
second
one.
B
Once
we
have
this
trust
to
set
up
based
on
the
the
dev
changes,
then
we
basically
run
the
interest.
The
conformance
test
stifles
step
test
and
and
there's
a
Lando
test,
and
we
do
we
actually
run.
We
create
one
cluster
per
test
case
of
her
these
three
tests
or
recreate
three
trusters
in
total,
and
this
is
just
to
speed
it
up.
If
one
of
them
fails,
then
you
have
would
have
to
like
wait
for,
for
a
long
time
to
re
run
all
the
tests
if
they
were
in
one
cluster.
B
If
only
one
of
the
tests
fails
here
is
only
one
cluster.
We
have
to
restart
and
the
other
test.
We
don't
have
to
rerun.
If
everything
works,
we
just
delete
the
classes
again
to
save
cost.
If
everything
doesn't
work,
we
leave
the
clusters
running
def
a
little
while
because
then
we
can,
then
we
can
go
and
inspect
and
see
if
there's
something
yeah
wrong
with
the
cluster
or
what
happened
it's
easier
than
looking
at
the
logs
to
actually
to
do
this
from
a
thing
as
part
of
you.
B
It
just
looks
like
this
I
think
this
one
it
takes
like
that.
20
minutes
here
to
set
up,
cast
and
update
it,
but
it
of
course
depends
on
how
many
changes
we
make
and
then
the
integration
test
here
this
London
one
switch
takes
like
10
minutes
to
run
mostly
waiting
for
TNS
stuff,
so
it
takes
like
40
minutes
or
so
total
and
looking
at
the
the
output
of
the
entrances
looks
something
like
this.
B
This
is
144
this
the
conformance
test,
and
it
says
that
it
skipped
a
bunch,
so
638
were
skipped,
and
this
is
because
you
can
you
can.
This
is
how
you
run
the
test.
Basically,
I
created
this
image
where
it's
just
pulled
out
the
entrances
from
the
upstream
kubernetes,
and
only
pack
this
into
one
image.
So
you
don't
have
to
have
the
holconius,
which
is
much
bigger,
I
just
produced
four.
If
anyone
is
interested,
but
basically
it
just
runs
the
in
trenches
that
are
obsolete
when
it
is,
it
has
some
flag.
B
P
means
parallel.
Focus
means
what
you
want
to
focus
on
and
skip
what
you
what
you
don't
want,
so
you
want
to
focus
on
conformance.
So
it's
each
test
has
this
naming
convention
that
it
has
this
square
bracket
and
then
some
some
value
that
you
can
target
beta
breaks.
Basically,
and
then
you
just
skip
the
zero
which
cannot
be
run
in
parallel,
so
they
have
to
run
one
by
one
so
and
and
these
we
don't
need
to
take
a
long
time.
So
we
skip
skip
these
the
stateful
set
similar.
B
You
just
pick
another
thing
to
focus
on
stateful
set
basic
here
and
stable
set
the
one
with
the
Redis
and
pretty
much
you
can.
If
you
find
some
tests
that
would
test
something
good
for
you
can
just
pick
it
out
and
and
and
use
that
yeah.
This
way,
a
few
hints
for
running
instances
that
we
found
is
that
it's
possible
to
run
with
flake
attempts.
This
is
a
flex
to
the
chest.
Orchestration
thing
it's
called
Kinkos,
encore
and
just
the
framework,
but
flaky
at
Sims,
basically
means
that
it
will.
If
it's.
B
If
the
test
first
fails,
then
we
will
actually
try
to
run
it
again,
and
this
is
useful
because
some
of
the
tests
are
just
creating
a
pot
or
creating
multiple
pots,
and
since
our
cluster
is
auto
scaling
during
the
test,
we
might
lose
a
part
because
it's
not
the
tests
are
not
really
built
in
a
reliable
so
having
the
FADEC
attempts
to
usually
it's
enough
to
make
sure
that
that
it's
working
and
what
we're
testing
is
the
law
of
chastity.
That
works,
not
that
the
interests
are
reliable
or
something.
B
So
so
that's
why
it's
fine
and
then
another
idea
is
to
update
the
interent
image
for
every
major
release.
You
do
you
role
also
going
from
one
a
to
1/9.
You
want
to
also
change
the
in
trend
image
you
use,
but
but
going
from
one
nine
to
one
nine
one
is
usually
fine.
That's
what
we
found
disabled
program
test
we
skip.
Sometimes
some
tests
are
just
broken
in
companies
release
and
you
want
to
skip
them
and
then
remove
all
completed
paths
from
the
cube
system
namespace.
B
B
The
the
really
interesting
part
is
actually
upgrading
the
nose,
because
one
thing
is
just
grading
faster,
updating
it
running
in
20s,
but
if
you
don't
have
anything
running
inside
you
don't
really
know
if
the
operate
went
really
well
in
between
so
first
we
went
with
a
naive
approach:
I
call
it
where
we
basically
just
have
an
outer
skin
group.
We
set
a
fixed
number
of
which
we
just
fixed
the
number
in
the
auto
skin
group.
When
you
want
to
do
an
update,
so
we
kind
of
lock
lock
it.
B
So
it
can't
auto
scale,
and
then
we
just
do
the
current
plus
one.
So
we
get
one
extra
note
and
we
basically
just
wait
for
it
obvious
to
bring
up
this
extra
note.
Then
we
train
a
random
node
in
the
cluster
and
this
will
then
terminate
it
and
it
obvious
will
bring
up
a
new
one
because
you
didn't
remove
one
and
the
desired
is,
is
list
and
that
the
current
is
less
than
the
desired.
And
this
you
do
again
for
all
the
notes-
and
this
was
of
course
fine.
B
B
B
Another
problem
was
that
the
volumes
like
I
always
set
up
her
availability
zones.
So
if
your
pot,
that
needs
a
volume
in
one
zone,
it
can't
get
scheduled
to
a
different
node.
So
if
you
don't
have
enough
nodes
in
a
particular
zone,
where
you
have
the
volume,
then
you
out
unlock
your
part
is
just
pending
and
you
you
can't
really
move
forward,
and
since
we
didn't
have
any
timer,
we
just
eventually
removed
the
next.
B
Another
problem
was
that
we
fixed
the
auto
scaling
group
size,
so
we
could
basically
not
scale
in
and
out
in
between
the
upgrades,
so
we
were
kind
of
fixed
and
we
would
have
like
pending
notes
from
the
autoscaler,
which
we
would
have
to
wait
until
after
the
upgrade
to
actually
scale
out,
which
is
not
good.
If
people
are
deploying
a
lot
and-
and
a
fourth
problem
was
that,
how
do
we
actually
handle
stateful
applications?
B
So,
in
our
case,
Proteus
is
used,
and
if
you
have
the
case
where
you
have
a
master
pot
and
some
replicas,
then
how
do
you
know?
If
you
do
this
to
drain
one
node,
you
don't
know
if
you're
actually,
training
the
master
and
if
you're
training
the
master,
then
your
scripts
person
becomes
unavailable.
So
this
is
not
the
ideal
way
to
ideal
way
to
go,
and
so
what
we
did
was
we
thought.
B
Okay,
we
need
to
design
a
new
upgrade
strategy
that
is
more
robust
to
these
issues
and
the
first
thing
that
we
thought
about
is
okay.
What
does
the
mean
mean
for
know
to
be
ready,
which
is
important,
because
then
we
can
know
when
we
can
move
on
to
the
next
one.
So
I
think
about
the
notes,
as
the
kubernetes
node
object
and
the
underlying
easy
to
instance,
in
our
case
or
VMO,
and
if
you
split
them
up
the
the
couplet.
Has
this
reporting
note
ready?
B
So
this
part
disrupting
packages-
and
this
is
a
way
to
say
that
in
this
case
mean
available
one
meaning
that
if
I
have
three
parts,
then
I
can
kill
two
and
then
I
still
have
one
available
and
and
and
our
infrastructure
will
not
continue,
deleting
more
parts
if
it's
disrupt
the
budget.
Basically,
so
you
have
a
control
that
evicts
their
part.
This
is
successful,
it'll
fix
the
next
one
successful
and
then
it
tries
to
evict
the
third
one
and
since
the
the
the
budget
would
be
disrupted.
B
If
you
were
to
remove
the
last
one,
then
it
fails
and
then
also
how
to
handle
the
the
Postgres
stuff.
Basically,
we
have
again
this
setup
with
the
master
and
replica
we
put
this
labels
on
the
on
the
pro
square
spots,
and
then
we
have
a
postage
operator
which
is
also
an
open
source
project
that
our
database
database
team
are
developing.
It's
a
basically
where
you
can
define
as
custom
resource
definition
for
postage
cluster,
and
then
it
get
gives
you
a
stateful
set
with
these
properties
and
manages
the
whole
cluster.
B
For
you
and
say
here
in
the
case
we
want
to
drain
the
master,
then
the
operator
will
put
a
disruption
party
saying
we
always
has
to
have
one
master
part
available.
So
if
you
try
to
drain
the
master
a
row,
basically
we
fail
the
eviction
and
then
the
prosecution
operator
in
the
meantime
will
see
that
we
are
doing
a
class
to
update.
So
we
see
the
green
node
is
a
new
node
and
it
will
notice
this.
It
will
evict
a
random
replica
of
the
class
time,
because
this
is
fine,
just
a
Republika.
B
It
will
move
this
by
being
evicted.
Companies
will
automatically
move
it
to
a
new,
empty
pot,
node
and
then
the
Postgres
operator
will
see
okay.
Now
that
is
a
replica
on
a
new
node.
It
makes
sense
to
promote
this
to
master,
so
it
promotes
this,
and
then
we
did
a
control
failover
and
the
one
that
was
the
master
before
we
can
now
evict,
because
it's
no
longer
has
this
disruption
part
it
that
you
that
you
can
only
that
you
have
to
have
one
master.
B
This
is
already
satisfied
by
having
moved
it
there,
and
this
would
look
something
like
this,
so
you
have
min
available
one
when
you
have
a
label
grow
master.
The
basic
way
to
do
it
just
to
quickly
summarize
how
this
rolling
upgrade
of
nodes
looks.
So
we
have
an
opal
or
a
skin
group.
We
have
a
bunch
of
availability
zones,
we
have
some
instances
there.
B
We
have,
of
course,
the
parts
running
and
on
the
instances-
and
we
have
this
volumes
persistent
volumes
attached
to
each
of
these
part,
and
when
we
do
an
update,
we
basically
find
all
the
old
nodes
in
the
cluster
we
put
a
chain
prefer
no
schedule.
This
is
also
a
kubernetes
feature
where
you
can
say
if
there
are
nodes
available
with
without
this
chain,
then
prefer
to
schedule.
B
The
new
parts
through
to
these
new
nodes
and
then
we
bring
up
three
new
nodes
because
then
we
get
one
in
each
availability
zone
and
even
if
we
don't
get
one
in
each
like,
if
it's
unbalanced
in
some
way,
then
we
actually
wait.
So
we
have
at
least
one
new
note
in
each
availability
zone,
because
then
we,
if
we
move
parts
from
an
old
notes
when
you
want
them,
we
always
have.
We
will
always
be
able
to
attach
the
persistent
volume.
B
So
if
we
then
train
a
node,
then
we
will
company
we'll
move
the
parts
to
something
else
and
the
ones
that
have
the
persistent
volume
will
stay
in
the
same
zone,
but
the
others
without
a
volume
might
go
to
a
difference
from
we
could
also
stay
in
the
same
and
then
kubernetes
will.
Basically,
our
AWS
will
bring
up
a
new
instance,
because
we
changed
the
value
in
the
auto
scaling
group
and
then
yeah.
It
basically
continues
from
there
and
the
way
we
do
it
is
that
we
don't.
B
We
don't
fix
the
auto
scaling
group
size
anymore.
We
just
always
find
the
old
nodes.
So,
even
if
we're
in
the
middle
of
an
upgrade
and
our
controller
dies,
it
will
just
start
up
again
and
find
all
the
old
nodes
and
continue
from
that.
So
our
scaling
can
happen.
It
was
just
bring
up
new
notes
in
the
meantime,
and
then
we
actually
have
less
work
to
do
in
the
controller
yeah.
B
And
if
you
just
switch
the
configuration
in
the
flannel,
then
you
would
have
some
information
LCD
and
some
in
kubernetes
and
they
wouldn't
match.
So
our
idea
was
to
migrate
this
correctly,
and
this
really
fails
to
do
so.
Basically,
we
had
on
each
node
two
different
networks,
one
from
it'sa
demon
from
kubernetes,
so
the
paths
couldn't
route
anywhere
because
they
have
to
miss
matching
networks
and
the
reason
this
happened
was
that
we
basically
didn't
test
the
upgrade
path
that
I
was
talking
about
in
our
intern
chest.
B
We
only
created
a
new
cluster
from
the
new
configuration
and
this
worked
fine
or
we
didn't
actually
create
an
old,
Rasta,
updated
and
then
just
this
would
have
caught
this
issue,
another
one
just
not
really
kubernetes
rate
or
anything.
It
was
just
that
we
have
our
images
on
our
docker
registry
internal
one
and
when
we
were
confident
at
some
point
that
we
could
roll
out
more,
it
trust
us
at
a
time
we
still
had
like
five
at
a
time
we
could
deal
with.
B
B
If
the
notes
were
real
ready
or
not,
so
we
just
kept
rolling
out
new
notes
that
didn't
get
any
any
couplet
running
or
anything
so
yeah
we
basically
removed
everything
from
the
faster
kind
of
it
was
easy
to
stop
the
beating,
because
we
could
just
remove
the
controller,
because
it's
a
single
instance,
so
we
just
scale
it
down
to
zero
and
then
everything
stopped
or
we
could
let
the
the
the
registry
recover
yeah.
This
is
pretty
much.
It
I
just
believe
some
links
here
for
us,
our
own
source,
particulated,
so
kubernetes.