►
From YouTube: Highly Available Kubernetes Clusters - Best Practices - Meaghan Kjelland & Karan Goel, Google
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Highly Available Kubernetes Clusters - Best Practices - Meaghan Kjelland & Karan Goel, Google
Everyone running a Kubernetes cluster in production wants reliability and high availability. Many clusters may implement a multi-master setup, but often this is not enough to consider a cluster highly available. So how can you truly achieve a highly available, highly reliable, and multi-master kubernetes cluster? In this talk, we will cover exactly that. The audience will walk away with a solid understanding of what HA is, what failure domains should be considered based on their level of risk tolerance, and actionable knowledge that they can implement in their own organizations. This talk will include examples and lessons learned implementing HA in GKE and for the open source cluster-api, but we will focus on using this knowledge in a vendor and environment neutral way.
To Learn More: https://sched.co/GrWQ
A
So
for
both
of
us,
it's
our
first
keep
God
and
naturally
first
keep
contact
so
we're
we're
excited
big
room.
Lots
of
people,
hopefully
you'll,
take
back
all
of
the
things
we
tell
you
and
actually
implement
them
in
your
organization,
so
I
want
to
start
off
with
kind
of
a
not
quite
relevant,
to
keep
kubernetes
or
cube
con
project.
This
was
a
github
repo
of
Brad
Fitzpatrick,
another
Googler,
and
he
was
writing
about
his
home
lab
setup
and
his
home
internet
setup.
Some
of
you
might
have
seen
this
unhappy
news
and
his
goal
was
hey.
A
I
want
hive
highly
available
internet
access
for
my
home,
lab
and
I.
Don't
want
any
single
points
of
failure,
and
so
you
know
if
you've
built
any
labs
or
any
systems
at
home
or
in
your
organization.
You
might
know
that
there's
redundant
power,
supplies,
redundant,
Wi-Fi,
switches,
network
access
points
and
so
on,
but
I
also
saw
he
had
two
ISPs.
A
lot
of
us
don't
even
have
two
eyes
piece
to
choose
from
and
I
thought
about
it,
and
it
makes
sense
if
there
is
no
single
point
of
failure.
A
It
makes
sense
to
prevent
yourself
head
yourself
from
a
rat
chewing
through
your
fiber
and
going
with
a
broadband
as
a
secondary,
and
so
what
does
this
have
to
do
with
kubernetes
we're
at
coop
con
not
lab
con?
After
all,
so
kubernetes
is
some
of
you
might
know
it's
a
distributed
container
orchestration
system.
It
is
great.
A
It's
based
on
Google's
knowledge
of
after
running
billions
of
containers
every
single
week
on
a
distributed
planet
scale,
but
really
planet
skill
doesn't
really
mean
anything
if
you
don't
have
redundancy
and
high
availability
if
one
of
the
components
of
your
system
goes
down
and
that
causes
all
of
the
other
systems
to
not
be
available,
doesn't
matter
how
much
you
scale,
it's
not
good
for
you
so
and
I
get
a
feel
for
the
room
of
feel
for
the
room
here.
How
many
people
use
kubernetes
right
now,
one
or
two
people?
A
Okay,
and
how
many
people
have
actually
deployed
a
kubernetes
cluster,
not
using
a
management
service,
quite
a
few
ops
people
and
how
many
have
done
h,
a
cluster
setups
Wow
a
lot
of
people
great
and
how
many
of
you
have
been
asked
to
raise.
Your
hand
at
a
conference
before
cool
so
hi,
my
name
is
Karan
Goyal
I'm,
a
software
engineer
at
Google
and
here
in
Seattle
and
I
work
on
gke
on
Prem.
Some
of
you
might
know
about
that
and.
B
I'm,
making
challenge
I'm,
also
a
software
engineer
on
the
GK
on
Prem
team
at
Google
and
before
I
joined
this
team
I
was
on
a
different
team
at
Google
that
worked
on
the
Cloud
Foundry
container
runtime.
So
that's
another
kubernetes
project,
so
I've
been
working
in
the
kubernetes
space
for
about
two
years.
B
B
So,
for
example,
GK
has
an
H
a
product
called
GK,
a
regional
clusters
and
the
control
plane
of
those
clusters
is
available
in
ninety
nine
point:
nine
five
percent
of
the
time
so
great
now
we
know
high
availability
means
you
want
to
be
available
a
lot.
That's
not
super
helpful,
but
basically
what
we
want
to
talk
about
today
is
how
we're
going
to
achieve
that.
How
do
we
make
sure
that
our
system
has
no
failure,
points
that
could
take
down
our
whole
system
that
weren't
redundant?
B
B
Before
we
get
too
far
into
that,
I
want
to
talk
about
multi,
master
or
multi
controller
nodes.
This
means
that
you're
running
multiple
copies
of
your
control
plans
so
that,
if
any
of
the
copies
fail,
you
have
other
ones
to
come,
take
up
the
work
and
that's
great,
that's
a
really
important
part
of
high
availability.
But
in
a
lot
of
cases
it's
not
enough.
B
B
So
we
need
to
make
sure
that
we're
spreading
our
workloads
across
VMs
as
well,
and
similarly,
if
all
of
those
VMs
were
running
on
the
same
physical
machine,
then
we
can't
handle
a
failure
of
that
physical
machine
either.
So
we
want
to
run
ideally
across
physical
machines,
and
then
you
can
imagine
there
are
other
things
that
could
fail.
Also.
So
if
we're
running
three
physical
machines
but
they're
all
connected
to
the
same
power
source
or
they're
all
using
the
same
cooling
system,
we
can't
handle
failures
and
those
them
either.
B
We
also
want
to
make
sure
we
handle
network
partitions.
So
if
a
network
switch
fails
in
our
lab,
the
nodes
can't
communicate
with
each
other.
We
want
to
make
sure
we
know
what's
going
to
happen
in
the
system.
If
that
happens,
and
storage
is
a
really
critical
one.
If
we
lose
storage,
we
need
to
make
sure
we
don't
lose
the
data,
because
that
can
be
irreversible
on
GK
or
a
little
bit
lucky,
because
we
have
this
concept
of
regions
and
zones,
and
this
encapsulate
Sal
OTT
of
the
things
that
I
just
talked
about.
B
Regions
are
geographical
areas,
so
those
are
very
isolated
from
each
other.
If
you're
running
across
regions,
you
can
even
handle
like
natural
disasters
in
one
area
and
then
zones
are
a
deployment
area
within
a
region
that
is
independent,
they're
independent
from
each
other.
In
a
lot
of
ways
like
if
you're
running
across
zones,
then
you
know
you
have
different
power
sources.
B
You
know
you
have
different
cooling
systems
and
usually
they're
running
in
different
areas
of
the
datacenter
as
well,
and
the
reason
I'm
telling
you
this
is
because,
through
most
of
this,
talk
we're
going
to
use
zones
as
our
failure
domain
and
I
just
want
you
to
remember
that
that
already
encapsulate
Sal
OTT
of
these
failures
that
I
talked
about.
But
you
can
also
replicate
this
in
your
own
setup
if
you're
running
on
Prem,
okay,
so
we've
divided
our
talk
into
three
sections.
First,
we
have
applications.
B
How
do
we
run
applications
on
kubernetes
in
highly
available
way?
Then
we'll
talk
about
kubernetes
itself,
the
control
plane?
How
do
we
run
the
controller
manager,
the
API
server
and
the
scheduler
for
kubernetes
in
a
highly
available
way,
and
we
split
out
at
CD
as
a
special
part,
because
it's
our
data
layer,
we
want.
We
have
a
couple
of
extra
considerations.
There.
A
A
So
we're
gonna
assume
that
we're
running
this
cluster
with
three
replicas
of
our
workload,
the
green
boxes
are
VMs
and
the
orange
yellow
boxes
are
workloads,
containers
pods,
and
you
can
see
that
we're
running
our
VMs
across
three
different
zones
of
a
region
and
like
Megan,
said
these
happen
to
be
in
different
data
centers
across
different
physical
locations,
but
in
the
same
region.
So,
let's
assume
that
we
are
running
these
workloads,
they
are
scheduled
over
three
different
PN's
over
three
different
zones.
This
concept
of
scheduling
across
different
zones
is
not
native
and
primitive.
A
In
kubernetes
I'll
talk
in
a
little
bit
about
how
to
go
about
achieving
that,
and
let's
say
one
of
our
zones.
The
one
on
the
further
right
goes
down
we're
lucky,
because
our
control
plane
is
not
running
there.
In
this
case,
the
scheduler
will
see
that
hey
I
was
supposed
to
run
through
your
replicas,
but
there's
only
two
running.
So
let
me
just
get
you'll
schedule
one
more
and
bring
up
the
running
count
to
three
and
it
happens
to
land
in
zone
D.
A
So
this
is.
This
is
probably
what
you're
most
familiar
with
as
a
developer.
As
someone
who
is
deploying
the
workload
on
kubernetes
is
a
concept
of
a
deployment
or
a
replica
set
or
a
replica,
you
would
run
your
pod
as
a
deployment
and
you
would
set
the
replicas
in
our
case
to
be
three.
The
second
more
important
thing
here
is
rolling
updates
or
just
update
type
in
general.
A
This
is
this
is
a
great
use
of
out
of
out
of
the
box
functionality,
because
you
don't
need
to
write
your
own
machinery.
There
is
a
replication
controller,
that's
watching
for
changes
to
your
either
deployment,
manifest
or
updates
to
your
pots
themselves
and
reconciling
any
errors
that
happen.
Similar
concepts
translate
to
stateful
sets
for
stateful
applications.
You
get
stable
storage,
stable,
persistent
storage
and
stable
Network
identities,
and
we'll
talk
about
staple
sets
in
a
little
bit
more
detail
later.
A
The
next
thing
is
the
concept
of
setting
zones,
and,
you
might
remember
I
said
it
is
not
intrinsic
to
kubernetes,
but
if
you
do
use
a
kubernetes
cloud
provider,
you
get
that
feature
again
out
of
the
box
there.
But
if
you
don't
use
a
cloud
provider
and
you
want
to
do
it
by
hand,
you
need
to
do
two
things.
A
The
first
is
setting
pod
and
I
affinity
rules
to
your
pods,
like
the
one
on
screen
where
we
set
the
specific
failure
domain
key,
and
the
second
is,
you
would
add,
labels
to
each
of
your
nodes,
where
the
key
happens
to
be
that
failure
domain
key
and
the
value
can
be
anything.
So
in
our
case
it
would
be
us
central
one,
a
B,
C,
D
and
so
on.
Once
you
do
that
the
scheduler
will
schedule
all
of
your
workloads
across
those
zones.
But
again,
if
you
deploy
a
cloud
provider
that
will
just
happen
for
you.
A
So
next
thing
I
want
to
talk
about
in
touch
here
is
node
upgrades
and
specifically
other
types
of
voluntary
disruptions
that
you
might
want
to
do.
So
if
you
want
to
do
kernel
upgrades
on
your
machine,
you
will
need
to
drain
all
of
the
pods
there
and
maintenance
and
then
join
it
back
into
the
cluster.
A
But
if
you
decide
to
drain
it
drain
the
machine
all
of
the
replicas
and
all
of
the
pods
running
there
will
be
drained
at
the
same
time
and
it's
possible
that
all
of
your
pods
were
running
there
and
all
of
those
are
gone,
and
so
your
services
unavailable
to
get
over
that.
You
can
use
pod
disruption
budgets
where
you
can
set
again
your
tolerance
for
downtime
and
if
the
tolerance
here
is
not
met
by
any
trained
commands.
The
pod
eviction
API
will
just
reject
the
call.
A
B
B
Let's
look
at
what's
actually
happening
on
those
VMs.
We
have
the
API
server,
which
talks
to
@cd.
Again,
that's
going
to
be
in
the
next
section,
and
then
we
have
the
scheduler
and
the
controller
manager.
The
API
server
is
really
easy
to
run
multiple
replicas
of
because
it
is
completely
stateless.
We
can
just
run
it
as
active
active,
so
all
of
them
can
take
the
same
request
and
all
we
have
to
do
is
put
a
load
balancer
in
front
of
API
servers
and
we
should
be
fine.
B
The
scheduler
and
controller
manager
are
a
little
bit
more
complicated
because
they
have
to
read
data
and
then
act
on
the
data
and
write
data.
Also,
we
want
to
make
sure
we're
only
running
one
of
them
actively
at
a
time.
So
what
we
do
is
we
have
a
locking
system
they'll
all
attempt
to
apply
on
the
lock.
One
of
them
will
get
the
lock
and
that
one
will
become
the
active
component
and
then
the
other
two
will
just
wait
for
that
one
to
fail.
B
So
if
it
fails,
they
can
they
can
take
over
by
acquiring
the
lock.
In
order
to
configure
that
we
have.
There
are
five
slides
on
these
components.
Leader
elect
all
you
have
to
do
is
set
it
to
true,
and
then
all
of
this
will
work,
but
the
other
ones
are
around.
They.
Let
you
configure
how
long
it'll
take
for
one
of
the
passive
components
to
take
over
if
the
leader
fails.
B
Ok,
so
we've
talked
about
kind
of
like
the
multi
master
setup
now,
but
we
haven't
really
talked
about
a
lot
of
other
things
that
are
important
in
H
a
if
one
of
our
party
M
fails
like
if
we
had
a
zone
outage
nothing's
really
like
health
checking
to
make
sure
that
we
always
have
three
available,
and
we
don't
have
anything.
That's
gonna
handle
failure.
Recovery.
In
that
case,
we
also
haven't
talked
about
how
to
upgrade
without
downtime.
There
are
actually
a
lot
of
solutions
to
this
and
a
lot
of
options
you
can
explore.
B
Depending
on
what
you
need.
Of
course,
there
are
hosted
solutions
like
gk
e
gk.
Regional
clusters
will
do
this
for
you
already
or
gk
on
prem,
which
is
what
we
work
on.
You
can
use
managed
instance,
groups
to
make
sure
you
have
the
correct
number
of
machines
available
at
all
times.
That's
a
GCE
feature
also,
you
can
build
your
own
monitoring
server,
but
that
would
be
really
hard.
We
would
know,
and
then
there's
also
kuru
knows
itself.
B
So
there
are
kind
of
two
different
ways
you
can
use
kubernetes
to
do
this,
for
you
or
two
schools
of
thought,
I
guess
and
like
we
just
saw
in
the
application
section
of
the
talk.
Kubernetes
helped
us
in
a
lot
of
ways
to
manage
these
things.
For
us,
the
first
way
of
doing
this
is
called
self
hosting.
This
is
when
you
run
kubernetes,
and
it
manages
itself
as
pods
in
the
same
cluster.
This
is
gonna.
Have
some
interesting
bootstrapping
issues
or
certain
failures
can
be
hard
to
recover
from
does
anyone?
B
B
Well,
yeah,
so
if
you
want
to
try
this
out,
though
cube
admin,
has
it
as
an
experimental
feature,
so
you
can
try
to
plug
a
cluster.
This
way,
I
think
the
way
that
has
gained
a
little
bit
more
traction,
though
recently,
is
the
management
cluster
idea.
So
you
have
two
clusters
now
cluster
a
is
unmanaged
and
it
manages
a
bunch
of
clusters
in
this
case,
just
one
cluster
B,
so
cluster
a
is
control.
Plane
is
managing
cluster
B's
worklet
cluster
B
is
control
plane
as
pods,
so
you
can
see.
B
Cluster
a's
node
is
the
same
as
cluster
b's
controller,
node
and
then
cluster
B
can
just
manage
its
own
workloads,
but,
like
I,
said,
cluster
a
is
unmanaged
itself.
If
that
fails,
though,
the
control
plane
of
cluster
B
will
continue
running,
but
you'll
need
to
be
able
to
have
a
good
recovery
story
for
that.
B
Gke
solution
is
regional
clusters,
like
I've,
said
before
regional
clusters
will
automatically
run
the
master
nodes
across
three
zones:
it'll
run
three
replicas
of
the
control
plane
and
it'll,
put
a
global
load
balancer
in
front
of
the
API
servers
for
your
cluster.
Also,
one
of
the
benefits
of
GK
regional
clusters
is
that
Google's
sres
are
now
managing
your
compute
networking
and
storage
resources
for
you
there's
also
the
cluster
API.
This
does
machine
and
cluster
management
in
kubernetes
style,
API
objects.
We
worked
up.
B
A
Thank
you.
So
next
we
want
to
talk
about
our
data
plane,
which
in
this
case
kubernetes
case
is
at
CD
and
at
CD
is
a
little
bit
of
a
snowflake
here,
because
it
is
backed
by
a
database
and
for
writes
to
a
database
to
succeed
in
a
distributed
environment.
There
needs
to
be
a
strict
majority
to
elect
a
leader
that
can
for
writes
to
succeed.
A
So
what
I
want
to
talk
about
here
is
why
city's
the
reason
reason
that
at
least
in
gke
we
decided
to
go
with
three
masters
instead
of
say
two
or
four
or
even
higher
than
that,
so
at
CDs
basis
is
distributed.
Consensus
algorithm,
called
raft.
I
won't
go
into
details,
there's
papers
about
that.
The
equation
that
governs
that
series
operate
or
raft
in
general.
Is
this
simple
n
over
2
plus
1
equation?
A
In
that
case,
we
have,
one
of
our
master
of
one
of
them
is
down,
which
means
only
one
out
of
the
table:
50%
are
able
to
reach
a
quorum,
but
we
need
a
strict
51%
majority
here
and
so
rights
to
our
database
will
not
succeed.
So
we
can
understand
why
one
is
not
good,
because
one
goes
down.
You
have
nothing.
We
can
see
why
2
is
not
good.
One
of
them
goes
down.
We
have
no
tolerance
after
that,
but
why
three?
Why
not
4?
A
A
What
you
can
see
is
for
a
cluster
size
of
three
two
out
of
the
three
members
need
to
be
in
consensus:
that's
60,
6%
of
the
members,
but
when
we
jump
to
four
three
out
of
those
four
are
needed
to
reach
a
quorum:
that's
75
percent!
So
we
by
increasing
our
cluster
size
from
three
to
four
we've,
actually
decreased
our
probability
of
reaching
a
consensus.
But
in
the
last
column
you
can
see
that
our
fault
tolerance
is
exactly
the
same.
We
can
still
only
tolerate
one
machine
failure
from
the
same
table.
A
You
can
also
notice
that
odd
numbered
cluster
sizes
beyond
cluster
size
of
two
is
strictly
better
in
terms
of
a
of
reaching
quorum
and
be
providing
failure.
Tolerance
than
even-numbered,
so
you
can
say
odd-numbered-
provides
us
with
better
odds
of
having
a
highly
available
cluster.
Thank
you.
So
why
not?
Why
didn't
we
choose
five,
then,
because
five
is
better
than
three?
Well
then
it
comes
down
to
decisions
like
is
it
how
expensive
is
it
going
to
be
to
run
five
machines
versus
through
versus
three,
and
is
that
cost
justified?
A
So,
let's
see
what
happens
in
case
of
three
replicas
now
that
we've
established
that
it's
a
good
good
first
step
say
one
of
our
nodes
goes
down
again.
One
of
our
masters
is
down
for
upgrades.
In
that
case,
we
still
have
two
available
replicas
and
those
two
can
still
now
reach
to
reach
a
quorum
because
we
have
a
strict
majority.
So
rights
to
our
database
will
succeed.
Do
note
that
if,
during
this
upgrade
one
more
of
the
replicas
goes
down,
only
one
one
of
our
masters
is
up.
A
We,
our
rights,
will
not
succeed,
since
we
do
not
have
a
strict
majority.
So
if
you
are
upgrading
a
three
master,
three
cluster
size
cluster,
you
do
not
get
a
high
availability
cluster
during
an
upgrade
itself.
If
you
want
that
well,
go
to
the
next
odd
number
replica
size
like
five.
Now
you
can
tolerate
two
machine
failures
and
still
be
high
availability.
So
this
is
what
EDD
recommends
for
a
lot
of
its
really
cluster
sizes.
But
again
it
depends
on
your
business
needs.
A
A
Api
server
with
those
@cd
instances,
there
is
a
linked
example
manifest
on
screen,
so
you
can
go
back
and
look
at
that
and
because
that
CD
is
backed
by
a
data
store,
we
do
recommend
having
preparing
preparing
for
any
downtime
that
you
might
have
because
of
disk
failures.
Data,
corruption
and
the
biggest
one
is
backup
in
backup
and
restore.
A
We
recommend
having
periodic
backups
monitoring
that
those
backups
are
having
an
alerting
when
those
backups
are
not
happening,
especially
for
production
systems
and
if
you've
run.
If
you
run
production
systems
where
you
are
taking
backups,
you
definitely
have
to
test
to
restore
randomly
in
your
backups.
If
you
do
lose
your
data
due
to
some
mishap
and
you
are
not
able
to
restore
or
people
have
not
been
trained
on
how
to
restore
it,
your
backup
is
useless.
A
So
there
is
that
linked
document,
documentation,
page
on
how
to
do
this,
backup
and
restore,
and
how
to
also
recover
from
when
your
cluster
cannot
reach
quorum,
so
dump
some
tool
recommendations.
Hefty
Oh
has
Ark.
Some
people
might
have
used
that
before
it's
great,
it's
a
server
client
where
the
server
lives
on
your
cluster.
The
client
makes
HCD
calls
to
basically
iterate
through
all
of
the
objects,
save
them
in
a
data
store
and
you
can
recover
them
by
just
posting
them
back
to
the
API
server.
Do
you
note
that
this
requires
the
API
server?
A
B
So
what
we're
gonna
do
is
deploy
an
application
across
zones
on
a
cross
zone,
multi
master
cluster
and
then
we're
gonna
simulate
a
zone
failure.
So
our
cluster
will
look
like
this.
We
have
three
VMs
at
our
controller
nodes,
three
VMs
at
our
worker
nodes.
We
deployed
this
cluster
using
kubernetes
the
hard
way,
but
we
made
it
cross
zone,
which
is
one
of
the
only
differences
and
then
we're
gonna,
deploy
our
application
across
them
and
see
what
happens
when
we
delete
the
VMs
in
one
of
the
zones,
which
is
simulating
our
zone.
Failure.
B
B
So,
first
on
the
Left
we're
going
to
show
what
nodes
we
have
running
in
our
cluster.
The
left
is
cute
cuddle,
get
nodes
and
then
the
right
is
the
Google
Cloud
console
and
we
can
see
that
the
zone
labels
have
been
applied
to
all
of
these
nodes.
We
had
to
apply
them
ourselves
because
the
way
we
deployed
it
didn't
use
a
cloud
provider
but,
like
we
said
earlier,
if
you
used
a
cloud
provider,
this
would
be
applied
automatically.
So
you
can
see
where
zero
is
in
u.s.
B
west
1a
that
matches
over
here
then
in
the
cloud
console
we'll
look
at
the
load,
balancer
that's
sitting
in
front
of
our
controller
nodes
and
what
we
should
see
is
three
healthy
controller
nodes.
So,
yes,
we
have
three
now
we're
gonna,
deploy
our
application,
so
we're
gonna
watch
what
pods
we
have
running
in
our
cluster.
Currently
we
don't
have
any
then
we'll
create
this
simple
nginx
deployment
and
we'll
look
at
what's
in
that
deployment.
B
So,
first
we
can
see
that
the
notes
or
the
pods
are
all
scheduled
across
the
three
different
worker
nodes,
because
they're
in
three
different
zones
and
down
here,
you
can
see
that
we
have
this
preferred
during
scheduling,
pod
anti
affinity,
rule
which
uses
this
failure
domain
zone
top
alot
topology
key,
and
since
it's
preferred
it
will
reschedule
two
pods
into
the
same
if
it
has
to.
If
we
make
it
if
we
made
it
required,
then
it
wouldn't
do
that.
Okay,
now
we
can
delete
our
zone.
B
We
can't
actually
delete
a
zone
because
we
would
probably
get
fired,
but
so,
okay
on
the
bottom,
we're
looking
at
the
pods
that
are
running
in
the
cluster
and
you
can
see
which
worker
node
they're
running
on
on
the
top.
We're
gonna
look
at
our
nodes
and
we're
gonna
make
sure
that
they're
all
in
a
good
ready
state
and
we'll
see
what
zone
they're
running
in
again.
So
you
can
see
right
here,
they're
all
ready
right
now.
B
Okay,
now,
the
scary
part,
although
this
is
a
video,
so
it's
a
little
bit
less
scary,
we're
gonna,
delete
controllers,
0
and
can't
work
or
0,
which
are
both
running
in
uswest
1a
once
we
delete
them.
The
first
thing
we'll
see
is
over
here
we'll
have
a
slight
and
then
a
down
time,
but
we
should
see
it
automatically
recover
itself
the
reason.
Well,
let's
see
it
should
happen
pretty
fast
yeah,
so
we
can
see
our
calls
to
q4q
pedal
are
failing,
but
then
they
come
back
pretty
quickly.
B
The
reason
for
that
is
that
the
load
balancer
that's
sitting
in
front
of
the
3
API
servers,
has
a
health
check
on
it,
and
it
takes
about
two
seconds
to
recognize
if
one
of
them's
failing
which
I
configured
myself,
so
you
can
configure
the
time,
amount
that
it
takes
to
recover
okay.
So,
while
they're
deleting
what
we'll
see
first
I
believe
is
that
worker
0
will
become
not
ready.
We
also
changed
the
pot
eviction
timeout
when
we
deployed
the
cluster
to
make
it
a
victim
pods
quicker
when
it
realized.
One
of
them
is
not
ready.
B
So
now
our
nodes
are
deleted
and
we
can
see
this
one's
not
ready
and
now
down
here
we'll
see.
A
new
pod
is
scheduled
on
work
or
two
really
quickly,
because
we're
core
zero
is
now
no
longer
ready,
and
now
we
can
look
at
the
load
balancer
again
to
make
sure
that
we're
not
actually
sending
traffic
to
controllers
0
anymore.
We
can
see
what
that
looks
like
so
yeah
we
can
see
controllers
0
is
now
a
yellow,
exclamation
point,
because
it's
not
healthy
cool.
That's
all.
We
have
we've
linked
some
additional
resources
here.
B
One
is
for
general
clusters
what
you
talked
about
already.
We
also
linked
the
kubernetes
high
availability
Doc's
here,
Cabrini's
the
hard
way,
if
you
want
to
try
the
demo
yourself
and
then
also
the
cluster
API
cig
and
we'll
be
in
the
at
the
Google
cloud.
Booth
downstairs
in
the
exhibition
hall.
So
if
you
have
any
questions,
I
hope
all
of
you
come
find
us
there.
Thank
you
thank.