►
From YouTube: What Happens When Something Goes Wrong? On Kubernetes Reliability - Marek Grabowski & Tina Zhang
Description
What Happens When Something Goes Wrong? On Kubernetes Reliability [I] - Marek Grabowski & Tina Zhang, Google
One of the best features of the Kubernetes is that it can automatically recover from various failures and keep your application working despite unfavorable circumstances. There are moments when this works like magic and operators won't even notice something was going on. Sadly, sometimes automation fails.
In this talk we're going to describe various policies and mechanisms that are implemented in the system designed to keep user applications and cluster in general running. We'll talk both about things that will happen automatically and those that users need to configure.
About Marek Grabowski
Marek is a Software Engineer turned Site Reliability Engineer late 2017. Currently he focuses on reliability of Kubernetes clusters. Since 2013 he has been working on Google’s Technical Infrastructure, where early 2015 he joined Kubernetes engineering team. In Kubernetes his main focus was scalability and machine management. Before Kubernetes he was working on Google internal orchestrator in Omega project.
About Tina Zhang
Tina joined the Google as a Site Reliability Engineer for GKE in March 2017 and has primarily been working on delivering High Availability Masters in GKE, bringing GKE to more cloud regions and improving monitoring and alerting for the system. Prior to this, she had a previous life as an investment banker at J.P. Morgan.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
Welcome
everyone
to
the
last
session
of
cube
con,
and
we
want
to
thank
you
for
well
staying
till
the
very
end
of
the
conference.
We
are
monographs
key
and
Tina
Zang.
We
are
Rica
e,
a
series
from
Google
London
and
we're
here
to
tell
you
a
bit
about
what
kubernetes
does
when
something
goes
wrong.
What
happens
when
there
are
some
failures
in
the
system?
How
system
tries
to
fix
itself
or
mitigate
the
problems?
What
what
kind
of
problems
can
be
fixed
this
way
and
what
are
the
limits
of
this
automation?
A
We
would
like
to
ask
you
to
keep
your
questions
to
the
end
unless
we
will
tell
something
that's
completely
unclear
and
with
just
clarifying
so
before
we
start
I
want
to
tell
you
a
bit
more
who
we
are.
Some
of
you
may
be
already
familiar
with
the
site:
reliability,
engineering,
engineering
role,
but
for
ones
who
are
not,
this
is
how
our
VP
described.
He
described
it.
He
said
that
a
Sri
is
fundamentally
what
happens
when
you
ask
a
software
engineer
to
design
an
Operations
function.
A
The
something
here
is
that
software
engineers
are
inherently
lazy
and
do
not
like
doing
the
same
thing
multiple
multiple
times.
The
result
is
supposed
to
be
that
the
s-series
are
there
to
resolve
the
problem
when
they
occur,
but
then
make
sure
that
they
never
occur
again
or
if
they
do
doesn't
do
not
require
human
action.
A
A
A
B
So,
let's
start
with
the
basics,
one
of
the
first,
automated
healing
mechanisms
that
new
users
of
kubernetes
learn
about
is
creating
pods
as
part
of
a
replica
set
typically
in
a
deployment.
So
the
controller
manager
here
will
be
helping
you
automate
the
process
of
keeping
the
desired
number
of
replicas
alive
so
that,
even
when
your
pod
containers
keep
crashing
due
to
a
seg
fault
memory
leak
or
some
other
applications,
specific
bug,
they
can
recover
automatically.
B
So
the
process
that
is
restarting
any
crashed
containers
is,
of
course,
cubelet.
It
will
restart
a
crash
container
according
to
the
pods
restart
policy,
which
is
always
defaulted
to
always.
We
start
cubelet
also
contains
logic
to
restart
pods
on
an
exponential
back-off
delay.
Hence,
when
you
describe
pods
using
cube
control,
the
name
of
the
pod
status
is
called
crash,
loop
back
off.
So
how
is
it
doing?
B
This
cubelets
is
doing
this
by
interacting
with
docker
in
order
to
fulfill
its
responsibilities
of
actuated
the
desired
pod
spec
on
the
node,
in
the
interests
of
fairness,
other
container
runtimes
are
available
and
the
introduction
of
the
container
runs
interface
as
an
abstraction
layer.
A
few
years
ago
made
people
Nettie's
really
extensible
in
terms
of
how
it
starts
and
stops
containers,
but
because
kubernetes
is
not
tied
to
docker.
It's
not
actually
using
Dockers
native
health
checking
features
such
as
the
auto
restart
policy.
B
Instead,
cubelet
periodically
calls
docker
PS
to
list
all
containers,
compares
it
with
the
previous
output
and,
along
with
docker,
inspect
Gabon's
the
state
of
containers.
So
if
there
is
a
change,
Keibler
triggers
a
synchronization
event
with
respect
to
the
pod
spec.
Some
operating
systems,
such
as
the
container
optimized
operating
system
on
gke,
also
use
system
D
to
run
health
checks
on
both
cubelet
and
docker.
Every
minute,
by
using
the
cubelets,
help
the
endpoint
and
calling
docker
PS.
B
If
the
response
time
times
out,
the
relevant
process
is
killed
and
a
separate
system,
D
process
restarts
the
relevant
process.
Unfortunately,
this
means
that
the
docker
daemon
lifecycle
is
tied
to
the
container
lifecycle.
So
if
this
health
check
fails,
all
your
containers
get
killed,
there
is
a
docker
live,
restore
feature
that
allows
containers
to
keep
running
even
when
the
docker
demon
has
died,
and
we
are
currently
working
on
enabling
this
in
a
forthcoming
release
of
the
container
optimized
OS.
B
This
is
especially
important
if
you
have
a
workload
or
an
application
that
can
get
into
a
broken
state
without
crashing
the
container,
so
maybe
just
as
a
show
of
hands
how
many
people
have
heard
of
liveness
probes,
okay,
great
so
I'll
go
through
the
slide
really
quickly,
and
there
are
three
types
of
liveness
probes.
Actually
how
many
people
are
using
live
probes
in
prod?
Okay,
that's
good
to
see
well,
it
could
be.
It
could
be
higher.
I
think
how
many
people
are
using.
B
As
with
any
code
bug
in
the
liveness,
configuration
can
be
extremely
problematic
as
it
can
lead
to
crash
looping
pods,
but
hopefully
this
is
something
that's
easy
to
spot,
and
ideally
your
liveness
config
should
be
as
simple
as
possible
and
I
guess,
with
HTTP
and
TCP
probes.
If
something
does
go
wrong,
you
also
have
to
debug
at
the
networking
layer
too.
B
So,
just
briefly,
here
are
some
example
configurations
from
the
official
documentation,
and
this
is
what
you
would
insert
into
the
pod
spec.
So
for
people,
perhaps
less
familiar
with
HTTP
probes,
you
can
set
custom
headers.
You
can
also
tweak
the
timing
of
how
long
cubelet
should
wait
before
performing
the
first
probe
and
how
frequently
probes
should
occur.
You
can
also
customize
the
number
of
failures
that
would
count.
B
The
cubelet
should
count
before
seeing
a
pod,
restart
and
there's
also
such
a
thing
as
readiness
probes,
which
is
very
similar,
and
they
define
health
checks
that
must
be
passed
before.
Traffic
is
sent
to
pods
and
just
quickly
an
example
of
when
liveness
probes
can
go
wrong.
It
can
go
wrong
when
you
have
a
container
that
takes
a
long
time
to
start
up
and
your
initial
delay
seconds
is
set
to
shorts.
A
Thank
you
so
Tina
told
us
what
happened,
what
how
kubernetes
try
to
solve
the
issues
when
there's
some
problem
in
the
binary
itself,
I
basically
tries
to
restart
attack
if
the
liveness
proper
training
is
probe
like
blindness
process
is
failing,
but
that's
not
the
only
reason
why
this
service
might
be
broken
or
if
I
kid,
the
binder
itself
might
be
perfectly
fine
and
correctly
written,
but
the
service
still
might
not
be
responsive.
The
reason
might
be
that
the
pod,
the
binder
itself,
is
lives
in
a
unhealthy
environment.
A
They
it
might
have
some
noisy
neighbor
that
just
consumes
all
the
resources
of
the
machine
and
does
not
leave
enough
for,
for
the
part,
that's
supposed
to
be
serving
well
in
principle.
Containers
should
be
good,
like
actually
do
offer
rather
good
isolation
on
many
resource
and
dimensions,
but
we
are
usually
usually
not
using
it
like
very
strictly.
We
try
to
like
relax
this
isolation
a
bit.
A
The
reason
is:
if
we
allow
some
some,
we
allow
container
to
share
some
full
of
resources.
We
can
achieve
much
higher
utilization
in
the
cluster
right
like
if
the
few,
not
fuel
pots
they
can
just
like
burst.
If
they
need,
we
can
pack
more
of
them
in
the
under
single
now
than
if
we
just
like
limit
all
of
them
strictly
to
like
the
highest
possible
utilization.
A
Hence
we
have
the
quality
of
service
concept
in
the
kubernetes,
my
back
to
the
noisy,
neighbor,
the
way
the
kubernetes
shows
like
tells
the
user,
that
there
is
something
wrong
with
the
node.
It's
through
node
conditions,
the
two
of
them
that
are
connected
to
the
noise,
never
problem,
our
disk
pressure
and
memory
pressure
they
well.
They
of
course
mean
that
the
note
is
running
low
on
disk
or
memory.
Some
of
you
may
ask
why
not
CPU?
A
A
So
luckily,
this
this
is
a
problem
that,
like
computer
science,
like
a
solar,
dissolve
right
in
the
or
even
on
a
single
master.
Single
single
single
machines,
like
the
even
single
machines,
can
run
out
of
memory
and
kernel
knows
how
to
handle
that
it.
Basically
it
kills
stuff.
So
maybe
we
would
be.
It
would
be
okay
to
just
leave
it
to
kernel,
to
just
kill
something
on
the
machine
and
make
sure
that
it's
stable.
A
The
better
way
to
do
it
is
to
allow
cubelet
to
do
the
killing
because,
like
it's
know,
it
knows
what
can
be
killed
safely.
Hence
it
can
be
like
more
logical,
not
completely
random,
when
when
killing
things,
so
it
will
be
killing
pots,
because
that's
the
thing
it
oversees,
but
not
all
the
pots
can
be
killed
safely.
A
Right,
like
you
probably
don't
want
to
kill
Q
proxy,
because
if
you
do
you,
your
services
will
stop
working
right,
like
nothing
will
update
the
iptables,
which
may
because
of
that
we
made
the
Q
proxy
special
in
in
the
default
configuration
by
special
I
mean
it
is
it
possesses.
It
has
two
properties.
In
the
same
time,
it
has
a
critical
pata
notation.
This
is
a
notation
you
may
know
of.
A
It
basically
says
the
system
that
this
is
very
critical
thing
that
means
to
run,
and
it
is
allowed
to
kill
other
things
to
make
a
space
for
itself
and
it's
also
a
static
part,
which
means
it
runs
from
the
manifest
start
on
the
machine
on
the
note
itself,
instead
of
reading
it
from
the
API
server.
So
if
any
part
has
those
two
properties,
it
is
safe
from
evictions
than
done
by
cubelets.
A
A
If
you
have
I,
if
we
run
out
of
put
a
certain
budget,
the
cubit
will
ignore
this
and
kill
the
pot
despite
not,
but
it's
not
that
it
will
kill
anything.
It
tries
to
respect
quality
of
service
classes,
so
it
will
start
by
killing
the
best
effort
pot.
First,
if
it
ran
out
of
them,
it
would
and
it
should
kill
worst
of
all
thoughts
and
only
after
they're
gone
killed,
the
guaranteed
ones.
So
now
you
may
understand
why
we
need
to
ignore
put
a
certain
budget.
A
If
we
didn't
it
was
put,
it
would
be
possible
to
gain
the
system
by
saying,
like
really
low,
for
the
certain
budget
for
the
pot
in
a
best-effort
class,
and
this
way
forced
the
cubelet
to
kill
the
post
from
the
higher
hierarchy
oasis,
which
we
want
to
want
to
avoid,
of
course
like
killing,
as
in
as
in
the
case
of
the
killer,
it's
a
slider
random.
So
it
pick
you
but
my
failed
to
kill
the
noisy
neighbor.
A
B
Okay,
but
even
if
all
of
the
neighbors
of
your
pod
are
nice
and
friendly
and
don't
try
to
use
up
extra
resources,
your
pod
might
still
be
experiencing
some
problems,
and
even
if
you
have
liveness
probes
configured
correctly,
it
doesn't
matter
if
there's
something
wrong
at
the
machine
level.
So
to
make
problems
at
the
level
of
the
node
more
visible,
we
have
a
kubernetes
add-on
called
node
problem
detector.
B
This
is
a
demon
that
can
make
automatic
diagnosis
of
node
problems
and
report
them
to
the
API
server
using
event,
objects
for
temporary
problems
or
node
condition
objects
for
more
permanent
ones.
This
is
enabled
by
default
on
GCE,
but
you
can
also
run
it
as
a
daemon
set
or
on
a
standalone
basis,
and
because
no
problem
detector
is
a
daemon,
albeit
a
very
small
one.
It
does
use
resources,
but
benchmark
testing
was
done
to
make
sure
that
the
footprint
was
relatively
minimal
and
in
any
case
you
can
limit
its
impact
in
config.
B
So
this
slide
shows
the
architecture
of
the
node
problem
detector,
which,
as
with
many
kubernetes
components,
was
designed
with
the
principle
of
extensibility.
There
are
sub
daemons
of
the
node
problem,
detector
called
problem,
daemons
that
report
on
a
very
specific
class
of
problem.
We
currently
only
have
the
kernel
monitor,
which
is
monitoring
kernel
logs,
to
detect
known
issues,
in
particular
kernel
deadlock
by
matching
against
predefined
rules,
and
these
rules
are
customizable
and
can
be
extended
to
include
new
sets
of
problems.
B
However,
you
can
add
custom
plug-in
monitors
that
can
deal
with
infrastructure
hardware
or
container
runtime
issues
and
in
the
future.
The
idea
is
that
any
node
problems
reported
to
the
API
server
can
be
self-healed
with
a
remedy
controller
that
could
perhaps
drain
or
reboot
the
node
as
Jessie
from
github
mentioned
in
his
keynote
this
morning,
they
actually
have
an
internal,
only
tool
that
is
doing
this
called
node
problem
healer,
and
this
would
really
close
the
on
having
diagnosis
and
treatment
fully
automated.
We
just
need
to
make
it
open
source.
B
However,
if
your
node
is
not
accessible
due
to
networking
issues-
or
there
is
a
more
severe
serious
hardware
malfunction
such
that
the
node
becomes
partitioned
from
the
rest
of
the
cluster,
the
node
problem
detector
can't
help
you,
as
its
diagnosis,
will
not
even
reach
master
and
unfortunately,
we
would
have
no
way
of
knowing
the
difference
between
a
networking
failure
or
a
node
failure.
We
simply
know
that
the
node
is
unreachable.
So,
in
order
to
account
for
this
scenario,
kubernetes
has
the
node
controller.
B
A
You
okay,
so,
as
Tina
said
like,
if
machine
is
unreachable,
we
need
to
do
something
having
in
mind
that
the
most
important
thing
for
the
whole
system
is
to
keep
user
workers
running
right.
This
is
the
goal
of
the
whole.
This
whole
experiment
right.
We
want
to
make
sure
that
your
pod
runs
as
such,
well
as
as
far
as
possible
so
like.
If
the
machine
is
unresponsive
or
we
are
actually
reports
itself
as
unhealthy,
we
need
to
move
parts
from
this
machine
somewhere
else.
Well,
in
companies
were
world.
A
We
basically
just
kill
the
parts
from
this
machine
and
allow
the
replica
set
controller
or
any
other
controller
to
does
create
replacement
parts
with
the
end
again,
this
will
be
scheduled
by
the
scheduler
on
some
healthy
ones.
There's
one
detail
here:
demons
are
not
evicted
which
might
get
be
confusing
from
time
to
time.
This
is
because,
like
they
like,
even
if
we
have
agreements,
they
won't
go
anywhere.
They
are
like
assigned
to
lick
this
note
all
right
like
so.
This
is
how
demons
work.
They
are
attached
a
note
and
can't
be
scheduled
anywhere
else.
A
So
that's
the
simple
behavior
in
case
of
like
a
small
scale
problems.
But
what
happens
if
a
large
number
of
nodes
is
unhealthy?
Can
we
do
exactly
the
same
thing?
Well
kind
of
can
write
like
we
can
evacuate
all
the
unhealthy
nodes
and
try
to
put
the
pods
that
well
are
pending
for
the
for
the
time
being
on
the
healthy
ones,
but
it
may
turn
out
that
there's
not
enough
space
for
everyone
all
right.
A
This,
of
course,
might
be
the
desired
behavior
like
if
the
pots
are
really
unhealthy.
Right,
like
we
want
to
put
this
workload
somewhere,
but
if
it
wasn't
really
the
problem
with
nodes,
but
for
some
reason
the
nodes
were
like
disconnected
from
the
master,
but,
like
all
other
communication
was
perfectly
fine.
The
pots
are
running
there
right,
they're
still
running.
A
This
is
not
a
huge
deal,
because
until
the
nodes
will
hear
from
the
master,
they
won't
kill
the
pots
cause
like
they
don't
know
they
should,
but
after
they
reconnect
to
the
to
the
cluster,
they
will
all
in
the
same
time
notice
that
they
should
kill
the
pots
and
do
it
in
the
same
time,
which
will
cause
the
egg.
The
massive
pod
eviction,
which
is
usually
undesirable
because
it
takes
some
time
to
start
them
up
again.
A
Lack
of
luckily,
the
experience
shows
that
if
the
large
number
of
nodes
is
unhealthy,
it
is
quite
unlikely
that
this
is
a
problem
with
nodes.
Actually,
luckily,
exam
of
you
were
the
talk,
101
ways
of
breaking
covers
with
companies
clusters.
They
were
showing
exactly
this
kind
of
failures
which
were
caused
by
some
networking
issues,
all
right.
So
this
is
exactly
what
I'm
talking
about.
A
A
The
bad
thing
here
is
that
machines
are
much
faster
than
people
right,
like
node
controller
can
go
and
kill
the
pause
before
anyone
notices,
because
of
that
we
change
the
behavior
of
node
controller
and
start
throttling
the
killing.
If
we
observe,
like
the
large
number
of
parts
of
nodes
being
unhealthy,
I
believe
default
is
like
a
third
it
like
more
than
a
third
of
the
cluster,
is
unhealthy.
It
will
start
slowing
down.
It
won't
be
killing
anything
very
quickly,
like
pause,
very
quickly,
I
just
configure
with
a
flag.
A
The
default
normal
eviction
rate
is
like
one
note
per
10
seconds
so,
like
a
hundred
node
cluster
will
be
evicted
in
like
20
minutes
give
or
take,
and
the
slower
one
is
one
note
per
hundred
seconds
which
will
bump
it
to
like
over
three
hours,
which
is
probably
enough
time
for
human
to
go
and
see
what's
going
on
and
decide
of.
What's
the
correct
course
of
action
is,
of
course,
if
the
human
wander
is
like,
eventually,
all
the
pods
will
get
evicted
and
created
somewhere
else.
A
Okay,
so
now
I
will
tell
you
how
to
make
this
work,
for
you
even
better.
The
issue
with
the
thing
that
I
just
said
is
that
it
is
not
really
configurable.
There
are
four
knobs
that
are
exposed.
You
can
tell
the
system
how
quickly
it
should
start
to
mark
notice
unreachable,
so
stop
scheduling
new
post
there,
how
quick
quickly
it
should
start
evicting
pots
from
the
front
from
the
given
notes
how
at
what
rate
it
should
evict.
A
Parts
from
the
nodes
it
just
paints
notes
that
applies
thanks
to
the
nose
like
the
not
control
paints.
The
note
that
has
no
component
is
dot.
Io,
/
initial
bottom,
not
ready
depends
account
of
what's
the
condition
of
the
note
and
leaves
the
killing
to
the
change
controller,
exactly
the
same
taint
controller.
That
does
it
when
you
apply
when
you
do
keep
CTL
paint.
A
For
those
who
are
not
familiar
tense,
this
is
something
that
you
can
apply
to
notes
that
will
generally
prevent
thoughts
from
being
scheduled
and
those.
No
on
this
note
or
like
a
big
stuff,
that's
on
that
ransom,
then
on
them,
and
the
second
part
of
this
picture
is
toleration
switch
you
can
apply
to
pots
and
tell
them
that
they
can
ignore
some
of
those
things.
A
When
there
are
so
problems
that
the
note
has
some
problems
like,
you
can
like
specify
that
this
pot
needs
to
live
there
forever.
I
don't
want
to
affect
it
ever
or
this
part
needs
to
be
evicted
as
quickly
as
possible.
Just
like
I
will
set
depolarization
seconds
to
1
and
it
will
be
evicted
one
second,
after
the
note
will
turn
will
become
unreachable.
A
The
important
detail
here
is
that,
because
of
backward
compatibility,
the
kubernetes
will,
by
default,
applied
the
Toleration
's
equal
to',
like
did
the
water,
the
behavior,
that
you
can't
observe
so
like
the
like
4
minutes,
20
seconds
of
toleration
for
the
unreachable
or
whatever,
to
to
talk
well
to
keep
the
current
behavior,
which
means,
if
you
want
to
evict
quickly,
you
can't
just
don't
set
it.
You
need
to
set
it
to
1
because
like
if
it
will
be
unset.
Therefore
default
will
kick
in
and
and
just
like,
your
default
behavior.
A
Well,
of
course,
all
the
controllers
and
like
that
normal
binaries,
which
can
die,
be
killed,
and
we
also
want
to
upgrade
them,
which
means
the
every
single
controller
needs
to
be
able
to
start
from
any
States.
No
matter
how
broken
the
state
of
the
cluster
is,
it
needs
to
be
able
to
like
after
being
started.
Let's
make
it
better.
A
This
is
like
the
standard
like
micro
services
and
stateless
architecture,
but
because
of
that,
we
can
use
the
exactly
the
same
logic,
the
same
behavior
periodically,
to
just
like
pretend
that
the
controller
is
being
restarted
so
it
instead
of,
but
instead
to
make
the
a
bit
lighter
on
the
API
server
side.
So
do
not
just
kill
it
within
a
lot
of
lists.
A
It
will
just
like
IT
right
through
internal
States
and
pretend
it
sees
the
every
single
object
for
the
first
time
and
like
what
it
will
do
with
that
and
if
it's
wrong
it
will
fix
it.
If
it's
right,
it
will
just
like,
as
its
general
in
the
idempotent
like
it
won't
do
anything.
This
might
sound
like
not
very
powerful,
but
it's
actually
really
strong
saying
it
is
so
strong
that
it
masks
a
very
serious
back
in
the
service
controller.
A
For
like
a
number
of
releases,
the
buck
was
that
when
we
had,
we
had
this
like
node
port
service
type
service,
which
assigns
a
port
number
to
a
service,
and
then
whatever
traffic
comes
to
like
for
any
connection,
that
every
connection
that
gets
to
any
poor
node
in
the
cluster
on
this,
given
part,
it
gets
forwarded
to
the
service.
So,
of
course
it's
bad.
It's
like
the
same
port
numbers
assigned
to
multiple
services,
but
component
is
doing
what
doing
that
for
like
number
of
releases,
really.
A
The
reason
why
no
one
noticed
is
because,
like
after
pretty
soon
after
it
was,
it
happened
like
this
reconciliation.
Look
pretend
that
you
see
this
for
the
first
time
kicked
in
and
what
it
was
noticing
that
something's
wrong,
because
this
this
particular
jerk
was
correct,
like
we
have
I,
have
two
services
assigned
the
same
port
and
just
change,
one
of
them
and
system
load
itself.
B
Summarize
the
self-healing
features
that
we
have
covered
in
this
talk.
We
have
cubelets
on
each
nodes
and
monitoring
your
pods
waiting
to
restart
any
crashing
containers
or
when
liveness
probe
health
checks
are
failing.
The
node
problem
detector
helps
to
diagnose
kernel
issues
and
report
them
to
the
api
server.
B
Node
controller
is
performing
pod
evictions
when
nodes
fail
and
with
taints
and
toleration
available
for
you
to
provide
more
subtle,
customized
ways
to
redistribute
pods
when
this
happens
and
as
Ken
mentioned
in
her
keynote
yesterday,
these
automated
healing
mechanisms
are
really
part
of
the
superpowers
of
kubernetes,
but
even
with
these
surprisingly
powerful
mechanisms
that
can
handle
a
great
number
of
problems
automatically,
it's
not
perfect.
There
are
bugs.
There
are,
maybe
health
checking
heuristics
that
don't
cover
all
cases,
and
we
really
hope
that
this
talk
is
inspired
you
to
raise
it
with
the
community.
B
If
you
think
that
there
is
a
feature
that
should
be
handled
automatically
or
a
bug
that
needs
fixing
and
finally,
just
to
wrap
up
going
one
level
further
up
from
kubernetes,
your
favorite
cloud
provider
is
also
having
some
automated
features
when
you
run
a
managed
to
kubernetes
cluster.
As
a
user
of
gke,
so
in
the
context
of
this
talk,
gke
provides
automated
repairs
of
masters
whenever
component
statuses
become
unhealthy.
The
repair
potion
is
very
simple:
it's
very
straightforward,
but
it's
actually
very
effective.
B
B
We
also
provide
automatic
master
and
node
upgrades
making
sure
that
they're
operating
on
the
latest
stable
minor
version,
up-to-date
with
security
and
bug,
fixes
and
we've
even
been
known
to
upgrade
clusters
in
between
open
source
kubernetes
releases
to
cover
important
security
vulnerabilities
such
as
the
DNS
mask
vulnerability
that
was
identified
by
the
Google
security
team
in
October,
but
we're
developing
tools
all
the
time
to
make
problem,
diagnosis
and
automated
healing
processes
more
intelligent.
So
we
would
love
to
hear
your
feedback
and
with
that
I
think
we're
out
of
time.