►
From YouTube: Stories from the Playbook - Tina Zhang & Fred van den Driessche, Google (Any Skill Level)
Description
Want to view more sessions and keep the conversations going? Join us for KubeCon + CloudNativeCon North America in Seattle, December 11 - 13, 2018 (http://bit.ly/KCCNCNA18) or in Shanghai, November 14-15 (http://bit.ly/kccncchina18).
Stories from the Playbook - Tina Zhang & Fred van den Driessche, Google (Any Skill Level)
Have you ever wondered how GKE Site Reliability Engineers (SRE) manage an entire fleet of GKE clusters in 15 regions around the world? This talk provides an overview on how the SRE team approach this challenge, what tools are used, the problems encountered and war stories/learning experiences. The talk introduces the most frequently used parts of our playbook and how SRE endeavours to save your cluster while oncall in an effort to meet our SLOs.
"About Tina
Tina joined the Google as a Site Reliability Engineer for GKE in March 2017 and has primarily been working on delivering High Availability Masters in GKE, bringing GKE to more cloud regions and improving monitoring and alerting for the system. Prior to this, she had a previous life as an investment banker at J.P. Morgan. Previous tech speaking experience: Kubecon (US) 2017: What Happens When Something Goes Wrong? On Kubernetes Reliability (co-speaker with Marek Grabowski)
About Fred
Fred is an SRE at Google working on Google Kubernetes Engine, primarily focused on improving system observability, both at single cluster and fleet-wide levels. Previously he worked at Microsoft, writing and wrangling Java web apps for their Yammer product.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
Hi
everyone
and
thank
you
for
coming
to
stories
from
the
playbook
I'm
Tina,
and
this
is
Fred
and
we
are
Google
coupe,
Nettie's
engine
site,
reliability,
engineers
from
London
and
today
we're
here
to
talk
to
you
about
how
we
approach
the
challenge
of
managing
the
fleet
of
Google
kubernetes
engine
hosted
masters.
So
when
you
run
a
coop
Nettie's
cluster
on
gke,
we
take
care
of
the
master
VM,
making
sure
that
all
of
the
core
control
plane
components
are
up
and
running.
A
We
also
control
the
upgrade
cycle
of
the
master
VM
and
we
foot
the
bill
for
hosting
the
master
on
GCP.
So,
in
this
talk,
we're
hoping
to
give
you
an
insight
into
how
we
do
the
alerting
and
incident
management
for
gke,
as
well
as
give
an
overview
of
some
recent
outages,
most
commonly
seen
tickets,
which
might
be
interesting
for
you
as
a
gk
customer
or
some
general
good
practice
tips
if
you're
running
your
own
cluster,
but
before
we
dive
into
the
gke
part,
what
is
sre
so
site,
reliability,
engineering.
A
So
this
is
much
in
keeping
with
the
self-healing
aasif
ii
behind
kubernetes.
The
components
of
the
system
can
detect
and
heal
themselves
without
manual
intervention,
and
this
means
we
need
to
iterate
on
the
pages
and
the
tickets
that
we
see
to
make
sure
that
they
don't
happen
repeatedly.
So
for
gke
we
have
two
teams,
one
in
Seattle
and
one
in
London
providing
24/7
coverage
for
both
gke
and
also
appengine,
flexible
environment.
A
So
actually,
our
team
looks,
after
all
of
the
managed,
compute
products
on
Google
cloud
platform
and
we
on-boarded
gke
as
a
product
around
18
months
ago.
So
to
give
you
an
idea
of
some
of
the
challenges
we
face
in
numbers.
As
I
said,
we
are
we're
one
team,
but
we
have
two
shots
globally
and
we're
supporting
a
cloud
product.
That's
growing
exponentially,
both
in
terms
of
the
number
of
users
and
also
the
number
of
cloud
locations
and
we're
backed
by
an
open-source
project
with
more
than
a
thousand
contributors.
A
We
have
a
pretty
rapid
release
cycle
its
weekly
and
the
rollout
occurs
over
four
days
across
the
different
cloud
locations
and
we're
supporting
the
latest
3
minor,
kubernetes
versions.
So
in
almost
every
release,
we're
doing
one
of
the
following
things:
either:
making
a
new
minor
version
available
as
a
preview,
early
access
version
bumping
a
previous
preview
version
to
a
generally
available
version,
once
we've
seen
enough
stability
or
promoting
a
previously
generally
available
version
to
the
default
version
for
all
newly
created
clusters.
Again,
once
we
have
enough
confidence
or
mass
migrating
clusters
from
the
oldest
version.
A
In
order
to
maintain
this
sliding
window
of
three
minor
versions,
so
one
of
the
central
tenets
of
site,
reliability
engineering
is
that
the
headcount
of
SRS
should
not
grow
linearly
with
the
number
of
developers
or
the
number
of
new
features.
So
we're
going
to
try
to
talk
to
you
about
how
we
approach
the
challenge
of
maintaining
a
99.5
percent
availability
SLO
for
zonal
clusters.
In
light
of
all
of
these
moving
parts.
B
Okay,
so
you're
here
for
some
stories
from
the
playbook.
So
it's
probably
a
good
idea
to
make
sure
we
start
off
on
the
same
page.
Describe
briefly
what
a
playbook
actually
is.
I
was
hoping
the
sre
book
would
give
a
nice
6
ink
definition
of
what
that
is,
but
it
doesn't
say
much
beyond
this,
which
is
fairly
light
on
the
subject
talking
about
best
practices
and
reducing
your
mean
time
to
recovery.
B
So
if
we
look
at
GK
specifically,
we
have
a
player
country
for
every
alert
defined
on
the
system,
so
that
might
be
high
error
rate
on
an
API
server
or
a
cluster
being
unavailable
for
too
long,
and
that
entry
documents,
the
signs
and
symptoms
to
look
for
when
an
alert
fires
and
also
the
possible
procedures
to
help
mitigate
and
eventually
fix.
The
issue
play
books
are
a
great
way
of
sharing
information
between
engineers
on
a
team
and
also
reducing
the
cognitive
load
of
the
on
call
or
carrying
the
pedra
quick
and
fluent
ation
detail.
B
Our
play
books
are
hosted
in
a
number
of
redundant
places,
so
they're
easily
accessible,
even
if
a
wide.
If
there's
a
wide
outage
in
Google,
if
we
look
briefly
at
what
an
idealized
play
book
entry
might
look
like
in
this
case
city
with
HCD,
we
can
see
some
signs
and
symptoms
down
the
left
hand,
side
and
some
procedures
to
fix
them.
On
the
right
hand,
side
we're
going
to
come
back
to
this
in
more
detail,
so
I'll
move
on
for
now.
B
As
I
mentioned,
the
core
part
of
an
effective
player
country
is
the
signs
and
symptoms
to
look
for
when
fires,
and
for
that
you
need
some
good
insights
into
how
your
system
is
performing
in
the
case
of
gke.
We
have
we
care
mainly
about
the
health
of
the
GK
system
overall,
so
that's
the
systems
that
serve
the
cost
cluster
lifecycle
and
we
also
care
about
the
health
of
each
and
every
cluster
running
within
the
fleet.
B
So
we
gather
metrics
for
both
of
those
aspects
showing
the
request
rate
for
the
control,
plane,
API
so,
for
example,
the
load
on
a
particular
master
instance.
These
are
all
shown
on
dashboards.
So
again,
there's
one
for
each
cluster
control,
plane,
service
and
per
cluster
dashboard.
So
we
can
see
exactly
what's
going
on
on
each
individual
cluster
and
we
aggregate
those
all
together
to
create
a
fleet-wide
view
of
different
clusters,
so
we
can
see
what's
going
on
between
different
kubernetes
versions
and
also
across
different
cloud
regions.
B
Finally,
we
have
a
set
of
probers
and
these
give
a
critical
black
box
view
of
the
performance
of
the
system,
as
a
user
would
see
it.
So,
for
example,
in
gke
we
have
a
proba
that
continually
creates
and
deletes
clusters
and
make
sure
that
works
and
others
that
make
sure
metrics
are
being
exported.
As
expected,.
A
So
how
did
we
gather
the
data
that
affords
us
this
availability,
what
this
observability
into
the
availability
and
the
state
of
gke
clusters?
So
in
every
cloud
location,
we
run
a
job
outside
of
Google
cloud
platform
called
the
monitoring
server,
and
this
collects
metrics,
metadata
and
logs
from
each
cluster.
A
So
our
SLO
definition
of
Master
availability
is
based
on
the
core
component
statuses
returning
healthy
for
every
component,
and
if
availability
is
down
for
even
a
short
period
of
time,
the
monitoring
server
will
enact
an
automatic
repair
by
recreating
the
master
VM
and
also
based
on
the
number
of
nodes
that
the
cluster
is
running.
We
can
also
automatically
resize
the
master
VM
so
that
it
can
handle
higher
workloads.
A
If
these
actions
don't
work
and
availability
remains
down
for
too
long
or
if
some
other
metrics
reach
some
undesired
status
that
like
under
some
predefined
conditions,
such
as
at
CD,
backups,
running
to
old
or
master
VM
discs
to
full.
An
alert
is
sent
to
the
on
caller
so,
depending
on
the
severity
of
the
condition.
This
can
either
be
a
ticket
or
it
can
be
a
page
that
requires
an
immediate
response
and
this
might
be
location,
specific
version
specific
or
cluster
specific.
So
each
and
every
cluster
has
the
ability
to
page
us
individually.
A
But
it's
on
a
rare
execution
pass,
but
due
to
the
sheer
number
of
clusters
that
we
manage,
it
gets
triggered
by
an
edge
case
and
wakes
up
and
on
caller
with
a
page.
In
a
couple
of
cases,
our
on-call
was
able
to
submit
a
PR
to
fix
the
issue,
and
then
we
just
need
to
wait
for
a
new
kubernetes
patch
release
in
order
to
migrate
the
problematic
cluster
onto
it
and
I
think
we're
quite
lucky
in
SRE
that
so
many
Googlers
are
open-source
contributors.
B
This
little
robot
character
represents
the
service
accounts
which
GK
uses
to
interact
with
clusters
as
they're
running
and
there's
one
of
these
in
every
customer
project.
It's
also
possible
that
a
customer
can
disable
these
for
whatever
reason,
and
this
limits
the
ability
of
the
master
instances
to
manage
the
cluster
and
also
the
ability
of
the
control
plane
to
interact
with
the
cluster
in
general.
B
A
Then
fluently,
you
might
be
surprised
to
see
that
or
may
not
be
surprised
to
see
that
on
this
slide,
and
so
we
use
this
to
export
logs
from
both
the
master
and
the
node
onto
our
logging
API,
and
this
creates
an
alert
because
at
some
point
in
time
it
was
the
largest
user
of
CPU
and
even
though
it's
not
a
core
component
in
itself,
we
definitely
don't
want
it
to
starve
other
critical
processes
on
the
master.
But
it's
also
vital
that
we
have
these
logs
for
debugging
purposes.
B
We
also
see
quite
a
few
instances
of
clusters
becoming
unavailable
due
to
the
workload
that
is
being
run
on
them.
Often,
this
is
simply
due
to
some
load
testing,
where
our
customers
kicking
the
tires
a
little
too
vigorously
on
the
cluster.
Sometimes
it's
due
to
a
slightly
pathological
configuration
of
the
workload,
for
instance
one
namespace
per
pod,
or
simply
too
many
jobs
simultaneously
submitted
this.
A
Finally,
everyone's
favorite
distributed
key-value
store
at
CD.
Let's
go
back
to
the
playbook
example
that
Fred
showed
earlier,
so
this
is
obviously
not
a
comprehensive
example
of
a
playbook
entry
for
at
CD.
It's
a
snapshot
of
what
our
playbook
looked
like
when
we
were
experiencing
two
symptoms
repeatedly
and
I
think
it
was
the
most
common
cause
of
pages
a
few
months
ago,
so
first
of
all,
MVCC
database
size
exceeded.
A
So
this
is
referring
to
bolt
DB
and
the
multi-version
concurrency
control
feature
that's
backing
at
CD,
and
this
is
what
allows
that
CD
to
handle
one
right
transaction
and
many
reed
transactions
concurrently.
So
what
we
were
seeing
was
that
there
was
a
bug
in
the
version
of
@
CD
that
we
were
running,
which
meant
that
this
database
would
expand
and
become
increasingly
defragmented.
If
you
happen
to
have
a
reasonably
long,
read
transaction
during
which
many
writes
occur,
and
the
chances
of
this
really
increase.
A
If
your
cluster
workload
is
relatively
high,
so
short
term
fix
for
the
on
caller,
who
gets
paged
was
to,
of
course
check
the
playbook
country,
then
log
in
to
the
container
and
then
call
a
manual
at
CD,
compact
or
defrag
command
and
or
resize
the
machine
disk
longer-term.
The
fix
that
we
actually
went
forward
with
actually
run
an
extra
component
on
the
master
that
were
just
simply
periodically.
A
Effective
another
symptom
that
we
saw
a
few
times
was
crash
looping
at
CD,
and
this
was
triggered
by
bug
in
the
writer
head
log.
So
what
happened
was
that
once
at
CD
crashed
it
would
replay
it's
right
ahead?
Blog
processing,
all
log
entries
in
order
which
can
include
some
long-running
reads,
and
this
could
take
much
longer
than
the
15
initial
delay
seconds,
setting
that
we
had
for
the
at
CD
containers,
liveness
probe
health
check
setting,
and
this
meant
that
cubelet
would
cause
at
CD
to
restart
again
and
therefore
result
in
this
crash.
A
Looping
behavior-
and
this
was
fixed
by
the
on
caller-
simply
removing
the
initial
delay.
Second
setting
from
the
health
check,
and
this
meant
that
city
had
an
indefinitely
long
time
to
restart,
but
the
underlying
fix
was
made
in
a
later
version
of
a
vet
CD,
and
the
last
section
on
vida
election
issues
is
an
example
of
a
preemptive
playbook
entry.
So
in
December
2017
we
launched
high
availability
clusters
as
a
product
in
gke
in
beta,
and
this
is
where
a
cluster
is
backed
by
three
masters
across
three
different
zones
in
one
region.
A
B
Okay,
so
here
are
some
other
common
procedures
that
we'll
use
that
are
somewhat
generic
but
apply
in
a
few
cases.
The
first
is
simply
to
roll
back.
This
is
something
we
can
do
with
the
GK
control
plane.
Quite
simply,
if
there's
a
regression
introduced
with
the
new
release,
you
simply
go
back
to
the
old
one.
It's
less
easy
for
kubernetes
specific
issues.
B
If
we're
rolling
out
a
new
patch
version
or
a
new
minor
version,
we
can't
really
roll
that
back
because
we
might
be
taking
away
features
from
a
customer
or
reintroducing
bugs
that
were
fixed
in
that
new
patch
release.
So
often
what
we'll
do
is
cut
a
new
release
internally
with
the
fix
roll
that
out
and
then
upstream
the
patches
to
the
relevant,
open
source
repo.
B
Another
option
we
have
for
clusters.
Specific
alert
is
simply
to
resize
the
master
VM
that
are
used
for
that
cluster
to
a
larger
size
and
often
if
the
cluster
is
overloaded
and
that's
causing
health
issues,
the
larger
size
will
give
it
the
space
to
breathe
and
become
healthy.
Again,
we
also
have
the
ability
to
tweak
our
on
alert
is
configured.
We
use
this
sparingly,
but
if
we
think
an
alert
is
misfiring,
then
we
can
change
the
conditions
on
the
which
should
shoot
fire
and
hopefully
reduce
the
that
the
pager
is
making
that
way.
B
For
example,
recently
we
had
a
configured
such
that
if
a
single
request
that
it
was
monitoring
failed,
it
would
fire
that's
clearly
not
a
scalable
situation
to
be
in.
Finally,
we
also
have
the
option
to
simply
silence
the
page.
We'll
do
this
very
sparingly,
but
if
we've
investigated
the
alert
to
its
full
extent
and
believe
we
can't
fix
it,
then
we'll
silence
it
we'll
use
this
in
the
case
of
the
controller
managing
controller
manager
crash
looping,
for
example,
where
we
knew
that
was
a
fix
imminent,
but
there
was
nothing
we
could
do
at
the
time.
B
So
what
can
you
do
to
make
sure
your
cluster
stays
happy
and
any
on
caller
behind
it
stays
happier
as
well.
First
and
foremost,
it's
important
that
your
cluster
is
provisioned
for
the
workload
that
it's
expected
to
run.
This
goes
for
nodes,
but
also
the
manage
the
masters
if
you're
managing
those
inso
yourselves.
B
Also,
the
disks
backing
the
underlying
at
CD.
Databases
need
to
be
well
provisioned
if
there's
any
throttling
there
and
the
discs
become
a
bottleneck
that
can
be
detrimental
to
the
cluster
performance
as
well.
You
can
use
resource
limits
on
your
pod
definitions
to
make
sure
the
scheduler
can
do
a
good
job
of
not
overloading
the
cluster,
and
this
should
be
a
no-brainer,
but
frequent
at
CD.
Backups
is
a
definitely
a
good
idea.
We
do
this
in
the
order
of
minutes
on
every
cluster
and
there's
an
alert
that
goes
off
if
these
become
too
stale.
B
A
We've
talked
to
you
about
some
of
the
routine
alerts
and
where
there
is
a
solution
in
the
playbook,
but
sometimes
a
large
incident.
Large
impact
occurs
and
that
must
be
route
cost
for
the
first
time.
So
in
sorry,
for
incident
management
and
follow
a
mantra
of
coordinate,
communication
and
control,
and
we
have
an
established
protocol
in
a
crisis
so
on
caller's,
are
encouraged
to
pull
in
other
team
members
for
help
and
assign
defined
roles.
So
there
should
be
an
incident
commander
to
manage
and
coordinate
efforts,
operations
lead
and
a
communications
lead.
A
So
we
have
established
communications
channels
to
identify
the
impact
and
notify
customers
via
the
support
channels.
There's
also
a
scalation
paths
to
other
parts
of
cloud
and
internal
teams,
as
well
as
our
developers.
He
maintained
their
own
on-call
rotation
for
the
different
subsystems
within
gke
and
kubernetes.
Then
after
the
incidents,
it's
very
important
to
write
a
post-mortem
that
we
can
learn
from,
and
it's
key
that
it's
blameless.
B
B
This
started
with
the
release
of
a
new
OS
image
underlying
the
nodes
and
that
started
to
roll
out,
and
it
was
noticed
that
if
the
docker
daemon
restarted
on
that
node,
then
pods
would
no
longer
be
scheduled
on
it.
It's
obviously
not
great
for
the
health
of
the
cluster
its
workload,
so
the
first
thing
to
do
is
to
try
and
mitigate
the
problem,
and
in
this
case
it
was
quite
simple.
We
roll
back
to
using
the
previous
OS
image
for
new
nodes
and
downgrade
any
nodes
that
had
been
pushed
forward,
so
the
problems
mitigated.
B
But
we
still
don't
know
exactly
what
happened
and
that's
when
we
start
looking
for
a
root
cause.
So
we
knew
it
was
something
to
do
with
the
new
OS
image,
so
we're
able
to
essentially
diff
the
configurations
between
the
and
the
new
one,
and
that's
where
we
see
the
doc
alive
restore
flag.
Come
into
play
so
then
we're
pretty
suspicious
of
that
the
the
life
restore
flag.
So
we
roll
out
a
new
image
with
that
disabled
and
everything
seems
to
be
fine
again.
B
B
Obviously
what
went
badly
well
those
lack
of
testing
here
this
should
never
really
have
got
out
into
into
the
live
world
and
in
some
cases
where
customers
didn't
have
Auto
upgrade
enabled
we
had
to
ask
them
to
downgrade.
Specifically,
we
actually
got
very
lucky
here
as
well,
because
there
were
no
alerts
firing.
At
this
point,
we
were
notified
by
one
of
our
node
team
engineers
that
this
was
an
issue
and,
yes,
we
were
able
to
solve
it
from
their
notification.
A
This
outage
was
one
of
our
greatest
hits,
arguably
one
of
the
longest
and
most
visible
outages
for
gke
so
far
fingers
crossed.
So
this
started
off
with
our
create
cluster
probers
for
the
entire
europe
geo
firing,
as
in
they
were
not
being
able
to
successfully
create
clusters,
and
our
logs
indicated
that
this
was
a
real
customer
issue
and
not
just
a
problem
with
our
prober
projects.
So
some
initial
investigation
showed
that
no
other
Google
cloud
platform
services
were
affected,
so
they
once
so.
It
wasn't
an
issue
with
the
network
or
quota.
A
It
was
a
gke,
isolated
incident.
Cluster
creation
was
failing
when
nodes
were
attempting
to
register
with
the
master
and
after
a
few
more
hours
of
root,
causing.
We
finally
found
that
the
error
message
indicated
that
the
certificate
signing
module
was
the
culprit,
so
the
certificate
signer
was
trying
to
pull
an
external
image
from
docker
hub
also
appeared,
and
the
image
was
corrupted.
So,
in
the
absence
of
an
escalation
path
to
docker
hub
asari,
we
first
explored
the
mitigation
option
of
configuring
clusters
to
pull
from
a
different
location
where
images
were
still
healthy.
A
A
So
in
terms
of
what
went
well,
we
had
pretty
good
metrics
that
assess
the
proportion
of
the
impact
and
unfortunately
it
was
pretty
large.
We
were
able
to
pull
in
help
from
the
incident
response
team,
who
are
an
internal
team
of
experienced
Google
sres
that
help
provide
emergency
assistance
in
the
case
of
a
major
outage,
and
we
were
pretty
lucky
that
once
we
root
cause
the
problem,
the
Google
container
registry
team
had
a
big
button
mitigation.
A
That
meant
we
were
able
to
easily
purge
the
problematic
image.
So
in
terms
of
what
went
badly
so
gke
has
many
dependencies,
which
meant
lots
of
paths
to
debug,
so
it
actually
took
quite
a
long
time
to
find
the
log
entry
that
suggested
it
was
the
certificate
signer
at
fault.
This
means
that
our
most
significant
action
item
was
to
create
a
more
comprehensive
list
of
our
dependencies.
A
This
is
still
quite
long
and
create
monitoring
that
shows
the
steps
with
success
and
failure,
while
we're
bootstrapping
a
cluster
and
even
though
we
didn't
need
the
binary
rebuild
process,
we
found
that
we
needed
to
create
a
fast-track
emergency
rollout
workflow.
So
in
case
of
any
future
outages,
we
can
roll
out
a
fix
in
a
more
speedy
fashion.
B
The
final
incident
we're
going
to
talk
about
is
to
do
with
the
NS
mask
vulnerabilities.
It
fits
really
poorly
into
this
instant
format
that
we
came
up
with,
but
it's
an
interesting
one
to
talk
about.
Nonetheless,
but
a
background
to
start
with
DNS
mask
runs
as
part
of
the
cube
DNS
add-on
on
every
class
of
the
GK
manages,
and
we
were
notified
by
our
internal
security
team
that
there
were
a
number
of
vulnerabilities
in
DNS
masks.
B
From
our
point
of
view,
there
were
no
symptoms,
that's
kind
of
a
given
with
security
vulnerabilities
like
this,
and
there
was
no
real
mitigation
here.
We
couldn't
really
roll
back
to
a
previous
version
of
DNS
mask,
which
would
be
our
usual
strategy
and
we
couldn't
stop
DNS
mask
running
on
all
of
our
clusters,
because
that
would
probably
make
no
one
happy.
B
So
what
did
we
do?
We
cut
GK,
specific
kubernetes
release
versions
with
fixes
included
in
them
and
rolled
those
out
just
before
these
vulnerabilities
were
publicly
disclosed.
We
had
to
do
this
internally
because
we
couldn't
leak
the
presence
of
these
vulnerabilities
before
public
disclosure,
but
after
we
were
able
to
upstream
the
patches
to
the
relevant
repos.
B
B
B
The
method
of
creating
these
patch
releases
is
pretty
heavyweight
as
well.
We
had
to
do
it
for
every
version
that
we
were
managing
and
it's
a
pretty
slow
process
just
to
change
the
version
essentially
of
one
container
running
across
our
cluster
masters.
So
when
I'm
currently
working
on
making
that
more
easy
to
change.
Of
course,
we've
got
incredibly
lucky
here
that
a
we
were
able
to
pull
this
out
with
no
one
noticing
and
without
Lee
the
presence
of
it
all
so
that
it
wasn't
leaked
by
someone
else.
B
A
So,
finally,
when
we're
not
putting
out
fires,
what
are
we
doing
so
right
now,
we're
very
focused
on
making
sure
that
the
support
model
for
gke
scales
as
the
products
continues
to
experience
exponential
growth
in
usage,
one
of
the
ways
that
we're
looking
into
doing
that
is
implementing
a
problem,
fingerprinting
tool
that
will
identify
a
smoking
gun
from
panics
and
stack
traces
for
reoccurring
issues
across
the
fleet
and,
as
discussed
earlier.
Our
standard
repair
procedure
is
an
entire
master.
Vm
recreate.
A
So
while
this
is
very
effective,
it's
a
very
big
hammer
and
we
want
to
do
this
in
the
smarter
way
and
really
pinpoint
the
individual
failing
components,
and
we
also
want
to
make
the
resizing
of
the
master
vm
a
little
bit
smarter
too
we're
looking
into
using
the
cloud
recommendations,
API
that
comes
with
Google
cloud
platform
that
gives
suggestions
on
the
CPU
and
memory
usage
and
rather
than
using
a
heuristic.
That's
based
on
the
number
of
nodes,
we're
also
working
on
the
private
clusters
product
which
launched
in
beta
recently.
A
So
the
challenge
here
for
sorries
is
to
again
provide
a
similar
gke
support
experience
when
a
customer's
cluster
is
not
visible
on
the
public
Internet.
And
finally,
we
really
we're
looking
into
how
we
can
contribute
to
cluster
API.
So
this
is
an
open
source,
kubernetes
project
and
it's
an
initiative
to
introduce
some
management
api's
to
enable
common
cluster
lifecycle
operations
such
as
install
upgrade
repair,
delete
for
masters
and
and
I.
A
A
So
if
you're
interested
in
any
of
these-
and
please
do
come
and
talk
to
us
after
we'll
hang
around
for
a
while
after
the
talk-
and
you
can
find
us
at
the
Google
cloud,
booth
later
and
we'd
also
really
be
interested
to
hear
about
any
sharp
edges
or
outages
and
learning
experiences.
You've
experienced
and
perhaps
howhow
you
guys
do
alerting
a
modern
Trinity
so
and
thanks
for
listening
and
if
we
have
time
we
can
take
a
couple
of
questions
or
we
don't
want
to
keep
you
from
from
lunch.