►
From YouTube: 101 Ways to “Break and Recover” Kubernetes Cluster - Suresh Visvanathan & Nandhakumar Venkatachalam
Description
Want to view more sessions and keep the conversations going? Join us for KubeCon + CloudNativeCon North America in Seattle, December 11 - 13, 2018 (http://bit.ly/KCCNCNA18) or in Shanghai, November 14-15 (http://bit.ly/kccncchina18).
101 Ways to “Break and Recover” Kubernetes Cluster - Suresh Visvanathan & Nandhakumar Venkatachalam, Oath (Yahoo) (Intermediate Skill Level)
At Oath (yahoo), we operate one of biggest on-prem kubernetes cluster. We manage about dozen kubernetes clusters and 7000+ pods are running across the dozen clusters, major application such as Yahoo Sports, Yahoo Finance are powered by kubernetes. Managing at scale has its own challenges, in this talk we will cover 101 Ways to “break and recover” kubernetes cluster.
About Suresh
Suresh Visvanathan, Sr Architect, has over 13 years of experience in IT and Software. Suresh’s current responsibilities include the architecture, vision, strategy and design of cloud platform as-a-service (PaaS). Suresh has been architecting solutions and building products around resilience engineering, Auto recovery Systems and is a frequent speaker at K8s meetup, Chaos Community Events and speaker at kubecon 2017.
About Nandhakumar
Nandhakumar Venkatachalam is a Princ Production Engineer, Lead for Kubernetes Infrastructure/ Cluster management team at Oath Media Build and Products. He is a subject matter expert and solution architect specialized in high availability. Nandha has been under Oath for 11 years and has been dealing with operations at scale for more than a decade. He has delivered and led high profile product launches for Yahoo Fantasy, Yahoo Sports and contributed to flawless Fantasy season year after year. Prior to Yahoo, he had worked as a Linux System Administrator at IBM.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
Good
afternoon,
thanks
for
joining,
my
name
is
Suresh
and
era
of
engineering
at
what-
and
this
is
under
Kumar
he's
Michael
a
part
of
my
team.
How
many
of
you
know
about
what?
What
is
what
good
someone
is
facing
my
hand
happy
about
that.
So
what
is
a
entity
which
was
formed
after
Verizon?
What
a
while
Yahoo
and
bunch
of
other
companies
and
merged
into
one
unit
called
oath,
so
technically
I'm
an
exe
hago.
Now,
today,
we'll
go
through
our
experience.
We
have
been
running
kubernetes
for
almost
two
years
now
we
will
share
our
experience.
A
What
are
the
outages
issue
which
we
run
into
and
then
what's
a
lesson
we
learned
out
of
it.
So
I'm
part
of
the
core
platform
team
I
lead
the
kubernetes
effort
at
what
most
of
media
products
such
as
sports
finance,
and
you
can
talk
homepage.
All
these
stacks
are
currently
sold
out
of
kubernetes.
Okay,
if
you
access
yahoo.com,
page
or
sports
page,
it's
completely
sold
out
of
kubernetes
tack
today,
so
my
team
manages
the
kubernetes
cluster
and
the
tools
are
on
kubernetes
cluster.
A
A
This
is
my
team.
You
guys
see
the
flag
kubernetes
flag,
we
really
love
kubernetes.
I
cannot
explain
more
than
this
flag
right.
Bunch
of
guys
are
missing
here,
but
pretty
much
anti
team
is
here
and
you
can
see
in
here.
We
have
a
fan,
a
dashboard
update,
so
we
extensively
use
graph
and
primitives
for
monitoring.
Tomorrow
we
have
a
talk
about
how
we
leverage
the
primitives
for
monitoring
or
2:45
please
to
join
the
session.
A
Okay,
where
are
the
Chile
right?
We
are
in
good
production,
like
most
of
our
applications
are
running
in
production
with
communities.
Today
we
have
12
clusters,
production
grade
cluster
distributed
across
six
regions
and
we
have
any
caste
routing
which
routes
a
road
user
to
the
process
data
center,
based
on
where
he
comes
from,
and
we
have
12
to
k
plus
welcome
nodes
in
a
cluster
and
20k
parts.
A
50K
containers
running
out
there
and
if
we
have
200
place
application
running
here
and
at
the
peak
pretty
much
we
do
is
folder
decay
RPS
made
this
particular
stack
does
and
we
do
believe
most
of
the
application
which
deployed
today
is
faithless.
When
I
mean
stateless,
it's
it's
not
storing
any
state
7
its
access,
yahoo.com
page,
there's
no
state
stored
there
right
and
we
are
exploring
stateful
applications
such
as
Redis
cluster
ring
and
Cassandra
as
we
speak,
and
we
will
definitely
share
more.
In
upcoming
cube
con
conference.
A
A
We
have
something,
but
the
orange
boxes
are
the
one
which
the
component
which
we
built
for
bringing
in
kubernetes
to
it
and
the
one
the
green
are
there
existing
both
services
which
we
live
reached
and
the
other
thing
you
pretty
much
know
this
is
from
kubernetes
stack,
kublai,
the
api
edulearn
and
things
let's
go
through
to
our
right.
We
have
custom
ingress
controller
right,
so
you
pretty
much
know
in
genetics
and
enjoy
and
things
here
we
use
Apache
traffic
server,
so
apache
trail
mix
over
is
we
have
so
much
dependency
on
apache
traffic
server.
A
It
used
to
be
a
go
traffic
server
when
it's
open
sourced
and
apache
license.
It
become
apache
traffic
circles.
We
have
lots
of
plugin,
which
we
have
dependency
on
traffic
server,
so
we
have
to
build
our
color
controller
english
controller
based
on
opposite
effects.
Allah
over
here.
So
you
can
see
this
there's
an
agent
running
in
the
controller.
What
is
there
to
do?
Is
it
like
to
look
for
the
kubernetes
api
it
hooks
to
kubernetes?
A
Api
looks
for
the
endpoints
gets
those
IPS
for
the
origin
for
the,
where
you
are
throughout
the
wreckage
and
pretty
much
dynamically
loaded
into
that
ATS.
It
does
all
in
memory,
so
you
can
think
of
here
like
we
have
further
decay
RPS
coming
in
and
there
are
parts
getting
had
that
removed
deployments
are
happening
that
IPS
are
getting
changed.
We
need
to
seamlessly
lower
this
information
to
the
ingress
layer.
We
cannot
reload
harvest
at
the
ingress
was
pretty
much
it
will.
A
It
will
kind
of
impact
your
life
traffic
energy,
a
long
tail
latency
will
go
up,
so
we
pretty
much
build
this
ingress
such
a
way
that
we
do
in
memory,
so
there's
no
impact
to
the
live
traffic.
So
that's
on
the
ingress
controller
which
you
have
built
and
if
you
look
at
your
left
over
here
custom
tool,
let's
talk
about
screwdriver
screwdriver
is
an
open
source
project
from
Earth.
This
is
our
CI
CD
platform.
So
we
build
custom
tools
on
top
of
screwdriver.
A
What
this
custom
tool
does
right
when
we
want
to
bring
kubernetes
to
earth
right,
we
don't
want
every
engineer
to
completely
understand
in
their
kubernetes
spec.
So
we
provided
a
simple
interface,
a
single
UML
file
for
end-users,
so
in
that
AML
file,
he
pretty
much
says
that
I
want
to
apply
this
particular
image.
That's
it
right
and
then
some
same
value
in
terms
of
CPU
and
memory
and
what
we
do.
We
have
our
own
templating
engine.
A
We
take
that
file
a
simple
interface
and
we
convert
into
a
bunch
of
common
additional
files
that
much
of
kubernetes
KML
files
like
in
service
deployment,
and
then
we
natively
apply
using
cube
cutter
to
any
cluster
which
we
built.
That
is
the
customs
tool
which
we
built.
This
enabled
us
to
rapid
onboarding
to
kubernetes,
because
the
interface
was
very
simple.
The
engineer
could
understand
the
simple
interface
rather
than
forcing
an
engineer
to
understand
every
aspect
of
kubernetes
and
all
the
resource
of
communities
and
while
we're
here,
we
build
an
earth
hook.
A
This
is
our
authentication
operation
hook,
it
is
integrated
with
Athens
is
pretty
much
open
sourced
as
well,
and
you
can
find
it
out.
Nathan's
is
similar
to
I.
Am
an
item
Leah's
fault,
so
this
who
pretty
much
when
you
exude
any
command
against
the
namespace
in
use,
or
that
means
it
checks
whether
you
are
authenticated.
Have
you
seen
a
valid?
We
have
our
own
attenuation
mechanism
and
it's
checks
again
whether
you
are
authorized
to
do
that.
A
Action
on
that
particular
resource
on
that
particular
namespace,
this
level
of
ISIL,
a
chronology
of
policy
which
you
can
define
in
Athens.
So
these
are
pretty
much
common,
which
we
built
and
leverage
to
get
kubernetes
into
what
so,
let's
go
through?
How
do
we
build
pipeline
right?
So
our
base
kubernetes
cluster
itself
right?
It's
built
using
Jeff,
so
I
was
remaining
everything
if
I
can
add
ons
and
application
developer
who's
deploying
their
application.
They
use
screwdriver
and
the
custom
tools,
but
the
base
kubernetes
cluster
is
used,
build
using
Jeff.
A
So
if
we
pretty
much
run
the
writer
Jeff
Wenberg
and
we've
pushed
the
changes
to
stage
be
baked
in
there,
and
then
we
push
the
changes
to
Kennedy
when
Kennedy
we
push
the
change
to
production,
it's
like
if
you
need
anything
to
override
Pacific
environment.
You
can
go
to
an
update
in
that
particular
environment,
the
override
files.
So
this
is
how
we
change
the
kubernetes
changes.
The
criminals
based
community
changes.
A
Okay,
we
are
operating
on
prompt
right,
Javier
anak
in
decals
are
aks,
so
this
pretty
much
my
team
managing
this
covenant
is
awesome.
We
all
love
it
right,
I
on
fact,
I
always
said:
I
really
kissed
kubernetes
all
the
day
right
so,
but
there
are
many
interdependent
components:
it's
which
is
very
hard
right
now,
when
you
try
to
operate
kubernetes
on
your
own
at
the
scale
right,
you
can
talk
about
etcd
controllers
edge,
Euler
API
cube
blade,
Tucker
runtimes.
What
are
not
now
there
are
so
many
control
loops
which
you
need
to
understand.
A
When
you
tries
to
operate
that
okay,
then,
when
we
try
to
manage-
and
we
try
to
scale,
we
ran
into
many
issues.
It
could
be
human
error
governor
this
bug.
Then
how
did
we
fail
and
how
did
we
recover?
We
will
go
through
that
now,
one
of
the
first
human
error
right
namespace,
right
I,
told
this-
is
a
single
cluster
right.
This
is
soft,
multi
tennis,
if
you
say
when
I
say
soft
multi-tenancy
today,
it's
like
there's
not
very
this
is
a
team
of
bigger
team
right.
Every
team
gets
their
own
name.
A
So
if
they
don't
get
a
separate
cluster,
they
get
their
own
namespace
for
them
to
operate
right.
So
only
it
is
sensitive
workloads.
Then
we
have
a
separate
cluster
for
them.
If
not
it's
pretty
much,
they
share
the
underlying
resource
with
the
namespace
isolation
right.
So
in
a
given
name
space
you
have
100
hundred
plus
deployments
going
on.
So
what
is
this
command
say?
Gibson
give
CTL
delete
as
an
F
directory.
You
are
deleting
pretty
much
whatever
there
in
that
particular
directory
all
the
resource.
A
So
one
of
the
cluster
I'd
mean
he
was
trying
to
operate
against
non
production
environment.
His
context
was
set
to
non
production,
ok
and
then
he
he
he
was
thinking
that
it
was
said
to
be
non
production
and
he
tried
to
delete
this
directory
right
and
sorry
that
he
was
pointing
to
production
and
he
executed.
This
comment
against
our
production
system
right
in
this
case,
one
of
the
file
in
that
directory
was
name
space,
so
you
pretty
much
deleted
a
namespace
which
had
100
plus
deployment
when
you
deploy
when
you
delete
a
namespace.
What
happens.
A
You
guys
know
that
is
pretty
much.
It
wipes
out
what
it's
trying
to
do.
It's
like!
Ok,
yeah
here
you
go.
It
has
a
cascading
impact
like
what
happened
was
like
all
the
deployment
and
part
everything
within
that
namespace
came
down
like
everything
was
deleted
right
and
there
is
no
way
for
us
to
stop
that,
because
the
command
has
been
executed
right
and
now
like
hey.
How
do
we
recover?
That
is?
The
impact
has
been
done.
This
is
like
you
executed
the
command.
A
The
deletion
is
happening
for
the
pause
and
deployment
and
all
the
resource
within
that
namespace,
and
then
the
namespace
finalize
our
weekly
in
that
namespace
right,
no,
nothing.
We
can
be
done.
It's
kind
of
complete
outage
what
that
particular
name
says
which
was
serving
a
critical
workload
around
that
particular
data
center.
It
was
complete
out
a
shortage
right.
How
did
we
recover?
Ok,
we
do,
as
I
was
mentioning.
We
failed
out
that
particular
data
center,
so
the
user
can
go
to
the
next
data
center
available.
A
It's
easy
quick
way
to
do
that,
and
then
we
worked
with
our
tenant,
saying
that
hey
we
apologize
how
we
made
this
mistake.
We
deleted
your
name
space
and
your
deployment.
Can
you
please
read
trigger
your
deployment
and
they
went
and
retrigger
their
deployment
and
the
things
came
in
right,
but
if
the
IRB's
are
like
hey,
this
accident
could
happen.
How
do
we
prevent
it
from
happening
right?
So
we
have
admission
control
in
kubernetes
we
have
the
admission
controller
right,
it's
pretty
much
you
can
validate.
There
are
two
types
of
mutating
and
validating
right.
A
You
pretty
much
validate
an
object
before
you
take
any
action
on
that
particular
resource.
So
in
this
case
we
built
this
an
admission
control
and
what
this
admission
controller
does
is
pretty
much.
If
you
run
an
operation
delete
on
the
resource
namespace
that
object
call
is
passed
through
this
admission
controller.
What
this
guy's
do
primarily
is.
It
will
check
if
you
have
any
other
resources
within
that
particular
name
space
if
you
have
part
or
if
you
have
deployment,
if
you
have
stateful
set
ingress
it'll
try
to
prevent
you
from
deleting
the
namespace.
A
It
will
say
you
please,
go
ahead
and
delete
all
those
resource
and
then
come
back
and
delete
your
namespace.
Then
we
know
that
this
action
has
been
deliberately
taken
by
the
users:
it's
not
an
accident.
Okay.
This
is
one
way
we
prevented
this
kind
of
accident
happening
and
again
we
also
provide
administrator
yeah
key.
Only
they
can
force
delete
it
so
that
we're
not
forcing
all
the
tenants.
So
if
you
have
a
special
key,
which
we
know
of,
if
we
passed
that
generally,
this
hook
will
not
be
executed
this
for
our
convenience.
A
In
case
we
run
into
any
kind
of
corner
cases.
The
next
issue
which
we
run
into
okay
I,
said
we
had
a
single
ingress
controller
right
today,
so
we
made
everything
self-serving
right.
Everyone
just
comes
in
creates
that
single
interface
EML
file.
We
convert
that
into
kubernetes
file
and
we
apply
the
change
machine,
cube,
cydia
right,
give
cutter,
so
ingress
resource
is
used
for
you
to
say
that
hey
onboarding
to
kubernetes,
saying
I
with
this
domain
name
or
hostname,
you
can
call
that
sports.com
right
and
then
this
is
the
back
end.
A
I
need
to
route
to
right.
This
is
how
you
on
board
to
kubernetes
and
create
a
ingress
resource.
You
can
think
of
this
one
what's
wrong
in
this
particular
things
right.
So
one
of
the
effective
tool
in
engineering
world
is
copy,
paste
right
and
clone
right.
In
this
case,
what
happened
was
there's
two
ingress
was
created
by
two
different
tenants:
okay
and
they
were
pointing
to
different
back-end.
This
is
app
2
and
that
is
app
1,
but
they
claim
the
same,
alias,
which
is
the
same
host
name
right.
So
now
what
happens?
A
Is
you
pretty
much
took
someone
else
domain
name
right
and
pointing
to
two
different
back-end?
So
what
happens
during
this
case?
If
you
of
the
ingress
controller
right
now
when
he
tries
to
get
their
requests
and
then
it
has
to
back
end
for
the
same
domain
right,
it
will
try
to
round-robin
between
those
two
back
ends
right,
so
you
can
think
that
I'm
accessing
sports.com
I'm
getting
Finance
content
because
it's
pointed
to
the
finance
back-end,
and
if
I
refresh
it
next
time,
you
get
a
sports
content
right.
A
So
this
is
because
of
accidentally
copy
pasting
or
ruining
someone's
get
repo
and
then
trying
to
extend
API
out
of
it
right.
We
would
like
to
avoid
this
from
happening,
and
it's
also
very
bad
user
experience,
so
yeah
most
of
the
investors
round-robin.
So
we
build
an
admission
controller,
which
we
called
ingress
claim
right.
What
what
this
guys
do
is
like
if
you
domain
name
are
a
host
name
is
being
claimed
by
some
other
name
space
on
a
mother
deployment.
It
will
not
allow
you
to
claim
it
back.
A
It
validates
it
and
say
that
hey
this
particular
domain
name
for
examples
for
Santiago
that
counts
is
being
used
by
sports
team.
Okay,
you
cannot
claim
that
we
try
to
avoid
this.
We
put
a
prevention
out
of
it
right.
This
is
nice
way
for
our
wedding,
anyone
from
hijacking
each
other's
domain-
okay,
scary
moment,
so
we
do
have
an
experience.
They're
like
hey,
I'm,
accessing
sports,
I'm,
singing
all
finance,
content
right
and
then
sick.
That's
bad!
A
So
again
the
same
thing
whenever
you
create
an
ingress
or
source
create
our
update,
and
we
pass
that
call
to
that.
To
the
Ingram's
claim
admission
controller
very
validate
this
and
you're,
not
copy-pasting
are
you're
not
duplicating
the
same
domain
name
and
it
denies
in
case
you
are
claiming
a
domain
which
does
not
belong
to
you.
Okay,
this
is
next.
We
kind
of
major
things
which
we
solved
next
thing,
this
coconut
is
node
right,
so
we
are
making
changes.
We
upgrade
we
started
with
1.6
and
then
now
we
are
in
1.9
1.5.
A
We
started
pretty
much
now
in
1.9,
and
so
we
did
lots
of
upgrades.
So
one
of
those
changes
are,
we
were
trying
to
make
a
change
to
cubelet
and
we're
trying
to
roll
it
out
right
and
we're
trying
to
roll
that
out,
and
then
we
ran
into
this
kubernetes
bug.
So
what
has
happened
is
when
the
cubelet,
when
is
trying
to
restart
it,
so
it
case
a
transient,
C
group
file
and
it
cleans
up
before
it
starts
up
right.
A
So
this
our
clean
way
of
studying
for
cubelet
so
because
of
this
particular
bug
in
kubernetes,
the
cube,
Latinas
stack
right
and
our
rolling
update
of
cubelet
change
went
across
the
fleet
in
one
particular
cluster.
So
what
happens
when
the
cubelet
is
not
coming
up?
What's
kubernetes
api
thinks
the
notes
pad
the
notes
went
down
so
in
this
case
hunter
pushed
off
the
node
we're
in
a
not
ready
state,
hi
cool.
A
All
the
nodes
are
not
ready
that
can't
a
processing
nodes
are
not
ready.
Yes,
but
let's
see
what
happened
right,
you
can
see
this
red
line
like
it's
create
four
things
happening,
and
this
red
line
see
that
this
number
of
nodes,
which
are
getting
into
the
eviction
queue
and
getting
being
evicted.
This
line
grows
up
right
and
then
you
can
see
that
everything
keeps
down,
and
this
guy
shows
up.
A
It
is
scary,
very,
very
scary,
right
when
a
node
when
a
node
mr.
Matt
Lee
himself
is
not
ready
right.
What
is
the
controller?
Does
the
controller
puts
that
node
into
a
eviction
queue
and
it
will
tie
to
a
weak
start
part
from
that
particular
node,
so
that
it
can
be
scheduled
in
other
available
nodes
right,
so
it
started
doing
that
we
need
to
observe
the
node
within
that
happen
right.
This
X
that'll
happen.
So
this
is
very
cool
feature
in
kubernetes,
I'm.
A
A
It
backs
it
off,
and
then
it
says
that
in
the
throttling
rate,
in
terms
of
eviction
rate,
instead
of
doing
every
seconds,
it
will
make
every
15
seconds
every
20
seconds
it
will
try
to
slow
down
the
eviction
of
the
part
from
the
node
right.
This
prevents
right.
We
cannot
evict
everything
right
if
we
evict
everything
across
the
cluster.
Where
will
this
go
because
all
the
nodes
are
not
ready
state
everything
will
go
in
the
pending
states
right
and
fine.
A
We
still
had
an
impact
like
50
percent
of
the
north
part
I
could
be
a
victor
they're,
still
getting
scheduled
in
the
other
remaining
set.
Of
course,
it's
not
scheduling.
It's
just
codes
on
the
pending
state
because
there
is
no
node
is
ready
same
time
when
the
controller
observes
that
of
the
notes
are
down
it
enters
into
something
called
full
disruption
mode
right
in
this
particular
mode.
What
is
steady?
It
will
stop
everything.
This
is
great.
This
is
what
we
want
right.
The
the
thing
is
the
communication
between
cubelet
and
api
is
broken.
A
It's
not
the
communication
between
the
ingress
to
the
part
right.
We
still
had
that
route
working.
The
part
was
up
and
running
just
think
cubelet.
The
process
was
down
right.
In
this
case,
we
had
a
degraded
experience,
because
50%
of
the
pod
cut
the
victor
remaining
50%.
We
were
able
to
sustain
because
kubernetes
entered
into
this
particular
mode.
Okay,
it's
a
pretty
cool
awesome
feature
in
kubernetes.
It's
not
well
documented.
You
can
take
a
look
at
that.
Ok,
what
we
did.
A
Okay,
two
quick
leave
it
to
Rico,
because
we
had
a
degrade
experience
another
completely
down
right,
because
we
have
many
data,
centers
and
many
clusters.
We
failed
over
that
particular
data
center
and
every
user
went
to
the
next
data
center.
That's
pretty
cool,
it
did
it
and
we
do
update
a
30%
concurrence.
If
you
don't
want
to
go
across
the
fleet
right,
we
do
that.
But
what
is
the
one
of
the
lesson
which
we
learned
out
here
is:
please
do
yeah
test
after
upgrading
cubelet
right,
yes,
simple,
curl
command
against
cubelets
api
is
sufficient.
A
Okay,
we
did
not
have
that
and
then
what
happened
was
like
there's
no
functional
test
post
upgrade
it
all
came
down
right.
It's
really
good
experience.
We
learned
a
lot
during
this
experience
and
now
we
pretty
much
when
the
cubelets
get
apply
updated.
You
have
a
framework
testing
framework
to
make
sure
that
that
cubelet
is
certified
before
the
next
30%
of
the
batch
is
being
picked
up.
A
Okay,
this
is
a
very
cool
experience,
ATC
the
upgrade,
how
many
of
you
guys
turned
etcd
upgrade
BOM
a
lot
right,
scary,
really
scary
right,
and
this
is
something
which
we
did
in
oven
kennedy
environment.
So
we
want
our
LED
CD
from
one
version
to
the
next
portion
and
one
of
the
things
which
we
did
was
we
left
the
kubernetes
api
up
and
running
during
this
process.
Okay,
fine.
We
will
take
the
etcd
down
the
communication
between
the
API
and
the
etcd
will
be
failing,
and
it's
not
going
to
take
any
action
right.
A
Okay,
we
did.
We
ran
a
script,
we'll
go
and
upgrade
the
etcd
and
puts
back
into
rotation
right
so
that
script
had
a
bug.
Okay,
we
need
state
to
upgrade
it
and
then,
when
it
came
up
in
the
production,
it's
supposed
to
point
to
a
production
directly
it
some
of
the
script
had
a
bug.
It
affected
that
as
a
non
production,
environment
and
pointer
to
a
directory
which
was
empty
because
that's
not
a
production
environment,
so
the
cubes
that
the
etcd
started
with
an
empty
directory.
What
happens
now?
A
A
Sorry
complete
right,
but
great
right.
Yes,
what
happened
here
now
that
API
was
able
to
connect
to
a
TC
d?
Okay
and
the
cubelet
came
up,
and
then
he
started
to
connect
with
the
API
and
a
PA
say
that
hey
I
have
100
parts.
Then
you
say
that
no
you're
not
supposed
to
have
under
parts
per
my
data
source.
Okay,
you
have
supposed
to
not
to
have
anybody
what
should
be
clean,
so
what
happens
across
the
fleet,
all
the
cubelet
it
have
leave
it
all
the
part
itself,
a
fix
all
the
parts
clean
slate.
A
The
clusters
complete
down
right.
This
is
awesome
right,
okay,
fine!
This
is
a
mistake.
This
is
script,
bug
right.
It's
supposed
to
find
to
production
directly.
It
pointed
to
a
non
production.
Detective
eachother
empty
in
production,
cluster
I
mean
it
came
out
empty.
This
is
expected
what
I
really
expect
her
like?
What?
How
did
we
recover?
Let's
go
through
what
how
we
could
recover
right?
Okay,
yeah.
We
pointed
to
the
right
directory.
A
There's
no
rocket
signs
here
and
also
we
periodically
take
a
snapshot
of
etcd
store
and
we
took
that
snapshot
and
stored
into
the
director
directly
and
recovered
that
once
you
put
the
data
back
now,
the
cube,
let's
synchronize
with
the
API
and
the
AP,
say
that
they
are
not
supposed
to
have
this
part
but
the
source
that
a
source
which
is
etcd.
Now
everyone
suffers
shadowing
this
one
right,
I
really
really
like
the
features
like
hey.
We
can
we
should.
We
want
kubernetes
api
to
know
that
we
are
doing
some
kind
of
maintenance.
A
There
should
be
a
flag
in
kubernetes
api,
which
you
are
also
looking
exploring,
which
says
that
I
am
right
now
in
maintenance
mode.
Do
not
sync
right,
so
that's
I
opened
a
bug
against
coming
and
we
are
discussing
their
way,
which
they
I
mean
a
maintenance.
Do
not
sink
so
this
kind
of
accidents
could
be
avoided.
A
B
You
today's
deal
a
certificate
in
reference.
How
many
of
you
have
done
till
a
certificate
rotation
of
your
existing
kubernetes
cluster?
Some
quite
brave
hands
there
right.
So
so
the
reason
why
we
do
a
periodic
TLS
certificate
rotation
for
the
kubernetes
cluster
is
the
primary
reason
being
is
that
it's
a
self-signed
certificate
and
it's
also
short-lived
close
to
one
year.
So
what
happens
when
we
do
the
TLS
rotation
pretty
much
the
whole
kubernetes
control
plane
components
needs
a
restart.
They
cubelet
HCD,
a
control
manager,
the
the
APS
over
all
these
component
needs
to
be
restarted.
B
To
avoid
communication
between
these
components
and
there's
a
one.
Another
component
called
service
account,
which
is
a
special
service
account.
Tokens
are
created
by
controller
manager
by
using
this
private
key
service,
account
private
key
and
the
current
is
API
server.
Authenticates
that
service
account
token
using
the
public
corresponding
public
key.
The
reason
I
am
saying
special
is
that
almost
every
controller
in
the
kubernetes
a
ecosystem
uses
the
service
account
to
have
indicated,
and
we
wanted
to
rotate
that
also
frequently
the
reason
being
the
service
account
once
you
created
it
and
it's
an
ever
expiry
token.
B
So
one
thing
we
observed
when
we
actually
rotated
this
certificate,
this
private
key,
that
all
the
controllers
stopped
working
like
horizontal
part,
autoscaler
stopped
working,
daemon
sets
controller,
stopped
working
deployment,
controller,
everything
and
including
all
the
add-ons
like
a
hipster
cube,
dns
and
everything.
You
know
it
also
had
some
impact
to
our
existing
clock
applications
where
we
had
an
an
identity,
D
component,
which
watches
the
kubernetes
api.
That
particular
communication
is
also
broken.
B
So
this
is
one
of
my
favorite
component
when
it
comes
to
equivalent
is
ecosystem
because
it
has
a
huge
integration
between
the
applications
running
and
the
equivalent
is
clustered
as
well.
Almost
every
part
majority
of
the
inducer
personally
in
the
kubernetes
cluster
uses
the
cube
DNS
as
a
resolver.
There
are
a
bunch
of
cases.
We
ran
into
problems
with
the
cube
DNS
naming
Lee
so
oftentimes,
not
all
the
applications
part
have
a
DNS
cache
built-in,
so
they
make
for
every
DNS
lookup.
They
make
a
call
to
cube
DNS.
B
If
you
have
it,
if
you
have
an
application
that
has
thousand
or
2,000
parts
and
when
they
have
a
deployment
going
on,
this
potentially
can
bring
down
the
Q
Tina's
part
and
there's
another
cool.
Another
interesting
thing.
The
second
thing
to
notice
that
whenever
there
is
a
cube,
DNS
pod
goes
down,
cube,
DNS
runs
in
a
deployment
right,
so
pod
can
go
down.
B
So
when
the
part
goes
down,
q,
proxy
updates,
the
endpoint
controller,
saying
hey
just
remove
this
particular
endpoint,
and
so
every
mission
in
the
cluster
gets
the
new
update
of
final
set
of
kubernetes
cube,
DNS
endpoints.
So
even
during
this
fraction
of
yet
you
would
probably
see
a
DNS
latency
in
your
in
your
applications
and
there's
an
one.
Another
critical
thing
is
because
we
run
in
our
on-prem,
so
we
don't
throw
the
host
away
when
the
host
goes
down.
B
We
wait
for
the
host
to
come
back
right,
so
oftentimes
a
node
goes
unresponsible
during
that
time.
If
that
node
B,
if
that
node
being
the
QB
in
a
spar
running
there,
then
we
have
another
interesting
problem
there:
that
there
is
a
flag
in
a
controller,
say
spot
eviction
time
or
it's
default
to
five
minutes
so
for
the
first
five
minutes
is
not
going
to
remove
that
the
failed
node
or
the
fail
depart
from
the
endpoint.
B
So
for
the
for
two
to
five
minutes,
you
will
see
the
all
the
application
routing
to
the
failed
cube,
DNS
part
just
having
some
outage
on
the
leaners.
How
many
of
you
know
dot
file
lookups
here
yeah,
it's
pretty
interesting!
So
that's
one
of
the
cause
in
kubernetes
that
it
adds
a
latency
for
external
DNS
lookups.
It
has
to
traverse
through
all
five
lookups
before
you
make
a
DNS
lookup
for
a
external
endpoint
when
I
say
external,
which
is
not
in
your
cluster
that
local
so
something
all
these
problem
we
wanted
to
avoid.
B
So
one
way
we
thought
we
could
resolve
this
issue.
Is
we
try
to
narrow
down
the
pod
addiction
time
odd?
How
much
we
can
go
thinner
right
so
because
we
now
on
Prem,
we
don't
want
to
be
getting
a
false
alert.
So
so
we
try
it
an
alternative
approach,
running
a
daemon
set
for
the
DNS
mask
so
that
every
node
has
a
DNS
mask.
So
all
the
external
calls
will
be
routed.
B
We
have
a
ruling
that
the
ants
mask,
so
external
calls
will
go
directly
from
the
node
and
it
it
will
forward
the
DNS
query
to
the
cluster
DNS
only
for
those
clusters
dot
local
DNS
lookups.
So
that
way
we
brought
down
the
impact
to
narrow
to
a
specific
node
instead
of
the
cluster
wide
impact.
On
the
other
case,
and
also
we
make
use
of
you
know
the
cluster
proportional
autoscaler
so
in
case
like
we
keep
adding
a
machines
right
so
for
the
applications
we
pretty
much
make
use
of
horizon
a
lot
of
our
scalar.
B
B
So
running
an
on-prem
is
definitely
a
challenging
thing
because,
as
soon
as
mentioned,
we
had
the
ship
pipeline,
we
have
a
robust
release,
guidance
and
everything,
but,
on
the
other
hand,
and
senior
partner
mention
in
the
keynote
today,
kubernetes
releases
almost
two
to
three
times
major
releases
in
a
year
right,
keeping
up
to
date
with
those
releases.
It's
often
a
challenging
task,
and
and
often
we
ran
into
an
issue
where
hey
there
are
some
flags
depreciated.
This
is
a
breaking
change.
B
You
may
have
to
make
things
so
I
just
wanted
to
make
sure
that
you
know
whenever
you're
doing
don't
file
up
those
releases.
Keep
whenever
you
get
a
major
release,
keep
a
practice
of
upgrading
to
the
major
releases
and-
and
we
have
also
bitten
by
a
couple
of
operating
system
packages.
So
we
tried
what
I
recommend
is
just
try
to
keep
those
kernel
and
operating
system
packages.
Up-To-Date,
like
there
were
a
couple
of
scenarios
where
diversity
man
was,
you
know
doing
a
file
descriptor
leak
and
the
darker
hole.
Darker
communication
was
broken.
B
B
So
it's
a
pretty
interesting
problem
in
a
multi-tenant
environment.
As
soon
as
mentioned,
we
have
a
namespace
isolation
right.
So
whenever
there
is
an
oem
happens,
c
group
is
going
to
notice
that
and
it
is
going
to
kill
that
process
right.
When
that
happens,
it's
going
to
trace
dump
a
lot
of
information
into
the
console
and
occasional.
B
Unfortunately,
in
our
setup
you
know
console
is
a
serial
port
and
it's
a
9600
baud
rate.
You
know
what
setup
so
during
that
the
log
is
written
into
the
console,
the
cpu
had
totally
choked
off
it's
effectively,
impacting
all
other
applications
running
on
that
particular
node.
So
we
took
an
approach
of
step.
Nasarah
of
you
know
reducing
the
last
bus
paucity
on
those
on
the
kernel,
verbose,
output,
it's
pretty
interesting,
a
future
in
the
kubernetes
like
suresh
mentioned
node
controller.
B
Has
the
nice
feature
about
the
eviction
throttling
and
and
the
couplet
side
in
a
multi-tenant
environment?
Right
as
we
mentioned,
we
have
2,000
nodes.
We
do
get
only
three
to
four
pages
a
week.
The
way
we
achieve
it,
we
silence
some
of
the
pages
through
you
know
some
of
the
cool
features
available
in
the
kubernetes
one
feature,
as
I
mentioned.
Just
this
out
of
resource
handling.
B
The
way
it
works
is
if
you
have
a
tenant,
and
if
you
have
a
user
causing
a
disc
face
to
fill
up
me
could
be
on
the
node
FS
on
empty
directory
volume.
It
could
be
even
in
the
darker
image.
You
know
image
directory
itself,
you
don't
want
to
be
woken
up
in
the
2:00
a.m.
and
just
forgetting
a
disk
full.
The
same
thing
could
happen.
You
know
you
know
you
have
a
more
nodes
available.
B
There
is
a
memory
pressure
you
wanted
to
make
sure
you
know
you
don't
want
to
get
a
lot
for
you
know
one
one
node
alert,
so
this
is
a
cool
feature.
What
we
can
make
use
of
it
basically
configure
the
eviction
threshold.
So
when,
when
this
particular
limit
hits
Kuebler
trance
the
pod
and
it
automatically
fix
the
bad
bad
part
so
that
you
don't
want
to
be,
you
know,
woken
up
in
the
middle
of
the
night
to
solve
this
one
single
node
issues.
So
so
it's
a
game
day
testing.
B
So
we
often
conduct
a
game
day
testing
for
our
kubernetes
environment.
So
what
I
mean
by
game
day?
Testing
is
basically
we
break
every
component
in
a
controlled
environment.
So
we
we
break
it
and
we'll
record
those
output,
and
so
we
pretty
much
do
that
for
all
the
control
planes
like
API
server,
bring
down
an
API
server
and
observe
the
metrics
on
the
node
side,
we
randomly
reboot
a
node,
that's
how
we
identify
if
the
DNS
is
good
or
if
the
customer
also
occurs.
Is
there
any
noise
from
the
customer
or
or
their
pod?
B
A
Hand
over
to
stress
thanks,
Rhonda,
okay:
this
is
like
a
scripted
environment
where
we
run
the
game-day
testing
like
it's
pretty
much
chaos
engineering,
you
can
think
of
alright.
We
would
like
to
if
we
are
planning
to
write
a
product
out
of
this
there.
It
can
be
used
across
everyone.
Really.
You
can
just
run
this.
It
will
it'll
produce
this
environment
with
scenarios
and
then
say
the
body's
lackluster
behavior
cool
before
I
go
here
I.
There
are
many
scenarios
which
we
run
into
there.
A
few
please
talk
to
us
I'll,
be
here
for
whole
day.
A
We
discussed
about
what
sadaqa
or
SUV
ran
into.
What's
the
English
you
should
be
ran
into
the
network
is,
should
be
ran
into
right,
so
pretty
much
all
those
things
which
I
discussed
about
the
complement
which
we
built
internal,
it's
pretty
much
out
here,
it's
kind
of
open
sourced
it
is
built
so
that
hey
we
can
contribute
back
to
the
community
rather
than
just
always
not
observing
it.
A
So
please
do
use
it
it's
pretty
much,
and
then
we
are
in
the
process
of
see
if
we
can
open
source
the
ingress
controller,
as
well
as
in
the
80s
Apache
traffic
server
and
kicking
out
cater
support.
Yeah
I
just
talked
about
the
other
experience
right,
but
we
have
done
templating
engine,
how
we
templated
arms
and
you
have
our
one
networking
model
of
how
did
we
do
the
networking
for
kubernetes
and
we
have
our
own
Top
Ramen
ingress
survey
of
how
do
we
manage
clusters?
What
are
the
tools
we
built
out
of
it?
A
How
is
our
identity
provided
right?
We
provide
a
very
unique
identity:
short-lived
identify
the
workloads
right,
so
our
engineers
just
deploy
their
application.
We
provide
an
identity
for
that
we
call
identity.
Other
courts
are
out
there
open
sourced
so
that
if
you
want
to
get
the
complete
view
how
we
are
doing
into
in
communities,
please
click
on
this
particular
link.
A
We
had
a
detailed
discussion
about
how
each
components
we
have
been
doing-
and
this
is
my
main
lady-
please
reach
out
to
me-
I'm
very
much-
happy
to
collaborate
I,
pretty
much
join
all
the
slack
channel.
I'm
always
active
out
there
in
case
I
want
to
talk
to
me.
Please
reach
out
to
me
and
important
I'm
wearing
right.
You
want
to
solve
the
challenge
with
through
automation.
Please
do
contact
me
and
we
can
talk
about
your
possible
option.
At
least
a
thank
you.