►
Description
101 Ways to Crash Your Cluster [I] - Marius Grigoriu & Emmanuel Gomez, Nordstrom
Running a kubernetes cluster requires operating many components. One must be good at running and scaling etcd, multiple control plane components, a monitoring system, a logging pipeline, Docker, rkt, and Linux itself. And this list isn't even close to being complete. With such a long list of technologies comes the potential to make a mistake that brings the whole cluster down. Come hear war stories from the Nordstrom's Kubernetes cluster admins. Each is a true story of how the cluster melted down, how they recovered, and what they did to prevent it from happening again. Don't let any of these happen to you...
About Emmanuel Gomez
Emmanuel initiated and served as tech lead on the Kubernetes platform efforts at Nordstrom for the last three years. He was working with and advocating for containers before the Kubernetes 1.0 release and has continuously (and tirelessly) developed, operated, educated, and led containerization efforts there.
This work has forced him to grapple with many of the challenges that come along with the opportunities of containers and container scheduling. Challenges both technical (ex: complex distributed systems, microservices observability), and organizational (ex: inertia, fragmentation, training). Despite these experiences, he wouldn't trade the new problems back for the old.
About Marius Grigoriu
Marius Grigoriu leads the teams responsible for all of the major tools along the software delivery pipeline: issue tracking, version control, continuous integration and deployment, and production through the use of Kubernetes. His focus is to help teams ship high quality systems on time, on budget, and with a smile.
Off the job, Marius can still be found at the keyboard, whether writing Golang or playing classical piano.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
B
A
Oh
hello,
I'm,
an
auto,
somehow
auto-advance
I'm
not
sure
how
or
why
this
might
be
scary.
Okay,
so
anyway,
not
only
do
we
use
the
dev
tools,
but
also
the
the
enterprise
logging
and
monitoring
stack
parts
of
that
are
running
on
kubernetes.
So
we
have
we
use
kubernetes
for
a
lot.
But
if
you're
interested
in
stories
with
happy,
endings
and
you'd
be
better
off
going
to
some
other
talk,
don't
get
me
wrong.
I
mean
kubernetes
is
great,
but
just
go
to
any
other
talk.
You'll
hear
about
how
wonderful
it
is.
A
A
So
meanwhile
we
go
digging
through
our
logs
and
unfortunately,
our
logs
are
scrolling
by
so
fast
and
we're
not
really
sure
what
we're
looking
for
and
eventually
ssh
gets
through,
and
everything
is
nice
and
good,
and
so
the
node
seems
to
have
become
happy
again
by
the
time
we
got
there.
So,
unfortunately,
we
couldn't
see
anything
when
we.
A
Finally,
got
onto
the
node,
but
then
we
looked
into
the
past,
so
we
collect
our
metrics,
they
go
into
Prometheus
and
and
so
what
we
found
out
was
memory
utilization
on
this
node
reached
up
into
the
90s
and
then
right
before
we
were
able
to
log
back
in
there's
this
crash
and
a
lot
of
memory
got
freed
up
and
suddenly
somehow
that
got
correlated.
So
we
knew
that
we
were
looking
for.
A
We
were
looking
for
something
that
was
related
to
memory
utilization
or
memory
memory
kills.
So
now
we
were
able
to
query
our
for
something
more
appropriate,
and
so
we
found
something
like
this
in
the
kernel
messages
we
found.
Oom
kills
and
we
found
page
allocation
stalls
on
that
node,
and
so
we
wanted
to
really
understand
the
mechanism
through
which
this
was
going
on,
because
we
didn't
run
out
of
memory
not
entirely
so
we
went
to
Google
about
page
allocation
stalls
and
back
then
there
was
nothing
if
you
actually
do
that
now.
A
You'll
find
some
other
unfortunate
kubernetes
user
who's
running
into
the
same
problem.
But
there
was
not
a
lot
of
information,
so
we
cracked
open
the
Linux
source
code
search
for
that
string
and
read
I
think
is
like
page
a
Lexi,
and
so
what
that
tells
us
is
that
when
you
go
and
allocate
a
page
of
Rams
kernel
is
going
to
go,
look
for
it.
It
may
need
to
perform
memory
compaction,
and
if
your
memory
utilization
is
really
high
well,
then
then,
then
it
could
take
a
really
long
time
to
get
that
memory.
A
Also
on
the
same
note
that
the
debt
that
were
dormant
and
one
of
our
users
would
initiate
a
job
and
all
the
pods
are
just
wake
up
at
the
same
time
and
they
would
wake
up
at
the
same
time
and
then
bursts
in
terms
of
memory.
Our
memory
utilization
goes
sky-high
and
then
the
machine
would
freeze.
So
that's
that's
what
happened
in
the
way
we
got
to
that
information
is
just
really
following
a
standard
checklist
of
troubleshooting
steps.
Now
this
is
really
pared
down.
I
mean
if
no
not
ready.
A
This
is
in
the
event
of
high
memory
utilization.
You
know
we
look
at
the
nodes
we
figure
out.
What's
going
on,
cubelets
stops
posting
nodes
status
and
then
so
then
we
look
for
signs
of
high
resource
utilization
and
finally,
the
own
kills.
So
what
was
the
fix?
The
fix
was
to
set
a
qubit
flags
correctly,
primarily
eviction
thresholds,
hard
eviction
actually
so
and
there's
documentation
that
tells
you
all
about
this.
A
A
So
that
was
that
was
our
kind
of
like
our
small
story
and
was
fortunate
that
doesn't
spread
to
multiple
nodes.
It
was
fairly
contained,
but
you
can
imagine
that
somebody
could
write
the
page
allocation
stall
or
daemon
set.
That's
just
ripped
through
the
cluster
and
cause
havoc.
Well,
we
didn't
need
that
for
something
to
rip
through
the
cluster
and
cause
havoc.
We
ended
up
seeing
this
one
day
and
there
were
more
nodes
down
there,
which
actually
were
ready.
These
are
the
not
ready
ones
which
is
about
half
of
it,
and
so
you
know.
A
So
we
have
to
resort
back
to
our
to
our
run
book
and-
and
you
know,
simplified
down,
we
take
a
look:
what's
a
node
up
to
a
look
at
stopped
posting
status.
Yet
again
we
want
to
look
for
signs
of
high
resource
utilization
and
in
this
case
we
found
that
resource
utilization
was
not
consistent
across
the
nodes
that
were
affected
and
in
fact
a
lot
of
the
nodes
were
not
utilized
at
all.
A
So
it
couldn't
have
been
that
you
know,
and
so
when
you
see
a
lot
of
nodes,
just
kind
of
pop
out
of
existence
all
at
the
same
time,
it
makes
you
wonder.
Maybe
this
is
like
a
networking
problem.
Cuz
like
nuts
split
from
the
IRC
days,
and
so
we
go
to
a
cloud
provider
happens
to
be
AWS.
We
take
a
look,
we
say:
what's
the
networking
status,
is
it
green?
Yes,
it's
green
or
they
claim
it's
green.
A
It's
always
a
network
right,
I,
always
blamed
on
the
network,
so
there's
only
one
way
to
find
out,
so
we
actually
go
try
to
log
in
machines
that
are
currently
posting,
not
ready,
no
problem
we're
able
to
log
in
and
then
we
say
well,
let's
try
to
hit
the
API
server
from
these
machines,
because
maybe
that
connection
is
somehow
down
and
no
problem
we're
able
to
reach
the
API
server
from
these
nodes.
So
what's
going
on,
doesn't
look
like
a
network.
Ignition
must
be
something
else.
So
we
look
through
our
cubelets
logs.
A
A
Usually
50%
of
the
nodes
go
not
ready,
all
at
the
same
time,
but
sometimes
33%
or
even
25%
go
not
ready
and
then
it
every
time
this
happens.
It's
always
all
the
nodes
kind
of
pop
at
the
same
time,
and
they
also
happen
to
come
back
at
the
same
time
and
in
fact,
every
time
they
go
not
ready
and
then
get
ready
again
that
time
window
the
time
period
where
they
are
not
ready,
always
happens
to
be
somehow
just
like
almost
exactly
15
minutes.
A
Now
it's
starting
to
feel
like
we're
dealing
with
a
time
out.
You
know-
maybe
maybe
it
is
Network
related
after
all,
but
maybe
we
looked
at
it
the
wrong
way,
so
we
scratch
our
headers
trying
to
say
well
what
did
we
not
look
at
yet
and
it
turns
out?
Well,
we
didn't
look
at
our
load
balance
or
logs,
and
so
let's
go
to
those
ELB
logs.
A
So
a
little
bit
of
our
architecture,
we
have
high
availability
masters,
which
means
that
we
have
multiple
IP
addresses
for
the
API
servers
and
the
cubelets
needs
to
go
talk
to
one
of
them
and
well.
We've
got
an
ELB
classic
ELB
in
between,
and
so
we
look
at
the
the
EOB
logs
and
they
said:
oh
we're
gonna
scale
down
one
of
the
instances
behind
the
scenes
transparent
to
you.
You
don't
know
about
this
until
you
know
about
it
and
and
that
preceded
the
incident
every
single
time.
So
what
is
this
all
about?
A
How
is
this
happening?
Well,
it
turns
out
that
there
was
actually
multiple
issues
over
on
get,
but
this
is
the
more
general
case
because
it
turns
out
that
this
issue
is
not
an
AWS
specific
problem.
The
problem
was
that
the
cubelet
didn't
have
a
timeout
in
the
heartbeat
over
back
to
the
API
server
and
so
the
15
minutes
without
TCP
retransmission.
A
A
Maybe
maybe
a
time
machine
time
machines
solve
all
problems,
but
but
it
was
finally
fixed
in
1,
7
8,
and
so,
if
you're
running
a
current
version
of
kubernetes,
you
probably
don't
have
to
worry
about
this
instance
of
this
happening,
but
you
never
know
so.
I'm
actually
gonna
hand
it
over
to
Emanuele.
Now
who's
gonna
tell
us
a
story
about
the
day
that
our
robots
turned
against
us.
B
So
yeah,
so
here
we
move
from
node
level,
outages
to
well
multi
node
level,
outages
to
cluster
components
going
haywire.
So,
as
Marius
mentioned,
we
run
on
iOS
as
a
cloud
provider,
and
we
have
run
the
cluster
autoscaler
component
as
a
way
to
manage
our
spend
and
kind
of
keep
our
infrastructure
provisioned.
According
to
our
workloads
and
one
day
we
came
to
find
that
our
cluster
was
shrinking.
B
B
B
Took
a
look
at
the
obviously
consulted
the
API
server
to
see
what
our
note
you
know,
what
nodes
were
being
reported,
found
the
the
shrank
size
and
then
proceeded
to
go
and
check
out
our
AWS
ASG.
The
auto
scaling
group
scaling
history,
which
helpfully
reported
that,
yes,
those
nodes
had
been
terminated
and
yes
that
had
been
requested
and
happily
complied
with.
B
Let's
see,
and
the
next
top
was
to
go
check
the
cluster
autoscaler,
because
knowing
that
this
component
is
running
and
interacting
with
the
auto
scaling
group
api's,
that
was
a
logical
culprit
and
autoscaler
group
are
the
autoscaler
logs
reported
that
utilization
was
observed
at
0.0.
So
we
have
the
autoscaler
set
to
both
scale
up
and
scale
down
when
it
observes
0
utilization,
it's
gonna,
happily
scale
down
problem
was
utilization
was
not
0,
so.
B
We
were
scratching
our
heads
about
this
for
a
while.
We
had
this
one
incident,
which
was
catice
significantly
impactful.
Our
cluster
was
still
operable,
but
number
of
workloads
were
impacted,
but
it
didn't
recur
at
the
same
scale,
and
this
is
truly
a
sad
tale
because
we
have
not
been
able
to
determined
true
root
cause
on
this
issue.
B
A
couple
of
things
that
made
that
difficult
was
that
we
had
diagnostic
data
aged
out
in
terms
of
the
numeric
telemetry
and
logs
that
were
aggregated.
We
were
trying
to.
You
know,
use
that
data
while
it
was
available
to
try
and
understand
the
issue,
but
we
didn't
get
to
the
root
cause
during
that
period
of
time,
and
we
didn't
do
the
work
to
durably
capture
that
in
a
long-term
way
to
be
able
to
go
back
and
look
at
it.
B
So,
furthermore,
we
didn't
have
the
data
to
go
open,
an
upstream
issue
about
this
and
potentially
save
somebody
else.
The
trouble
now
a
couple
of
things
about
how
we
worked
around
it.
We
well,
as
it
says
here,
we
extended
the
smoothing
function,
so
we
so
we
slowed
down
its
scale
and
behavior
both
reducing
the
number
of
nodes.
B
It
would
scale
in
it
at
once,
as
well
as
the
interval
between
scale
and
events,
so
that
gave
us
time
to
respond
so
that
we
wouldn't
get
it
sort
of
bulk
bulk
killed,
which
is
clearly
a
workaround,
but
at
least
gives
us.
You
know,
even
if
this
is
a
middle-of-the-night
incident,
gives
us
time
to
kind
of
get
somebody
at
a
keyboard
and
intervene
before
before
we're
in
a
catastrophic
situation.
B
In
addition,
we
upgraded
the
cluster
auto
scalar
component
to
a
version
which
supports
a
significantly
better
observability
in
terms
of
the
metrics
that
it
exports
and
makes
visible,
but
there's
still
a
couple
of
metrics
that
we
haven't
yet
been
able
to
introduce
to
the
cluster
autoscaler
we'd
like
it
to
be
able
to
report
what
will
happen.
It
reports
what's
unneeded,
but
not
the
actual,
intended
scale
in
that
it's
going
to
perform.
This
is
something
that
we
know
how
to
do.
B
We
just
haven't
gotten
a
chance
to
get
to
yet
and
along
the
way
we
learned
some
interesting
things
specifically,
and
this
may
be
germane.
It's
somewhat
specular
because
we
can
couldn't
conclusively
prove
this,
but
this
I
suspect
is
germane
to
to
what
hit
us
that
day
that
so
first
first
interesting
thing
we
learned
in
this
investigation
was
that
the
kubernetes
service,
the
cluster
IP
service,
that
kubernetes,
creates
by
default
to
be
able
to
address
the
API
server.
Often
it
shows
up
as
kubernetes
default
service
cluster
local.
There
is
session
affinity
set
on
that
service.
B
Implicitly,
you
don't
create
that
service.
That
service
is
created
by
the
API
server.
If
you're
running
in
a
high
availability
configuration-
and
you
have
multiple
API
servers
present
in
your
cluster
there's
a
number
of
different
knobs
and
possible
configurations
you
could
be
in
you
could
be
advertising
a
load
balancer,
but
if
you're
connecting
directly
to
an
API
server,
your
clients
that
are
addressing
it
in
that
way
will
be
pinned
to
one
specific
API
server
instance.
On
the
face
of
it.
That's
not
so
surprising,
there's
a
number
of
reasons.
Why
that's
the
case?
B
B
So,
in
short,
what
happens
here
is
readiness
is
not
respected
when
the
API
server
count
flag
is
in
play.
So
in
our
case
we
run
five
node
control,
plane.
Five
instances
of
Etsy
d5,
API
servers,
scheduler
and
controller
manager
in
HT
configurations,
and
we
run
with
the
API
servers.
Having
the
API
server
account
set
to
five
seems
seemed
logical.
Now
it
turns
out
that
the
behavior
with
that
setting
is
that
all
five
of
those
API
servers
remain
in
the
endpoints
set
for
that
service
at
all
times.
B
So,
if
you're
in
a
maintenance
event-
and
you
don't
change
the
count
value
bringing
down
one
of
your
API
servers,
20%
of
your
traffic,
that
is
addressing
those
API
servers
over
that
cluster
service
are
going
to
fail
and
it
could
actually
be
more
because
of
the
client
because
of
the
session
affinity
based
on
client
IP.
You
can
end
up
in
situations
where
you
have
hotspots,
because
the
clients
could,
during
maintenance
operations,
you
could
consolidate
down
to
less
than
your
total
set
of
API
servers.
B
Now
the
API
server
account
issue.
The
referenced
pull
request
here
helpfully
suggests
that
the
behavior
with
API
server
account
set
greater
to
one
is
worse
than
just
leaving
it
set
to
one
I
tend
to
agree.
I
think.
Basically,
this
is
something
that
I
there
are.
There
is
upstream
work
to
change,
but
it's
something
to
be
aware
of
where
you
have
components
address
in
the
API
server
over
that
service,
we've
seen
a
couple
of
issues
around
the
schedule
and
controller
manager
as
well,
especially
during
maintenance
operations.
B
Don't
let
this
happen
to
you
and
as
a
side
note
we
have
actually.
So
we
mentioned
that
the
smoothing
function.
Where
were
we
here?
Yes,
that
the
smoothing
function
slowdown
we've
upgraded
to
the
newest
release
of
the
cluster
autoscaler
and
we
haven't
been
hit
by
this
behavior,
but
we
have
seen
intermittent
only
very
occasional,
but
we
have
once
or
twice
seen
reported
in
the
cluster
autoscaler
logs
that
observe
utilization
was
zero
when
in
fact
it
was
not
so
I
thrown
around
a
little
bit
of
flood
there
and
I
guess
I.
B
We
had
a
truly
catastrophic
outage,
which
we
calling
the
split
personality,
SCD
cluster,
so
I
was
describing
a
bit
about
how
our
control
plane,
maintenance
procedures
had
grown
to
kind
of
work
around
this
interaction
of
the
API
server
account
and
session
stickiness,
or
session
affinity
of
the
API
server
communications
from
components
that
talk
to
the
API
server
over
the
built-in
cluster
IP
service.
So
our
procedure
for
performing
control,
plane,
node
maintenance,
which
we
occasionally
have
to
do
due
to
configuration
updates.
B
We
treat
our
nodes
control,
plane,
nodes
included
as
sort
of
immutable,
and
when
we
need
to
update
the
underlying
node
configuration,
that's
typically
will
almost
universally.
We
will
terminate
one
by
one
and
to
roll
through
the
roll
through
the
set
and
replace
them
with
the
with
nodes
of
the
new
configuration
so
given,
given
what
I
just
described.
That
means
removing
a
member
from
the
NCD
quorum
starting
a
new
machine.
It
also
involves
actually
decrementing
the
API
server.
Api
server
account
flag
on
all
the
running.
Api
servers
then
I
say
starting
a
new
machine.
B
We
start
a
new
machine
that
new
machine.
Then
we
we
manually
add
that
to
the
sed
quorum
and
then
go
back
to
all
of
the
API
servers
and
increment
that
API
server
count
flag
and
that
allows
us
to
work
around
the
negative
behavior
that
I
described
a
moment
ago.
I'm
telling
you
all
this,
it's
gonna,
it's
gonna
come
back
later
now
in
this
incident
we
had
a
situation
where
nothing
made
sense.
B
We
were
deeply
confused,
we
were
on
cube,
control,
get
pods
and
we
would
get
two
different
sets
of
data
and
it
would
alternate
back
and
forth.
We
would
run
cube
control,
get
nodes
and
we
would
get
two
different
sets
of
data
and
it
would
alternate
back
and
forth
and
we
couldn't
make
sense
of
it.
It
seemed
like
there
were
two
opinions
of
what
the
state
of
the
cluster
was.
What
was
going
on
and
here's
how
it
came
up
to
us
our
user.
B
This
is,
this
is
not
ever
a
conversation
you
want
to
have
as
a
cluster
operator
and
not
because
this
is
a
self-identified
Cubs
fan,
but
because
pods
starting
up
and
vanishing.
It's
not
supposed
to
something.
That's
supposed
to
happen
here.
So
when
we
went
and
looked
we'd
see
this
and
then
this
and
that
flashed
by
pretty
quickly.
So
it
would
go
back
and
forth
between
these
two
states.
Something
like
this,
although
obviously
depending
what
resources
you
were
looking
at
you'd,
see
different
things.
B
This
was
confusing,
to
say
the
least,
as
I
already
said,
and
it
wasn't
just
about
cube,
control
and
reporting
to
our
sort
of
interactive
query
of
the
state
of
the
cluster.
The
control
loop
started
misbehaving,
so
this
manifested
as
the
controller
manager
acting
on
incorrect
data-
and
this
was
thousands
of
pods
getting
spun
up
and
terminated,
I,
don't
that
basically
the
cluster
or
the
controller
manager
would
be
querying
API
servers
and
getting
alternating
sets
of
data
and
then
hopefully
rapidly
acting
on
alternating
sets
of
data
trying
to
converge
to
some
state.
B
But
it's
hard
to
converge
when
you're
getting
conflicting
views
of
the
world.
The
next
set
of
you
know
horrible
symptoms
asserted
manifesting.
Was
you
know?
Service
endpoints
were
thrashing
our
English
controller
started
to
do
very
bad
things
to
the
traffic
that
should
have
been
flowing
through
it,
or
rather
the
ingress
controller
itself
was
reasonably
okay,
but
it
just
it
was
responding
to
these
conflicting
signals.
B
So
there's
bad
news
here.
This
was
a
full
cluster
outage
on
our
primary
production
cluster.
We
weren't
service
simply
out
of
service.
It
wasn't
just
sort
of
went
away,
but
it
was
sort
of
violently
wrong
and
our
time
to
resolution
was
brutal.
It
was
about
a
four
hour
outage
for
our
until
we
were
able
to
restore
cluster
or
restore
service
by
way
of
a
replacement
cluster.
B
There
are
a
number
of
contributing
factors
that
kind
of
led
to
that
significant
of
a
tender
resolution.
One
was,
you
know,
simply
confusion,
but
also
we
were
reluctant
to
replace
the
cluster
and
spend
in
quite
a
while
troubleshooting
and
diagnosing.
We
knew
that
replacement
would
mean,
read
basically
redeploying
applications
on
the
new
cluster
and
there
was
a
fairly
distributed
relationship
in
terms
of
teams.
B
Utilizing
features
that
are
challenging
to
simply
migrate.
Pick
up
and
move
from
cluster
cluster,
we
didn't
end
up
replacing
the
cluster,
so
Oliver
lectins
ended
up
being
for
naught,
and
that
was
also
contributing
contributed
to
the
delay
and
the
duration
of
our
outage.
So
some
specific
things
here
volumes
are
challenging.
Volumes
are
exclusively
by
own
to
one
node.
B
That
pointed
to
the
sort
of
ephemeral
name,
that
kubernetes
creates
the
load
balancer
with,
and
they
had
to
manually
update
those
DNS
records
to
point
to
the
new
load,
balancer
names
that
were
created
on
the
new
cluster
and
one
could
potentially
wish
for
a
little
bit
better
support
on
kubernetes
side
in
terms
of
making
it
possible
to
kind
of
claim
ownership
of
an
existing
load.
Balancer
I
think
it's.
Maybe
possible,
but
it's
not
a
supported
use
case,
so
basically
migrating
load
balancers
is
not
realistic.
B
B
Hooray,
nobody
got
woke
up
in
the
middle
of
the
night,
so
we
had
full
team
presence
and
we
were
able
to
kind
of
bring
our
full
resources
to
bear,
and
another
bit
of
good
news
is
that
we
were
able
to
analyze
this
one
and
and
actually
get
down
to
a
root
cause
and
and
that
root
cause
understanding
did
Drive
or
did
lead
to
significant
improvements
in
our
understanding
of
the
system,
its
behavior
under
this
failure
mode,
our
code
and
and
our
procedures
around
that
now
you
may
be
asking
but
wait.
How
could
this
possibly
happen?
B
That's
it
is
a
consistent
key
value
store
right,
that's
the
whole
point
it
is,
it
is
and
I'm
not
making
any
claim.
We
didn't
observe
sprit
split-brain
as
described,
which
is
a
breakage
of
the
consensus
model
itself,
but
what
we
did
see
was
stale
data
and
it's
not
super
widely
known
I,
don't
think,
but
it
is
indeed
documented
that
etsy
D
will
return
steal
data.
B
There
are
known
conditions
under
which
that
can
occur,
and
typically
this
is
very
small
increments,
where
the
sed
members
I,
have
you
know
some
small
degree
of
replication
lag
and
between
them,
a
leader
will
accept
rights
and
then
replicate
those
two
followers,
but
under
certain
sets
of
scenarios,
that
lag
can
get
very,
very
significant
and
it
can
also
get
if
you
have
very
significant
lag.
You
can
also
in
inter
situations
where
you're
having
leader
thrashing
because
leader
election
timeouts
can
be
repeatedly
crossed,
causing
multiple
elections
back
to
back.
B
To
back
and
like
I
said
this
is
documented
and
in
fact
we
had
even
read
the
manual
and
knew
that
there
was
at
least
some
theoretical
possibility
of
this
at
the
time,
which
is
to
say
earlier
this
year,
six
months
ago
there
wasn't
widespread
agreement
that
this
that,
oh
sorry,
let
me
back
up
for
just
a
second.
So
one
piece
of
mitigation
here
resolution
actually
is
to
always
ensure
demand
that
HCD
perform
quorum.
Operation
when
reads
are
are
happening,
so
stale
data
is
in
the
read
path,
not
in
the
right
path.
B
B
It
wasn't
mentioned
in
the
a
che
docs
there's
another
may
of
culpa
here
we
after
having
observed
this
and
then
immediately
going
after
resolving
the
issue
immediately
going
to
the
HJ,
docs
and
scratching
our
head
saying.
How
did
we
miss
that
found
that
it
wasn't
there?
It
is
now
so
read
the
docs
read
the
box.
B
You
know
a
quorum,
so
majority
of
your
Etsy
team
members
and
xev
3.1
greatly
reduces
the
impact
of
that
because
reads:
don't
have
to
hit
discs
anymore
previously,
that
meant
a
disk
hit
on
on
you
know,
majority
of
your
got
CD,
cluster
and
and
1.9
quorum
reads
become
the
default
behavior.
So
if
you're
running
1.9
you're
safe,
if
you're
not
running
1.9,
you
may
not
be
safe.
B
There
yeah.
This
might
be
something
to
go.
Look
at
in
your
configs
if
you're
running
your
own
API
server
configs.
So
in
our
case
we
know
what
happened.
We
got
hit
by
right,
latency,
so
exit
II
is
extremely
sensitive
to
right
latency.
This
is
healthy,
right,
latency,
we're
talking
single-digit
milliseconds
in
this
case.
This
is
this
is
a
happy
Etsy,
the
instance
or
a
cluster.
Actually,
this
is
five
Exedy
instances
this.
This
is
not
a
happy
at
CD
instance.
B
So
if
you,
if
you
start
crossing
double
digit
or
multiple
hundreds
of
milliseconds
of
right
latency-
and
this
is
specifically
the
database-
sync
lag-
you're
you're
entering
a
world
of
hurt
and
that
CDs
gonna
start
to
do
well,
if
you're
not
using
core
memories,
you're
potentially
going
to
start
to
see.
Behavior
like
we
saw
even
with
corn,
beats
you're
gonna
start
to
see
you.
A
How
is
it
that
we
have
missed
such
an
important
thing
on
such
a
critical
database,
and
the
answer
is
really
that
there's
just
so
much
to
to
go
figure
out
if
you're
running
a
kubernetes
cluster
and
to
kind
of
demonstrate
kind
of
what
I'm
trying
to
talk
about.
I
mean,
let's,
let's
talk
about
what
happens
at
the
node
level,
so
this
is
from
a
great
site
post
by
Brendan
Gregg,
who
posts
a
lot
of
topics
about
Linux
performance
and
if
you
need
to
figure
out
how
to
troubleshoot
performance
issues
on
a
Linux
node.
A
This
is
a
pretty
good
kind
of
road
map
to
that
and,
as
you
can
see,
there's
just
a
lot
of
different
components.
You
have
to
worry
yourself
about
and
for
each
component
there's
a
different
set
of
tools
right
so
now,
as
cluster
operators,
we
don't
run
one
node.
We
run
dozens
of
nodes
and
dozens
of
nodes
each
in
dozens
of
clusters
right
and-
and
so
you
really
need
to
stay
on
top
of
this.
But
this
isn't
a
small
right.
This
is
zoomed
in
down
to
the
node.
A
If
we
take
a
zoom
all
the
way
back
out
to
the
ecosystem,
this
is
what
we
have
to
deal
with
and
in
fact,
Linux
isn't
even
anywhere
on
the
slide,
so
so
not
only
needs
to
deal
with
the
operating
system,
but
the
cloud
level
and
provisioning.
You
need
to
know
the
quirks
of
your
cloud
providers
because
maybe
they're
going
to
transparently,
replace
or
scale
in
some
of
the
nodes
behind
your
load,
balancer
doing
crazy
things
to
your
network
traffic.
You
need
to
know
about
your
container
runtime.
A
You
need
to
know
about
that
overlay
network
plugin
that
you're
running
how
that
performing
for
you.
You
need
to
know,
obviously
about
kubernetes
itself,
how
the
scheduler
interacts
with
the
API
server
and
how
the
cubelets
talk
to
each
other
and
the
controller
managers
and
everything,
and
then
don't
forget
that
super
critical
300
megabyte
database
called
Etsy
D.
That's
really
sensitive
too
right.
A
That
deployments
the
cloud
the
provisioning,
the
operating
systems
of
virtual
machines,
the
networking,
the
DNS
load,
balancers
all
of
that
stuff
and
what
kubernetes
provides
for
the
application
teams
at
least,
is
a
simple
interface
through
which
to
run
their
applications
and
then
push
all
that
responsibility
on
to
somebody
else.
The
cluster
operators.