►
From YouTube: Preventing Controller Sprawl From Taking Down Your Cluster - When a Scalable Pattern... - Madhu C.S.
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe in Amsterdam, The Netherlands from April 17-21, 2023. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Preventing Controller Sprawl From Taking Down Your Cluster - When a Scalable Pattern Stops Being Scalable - Madhu C.S., Robinhood Markets
Speakers: Madhu C.S.
The vast majority of Kubernetes controllers make use of a WATCH and UPDATE pattern, which is a highly scalable client-pull based pattern. “Highly” does not mean “infinite”, and the spread of this pattern has led to a number of implicit design guarantees that operators build on. In this talk, the Container Orchestration team at Robinhood will cover the exploration of the boundaries of this pattern, how second order effects result in service degradation in production, and best practices for monitoring, detecting, debugging and addressing these issues. With examples drawn from real outages, the team will present lessons learned for organizations of all sizes.
A
Hello,
everyone,
I'm
madhu,
CS
I'm,
a
software
engineer
at
Robinhood.
We
are
a
financial
services,
company
and
I'm,
a
tech
lead
for
this
team
or
our
called
as
container
orchestration,
which
is
responsible
for
running
all
of
compute.
At
Robinhood
and
Robin
Hood
has
chosen
kubernetes
as
the
compute
platform
of
choice.
So
pretty
much
all
of
our
compute,
except
for
some
Legacy
stuff,
is
all
on
kubernetes.
A
A
Everybody
is
welcome,
but
that
said,
if
you
are
an
experienced
cluster
operator
here,
you
might
still
some
of
the
overview.
Concepts
here
might
be
a
reputation
for
you,
but
we
really
hope
that
you
can
take
some
of
these
things
home
as
an
inspiration
or
to
your
company
back
as
an
inspiration
and
pay
a
little
bit
more
closer
attention
to
some
of
these
areas,
or
maybe
do
I
mean
revisit
some
of
these
Concepts
and
see
what
is
working
and
what
is
not.
A
If
you
are
a
beginner,
there
is
definitely
an
overview
in
the
beginning,
but
some
of
these
specifics
may
go
over
her
head,
but
that's
okay!
You
don't
have
to
take
everything
here,
but
we
still
feel
or
hope
that
it
will
give
you
a
solid
grounding
to
the
concept
so
that,
when
you're
ready
to
write
your
own
controllers,
you,
you
can
use
all
these
lessons
or
learnings,
and
hopefully
it
will
it'll.
A
Let
open
the
hood
a
little
bit
and
show
the
Machinery
inside
so
that
you
know
how
exactly
these
things
work
and
coming
back
to
the
topic
itself.
Nothing
kubernetes
controller
pattern
is
a
highly
scalable
pattern
right.
We
have
seen
how
kubernetes
scales,
but
nothing
is
infinitely
scalable,
not
even
slash,
never
slash
null
right,
so
we
I'll
talk
about
how
we
have
pushed
the
limits
of
this
pattern
and
the
incidents
or
outages
that
we
ran
into
when
we
try
to
do
that.
A
The
lessons
we
have
learned
from
it
and
what
we
are
do,
how
how
we
solve
those
problems
and
the
ongoing
work
right
now
to
ensure
that
we
don't
we
don't
run
into
these
limits
again
and
again.
So
we'll
begin
with
a
quick
overview
of
overview
of
controllers
and
then
we'll
do
two
case
studies,
that's
where
I'll
be
spending
most
of
my
time
on
case
studies
about
where
we
went
too
far
or
when
we
hit
the
limits
of
the
patterns
and
we'll
talk
about
the
best
practices
or
Lessons
Learned.
So
this
is
the
overview
right.
A
Kubernetes
philosophy.
Is
you,
the
user
of
kubernetes
submits
a
declarative
intent
as
in
they
submit
the
intent
of
what
they
want
the
world
to
look
like,
which
is
called
as
the
desired
State?
And
then
there
are
controllers,
reading
this
intent
and
taking
action
to
bring
the
current
state
of
the
world
to
the
declared
state
of
the
world.
A
The
controllers
are
constantly
observing
both
states
of
the
world,
the
Absurd
state
of
the
world,
as
it
is
in
the
intent
store,
which
is
the
epis,
and
then
it's
also
observing
the
current
state,
and
it's
always
always
trying
to
bring
the
current
state
of
the
world
to
the
desired
state
of
the
world.
I
think
the
key
concept
here
to
remember
is
it's
always
in
that
direction.
The
current
state
is
always
brought
to
match
the
desired
State,
not
the
other
way
around.
A
If
the
desired
State
changes
controller
does
its
work
to
bring
the
current
state
to
the
newer
desired
state
or
if
the
current
state
changes
for
some
reason,
because
somebody
went
and
did
something
manually
in
the
infrastructure,
it
again
tries
to
bring
the
current
state
to
match
the
desired
State.
This
is
the
key
concept,
and
this
is
this
is
the
philosophy
on
which
everything
in
kubernetes
is
built.
There
is
some
documentation,
as
in
there
is
official
documentation
around
the
concepts
of
controller.
A
If
you
go
look
at
the
documentation,
one
of
the
interesting
examples
that
it
talks
about
the
classic
example
is
the
example
of
a
thermostat
in
the
physical
world.
Right,
if
you
set
the
temperature
of
your
thermostat
to
70
degree,
Fahrenheit
Fahrenheit
is
constantly
watching
for
or
observing
the
temperature
of
the
room.
It
does
not
immediately
make
the
temperature
75
degree
Fahrenheit
in
the
room,
but
it's
observing
the
temperature
of
the
room
and
every
time
it's
not
70
degree
Fahrenheit.
A
It
either
turns
on
the
heater
to
emit
Heat
or
the
AC
to
cool
down
the
room,
to
bring
the
room
temperature
to
70
degree
Fahrenheit.
There
is
more
about
this
analogy
in
the
documentation
you
can
check
out
the
the
slide
deck
is
shared,
so
you
should
have
the
link
but
bringing
this
to
kubernetes
world
and
to
be
more
specific
cubelet,
although
not
a
not
called
a
controller.
The
agent
that
run
on
the
agents
that
run
on
kubernetes
nodes
are
controllers
too,
although
they
are
node
agents,
they
are
controllers
too.
A
They
watch
the
API
server,
which
is
the
intent
store
to
see
if
the
cube,
let's
have
to
run
new
pods
and
if
any
parts
get
assigned
to
the
node
those
and
those
parts
aren't
running
on.
You
know
the
cubelet's
job
is
to
instruct
the
local
container
runtime
on
the
Node
to
start
those
new
parts,
so
cubelet
circuit
or
controllers
too.
With
that
we'll
talk
a
little
bit
about
the
specifics
of
the
implementation
of
a
controller.
This
talks
about
the
semantics
of
the
controller.
So
far
we
talked
about
the
semantics
of
the
API.
A
A
It
is
interested
in
the
key
here,
and
this
goes
into
where
we
push
the
limit,
so
I
want
the
one
thing
that
I
want.
All
of
you
to
remember
is
this:
listing
operation
is
a
very
expensive
operation,
so
what
controller
does
is
well,
it
can
continue
listing
once
in
a
while
periodically
pull
the
API
server
or
the
intent
store
and
keep
getting
the
desired
state
of
the
world,
but
we
have
established
that
it's
already
an
expensive
operation
right.
So
you
don't
want
to
be
doing
often
so
the
Machinery
lets
you
use.
A
This
API
called
as
the
watch
API,
where
the
controllers
can
say.
I
now
have
the
entire
snapshot
of
the
state
of
the
world
or
the
desired
state
of
the
world,
and
from
now
on
start
giving
me
only
the
changes
for
the
desired
state
of
the
world,
which
happens.
Let
me
use
my
Mouse
pointer
here.
Give
me
a
sec,
so
APS
I
mean
controller,
says
I
have
listed,
give
from
now
on
give
me
watch
events,
which
is
the
changes
to
the
objects
that
I'm
interested
in
or
the
resources
I'm
interested
in.
A
But
that's
not
it
right
if
it
were
only
this,
this
Pro
for
today's
computers,
it's
not
a
big
deal.
We
are
replicating
data
here,
it's
not
really
expensive,
but
that
said,
controllers
also
have
to
do
some
additional
work
right
as
in
they
are
not
just
replicating
or
duplicating
data
or
listing
data
from
the
API
server.
They
do
some
real
work
for
the
change
events
or
for
the
resources
and
then
update
the
status
back
and
the
order
of
magnitude
of
resources.
Every
controller
watches
for
is
also
very
different.
A
There
are
some
apis
or
controllers
where
there
are
only
order
of
hundreds
of
resources.
If
you're
talking
about
something
like
crds
or
something,
the
crd
control
is
only
watching
for
hundreds
of
resources.
There
can
be
thousands
of
resources
or,
if
you
talking
about
things
like
pods,
there
can
be
half
a
million
Parts
within
a
cluster
because
that's
the
scalability
limit
of
kubernetes
in
large
clusters
today
right.
So,
if
you
are
getting
tens
of
thousands
or
hundreds
of
thousands
of
parts
and
then
doing
some
work
on
those
parts,
it
gets
really
expensive.
A
If
you
keep
pulling
all
the
time
and
going
through
all
the
polls
all
the
time,
even
though
if
they
haven't
changed,
so
that's
why
we
need
this
complex,
Machinery
of
listing
and
Then,
followed
by
watches.
One
other
important
thing
to
remember
is
kubernetes
API
Machinery.
Does
this
or
the
client
API
missionary?
A
Does
this
so
well
that
after
it
gets
the
snapshot,
it
can
clearly
instruct
the
API
server
to
send
all
the
events
since
that
snapshot,
the
consistency
is
maintained
in
the
sense
that
you
are
guaranteed
or
the
controller
is
guaranteed
to
not
miss
any
of
the
changes
that
happens
to
the
resources
since
the
list
response
was
sent
out.
So
these
are
the
key
Concepts,
and
here
is
a
hilarious
example
that
we
have
come
across
right.
Oh
one,
other
thing
that
I
did
not
introduce
to
was
kubernetes.
A
Also
has
this
concept
of
crd
custom
resource
definitions
that
which
is
the
key
or
the
core
foundation
of
the
kubernetes
extensibility
model.
It
lets
you,
the
user
of
kubernetes,
to
extend
the
kubernetes
the
way
you
want
to
extend
it.
You
can
Define
your
own
apis
as
a
schema,
and
you
can
have
your
users
use
those
schemas
or
apis
for
getting
something
done,
and
in
this
hilarious
example,
oh
actually,
even
before
I
get
there
I'm.
Sorry,
you
can.
A
You
can
Define
your
own
schemas
and
then
you
can
write
your
own
controller
because
schemas
themselves,
as
we
have
seen,
don't
do
anything
right.
They
just
get
stored
as
an
API
resources
just
get
stored
in
the
intent
store.
You
need
a
controller
to
take
some
action
or
do
something
on
top
of
those
apis.
So
kubernetes
allows
you
to
write
those
controllers,
as
you
want
and
the
interesting
thing
about
kubernetes.
Is
it
doesn't
let
you
write
your
controller
against
just
your
own
crds
or
apis?
A
It
lets
you
write
controllers
against
existing
native
apis
as
well,
and
you
can
write
controllers
that
cross
the
boundaries
between
your
custom
resource
or
custom
apis
and
the
native
apis,
and
that
leads
to
an
extremely
powerful
pattern.
So,
coming
back
to
this
hilarious
example,
there
is
a
this
is
a
real
GitHub
project.
By
the
way
you
can
follow
the
link.
There
is
this
pizza
controller.
There
are
two
crds
here:
pizza
store,
crd
and
pizza
order,
crd
I,
don't
think
anybody's.
A
Opening
app
is
a
store
here,
so
I'm
not
going
to
talk
about
it,
but
we're
all
interested
in
eating
pizza
generate.
So
there
is
a
pizza
order,
crd.
What
it
does
is,
if
you're
interested
in
eating
a
pizza.
If
you
want
a
pizza,
you
can
go
submit
a
pizza
order,
resource
to
the
API
server
where
the
crd
exists
and
the
controller
makes
the
call
to
Domino's
API
to
order
a
pizza,
but
unfortunately
that's
all
it
can
do
today.
It
can't
tell
you
whether
you're,
full
or
not,
or
if
you
are
satisfied
with
the
pizza.
A
We
constantly
keep
hearing
from
the
authors
that
that's
coming
in
V2,
but
let's
see
when
we
do
happens,
but
this
talks
about
both
crds
and
controllers
and
taking
this
example
little
bit
little
bit
further,
you
could
consider
of
writing
your
own
controller,
that
watches
for
new
service
accounts
that
get
created
in
your
kubernetes
cluster
and
order
a
complimentary
visa
for
every
new
person
who
comes
into
the
cluster.
That's
the
extensibility
power
here
with
all
this
introduction.
A
So
I
was
on
call
a
few
months
ago
and
I
got
a
bunch
of
alerts,
and
people
started
complaining
that
Quran
jobs
could
no
longer
resolve
DNS.
That
was
interesting
because
anybody
has
worked
in
infrastructure,
recognizes
this
sort
of
a
page
right
or
an
alert
right.
What
is
it
about
KRON
jobs
and
DNS
like?
Why
would
DNS
resolutions
fail
for
cron
jobs?
Well,
the
real
answer
is,
it
does
not
make
any
sense.
There
is
no
correlation
like.
If
you
are
experienced
not
experienced,
it
really
does
not
make
sense.
A
So
we
started
digging
deeper
I
gathered
around
an
incident
response
team.
We
started
hearing
from
more
and
more
application
developers,
back-end
service
developers
that
they
were
seeing
this
and
we
quickly
discovered
why
it
was
cron
jobs.
The
thing
was,
it
was
not
not
just
cron
jobs,
any
new
pod
or
any
new
workload
that
came
up
had
this
problem
and
the
actual
problem
was
not
even
just
DNS.
Any
new
part
that
came
up
would
have
a
total
Network
blackout
for
a
few
minutes,
and
then
it
would
resolve
itself.
A
It
was
anywhere
between
two
to
ten
minutes
and
no
packets
would
go
no
pack,
I
mean
packets
would
actually
go
out,
but
they
would
not
come
back
and
The
Return
part
wasn't
working
and
the
problem
was
the
blackout.
Time
was
typically
longer
than
the
Readiness
probes
for
most
of
these
pulse,
the
pause,
so
pause
would
constantly
die
or
the
containers
would
constantly
die
and
come
back
up
and
they
would
see
the
problem
again,
and
this
would
happen
over
and
over
and
over.
Fortunately,
for
us
running
pulse
continue
to
work.
A
This
is
the
key
thing
or
the
core
tenant
of
kubernetes
container
orchestration
design.
Even
if
there
is
a
problem
with
your
control
plane,
the
programming
of
PODS
and
everything.
The
data
plan
is
supposed
to
continue
to
work
right,
so
the
data
plane
was
held.
The
existing
parts
continue
to
work.
We
just
couldn't
do
scaling
operations
or
deploy
new
version
of
the
software.
Think
things
like
that,
so,
fortunately
we
did
not
have
any
customer
impact
with
that.
A
I
want
to
take
a
slight
detour
to
introduce
you
to
the
networking
stack
that
we
run
so
that
you
can
understand
what
happened
here
better.
So
we
are
a
full
AWS
shop
and
we
started
our
kubernetes
journey
about
four
years
ago
or
a
little
longer
than
four
years
ago,
and
we
have
been
migrating
all
the
services
from
raw
ec2
VMS
to
our
self-managed
kubernetes
clusters.
A
A
At
the
moment
we
have
migrated
pretty
much
all
our
stateless
workloads
onto
kubernetes
and
there
are
only
a
few
stateful
workloads
on
ec2,
but
we
still
continue
to
live
in
this
world.
So
the
solution
that
we
have
built
I
mean
in
the
AWS
world.
If
you
want
to
do
authentication
or
authorization,
the
easiest
way
to
do,
especially
in
the
Raw
ec2
world,
is
to
use
AWS
security
groups
right
security
groups
is
essentially
a
network
segmentation
as
a
proxy
for
your
authentication
and
authorization.
A
A
So
what
this
does
the
Calico
Cloud
controllers
do?
Is
they
read
the
security
group
configurations
in
AWS
as
and
it
periodically
pulls,
AWS
makes
the
describe
security
groups
and
describe
network
interfaces,
call
and
it
sinks
that
information
into
the
kubernetes
Clusters
as
in
it,
creates
the
global
Network
policy
or
network
policies.
Custom
resources
in
the
cluster,
which
is
there
is
this
component
called
as
typher
here,
but
I'll
get
into
that
a
little
bit
later,
but
overall
Calico,
which
runs
the
node
agents.
A
These
node
agents
are
essentially
controllers
as
well
they're
watching
for
these
Network
policy
resources
and
every
time
a
new
pod
comes
in
it
programs,
the
IP
tables
on
the
local
nodes,
to
ensure
that
the
security
group
rules
are
enforced
at
the
IP
tables
layer,
as
in
pods
can
either
send
packets
or
not
send
packets,
depending
on
the
IP
table's
rules
configuration,
but
these
iptables
rules
configuration
itself
is
a
translation
of
our
security
groups
configuration.
So
this
is
the
networking
stack
we
have
and
this
how
it
looks
like
roughly
right.
A
A
We
run
a
kxcni
plugin,
which
is
responsible
for
doing
IP
acquisition
for
every
part
that
comes
up
and
setting
up
the
routes
for
the
pods
as
the
IPS
are
acquired
and
allocated
or
assigned
to
the
pause
that
is
supposed
to,
as
I
said
supposed
to
do,
the
full
route
management
for
the
routing
rules,
management
for
the
Pod
right.
So
we
looked
at
cni
logs.
We
looked
at
Calico
logs
to
try
to
understand
what
was
going
on,
and
then
we
found
this
I,
don't
know.
If
you
all
can
read
this.
A
That
was
interesting
because
what
is
an
old
Route,
because
these
parts
are
brand
new,
very
fresh
right,
so
the
trigger
was
unclear,
but
we
at
least
understood
what
was
going
on
with
Calico.
So
we
put
some
mitigations
in
place
so
that
we
could
open
for
Market
the
following
day.
We
also
have
a
big
stock
brokerage
product
right,
so
we
put
this
medication
so
that
it
would
give
it
would
give
us
some
more
time
to
go
root
cause
what
was
going
on
and
fix
the
root
cause.
A
A
It
sends
ping
requests
off
onto
the
node
agents
to
make
sure
that
the
node
agents
are
fine,
so
we
saw
the
pink
latencies
from
the
taifa
side
go
up
and
if
you
look
at
this
graph,
the
the
time
at
which
it
started
going
up
perfectly
correlated
with
the
time
the
incident
started.
This
is
exactly
when
I
started
getting
paged
and
people
started
complaining.
So
that
was
interesting.
A
So
at
this
point
everything
is
pointy
pointing
to
Calico
being
the
problem,
but
we
are
not
still
not
sure.
What's
up
with
Calico,
so
we
started
reading
the
Calico
code.
The
other
concern
here
is
why
now
we
have
handled
much
higher
loads
during
previous
Black
Swan
events,
we
looked
at
part
churn
rate.
We
looked
at
Security,
Group
Change
rate.
There
wasn't
enough
Fortune
rate
or
Security
Group
changes
at
the
time,
so
it
did
not
make
sense.
A
There
is
a
part
of
work
stream
within
the
incident.
Response
Group,
that's
reading
the
code
and
trying
to
make
sense
of
Calico
type.
There
is
this
other
group
of
people
who
are
just
looking
at
the
metrics,
and
somebody
comes
across
this
graph,
and
this
is
the
QPS
rate
for
the
service
accounts
graph.
A
It's
hard
to
come
across
this
graph,
because
you
kubernetes
has
hundreds
of
resources
or
API
types
right.
You
can't
be
building
separate
graphs
for
it's
just
not
practical
to
build
separate
graphs
or
panels
for
every
single
API
type,
and
if
you
aggregate
everything
into
a
single
panel,
the
problem
is
the
rate
of
changes
or
the
number
of
requests
that
flow
through
for
pause
is
so
high
that
it
basically
drowns.
Everything
else
in
the
graph
like
every
single
other
API
request
type,
will
look
like
this
zero
line
in
that
graph.
A
Interestingly,
we
saw
this
across
all
our
production
clusters
and
then
we
started
digging
deeper
and
deeper
and
we
started
pulling
in
or
paging
people
in
from
various
partner
teams,
the
on-calls
from
various
on
code
teams,
and
then
we
realized
that
there
was
a
potentially
faulty
change
that
was
introduced
to
our
in-house
application.
Development
framework
and
I'll
take
a
minute
to
explain
what
our
framework
is.
We
have
this
framework
called
as
archetype,
which
is
an
abstraction
that
we
provide
for
application
teams
to
develop
their
applications.
Applications
package
Services.
A
Whatever
you
want
to
use
the
terminal
in
the
lingo
or
terminology
archetype.
We
are
completely
bought
into
the
kubernetes
style
of
APs.
The
declarative,
modeling,
declarative
style
of
apis
controllers
and
everything.
So
archetype
are
essentially
a
bunch
of
crds.
We
specifically,
we
have
a
crd
called
as
the
application
crde
and
we
have
a
crd
called
as
the
component
crd,
and
the
idea
here
is
that
a
given
application
is
composed
of
a
bunch
of
components,
and
there
are
controllers
for
each
of
this
and
different
components
have
different
types
of
controllers,
and
things
like
that.
A
Interestingly,
and
unfortunately,
at
that
point
in
time
there
was
ongoing
work,
I
mean
the
ongoing
work.
Hadn't
landed,
yet
a
single
service
account
could
be
shared
by
multiple
components.
We
were
working
on
or
we
were
discussing
doing
designs
or
splitting
that
out,
as
in
we
wanted
to
have
one
service
account
for
component,
but
that
work
hadn't
landed.
So
what
happened
was-
and
this
is
where
the
dueling
in
the
introduction
or
the
title
comes
in
right.
The
controllers
ended
up
fighting
with
each
other.
A
One
One
controller
would
come
in
and
say
this
is
the
owning
component
and
add
the
signature
for
it.
Another
controller
would
see
that
no,
that's,
not
the
owning
component
and
it
would
rewrite
the
signature,
and
this
generated
a
significantly
large
number
of
update
orientation
operations
on
the
service
account
objects
that
it
completely
overwhelmed
the
system,
and
this
is
where
things
get
really
interesting.
What
do
service
accounts
have
anything
to
do
with
networking
or
anything
right?
A
We
were
confused
at
the
point
and
then,
as
we
were
reading
the
code
I
mean
somebody
on
the
team
also
knew
about
this,
but
we
also
realized
at
the
point.
This
is
this
is
where
all
this
incident
response
and
interest
saying
out,
War
Stories
come
in,
even
though
you
theoretically
know
about
things
it's
hard
to
remember
or
recollect
some
of
these
things
when
you
are
in
this
high
stress
situations
like
incident
I
mean
like
these
production
incidents
right.
A
Somebody
realized
that
kubernetes
Network
policies
also
have
this
mod,
where
kubernetes
implementations
of
network
policies
can
program,
policies
based
on
service
accounts
and
Calico
does
that,
unfortunately,
there
is
no
way
in
Calico.
We
don't
use
network
service
account
based
Network
policies
at
Robinhood,
but
unfortunately
there
is
no
way
in
Calico
to
tell
Calico
to
not
watch
for
service
accounts,
even
though
we
don't
use
it.
It
just
continues
watching
for
service
accounts.
A
So
when
the
mutation
rate
started
going
up
calicos
in
in
the
Happy
State,
it's
not
a
problem
at
all,
because
there
are
not,
as
many
mutations
to
service
account
objects,
but
in
this
unhappy
state
it
was
a
problem
because
calicos
in
memory,
controller
queue
which
is
based
on
the
kubernetes
client
go
Machinery
got
so
backed
up
that
it
started
the
important
updates
and
those
important
updates
was
new
pod
creation.
Although
Calico
was
not
supposed
to
be
doing
any
route
management
for
us,
the
AWS
kcni
was
supposed
to
be
doing
the
route
management
for
us.
A
There
was
a
bug
in
Calico
that
did
not
read
the
configuration
correctly.
The
configuration
basically
asked
Calico
not
to
do
any
route
management,
but
the
bug
did
not
translate.
I
mean
did
not
let
the
configuration
to
translate
correctly
and
we
hit
that
bug,
and
it's
now
a
known
bug
in
both
AWS
and
AWC
and
I
and
Calico
communities.
So
we
have
currently
patched
the
bug
in
our
local
implementation,
and
there
is
some
Upstream
work
going
on
right
now,
so
Calico
would
still
do
the
route
management.
A
It
was
started
off,
Newport
updates,
so
it
wouldn't
know
anything
about
the
pause.
Cni
would
come
in,
do
the
IP
assignments
and
all
the
IP
management
stuff
and
then
set
up
the
routes.
Calico,
not
knowing
anything
about
these
new
parts
because
it
was
so
backed
up.
It
would
just
go
and
delete
the
routes
it
wouldn't
recognize,
and
that
would
cause
a
total
Network
blackout
for
a
few
minutes
and
then
eventually
Calico
would
catch
up
anywhere
between
two
to
ten
minutes
depending
on
the
Node
and
what
else
is
going
on
the
Node
and
the
route.
A
A
This
happened
a
while
back
I,
don't
remember
exactly
remember
when
we
scaled
up
our
systems
in
preparation
for
something
we
scaled
up,
our
kubernetes
clusters
and
everything
applications,
everything
for
some,
something
that
we
were
anticipating
and
and
then
we
did
a
production
freeze.
We
said
we
were
anticipating
this
event,
so
we
are
not
going
to
roll
out
any
new
software
and
then
the
event
happened
and
we
wanted
to
scale
down
because
it
did
not
make
sense
to
be
at
that
scale
and
spend
money.
A
We
had
other
run
those
to
go
directly
onto
the
nodes
and
look
for
things
and
basically
that's
what
we
used,
but
we
couldn't
use
a
lot
of
our
other
run
books
again.
Fortunately,
this
goes
back
to
the
core.
Community
standard
right
control.
Plane
is
down,
the
airplane
should
continue
to
function,
so
there
was
no
customer
impact
again.
A
So
we
continued
doing
what
we
did
as
as
in
the
previous
incident
or
case
study.
Interestingly
also,
we
went
to
the
API
server
was
the
problem.
We
went
to
the
API
server
dashboard
and
these
two
are
literally
the
first
two
graphs
that
we
see
on
our
API
server.
Dashboard
is
the
CPUs
Edge
and
memory
usage
and
again
same
story
right
the
moment
where
it
starts
going
up,
as
in
the
CPUs
Edge
and
memory
usage
start
going
up
correlates
with
when
the
incident
started.
A
So
we
had
a
bunch
of
theories.
We
started
looking
at
logs
again
like
in
the
previous
incident,
and
there
was
this
particular
log
line
in
the
API
server
logs
and
again,
we
run
self-managed
because
of
these
reasons,
because
we
want
to
have
access
to
these
kinds
of
things
during
incidents
and
because
it's
our
self-managed
cluster,
we
have
access
to
all
these
logs,
so
we
went
and
started
looking
at
these
logs
and
there
was
this
particular
log
line.
It
said
something
about
the
audit
buffer.
A
This
is
for
the
audit
logs
that
kubernetes
API
server
generates
right.
This
throws
off
a
little
bit
because
this
was
a
red
herring.
We
thought-
or
we
started
thinking
that
we
have
an
in-house
implementation
of
the
logging
web
audit
logging
web
hook
that
ships
the
audit
logs
to
our
logging
system.
So
we
thought
that
implementation
was
buggy
or
something
there
was
some
problem
with
the
implementation
and
it
wasn't
receiving
the
audit
logs
so
that
got
backed
up
in
memory
in
the
API
server
and
API
server.
A
We
started
reading
the
apsr
code,
I
mean
I'm
personally
familiar
with
the
aps
server
code,
because
back
at
Google
I
worked
on
kubernetes
and
control
plan,
so
we
started
reading
the
kubernetes
API
server
code
and
we
realized
that
kubernetes
API
server
does
not
do
any
circuit
breaking
when
the
audit
web
hooks
don't
behave
properly
or
misbehave.
We
had
the
theory,
but
that
did
not
fly
for
long.
We
looked
at,
we
were
looking
at
cubelet
logs.
A
We
saw
a
lot
of
config
map
secret,
mounting
log
lines
in
cubelet
logs,
but
that
was
another
theory
that
did
not
work
out
and
then
because
the
usage
had
increased
in
the
API
server.
We
said
well,
let's,
let's
throw
a
little
bit
more
money
into
the
API
servers
right
as
in
make
the
API
servers
someone
as
in
the
nodes
larger
than
what
they
are
right
now.
This
is
the
cultural
thing
in
the
team.
This
is
ambiguant
is
the
term
that
we
use
for
these
kinds
of
things.
A
It
comes
from
The
Simpsons,
so
we
said
well,
let's
embed
on
the
API
server,
and
maybe
it
will
make
the
problem
go
away
for
now
it
did
not
make
the
problem
go
away.
It
made
the
problem.
Api
servers
were
still
dying,
but
not
as
frequently,
and
then
we
started
looking
into
the
aps
over
logs
again
and
again,
and
then
we
saw
that
we
were
getting
this
HTTP
410
errors
in
the
API
server
HTTP
410
gone
is
also
a
well-documented
semantic
in
the
kubernetes
documentation.
A
I
won't
read
out
the
entire
thing,
but
the
tldr
here
is
actually
even
before
the
tldr
a
little
bit
of
conceptual
introduction
right.
When
the
controllers
do
a
list
and
Then
followed
by
a
watch,
they
have
to
get
the
watch
events.
So,
in
order
to
send
those
watch
events
to
all
the
controllers
or
clients,
the
API
server
keeps
a
watch
of
all
the
historical
versions
or
changes
so
that
it
can
send
those
changes
to
all
the
clients
and
that
is
stored
in
a
cache
in
the
API
server.
A
A
Do
a
list
operation
on
Startup
and
then
they
begin
watching
and
then
APS
server
Returns
the
changes,
occasionally
API
servers
can
say:
full-time
gone,
it's
not
a
problem,
but
if
it
starts
happening
frequently
it
is
a
problem
and
as
we
started
looking
at
the
cubelet
logs,
it
started
becoming
increasingly
clear
to
us
that
the
client
in
this
case
or
the
controller
in
this
case
was
actually
cubelets
and
because
the
control
or
clusters
had
thousands
of
nodes.
Thousands
of
keywords
right,
thousands
of
cubelets
were
getting
HTTP
410
gone
and
then
they
had
to
restart
their
watches.
A
By
doing
this
expensive
realist
operation
against
the
API
server
so
and
this
hap,
this
actually
happened
when
we
were
trying
to
either
run
large
load
tests
or
when
we
were
trying
to
redeploy
or
update
the
versions
of
larger
services
that
we
have
and
because
we
were
in
this
production
freeze
and
then
now
we
were
in
the
top
period.
There
was
this
bug
that
was
introduced
in
our
Frameworks,
where
the
number
of
parts
that
were
created
for
some
of
these
larger
Services
were
in
the
order
of
thousands
in
a
matter
of
minute.
A
So
this
our
systems
would
create
thousands
of
parts
in
a
matter
of
minutes.
Oh
my
God,
that's
not
good
I
can
do
this
later
and
because
the
number
of
parts
that
were
getting
updated
were
so
high,
the
watch
cache
would
have
a
very
high
Miss
rate,
as
in
things
would
go.
The
historical
versions
would
go
stale
really
really
really
quickly
because
they
were
getting
updated,
as
in
the
versions
were
getting
updated
for
given
Parts
at
a
very
high
rate
and
every
client
in
this
case,
cubelets
were
doing
this
right.
A
Thousands
of
cubelets
are
now
trying
to
are
getting
410
guns
because
historical
versions
are
not
there
in
the
API
server
and
they
miss
their
watch.
Events
and
every
every
cubelet
is
trying
to
do
a
realist
and
followed
by
a
watch
this
further
further
overloaded,
the
API
server
and
sent
them
in
into
a
downward
spiral
and
eventually
to
their
dads.
So
that's
the
two
incidents
here
here
are
the
takeaways,
at
least
from
I
mean
not
from
these
two
incidents.
The
takeaways
are
mostly
from
all
the
production,
outages
or
incidents
that
we
have
run
into.
A
Hopefully
these
are
some
Inspirations
for
us.
The
number
one
takeaway
for
us
is
observability
is
the
key
to
operational
success
and
there
are
three
dimensions
to
observability
right
there.
There
are
metrics
logs
and
in
kubernetes
case
audit
logs
I'll
talk
about
all
of
them,
I
mean
there
is
tracing
as
well,
but
I
won't
get
into
tracing,
because
the
Upstream
support
for
tracing
of
kubernetes
components
has
not
been
great.
So
far.
A
There
is
a
cap
right
now
and
I
know
David,
Ash,
Paul
and
other
things
are
working
on
bringing
tracing
into
kubernetes
components,
but
because
at
least
at
the
time
there
was
no
prior
art.
We
also
haven't
done
a
lot
of
work
in
terms
of
distributed,
raising
in
the
kubernetes
components
or
the
extensions
or
controllers
that
we
have
written.
We
hope
to
do
that
as
as
the
Upstream
work
lands
so
leaving
that
aside,
metrics
right,
kubernetes
components,
export
a
ton
of
metrics
tons
and
tons
of
them.
A
A
So
if
you're
scraping
everything
you
should
you,
you
should
be
able
to
build
these
graphs.
Ad
hoc
and
cutting
the
logs
logs
from
kubernetes
components
are
really
useful,
at
least
in
our
experience.
But
if
you
are
reading
the
logs
only
from
the
logs
system
without
actually
looking
at
the
code,
it's
quite
a
fee.
Doesn't
these
were
the
logs
written
by
someone
else
right,
the
author
of
The
Code
by
the
developer.
So
you
need
to
train
yourself
to
understand
what
those
logs
mean
to
understand
what
the
system
is.
I,
think
everybody
not
kubernetes
right.
A
Everybody
who
has
operational
experience
knows
about
this,
so
you
got
to
train
at
least
our
lesson
learned
here,
as
we
got
to
train
our
teams
to
make
sure
that
they
understand
what
these
logs
are
for
these
particular
components
as
in
that
muzzle
memory
or
that
muscle
has
to
be
built
within
the
team.
Api
server
logs
haven't
been
that
useful
for
us,
I
mean
generally,
they
are
noisy
and
haven't
been
useful
for
us
in
practice.
Don't
don't
get
me
wrong.
I
have
contributed
to
the
API
server
myself
so
I
understand
and
how
the
code
is
written.
A
At
least
it
feels
to
me,
like
the
API
server
logs
are
written
for
developers
to
debug
the
system
than
for
The
Operators
of
the
Clusters.
So
we
usually
generally
don't
run
to
the
API
server
logs
as
the
first
thing,
but
but
that
aside,
having
a
good
law
querying
and
filtering
system
or
the
aggregating
UI
comes
in
really
handy
coming
to
audit
logs.
This
is
where
I
think
things
are
super
helpful.
A
As
for
API
server
is
concerned,
kubernetes
exports
audit
logs
right
and
it
fills
the
gaps
left
by
the
debug
logs
that
the
API
server
provides.
The
audit
logs
haven't
just
helped
with
security
stuff.
For
us,
it
has
also
been
an
extremely
powerful
debugging
tool.
It
tells
you
who
made
the
calls.
How
frequently
were
the
calls
made
and
at
what
times?
A
This
is
how
we
found
that
there
were
tons
and
tons
of
realists
that
were
happening
and
those
realists
were
made
by
the
cubeless
during
the
second
case
study
and
have
a
similar
or
same
aggregating
wearing
and
filtering
logging
system
or
UI
for
audit
logs
as
well,
because
it
matters
you
need
to.
You
need
to
understand
how
these
audit
logs
work
and
you
need
to
be
able
to
filter
these
logs
and
view
them
during
incidents.
A
So
that's
the
first
takeaway.
The
second
takeaway
here
is
around
change
management
and
having
some
visibility
around
the
changes
that
go
into
the
system,
ensure
that
you
have
some
way
to
or
some
visibility
into
all
the
changes
that
are
pushed
into
your
cluster.
Here
it
could
be
native
kubernetes,
cluster
components.
It
could
be
extensions
as
in
controllers
or
other
things,
that
you
write
and
have
some
mechanism
to
have
those
notifications
somewhere.
As
in
all
the
change
notifications,
it
could
be
a
very
simple
system
just
throwing
out
ideas
here.
A
It's
not
just
you
or
The
kubernetes
Operators
who
can
write
these
extensions.
Anybody
who
is
using
kubernetes
can
write
these
extensions
right,
and
that
means
anybody
who's
writing.
These
extensions
can
make
mistakes
so
ensure
that
you
work
closely
with
your
partner
teams.
For
writing.
These
extensions
understand
their
use
case
help
them
build
similar
systems
as
yours,
for
change
tracking,
so
that
you
have
visibility
into
them
as
well.
A
And
finally,
this
is
really
the
last
thing
here,
don't
be
afraid
to
read
the
code
I
think
at
least
our
personal
I
mean
detect
here
on
my
personal.
Take
here
as
God
does
not
lie
as
in
lie,
meaning
there
might
be
bugs
in
court.
But
when
you
read
the
code,
you'll
see
the
bug
as
well
as
if
you
are
paying
attention,
documentation
can
be
out
of
date
or
can
have
typos
a
bunch
of
things
right,
but
code
is
exactly
what
is
running
in
your
system.
A
Make
sure
that
you
have
the
core
I
mean
if
you're
running,
open
source
components
at
least
make
sure
you
have
your
code
checked
out,
add
the
version
that
you
are
running
and
don't
be
afraid
to
read
the
code.
Kubernetes
is
a
very
large
code
base.
I
understand
that
I
did
some
analysis
of
API
server
when
I
was
back
at
Google
and
it
was
already
just
the
API
Machinery
was
around
1.1
or
1.2
million
lines
of
code
back
then
this
is
excluding
all
the
controllers
and
all
the
other
pieces
in
kubernetes.
A
So
it's
a
large
surface
area,
not
not
everybody
will
become
an
expert
in
everything
in
kubernetes
is
just
not
practically
possible,
but
try
to
we.
We
have
tried
to
do
this,
and
my
recommendation
or
advice
would
be
to
build
expertise
within
each
of
these
areas.
In
in
your
team,
the
the
expertise
the
areas
could
just
follow
the
Sig
model
that
the
kubernetes
has
sign.
Networking
you
have
a
parallel
analogous
group
within
expertise
group
within
and
is
being
raised,
I'll
wrap
up.
This
is
the
last
slide,
so
yeah
build
expertise.
A
Make
sure
that
people
in
those
areas
are
familiar
with
the
code
for
the
components
that
they
own
both
internal
and
open
source
and
then
also
ensure
that
there
are
debugging
tools.
I
mean
people
within
those
groups
are
comfortable
with
the
debugging
groups,
debugging
tools
within
that
area,
tsp,
dump,
IP,
etc.
For
networking
people
of
frames,
etc,
etc
for
all
components
with
that,
that's
the
end
of
my
talk.
Thank
you.