►
From YouTube: Who Killed My Pod? #Whodunit - Suneeta Mall, Nearmap
Description
Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Who Killed My Pod? #Whodunit - Suneeta Mall, Nearmap
A few weeks ago, we deployed a brand new thoroughly tested, and profiled application onto a self-managed Kubernetes cluster. Suffice to say, all hell broke loose. The pods were getting OOMKilled with error code 137 left and right. This sparked a massive crime scene investigation and some interesting insights were discovered. In this Kube-CSI [crime scene investigation] episode, we will talk about exactly whodunit, why, and the fix!
A
A
Every
intriguing
investigation
begins
with
a
disaster,
and
ours
is
no
different.
It
started
off
by
cube.
Control
apply
my
pixie
dust
magical
app,
which
is
we
thought
is
well
tested
in
local
environment
and
bare
metal,
and
as
soon
as
we
deployed
deployed
excuse
me
all
hell
broke
loose,
we
started
to
see
a
container
being
killed,
left
and
right
with
om
kill
as
a
reason
and
error
code
exit
code
137,
and
this
is
then
going
into
crash
loopback,
which
is
a
pure
joy
to
look
at
when
you
are
getting
things
out
into
production.
A
So,
let's
start
with
the
investigation
bone,
we
have
a
particular
container
process,
that's
being
killed
repeatedly,
potentially
because
it's
being
a
memory
hogger
now
we
know
that
the
kill
is
a
forceful
brutal
and
it's
killed
with
a
sickle
as
a
signal.
So
it's
either.
Somebody
in
somebody
with
more
power
has
killed
the
process
without
the
the
desired
wishes,
and
it's
a
runtime
error.
A
Then,
okay,
so
we
we
know
some
context
and
let's
look
at
the
details
for
how
the
containers
run
on
a
host
machine
and
then
who
are
all
in
charge
in
making
the
container
process
life
cycle
run
on
a
host.
So
we
obviously
have
a
host
the
the
their
metal
or
the
instance
or
virtual,
and
then
on
top
we
have
the
container
runtime
and
container
runtime
is
comprised
of
two
different
levels:
the
lower
level
runtime
and
higher
level
and
time.
A
The
lower
level
deals
with
interacting
with
systems
host
using
system
calls
and
making
use
of
kernel
and
os
features.
Container
d,
which
is
the
higher
level
runtime,
then
deals
with
a
higher
level
apis
api
that
interfaces
users
and
deals
with
image,
management,
etc
aspects.
So
then
we
have
the
runtime,
that's
broken
down
into
two
levels:
low
level
and
high
level
run
c
and
container
d
are
two
very
popular
examples
of
these
runtimes.
Obviously,
there
are
many
different
other
runtimes
available
too.
A
So
now
we
have
container
on
time
and
then
your
process
that
we
call
container
then
runs
on
top
of
it
and
the
the
layer
underneath
the
runtime
and
the
host
then
guarantees
the
isolation
for
your
container
process,
so
they
are
kind
of
not
stepping
onto
each
other's
toes.
So
this
is
what
happens
on
the
host
level.
A
Now,
when
we
put
kubernetes
in
the
mix,
that
is
a
container
orchestrator,
not
just
on
one
host
on
many
host,
then
we
need
another
process
who
is
responsible
for
managing
container
processes
on
the
host
on
its
own
way
and
also
managing
across
all
the
other
servers
which
is
essentially
what
kubernetes
is
a
part
of
it,
I
should
say,
has
become
very
big
so
that
process
that
we're
talking
about
that
deals.
The
management
at
a
host
level
is
called
tubelit.
That
is
a
systemd
process
that
manages
your
process
running
and
the
way
it
does.
A
It
is
by
implicitly
talking
to
your
container
runtime,
that's
provisioned,
on
your
node.
Now
then
that
then
talks
to
image
registry
and
deals
with
the
container
process
life
cycle
now
c
advisor
is
is,
is
a
is
a
way
to
grab
container
resources,
utilization
and
stats
about
it,
which
is
a
part
of
a
cubenet.
So
cubelet
then
talks
to
the
container
runtime
to
facilitate
the
lifecycle
of
container
processes,
and
it
also
has
the
advisors
for
the
matrix
collection.
A
Now
there
are
other
parts
in
that
figure
that
I'm
not
going
to
touch
upon
very
much,
because
that's
kind
of
not
the
focus
of
the
discussion
now
in
current
version,
all
the
way
up
to
k20
the
the
way
cubelet
talks
to
container
runtime
is,
via
a
thin
layer
called
docker
ship,
which
is,
which
is
a
means
to
mitigate
the
communication
or
overhead
with
talking
to
docker.
This
is
all
going
to
change
a
lot
in
k,
20
and
22,
but
for
what
we're
talking
about?
It's
not
so
much
relevant.
A
Okay.
So,
coming
back
to
to
to
our
actual
investigation,
then
we
know
that
our
containers
are
being
forcefully
killed,
but
it's
not
to
manual
intervention,
then,
based
on
what
we
have
discussed,
it's
either
the
cubelet
who
is
in
charge
of
managing
the
process
running
from
a
cube
viewpoint
or
an
os
kernel.
Then,
who
is
in
charge
of
running
processes
on
the
host?
One
of
the
twos
are
killing
the
process
they
they
it
doesn't.
Look
like
there
could
be
other
suspects
that
we
would
have
to
deal
with.
A
So
let's
talk
about
what
container
processes
and
the
isolation,
how
that's
guaranteed
at
the
os
level,
namespace
and
c
group.
These
are
the
two
important
aspects
that
allow
for
the
isolation
of
the
container
process.
Now
namespace
is
responsible
for
isolations
from
a
file
system.
Networking
in
other
aspects,
c
group,
on
the
other
hand,
is
a
feature
of
kernel
that
allows
for
hierarchical
management
of
resource
and
resource
requirements
and
c
group
memory
and
other
features
or
resources
can
be
used
to
to
to
to
manage
viac
group.
A
A
So
c
group
is
managed
through
a
pi
system
in
a
way
like
this,
and
you
can
see
all
the
kind
of
resources
that
cgroup
then
would
manage.
Now,
if
I
look
into
sorry
look
into
memory,
I
would
see
these
are
the
kind
of
aspects
of
memory.
That's
then
managed
by
c
group.
What's
of
interest,
is
kernel
memory
and
also
memory
limits
and
in
few
other
aspects.
A
A
A
Now,
let's
have
a
bit
of
a
look
into
what
the
memory
limits
for
the
containers
are,
and
I'm
just
going
to
cut
the
memory
limit
type
and
all
I'm
doing
here
is
say-
give
me
full
full
id
of
my
container.
That's
40
meg
limit
and
you
can
see
when
I
do
this.
I
am
getting
about
40
mag
as
a
process.
Now,
if
I
just
go
in
here
and
make
this.
A
A
A
Now.
What
we
want
to
do
is
look
at
the
inspect
this
container
and
all
I'm
doing
doing
is
saying.
Do
I
inspect
on
this
container,
and
I
want
to
only
look
at
the
state
measure
and
you
can
see
it's
saying.
This
process
was
actually
omkil
with
an
exit
code
of
137
and
we
could
do
the
same
thing
with
40
mag
container
and
we
would
see
then
it's
not
omt,
it's
still
running,
etc.
A
A
My
container
is
only
going
to
get
limited
to
20
meg,
because
the
swap,
which
is
a
capability
of
systems
to
go
beyond
what's
available
in
physical
ram
and
and
extend
use
a
little
bit
more
than
what
the
capacity
is,
and
that
aspect
is
disabled
now,
so
we
we
can't
really
use
more
than
what
we
have.
But
this
is
my
mac
now
and
I
would
run
the
same
two
command
like
before.
A
A
A
A
A
That
is
that
belongs
to
this
particular
limitless.
Kcsi
container,
so
in
a
way
of
saying,
is
give
me
we
ask
for
the
process
id
by
doing
a
docker
inspect
using
the
container
id,
and
then
we
do
a
process
tags,
and
you
can
see
that
three
five
zero
five
is
the
process
id
on
the
host
level,
as
in
the
host
sees
it
as
this
three
five
zero
five
id-
and
this
is
pointing
to
an
ipython
process.
A
A
A
Or
om
one
is
the
score
which
actually
is
used
to
manage
this
code
or
record
the
score.
The
other
are
two
adjustment
files.
As
in
how
much
you
want
to
offset
the
score
and
there's
two
for
different
linux
versions,
but
that's
not
so
much
relevant
for
what
we're
talking
about.
So
we
had
to
look
at
how
cgroup
works
and
allows
for
container
processes
isolation,
and
we
also
look
through
how
om
scores
are
managed
and
how
container
knows
which
process
to
then
kill.
A
Okay,
it's
slightly
changing
the
gear
and
talking
more
about
how
do
we
ensure
the?
How
do
we
ensure
the
quality
of
service
and
resource
requirements?
So
when
a
pod
is
running
under
kubernetes,
it
allows
users
to
specify
the
resource
requirements
and
it
does
this
in
three
different
fashion.
One
is
you
basically
don't
specify
any
limits,
and
in
that
case
the
quality
of
service
becomes
the
best
effort
and
then
the
other
one
is
bustable,
where
you
give
it
a
range
and
based
on
the
range,
the
cube
will
try
to
best
place
it.
A
A
Incompressible,
on
the
other
hand,
says
that
if
you
breach
your
limits,
c
group
or
something
would
kill
your
process
and
memory
is
an
incompressible
resource
and
cpu,
then
is
is
a
compressible
resource.
So,
based
on
what
we
have
discussed,
I
think
the
question
here
is:
can
cube
overcome
it.
Or
can
you
allow
your
commit
and
how
about
linux
kernel-
and
this
is
a
very
important
question
and
it's
not
we
are
posing
now.
A
A
Now
sure
same
thing
applies
to
cpu
as
well.
You
can
can
overcome
it
from
cpu
and
a
memory
viewpoint,
and
the
os
are
configured
to
overcome
it
as
well
by
default,
but
they
do
have
a
soft
space
to
fall
back
on
to
to
kind
of
use
a
little
bit
more
space
when,
when
the
physical
memories
is
reached,
okay,
so
going
back
to
our
actual
crime
scene,
when
we
profile
locally,
we
never
really
saw
the
resource
requirement
to
go
beyond
six
gig.
A
A
It
had
enough
memory
left
from
the
monitoring
viewpoint
that
it
was
never
really
reaching
its
own
to
the
limit,
so
just
reviewing
our
board
back.
We
know
that
container
behaves
kind
of
reasonably.
We
are
still
getting
a
kill.
The
quality
of
service
is
configured
to
the
best
capacity.
Still
there
is
a
kill,
it
is
likely
to
be
an
over-the-middle
related
issue,
and
we
know
that
the
suspect
is
now
either
the
kernel
or
cubelet
okay.
So
we
had
a
bit
of
a
lucky
draw.
A
Okay,
so
we
talked
about
om
scores.
Before
what
happens?
How
does
the
om
workflow
gets
triggered?
So
linux
manages
this
k
message,
kernel
messaging
stream,
where
all
these
events
are
then
put
in
now
k
message:
kernel
messaging
parser
then
passes
it
and
translate
it
into
events
through
event
handler
now.
The
two
events
that
we
are
interested
in
is
om
event
and
omg.
11
om
event,
kind
of
indicating
hey
memory
surge
is
happening
now
there
is,
there
is
actually
an
oven
situation
ahead
and
the
other
one
is
okay.
We
have
an
om.
A
A
Okay,
so
there
is
nothing
in
application
log
that
would
indicate
any
problem
with
memory
search,
but
we
know
there
is
a
search
very
rarely
we
don't
see
any
event
of
any
kind
of
om
on
cube
system,
pods,
cube
system
processes
or
in
any
of
the
event
log.
As
far
as
cube
is
concerned,
the
node
is
all
healthy
and
it's
all
hakuna
matata
okay,
coming
back
to
again
so
then,
based
on
that,
we
conclude
that
it's
not
actually
tubelet
who
is
killing
the
process.
A
So
let's
look
at
kernel
logs
then,
and
this
is
where
we
landed
up.
Then
this
is
an
excerpt
from
our
kernel
log
and
we
can
see
that
actually
om
kill
was
invoked
and
it
was
ignored
because
c
group
ran
out
of
memory-
and
you
can
see
here
in
here
that
it's
actually
omkil
invoked
and
then
q
pod.
This
is
the
application.
Part
was
killed
as
a
result
of
memory
limit
and
the
actual
usage
was
31,
4,
55
or
31.4
gig
to
31.45
gig.
A
Again,
there
is
a
difference
of
about
1.5
1000
kb,
now
that
it's
still
kind
of
short
of
the
reaching
memory
limit,
but
it's
almost
reaching
and
you
can
see
that
swap
usage
is
still
zero
and
on
the
os
level,
the
the
swap
is
unlimited.
So,
like
you
know,
memory
usage
is
still
unlimited
from
from
a
system
viewpoint.
Kernel
is
using
a
little
bit
of
memory,
but
it's
it's.
It's
not
the
application
usage
so
much
and
it
has
its
own
limit,
as
you
were
seeing
earlier.
A
A
So,
coming
back
to
the
second
part
of
the
log,
as
you
can
see
here,
we
are
saying
that
the
pod
usage,
the
actual
usage,
is
this
and
then
within
that
part
there
are
two
containers.
One
is
the
pause
container,
which
is
really
using
very
little
memory,
so
not
so
much
the
focus
right
now,
but
this
is
the
application
part,
and
you
can
see
it
is
actually
using
an
active
anonymous
memory
of
31.38
gig
and
cash
is
a
little
weight.
A
The
rss
memory
is
also
then
31.38
gig
and
the
the
huge
page
memory
is
29
gig.
So
all
in
all,
there
is
only
very
little
memory,
that's
being
used,
that's
not
on
the
huge
pages,
and
but
we
are
still
very
much
within
the
limit
of
container,
which
was
31.45
as
it
was
said.
So
so
this
is
where
we
are.
We
are
almost
the
process
is
almost
reaching
the
limit
and
it's
getting
killed.
The
om
is
getting
invoked
and
it
is
getting
killed.
A
There
is
still
enough
memory
available
on
the
node,
so
node
is
not
under
duress,
and
if
you
look
at
this
total
vm,
we
could
see
that
we
have
a
really
ridiculous
amount
of
virtual
memory.
That's
being
allocated,
62.59
meg
is
being
allocated
and
it
does
look
like.
We
have
some
kind
of
unused
memory
allocation.
That's
that's
going
on,
but
coming
back
to
like
you
know
why
the
process
was
killed.
We
know
that
it
was
because
it
was
reaching
the
the
memory
limit.
It
was
just
a
little
bit
shy.
A
I
to
be
honest,
I'm
not
sure
why
it
was
still
getting
killed
when
it
was
just
shy
of
the
requirements,
but
it
is
getting
killed
just
being
on
on
the
shy
of
requirements
and
yeah.
I
must
mention,
if
you
haven't
seen
ian
lewis
blogger
on
almighty
container
and
if
you
I
want
to
know
more
about
what
pause
container
is
I
highly
recommend
reading
through
that
blog?
A
So
then,
as
I
mentioned,
we
were
quite
confused
why
the
process
is
getting
killed
despite
being
shy
of
the
limit,
but
given
it's
only
a
few
few
kb
shy,
we're
just
going
to
accept
that
it
is
getting
killed
there.
Now.
The
problem
we
have
is
when
we
get
overkill.
The
behaviors
are
exactly
like
that.
It's
a
runtime
error.
You
can't
take
away
from
it
gracefully
and
you
can't
even
fail
safely,
and
this
is
exactly
not
what
we
want.
A
So
we
looked
at
disabling
the
over
capital
and
saving
failing
more
at
an
application
level,
rather
than
abruptly
being
killed
and
there's
two
settings
on
vm
level,
where
you
could
say:
oh
computer
memory
is
disabled,
two
for
disabled
here
there,
two
for
disabled
and
then
this
another
ratio
aspect
which
which
is
only
used
when
you
disable
the
over
committee,
and
it's
a
way
of
saying
that
I
don't
want
my
system
to
overcome
it
and
only
commit
to
their
total
allowed
memory
up
until
cube18
this.
There
was
no
way
to
provide
this
setting.
A
If
you
provide
this
via
user
data
script
on
your
node
provisioning
site,
it
would
essentially
get
overwritten
in
cube
18
onwards.
This
system
parameters
property
that
now
k
ops
allows
that
you
can
use
to
configure
your
cube
clusters,
but
even
in
that,
the
over
committal
setting
explicitly
overcome
it
memory
is
equal
to
two
is
actually
disabled.
There
is
no
way
to
systematically
provide
this
setting
and
and
make
sure
your
your
nodes
are
actually
disabling
the
over
committee.
A
So
really
the
only
choice
we
have
here
is
to
to
on
the
fly
update
the
nodes
after
they
have
come
up
and
they've
joined
the
queue
cluster
to
override
this
over
commercial
setting.
So
we
kind
of
follow
the
script
to
to
update
all
the
nodes.
With
this
overcommittal
setting,
we
obviously
there's
probably
a
reason
why
it's
disabled,
but
we
don't
know
the
the
details
and
that's
why
we
selectively
apply
only
for
a
particular
fleet
of
nodes
where
this
particular
process
goes
in.
A
So
just
revealing
our
board.
We
know
who
are
who
who
the
killer
was.
It
was
os
kernel
and
we
also
know
that
actually,
our
well-behaved
application
was
not
truly
well
behaving,
but
disabling.
The
over
committee
definitely
mitigated
the
issue.
We
can
see
that
the
process
has
been
running
for
about
nine
hours
without
any
interruption
when
the
kills
would
happen
at
least
15
20
minutes
so
so
frequently.
A
A
So
all
in
all,
in
summary,
we
have
a
container
that
was
killed
because
it
was
reaching
its
memory
limit
and
it
was
killed
by
os
kernel
and
disabling.
The
old
committee
was
mitigating
the
issue
and
but
the
actual
fix
is
to
reduce
the
memory
footprint
and
guarantee
the
the
resource
quality
of
service
and
resource
requirements
on
the
part
to
have
more
reliable
experience
on
divinities,
and
that's
all
I
had.
Thank
you
very
much.
I
hope
this
was
an
interesting
listening
in
to
our
experience,
and
it
was.