►
From YouTube: Look Ma, No Pause! - Mrunal Patel & Peter Hunt, Red Hat
Description
Don’t miss out! Join us at our upcoming events: EnvoyCon Virtual on October 15 and KubeCon + CloudNativeCon North America 2020 Virtual from November 17-20. Learn more at https://kubecon.io. The conferences feature presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Look Ma, No Pause! - Mrunal Patel & Peter Hunt, Red Hat
The pause container is a quinnessential hack for Kubernetes pods. A pause container is the first container in a pod, pauses until it is terminated, and is used to hold Linux namespaces open for the lifespan of the pod. Even though the pause container is as minimal as possible, it still has overhead, and we can do better. CRI-O now supports running the pause container only when absolutely necessary. Join Peter Hunt and Mrunal Patel for a deep dive into the implementation details of dropping the pause container, as well as a performance comparison and discussion of other benefits. Attendees will participate in a conversation about the removal of this hack from Kubernetes.
https://sched.co/Zenv
A
Right
we
can
get
started.
So
what
is
pause?
So
if
you
ever
run
docker
ps
on
a
kubernetes
node,
you
must
have
seen
a
mysterious
container
running
with
a
pause
image
for
each
one
of
your
pods.
So
if
you
take
a
look
at
the
example
on
the
slide,
I
am
running
two
parts,
the
dns
port
and
the
dashboard
part,
and
there
are
two
pause
container
one
for
each
of
them.
So
today,
in
this
talk
we're
going
to
cover
what
the
pause
container
is.
A
So
so
what
exactly
is
the
pause
container
doing
so?
Okay?
So
we
can
talk
about
how
a
cuban
at
this
pod
works.
So
a
kubernetes
pod
consists
of
multiple
containers
that
share
linux
name
spaces
and
are
nested
under
a
common
port
c
group.
The
pause
container
holds
the
shared
namespaces
for
the
containers
in
a
pod,
and
it
is
referred
to
as
a
pod
infra
container
in
the
kubernetes
code
base.
So
on
the
right,
you
can
see
a
picture
of
what
a
pod
looks
like
when
using
cryo
as
your
container
runtime.
A
A
So
now,
let's
talk
about
the
history
of
kubernetes
pods,
so
kubernetes
was
started
after
the
container
revolution
was
brought
about
by
docker
kubernetes
aimed
to
solve
the
problem
of
orchestrating
containers
across
a
cluster
of
nodes.
Note
this
this
was
way
before
the
container
runtime
interface
of
the
cri
was
introduced
to
support
plugging
in
different
container
runtimes,
like
you
can
do
today,
with
cryo
or
container
d,
and
one
of
the
core
kubernetes
decisions
initially
was
to
give
each
pod
a
unique
ip
address.
Now,
how
would
you
do
that
using
just
docker
containers
well
to
do
that?
A
A
Congratulations
so
you
have
invented
the
pause
container
and
that's
how
it
came
to
be,
and
initially
it
was
referred
to
as
the
network
container
in
the
code
base,
because
that's
all
its
job
was
at
that
point
and
then
soon
after
it
made
sense
to
enable
sharing
ipc
namespace
inside
a
pod
to
allow
the
containers
in
a
pod
to
communicate
using
ipc,
cues
and
shared
memory.
So
with
that
change,
the
net
container
was
renamed
to
the
much
more
appropriate
pod
infra
container
in
the
code
base.
A
So
you
can
take
a
look
at
that
pull
request
that
made
some
of
those
changes
to
see
the
history
and
the
discussions
that
happened
back
then,
and
finally,
support
for
sharing
with
namespaces
was
added,
and
at
that
point
the
job
of
the
pause
container
became
a
bit
more
involved
than
just
holding
namespaces.
It
was
also
responsible
for
reaping
pits
for
all
the
containers
in
a
pod,
basically
acting
as
in
it
for
the
pod.
A
So
now
we
talked
about
the
purpose
of
the
pause
container
and
what
it
does.
So
why
would
we
want
to
drop
that
container?
So
there
are
a
few
reasons
and
let's
talk
about
the
reasons.
The
first
is
that
it
uses
up
space
for
the
binary
and
for
the
containers
so
on
its
own.
The
pause
container
doesn't
occupy
a
lot
of
disk
space,
but
if
you
are
starting
a
pause
container
for
200
pods
on
your
note,
then
you
are
using
a
bit
of
memory
which
could
be
used
elsewhere.
A
The
second
reason
it
takes
time
to
create
the
root
file
system,
create
start
mount
the
pause
container
and
so
on,
and
you
can
see
a
small
micro
benchmark
of
that
with
and
without
and
later
on,
in
the
talk
peter
is
going
to
cover
some
more
performance
numbers
and
the
final
reason
for
dropping
pause
is
to
just
get
rid
of
the
additional
process
management
overhead
right.
We
have
code
today
to
set
up,
monitor
and
clean
up
the
pause
process.
We
have
to
keep
tracking
that
the
pause
container
is
still
running.
A
Cubelet
has
custom
code
in
the
image
gc
code
part
to
make
sure
that
the
pause
controller
image
is
available.
So,
if
there's
no
such
process,
we
just
get
rid
of
the
process
management
and
our
goal.
If
you
look
at
the
two
diagrams
below
is
to
get
rid
of
the
pause
container
now
peter
is
going
to
talk
about
how
and
in
what
situations
we
can
get
rid
of
the
pause
container.
B
B
Oh
next,
okay,
sorry
about
that.
So
first
we
need
to
emulate
what
the
runtime
does
when
creating
a
pause
container.
We
need
to
create
a
new
namespace
for
the
pod,
so
luckily
linux
supports
bind
mounting
namespaces.
So
here
in
this
diagram
below
we
have
a
little
example.
So
unshare
in
linux
is
create
me
a
new
name
space
and
put
my
process
inside
of
it.
B
So
unshare
here
we're
going
to
create
a
net
name
space,
and
then
the
next
step
is
to
bind
mount
that
our
new
next
namespace
proc
self
and
s
net
to
a
specified
location
bar
one
net,
one
underneath
that
we
have
another
process
and
that
process
is
unsharing
or
joining.
In
this
case,
a
specified
name
space
here,
far
run
net1.
So
now
both
of
these
windows
are
in
the
same
net
name
space.
B
The
second,
which
was
kind
of
discovered
a
little
bit
later,
is
applying
cis
cuddles,
so
cis,
cuddles
and
kubernetes
are
at
a
pod
level,
and
typically
you
pass
the
cis
cuddles
to
the
creation
of
the
pause
container
and
then
the
runtime
configures
them
on
behalf
of
the
pod.
But
here
we
don't
have
a
runtime.
So
we
need
to
do
it
ourselves,
so
though,
the
time
you
do
it
is
after
you've
unshared
your
pod,
you
need
to
write
a
file
within.
B
B
The
third
step
is,
you
need
to
make
the
pod
containers
use
it
so
luckily
run
c
allows
you
to
configure
the
namespaces
inside
of
a
config
json.
So
here
we
have
the
two
examples
on
the
left.
We
have
the
old
way
of
doing
it,
where
we
have
the
proc
pid
entries
for
each
of
the
types
of
namespaces
that
we're
sharing
on
the
pod
level.
B
These
pod
level
name
spaces
were
created
on
behalf
of
the
by
the
runtime
when
the
pause
container
was
created
and
now
we're
using
the
pit
of
the
pause
container
to
reference
those
namespaces
in
cryo.
The
new
way
of
doing
it
is
referring
to
the
file
that
was
bind
mounted
so
on
the
right.
We
have
these
different
paths
that
are
the
bind
mounted
locations
of
the
newly
created
run
namespaces.
B
B
So
we
we
decided,
it
would
be
easier
and
you
know
more
convenient
to
do
and
see
so
we
exec
from
cryo
and
then
so
pin
and
s
unshares
its
own
name,
space
create
a
new
one.
Fine
mounts
it
into
a
specific
location,
then
returns
cryo,
take
oh
and
it
after
unsharing
it
configures
the
ciscos
and
then
it
returns.
Cryo
takes
those
newly
mounted
namespaces
and
hands
them
to
the
new
container
and
boom.
We
did
it
pause,
dropped
problem
solved
for
pretty
much
every
case.
B
However,
we
still
have
the
last
case
the
faded,
pid
namespace.
So
when
you
created
a
new
pid
namespace
you're
given
pid1
within
that
pid
namespace
and
pin
one
has
a
couple
of
special
responsibilities.
It's
responsible
for
reaping
all
of
the
children
within
that
pit
name
space
from
the
kernel's
process
table.
B
B
What
about
having
one
of
the
containers
be
pig
one
within
the
pit
name
space
you
know,
say
we
have
two
containers
in
our
pod,
an
nginx
container
and
some
sidecar
container.
Why
not
have
the
nginx
container
just
do
the
reaping
and
everything
well
number
one
we'd
have
to
ensure
ordering
the
pid
name.
Space
we'd
have
to
make
sure
that
the
pid
one
within
the
pid
name
space
exits
after
everything
else
to
actually
be
able
to
do
the
reaping
of
all
of
its
children,
or
else
its
children
will
become
zombified.
B
Also
we're
not
totally
positive
that
every
container
has
a
reaping
capability,
so
it
you
actually
have
to
you
know
specif,
there's
some
code
that
you
need
to
call
to
actually
wait
for
all
of
your
children
to
exit
and
that
reaps
them
from
the
process
table.
So
you
know
joe
schmoe
container
may
not
have
the
reaping
capabilities,
and
ultimately
the
pause
container
already
does.
All
of
this.
We
already
make
sure
that
the
pause
container
is
the
first
one
to
enter
the
pod
and
the
last
one
to
leave.
B
So,
unfortunately,
you
know
we
can't
drop
the
pause
container
for
this
one
case,
the
pit
name
space
case,
but
luckily
most
most
containers
have
a
private
pit
namespace,
which
means
they're.
They
create
one,
a
new
one
for
each
new
container
as
the
private
bid
name
space
is
the
default.
So
it's
only
when
a
user
specifically
specifies
hey.
I
need
a
pod
level
name
space.
Can
we
not
drop
the
pause
container?
B
So
let's
talk
a
little
bit
about
cryo's
journey
from
you
know
the
original
way
to
the
new
world
of
dropping
the
pause
container.
So
we
started
out
configuring
pods
with
namespaces
via
the
proc
entry
for
the
pit
of
the
pause
container,
as
we
saw
in
the
diagram
showing
the
two
different
name,
space
configurations
for
the
newly
new
container
config
json.
B
That
was
represented
the
option
with
the
pause
container,
which
was
the
original
option,
then
to
add
support
for
kata
containers.
We
added
an
option:
manage
network
ns
lifecycle
which
pre-configured
the
network
namespaces
and
did
a
similar
thing
to
what
pin
in
s
does,
but
only
for
the
network
namespace
that
was
introduced
by
the
time.
Cryo
was
had
its
initial
release
and
that
allowed
cod
of
vms
to
work.
B
So
now
you
can
run
cryo
crowd,
sandbox
inside
of
a
kata
vm
later
we
added
the
option:
managenice
lifecycle
to
be
able
to
configure
most
other
namespaces,
so
the
manage
ns.
So
we
had
a
security
vulnerability
about
a
a
little
bit
less
than
a
year
ago.
Now
that
was
discovered
where
containers
would
try
to
join
a
proc
entry
that
was
stale,
so
cryo
had
some
stale
state
in
it,
and
this
happened
through
some
weird
interactions
with
the
killer,
and
then
the
the
container
could
join
a
namespace.
B
B
The
security
vulnerability
was
also
mitigated
in
a
couple
of
other
ways.
So
it's
you
don't
need
to
set
manage
ns
life
cycle
to
be
mitigated,
but
it
is,
you
know
the
final
nail
in
the
coffin
to
defeat
that
vulnerability
and
finally
for
performance
we
use
the
infrastructure
already
created
from
anagenis
lifecycle,
mainly
pin
and
s,
and
then
we
reconfigured
some
things
in
cryo
to
drop
the
infra
and
we're
we
are
pushing
for
that
to
make
it
in
for
cryo
119,
which
releases
with
cube
119..
B
It
also
has
a
an
infra
container
to
hold
the
name
spaces
for
the
pod,
so
we're
going
to
investigate
using
pin
and
s
for
that
case
as
well
as
well
as
for
the
rootless,
pod
man,
there's
a
username
space
that
every
container
is
put
into
and
we're
going
to
look
at.
Maybe
having
pin
and
nest
can
figure
that
as
well.
B
B
B
B
B
B
B
Notice
here,
if
we
remember
our
pid
value,
we
have
the
proc
pid
and
that's
net.
So
this
is
the
net
name
space
that
runs
he
created
for
the
pod.
That
crowl
then
took
from
the
pause
took
with
respect
to
the
pause,
containers,
pid
and
passed
it
down
to
run
c,
to
create
and
to
in
creating
a
container
use
that
network
namespace
same
with
ipc
and
ets.
B
So
we
have
hello
to
create
and
then
so,
let's
do
a
cry:
cuddle!
Yes,
pods!
So
notice.
We
have
hello
2
here.
This
is
our
id.
Let's
do
a
run
c
list
and
look
for
our
pause
container.
Well,
we
have
no
pause
container.
We
asked
it
to
drop
it.
Of
course
we
did
so
in
in
this
situation.
We've
dropped
the
pause
container
and
there's
no
config
blob
that
we're
keeping
associated
with
it
all
we've
done
is
have
pin
and
s
create
the
namespaces.
So
let's
look
instead
at.
B
Looking
for
its
config
blob,
let's
take
a
look
at
it,
so
here
notice
we
have
our
newly
our
our
newly
formatted
namespace
paths.
So
this
was
a
namespace
that
was
unshared
from
pin
and
s,
and
then
we
have
we
configured
the
ciscos
and
mounted
the
mounted
it
at
this
location,
and
then
cryo
took
that
location
and
passed
it
down
to
the
newly
created
container
and
now
the
container
is
operating
within
the
pods
namespaces
and
any
old
process
can
unshare
into
if
I
copy
it.
B
B
So
here
we
have
a
script
that
was
run
so
here
it's
just
creating
a
hundred,
unique
pods
waiting
for
them
to
be
done
being
created
and
then
removing
the
pods.
I
there's
actually
a
typo
here
that
I
missed
that
all
these
pod
creations
were
done
in
the
background,
so
wait.
It
was
waiting
for
all
of
the
children
to
be
created,
so
the
we
have
all
of
those
pods
created
and
then
we
wait
and
then
we
remove
them
all
and
I
ran
so.
B
B
So
here
we
have
some
results.
We
used
multi-time
to
compile
and
do
all
the
math
for
us,
so
the
first
one
is
with
the
pause
dropped
and
the
second
one
is
with
keeping
the
pause
now,
a
couple
of
things
to
note
so
notice
how
the
the
real,
the
user
and
the
cis
time
for
both
of
them
are
actually
pretty
similar
and
that's
expected
cry.
Cuddle
just
makes
a
request
to
cryo
cryo
does
all
the
actual
work
and
then
returns
a
response
to
crycuddle.
B
So
time
won't
be
calculating
the
user
in
the
system.
For
the
actual
operation,
because
that's
happening
in
cryo,
not
cry
cuddle
and
we're
only
doing
time
on
cry
cuddle,
so
the
only
metric
that
we
really
get
is
with
the
real
time.
Unfortunately,
the
real
time
is,
you
know
a
little
bit.
It's
not
like
the
most
accurate
because
it
accounts
for
kernel,
interrupts
and
everything.
B
But
you
know
I
I
run
it
a
couple
of
times
and
there
has
been
a
little
bit
of
variation,
but
ultimately
there
is
a
savings
in
runtime,
there's
also
a
savings
in
overhead
which
we
have
we
don't
show
here.
But
if
900
kilobytes
for
every
pause
container
and
say
you
know
max
200
pods
on
a
node,
that
means
we
have
200
megs
for
our
pause
container.
B
All
of
the
pause
containers
on
the
node,
which
you
know,
isn't
a
ton,
but
all
of
that
could
be
better
used
by
you
know
the
workloads
instead
of
the
infrastructure,
so
dropping
the
pause
container,
fix
that
and
also
we're
saving
on
all
of
the
code.
That
would
go
through
to
create
the
pause
container.
We're
saving
on
the
run
c
exec
we're
saving
on
a
con
mon,
paying
attention
to
the
pause
container,
accounting
for
when
it
spuriously
gets
killed.
All
of
these
different
things
totally
avoided.
A
B
B
B
A
There's
one
question
on
the
kubernetes
slack
that
why
don't
we
use
ipnetinus,
so
the
response
is
like
ipnet10s
only
handles
a
network
namespace,
so
we
wrote
this
custom
toolpin
ns
that
can
handle
all
the
namespaces
and
also
we
don't
want
to
stomp
on
the
system
directories.
So
we
allow
you
to
customize
the
directory
in
which
the
namespaces
are.