►
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe in Amsterdam, The Netherlands from April 17-21, 2023. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
CRI-O’s Senior Year - Peter Hunt, Urvashi Mohnani, Mrunal Patel, Red Hat
Speakers: Urvashi Mohnani, Peter Hunt, Mrunal Patel
As CRI-O approaches CNCF graduation, it continues in its mission to provide a stable and secure OCI implementation of the Kubernetes CRI. Join the CRI-O team as they give an overview of CRI-O as well as talk about some new work, such as the progress on the new container monitor conmon-rs, rewritten completely in Rust. The team will also talk about the integration with sigstore to secure CRI-O’s supply chain as well as some of the interesting work being done in CRI-O to stay in-line with upstream Kubernetes. These include the stats collection rework as well as the work to support evented PLEG. Audience members will leave with an understanding of what CRI-O is, and where it is going.
A
Hey
everyone.
Thank
you
for
joining
Peter
hunt,
I'm.
Also
a
software
engineer
at
Red
Hat,
working
primarily
on
cryo,
but
sometimes
on
keyboard
and
signode,
and
sometimes
run
C.
Podman
container
related
Technologies
appreciate
everyone
coming
out
right
before
the
party
hope
to
get
you
guys.
You
all
pumped
up
ready
to
go
for
this
evening
and
we're
gonna
be
talking
about
cryo
senior
year,
so
I'm
going
to
start
off
quick
introduction.
What
is
cryo
it
gets.
A
You
know
and
cryo
is
an
implementation
of
the
kubernetes
container
runtime
interface,
compliant
with
the
open
container
initiative.
That's
a
lot
of
jargon,
a
lot
of
acronyms
what
you
can
take
away
from
that
is.
It
takes
the
spot
and
the
stack
that
Docker
used
to
occupy,
so
it
pulls
the
oci
container
images
and
starts
the
containers
and
pods
it's
responsible
for
all
the
operation
underneath
the
cubelet,
but
before
the
oci
container
runtime.
A
So
here's
a
quick,
architectural
diagram
of
what
cryo
is
so
you
see
on
the
left
here
cry
a
cubelet
talks
via
grpc
to
cryo
cryo.
You
know,
cubelet,
ask
cryo
to
pull
an
image
or
create
a
container
or
start
a
pod.
Cryo's
image
and
runtime
Services
then
are
responsible
for
doing
that,
underneath
the
so
for
the
image
service.
Crow
has
a
library
containers
image
which
actually
does
the
pulling
and
then
Crow
has
for
the
runtime
service
it
uses.
A
You
know
OSI
runtime
generation
to
actually
generate
the
container
spec
and
then
uses
an
oci
runtime
to
actually
start
the
container
for
a
pod.
It
uses
cni
to
provision
the
networking
resources
and
underneath
on
the
disk,
you
know
it
uses
container
storage
to
allocate
the
you
know.
The
disk
resources
like
the
copy
on
write
file
system
cry
also
has
an
entity
called
kanman
which
we'll
be
talking
about
a
little
bit
later,
but
it's
a
container
monitor
that
actually
pays
attention
to
the
life
cycle
of
the
container.
A
Why
would
anyone
use
cryo
and
why
do
we
implore
that
you
do
so
cryo?
Is
we
aim
to
make
cryo
secure
by
default?
So
you
know
we
try
to
have
a
minimal
attack
Surface
by
reducing
the
set
of
operations
that
cryo
is
responsible
for
so
with
other
generic
container
run
times
as
responsible
for
a
lot
of
a
lot
more
operations
in
cryo,
which
is
only
looking
to
satisfy
the
kubernetes
CRI
and
the
things
that
the
cubelet
wants
it
to
do.
A
A
We
had
experimental
support
for
user
namespaces
for
years
before
we
just
recently
added
it
to
the
actual
CRI
and
that
allowed
folks
to
test
it
out,
and
you
know,
work
out
some
issues
with
it
beforehand
and
also
you
know
it
was
specified
through
annotation,
so
it
was
actually
consumable
by
you
know.
In
kubernetes
we
also
ship
with
a
smaller
capability
set,
because
we
don't
expect
you
know
we
want
the
containers
that
are
run
in
production,
which
is
cryo's
main
priority
to
be
as
secure
as
possible.
A
Cry
also
aims
to
be
as
performant
as
possible,
specifically
for
kubernetes,
because
crowd's
behavior
is
customized
for
kubernetes,
common
operations
are
optimized,
for
so
you
know
currently
with
the
generic
plague.
The
Pod
life
cycle
event
generator
the
way
that
the
cubelet
maintains
the
state
of
what
the
status
of
the
pods
and
containers
are
is
by
frequently
re-listing
and
asking
cryo
hey.
What's
my
containers,
what
are
my
containers?
What
are
my
pods?
A
What
are
my
containers,
and
it
does
that
over
and
over
again
fairly
frequently
to
maintain
its
state
machine
while
we're
improving
that
asynchronously
throughout
this
time,
cryo
has
optimized
for
that
use
case,
because
we
know
qubit
is
going
to
be
doing
that.
A
lot
I
also
be
talking
about
another
optimization
that
cryo
another
set
of
optimizations.
That
cryo
has
made
a
little
bit
in
a
little
bit.
A
Ultimately,
what
we
want
for
cryo
is
for
it
to
be
boring.
So
you
know,
even
though
we
have
plenty
of
these
exciting
features
that
you
know
optimize
for
kubernetes.
What
we
want
really
is
that
an
admin
chooses
cryo
as
their
container
runtime
after
deliberating
over
the
options
and
then
promptly
forgets
about
it,
because
it's
doing
the
job
that
it
needs
to
and
really
nothing.
A
And
we're
excited
to
announce
that
we
are
in
our
senior
year.
Hopefully
we
have
just
recently
applied
for
graduation
and
you
know
we're
excited.
We've
been
in
the
as
the
CNN.
We
cry
was
born
in
the
kubernetes
sandbox
kubernetes
incubator
in
2016,
and
then
it
was.
It
moved
up
to
a
Sandbox
project
which
it
is
now
in
2019
and
we're
excited.
You
know
we
have
a
lot
of
production
users
that
are
using
cryo
openshift.
A
You
know,
Susan
was
for
a
while
lift
and
so
we're
we're
happy
with
those
relationships,
and
we
think
that
cryo
is
very
ready
for
production.
Now
it
has
been
for
a
long
time,
but
I
think
we've
really
proven
it.
At
this
point.
We
got
a
security
audit
in
the
spring
and
they
had
mostly
pretty
good
things
to
say,
and
you
know
the
couple
of
bad
things
we
fixed
already.
So
now
it's
all
good
things,
and
so
we're
excited
for
graduation
and
hope
to
have
that
go
through
soon.
A
So
next
up,
we
are
going
to
talk
a
little
bit
about
the
container,
monitor
that
I
mentioned
earlier
kanman
and
you're,
going
to
talk
about
a
rewrite
of
kanman.
First
off
we're
going
to
start
out
what
is
kanman
kanman's
a
little
helper
agent
that
manages
the
life
cycle
of
the
container.
Specifically,
it
actually
starts
the
oci
runtime
process
and
it
watches
for
the
exit
of
the
container.
A
It
manages
the
logs
of
the
container,
takes
the
logs
from
standard
out
and
writes
them
to
disk
there's
one
instance
of
kanman
per
container,
and
also
one
per
exact
sync
request.
So
if
you
have
an
exact
probe,
there's
actually
a
common
one
running
underneath
there,
that
will
describe
why
that's
important
later
kanban
is
called
by
cryo
over
the
CLI.
So
it
has
this
pretty
large
CLI
set
now,
and
it's
currently
written
in
C.
So
we
have
a
couple
of
reasons
that
we
want
to
go
through.
The
process
of
re-reading.
A
Kanman
specifically,
some
of
the
points
that
I
mentioned
earlier
and
why
the
consequences
of
those
like
one
container
per
kanman,
one
come
on
per
container
and
per
exact
session,
means
that
there's
a
lot
more
process
overhead
than
we'd.
Ideally
like
everyone
here
knows
now
that
the
standard
unit
of
container
is
POD,
you
know
it's
2022
kubernetes
is
largely
one.
A
You
know
with
the
more
versioned
you
know,
you
know
smart
IPC
mechanism,
and
you
know
it's
a
it's
a
bit
of
a
clunky
program.
You
know
it's
a
little
tough
to
work
with.
Sometimes
it
was
written
in
C
and
it's
you
know
started
a
long
time
ago,
and
you
know
we.
It's
really
served
us
well
and
it's
very
stable,
but
you
know
we're
looking
forward
and
looking
up
and
trying
to
do
new
things.
A
You
know,
C
has
served
us
well,
but
everyone
knows
the
common
pitfalls
of
c
and
a
lot
of
the
difficulties
that
can
arise
from
using
it,
and
so
we
want
to
kind
of
work
around.
You
know
work
around
those
by
having
a
language
that
works
a
little
bit
better.
So
I
am
happy
to
introduce
to
you
a
program
that
actually
satisfies
all
of
these
constraints
and
more
we
have
kanman
RS,
you
know
I.
A
Originally,
it
would
have
been
cool
if
we
could
have
named
it
podman,
because
you
know
it's
a
pod
monitor
after
all,
but
for
some
reason
people
said
that
that
would
be
confusing.
You
know,
I
wonder
why.
So,
instead
we
settled
on
kanman
RS,
a
rust
implementation
of
kanman.
It
currently
covers
the
scope
of
the
existing
kanban,
but
with
a
couple
of
differences,
some
highlights
of
common
RS
is
it?
Has
we
have
a
native
golang
client
API,
which
wraps
Captain
Proto
as
a
protocol?
A
Cam
Proto
is
an
API
framework,
that's
used
by
cloudflare
and
a
couple
of
other
folks
that
basically
advertises
zero
serialization
between
the
different,
even
different
languages.
So
we
can
speak
between,
go
and
rust
and
it
basically
passes
a
block
of
memory
over
and
those
are
each
and
mapped
into
the
different
languages.
So
it's
very
fast
and
also
has
less
complexity
than
something
like
you
know,
grpc
or
tgrpc,
or
something
it
come
on.
A
It
aims
to
keep
a
low
memory
usage,
so
kanman
used
something
around
two
Megs
per
container
and
we're
aiming
you
know
something
between
four
and
six.
It
would
be
better
if
it
was
lower,
obviously,
and
we're
working
on
that,
but
we
really
want
there
to
be
no
memory
penalty
for
using
you
know,
I.
Theoretically,
there
should
not
be
a
memory
penalty
for
using
rust,
and
you
know
if
you
had
a
pod
with
you
know
two
kanbans
for
one
for
each
container
and
then
maybe
you
know
a
couple
of
con
mounts
for
the
execs.
A
We
aim
for
common
RS
to
be
much
less
memory
than
all
of
that
would
ultimately
end
up
using
common
RS
to
be
able
to
support
pod
level.
As
you
know,
to
support
all
the
containers
in
the
Pod.
It's
going
to
be
multi-threaded
where
kanman
used
to
be
single
threaded
or
still
single
threaded,
and
we
have
some
exciting
features
that
we're
able
to
enhance
kanman
RS
with
because
is
written
in
Rust.
A
So
it's
a
little
bit
easier
for
us
to
add
new
features
to
it
to
be
able
to
use
common
RS
in
cryo.
All
you
have
to
do
is
add
a
drop
in
file
to
crowds,
config
directory
and
specify
the
runtime
type
as
pod,
and
then
point
it
to
con
Monarch,
the
location
of
con
man,
RS
and
then
restart
cryo
and
cryo
will
come
up
use
column
on
our
acid.
Hopefully,
you
can
also
promptly
forget
about
that
choice.
A
Right
now
we
have
passing
CRI
tests
and
integration
tests,
kubernetes
and
and
no
tests,
basically
all
of
the
tests
that
cryo
expects.
For
you
know
a
fully
functioning
you
know
to
be
fully
compliant
with
kubernetes
common
RS
is
passing
now
we're
planning
for
integration
into
podman.
There's
a
couple
more
pieces
that
we
need
for
podman,
which
we've
not
quite
gotten
to
yet
so
we'll
talk
about
that
a
little
bit.
A
We
have
RPM
packages
available
and
we
also
have
static
binaries
available
for
each
commit.
So
if
you
want
to
download
and
use
it,
you
can
currently
and
we're
looking
for
adding
more
distributions
in
the
future,
as
we
stabilize
in
the
future
I'd
like
to
describe
to
you
a
world
in
which
we
have
all
the
pieces
that
kanman
used
to
have
so
there's
currently
some
gaps.
A
A
We
need
checkpointing
for
podman
and
also
actually
soon
for
cryo,
because
we're
adding
checkpoint
and
restore
functionality.
We
need
support
for
SEC
comp
notify
to
be
able
to
be
notified
when
a
container
uses
Asis
call
that
it
wasn't
supposed
to
or
wasn't
expected
to,
and
the
journal
D
log
driver,
which
you
know
just
adds
compatibility
to
what
podman
currently
uses.
A
We
also
looking
forward
have
some
features
that
we're
pretty
excited
about
hidden
name
space
holding
is
one
that
I
personally
am
really
excited
about
So.
Currently,
if
you
have
a
pod
level
pin
namespace,
you
need
the
infra
container
to
hold
open
that
namespace,
because
you
need
to
have
the
pid1
in
the
namespace
to
stay
alive
for
the
duration
of
all
of
the
processes
within
the
pin
name
space.
A
Currently,
the
infra
container
I
mean
it
works.
This
worked
for
a
long
time,
but
it
adds
a
little
bit
of
process,
complexity
and
overhead,
and
it
would
be
nice
if
we
just
had
some
process.
Imagine
what
a
process
that
survived
for
the
duration
of
the
Pod
and
was
able
to
hold
the
Pig
namespace
for
it
and
con
man
RS
satisfies
those
requirements,
so
I
think
it'd
be
cool
to
be
able
to
have
the
pin
name
space
held
by
kanman
RS
and
then
we'd
be
able
to
drop
the
input
container.
A
We
want
a
logging
rate
limiting
so
and
just
additional
log
drivers
in
general.
We
have
you
know
right
now,
there's
like
the
kubernetes
log
format
and
we're
looking
towards
Journal
D,
but
also
we
have
the
opportunity
for
a
Json
logging
driver
which
podman
has
wanted
for
years
ever
and
has
never
actually
gotten
and
so
being
written
in
a
more
modern
language.
It'll
be
easier
to
integrate
some
of
these
new
features
and
open
Telemetry
tracing
which
I'm
actually
excited
to
say.
A
So
that's
come
on
RS,
we're
pretty
excited
for
it
and
now
I'm
going
now
for
something
completely
different.
We're
going
to
talk
a
little
bit
about
some
load
optimizations
and
specifically,
some
better
reporting
mechanisms
that
cryo
has
added
semi-recently.
So,
if
you've
attended
some
of
these
talks
before
you
might
have
seen
me
talking
about
situations
of
load
and
kubernetes
and
specifically
between
cryo
and
cubelet,
but
I'll
give
a
refresher
for
anyone
who's
new
here.
A
So
the
problem
is
in
situations
of
load,
cryo
and
Cuba
get
into
this
bickering
match
where
they
have
trouble
syncing
up
between.
So
basically,
the
cubelet
needs
to
have
a
Timeout
on
each
container
and
pod
creation
request,
because
it
needs
to
know
that
when
it
sends
out
that
request
that
request-
and
it
just
disappear
into
the
void.
The
problem
about
that
is
that
cryo,
you
know,
can
take
an
undetermined
amount
of
time
to
create
a
pottery
container,
especially
under
load,
usually
there's
some
entity.
That
is
the
bottleneck.
A
So
this
has
been
largely
solved
in
cryo.
As
of
you
know,
119,
or
something
like
that
and
I'll
describe
so
it's
quite
old
now,
but
I'll
describe
the
solution
because
we've
added
a
little
bit
to
that
solution.
So
the
solution
to
it
is
to
fine-tune
our
Behavior
to
the
cubelet,
which
cry
I
was
able
to
do
because
it's
you
know
only
for
the
only
client
that
cryo
materially
cares
about
is
the
cubelet
and
running.
You
know:
containers
in
production,
so
the
cube.
A
The
the
basic
idea
is
to
finish
creating
the
resource
and
then
save
it
into
the
cube.
Last
again
so
walk
us
through
that
example.
So
cubelet
asked
Crow
hey.
Can
you
create
me
this
pod
and
Kyle's
like
sure
I'll
Reserve
this
name,
so
that
no
one
else
know
cry
cuddle
or
someone
can
come
in
and
try
to
take
that
pod
from
you
at
some
point
in
that
process?
Maybe
cryo
gets
stuck
on
sdn
because
it's
taking
a
real
long
time,
there's
a
lot
of
pause
being
created
at
the
same
time.
A
Something
crowd
takes
too
long
and
the
cubelet
times
out
Cuba's,
not
sure
whether
the
you
know
timeout
is
because
the
request
disappeared
or
because
you
know
it
actually
is
just
taking
too
long.
So
just
in
case
qubit
re-requests.
Hey.
Can
you
create
me
this
pod?
The
second
routine
is
aware
that
the
first
routine
is
working
on
creating
that
plot,
because
it
can
tell
that
the
name
is
already
reserved.
So
what
it
does
is
it
waits
and
basically
tells
the
first
routine
hey
when
you're
done,
creating
that.
A
Let
me
know
because
I'm
trying
to
give
it
to
the
cubelet.
Eventually
the
bottleneck
either
clears
up,
or
you
know,
the
networking,
your
pot
you're.
Finally,
you
know
on
gums
and
crowd's
able
to
create
the
resource
so
crowd
that
the
first
routine
detects
that
there
was
a
timeout-
and
you
know,
pings
the
second
routine,
saying
hey
I'm
done
my
thing
now
and
returns
to
the
cubelet.
You
know
Cuba's
not
paying
attention
anymore.
It
assumed
that
the
request
disappeared.
A
So
this
dotted
line
represents
Cuba's,
not
really
listening
anymore,
but
what
it
is
listening
to
is
the
second
routine,
which
has
said
you
know:
hey
I
have
this
resource
for
you,
but
you
need
to
request
again
the
reason
we
do
another
round
of
requests,
even
though
the
resources
already
made
is
because
you
know
the
resource
there
might
be
a
race
between
the
Cuba
timing
out
when
cryo
sends
it.
So
we
worry
about
you
know
having
this
resource
disappear.
A
So
instead
we
we
time
out
one
more
time
in
the
cubelet
and
then
await
the
inevitable
cubelet
re-request,
that
resource
from
another
cryo
routine.
That
routine
sees
that
it
has
the
resource
and
it's
able
to
return
it
no
problem.
So
this
this
was
a
good
optimization
because
it
changed
the
situation
where
there
used
to
be
a
lot
of
thrashing
on
the
Node.
Where
container
would
be
created,
it
would
take
some
time
and
then
it
would
have
to
be.
A
You
know
the
the
creation
would
time
out
and
at
the
time
we
actually
removed
the
container
because
it
aired
out,
and
that
was
bad
because
that
actually
made
the
the
the
resources
be
even
more
constrained.
So
now
you
know,
we've
improved
the
situation
a
lot
now
we're
basically
returning
the
resource
just
about
as
soon
as
it's
actually
done.
A
Another
you
know,
small
piece
of
it
is
we're
actually
throttling
the
requests
from
the
qubit.
So
if
the
cubelet
had
its
way
or
the
way
that
it
used
to
work,
is
you
know
if
cryo
was
just
like
immediately
after
getting
the
duplicated
request?
Hey
I'm,
already
working
on
it,
I'm
gonna
return
an
error
to
you
cubelet,
then
the
qubit
and
cry
like
cubelet
would
propagate
through
the
event.
Api
like
a
whole
bunch
of
like
names,
reserved
name
is
reserve
because
cry
would
be
returning.
A
These
errors
really
quickly
and
Cuba
is
trying
to
create
this
pot
as
fast
as
it
can.
What
because
we
know
that
the
Cuba
is
going
to
re-request
this
object,
you
know
as
long
as
the
air
keeps
being
time
out.
What
crowd
can
do
is
it
can
actually
throttle
the
cubelet
and
wait
on
in
this
routine
too
saying
hold
on
Cuba
slow
down
we're
working
on
it?
Don't
worry
taking
as
long
as
it
can
so
that
reduces
the
number
of
events
in
the
Pod
API.
A
So
that's
very
nice
and
reduces
churn
and
makes
the
admins
not
as
scared,
because
there's
not
a
bajillion
messages,
saying
God's
name
is
reserved
a
new
thing
that
we've
done
semi
recently,
which
I'm
also
very
excited
about.
Is
we
return
where
the
Potter
container
creation
is
stuck?
So
we
used
to
just
say
this
kind
of
generic
error,
like
famous
Reserve,
we're
working
on
a
cubelet
chill
out,
but
now
we're
also
keeping
track
of
where
in
the
container
and
pod
creation
process,
that
resource
is
stuck
and
that
way
we're
able
to
return.
A
So
when
resource,
when
you
know
one
of
the
one
of
the
crowd
routines
that
isn't
the
original
one
returns,
an
errors,
saying
hey,
you
know
we're
working
on
it,
but
it's
not
done
yet.
It
also
says-
and
it's
actually
stuck
at
you
know
something
like
currently
at
stage
sandbox
Network
created.
So
it's
saying,
like
you
know,
sometime
after
the
sandbox
network,
was
created,
we're
stuck
or
says
something
like
you
know:
sandbox
storage
creation.
A
B
Folks,
everyone
must
have
six
tours
well
I'm,
happy
to
happy
to
cryo
her
support
for
six
tours
style
signatures,
so
the
containers
image
Library
the
trial
uses
for
pulling
images,
merge
support
for
verifying
those
signatures.
Oddman
fully
supports
it.
Supportment
can
be
used
to
sign
and
push
those
images
and
the
signatures
to
a
registry,
and
then
you
can
use
cryo
to
verify
those
signatures.
When
you
pull
the
images,
however,
we
need
to
improve
the
UI
of
what
happens
when
a
signature
verification
fails.
B
B
B
B
So
first
of
them
is
username
spaces,
so
so
far
like
the
like,
this
username
is
supported
in
the
Linux
kernel,
but
kubernetes
has
not
been
able
to
take
advantage
of
it.
So
Peter
mentioned
that
we
had
annotations
based
support
in
cryo,
but
finally
in
125
we
got
phase
one
alpha
for
username
spaces
merged
into
kubernetes.
B
So
this
phase
one
supports
stateless
spots.
It
means
any
pods
that
don't
use
persistent
volumes
will
work
with
username
spaces,
so
the
supported
volume
types
are
like
empty:
the
secrets,
config
maps
and
so
on.
So
with
username
spaces,
you
get
an
additional
layer
of
security
in
your
parts,
so
you
can
be
root
inside
your
pod,
while
being
non-root
on
the
host.
What
it
means
is
if
the
process
is
able
to
break
outside
of
the
container,
it's
not
able
to
attack
the
host
or
other
containers
running
on
the
Node.
B
B
B
Restore
so
this
is
another
feature
that
was
merged
into
the
cubelet,
so
basically
it
uses
the
criu
to
checkpoint
container
state.
It's
only
a
cubelet
API
at
the
moment,
so
there's
no
kubernetes
API
for
it.
So
if
you
want
to
use
this
feature,
you
have
to
hop
on
a
node
and
directly
hit
the
queue
the
cubelets
endpoint
to
checkpoint
the
container.
B
B
Then
checkpoint,
the
state
of
that
quad
and
then
you
can
move
that
checkpointed
state
to
another
node,
and
then
you
can
start
that
pod
back
up
again,
so
you
can
analyze
what's
what
was
happening
inside
that
ball,
and
this
can
happen
without
the
knowledge
of
the
attacker
like
they
can
continue
being
in
that
part.
So.
B
Like
this
is
like
a
security
feature,
so
other
use
cases
of
checkpoint
restore
like
faster
startup
for
parts
such
as
jvm.
So
we
know
that
Java
takes
time
to
start
up
right.
So
potentially
we
can
checkpoint
a
pod.
After
that,
startup
phase
is
done
and
then
you
can
launch
hundreds
of
copies
of
such
pods
right,
so
that'll
be
a
net
Improvement
in
startup
time.
B
B
And
finally
invented
like
so
today,
the
cubelet
uses
the
generic
plate.
What
is
plague
so
it's
a
pod
life
cycle
event
generator.
So
that's
what
is
being
used
by
the
cubelet
to
materialize,
the
life
cycle
of
a
pot.
So
once
the
pubelet
starts
a
pod,
it
needs
to
be
aware
when
it
dies
or
it
gets
killed
for
some
reason,
because
you
know
whenever
it
dies,
cubelet
starts
it
back
for
you
right
and.
B
It
know
that
it
knows
it
through
the
plague.
The
way
the
generic
flag
works
is
the
cubelet
periodically
realists
all
the
pods
and
containers
from
the
runtime
over
CRI.
Now
the
overhead
of
doing
this
is
very
high.
When
there
are
lots
of
faults,
like
imagine,
you're
trying
to
push
the
boundaries
of
how
many
pods
you
can
run
on
a
road,
you
are
at
600,
700
nodes
and
very
frequently
a
cubelet
is
requesting
this
list
of
pods
from
the
runtime.
So
this
adds
a
lot
of
overhead
to
both
the
cubelet
and
the
container
runtime.
B
B
A
You
also
so
in
in
a
tragic
event
where
conman
dies
before
the
container
does
then
also
we
don't
catch
the
container's
exit,
because
kanman
is
the
parent
of
the
container
process.
So
it's
the
one
that
can
catch
the
Sig
child
that
the
kernel
sends
when
the
process
ends.
Cryo's,
not
you
know
at
all
at
common,
actually
demonizes.
So
conman
is
a
child.
That
system
B,
not
even
of
cryo,
so
crowd
can't
catch
those.
A
So
you
know
at
one
point:
we
had
a
kanman
monitor,
but
con
man
we
called
it,
but
it
was
racy
and
kind
of
difficult
to
work
with
so
we've
you
know
made
it.
We
worked
hard
to
make
sure
that
conman
isn't
going
to
segfault
or
anything
like
that
or
exit
before
the
container
does
and
it
actually
can't
be
boom
killed
either,
because
that
would
be
in
a
bad
State.
We
hope
to
we're
looking
at
situations.
Pid
fds
would
help
with
that.
Where
we
can.
A
A
A
As
of
now
it
does
not,
it
would
I
agree
that
that
would
be
even
more
problematic
of
a
situation
and
so
we're
you
know
it
is
on
our
radar
trying
to
figure
out
the
way
to
actually
pay
attention.
You
know,
I
really
look
forward
to
Pitt
fds
in
general.
Konmon
could
also
use
them,
but
you
know
before
that.
Maybe
we'll
come
up
with
some
ebpf
thing
so
that,
like
that
risky
situation
doesn't
come
up.
A
Because
I
wanted
to
have
a
good
resume,
no,
because
the
well
because
it
like
was
a
natural
fit
for
kind
of
the
the
piece
of
the
stack
it
was.
You
know
we
already
had
conman
NC,
but
it
would
have
been
kind
of
clunky
to
add
you
know
asynchrony,
and
you
know
all
of
these
things
and
go
would
be
the
go
run.
Time
uses
too
much
memory
for
the
number
of
con
mounts
that
we
want.
A
D
B
I
think
that's
just
something
that
we
ship,
so
it's
easy
for
you
to
get
started
with
cryo
right.
So
you
can
easily
start
up
a
local,
appluster
and
you'll
be
able
to
see
your
parts,
but
that's
something.
We
never
recommend
for
production
use
cases
and
we
make
it
easy
for
you
to
like
Drop
in
any
see
an
icon
and
the
binary,
and
it
should
just
work.
D
D
B
The
second
part
of
it
like
improving
what
we
log
on
the
cubelet
side
of
things
like
the
whole
cubelet
code,
that
manages
sync
pod
and
generates
all
this
is,
is
hard
and
what
we
are
hoping
is
with
events
like.
We
have
some
ideas
on
how
to
simplify
it
and
like
make
it
better
and
also
folks
on
the
community.
Folks
on
Google
are
also
working
on
some
documentation,
so
more
people
can
come
up
to
speed,
contribute
and
simplify
that
whole
area.
Yeah.
A
The
the
existence
of
tracing
support
into
Cuba.
We
also
had
the
idea
we
were
talking
about
it
today,
like
improv,
like
instrumenting,
sync
pod,
and
you
know,
or
the
the
Pod
workers
that
are
actually
doing
the
plague
you
know
generating
and
managing
if
we
were
instrumenting
them
with
open
Telemetry
like
maybe
that
could
also
give
some
better
insight
into.
What's
going
on.
There
you'll
be
able
to
watch
a
span
and
see
Its
Behavior
so
that.
A
It
so
it
does
have
some
preliminary
support.
It,
like
I,
think
we're
still
working
through
trying
to
get
like
figure
out
exactly
the
granularity
of
spans.
We
have
it
on.
You
know
we
have
a
number
of
functions
that
are
like
emitting
them.
I
think
that's,
but
it's
in
like
the
main
branch.
It's
like
not
yet
been
released,
I
mean
kanman
still
has
only
it's
only
on
like
0.3.0
or
something
like
that,
so
it's
pretty
young
there
could
have
been
a
demo
that
we
did
today,
but
I
didn't
have
time
to
put
it
together.
A
A
Yeah
yeah
I
would
say
long
term.
You
know,
assuming
that
we
can.
You
know,
make
sure
that
and
I'm
pretty
sure
that
we
could.
This
can
be
the
case,
but
you
know
I
just
want
to
maintain
that
one
of
the
main
ideas
is
to
make
sure
that
the
ultimate
memory
usage
is
not
egregiously
more
than
it
was
with
kanman,
assuming
that
which
I
think
will
be
the
case,
then
eventually
we'll
deprecate
and
remove
kanman
and
it'll
be
convon
RS
moving
forward.
C
I
wanted
to
say
very
impressive
work
with
kanman,
RS
and
I
appreciate
the
description
of
why
rust
might
be
the
good
tool
for
the
job.
Could
you
show
the
slide
again
on
how
to
switch
the
crio
runtime
and
then
maybe
elaborate
on
like
what
future
scenarios
people
might
consider?
Switching
to
conman
RS
to
be
an
early
adopter.
A
Yeah,
so
this
this
slide
is
the
one
you
were
looking
for
yeah.
So
there
you
go
what
so,
why
you
would
want
to
early
adopt
it.
So
so
what
are
the
things
that
we
kind
of
triggered
our
desire
to
do
this?
Is
we
wanted
better
C
group
accounting
of
the
kanman
resources?
So,
like
you
know,
in
situations
where
you
want
to
keep
the
kanman
on
a
separate
CPU
set
than
you
know,
the
container
is
running.
If
you
have
a
real-time
pod,
that's
doing
some
Network
latency
stuff.
A
You
could
you
know
that
was
originally
what
kind
of
got
us
started.
Thinking
about
having
a
pod
level
kanman
to
be
able
to
isolate
that.
So
that's
one
use
case.
I
mean
you
know,
we'd
be
happy.
If
folks
wanted
to
try
out-
and
let
us
know
you
know-
even
if
you
just
want
to
be
at
the
bleeding
edge,
because
you
know
that'll
help
us
iterate
on
getting
it
better
and
stuff,
I
would
say.
B
Yeah
I
mean
so
one
more
thing
that
we
wanted
to
tackle
is
like
right
now
when
we
do
an
exec
right,
we
are
spawning
a
new
common
and
its
study
State
on
a
node.
Your
execs
are
the
most
expensive
operation
that
a
container
runtime
is
doing.
So
we
want
to
reduce
that
process
hop
here.
So
with
common
RS
like
there
won't
be
an
additional
common,
we'll
talk
over
rpc2,
kanban
and
it'll
directly
do
a
run
C
or
a
c
Run
exit
drastically
reducing
your
CPU
usage.
A
The
the
other
other
thing
that
I
just
remembered
is
actually
it
can
help
with
CPU
and
memory
accounting
on
the
Node
so
because
we
have
multiple
kanmans
now
it's
hard
for
us
to
use
the
Pod
overhead
feature
which
was
originally
made
for
Kata,
because
you
have
like
a
VM
that
has
some
sort
of
standard
amount
of
memory
and
CPU.
But
you
know
if
we
have
a
variable
number
of
kanbans.
A
We
can't
really
well
guess
how
much
memory
and
CPU
they'll
all
use,
because
it's
not
proportional
to
the
number
of
containers
with
pod,
a
kanban
RS,
it's
one
per
pod.
We
can
guess
like
okay,
it's
not
going
to
use
more
than
eight
Megs.
Definitely
so
you
know
that's
the
pot
overhead
and
then
you
can
more
tightly
fit
your
containers
on
the
Node,
because
you
know
they're,
not
there's
not
this
like
mysterious
amount
of
memory,
that's
used
by
kanman.