►
From YouTube: CRI-O Still Loves Kubernetes - Sasha Grunert, Peter Hunt, Urvashi Mohnani & Mrunal Patel, Red Hat
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon North America 2021 in Los Angeles, CA from October 12-15. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
CRI-O Still Loves Kubernetes - Sasha Grunert, Peter Hunt, Urvashi Mohnani & Mrunal Patel, Red Hat
In Kubernetes 1.20, support for the dockershim was deprecated, leaving many wondering what will take its place. Wonder no longer: CRI-O is a container runtime written exclusively for Kubernetes, and is ready to take the dockershim’s place. In addition to being a standard component for deploying secure and stable Kubernetes clusters, CRI-O has the unique advantage of being able to tailor its behavior to the needs of the Kubelet. In this talk, the maintainers of CRI-O will provide an update about the latest feature developments, as well as live demonstrating typical real world use cases around them. Join the CRI-O maintainers as they walk through the latest improvements in communication with the Kubelet under load, container stats reporting, user namespaces, and seccomp profile generation, as well as a general project update. After this session, you should know more about how CRI-O works in action and why it’s the perfect choice for your Kubernetes cluster!
B
D
I
will
first
give
a
brief
overview
of
the
latest
developments
happening
in
cryo
and
then
my
co-presenters
will
dive
in
deeper
into
some
of
the
notable
features
we
have
added.
We
have
been
adding
more
metrics,
such
as
improved
cra
level.
Stats
that
could
be
exported
and
queried
through
meteors
sasha
is
also
working
on
adding
related
metrics.
That
will
give
a
better
visibility
into
room
kills
of
your
parts.
These
metrics
provide
a
good
foundation
to
add
custom
alerts
to
your
cluster.
D
D
Hardman
added
a
feature
called
short
name
aliases
that
we
have
now
integrated
into
cryo
as
well.
This
feature
allows
you
to
create
aliases
for
your
preferred
images
and
prevents
registry
image
squatters
from
running
malicious
images
on
your
cluster.
You
can
configure
aliases
for
your
critical
images
to
better
protect
your
cluster.
You
can
read
more
about
this
feature
from
the
article
linked
in
the
slides.
D
Another
small
feature
that
I
would
like
to
highlight
is
the
p
prof
over
unix
socket
support
by
default.
Pprof
works
over
http
endpoint
and
that
needs
to
be
protected,
enabling
it
over
the
unix
socket
keeps
the
endpoint
access
locked
down
by
default
and
provides
valuable
profiling
information
to
admins
and
developers
more
easily.
D
We
have
been
using
the
pattern
of
enabling
experimental
features
using
annotations
and
cryo
to
test
and
refine
them
before
proposing
them
for
inclusion,
as
first
class
features
in
kubernetes
ubershe
will
cover
username
spaces
and
sasha
will
cover
sitcom
profile
generation
in
the
talk
today.
Besides
those
we
have
support
for
specifying
shim
size
and
enabling
devices
and
containers
using
annotations,
you
can
check
out
our
docs
for
more
details.
D
If
you
aren't
aware,
docker
shim
is
slated
to
be
removed
from
the
cubelet
in
kubernetes
124..
You
can
read
the
linked
cap
for
more
information.
The
guidance
from
sig
node
is
to
move
to
a
cri
based
runtime,
and
we
want
cryo
to
be
your
preferred
runtime.
Now,
how
can
you
actually
switch
to
cryo
simple?
Actually,
there
are
two
steps.
D
The
first
step
is
to
install
and
start
cryo
service
on
your
node,
we
package
cryo
for
several
distributions.
The
second
step
is
to
configure
the
cubelet
to
point
to
the
cryosocket
as
a
container
runtime
endpoint
and
you
are
all
set
folks-
are
used
to
the
docker
cli
for
debugging
and
building
images.
Today
we
have
alternatives,
for
that.
Cryctl
is
a
debugging
tool
that
can
be
used
to
interact
with
a
cri
runtime
directory.
You
can
list
pods
list
containers,
inspect
them
exact
into
them.
Using
chrysatel.
D
B
So
we're
going
to
talk
through
the
fundamental
problem,
much
like
the
docker
shim
and
other
cri
implementations,
gilbert
and
cryo
have
a
client
and
server
relationship
because
of
that
the
cubit
must
set
timeouts
on
resource
creation
requests.
This
is
to
guarantee
the
eventual
part
of
eventual
consistency,
but
what
happens
when
one
of
these
timeouts
is
actually
hit
when
a
potter
container
is
being
created,
so
I'm
going
to
walk
through
this
scenario.
B
This
is
the
behavior
that
cryo
did
before
crow
119.,
so
the
the
cubelet
requests
a
resource
from
cryo
early
on
in
that
process,
crowl
reserves,
the
name.
This
is
a
primary
key
that
is
used
to
make
sure
multiple
clients
would
try
to
create
the
same
thing.
It
includes
the
attempt
number,
the
namespace,
the
pod
name
or
the
container
name
depending
on
what
type
of
resource
it
is
due
to
system
load.
At
some
point,
the
cryo
request
takes
longer
than
the
cubit
is
expecting
and
the
cuba
times
out.
B
Whatever
the
reason
is
the
cubelet
that
the
original
request
fails
from
the
perspective
of
the
cubelet,
because
the
timeout
happened.
It
doesn't
know
that
the
request
didn't
just
go
into
the
void
because
of
that
it
sends
a
new
create
resource
request,
but
cryo
is
already
in
the
middle
of
creating
the
original
resource
and
because
of
that,
crowd
returns
a
name
as
reserved
error,
because
that
unique
name
is
already
actively
being
created
in
another.
B
Go
routine
eventually,
cryo
detects
cryo
finishes
creating
the
resource
and
detects
that
there
was
a
timeout
and
cleans
up
all
of
its
work.
This
is
to
prevent
the
resource
from
leaking
until
the
cubic
garbage
collector
has
to
come
in
and
clean
it
up,
but
this
is
not
very
elegant
for
a
couple
of
reasons.
B
Number
one
during
this
time:
the
cubelet
where
the
cuban
cryo
are
doing
this
dance,
where
the
cupid
requests
the
resource
and
crowd
returns.
Saying
name
is
reserved
that
happens
every
10
seconds
and
that
every
time
that
happens,
an
error
propagates
through
the
kubernetes
event
system,
saying
name
is
reserved
the
pod
sandbox
creation
failed
or
container
creation
failed.
For
that
reason,
it's
kind
of
an
ugly
user
experience
on
top
of
that
cryo
completely
undoes.
All
of
its
work
after
the
timeout
happens,
crowd
cleans
up
the
resource
so
that
nothing
leaks.
B
B
So
one
thing
that
we
can
do
to
mitigate
one
of
the
problems
is
finish:
creating
the
resource
and
then
until
the
cubelet
asks
a
gun.
So
the
process
is
largely
the
same.
The
reserve,
the
name
is
reserved
timeout
happens,
resources
create
resource
comes
in
again,
name
is
observed,
error,
so
those
errors
are
still
propagating
through
the
system,
but
when
crowd
protect,
detects
the
timeout,
instead
of
undoing
all
of
the
work
that
it
had
previously
done,
it
stashes
it.
It
saves
that
resource.
B
It's
able
to
do
this
because
the
kubelet
is
a
you
know
a
special
client
in
that
it
has
very
specific
behavior
that
it
does.
We,
the
crowd
developers,
know
that
the
cubelet
is
going
to
re-request
that
same
resource
until
it
gets
created.
It's
going
to
keep
going
going
going
going,
not
only
that,
but
it's
going
to
keep
the
same
unique
name,
the
name
that
we
had
reserved
unless
the
api
server
changes
the
request.
But
then
it's
a
new
attempt
number,
which
is
a
new
name.
B
B
So
we
got
one
of
those
problems,
we're
no
longer
doing
redoing
work,
but
there's
still
the
problem
of
the
api
events
where,
during
this
process,
if
cryo
is
taking
five
minutes
to
create
this,
this
resource,
the
cubelet
is
going
to
be
attempting
to
get.
You
know
about
every
10
seconds,
cuba's
gonna
re-ask
for
this
thing,
which
you
know
four
minutes,
minus
five,
that's
one
left
and
then
so
that's
about
six
times
we're
gonna
get
this
event
through
the
system
and
that's
not
a
very
pretty
user
experience.
B
B
So
another
thing
that
cryo
is
able
to
do
and
has
done,
is
we're
going
to
throttle
new
requests
and
I've
kind
of
changed
up
the
diagram
a
little
bit,
because
it's
not
entirely
clear
if
there
aren't
different
routines.
So
cryo
r1
stands
for
routine
one,
the
routine,
the
go
routine.
That
is
processing
the
original
request.
B
B
Basically,
what
we're
doing
is
we're
stalling
the
cubelet
who
would
otherwise
be
sending
these
requests
every
10
seconds,
but
now
is
forced
to
wait
for
its
own
timeout
length,
at
least
or
at
most
its
own
timeout
length,
eventually
cryo,
the
original
routine
detects
that
the
timeout
happened
and
it
saves
the
resource.
This
is
consistent
with
what
we
did
before
and
cryo
returns
an
error.
B
B
Cuba
cancels
cryo
attempts
to
send
and
that
resource
gets
lost
so
to
protect
against
races
and
to
with
cryo.
This
second
one
returns
an
error.
Luckily,
cryo
knows
that
the
cube
is
going
to
re-request
that
resource
pretty
much
immediately
10
seconds
later
and
on
that
next
re-request
cryo
can
look
into
his
resource
store,
realize.
Oh
I've
already
got
this
and
return
it
immediately.
B
So
this
is
a
much
better
user
experience
and
also
mitigates
against
the
load
we're
now.
You
know,
instead
of
returning
one
of
these
errors,
every
10
seconds,
it's
once
every
four
minutes
and
we're
not
re
doing
all
of
the
work
that
we
did
in
the
original
routine.
We're
saving
that
resource,
just
as
we
you
know
should
and
the
moment
that
we're
able
to
we
send
that
resource
back
and
we're
only.
This
is
all
possible
because
cryo
is
able
to
tune
its
behavior
to
the
cubelet.
B
So
next
up
I'm
going
to
go
through
a
quick
demo.
So
here
we
have
a
kubernetes
local
kubernetes
cluster
setup,
we're
going
to
have
two
different
versions.
One
version
is
one
where
we
throw
away
the
progress
and
have
this
namer's
reserved
error
pop
up
and
the
other
one
is
going
to
be
one
where
the
new
version,
where
we
we
can
save
the
results
and
don't
throw
away
it
and
also
stall
the
the
cubelet.
B
So
here
the
note
is
just
bootstrapping,
so
I
have
to
admit
I
was
not
able
to
accurately
create
this
situation.
It's
hard
to
put
a
node
under
load
to
the
point
where
this
situation
pops
up,
but
the
node
is
not
totally
overwhelmed.
It's
hard
to
artificially
do
that,
so
I
have
spoofed
it.
I.
I
hope
that
you'll
trust
me
that
this
situation
is
largely
mitigated
in
production.
It's
a
situation
that
came
up
quite
a
bit
and
we're
seeing
a
lot
of
improvements
with
it.
B
But
yes,
it's
difficult
to
artificially
do
so.
The
way
that
I
artificially
did
it
was.
I
have
a
cni
plugin
that
I
created,
which
is
called
bridge
timeout
and
all
that
it
does
is
if
the
cni
command
is
add,
which
is
called
on
a
run.
Pod
sandbox
request
it
sleeps
for
four
minutes
and
then,
after
that
it
calls
the
bridge
plug-in.
It's
very
simple,
basically
does
what
the
bridge
plug-in
does,
but
a
lot
worse.
B
The
the
motivation
here
is:
we
have
seen
in
production
situations
where
our
cni
plug-ins
are
overwhelmed
by
the
rate
of
sandbox
creation
and
they
stall
the
node
from
being
able
to.
Actually
you
know,
create
the
resources,
so
we
ran
into
a
lot
of
names,
reserved
errors
from
cni
plug-ins
being
the
bottleneck.
B
B
So
this
is
our
first
failure
and
this
is
expected
because
the
cni
plug-in
took
that
long
and
then
what's
going
to
happen,
is
the
qubit's
going
to
attempt
to
recreate
that?
So
here
we
see,
this
name
is
reserved
air
and
this
propagates
all
through
the
system-
and
this
is
basically
the
cubit.
You
know
re-requesting
and
proud
being
like
I'm
in
the
middle
of
it
calm
down,
relax,
doing
nothing
to
actually
slow
down
the
cubelet.
So
this
air
is
gonna
pop
up
a
couple
of
times.
B
As
we
can
see
here.
We
have
a
you
know,
they're
separated
by
like
15
seconds,
and
so
this
is
a
bad
user
user
experience.
Eventually,
this
pod
will
attempt
to
be
recreated
and
that
could
pass
or,
as
we
know
it
will
fail,
because
four
minutes
will
have
been
exceeded
again.
So
in
this
situation
the
cubelet
and
cryo
would
never
reconcile,
but
we
can
do
better.
So
next
up,
we
have
an
example
with
our
new
cryobinary
and
we're
going
to
do
the
same
thing.
B
Then,
let's
fast
forward
to
when
the
timeout
is
about
to
happen,
so
the
timeout's
about
to
happen
we're
332,
348
355..
So
here
we
have
the
expected
context
deadline
exceeded,
which
we
can't
do
anything
about.
I
mean
it's,
the
cni
plug-in
that's
taking
a
long
time.
Cryo
can't
make
it
go
any
faster.
It's
the
same
way
about
you
know
the
latency
on
the
cloud
provider's
perspective,
but
what
kryol
can
do
is
successfully
save
that
resource
and
be
able
to
move
like
allow
the
cubelet
to
move
forward.
B
So
what
we're
about
to
see
is
so
it's
kind
of
confusing
here,
but
we're
gonna,
we're
gonna
show
the
described
pods.
So
here
we
have
the
failed
sandbox
request
19
seconds
ago,
and
then
we
see
the
pulling
message.
What
this
means
is
in
between
that
time.
The
creates
on
pod
sandbox
succeeded
because
the
cubelet
wants
the
pod
sandbox
request.
Succeeds.
The
cuba
then
starts
pulling
the
images
for
the
containers
themselves.
So
we
see
that
we're
pulling
the
alpine
container
it's
created
and
it
started
so
we
have
a
running
pod.
B
So
that's
it
for
the
demo
and
all
in
all.
We
have
a
situation
where
cryo
is
able
to
better
tune
its
behavior
to
the
needs
of
the
cuba,
and
this
is
uniquely
helpful
in
this
situation,
where
we
can
mitigate
the
problems
of
heavy
load
due
that
come
and
a
lot
of
the
pitfalls
that
resource
creation
falls
into.
A
All
right
so
now
we're
going
to
talk
about
username
spaces.
A
lot
of
work
has
been
done
on
enhancing
the
username
space
support
with
kubernetes
and
cryo
over
the
past
few
months.
The
username
space
support
is
currently
an
experimental
phase,
as
we
continue
to
polish
it
up,
but
it
is
something
that
you
can
try
out
and
use
today.
A
There's
also
a
lot
of
discussions
going
on
in
the
upstream
kubernetes
enhancement
plan
for
username
spaces
to
continue
to
drive
this
to
be
a
fully
supported
feature
in
kubernetes,
so
you
can
enable
username
spaces
for
a
certain
runtime
by
adding
the
io.kubernetes.cryo,
the
userness
mode
annotation
to
the
list
of
allowed
annotations.
For
that
one
time
and
the
picture
on
the
left
here
is
an
example
of
such
an
entry
in
the
cry.com.
A
You
can
also
use
runtime
classes
to
create
a
custom
runtime
with
this
annotation
enabled
so
that
if
you
want
to
use
username
spaces
on
certain
parts
such
as
your
billboards,
you
can
spec.
You
can
specify
that
one
time
class
for
that
body
ammo
and
while
your
other
workloads
run
without
username
spaces,
and
the
picture
on
the
right
is
an
example
of
a
party
ammo
that
has
the
usernamespace
annotation.
A
There
are
certain
options
you
can
pass
to
this
annotation,
so
one
of
them
is
the
size
which
sets
the
size.
The
range
of
uids,
the
username
space
you
want
to
have
the
map
to
root
option
means
that
it
would
map
it
to
the
uidsu
and
your
container
if
you
want
to
run
your
container
without
roots.
So
with
any
other
users
such
as
1000,
you
can
use
the
keep
id
option
here
and
it
will
keep
that
uid
and
then
some
other
options
are.
A
A
A
A
A
C
All
right,
I
would
like
to
talk
about
how
we
can
record
second
profiles
using
cryo
and
kubernetes.
The
first
thing
I
have
to
notice
is
that
second
profiles
for
kubernetes
have
to
be
available
as
json
files
on
disk.
So
we
need
something
like
this.
We
have
to
specify
a
default
action.
We,
which
is
in
our
case,
throwing
an
error.
C
C
This
is
a
main
drawback,
because
if
you
now
want
to
create
second
profiles,
we
have
to
require
detailed
knowledge
about
the
underlying
or
zero
in
term
used
by
cryo,
how
it
is
how
the
cluster
is
configured
and
which
version
we
actually
use.
So
we
also
need
necessarily
the
application
systems
which
may
be
executed
in
all
code
paths,
and
if
we
miss
a
code
path,
then
it
may
be
possible
that
the
workload
gets
terminated
during
runtime,
which
is
actually
not
what
we
want.
C
So
what
is
the
solution
around
this?
I
think
a
good
solution
around
this
would
be
to
record.
Second
profiles
and
a
test
environment
and
then
applic
apply
it
to
the
production
workload
and
how
this
works
is
part
of
my
demo
for
today.
So
this
demo
should
show
you
how
we
can
record
those
second
profiles
using
cryo
and
to
be
able
to
run
this
demo.
C
So
it
is
also
necessary
to
use
a
project
like
the
oci
second
bpf
hook,
and
it
has
to
be
installed
and
configured
on
the
system.
So
this
is
a
pre-start
oci
hook,
which
actually
runs
exactly
before
the
container
workload
would
start,
and
it
creates
a
different
child
process
which
compiles
a
bpf
module
and
then
attaches
a
trace
point
to
our
running
workload.
And
then,
if
the
application,
the
hook
is
possible,
is
able
to
record
all
our
syscalls
into
an
intermediate
structure.
C
C
C
C
C
This
call
in
our
case-
and
this
would
be
actually
already
a
usable
second
profile,
but
usually
we
run
multiple
containers
within
a
pot,
and
we
cryo
was
also
able
to
separate
those
containers
and
record
into
different
files.
So
for
that
we
made
a
custom
modification
to
cryo.
That
cryo
has
the
intelligence
to
split
up
the
annotation
based
on
the
actual
container
workload.
C
For
example,
if
we
now
run
a
pot
like
this,
we
can
specify
multiple
annotations
using
the
tray
syscon
annotation
referencing,
the
container
name.
So
we
now
use
the
engine
x
and
redis
containers
which
are
down
here
and
reference
them
by
their
name
to
specify
different
output
files.
Otherwise
they
will
overwrite
each
other
if
we
specify
the
same
output
file,
so
this
wouldn't
make
any
sense
at
all.
C
C
Then
we
can
see
that
we
have
two
instances
of
the
hook
running
and
recording
into
different
files,
so
yeah
for
sure
we
have
to
delete
the
hook
again
to
be
able
to
see
what
is
actually
written
in
the
into
those
files.
Now
the
pod
got
deleted
and
cryo
uses
the
annotation
value
f
again
to
look
to
separate
our
profiles.
C
C
But
in
our
case
I
just
have
to
copy
our
second
profiles
into
wallet
cube.com,
and
if
we
did
this,
then
we
are
able
to
use
both
profiles
in
our
workload.
So
if
we
now
use
create
a
deployment
workload,
for
example,
which
uses
two
replicas,
then
we
can
just
use
the
new
security
context.com
profile
which
got
introduced
in
kubernetes.
1.19
guru
reference
our
localhost
profile,
and
now
we
can
use
the
nginx
profile
for
the
nginx
container
and
the
writers
profile
for
the
radius
container.
C
And
if
we
now
create
this
deployment
and
wait
for
it
to
be
available
in
my
local
cluster
setup,
then
we
can
look
at
our
pots
which
got
created.
And
if
we
just
choose
one
of
those
spots,
then
we
can
see
that
we
also
have
the
container
annotation
for
the
second
profile
now
available
for
both.
So
we
have
the
nginx
annotation
for
the
nginx
profile
and
the
redis
annotation
for
the
radius
profile.
C
And
if
we
want
to
check
how
this
actually
looks
like
and
on
a
low
level,
oci
runtime,
then
we
can
use
crystal
inspect
and
in
our
case
we
just
look
for
our
nginx
profile
engine
x
workload
and
one
of
those
containers.
We
just
choose
one,
because
we
have
two
running,
then
we
can
see
that
we
have
a
second
profile
applied
which
actually
matches
the
same.
C
In
the
same
way
as
we
have
recorded
it,
and
that's
my
demo
for
recordings.com
profiles
of
cryo,
one
more
thing
I'd
like
to
mention
is
that
cryo
targets
to
make
kubernetes
more
secure
by
default.
So
our
overall
idea
was
to
make
the
runtime
default
profile
for
second,
the
default
profile
for
all
workloads.