►
From YouTube: NRI: Extending Containerd And CRI-O With Common Plugins - Krisztian Litkey, Intel & Mike Brown, IBM
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe in Amsterdam, The Netherlands from April 17-21, 2023. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
NRI: Extending Containerd And CRI-O With Common Plugins - Krisztian Litkey, Intel & Mike Brown, IBM
Speakers: Mike Brown, Krisztian Litkey
The Node Resource Interface (NRI) was started as a subproject of containerd to provide OCI hook-like extensions as plugins which can perform extra actions outside the scope of CRI. This idea has since evolved into providing a common infrastructure for plugging extensions into CRI runtimes. Our proposed revision, which hopefully gets merged by the time of this presentation, turns NRI into a common runtime agnostic extension interface for plugging custom container adjustment logic into runtimes. It defines NRI using a formal protobuf protocol description and implements it as a set of ttRPC services. This improves the efficiency of plugin-runtime communication and enables straightforward implementation of stateful plugins. Our additions also expand the original scope of NRI, by allowing plugins to hook into virtually any of the pod and container life-cycle events and adjust a carefully chosen subset of container parameters. Integration to both containerd and CRI-O is available as experimental features. In this talk we'll take a closer look at how NRI and the plugins fit into runtimes, then cover some common and novel use cases, and example plugins for OCI hook injection and custom resource assignment.
A
Thanks
for
coming
to
our
pitch
we're
going
to
talk
about
node
resources,
it's
a
node
resource,
well,
you've,
probably
heard
of
kubernetes
and
kublet
and
container
run
times.
Maybe,
and
anybody
needs
to
know
a
little
bit
more
about
container
runtime,
raise
your
hand
if
you're
good.
If
you're.
If
you
understand
what
those
are
that,
then
that's
great
hey
about
I'd,
say
about
six
years
ago,
I'm
just
gonna
do
a
little
intro
a
little
history.
Before
handing
off
to
Christian.
A
About
six
years
ago,
we
were
sitting
around
a
round
table,
doing
the
open
container
specification
for
runtime
specifications
and
a
guy
named
Bernal,
Patel
I,
don't
know
if
he's
here
or
not,
but
Ronald
Patel
came
up
with
an
idea
to
do
hooks
in
inside
of
the
Run
C
runtime,
using
some
text
that
would
go
into
the
oci
specification
and
those
hooks
would
be
executable
programs.
That
will
be
run
at
particular
points
at
which
a
container
is
executed,
for
example,
pre-start
post
start.
You
know
that
kind
of
thing,
but
how
would
people
Define
were
that?
A
A
For
you
know,
other
kinds
of
container
runtimes
that
are
running
in
parallel
on
your
node
paired
in
parallel
with
kubelet
and
worker
node
pods
that
have
been
assigned
and
Intel
is
one
of
the
the
groups
that
always
needed
to
be
able
to
have
their
own
resource
managers
and
so
they're
heavily
involved.
We
we
did
do
Beyond.
This
whole
thing.
We
didn't
know
ceiling
spec,
we
contained
runtime
groups
put
together.
A
This
runtime
schema
for,
for
our
hook
schema
sorry
that
would
sit
on
a
directory
and
then
a
container
runtime
could
bring
in
that
schema
or
look
at
look
at
the
Json
for
anybody
who
wants
to
do
a
resource
book
inside
of
the
runtimes,
and
some
of
the
people
didn't
really
want
to
just
use
the
runtime
schemas.
They
wanted
to
be
able
to
hook
into
container
run
times
themselves,
so
that
the
the
modifications
of
the
container
runtime
can
be
adjusted
as.
B
A
A
The
being
being
able
to
you
know,
modify
those
configs
anyways
and
set
dependencies
I,
don't
probably
into
too
much
detail,
but
Christian
has
put
together
a
great
little
plugin
that
will
run
in
container
run
times
so
that
you
can
actually
have
a
process.
That's
loaded
by
systemd,
for
example,
and
then
call
into
the
Container
runtime
over
a
ttrpc
interface.
B
B
B
B
So
the
benefits
some
of
the
benefits
of
NRI
so
for
run
times,
it's
I
think
obvious.
It
is
another
way
to
share
and
reuse
code.
So,
in
addition
to
just
using
the
same
packages
in
the
runtime
implementations,
there
is
now
a
common
interface
to
plug
in
your
functionality,
which
can
be
implemented
once
and
then
plugged
into
any
of
the
NRI
enabled
runtimes
for
cluster
administrators
NRI
provides
the
necessary
Plumbing
to
customize
or
enforce
how
containers
are
configured.
B
So
essentially,
if
you
plug
in
your
plugin
so
plug
in
your
MRI
plugin
your
extension
into
the
runtime
in
your
cluster,
then
you
are
free
to
implement
your
own
custom
Logic
for
container
configuration,
but
only
within
the
boundaries
imposed
by
NRI.
So
there
are
certain
things
which,
even
with
NRA,
you
are
not
not
able
to
or
not
allowed
to
do.
B
So.
How
does
NRI
work
to
explain
how
Anri
works?
We
need
to?
First,
take
a
look
at
how
the
content
accusation.
Signaling
flow
looks
like
from
a
high
level
in
our
kubernetes
cluster,
so
this
is
going
to
be
a
lost
over
simplification
of
what
is
going
on,
but
it
should
be
sufficient
to
understand
in
this
context
the
necessary
details.
B
So
first,
a
port
spec
is
created,
and
this
typically
happens
as
a
end
result
of
creating
some
higher
level
object.
For
instance,
a
service
demon
said
replica
set
or
something
similar
So.
Eventually,
this
puts
back
ends
up
in
the
kubernetes
node
agent,
the
cubelet
on
a
worker
node,
the
cubelet
uses
the
container
runtime
interface.
B
To
request
the
creation
of
a
pod
and
one
or
more
containers
within
that
port,
so
the
container
runtime
constructs
from
this
CRI
request
and
oci
specification.
This
also
a
specification
describes
the
containers
parameters
for
the
low
level
runtime,
and
this
oci
spec
is
then
passed
on
to
the
low
level.
Runtime,
for
instance,
runs
your
cutter
containers,
which
then
creates
the
container.
B
B
So
there
are
other
container
lifecycle
events
wherever
I
can
hook
in
and
basically
the
details
of
the
signaling
flow
are
slightly
different.
But
the
important
thing
from
the
general
point
of
view
is
that
usually
a
request,
or
always
a
request-
comes
in
via
the
Sierra
interface.
As
long
as
we
are
talking
about
kubernetes
clusters
and
and
kubernetes
containers.
B
B
Because,
instead
of
each
plugin
author
having
to
implement
all
the
low
level
necessary
functionality
which
are
common
to
our
plugins,
they
can,
by
using
the
stuff
they
can
directly
dive
in
and
start
focusing
on
the
custom
logic
that
they
need
to
implement.
So
this
tab
takes
care
of
things
like
connection
establishment
to
the
runtime
communication,
plugin,
registration
and
so
forth.
B
There
is
the
NRI
protocol
itself,
which
this
is
above
or
previously
mentioned
two
components,
so
the
adaptation,
client
and
the
plugins
the
abuse
to
communicate
and
interact
with
each
other.
So
the
NRI
protocol
is
defined
as
a
protobuf
based
API
with
ttrpc
bindings,
the
protocol,
defense
and
execution
model
and
the
data
model
for
NRI.
B
So
if
we
look
into
the
events
in
more
detail,
so
the
NRI
API
defines
an
event
subscription
mechanism
and
plugins
only
need
to
subscribe
to
those
potent
container
lifecycle
events
which
they
are
interested
in
and
they
only
get
notified
about
those
events
which
they
have
subscribed
to
So.
Currently,
ports
cannot
be
modified
by
any
right.
B
B
B
B
And
actually
this
so
for
Content
creation
is
the
one
where
the
largest
amount
of
modifications
can
be
performed
by
a
plugin,
and
this
directly
comes
from
the
underlying
container
model
defined
by
the
oci
specification.
So
once
the
container
has
been
created,
then
most
of
its
parameters
cannot
be
changed,
but
there
is
a
subset
which
can-
and
this
is
reflected
in
the
possible
responses
to
these
various
elements.
So
for
contact
for
container
creation.
B
B
Plugins
can
also
request
updates
to
other
existing
containers
in
the
runtime,
not
only
the
one
which
is
being
created
so
for
container
updates.
The
requested
update
can
modify
the
resource
related
signal
parameters
both
for
the
container
being
updated
and
other
existing
containers,
and
this
is
the
same
for
stopping.
B
B
The
plugin
is
offered
a
chance
to
synchronize
its
state
with
the
runtime,
so
during
the
synchronization
it
receives
a
full
set
and
a
full
dump
of
all
existing
ports
and
all
existing
containers
in
the
runtime,
and
in
response
to
this
it
has
the
chance
to
do
customizations
to
the
configuration
of
any
of
these
containers,
and
these
customizations
are
exactly
the
same,
as
can
be
done
for
a
stop
or
an
update
event.
So
not
the
same
as
for
Creative.
Basically,
resources
can
be
updated,
but
not
much
else.
B
There
are
also
a
number
of
things
that
NRI
plugins
can't
or
shouldn't
be
doing
so.
Multiple
plugins
cannot
make
simultaneous
conflicting
changes
to
the
same
container,
so
this
is
checked
caught
by
the
NRA
infrastructure
and
it's
like
the
semi
error.
So
basically,
if
this
happens
during
Continental
creation
or
container
update,
then
it
is
sort
of
transactionally
rejected
with
an
error.
B
So
we
try
to
be
careful
to
include
everything
that
we
think
is
needed.
Of
course,
there
it
is
possible
that
something
we
Overlook,
but
the
rest
of
the
parameters
which
cannot
be
controlled
by
any
right.
Plugins
are
international
and,
although
I
said
that
energy
plugins
cannot
do
such
control
in
reality,
this
is
only
true
if
any
plugins
are
not
trying
to
bypass
the
runtime
and,
for
instance,
by
direct
sake
of
manipulation,
make
change
changes
on
their
own
to
these
parameters,
which
they
should
not
do.
B
B
B
So
and
the
easiest
way
to
get
up
and
running
is
that
you
clone
this
one
and
then
you
start
modifying
it
and
how
you
need
to
modify
it
well.
First,
you
need
to
subscribe
to
the
events
that
you
need,
because
you
probably
don't
want.
Maybe
you
don't
want
to
receive
all
the
events
and
then
you
need
to
implement
actual
customization
Logic
for
containers.
B
B
So
if
you
do
this,
if
you
are
modifying
modifying
other
containers,
so
existing
containers
for
like
a
create
container
request,
then
you
typically
want
to
subscribe,
also
to
a
stop
container
event,
because
then
you
need
to
undo
somehow
those
modifications
that
you
did
for
the
creation
events.
So
in
a
way
the
other
direction.
B
So,
in
addition
to
this,
so
these
are
all
reactive
changes
or
reactive
customizations.
So
something
NRI
is
telling
you
something
that
something
is
happening
in
the
runtime.
A
content
is
being
created,
updated
or
stopped,
and
then
you
react
with
with
customizations
in
the
response,
but
then
I
also
provides
a
mechanism
for
doing
unsolicited
customizations.
B
So
this
you
can
do
that
is
with
the
update,
update
containers,
a
call
which
is
actually
control
flow
wise
in
the
other
direction.
So
an
error
is
not
calling
you,
but
you
are
calling
into
NRI
tell
your
customizations.
These
customizations
are
exact
same
as
for
stopping
or
update,
so
they
are
only
to
the
clauses.
B
So
those
are
the
short
guidelines:
how
to
write
the
plugin
and
of
course
one
good
idea
is
to
take
a
look
at
our
sample
plugins
that
we
have
in
our
repository.
We
have
a
few,
not
too
many,
but
those
can
be
used
for
inspiration
and
guidance
and
of
course
we
have
some
some
set
of
documentation
a
little
bit
limited,
but
we
are
working
on
that
so
examples
and
use
cases.
So
we
have
a
few
plugins
already
available.
B
To
prevent
this
kind
of
I,
don't
know
whether
I
should
call
them
customers
specific
additions,
but
that's
a
little
bit
how
those
look
like
to
me
so
keep
those
out,
and
rather
Implement
them
as
an
Android
plugins.
So
these
real
world
ones
that
we
have
so
one
is
an
annotation-based
device
injection.
So
if
I
recall
correctly,
I
saw
this
code
in
cryo,
then
I,
but
okay
I
will
implement
this
in
a
right
plugin.
B
Then
we
have
another
one:
the
CDI
device
injection,
which
is
by
now
obsolete,
but
it's
still
available.
So
this
is
basically
so.
This
is
how
we
initially
Implement
CDR
device
injection.
So
if
you
know,
if
you
don't
know
what
CDI
is
so
Celia
is
basically
the
low
level
device
description
format
used
by
the
upcoming
dra
or
dynamic
resource
allocation,
featuring
kubernetes
the
inject
devices
into
containers
and
manage
them
a
little
bit
better
than
you
can
do
with
or
actually
a
lot
better
than
you
can
do
with
with
traditional
device
plugins.
B
So
this
was
easy
to
implement
as
as
a
snmri
plug-in.
So
it
was
one
of
the
first
sort
of
test
cases
for
ourselves
that
do
we
have
in
our
functional
coverage
that
some
useful
things
can
be
done,
but
because
we
did
not
want
to,
we
did
not
want
to
add
the
dependency
between
from
the
array
to
MRI,
so
we
wanted
to
completely
decouple
those.
Then
eventually
we
implement
it
natively
this
in
both
cryo
and
content
added.
That's
why
this
is
now
obsolete.
B
Then
the
third
one
I
think
it's
Mike's
favorite,
it's
the
oci
hook
injection,
and
that
is
basically
so
in
cryo
you
have
oci
hook
injection
support,
but
you
don't
have
it
in
container
day,
and
this
has
been
a
long
standing
discussion.
Several
times
have
been
attempted
in
one
way
or
another,
but
the
pr
did
not
really
ever
go
in
yeah
so
and
then
we
decided
that
maybe
another
plugin
would
be
a
good
idea,
because
then
it's
not
a
configuration
option,
but
you
can
completely
leave
out
the
code.
B
So
if
you
need
to
lock
down
some
in
some
environments
continually
so
that
this
is
absolutely
not
possible,
then
to
access.
This
is
a
little
bit
easier.
If
you
simply
don't
have
the
plugin
present
on
the
system,
then
just
take
a
look
at
the
configuration,
so
this
code
has
almost
heard
about
them
been
taken
from
the
cryo
implementation
turned
into
Anri
plugin.
It's
really
just
the
fuel
lens
of
code,
and
if
you
have
a
Anri,
enabled
containerd
and
download
this
plugin,
then
it
works
exactly
identically.
B
B
They
call
it
high
performance
hooks.
So
this
is
a
really
good
example.
Actually
of
some
functionality,
value
react
to
some
container
life
cycle
event,
but
then
you
perform
actions
which
are
outside
the
realm
of
the
OC
specification.
So
this
piece
of
code
is
not
there
for
fun,
but
it's
there,
because
the
same
thing
cannot
be
done
by
the
ocean
specification
and
the
reason
for
that
is
that
nowadays
it
does
also
other
things,
but
but
initially,
when
I
took
a
look
at
it.
B
So
this
is
providing
even
more
isolation
from
other
containers
and
system
and
the
processes
running
on
the
system,
then
just
by
getting
a
normal
exclusive
CPU,
and
when
that
container
is
stopped,
then
this
migration
is
done
in
the
other
way
around.
So
I
put
the
so
the
higher
grinding
state
is
restored
to
the
origin.
B
B
Then
we
have
a
couple
of
debugging
and
development
plugins,
so
we
have
event
logger.
So
that's
an
easy
way
to
take
a
look
at
what
is
going
on
in
the
NRI
sense
in
a
runtime,
so
it
registers
itself.
So
the
logger
registers
itself
to
the
runtime
hit
subscribed
to
all
events,
and
then
it
does
a
full
dump
of
every
single
event
it
receives
and
it
never
request
any
customizations
of
custom
containers.
So
in
that
sense
it's
a
knob
plugin.
B
B
It's
not
doing
well,
it
was
logging,
but
otherwise
it's
almost
the
same,
and
then
we
have
one
more
experimental
plugin,
and
so
that
is
a
modified
version
of
a
I
would
call
it
an
experimentation
vehicle
that
we
have
in
now
for
a
couple
of
years,
using
in
our
team
to
experiment
with
various
resources,
assignment
policy,
improvements
for
orchestration
and
the
runtime
space.
So
originally
it
was
written
so
that
it.
B
So
if
you
remember
the
cubelet
and
the
runtime
communicates
with
something
called
the
CRI,
so
the
container
random
interface
and
the
original
implementation
of
this
is
a
resource,
so
I
say
right
proxy.
That's
why
it's
called
the
Sierra
resource,
how
it
works
towards
the
cubeletic
presence
to
be
the
runtime
towards
the
runtime.
It
presents
to
be
the
cube
right
and,
as
the
requests
are
passing
by,
it
is
just
Consulting
its
own
policy
that
hey
what
should
happen
and
then
that's
the
customization
by
modifying
the
CRI
requests
and
he
goes.
B
This
is
a
some
kind
of
a
hack.
So
we
decided
that,
since
we
have
possibility
to
do
to
do
much
of
these
same
things,
we
then
arise,
so
we
modified
it
that
we
added
an
array
plug-in
support
to
it.
So
we
give
it
a
special
command
line
option.
Then
it
registers
itself
to
the
runtime
with
another
interface
and
then
does
all
the
same
things
that
it
will
do
over
the
CRI
proxy
method,
but
using
Android
customizations.
C
B
A
B
So
we
want
it's
a
good
question.
We
want
it
to
start
very
simple,
so
we
recognize
that
it
might
be
possible
that
you
want
to
split
up
your
full
processing
chain
into
smaller
components,
because
it
might
be
that
you
want
to
just
in
some
configurations
do
part
of
it,
but
not
all
we
have
done.
We
have
our
own
test
for
that,
but
we
wanted
to
start
very
simple.
A
B
A
B
Yeah,
so
basically,
okay,
sorry
I
forgot
to
say
that
that's
how
it
goes
that
the
plugin
and
the
plugin
registers
itself
it
gives
an
index
and
the
name
yep
and
the
index
is
used
to
order
them
and
that's
it.
A
name
is
used
to
basically
pass
it
some
configuration.
If,
if
you
want
to
manage
your
configuration
right
that
way
and
that's
it.
B
So
for
the
security,
so
we
try
to
live
out
everything
which
is
directly
related
in
Security
in
the
OC
aspect,
so
second
filters
system
course
none
of
it.
You
can
touch
my
NRI,
but
currently
it
is
so
that
the
NRI
has
no
any
kind
of
excess
control
with
various
privilege
levels.
So
once
if
you
are
able
to
connect
to
the
NRI
socket,
if
you
are
able
to
register
yourself,
then
you
can
do
whatever
is
possible.
B
B
Yes,
so
we
yes,
so
this
we
had
this
one,
so
the
Sierra
resource
manager,
that's
basically
work
so
that
it
has
several
built-in
policies.
So
what
we
were
thinking
is
that
we
we
would
like
to
so
as
this
is
merged,
then
then
we
would
like
to
rework
that
so
that
we
rip
out
all
the
CRI
based
code.
We
clean
it
up
and
then
we
would
like
to
split
it
up
to
smaller
pieces
and
then
make
it
available
a
little
bit
like
the
stub.
B
But
if
you
want
to
do
with
Resource
Management,
if
you
want
to
experiment
with
that,
if
you
would
like
to
you
know,
disable
the
corresponding
components,
the
CPU
manager,
memory
manager,
topology
manager
in
the
cubelet,
and
you
would
like
to
try
doing
it
here,
then
the
idea
is
that
we
will
turn
that
into
something
that
you
could
start
from
and
just
you
know
it
has
a
little
bit
higher
level
abstraction
than
this
one.
So
it
has
a
much
closer
to
so
it
has
things
like
allocate
and
release
resources
and
that's
it.
B
A
This
it's
a
great
question
right,
it's
going
to
be
default
off,
however,
the
hope,
well,
yeah
and-
and
the
point
of
this
right
is
some
of
those
hooks.
We
talked
about
some
of
the
ways
that
you
do
Resource
Management
already
I
mean
it
already
exists
there
for
real
right,
so
we're
trying
to
put
together
a
higher
level
process
where
we
can
start
managing
this
with
policies.
For
example,
this
pod
is
immutable.
That
could
be
a
policy
that
we
could
hand
through
right
and
then
say
sorry,
no
hooks,
because
it's
immutable.
E
A
F
Thank
you.
First
off
my
apologies.
I
missed
your
whole
talk.
I
just
came
here
cool,
so
here's
my
question:
here's
my
the
problem
that
I
have
and
I'm
just
curious.
If
this
would
help
I'm
with
the
company,
that's
a
storage
solution
and
we
need
to
install
a
kernel
driver
right
now.
There's
there
was
SRO,
which
was
the
special
resource
operator.
Openshift
specific,
it's
a
genealogical
dead
end
kmmo
is
the
replacement
if
you're
familiar
with
kmmo.
A
C
A
B
C
Thank
you
very
much
for
delivering
a
great
talk.
I
really
appreciate
that
I
have
a
question
related
to
container
and
cryo
configuration
itself.
If
you
look
at
how
to
integrate
that,
there's
a
lot
of
manual
steps
go.
Do
that
systemd
and
blah
blah
blah
kanji.
Are
you
stand
load
map
that
to
simplify
that
step?
Maybe
some
hook,
just
like
you
mentioned,
do
something
because
manually
changes
wherever
you're
making
there
are
greater
chances
that
failure
as
something
goes
with
you.
A
Yeah
it'll
it
for
administrators
there's
going
to
be
more
work
to
do
different
work
to
do
right.
You
can
currently
configure
your
system
in
in
cryo
to
do
this
by
putting
your
your
your
hook,
schemas
in
a
particular
location,
just
the
administrator
has
to
set
it
up,
and
you
know
with
the
defining
jsons
for
what's
going
to
run
at
a
particular
point
in
time,
and
this
just
sort
of
modifies
that
so
you
no
longer
have
to
do
that
administratively.
But
again,
each
resource
manager
may
have
its
own
complexity.
We
haven't.
A
We
haven't
made
this
simple
for
users,
yet
this
is
more
of
a
for
developers
and
the
owners
of
the
resource
managers.
At
this
point
we're
we're
going
to
have
to,
as
he
mentioned
earlier,
we're
going
to
have
to
look
work
on
policies
and
that's
going
to
it's
going
to
mean
we
need
more
declarative
rate
specifications
that
are
going
to
be
setting
your
plots
back,
make
it
simple,
bring
it
down,
pass
it
to
The,
Container
runtime
through
CRI
and
then,
and
we
can
just
execute
it
with
the
plugins
receiving
the
information
for
the
policy
requirements.