►
From YouTube: "My CNI Plugin Did… What?!": Debugging CNI with Style & Aplomb - Douglas Smith & Daniel Mellado Area
Description
Don’t miss out! Join us at our upcoming hybrid event: KubeCon + CloudNativeCon North America 2022 from October 24-28 in Detroit (and online!). Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
"My CNI Plugin Did… What?!": Debugging CNI with Style and Aplomb - Douglas Smith & Daniel Mellado Area, Red Hat
Just because CNI is simple – doesn't mean it's easy. We're going to hook you up with the tools of the trade to analyze what's up when your CNI plugin is feeling down. It sure is easy to speak STDIN and STDOUT and write CNI plugins, but debugging CNI plugins in production takes more than brute force and will power – it takes a toolbox. From cnitool, to dummy CNI plugins, to enhancing your logging, we'll show the tools we use every day to make zeroing in on CNI problems a cool breeze. Not only will we crash CNI plugins on-the-fly to show you how we inspect what's happening, we'll also show you how we handle logging so you can analyze reports from your users when you don't have direct access to systems. We'll talk about how we architect CNI plugins to make debugging easier, and talk about thin plugin vs. thick plugins programming patterns, and show you how it impacts debuggability. Not only does it make debugging easier, it'll make developing easier.
A
B
A
We're
also
going
to
then
kind
of
feed
this
information
back
into
what
we
think
needs
to
happen
for
cni
2.0
and
how
we
can,
as
a
community,
make
this
better
as
we
come
together.
So
cni,
it's
the
container,
networking
interface
it.
What
cni
is
is
an
api
and
each
of
the
cni
plug-ins
are
discrete
binaries
that
live
on
disk,
that
you'll
see
here
by
the
cni
binary
directory.
That's
something
you
you
got
to
know
where
it
is,
and
you
also
have
cni
configurations
that
feed
into
those
those
are
executed
by
your
container
runtime
implementation.
A
So
between
your
container
runtime
implementation,
the
cubelet
and
your
cni
plugins
themselves.
Those
plugins
will
then
go
and
manipulate
your
pod
sandbox.
So
that's
where
you're
going
to
have
a
network
namespace
and
that's
where
your
cni
plug-in
is
going
to
do
its
primary
work
of
creating
the
network
interfaces
that
you
see
in
your
pods
when
you've
got
a
cni
plug-in
call,
it's
really
kind
of
a
simple
flow.
A
What
happens
is
your
runtime
implementation
using
libcni
is
going
to
call
a
binary,
that's
on
the
host
and
it's
going
to
call
it
with
some
environment
variables
that
give
you
some
information
about
the
call
that's
being
made,
and
then
it's
going
to
feed
in
a
configuration
through
standard
in
your
plugin
itself,
then
will
speak
via
standard
out
and
it'll,
give
a
cni
result
as
specified
by
the
cni
spec
and
then
additionally,
it'll
have
an
exit
code
which
counts.
So
if
you
don't
exit
zero,
that
pods
gonna
go
off
into
a
crash
loop
and
try
again.
B
Yeah
so
as
dark
was
mentioned
in
here,
let
me
see
any
specification.
We
got
to
implement
basically
four
different
differently
from
different
calls.
So
we'll
see
now
add
del
check
and
version,
even
though
this
is
somehow
mandatory
from
this
specification,
you
don't
really
most
of
the
plugins,
they
don't
implement
checks,
so,
as
the
name
implies,
I
would
just
basically
add
a
container
for
a
network
or
apply
modifications
on
your
plugin
delete
would
remove
a
container
for
the
network.
B
Whatever
buys
the
dell
operation
should
return
zero
or
you
may
get
into
a
terrible,
terrible
crashly,
backup
that
we
will
speak
about
later
and
also
the
delete
operation
may
not
be
successful.
Should
some
operations
in
the
node
be
happening
such
as
a
rebooting?
So
then
you
would
need
to
delete
basically
the
remaining
things
on
your
own.
Then
you
got
check
check.
As
I
said,
it's
not
really
implemented.
B
I'm
going
to
be
showing
an
example
about
the
cna
plugin
that
we
developed
specifically
for
cubeco,
which
is
called
cubeconcinia,
and
that
does
automatically
nothing,
but
I
think
it's
a
good
example
to
see
how
the
you
know
seen
applying
could
be
bootstrapped
and
version.
Will
be
just
you
know,
getting
you
back
to
with
the
virtual
name
from
the
ci
plugin,
so
c9
plugins
get
their
configuration
by
a
json
file,
even
though
there's
a
lot
of
fields
in
the
json
file.
B
So
how
do
I
really
make
a
ci
plugin?
As
I
was
saying,
this
does
nothing
but,
as
you
can
basically
figure
it
out,
we
are
implementing
the
scale
from
the
plugin
and
we
got
the
four
meters
there.
We
got
add
dell,
the
version
and,
as
I
was
commenting,
we
don't
are
not
going
to
really
be
really
implementing
check
at
all
in
this
case.
But
if
you
just
go
and
check
the
cm,
the
add
version,
you
see
that
it's
basically
loading
the
netcode
data
from
there.
B
It's
going
to
be
getting
you
an
error
should
that
happen,
and
we
are
just
basically
returning
a
json.
So
nothing
really
fancy.
I'm
mentioning
this
because
developing
a
scene
plugin
is
not
straightforward,
even
if
this
is
just
so
easy
because
there's
seldom
to
know
documentation
and
how
to
do
this.
So
most
of
the
people
just
read
the
spec,
but
there
is
no
a
real
example.
This
is
something
that
someone
should
improve.
You're
happy
to
you
know,
fill
any
request
request.
A
Just
something
to
mention
quick
about
this
plugin
that
daniel
is
showing
is
we've
got
this
code
available
on
github.
We've
got
a
link
later
in
the
presentation,
but
this
is
what
I
would
call
a
dummy,
cni
plug-in
and
it's
something
that
you
could
take
just
as
a
skeletal
structure
and
put
in
some
logging
and
use
it
to
substitute
in
for
a
currently
running
cni
plug-in.
If
you
wanted
to
get
some
debug
data
as
well.
A
Carry
on
and
one
other
cni
configuration
just
to
go
over
briefly,
is
what
we
call
a
conflict,
otherwise
known
as
chained,
cni
plugins.
This
is
where,
instead
of
having
the
type
field
you're
going
to
have
this
plugins
array-
and
I
wanted
to
show
this
specifically
because
in
the
cni
1.0
spec-
it
will
be
all
configurationless
all
conflicts,
so
you're
always
going
to
see
this
list,
even
if
it's
a
list
of
one
in
the
future.
So
just
to
give
you
a
quick
example
of
that.
A
A
A
However,
they're
configurable
and
since
the
cubelet
is
going
to
be
in
charge
of
kicking
off
this
process,
that's
where
you're
going
to
find
the
configuration
for
these
so
two
places
that
you
want
to
look.
You
want
to
look
at
your
cubelets
command
line
arguments
so
just
do
a
ps
look
for
that
see
how
it's
actually
running.
A
If
it's
not
specified
there,
don't
assume
it's
the
fault.
The
defaults.
What
you're
going
to
want
to
do
is
actually
pull
up
the
cubelet
configuration.
So
if
you're
using,
if
your
distribution
happens
to
use
a
non-default
location,
you
can
really
burn
some
cycles.
Trying
to
look
and
just
to
the
wrong
place.
A
Something
that
you've
got
to
realize
related
to
your
binder
is
that
the
type
field
itself,
where
you've
got
you
know
flannel
calico,
psyllium,
whatever
it
may
happen
to
be,
that's
actually
referring
to
a
binary
in
your
bender
path
itself.
So
that's
what
it's
actually
going
to
execute
so
be
mindful
of
the
fact
that
that
has
to
be
an
actual
file
that
lives
on
disk
and
that
matches
the
value.
That's
there
something
that
I've
seen
happen.
Many
a
time
is
that
you're
having
a
failure,
and
it's
because
you've
specified
this
configuration.
A
Yet
your
c
and
cni
plugin
itself
hasn't
installed
properly
or
isn't
installed
on
disk.
So
that's
one
thing
to
take
a
look
for
and
related
to
your
configuration
directory.
This
is
going
to
the
cni
configuration
that's
alphabetically.
First,
there
is,
the
presence
of
that
file
is
going
to
determine
if
your
node
is
marked
ready.
So
if
you
do
a
cube,
cuddle,
get
nodes,
you'll
see
a
status
field
ready
or
not
ready,
and
that
cni
configuration
is
a
semaphore
for
the
cubelet
to
know
that
this
node
is
ready
to
handle
network
traffic.
A
So,
as
a
cni
developer,
you
want
to
be
mindful
of
how
you
lay
down
your
cni
configuration
on
disk
and
as
an
administrator,
you
want
to
be
mindful
of
how
this
process
works
so
that
you
can
know
if
your
nodes
are
tainted
properly
or
not
in
order
to
accept
these
pods.
So
you
can
of
course,
set
up
pods
that
you
may
want
to
have
run
very
early
in
your
cluster
life
cycle
to
tolerate
the
not
ready
state.
A
So,
for
example,
if
it's
flannel
that's
installed
with
a
daemon
set
so
check
out,
what's
going
on
with
that
daemon
set
and
then
enter
your
node
to
check
out.
What's
up
a
few
more
just
quick
tips
for
debugging
the
number
one
thing
that
will
bite
me
is
json,
so
it's
great
for
parsing,
it's
not
so
great
for
humans.
So
whenever
you're
mucking
around
with
your
cni
configs
in
json,
just
send
them
through
a
linter
all
the
time.
A
A
Last
but
not
least,
when
you're
debugging
cni
you're,
often
doing
it
specific
node
to
node
and
you'll,
be
on
one
particular
node,
and
if
you're
making
manipulations
to
that
node
you'll
find
yourself
having
to
muck
around
with
labels,
node
selectors,
all
of
that
kind
of
stuff.
You
may
want
to
just
use
static
pods
which
allow
you
to
put
a
yaml
file,
specifying
your
pod
on
that
particular
node
and
then
it'll
spin
up
on
that
node.
So
it's
a
handy
way
to
kind
of
get
around
that.
B
So
I
also
wanted
to
speak
a
little
bit
about
cna
tool,
which
is
a
tool
that
comes
within
the
repo
of
the
cni
itself,
because
so
recapping.
So
how
do
we
get
the
cnip
plugins
to
work?
So
you
can
see
in
a
plugin
it's
a
binary
which
may
get
installed
into
all
the
nodes
from
the
cluster,
using
the
demon
set
of
whatever.
B
Then
you
may
need
to
go
see
what
happens
when
you
are
going
to?
You
know,
go
and
develop
the
binary.
How
do
you
copy
that
then?
Also
you
would
need
to
check
the
logins.
Maybe,
as
this
is
executed
by
the
cubelet,
you
may
want
to
go
to
the
cubelet
or
cryo.
So
this
is
maybe
a
little
bit
too
excessive
for
a
developer,
who
might
not
even
have
access
to
a
quantitative
cluster
and
maybe
not
even
kind.
B
So
the
cni
repo
comes
with
a
tool
which
is
a
swiss
army
knife,
which
is
basically
getting
your
binary
and
getting
your
configuration
that
I
spoke
before,
which
is
basically
a
json
file
with
all
the
tooling
from
there.
It
will
create
a
network
name
and
space,
and
then
you
will
be
able
just
to
test
that
out.
B
Rather
than
just
you
know,
a
crash
loop
on
the
pod
or
yes,
a
sandbox
could
because
sandbox
couldn't
get
created
and
so
forth.
So
again,
and
as
we
were
commenting
before,
one
of
the
things
that
these
cna
plugins
don't
usually
have,
especially
the
reference
ones
are,
is
login
a
login
boot
just
makes
your
life
so
much
easier,
and
even
the
user
or
the
administration,
because
they
will
know
how
to
debug
the
thing
and
what's
going
on
so,
let's
get
you
started
with
some
demo.
C
B
B
B
B
So
there
we
go
running
so,
okay,
how
modulus
works.
How
do
I
add
a
secondary
network
interface
or
a
secondary
network?
So
you,
you
need
to
use
a
network
crd,
which
is
called
a
network
type
definition
which
has
this
way.
If
you
see
I'm
also
injecting
the
copy
from
the
cinema
plugin,
we
are
going
to
be
using
the
plugin
that
we
just
bought
before
that
we
developed
for
this
for
this
conference,
the
keepcon
cni
and
again
this
is
a
binary
which
is
going
to
be
accepted
later
on.
B
But
let's
what
happens
if
you
mess
up?
Let's
assume
you
create
a
secondary
network
with
an
unvalidated
configuration.
How
would
you
deback
this
okay
here
we
are
going
to
create
yet
another
network
attachment
definition
which
is
going
to
be
nonvalid,
because
this
is
just
not
there
and
if
we
get
to
see.
B
The
pods
okay,
let's
see
there,
we
go,
those
are
the
annotations
filled
from
the
first
spot.
This
is
running
it's
okay.
If
you
can
see
it
has
you
know
two
different
interface
as
well
from
the
ceiling
scene
eye
and
the
other
one
from
the
give
concealer
and
that's
okay,
but
what
happens
with
the
other
one.
B
So
getting
back
to
the
cni
plugin
path,
and
let's
assume
that
we
are
using
the
default
path,
you
could
get
to
see
that
most
of
the
blanks
are
there.
You
know,
we've
got
cylinders
and
I
we
go.
The
cube
can
see
now.
Why
do
we
don't
have
the
other
one?
We
are
referencing,
so
this,
of
course,
would
always
fail.
B
I
also
wanted
to
speak
a
little
bit
about
okay.
How
would
I
test
this
so
keep
can
see,
and
I,
if
we
are
on
the
cni
plugin
path,
you
could
always
just
go
ahead
and
you
know
call
the
plugin.
Those
are
all
binaries
and
if
you
are
on
each
node,
you
can
just
you
know,
execute
them.
If
you
see
the
this,
hypocrite
is
just
kicking
out
like
today's
date.
B
So
it's
that's
nothing
fancy,
but
I
think
it's
a
great
example
on
how
to
handle
this
on
so
feel,
free
to
check
that
in
github-
and
I
also
spoke
about
cni
tool.
So
how
would
I
check
locally
and
without
having
to
deal
with
any
cluster,
my
genetic
plugin
and
the
configuration?
So,
for
instance,
this
is
a
sample
configuration
that
they
took
out
for
the
router
cni,
which
is
also
available
on
github,
so
basically
it
it's
meant
to
be
expecting
a
few
and
ip
addresses
and
a
few
ports.
B
B
So
I
got
this
script,
which
is
just
basically
first
I
do
create
a
network
timespace.
Then
I
do
call
cna
plugin.
If
you
check
there,
I'm
just
asking
for
it
to
do
a
cna.
Add
command
using
the
error,
sorted
binary,
so
irrestorative
binary
is
again
the
name
and
the
type
for
that
cni
plane
that
we're
testing.
B
Here
this
is
one
of
the
plugins
that
has
loggings,
so
I
again
the
login
is
enabled.
So
this
is
again
an
awesome
theme
because
you
would
be
able
to
see
what's
going
on
so
it's
we
are
calling
out.
We
are
trying
to
set
a
gateway
which,
of
course,
doesn't
work,
but
you
can
see
some
stuff.
This
doesn't
happen
for
any
of
the
reference
plugins
and,
as
that
was
commented
before
it's
and
as
of
now,
and
we
hope
that
this
is
going
to
be,
you
know,
being
improved
by
the
cine2o
con
specification.
B
B
Let's
try
to
do
the
same
with
the
keep
consumer
that
we
just
had
here
again,
as
I
was
commenting
before
you
just
need
to
put
in
the
type
and
the
name
and
the
cni
version,
it's
nothing
much
more.
So
that
would
be
the
scale
of
the
first
inner
plane
you're
going
to
be
playing
with
afterwards.
You
could
just
go
and
add
any
fields
that
you
would
be
happy
to.
B
This
is
just
the
same
script,
adapted
to
the
other
ci
plugin,
and
what's
going
on
there,
you
go,
it
gives
you
a
valid
output,
so
should
it
has
failed,
we
wouldn't
have
any
way
of
knowing
that
because
there's
no
login.
So
that's
why
I
do
really
encourage
you
to
please
please,
please,
when
developing
cnn
plugins,
please
implement
login.
A
Cool
thanks
daniel,
really
appreciate
it
all
right
so
to
briefly
try
to
wrap
this
all
together
and
talk
about
how
we'd
like
to
see
cni
2.0
improve,
based
on
our
experience
with
debugging
cni
2.0.
One
of
the
main
things
I'm
excited
about
in
cni
2.0
is
the
prospect
of
having
the
plug-ins
themselves
running
in
pods.
So
since
we
have
these
plugins
running
as
binaries
on
disk,
it's
kind
of
a
non-sequitur
for
how
you
interact
with
your
cluster
as
an
administrator.
A
It's
also
not
exactly
the
same
as
what
we're
used
to
with
our
yaml
type
of
specifications
that
we're
working
with
additionally,
devices
are
a
difficulty,
so
we're
trying
to
figure
out
if
there's
a
kind
of
a
better,
more
collapsed
way
to
work
with
devices
and,
lastly,
is
to
have
something
that
better
handles
a
network
life
cycle
during
a
container's
run
time.
So
when
we
mention
the
commands,
we've
got
cni,
add
delete,
etc.
Those
happen
in
discrete
moments
when
your
pod
is
created
when
your
pod
is
deleted.
A
But
what
if
something
happens
between
those
two
detent
points,
we'd
like
to
see
something,
that's
a
little
bit
richer
in
that
sense,
and
just
briefly
something
that
is
kind
of
a
idea
in
terms
of
design
paradigm
is
the
idea
of
a
thin
plug-in
and
a
thick
plug-in.
A
thin
cni
plug-in
is
just
the
binary
on
disk
that
runs
as
a
one-shot.
It
starts
it
does
its
work
and
it
exits.
A
You
could
have
a
richer
design
that
we
call
a
thick
plug-in
and
the
idea
is
you
replace
that
thin
plug-in
that
runs
on
disk
with
what
we
call
a
cni
shim
that
shim
takes
that
standard
in
the
environment
variables
and
it
packages
up
and
sends
it
to
a
process,
that's
usually
running
in
a
in
a
pod
as
a
long-running
process.
That's
where
we
call
it
a
thick
plug-in
and
it
takes
that
information
does
the
work
that
it
needs
to
do
and
then
sends
the
results
back
to
the
shim
and
the
shim
exits.
A
All
of
that
communication
and
have
and
have
your
plugins
running
kind
of
natively
as
a
thick
plug-in
and
then
in
this
sense,
instead
of
all
of
that
work
that
daniel
had
to
do
in
order
to
figure
out,
is
my
pod
running
and
not
running.
How
did
I
get
the
logs
from
the
cubelet?
What
you
can
just
do
is
more
like
a
cube
cuddle
get
logs
and
get
your
logs
out
of
the
pot.
A
So
yeah,
thank
you.
If
you
have
interest
in
any
of
this
and
in
cni
2.0
as
well,
there's
a
working
group
that
I
work
with
that
we
would
love
to
have
you
join
with
any
questions,
comments,
ideas,
contributions
etc,
as
well
as
the
cni
maintainers.
I'm
sure
would
be
happy
to
hear
from
you
there's
a
number
of
issues
in
the
github
issues
that
are
tagged
as
a
cni
2.0.
If
you're
interested
in
that
as
well-
and
we
have
the
contents
available,
that's.
B
D
A
A
If
you
have
these
plugins
running
in
pods
they're
going
to
need
host
access
as
well,
and
I
think
that
that's
going
to
be
something
tricky
to
be
able
to
expose
that
host
access
to
the
pods
and
then
it's
also
going
to
have
security
implications.
So
I
think
that
there's
a
number
of
interesting
challenges
there.
Thank
you
yeah.
A
A
B
Also,
one
of
the
things
that
is
being
discussed
lately
is
that
you
know
so
far.
You
need
to
do
an
add,
a
network
test
definition.
So
it's
a
work
in
progress
unless
an
idea
to
have
a
direct
network
primitive
within
you
know
kubernetes
directly,
without
having
to
do
all.
You
know
this
plumbing
under
the
hood.
E
B
F
A
B
There's
also
another
option,
which
is:
let's
say
you
got
your
own
plugin
you've
got
to
have
any
plugins.
That,
for
instance,
would
attach
your
new
secondary
nic
to
obs
dpdk,
then
you're
getting
that
for
free
or
you
can
also
have
a
secondary
plug-in.
That
would,
you
know,
attach
the
magnet
interface
and
then
you
can,
you
know
just
get
the
sr
we
pass
through
and
so
forth.
So
I
mean
it
kind
of
depends
on
what
you
want.