►
Description
Want to view more sessions and keep the conversations going? Join us for KubeCon + CloudNativeCon North America in Seattle, December 11 - 13, 2018 (http://bit.ly/KCCNCNA18) or in Shanghai, November 14-15 (http://bit.ly/kccncchina18).
Kubernetes and the CNI: Where We Are and What's Next - Casey Callendrello, CoreOS (Intermediate Skill Level)
The Container Networking Interface, or CNI, is a standard for networking vendors and projects to integrate with Kubernetes. First released in 2016, CNI has become the default way to network a Kubernetes cluster. In this talk, I'll explain why CNI is designed the way it is. I'll talk about how CNI is typically used in a Kubernetes installation, including some common and not-so-common gotchas and pain points. I'll go into detail about the best-practices for writing a CNI plugin with Kubernetes. I'll also discuss the future of the project, some possible improvements, and next steps for the ecosystem as a whole.
About Casey
Casey Callendrello is an open-source developer at CoreOS. He is a maintainer for the Container Networking Interface (CNI) project. He also contributes to the the Kubernetes project and the Rkt container runtime.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
All
righty
I
think
we'll
get
started
thanks
all
for
coming
just
before
lunch.
I
know,
but
hopefully
I'll
hold
your
interest
a
little
bit
better
than
the
thoughts
of
thoughts
of
lunch.
So
my
name
is
Casey
calendar
ello
I
am
at
Red
Hat,
formerly
core
OS
and
I
want
to
talk
about
kubernetes,
the
CNI
and
how
these
two
interact.
This
is
an
intermediate
level
talk,
so
I'm
gonna
assume
that
you
more
or
less
have
an
idea
of
what
what's
going
on
and
try
and
get
sort
of
technical
trying
to
go
into
some
details.
A
A
So
this
is
just
what
we're
going
to
talk
about
today.
Four
basic
topics
plus
a
quick
introduction,
so
starting
out
what
is
CNI
actually
and
it's
okay,
that
it's
a
little
bit
of
a
confusing
term,
because
in
this
particular
community
this
term
has
come
to
mean
a
couple
different
things:
it's
a
bit
overloaded
and
then
another
question
is
sort
of
a
historical
question
of.
Why
is
it
designed
the
way
that
it
is
so
in
this
particular
community?
Cni
means
three
different
and
slightly
related
things.
A
First
of
all,
and
most
formally
it
is
a
vendor-neutral
protocol
that
is
used
by
container
run
time
engines
to
make
requests
to
networking
providers.
That
is,
by
definition
what
CNI
is
right
so
and
it's
not
just
for
communities.
In
fact,
it
was
started.
Part
of
the
rocket
project.
Initially
CNI,
has
also
come
to
mean
a
set
of
commonly
used.
Networking
plugins
that
are
maintained
by
the
CNI
community,
as
opposed
to
the
kubernetes
community
or
by
a
specific
vendor
and
CNI,
is
also
shorthand
for
a
plug-in
for
the
cubelet
called
CNI
oops.
A
In
other
words,
when
you
type
Network
plug-in
equals
CNI
on
the
command
line,
the
cubelet
is
going
to
run
one
code
path
versus
another
one,
and
how
does
it
all
come
together
then,
with
kubernetes?
This
is.
There
are
other
talks
that
have
outlined
this
a
little
bit
better,
so
I'm
going
to
go
very
quickly.
Does
three
thing:
it
does
these
things
you
write
a
configuration
file,
you
put
it
on
disk.
A
You
put
some
binary
on
disk
at
some
point
in
time
later,
you
use
to
create
a
pod
kubernetes
executes
those
binaries
passes
this
configuration
file
into
standard
in
and
then
your
pod
is
online
like
magic,
okay.
So
this
is
the
basic
of
this
is
the
base
of
where
we
are
today.
As
for
why
it
is
designed
the
way
it
is,
if
you
think
about
every
other
extension
point
in
the
kubernetes
ecosystem.
It's
always
always
always
protobuf
G
RPC,
daemon,
CSI
device,
plugins
everything.
So
why
isn't
CNI
like
this?
Why
is
you?
A
Why
does
CNI
unique
well
CNI
came
out
originally
out
of
rocket
and
rocket?
Is
a
daemon
'less
container
runtime
engine,
so
it
wouldn't
really
have
matched
the
architecture
for
CNI
to
have
a
deeper,
to
require
plugins
to
be
a
demon
and
rocket
has
always
focused
on
really
clean,
really
simple
integration
points
and
C
and
I
fell
out
of
that
effort.
Additionally,
as
some
of
you
may
have
discovered,
either,
hopefully
with
the
blog
post
or
not
so,
hopefully
with
bugs
in
production,
managing
network
names
faces
managing
any
sort
of
namespace
and
go
is
not
reliable.
A
This
is
only
fixed
and
go
1.10,
which
kubernetes
still
doesn't
build
with
right,
so
you
have
to
use
sub
priceless.
You
have
to
fork
an
exec
to
safely
manage
networking,
namespaces
and
golang,
so
you
might
as
well
just
make
that
fork
and
exec
protocol
your
execution
protocol,
so
that
is
in
effect,
CNI
all
right.
So
now
we
all
are
on
the
same
page.
We
all
know
everything.
A
Let's
talk
about
how
CNI
is
used
in
kubernetes
and
I
really
want
to
emphasize
that
CNI
is
not
just
for
creepy
Nettie's,
and
so
this
is
so
how
kubernetes
uses
CNI
might
be
different
from
how
other
container
runtime
engines
use
CNI
just
to
get
some
terminology
really
quickly.
You
probably
have
seen
this,
but
on
the
right
we
have
a
node
and
it
has,
and
it
has
an
Ethernet
interface.
It
has
an
IP
address
on
that.
Node
is
some
pod.
That
pod
also
has
an
address
and
IP,
and
this
state
corresponds
in
the
API
server.
A
It's
reflected
here.
This
is
the
node
object
and
the
important
things
you
can
see
that
it
is.
The
node
has
reported
its
internal
IP
and
the
node
has
been
assigned
something
called
a
pod
cider,
and
this
is
the
block
of
IP
addresses
that
the
node
is
going
to
use
for
the
IP
addresses
in
the
pod.
Likewise,
this
is
reflected
in
the
pods
data
structure
in
the
server
it
has
a
host
IP,
which
is
useful,
and
it
has
the
pod
IP
that
has
been
assigned
to
the
east
zero
inside
the
pod.
A
A
This
slide
is
for
the
benefit
of
people
who
have
downloaded
the
slides
and
don't
feel
like
watching
through
the
video
skip
that.
So
how
does
C
and
I
play
into
this
little
dance
here?
So
when
a
pod
is
added
to
a
node,
a
C
and
I
plugin
needs
to
do
three
things.
A
CNI
plugin
for
kubernetes
needs
to
do
three
things.
It
needs
to
create
this
easy
row
inside
the
container
inside
the
pods
network
namespace.
In
other
words,
it
needs
to
connect
the
pod
to
the
network.
A
It
needs
to
allocate
the
pod
IP,
so
CNI
plugins
are
responsible
for
IP
address
allocation.
The
pod
cider
is
a
hint.
However,
Cadiz
does
not
actually
require
that
the
pods
IP
be
inside
the
pod
cider
and
the
third
thing
that
the
CNI
plugin
needs
to
do
is
it
needs
to
make
this
pod
IP,
reachable
by
the
whole
cluster.
A
So
in
other
words,
a
CNI
plugin
is
expected
to
do
two
related,
but
distinct
tasks.
It
needs
to
provide
connectivity
to
an
isolation
domain.
In
other
words,
it
needs
to
put
each
0
inside
this
network
namespace,
or
it
could
also
be
required
to
do
each
0
inside,
say
a
virtual
machine.
That's
connectivity
from
the
domain
to
the
host.
It
also
needs
to
provide
reach
ability.
In
other
words,
this
pot
IP
needs
to
be
reachable
to
the
entire
cluster.
A
Now
a
question
that's
probably
worth
asking
just
so
we
all
understand
why
this
is
the
way
it
is
is
why
do
we
have
plugins
to
do
this
at
all,
and
the
answer
is
that
it's
a
really
diverse
space
and
everybody
has
quite
distinct
requirements
for
how
to
solve
these
problems.
Connectivity,
in
other
words,
bringing
your
pod
online,
so
it
can
go
to
Google
or
whatever
it
might
defend.
It
depends
on
your
networking
interface
hardware
on
the
host.
A
It
depends
on
the
isolation
technology
that
you're
using
whether
you're,
using
namespaces
or
virtual
machines,
and
it
depends
on
performance
requirements,
and
that
of
course
means
it
depends
on
your
budget.
Reach
ability
depends
on
your
networking
hardware
just
the
same,
but
not
your
networking
hardware
on
the
host.
It
depends
on
your
networking
hardware
inside
your
broader
network
right.
So
can
you
touch
your
networking
hardware?
Can
you
touch
the
people
who
can
touch
your
networking
hardware
are
the
as
the
networking
hardware
and
are
the
people
that
can
touch
a
networking
hard
there.
A
Are
they
yours
or
are
they
in
the
cloud
right
like
there's?
No
such
cloud?
That's
just
other
people's
routers,
but
nevertheless,
how
do
you
talk
to
your
networking
hardware?
Do
you
talk
something
sort
of
traditional
like
BGP
or
OSPF?
Do
you
have
a
custom,
rest
interface
and,
of
course,
what's
your
performance
requirements
and
what's
your
budget
right?
All
of
these
things
have
wildly
different
answers
for
wildly
different
circumstances.
So
it's
natural
to
think
that
we
should
have
a
plug-in
system
for
that.
A
Reach
ability
really
is
a
really
rich
topic
and
it
is
important
to
understand
that
kubernetes
doesn't
match
sort
of
the
traditional
networking
environment,
traditional
networking
environments,
the
general
practices,
nodes
or
participants
in
the
network
have
long
relatively
predictable.
Lifetimes
importantly,
address
allocation
is
semi
manual
and
is
most
importantly
as
topology
aware,
so
you
have
a
subnet
for
your
rack.
You
have
a
subnet
for
your
row.
You
have
a
subnet
for
your
data
center.
Everything
is
nicely
aggregated
and
this
is
the
big
one
for
ipv4
environments.
A
How
many
of
you
have
run
out
of
net
ten
right,
one
interface,
even
if
you're,
using
entirely
private
ip's
one
interface
in
your
network
generally
equals
one
IP
address
till
we
all
move
to
v6,
whereas
kubernetes.
Basically,
flips
this
entirely
on
its
head
right.
It's
generally
IP
allocation
is
topology
unaware.
There
are
cases
where
this
is
not
the
case.
Generally,
it's
topology
unaware
kubernetes
will
happily
allocate
hundreds
or
more
of
IP
addresses
per
node,
which
means
that
each
physical
port
on
your
router
infrastructure
needs
to
be
able
to
handle
much
more.
A
Addressing
and
lifetimes
are
short,
churn
is
high,
addressing
is
constantly
changing
nodes
or
participants
in
the
network
are
coming
and
going
so
there
really
is
a
disconnect
between
how
traditional
networking
environments
work
and
how
kubernetes
intends
to
use
those
networking
environments.
All
this
is
to
say
that
reach
ability
is
a
very
rich
and
complicated
topic
when
you're
trying
to
bring
kubernetes
into
into
your
world.
So
how
do
CNI
plugins
work
with
kubernetes?
Let's
talk
a
little
bit
more
in
detail
like
I
said:
CNI
plugins
need
to
do
two
things.
They
often
have
two
separate
components.
A
They
typically
have
a
CNI
binary,
so
this
is
the
CNI
spec
binary
executed
by
the
cubelet
that
configures
the
pods
interface.
This
is
for
technical
reasons.
This
is
for
lifecycle
reasons.
This
is
the
code
that
is
actually
going
to
be
doing
the
namespace
switch,
so
this
needs
to
be
a
separate
process
that
is
providing
your
connectivity
separately.
A
Essentially,
all
CNI
systems
for
kubernetes
also
include
a
daemon
that
is
going
to
manage
routing,
in
other
words,
you're
going
to
have
a
daemon
that
is
going
to
be
providing
the
reach
ability
solution
for
your
network.
Connectivity
is
almost
exclusively
done,
the
same
way
if
you're
using
network
namespaces
you're
going
to
create
a
Vieth
pair,
which
is
a
point-to-point
virtual
tunnel.
Most
of
you
probably
seen
this.
If
not,
there
is
an
excellent
tutorial
by
lwn,
where
you
can
do
this
in
a
couple
command
lines.
A
This
is
all
entirely
supported
with
the
IP
binary,
so
IP
like
the
IP
binary,
the
IP
binary
is
effectively
a
container
runtime
because
you
can
do
IP
net
NS
new.
So
now
you
have
a
command
line.
Container
runtime
engine
I
highly
recommend
all
of
you
just
search
for
namespaces
and
operation.
You
all
should
subscribe
to
LW
N
or
Linux
weekly
news
anyways.
They
are
a
really
important
source
of
news
and
and
a
commentary
for
all
of
us
in
the
open
source
and
Linux
community.
So
connectivity
almost
always
solved
in
this
particular
manner.
A
Reach
ability
has
one
reach.
Ability
is
much
more
diverse.
Reach
ability
has
one
goal,
which
is
to
make
every
pod
cider,
which
is
assigned
dynamically,
reachable
from
every
node
in
your
kubernetes
cluster.
Another
way
of
thinking
about
this
is
that
reach
ability
is
you
need
to
announce
some
semi
dynamic
routes
to
some
peers?
So
does
this
sound
familiar
right?
This
is
every
routing
protocol
ever
existed.
A
This
is
the
solution
they
need
to
solve
and
essentially
all
CNI
plugins
contain
a
daemon
that
the
host
that
programs,
the
network
with
the
pod
cider,
however
programming
the
network,
in
other
words
programming,
this
pod
cider,
can
take
a
lot
of
different
forms.
You
can
program,
not
your
network,
but
you
can
program
the
routing
tables
on
every
node.
That
is
an
overlay
network,
you,
the
excellent
IP
and
IP,
or
even
no,
no
no
encapsulation.
However,
if
your
routers
don't
know
about
your
pod
cider,
then
you
have
an
overlay
Network.
A
You
can
program
your
networks,
routing
tables
via
rest,
in
which
case
you
are
paying
Amazon.
That
is
any
sort
of
cloud
provider
based
API
route
table
manipulation.
You
need
to
be
careful,
you
need
to
watch
your
resource
limits,
but
this
is
a
very
effective
way.
It's
a
great
way
to
avoid
encapsulation,
while
still
having
full
reach
ability.
A
You
can
also
program
your
network
using
traditional
networking
protocols,
in
other
words
OSPF
or
bgp,
and
some
of
the
plugins
that
provide
this
are
calico
and
Romana,
and
this
has
an
awesome
feature
where
your
pod
ciders
are
now
real
addresses
inside
your
network
and
they're
reachable
not
just
internal
to
your
cluster
but
external
to
your
cluster.
But
this
comes
at
a
cost.
You
need
to
have
routers
that
you
can
talk
to
and
you
need
to
make
sure
that
your
routers
have
enough
memory
and
hardware
to
be
able
to
accept
the
scale
and
churn
of
kubernetes.
A
So
it's
a
trade-off.
So
a
important
thing
to
understand
is:
why
are
these
two
separate
components?
Why
do
we
have
this
daemon
and
why
do
we
have
a
binary
while
we
can't
make
them
the
same
thing,
and
it
really
comes
down
to
the
fact
that
the
cni
spec
only
covers
the
connectivity
case.
It
doesn't
cover
reach
ability,
but
kubernetes
has
broader
expectations
right.
So
CNI
has
essentially
two
verbs
that
has
ADD
container
and
the
lead
container,
and
it
only
understands
pod
lifecycle
but
route
lifecycle,
which
is
to
say
the
lifecycle
of
the
pod.
A
Sider
is
different
from
that
it's
per
node,
so
you
can't
get
away
with
having
you
have
to
have
a
long-running
process
right
if
you
have
an
overlay
Network.
You
already
know
this
because
you
need
to
program
every
node
anytime,
any
node
changes,
so
you
have
to
have
a
demon
that
is
programming
every
node
and
watching
every
node
in
the
API
server.
If
you're
talking
a
routing
protocol,
if
you're
talking
be
droopy
or
OSPF,
those
protocols
also
require
persistent
sessions
right.
A
If
your
bgp
session
goes
down,
your
routes
are
withdrawn,
your
notice
gone
and,
more
importantly,
deletion
is
always
hard.
You
never
really
know
when
you
can
delete
the
pod
cider,
because
you
need
to
delete
the
pod
cider
when
the
node
goes
down,
but
C&I
doesn't
tell
you
when
the
node
goes
down,
because
no
it's
got
down
all
right.
A
So
that's
what
CNI
and
the
CNI
infrastructure
is
expected
to
provide
into
kubernetes
now
I'm
going
to
talk
about
the
some
best
practices
for
operators,
the
best
practices
for
users
of
kubernetes,
because
it
is
still
a
little
rough
around
the
edges.
There
are
still
a
lot
of
gotchas.
The
first
best
practice
is
to
understand
how
the
cubelet
is
configured
for
CNI.
So
this
is
a
bit
of
a
pop
question.
This
is
a
bit
of
a
quiz
which
of
these
command
line.
Flags
tells
you
which
CNI
configuration
file
to
use
for
the
cubelet.
Anybody
know.
A
None
of
them
it
doesn't
take
it.
The
couplet
does
something
that
seems
insane,
but
is
actually
for
a
very
good
reason,
and
that
is
the
cubelet
scans,
a
directory
every
5
seconds
and
picks.
Whichever
configuration
file
has
the
lowest
ordered
name.
It
keeps
scanning
and
re
parsing
forever,
so
the
first
best
practice
is
to
only
ever
write
one
CNI
configuration
file
to
this
directory,
a
corollary,
is
to
don't
test
in
production.
If
you
want
to
test
this
deference
different
anyways,
why
does
it
do
this?
A
It
does
this
so
that
you
can
start
the
cubelet
and
then
use
a
pod
to
deploy
a
CNI
configuration
file.
So
it
seems
a
little
awkward
but
in
fact
there's
a
very
good
reason.
Right
now
you
have
this.
Nice
decoupling
comes
in
a
bit
of
a
cost,
though,
which
is
if
you
accidentally
write
to
CNI
configuration
files
to
disk.
You
may
get
yourself
into
a
bit
of
a
race
condition
where
you
parse
Beacom,
add
a
pod
onto
the
B
Network.
A
A
The
second
thing
is
to
understand
some
of
the
new
changes
in
CNI
CNI
now
supports
something
called
plug-in
chaining,
which
is
a
really
powerful
tool
for
you.
If
you
want
to
tweak
your
network,
plugin
chaining
operate,
execute
multiple
plugins
for
a
single
container,
execute
them
in
a
row,
and
it
passes
the
result
of
the
previous
one
into
the
next
one,
and
what
this
means
is
that,
if
you
as
an
operator
want
to
do
something
very
simple,
let's
say
that
you
want
to
tag
all
of
your
pods
with
a
certain
QoS
class.
A
It
means
that
you
can
write
a
small
amount
of
go
code.
Call
it
100
lines
of
code
maximum
to
accomplish
this
without
having
to
know
how
to
write
a
full
cni
plug-in
without
having
to
know
how
to
write
a
full
interface.
Plugin
plugin
chaining
means
that
you
can
adjust.
Your
network
means
you
can
tweak
your
network
with
a
few
JSON
changes
as
opposed
to
building
a
whole
new
binary.
There
are
some
existing
production
plugins
now
that
rely
on
plug-in
chaining,
the
big
one,
the
first
one
that
really
motivated
this
is
host
port
forwarding.
A
Any
of
you
want
to
have
something
listening
on
port
80
on
the
host.
You
need
to
do
port
forwarding.
You
can
write
in
a
second
plug-in
in
the
chain
that
just
as
host
port
forwarding,
otherwise
it's
entirely
interface.
Agnostic
also
is
you
can
tweak
CTL
parameters
right?
This
is
sort
of
a
typical
CNI
configuration
file.
It
might
look
something
like
this.
It's
a
bit
extreme,
but
only
the
first
one
creates
the
interface
all
of
the
other
ones
might
adjust.
Firewalls
might
adjust
CTLs
and,
like
I
said,
these
are
really
easy
to
write.
A
We
have
a
skel
plug-in
that
you
can
copy
and
paste
and
hundred
lines
of
code.
You
can
do
almost
anything
you
want
and
adjust
your
network.
Another
best
practice
with
CNI
is
to
self
host.
You
know
we
need
to
run
a
daemon
on
every
node
and
man
like
does
anyone
know
how
to
run
a
daemon
on
every
node
right,
so
you
should
deploy
your
plugins
using
kubernetes.
This
takes
a
little
bit
of
subtlety
to
get
it
to
use
the
binaries.
A
Your
manifest
needs
to
do
this
sort
of
funny
dance
where
the
first
thing
they
do
is
install
and
activate
the
routing
daemon.
Then
they
copy
the
plug-in
binaries
onto
the
host
into
a
configured
directory,
and
then
they
copy
the
configuration
file,
and
it's
really
important
that
the
last
thing
you
do
is
to
copy
the
configuration
file
because
copying
the
cni
configuration
file
to
disk
is
the
secret
signal
that
says
to
the
cubelet.
The
network
is
up,
and
you
may
now
start
pods
that
are
not
host
networking
pods.
A
This
is
a
little
bit
of
an
awkward
API,
but
this
is
the
API
that
we
use
to
say.
Yes,
the
host
is
up,
give
it
to
me
some
important
things
to
get
right
when
you're
writing
these
manifests,
especially
for
the
daemon.
This
might
seem
obvious,
but
should
be
said.
You
need
to
run
the
daemon
in
the
host
Network.
A
You
also
need
to
make
sure
that
your
daemon
ignores
all
taints,
because
master
should
be
able
to
reach.
Pods
master
in
fact
has
to
be
able
to
reach
pods.
So
you
definitely
need
to
run
your
routing
daemon
on
master,
and
this
is
a
little
bit
of
a
funny
tweak
you
need
to.
You
should
use
the
magic
grace
period
here,
which
will
let
you
restart
your
daemon
very
quickly,
minimizing
the
amount
of
time
that
your
daemon
is
down
in
the
event
of
an
upgrade
or
a
crash.
A
A
Another
best
practice
that
requires
a
lot
of
careful
thought
is
disaster
recovery,
especially
if
you
have
a
self
hosted
control
plan.
You
need
to
understand
the
dependencies
that
your
CNI
plugin
has.
You
need
to
always
run
your
API
server
as
well
in
host
networking,
because
if
your
API
server
is
down,
nothing
may
start
and
you
need
to
understand
the
exact
dependency
chain
and
make
sure
that
your
daemon
set
your
routing
daemon
will
start
up
without
any
other
running
services.
A
You
know
in
system
D,
if
you
have
a
dependency
system,
people
very
happily
tell
you
that
a
transaction
is
not
achievable
right.
It
will
tell
you
nope
can't
do
it,
whereas
the
kubernetes
general
practice
is
to
just
if,
if
the
conditions
are
not
met
for
whatever
your
conditions
are
retry,
retry
retry,
so
you'll
never
actually
get
any
sort
of
fairly
failure.
A
All
you'll
know
is
that
all
of
a
sudden,
your
control
plane
is
down,
and
you
have
no
idea
why
and
you
have
no
idea
how
to
start
it
so
be
really
careful
think
about
this
very
carefully.
All
right.
Yes,
exactly
the
time.
So
now
that's
best
practices
for
run
executors,
but
this
is
cube
con
where
your
cube
root
of
these
developers
and
so
I'm
really
excited
to
talk
about
best
practices
for
plugin
developers.
A
The
first
thing
you
need
to
do,
and
we
should
all
really
do
this-
is
to
understand
the
lifecycle
of
pods
and
cubelet
and
how
things
are
executed.
The
cubelet
is
massively
parallel,
has
hundreds
of
go
routines
and
it
has
very
few
guarantees.
It
gives
you
exactly
one
guarantee
in
the
world
of
CNI
C&I.
Has
the
notion
of
a
sorry
about
that.
Cni
has
the
notion
of
a
container
ID,
some
sort
of
UUID
a
free
text
field
that
identifies
a
container.
It's
a
meaningless
thing.
A
You
cannot
assume
that
you
will
see
a
delete,
a
short
amount
of
time
after
a
sandbox
or
a
pod
is
deleted.
You
can't
assume
that
you'll
see
a
delete
before
the
node
has
been
rebooted.
You
may
see
the
delete
after
the
reboot.
You
might
not
see
a
delete
before
you've
got
a
crash.
Looking
pod
that
runs
a
thousand
times
and
you
run
out
of
IP
addresses
on
the
node
no
guarantees,
and
you
might
not
see
a
delete
ever
because
your
node
might
get
chaos.
A
If
you
use
any
other
identifier
as
your
primary
key,
it
might
be
reused
by
the
cubelet.
You
cannot
doubt
you
even
pod.
They
may
be
reused.
This
has
caused
production
bugs
in
in
commercial
CNI.
Plugins
the
where
the
race
condition
is
something
like
this.
If
you
have
a
pod
with
the
same
name
but
two
different
pot,
IDs,
add
a
one,
a
Crash's
at
some
point
in
time.
A
The
cubit
is
going
to
restart
it
and
it
might
restart
it
and
do
an
ad
of
a
two
before
it
does
the
delete
you
delete
a
one,
but
if
you're,
using
a
as
your
primary
key,
a
two
is
now
offline
right.
All
this
comes
from.
You
need
to
understand
how
the
cubelet
works.
The
second
thing
you
need
to
understand
is
you
need
to
understand
the
host
network.
The
host
networking
environment
kubernetes
generally
expects
that
all
packets
leaving
a
pod
traverse
the
host,
and
this
has
a
lot
of
subtle
implications.
A
So
most
plugins
for
reasons
of
efficiency
do
all
of
their
work
in
the
host
networking
namespace
for
a
lot
of
reasons,
right,
duplicating
functionality,
duplicating
IP
tables
rules,
duplicating
routing
tables
rules
and
every
single
pods
networking
namespace
would
be
a
waste.
These
are.
This
is
all
non
payable,
memory
right
and
more
IP
tables
rules.
Mulch
more
traversal
is
a
waste
of
time
and
energy.
A
There's
also
a
requirement
that
host
networking
pods
be
more
or
less
equivalent
right,
so
anything
that
a
non
host
networking
pocket
access
a
host
networking
pod
should
also
be
able
to
access
right.
They
should
be
uniform
and
also
in
some
cases.
It
may
be
that
if
you're
programming,
the
pods
networking
namespace
it's
a
little
bit
of
a
leak
right,
you
don't
necessarily
want
to
set
up
your
service
rules
for
every
single
service
inside
every
single
pod,
because
now
those
pods
can
enumerate
all
your
services.
So
you
have
to
program
the
host
network.
A
You
might
as
well
only
program
the
host
network,
not
only
in
CNI
but
also
non
CNI.
Functionality
is
also
generally
generally
only
implemented
on
the
host
network,
so
cube
proxy
love.
It
or
hate
it
cube
proxy
does,
but
it
does
Dena
it
rewrites
destinations.
These
rules
are
almost
always
only
installed
in
the
hosts,
IP
tables
space
right
uses,
IP
tables
or
now
uses
IP
vs,
which
is
awesome
and
really
high
performance.
A
It
only
does
this
in
the
hosts
networking
namespace,
most
Network
policy
providers
also
only
provision
only
program,
their
firewall
rules
in
the
hosts,
IP
tables,
namespace
traffic,
shaping
and
overlay
networks.
Of
course,
if
you're
programming
a
routing
table,
you
only
want
to
do
it
once
so.
You
did
in
the
host
networking
namespace.
So
once
you
understand
what
the
hosts
networking
names
what-what-what
has
generally
done
inside
the
host
network,
you
can
decide
whether
or
not
you
want
to
give
it
up.
You
can
choose
to
re-implement
this.
For
example,
psyllium
reimplemented
I.
A
Don't
care
about
this
now
that
you
know
I,
don't
care
about
this
I
want
to
do
something.
I
want
to
connect
my
container
directly
to
the
network.
This
works
really.
Well,
it's
really
fast.
It's
very
high-performance.
You
can
do
Mac
VLAN,
you
can
do
SR,
iov
and
and
pull
off
and
give
it
a
chunk
of
a
PCI
device,
and
so
lesson
from
all
of
this
is
that,
as
with
all
systems,
you
can
take
these
nice
clean
abstractions
and
you
can
give
them
up
a
little
bit
in
exchange
for
speed.
A
A
So
how
do
you
solve
this?
The
solution?
If
you
want
to
talk
technical,
the
solution
is
to
masquerade
the
source
IP
address,
but
the
real
solution
is
to
make
sure
that
you
have
a
test
case
for
this.
If
your
C
and
I
plug
an
author
make
sure
you
test
for
this,
if
you
are
evaluating
cni
plugins,
make
sure
that
your
acceptance
criteria
includes
this
as
a
test.
Otherwise
you
will
get
bitten
some
CNI
plugins.
This
does
work
only
if
you
configure
it
in
the
correct
manner.
A
So
you
absolutely
need
to
test
for
this
when
you're
evaluating
evaluating
your
CNI
plugins
and
the
fourth
best
practice.
If
you
are
a
CNI
plugin
author,
is
to
use
this
cool
new
feature
that
came
with
C
and
is
only
a
year
old,
called
capability
arguments,
so
the
cubelet
now
will
fill
in
dynamic
configuration
into
the
cni
configuration
file.
A
Previously
it
basically
just
copied
the
file
from
disk
and
spit
it
right
out
directly
to
the
plugins
standard
in
this
is
really
awesome,
because
it
means
that
you
can
reduce
your
dependency
in
the
API
server,
which
means
that
you
have
less
of
a
bootstrapping
problem
and
you
have
higher
performance
capability.
Arguments
are
defined
in
the
cni
spec
and
need
to
be
implemented
in
the
cubelet.
The
current
capability
right
now
is
host
port
mappings.
A
A
If
we
get
our
stuff
together,
is
going
to
be
passing
the
pod
cider
inside
the
network,
configuration
and
bandwidth
limits
inside
the
networking
configuration
and
the
real
goal
of
the
cni
project
is
to
be
able
to
get
up
a
pod
using
kubernetes,
get
it
bring
up
at
kubernetes
pod
without
having
to
talk
to
the
api
api
server
at
all
right.
This
is
a
any
sort
of
dynamic
configuration
that
communities
passes.
We
should
be
able
to
represent
in
our
configuration
file.
A
So
if
you
find
something
where
this
isn't
the
case
file
a
bug
and
we
will,
we
will
try
and
fix
it
capability
arguments.
Look
a
little
bit
like
this.
This
is
the
structure
on
disk.
You
need
to
flag
as
an
administrator
writing
your
configuration
file.
You
need
to
flag
which
capabilities
your
plugins
are
capable
of
your
vendor
should
tell
you
this
if
you
are
using
any
sort
of
vendored
manifest,
and
the
reason
we
do
this
is
that
we
don't
necessarily
want
to
care
in
the
kubernetes
side
about
how
port
mappings
are
implemented.
A
That
depends
on
if
you're,
using
BPF
that
depends
on.
If
you
get
two
pass
of,
the
hosts,
IP
tables
or
kubernetes
can
make
no
assumptions
as
to
how
this
can
actually
be
accomplished.
So
instead,
kubernetes
only
can
see
if
a
particular
configuration
declares
that
it
supports
port
mappings.
So
if
you
say
that
my
configuration
has
a
port
mapping
capability,
what
kubernetes
will
do
is
the
standard
in
that
is
passed
to
the
actual
plugin.
A
Another
best
practice
for
the
developers
is
to
be
aware
of
plug.
I
mentioned
it
on
the
administrator
side,
plug-in
chaining
is
awesome.
Plugin
chaining
is
a
great
way
for
you,
as
an
administrator
to
write
a
very
simple
plugin
that
does
just
what
you
want
plug-in.
Chaining
is
also
super
useful
if
you
are
a
networking
vendor,
because
sorry
about
that,
because
it
means
that
you
can
now
expose
your
functionality
as
to
something
you
provide
say,
firewall
rules
without
necessarily
having
to
dictate
the
entire
networking
stack.
A
It
allows
you
to
be
flexible
and
allows
your
plugin
to
be
more
available
to
more
customers.
So
this
is
super
common
for
post
setup
tweaking.
This
is
how
it
looks.
This
is
the
cni
plugin.
This
is
the
CI
configuration
file
on
disk,
and
then
this
in
green
is
what
your
binary
is
actually
going
to
see
from
the
cubelet.
You
can
see
there
I'm
imagining
the
second
plug-in.
That's
going
to
be
executed.
It
executes
some
in
row
from
top
to
bottom.
A
This
plugin
is
called
IP
tables
allow,
and
what
that
plugin
is
going
to
see
is
it's
going
to
see
its
configuration,
and
it's
also
going
to
see
the
previous
result
from
the
previous
plug-in,
and
this
is
really
cool,
because
now
you
can
do
things
that
rely
exactly
on
the
previous
result.
You
can
do
things
like
in
this
case.
This
plugin
is
going
to
do
add
an
IP
tables
rule
to
bring
to
allow
either
this
interface
or
to
allow
this
IP
address.
A
This
is
something
you
can
only
do
with
dynamic,
plug-in
chaining
the
details
of
this
specified
but
important
to
understand
that
the
plugins
are
executed
first
to
last,
and
the
previous
output
is
passed
to
the
next
one,
and
only
the
last
plugin
dictates
what
cubelet
understands
to
be
the
IP
address.
So
the
last
plug-in
writes
the
result
back
to
the
cubelet
now
for
delete.
Plugins
are
run
in
reverse
order.
A
This
is
a
problem
with
the
spec
as
written,
which
is
that
they
plugin
that
runs
last
gets
to
know
the
IP
address
on
add,
but
it
doesn't
know
the
IP
address
on
delete,
so
we're
actually
fixing
this
in
spec
as
you're
0.4
with
essentially
cached
results.
So
this
is
going
to
be
a
lot
more
usable
if
you're
writing
a
plug-in
in
I.
Think
0.4
is
going
to
be
released
in
the
next
month
or
two.
A
So
what
this
means?
What
I
really
want
to
emphasize
for
networking
vendors
is
that
if
your
plugin
only
touches
the
host
side
or
if
it
operates
in
a
mode
where
it
only
touches
the
hoe
side,
then
it
can
easily
be
a
chained
plug-in
right
and,
for
example,
imagine
a
firewall
plugin,
let's
say
calico
right:
calico
is
a
networking
network
policy
provider,
but
calico.
Could
it
doesn't
right
now,
hey
Brian,
it
doesn't
right
now,
but
it
should
be
able
to,
for
example,
say
work
with
virtual
machines.
A
You
should
allow
you
should
use
chain
plugins
to
allow
administrators
to
swap
out
the
connectivity
plug-in
wire
as
you
can
provide.
Perhaps
your
policy
or
reach
ability
plugins,
that's
separate
from
connectivity,
cool,
there's
a
lot
of
information,
but
I
kind
of
wanted
to
get
technical
very
briefly,
I'm
going
to
talk
about
what's
next
CNI
0.4
is
going
to
be
released
in
the
next
month
or
so,
and
the
big
thing
that
it
has
is
a
get
verb
result.
A
Caching
and
more
capability
arguments
get
has
wound
up
with
a
lot
of
corner
cases,
so
it
took
us
a
really
long
time
to
figure
out
how
to
do
this.
Exactly
correct.
The
spec
language
still
needs
a
little
cleanup.
The
big
one
is
result,
caching,
which
shouldn't
matter
to
most
administrators,
but
it's
going
to
allow
chain
plugins
to
provide
a
lot
more
functionality
and
for
them
to
not
necessarily
have
to
write
state
to
disk.
If
anybody's
writing
say
it's
a
disk,
it
should
be
the
runtime
or
a
stable
layer.
A
Also
coming
soon
is
CNI
1.0.
Hopefully
it
will
have
no
new
features.
Hopefully
it
will
just
have
a
better
specification.
What's
next
ipv6
CNI
has
supported
ipv6
for
years.
Kubernetes
has
not
kubernetes
has
alpha
p,
v6
points,
alfe,
ipv6
and
1.9.
Dual
stack,
support
is
sort
of
being
designed,
but
not
a
lot
of
progress
has
been
made
on
that
yet
I'll
still
a
lot
of
work
to
do
still.
A
Looking
for
contributors
still
looking
for
people
to
review
the
design,
a
big
one
that
everybody
wants,
that
everybody
in
this
particular
community
is
working
very
hard
on
is
multi.
Networking
CNI
always
has
been,
and
always
will
be,
a
single
network
per
configuration
file
more
or
less.
There
should
be
one
interface
in
your
pod
for
a
single
network
configuration
file.
However,
there
are
is
a
use
case.
There's
lots
of
use
cases
for
multiple
networks.
A
So
there's
a
lot
of
things
to
figure
out
in
this
design
a
lot
of
subtle
questions
that
need
to
be
answered.
For
example,
well,
pods
always
have
this
default
network.
Well,
pods
always
have
this
a.com
and
the
answer
that
we've
decided
on
is
yes
pods,
always
pods
should
assume
that
any
additional
CNI
configurations
that
are
executed
will
be
additional.
It
real,
always
be
an
easy
row.
This
will
be
directly
connected
to
the
node.
A
A
big
question
that
we've
decided
to
punt
on
for
now
is
how
to
expose
additional
interfaces
or
how
to
expose
additional
IP
addresses.
If
you
remember
way
back
in
the
beginning,
PAH
type
II
was
a
single
object
that
is
not
changing
for
now,
so
you
will
need
to
use
these
endpoints
manually.
That
problem
is
not
going
to
be
solved
for
you
right
now.
This
might
change
we're
looking
for
good
solutions
for
this.
What
about
security?
A
No
idea,
not
on
the
plan
for
right
now
there
is
no
plan
as
of
yet
to
implement
to
include
this
in
for
something
along
the
lines
of
pod
security
policy.
So
if
you
don't
trust
your
administrators,
if
you
have
access
to
a
network
that
you
think
is
needing
to
be
need
need
to
be
secure.
That
is
not
yet
supported
in
the
current
draft
for
multi
networking
and
then
what
about
hairpin
yeah
good
question?
We
figured
it
out
yet.
A
Part
of
the
multi
networking
app
wrote,
part
of
the
multi
networking
working
group
is
also
a
lot
seeking
a
way
to
combine
device
plugins.
So
the
device
plug-in
interface
inside
kubernetes
combine
that
with
multi.
Networking
right
makes
a
lot
of
sense.
There's
a
lot
of
common
features
that
we're
that
could
be
thought
of
functionality
that
should
be
shared
right
device,
plug-ins
allocated
device
device,
plugins
managed
schedule
resources.
None
of
these
things
are
in
CNI.
Cni
knows
how
to
initialize
the
network.
A
Cni
knows
how
to
hand
an
IP
address
back
to
the
cubelet
for
it
to
expose
right.
So
there
is
a
sort
of
an
interesting
daemon
right
now
that
works
entirely
with
annotations.
If
this
is
something
you're
interested
in
and
getting
it
into
core
kubernetes
join
sig
network,
because
this
is
InDesign,
and
this
is
a
really
cool
feature.
That's
coming!
So,
what's
next,
really
that's
up
to
you
right!
This
is
a
wide
open
community.
We
take
lots
of
pull
requests.
We
have
hundreds
of
contributors
and
we're
going
to
have
a
meet
the
maintainer
x'
later
today.
A
It's
an
active
specification.
There's
lots
to
do
so.
I'm
really
excited
this
community
knows
so
much
there's
so
much
more.
We
can
be
so
much
more.
We
can
do
that.
Working
is
really
complicated.
It's
really
diverse
and
I'm
really
excited
to
see
what's
next,
so
thank
you
so
much
for
sticking
through
these
technical
details.
Are
there
any
questions
so.
A
The
question
was:
what
happens
when
you
change
the
configure
the
CI
configuration
file
on
disk?
What
happens
one
of
the
how
those
changes
propagated
and
the
answer
is
for
existing
networks
for
existing
interfaces.
Those
changes
are
never
propagated
because
there
is
no
change
verb
in
CNI.
Essentially,
only
all
new
interfaces
going
forward
will
have
the
configuration
changes.
So
if
you
change
your
configuration
file
in
a
substantial
way,
you
should
just
reboot
the
node.