►
From YouTube: Stretching CNI Boundaries with Service Meshes, a Roadmap for the Future - Alex Leong, Buoyant
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2023 in Amsterdam, The Netherlands from April 17-21. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Stretching CNI Boundaries with Service Meshes, a Roadmap for the Future - Alex Leong, Buoyant
Container Network Interface (CNI) plugins such as Calico or Cilium are typically used to provide container network connectivity and network policy. However, service meshes such as Linkerd and Istio also use CNI plugins to configure the networking rules that allow their sidecar proxies to intercept incoming and outgoing traffic. This means that it is increasingly common to have more than one CNI plugin installed at a time, which can lead to race conditions where the CNI plugins overwrite each other's configuration. In this talk, Alex Leong will demonstrate how to detect and resolve these problems and suggest a set of best practices for CNI plugins to ensure compatibility with other plugins. She'll also explore some potential changes to the CNI plugin specification, which could solve these problems at a structural level.
A
Thanks
everybody
for
coming
this
talk
is
called
stretching
the
cni
boundaries
with
service
meshes
a
roadmap
for
the
future.
So
first
a
little
bit
about
myself.
My
name
is
Alex
I'm,
a
software
engineer
at
a
company
called
buoyant
I'm
one
of
the
Linker
D
maintainers,
so
I
feel
very
lucky
that
I
get
to
spend
my
day.
Job
working
on
open
source
software,
which
is
awesome.
A
A
A
So
this
is
basically
what
we're
going
to
cover
we're
going
to
talk
about
what
what
is
Linker
D
and
what
is
a
cni
plug-in
to
give
you
some
background,
so
that
this
stuff
makes
sense
and
I'm
going
to
talk
about
two
problems:
kind
of
in
particular
that
we
ran
into
what
happens
when
there's
multiple
cni
plugins
installed
and
there's
conflicts
between
them.
And
how
do
you?
A
Okay,
so,
first
and
background
what
is
Linker
d
I'm
not
telling
this?
Hopefully
you
know
if
you're
here
at
255
at
service
mesh
con
you've
at
least
heard
of
linker
D,
you
at
least
know
the
concept
behind
a
service
mesh,
and
so
for
this
talk,
I
really
want
to
focus
on
how
service
measures
use
cni,
rather
than
spend
a
lot
of
time.
A
On
the
background,
if
you
are
interested
in
learning
more
about
this,
there's
a
really
good
talk
on
Wednesday
called
whose
packet
is
this
life
of
a
packet
through
the
service
mesh
from
Kevin
and
Doug.
That's
going
to
go
into
a
lot
more
detail
about
the
iptables
rules
and
the
routing
that
Linker
uses
I'm
going
to
talk
about
that
at
a
high
level
today
and
talk
about
how
those
things
get
set
up.
A
But
if
you
really
want
to
get
into
the
nitty-gritty
of
what
those
rules
are
and
how
they
work,
this
is
going
to
be
a
really
really
good
talk
for
something
a
little
more
high
level.
I
also
have
a
talk
on
Thursday,
which
is
overview
and
state
of
liberty.
That'll
give
a
higher
level
talk
about
what
Linker
D
is
how
it's
used.
What
is
new
in
the
past
year
or
so,
and
kind
of
what's
coming
up
on
the
road
map,
so
I
highly
recommend
both
those
talks.
A
So
you
do
need
a
little
bit
of
background,
so
I
do
have
to
tell
you
a
little
bit,
which
is
that,
for
the
purposes
of
this
talk,
all
we
really
care
about
for
service
meshes
is
that
they
are
something
that
runs
a
sidecar
proxy.
Here's
the
Lincoln
proxy
in
each
pod
and
that
proxy
is
responsible
for
intercepting
all
inbound
and
outbound
traffic
from
that
pod
and
doing
stuff
with
it,
and
so
the
way
it
accomplishes.
A
That
is
that
traditionally,
there
is
an
init
container,
which
also
runs
in
the
pod
and
that
init
container
runs
before
any
of
the
the
main
containers
and
its
job
is
just
to
set
up
ibtables
rules
which
configure
that
routing.
So
anytime,
someone
tries
to
establish
a
connection
to
anything
in
that
pod.
A
Those
ibtable
rules
will
reroute
that
connection
to
the
proxy
and
the
proxy
can
handle
it,
and
similarly,
on
the
outbound
side,
whenever
anything
in
the
Pod
tries
to
connect
outside
of
it,
those
ibtable
rules
will
intercept
that
connection
and
rewrite
it
so
you're
connecting
to
the
proxy
instead.
So
the
proxy
intercepts
in
both
directions.
A
That's
fine,
that's
no
issue,
but
if
you
really
want
to
lock
things
down-
and
you
don't
want
every
pod
in
your
cluster
to
necessarily
have
these
privileges,
that
can
be
a
problem,
and
so
the
question
we
ask
ourselves
is
to
get
around
that.
Is
there
any
way
to
set
up
these
iptable
rules
ahead
of
time
so
that
we
don't
have
to
Grant
these
privileges
to
everything
in
the
cluster?
Is
there
some
way
we
can
get
this
set
up
for
every
Pod
without
needing
those
additional
Privileges
and
cni
container?
A
So
what
is
cni
so
cni
stands
for
container
networking
interface
and
it's
basically
the
system
where
you
can
have
these
things
called
cni,
plugins
and
cni
plugins
are
installed
on
the
host
and
they're
called
by
the
container
runtime
whenever
a
new
pod
is
created
and
their
job
is
to
set
up
the
network
appropriately
for
that
pod,
and
so
what
that
means
can
vary
a
lot
depending
on
the
plugin
different
plugins
can
do
different
things.
You
know
they
may
set
up
the
Pod
Network.
A
It's
mostly
made
up
of
just
two
files,
so
there's
a
configuration
file
and
there's
an
executable
binary
and
those
live
on
the
Node
file
system,
and
so
anything
that's
kind
of
in
the
right
place
and
in
the
right
format
on
the
Node
file
system
will
get
recognized
as
a
cni
plugin
and
whenever
a
new
pod
is
created
on
that
node,
the
container
runtime
is
going
to
call
that
plug-in
and
say:
hey,
you're,
a
plug-in
you're
supposed
to
set
up
the
network.
Do
your
thing.
A
So
right,
like
I,
said
we
can
have
this
Linker
dcni
plugin,
which
configures
the
iptable's
rules
for
a
pod,
and
if
we
do
that,
then
we
no
longer
need
that
init
container
anymore,
because
those
rules
are
set
up
by
cni
when
the
Pod
is
created.
We
don't
need
an
init
container
to
do
that
and
we
also
don't
need
those
elevated
privileges.
A
So
it's
great
and
it
works
great
liquidy
has
been
doing
this
for
a
long
time
and
and
it's
fantastic,
but
it
does
raise
a
question
of
how
do
you
get
that
cni
plugin
set
up
in
the
first
place?
Remember
I
said
that
that
was
made
up
of
files
on
the
Node
file
system.
So
how
do
they
get
there?
Who
puts
them
there.
A
So
I
think
traditionally,
when
you're
using
cni
plugins,
it's
the
responsibility
of
the
cluster
administrator
to
to
set
that
up.
But
you
know
it's
a
little
bit
involved.
You
have
to
take
these
files,
you
have
to
put
them
on
the
host
file
system
or
the
node
file
system.
In
order
to
know
how
to
do
that,
you
need
some
amount
of
cni
expertise.
You
could
make
a
mistake
doing
that.
Where
do
you
put
the
files?
A
What
format
do
they
have
to
be
in,
and
you
probably
want
to
integrate
this
with
your
Automation
in
some
way,
so
that
you're
not
manually
doing
this
for
every
for
every
node?
So
it's
like
it's
a
non-trivial
task
to
get
this
set
up
and
one
of
the
philosophies
behind
Linker
D
is
we
wanted
to
alleviate
as
much
as
possible
the
kind
of
manual
administrative
work
that
can
be
automated
and
take
that
burden
off
of
cluster
administrators
wherever
it
makes
sense
to
so
we
asked
the
question:
well
what
if
a
cni
plug-in
could
install
itself.
A
Is
it
mounts
the
node
file
system
and
copies
those
files
into
place
very
simple,
but
what
it
means
is
that,
once
you
run
that
those
files
will
be
copied
onto
every
node
and
then
from
that
point
forward,
any
pods
that
are
created
will
be
running
on
that
node
with
that
cni
plugin
installed,
and
so
the
container
runtime
will
call
into
those
that
plugin
and
say
hey,
there's
a
pod
being
created.
I
need
you
to
set
up
the
network.
A
So
this
works
wonderfully
most
of
the
time,
but
we
did
run
into
some
problems,
especially
for
people
who
were
running
multiple,
cni
plugins.
You
know
we
would
get
these
error
reports,
we're,
saying:
hey
I'm
running
with
Calico
or
I'm
running
with
you
know,
psyllium
or
I'm
running
with
whatever
and
things
aren't
working.
A
So,
like
I
said,
cni
is
designed,
so
you
can
have
multiple
cni
plugins
installed
at
the
same
time
and
that's
fine,
because
they'll
run
in
chain
mode
and
and
they'll
get
consequently
and
everything's
great.
But
if
you
have
two
different
cni
installers
that
are
unaware
of
each
other.
Well,
what
are
they
going
to
do?
They're
both
going
to
run
on
the
Node
they're,
both
going
to
mount
the
file
system,
and
you
know
Linker
dcni-
is
going
to
copy
these
files
into
place.
A
A
So,
since
those
installers
are
not
kind
of
aware
of
what
the
other
one
is
doing,
you
can
get
in
the
situation
where,
where
they're
overriding
each
other,
remember
that
when
you
just
have
a
single
cni
plugin,
you
have
what's
called
a
dot-comp
file.
But
if
you
have
multiple
cni
plugins,
you
need
to
have
a
conf
list
file
and
So,
since
neither
those
plugins
are
aware
of
each
other.
They
don't
know
to
create
that
conflict
file.
A
They
both
think
they're,
the
only
one
there
and
they
step
on
each
other's
toes,
and
so
the
result
is
that
you
end
up
in
these
very
broken
Network
states,
where
you
think
you
have
multiple
plugins
installed,
but
only
one
of
them
is
running
and
you
don't
know
why,
and
you
know
anything
that's
expecting
to
be.
There
is
going
to
get
confused,
and
so
what
we
really
want
is
we
really
want
to
be
in
a
chained
configuration
we
want
accomplished,
but
neither
of
these
things
know
that.
A
So
the
solution
we
implemented
in
linkerty
was
basically
to
become
a
cni
plugin
manager.
So
what
we'll
do
is
that
whenever
you
have
that
Linker,
DC
and
I
damage
that
that's
installing
the
cni
plugins,
what
it's
going
to
do
is
it's
going
to
first
look
at
that
directory
and
say:
hey:
is
there
any
cni
plugin
already
installed
here?
A
You
know
this
works
but
like
it
uses,
I
notify
weight,
there's
a
whole
bunch
of
gross
bash
and
in
fact
the
majority,
maybe
even
the
vast
majority
of
the
code
in
the
cni
linked
cni
plugin
installer
is
dealing
with
parsing
config
files
and
watching
directories
and
rewriting
them
and
doing
all
of
that
kind
of
plug-in
management.
Rather
than
doing
its
own
job,
which
is
unfortunate.
A
So
this
works,
but
but
how
could
it
be
better-
and
these
are
just
kind
of
some
half-baked
ideas
about
about
how
we
could
make
this
better?
You
know
there's
a
lot
of
stuff
here
still
to
figure
out,
but
it
would
be
cool
if
we
had
some
kind
of
convention
for
how
cni
plugins
should
interact
with
each
other
when
they're
being
installed,
if
you're
a
cni,
plug-in
and
you're
installing
yourself,
and
you
notice
that
there's
already
some
plugins
there,
what
should
you
do?
A
What
should
the
conventions
be
around
chaining
to
those
or
or
somehow
integrating
with
them,
rather
than
just
kind
of
blindly
overriding,
but
there's
a
lot
of
questions
there
like?
How
do
you
decide
what
order
to
chain
those
in
because,
of
course,
these
cni
plugins
they
do
things
to
the
network.
The
orders
that
they
run
in
is
very
important.
So
how
do
you
determine
that
order?
How
do
you
deal
with
atomicity
if
you
have
two
plugins
installing
themselves?
At
the
same
time,
how
do
you
guarantee
that
that's
going
to
work
out?
A
Another
potential
idea
would
be
to
have
this
be
handled
sometime
somehow
by
the
runtime
to
basically
move
this
down
to
the
infrastructure
rather
than
having
the
Linker,
DC
and
I
be
a
plugin
manager.
You
know
what,
if
there
was
something
analogous
to
a
plugin
manager
that
would
take
all
those
configuration
files
and
know
how
to
combine
them
in
an
intelligent
way.
But
again
you
still
have
some
of
these
questions,
like
the
order
that
they're
combined
is
very
important.
So
you
need
some
way
to
specify
that.
A
Okay
and
the
other
issue
that
I
really
want
to
talk
about
was
a
cni
plug-in
race
conditions,
so
this
is
something
that
happens
when
a
node
joins
the
cluster,
and
this
can
happen
for
a
variety
of
reasons.
You
can
have
just
a
new
node
spin
up
because
you're
adding
expanding
capacity
or
something
like
that.
It
also
happens
when
a
node
restarts.
So
when
a
node
restarts
it's
going
to
leave
the
cluster
and
then
once
it
restarts
it'll
it'll
join
again,
and
so
this
is
something
that
we
saw
happening
a
lot
during
node
restarts.
A
A
And
kubernetes
says:
okay
well,
I've
got
some
pods
in
particular,
I've
got
the
Linker
dcni
pod,
because
this
is
a
demon
set
and
we
need
to
run
one
of
these
on
every
node.
So
here
you
go
and
potentially
I
have
some
other
pods
that
need
to
be
scheduled
so
I'm
going
to
schedule
those
as
well
and
there's
no
ordering
constraint
on
how
these
two
pods
come
up.
They're
just
both
supposed
to
be
scheduled
there.
So
they
are
and
remember.
A
A
It
could
send
traffic
there.
That's
had
a
protocol
upgrade
that
only
Linker
D
is
aware
of,
but
because
those
iptable
rules
aren't
set
up
on
that
pod.
All
that
traffic
is
going
to
go
directly
to
the
application
and
the
application
is
not
going
to
know
what
to
do
with
it,
and
things
will
be
very
broken.
A
So
the
solution
where
we
come
up
with
in
Linker
D-
and
this
is
something
that's
still
in
progress-
I
feel
work
is
mostly
done.
It's
just
a
matter
of
rolling
it
out
is
that
we
want
to
validate
the
routing
on
pod
startup.
So
in
other
words,
when
a
pod
starts
up,
we
want
to
validate.
Does
the
routing
rule?
Do
the
routing
rules
here?
Make
sense?
Are
they
what
they're
supposed
to
be?
A
Have
things
been
set
up
correctly
and
if
they're
not,
we
need
to
fail
the
Pod
and
start
over
again
and
try
again,
and
this
is
very
reminiscent
of
something
that
cni
itself
actually
does
so
cni.
Has
this
command
called
check,
I
believe
where
the
container
runtime
can
call
into
each
cni
plugin
with
a
check
and
say,
hey
I
know
you're
already
done
setting
up,
but
are
things
still
the
way
you
think
they
should
be?
Is
you
know,
are
you
still
set
up
correctly
and
the
cni
plugin
can
respond?
Yes,
things
are
good.
A
Everything
looks
fine
or
no
something's
gone
wrong.
We
need
to.
We
need
to
fix
things,
and
so
this
is
a
very
similar
idea
by
having
pods
check
their
own
routing
and
saying
is
the
routing
on
this
pod
set
up
correctly?
Yes,
it
is
we're
good
to
go,
or
no
it's
not.
We
need
to.
We
need
to
restart
this
pod
and
try
again.
A
A
So
that's
going
to
run
before
any
of
the
main
containers
it's
going
to
run
before
the
proxy
it's
going
to
run
before
the
application
and
it
what
it
does
is
it
sets
up
a
server
that
listens
on
Port
4140
4140
is
the
outgoing
Port
of
the
proxy.
So
what
that
means
is
that
when
things
are
running
correctly,
all
of
the
traffic
which
goes
out
of
the
Pod
is
redirected
to
Port
4140
on
the
proxy.
A
So
things
are
set
up
correctly,
that
should
redirect
back
to
there
and
we
can
detect
that
we
can
say
hey
we
connected
to
our
own
thing,
and
that
means
the
routing
rules
are
set
up
correctly,
we're
good
to
go.
The
init
container
can
terminate
successfully
and
then
the
Pod
can
spit
up
spin
up
as
normal.
A
If
this
doesn't
happen,
if
that
connection
kind
of
just
goes
through
to
its
original
destination
and
nothing
connects
to
4140.,
we
know
that
something's
gone
wrong.
We
know
that
the
iptables
rules
have
not
been
set
up
correctly
for
this
pod,
so
we
need
to
fail,
and
so,
when
that
IP
or
when
that
validator
container
fails,
the
whole
pod
will
fail.
Startup
kubernetes
will
kill
it
and
try
and
start
again
try
and
create
a
new
one,
and
hopefully,
by
the
time
that
pod
comes
up
a
second
time.
A
A
So
this
again,
this
is
something
that
works.
It
solves
the
problem.
It's
a
little
bit
gross
it's
a
little
bit
messy.
It
has
this
eventually
consistent
nature
where
pods
may
have
to
restart
several
times
if
they
are
scheduled
on
a
node
that
is
in
the
process
of
coming
up,
and
ideally,
there
would
be
kind
of
a
cleaner
solution.
A
So
what
we
would
really
love
to
see
is
if
there
was
some
way
to
add
constraints
on
The
Ordering
of
these
pods
so
that
we
could
say
yes,
we
know
that
the
cni
plugin
has
been
installed.
All
the
networking
stuff
is
set
up
before
we
try
to
schedule
any
other
pods
on
that
on
that
node,
and
so
that
could
be.
A
There
could
be
some
kind
of
affordance
for
that
in
kubernetes
to
say
that
hey
these
are
cni
set
up
jobs
or
pods,
and
we
need
to
let
those
run
first
or
more
generally.
You
know
some
other
way
of
specifying
pod
ordering
constraints
so
that
we
can
guarantee
that
certain
pods
are
run
and
complete
before
other
pods
are
scheduled,
but
as
far
as
I
know,
there
isn't
anything
quite
like
this.
A
If
you're
interested
in
kind
of
learning
more
about
the
nitty-gritty
of
how
Lincoln
it
works,
how
cni
plugins
work
and
really
dive
deep,
there's
a
very
good
service
mesh
academy
that
you
can
sign
up
for
it's
completely
free
I
think
the
next
one
runs
from
November
11th
to
17th,
it's
very,
very
Hands-On,
so
it'll
help
you
kind
of
get
into
some
of
the
details
there
that
I
glossed
over
in
this
talk.
So
that's
very
interesting.
A
If
that
is
something
that
you're
interested
in
buoyant,
also
offers
fully
managed
link
reviews,
so
if
you're
interested
in
running
Lincoln
production-
and
you
want
some
of
that
administrative
burden
alleviated,
this
should
be
of
interest
to
you.
Boinkcloud
can
handle
things
like
automatic
upgrades
and
and
other
administrative
tasks.
So
if
that's
interesting,
you
can
find
the
buoyant
Booth
or
you
can
book
a
demo
online
other
than
that.
I
am
going
to
be
around
to
answer
questions.
You
can
also
find
me
in
the
Linker
D
booth
in
the
project.
Pavilion.
A
C
A
Yeah,
that's
right,
so
it
only
tests
the
kind
of
the
outbound
Direction,
but
is
kind
of
as
there
as
a
proxy
right
it'll
tell
us
whether
the
rules
have
run
at
all
and
the
rules
you
know
are
intended
to
set
up
both
inbound
and
outbound.
So
this
kind
of,
if
we
see
that
outbound
is
working,
we
can
kind
of
assume
that
inbound
is
also.
B
That
was
a
great
talk.
Thank
you,
the
kind
of
issue
of
I
guess
all
your
critical
demon
sets
not
being
ready
until
and
then
user
applications
come
on.
I
remember
like
reading
a
lot
about
it
when
I
was
looking
to
the
storage
stuff
because
they
have
a
very
similar
problem
there
and
then
like
there
was
this
GitHub
issue
and
a
cat
floating
around
about
like
tating
nodes
and
all
that
kind
of
stuff
and
I
was
curious
about.