►
From YouTube: Multiple Networks for Kubernetes Workloads - Piotr Skamruk, CodiLime & Doug Smith, Red Hat
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Multiple Networks for Kubernetes Workloads - Piotr Skamruk, CodiLime & Doug Smith, Red Hat
Embark on a tour of CNI multiplexers -- an adventure in attaching multiple network interfaces to pods. We'll show the advantages of each and provide examples to get you started using them. We'll also talk about the history and future of multiple network attachments in Kubernetes. Kubernetes is based on simplicity, and Kubernetes networking is no different-- simplicity is king. Each pod is given a single IP address, and a single network in which “everything sees everything”. This model is not always what consumers expect, especially for high performance networking. In this world we need to have network isolation (to isolate traffic between control & data planes) or to have multiple interfaces in pods. This provides operators better control over functionality, latency and throughput. We'll make sure you're geared up for the adventure with CNI multiplexers and multiple network attachments!
https://sched.co/UabS
A
All
right,
thank
you,
guys
appreciate
making
it
here
after
the
big
switchover
appreciate
the
intrepid
adventurer
that
you've
been
on.
So
my
name
is
Doug
Smith
joined
by
peyote
and
I
am
one
of
the
maintainer
zuv
multi
CNI,
which
is
a
meta
plugin
for
CNI
that
will
get
into
that
and
I'm.
Also,
a
member
of
the
network,
plumbing
working
group
that
works
on
plumbing
multiple
interfaces
or
network
attachments,
as
we
like
to
say,
into
pods
and
kubernetes
and
Pio
an
introduction
and.
A
All
right,
so
here's
a
quick
agenda:
we're
gonna
talk
about
CNI
multiplexers,
some
of
the
plugins
that
we
will
talk
about
live
on
kind
of
a
spectrum
of
what's
strictly
a
multiplexer.
So
we're
gonna
talk
about
these
other
approaches,
we'll
talk
about
network
attachments
as
a
theme
and
then
we're
gonna
talk
about
some
native
object
proposal
that
Pio
has
presented
for
us.
A
So
why
do
we
really
care
about
having
multiple
network
attachments?
Sometimes
what
we
get
in
the
world
of
kubernetes
is
a
focus
on
web
scale,
applications
and
in
web
scale.
All
you
care
about
is
just
some
traffic
going
out
and
getting
traffic
back.
So
your
pot
has
this
one
singular
network
attachment
that
looks
like
an
interface
in
your
pod
and
it
just
attaches
you
to
everything.
A
But
once
you
start
looking
at
workloads
that
are
Network
specific,
you
have
different
type
of
architectural
considerations
that
you
need
to
manage
and
what
I've
Illustrated
here
is
this
idea
of
control,
plane
data,
plane
separation
and
when
you
just
have
that
single
interface,
you
start
the
ask
the
question
of
how
am
I
going
to
approach
this
problem.
So
we're
going
to
talk
about
some
of
the
approaches
that
are
used.
B
Kubernetes
knows
almost
nothing
about
network
which
is
prepared
by
System
Operator
on
the
nodes
on
which
the
part
of
kubernetes,
which
is
working
his
cubelet
cubelet,
is
not
configuring
networking
for
pot
it's
asking
for
that
around
time
and
that
runtime
could
be
container
d-doc
regime
or
even
via
that
for
virtual
machines.
So
everything
what
could
speak
using
CRI,
what
means
container
time
interface
protocol
and
this
runtime
when
it's
asked
to
prepare
sandbox,
which
is
a
place
where
pod
containers
will
be
running,
is
somehow
configuring.
B
Networking
that's
how
it's
done
is
dependent
on
the
runtime
most
runtimes
are
using
CNI
scene
is
what
means
container
networking
interface
and
it
will
be
discussed
later,
but
it's
not
a
hard
dependencies.
So
it's
up
to
implementer
of
runtime
how
it's
done
in
FX
q
Burnett
is
not
only
about
single
IP
and
it's
interested
only
about
single
IP
for
the
pod,
and
it's
not
always
what
we
expect,
because
we
have
bigger
needs.
B
Basically
I
say
it
already
about
this
assumption,
so
kubernetes
is
expecting
from
han
time
that
it
will
add
to
port
some
IP
address
and
that
it
will
be
available
without
and
Network
translation.
Not
every
port
should
have
a
possibility
to
see
everything
on
the
cluster.
So
all
services
services
are
not
services
exposed
by
kubernetes
and
in
most
cases,
that's
configured
by
CNI.
B
A
Absolutely
so
something
that
we'll
talk
about
a
lot
is
CNI,
which
is
the
container
network
interface
and
really
what
CNI
is?
Is
it's
it's
an
API.
It's
a
way
to
describe
how
you
can
make
a
pluggable
framework
in
order
to
extend
the
network
capabilities
of
a
container
platform.
Cni
itself
is
platform
agnostic.
So
it's
not
purpose-built
for
kubernetes.
It's
for
any
container
platform,
however.
A
Kubernetes
does
implement
it,
and
some
of
the
things
that
we
really
like
about
CNI
is
that
it's
really
easy
to
use.
If
you
can
have
a
programming
language
that
can
talk
standard
in
standard
out,
you
can
use
it
to
interface
with
CNI
now
granted
most
of
the
time,
you'll
see
CNI
plugins
that
are
written
in
go
line,
and
this
is
primarily
because
the
maintainer
zuv
see
and
I
have
made
some
really
great
libraries
for
us
to
use.
A
That
makes
it
easy
to
leverage
something
else
that
we
really
like
about
CNI
as
well,
is
that
it
allows
us
to
offload
the
kind
of
network
intelligence
that
you
need
away
from
kubernetes
and
into
your
own
specific
plugins.
So
you
can
get
plugins
that
are
made
by
you
by
your
favorite
vendors
and
by
the
community,
instead
of
having
all
of
that
knowledge
packed
right
into
kubernetes
itself.
So
you
can
meet
your
specific
demands
for
networking
in
particular,
so
just
to
go
over
a
quick
example.
A
Reference
see
and
I
plug
in
this
one
goes
over
bridge
and
basically
the
flow
of
what
happens
is
that
you'll
have
a
container
run
time.
It's
gonna
call
CNI,
which
in
turn
will
run
your
plug
in
and
it's
there's
gonna
be
a
series
of
variables
that
are
sent
along
to
the
C&I
plugin.
It's
gonna
contain
an
entire
configuration.
It's
gonna
contain
your
container
ID
and
it's
going
to
contain
a
method
or
operation,
that's
currently
being
performed,
such
as
add
or
delete,
and
there's
a
number
of
other
methods
that
are
there.
A
Then,
once
your
plugin
runs,
you
basically
are
free
to
do
whatever
you
need
to
plumb
your
particular
pot.
So
in
the
case
of
the
bridge
CNI,
essentially
it
takes
that
a
container
ID
and
it'll
create
a
bridge
on
the
host
node,
and
then
it
will
make
up
these
pair
between.
So
that's
a
like
virtual
Ethernet
connection
between
your
pod
and
the
bridge
on
the
host
and
that'll
be
presented
as
your
typical
ethernet
device
inside
your
pod.
A
That's
like
a
CNI
plugin
that
can
be
run
to
perform
IP
address
management,
that's
where
our
ipam
comes
from
and
the
rest
is
up
to
you.
So,
for
example,
in
the
bridge
field,
that's
there
that's
specific
to
the
bridge,
see
and
I
plug
in,
and
it
will
denote
the
name
of
the
bridge
device.
That's
created
on
the
note,
so
you
can
use
those
essentially
in
a
free-form
fashion.
So
let's
talk
a
bit
about
CNI
multiplexers,
which
is
getting
into
more
of
the
core
of
what
we
came
to
talk
to
you
about.
A
So
a
CNI
multiplexer
is
also
sometimes
known
as
a
CNI
meta
plug-in
and
we
say
it's
a
meta
plug-in
because
it's
a
CNI
plug-in
that
can
run
other
CNI
plugins,
so
that's
kind
of
the
strict
definition
of
a
CNI,
multiplexer
or
meta
plug-in.
So
on
the
left
here.
What
we've
got
is
we've
got
your
runtime
calling
what
we'd
say
is
a
default
CNI.
A
So
like
a
CNI
that
provides
pod
to
pod
connectivity
that
allows
your
pods
to
communicate
over
the
network
to
other
pods,
so
it
calls
your
default
CNI
and
then
it's
gonna
plum
that
pod
with
an
east
zero
right.
So
that's
where
we
get
this
one
interface
that
we
see
well.
Technically,
if
you
go
into
a
pod,
execute
IPA,
you're
gonna
see
a
loop
back
and
at
zero
right.
So
that's
what
you've
got
by
default.
So
one
of
the
solutions
to
plumbing
your
pod
to
attach
to
multiple
networks
is
to
use
a
CNI
multiplexer.
A
So
on
the
right
we've
got,
the
cni
multiplex
are
being
called
by
the
container
runtime
and
then
it's
gonna
call
your
default
CNI.
So
you
still
have
that
pod
pod
connectivity.
Then
it's
gonna
call
n
number
of
additional
CNI
plugins
in
order
to
plumb
additional
interfaces
or
in
the
case
of
technologies
that
don't
use
a
traditional,
you
know
Linux
kernel
type
interface.
It
may
perform
other
actions
like
say
it's
for
user
space
like
a
DVD,
Kay,
etc.
A
It
may
not
present
itself
that
way,
but
it
can
do
whatever
plumbing
it
needs
to
do
something
that
we'll
talk
a
bit
about
as
we
continue.
The
rest
of
the
conversation
here
is
about
custom
resources,
so
this
is
something
that
used
heavily
by
a
number
of
CNI
plugins
that
perform
this
kind
of
functionality
and
the
kind
of
fancy
way
to
talk
about
custom
resources.
A
Pardon
me
is
to
say
that
custom
resources
are
an
extension
of
the
kubernetes
api
and
to
me
that
sounds
pretty
highfalutin
when
really
what
it
comes
down
to
in
the
end
is
a
pretty
simple
thing:
it's
a
way
to
store
some
information
in
the
kubernetes
api
and
let
your
application
query
for
it.
So
it
can
provide
us
kind
of
a
lingua
franca
for
your
applications
to
speak
amongst
themselves.
So.
A
On
the
network
attachment
definitions,
this
is
something
that's
a
standard.
That's
been
proposed
by
the
network,
plumbing
working
group,
and
essentially
the
idea
is
to
create
this
way
to
express
that
you'd
like
to
attach
to
multiple
networks
and
the
reasoning
behind
creating
this
type
of
specification
is
to
create
a
more
uniform
user
experience
for
people
that
are
using
these
kind
of
technologies.
So
this
way,
if
the
technology
changes
behind
the
scenes
as
a
user,
that's
really
annoying
for
you.
A
If
you
have
to
change
all
of
your
configurations
to
a
different
kind
of
standard,
so
this
kind
of
idea
is
to
say:
hey:
let's
come
up
with
a
community
accepted
de-facto
standard
that
lets.
You
express,
hey,
I'd
like
to
attach
the
multiple
networks,
and
you
can
let
the
technology
behind
the
scenes
perform
the
dirty
work
there.
Instead
of
putting
that
right
in
the
users
face,
multi
CNI
is
a
CNI
multiplexer.
It's
a
meta.
A
Plugin
I
am
one
of
the
maintainer
x'
of
this
particular
technology,
and
it
is
a
reference
implementation
for
the
working
group
that
came
up
with
network
attachment
definitions
and
kind
of
the
core
behind
it.
Is
you
take
a
CNI
configuration,
so
that's
that
kind
of
freeform,
Jason
and
I
was
talking
about
and
it's
packaged
inside
custom
resource.
So
since
that
custom
resource
is
stored
in
the
kubernetes
api,
it
can
be
accessed
from
any
particular
node,
and
it
allows
you
to
say
hey
this
particular
configuration
in
this
example.
It's
for
Matt
VLAN
type,
CNI
plugin.
A
B
B
So
it's
configured
by
additional
annotation
and
in
this
fewest
simple
case,
it's
using
its
custom
annotation
named
CNI,
in
which
you
have
comma
separated
list
of
names
of
configuration.
We
should
be
attached
to
a
pod,
and
this
case
is
also
interesting,
because
with
this
list
you
can
skip
default
network
configuration.
So
you
can
configure
your
pod
to
have
an
access
to
your
private
network,
which
is
somehow
disconnected
from
kubernetes
api.
B
So
we
have
this
custom
CNI
annotation,
and
if
we
define
another
multi
IP
preferences
in
the
form
which
is
presented,
it
will
be
filled
by
CNI
Jenny,
with
information
about
what
was
really
added
to
the
pot.
So
that's
the
way
to
inform
what
was
really
configured
because
in
normal
pod
status
you
have
only
a
single
IP.
So
that's
the
place
where
we
will
have
full
information
about
different
IPS.
B
Another
option
is
to
use
network
attachments
so
in
case
of
network
attachments,
the
configuration
which
is
shown
there
it's
based
on
other
name
for
annotation
and
this
example
will
work
also
with
motifs.
That's
the
wife.
These
network
attachments
were
prepared
to
have
a
common
way
to
configure,
even
if
the
technology
under
the
hood
is
different,
and
it
also
have
this
name
for
networks.
So
that's
the
name
of
network
attachments
configurations,
as
shown
by
DAC,
and
also
additional
annotation
for
status
and
information.
What
is
really
configured
for
that
bot.
A
Another
technology
that
acts
in
a
similar
fashion,
but
is
is
different,
is
Nokia's
danam
and
something
about
danam.
That,
I
think
is
interesting
is
that
it
provides
maybe
a
more
opinionated
way
about
how
to
attach
to
a
multiple
network.
So
instead
of
having
arbitrary
c-
and
I
plugins
that
you
can
attach
to
it-
has
its
own
customized
custom
resources
that
speak
about
specific
aspects
of
those.
A
B
So
there
are
also
another
option
to
configure
multiple
interfaces.
One
of
them
is
tungsten
fabric.
It's
a
bigger
project
to
interconnect
with
different
types
of
clouds.
You
can
connect
your
kubernetes
to
open
ships
to
OpenStack
to
VMware
or
other
oceans,
and
it
has
a
way
like
CNI
Jenny,
to
configure
your
network
in
using
their
custom
annotation,
but
it
also
has
a
possibility
to
leverage
Network
attachments,
but
it's
not
a
multiplexer.
B
B
Another
option
is
that
in
neither
it's
also
speaking
CNI
protocol,
but
most
of
configuration
is
done
outside
of
CNI
configuration
piles
and
even
if
it's
speaking
in
a
protocol,
it's
always
returning
the
result
with
information
of
configuration,
hard
code
with
hard
coded
values
so
which
are
incorrect
in
the
time.
So
it
cannot
be
used
by
all
runtimes,
because
sometimes
runtime
will
expect
correct
values,
for
example
via
cloud
for
virtual
machines
will
expect
really
Watts
configurate
on
hosts
and
this
kind
of
hard
core.
That
data
will
simply
not
work.
A
So
I
just
wanted
to
speak
for
a
minute
about
the
network,
plumbing
working
group
that
came
up
with
these
this
de
facto
standard
for
a
network
attachment
definition.
This
was
a
community
founded
effort
that
happened
at
Q
Khan
in
Austin
in
2017,
and
essentially
there
was
kind
of
a
bit
of
a
frustration
with
how
we
could
take
this
concept
of
multiple
networks
and
how
were
we
going
to
approach
this
as
a
community?
A
So
what
we
decided
to
do
is
we
got
together
and
we
formed
an
out
of
tree
solution
for
how
to
express
your
intent
for
this
attachment
to
pull
multiple
networks
and
I'm
really
proud
of
the
work
that
our
group
is
done.
It's
a
easy
to
get
along
with
group
of
guys
and
everyone
else
who
joins
the
effort.
However,
I'd
really
encourage
you
to
come
and
join
some
of
our
meetings.
A
This
time
of
year,
we
tend
to
operate
on
kind
of
a
cube
con
to
coop
con
schedule,
and
some
of
the
recent
improvements
that
we've
made
over
this
last
year
is
to
specify
how
you
could
change
your
default.
Network
selection
for
routing
to
a
default
gateway.
We've
also
figured
out
how
to
approach
the
use
of
CNI
args,
which
allows
you
to
specify,
on
a
pod
per
pod
basis,
some
exceptions
to
your
general
custom
resource
that
you've
used
and
so
definitely
join
up
with
us
and
see.
A
What's
going
on,
we
meet
on
opposite
weeks
from
when
sig
network
gets
together
and
probably
around
this
time
of
year,
we'll
start
to
figure
out
what
our
goals
are
for
the
next
year.
We
have
some
goals
that
are
overlap,
we'll
have
some
new
stuff.
So
if
you
have
considerations,
we'd
love
to
hear
them
and
we'd
love
your
participation
in
the
formation
of
these
specifications
and
the
reference
technologies
that
we
build.
So
we
don't
just
make
the
spec.
We
also
make
the
technologies
and
we
have
this
feedback
loop
going
into
it.
B
B
So
this
is
the
list
of
URLs
of
technologies
used
in
our
presentation.
The
presentation
is
available
on
sketchcam,
so
you
can
simply
reuse
these
links
and
also
there
is
a
link
to
youtube
to
some
small
demo
about
how
you
can
switch
on
the
fly
where
traffic
should
go
using
tungsten
fabric
and
under
the
hood.
You
have
cue
Benetti
spots
and
open
star
crystal
machines,
so
you
have
a
choice
to
switch
on
the
fly
from
web
interface,
where
the
workload
shoot
oh
well,
the
traffic
should
go,
and
basically
that's
it.
C
C
A
Yes,
so
you
can
so
that
is,
it
probably
should
be
diagrammed
a
little
bit
better,
but
that
is
intended
to
represent
n
number
of
additional
attachments,
and
so
there
may
be
some
dependency
on
the
particular
plug-in
the
that
you
use.
However,
it
was
certainly
one
of
the
first-class
considerations
when
we
first
started
writing
the
specification
was
to
say
I'd
like
to
attach
the
same
definition
multiple
times
as
multiple
interfaces.
B
The
specification
says
about
configuration
and
this
configuration
did
this
different
configuration
can
use
the
same
binary,
which
is
under
the
field,
and
this
is
different
configurations
probably
will
have
configuration
for
different
subnets,
but
still
they
can
use
under
the
hood.
The
same
binary
from
the
same
plugin.
A
I
and
so
third-
the
yes,
so
the
question
is,
is
ipv6
supported
under
CNI
multiplexers
and
the
way
that
the
specification
is
written.
We
have
certainly
considered
ipv6
from
the
get-go
and
for
each
iteration
that
we've
made
on
the
specification.
So
just
as
a
quick
example
is
we
made
a
proposal
that
we
accepted
recently
that
was
for
default
route
selection
and,
as
you
read
through
that
you'll
see
how
we
addressed
it
in
order
to
account
for
ipv6.
So,
for
example,
under
ipv6,
you
can
have
multiple
gateways
for
a
default
route.
A
So
we
were
like
that's
got.
That's
got
to
be
in
there
right
now,
particular
CNI
plugins
may
have
so
probably
your
CNI
multiplexer
will
be
able
to
pass
through
that
information
to
each
additional
CNI
plug
in
some
CNI.
Plugins
may
have
limitations
based
on
ipv6,
but
I
think
that
if
you
look
at
the
reference,
CNI
plugins,
as
put
together
by
the
scene,
CNI
maintainer
Xin
that
community
I
think
that
you'll
see
a
strong
support
for
ipv6
in
those.
So.
B
Simply
multiplexers
are
a
way
for
providing
configuration
to
underlying
plugins
and
it's
up
to
this
plug-in
if
it
will
support
ipv6
or
not,
and
in
this
area,
multiplexers
will
gather
information
about
what
was
configured
if
plug-in
is
returning.
Cni
result,
which
is
correct
and
to
be
more
precise.
Molto's
in
the
mine
line,
has
not
support
to
combining
results
from
different
plugins,
but
it
has
a
branch
which
is
providing
such
option.
B
So
a
D
and
a
result
will
have
whole
information
from
all
blackens
running
under
the
hood
and
see
mi
chinese
has
this
possibility
from
the
start
so,
but
still
most
kubernetes
runtimes
are
not
looking
on
this
result
and
they
are
looking
inside
container
namespace
to
guez
wutz
configured,
so
it's
kind
of
out
of
scope
of
multiplexers
how
it's
done
between
CNI
plug
in
and
around
time.
How
runtime
is
using
this
information,
so
multiplexers
are
returning
CNI
result,
but
it's
up
to
runtime
how
this
information
is
used.
B
A
A
So
the
question
is:
will
there
be
a
convergence
between
the
work,
that's
being
done
in
core
kubernetes
for
ipv6
ipv4,
dual
stack
and
the
efforts
that
are
being
done
here
and
I
think
that
there
are
aspects
of
the
dual
stack
work
that
are
very
interesting
from
a
multiple
network
attachments
perspective,
I.
Think
to
me,
one
of
those
that's
very
interesting
is
this
kind
of
idea
of
the
singular
versus
ploor?
Plural
IP
addresses,
for
example,
so
something
that's
been
a
challenge
that
we've
explored
to
an
extent
in
the
network.
A
Plumbing
working
group
is
the
idea
of
having
services
on
your
additional
network
attachments,
and
certainly
you
know
right
off
the
bat.
We
took
a
look
at
this
and
we
said
well
in
a
service
object
in
kubernetes.
It's
a
singular
IP
address,
so
I
think
that
there
are
aspects
of
both
that
can
interoperate
very
well
and
may
be
very
interesting
in
the
future.
However,
I
think
that
there
is
a
slight
difference
in
the
idea
of
attaching
to
these
networks
that
may
be
used,
for
example,
for
data
plane,
control,
plane.