►
From YouTube: Five Ways With a CNI: Understanding Kubernetes Networking For Performance..-Stig Telfer & Erez Cohen
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe in Amsterdam, The Netherlands from April 17-21, 2023. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Five Ways With a CNI: Understanding Kubernetes Networking For Performance-Intensive Workloads - Stig Telfer, StackHPC Ltd & Erez Cohen, Nvidia
Speakers: Stig Telfer, Erez Cohen
Network abstractions are a cornerstone of Kubernetes, interconnecting containers, pods and services, both across the cluster and beyond. Historically this rich functionality often came at the expense of performance. However, recent innovations and new implementations of the Kubernetes Container Network Interface (CNI) have transformed the Kubernetes networking landscape. Some classes of scientific computing workloads are highly network-intensive: sensitive to network performance to a degree that renders them unsuitable for execution in a Kubernetes context. In machine learning, the most demanding distributed training applications are also becoming increasingly network-intensive. The CNIs now available offer different architectures and technologies. Many claim performance advantages over their predecessors. How can we distinguish between them? In this talk we describe the leading CNIs, and use real-world benchmarks to compare and analyze performance for demanding workloads.
A
B
B
High
performance,
Computing
or
HPC
is
a
field
of
computer
science
that
solves
complex
problems
such
as
fluid
dynamics
for
aircraft
design,
for
example,
large-scale
weather
forecast
or
drug
Discovery
through
the
use
of
large-scale
compute
simulations
HPC
is
one
of
the
most
compute
Network
and
storage.
Intensive
workloads,
Technologies
developed
for
HPC
often
make
their
way
to
more
standout
data
center
applications.
B
B
B
B
When
we're
looking
at
the
natural
consideration,
the
first
and
foremost
is
throughput
or
bandwidth,
and
here
we
would
like
to
have
as
much
as
we
can
today
we're
running
at
100,
Gig
and
going
forward.
We're
already
deploying
400
gigabit
per
second
but
latency,
and
the
predictability
of
latency
is
is
as
important
for
those
kind
of
workloads
and
obviously
we
would
like
to
have
low
and
predictable
latency
across
all
packet
types.
B
B
B
Now,
let's
review
the
various
Network
configurations
that
we
will
use
for
this
stock
and
we'll
start
with
Calico
is
a
popular
open
source.
Networking
and
network
security
platform
for
kubernetes,
created
by
tigera
Calico
addresses
connectivity
between
pods
to
pods
and
pods
to
service
and
service
to
ponds
and
I.
Think
in
it,
and
it
includes
Ingress
and
egress
load.
Balancing
Calico
also
provides
network
security
features
which
is
out
of
the
scope
of
our
talk
today.
B
Calico
has
a
central
controller
that
calculate
the
policies
and
routes
and
stored
them
in
an
etcd
data
store
on
each
node.
Calico
runs
three
components:
conf
day,
which
monitors
the
etcd
data
store
for
configuration,
changes
and
update
bird
configuration
files,
Felix,
which
configures
iptables
and
Linux
routes
based
on
the
policies
and
connectivity
and
bird,
which
is
an
open
source
bgp
agent,
which
advertise
Felix
routes,
conical
support,
both
bgp
and
Vixen
routing.
In
our
talk,
we
will
use
vx9
overlay.
B
B
B
B
However,
slav
provide
us
with
very,
very
high
performance,
close
to
Bare
Metal,
as
it
bypass
the
entire
virtualization
stack
to
use
sriv
in
kubernetes
we're
using
SDS
or
vcni,
and
also
in
addition,
the
SRV
device.
Plugin
slav
C90
support
the
all
the
network
requirements
of
a
pod
and
therefore
it
is
used
typically
as
a
secondary
cni
and
apparently
like
Calico
or
rovian
it
together
with
parameters
in
cni
as
Calico
ovn
through
the
use
of
multis,
which
is
a
meta,
Plugin
or
meta.
Cni.
B
B
When
a
packet
comes
in,
it
will
go
to
the
kernel,
module
or
the
fast
path
and
classify
the
packet
and
see
if
there
is
a
rule,
what
to
do
with
that
pocket.
If
there
is
a
rule,
the
packet
will
be
executed,
meaning
will
be
sent
to
the
Pod
or
do
something
else.
If
there's
a
miss,
this
packet
will
go
to
the
user
space
Daemon
for
further
analysis,
and
once
a
decision
is
made
what
to
do
with
this
new
flow.
B
It
will
program
the
the
kernel
data
plane
and
from
that
point
on
packets
will
just
go
through
the
fast
path.
Now,
both
the
kernel,
the
kernel
module
and
the
user
space
demon
are
CPU
running
on
the
CPU
and
the
general
purpose.
Cpu
is
not
very
good
in
processing,
High
rate
of
packets
and
therefore
ovs
and
many
of
the
other
virtual
switches
are
have
very
low
bandwidth
and
very
high
latency
and
high
CP
utilization.
B
In
addition,
because
the
interface
to
the
Pod
or
to
the
VM
is
eth
pair,
none
of
the
advanced
features
of
the
modern
X
are
supported
like
RDMA
and
dpdk,
and
so
on.
To
address
these
challenges.
Nvidia
added
to
our
NYX,
an
embedded
switch
ovs
was
enhanced
through
a
set
of
API
to
program
that
embedded
switch
on
the
neck
and
program
its
data
plane.
B
The
pods
are
now
connected
directly
to
the
neck
through
advanced
sriov
mode
called
e-switch
mode.
Every
packet
that
comes
from
the
network
will
hit
the
e-switch
first
and
will
classify
the
packets
there
if
there
is
a
rule
that
defines
what
to
do
with
that,
the
packet
will
be
executed
and
be
sent
to
the
Pod
and
vice
versa.
B
B
We
have
one
primary
cni
ovn
in
this
case
that
supports
all
the
capabilities
needed
for
kubernetes
all
the
kubernetes
services,
all
the
Network
Services,
the
quality
of
service
and
visibility,
and
so
on.
It
support
bonding
which
Legacy
srav
does
not,
and
it
supports
security,
but
unlike
the
virtual
switches
and
other
cnis
that
are
software
based,
it
provides
bare
metal
performance
of
the
server
and,
in
addition,
it
provides
additional
capabilities
that
the
network
adapter
may
have
such
as
RDMA
and
dpdk.
B
B
The
servers
are
quite
high-end
for
the
performance:
benchmarking,
we're
running
HP,
DL,
380
gen10,
with
8380
CPUs,
two
pneumas,
each
one
of
40
core,
so
overall
80
cores
and
half
a
terabyte
of
RAM
and
those
devices
are
and
those
servers
running
in
video,
connecting
6dx
with
two
ports
of
100
gigabit
each
on
the
OS,
we're
running
Ubuntu,
2004
with
latest
kernel,
and
then
video
offered
for
latest
networking
drivers.
Kubernetes
version
is
1.23.7
and
we
deployed
with
Cube
spray,
and
the
port
is
running
Ubuntu
again
with
the
offered
packages.
B
The
network
interfaces
are
bonded
and
we're
running
at
9kmtu
connecting
to
a
switch
and
a
SN
through
3700
32
volts
of
up
to
200
gigabit
per
second,
so
plenty
of
interfaces
running
cumulus,
Linux
4.4,
the
master
node
is
a
lower
grade,
server,
running
Ubuntu
2004
and
the
same
kubernetes
version.
So
this
is
the
bare
metal
environment.
B
The
kubernetes
over
openstack
environment
is
obviously
the
same
Hardware
two
compute
nodes,
but
this
time
on
the
operating
system,
we
did
install
yoga
openstack
and
we
created
a
VM
running
Ubuntu
2004
and
the
latest
kernels
and
and
offered
packages
and
inside
the
VM.
We
basically
deployed
the
same
kubernetes
as
before,
and
then
obviously
the
pods
running
in
that
kubernetes.
The
switches
and
all
the
rest
is
exactly
the
same.
Of
course,
in
the
controller
node,
we
did
need
to
install
openstack
yoga
controller.
B
We
don't
have
the
connectix
device
and
inside
the
holster
running
ovn
with
ovs
offload,
so
this
ovs
is
offloaded
the
data
plane
into
the
connectics,
as
I
explained
before
inside
the
VM,
we're
running
Calico,
which
is
connected
to
the
obvious
non-accelerated
and
that's
the
primary
Network
for
the
Pod.
And
in
addition,
we
have
a
secondary
ovn,
SRI,
V
Network,
going
to
the
Pod,
and
so
the
application
can
use
RDMA,
which
basically
goes
to
the
virtual
function
and
then
we'll
go
out
through
the
neck.
With
the
rules
and
the
sdn
layers
of
the
OBS.
A
B
A
A
suite
of
test
cases
that
should
be
both
insightful
and
representative
for
HPC
and
AI
applications.
This
Benchmark
Suite
is
also
open,
source
and
available
on
GitHub.
Today,
under
this
link,
it
uses
the
volcano
drop
scheduler
to
run
through
a
set
of
test
cases
and
extract
results
in
an
automated
and
repeatable
process.
A
A
A
A
A
On
this
slide,
we
have
two
charts
measuring
the
effect
of
a
9000
byte
MTU
and
its
uplift
on
performance.
It's
actually
not
such
a
massive
difference
in
results
here,
due
probably
to
the
high-end
network
cards
that
we're
using.
But
what
is
not
shown
is
the
extra
work
behind
the
scenes
that
the
host
kernel
is
doing
in
packet
processing
load
for
standard
frame
sizes.
A
This
hidden
cost
is
not
shown
in
the
end
results,
but
will
return
to
expose
and
explore
this
a
bit
later,
so
we've
adopted
a
9000
byte
MTU
for
the
benchmarks
in
this
presentation
as
being
considered
representative
of
HPC
networking
use
cases
before
we
go
on,
though
it's
just
amazing
to
see
that
Modern
Hardware,
a
Modern
Hardware
server,
can
readily
saturate
200
gigabits,
a
second
of
networking
foreign
first
up.
Let's
start
with
cni
results
with
a
couple
of
standard
choices.
A
Two
of
the
most
popular
cnis
for
kubernetes
are
calico
and
ovn.
In
this
slide.
Calico
is
the
chart
on
the
left
and
ovn
is
the
chart
on
the
right.
Color
code
is
configured
with
default
settings.
We've
done
nothing
special
here.
It's
using
vxlan
as
the
transport
layer.
Ovn
is
configured
without
sdn
acceleration.
At
this
stage.
It's
also
at
default
settings
we'll
return
to
the
sdn
acceleration
shortly.
A
A
Both
cnis
get
30
and
a
half
gigabits,
a
second
for
a
single
iperb
TCP
stream,
but
for
ovn
Beyond,
a
small
number
of
concurrent
TCP
streams,
our
performance
tails
off
to
a
fraction
of
peak
bandwidth.
Why
is
that
we
can
speculate
about
the
different
fortunes
of
these
two
cnis
for
ovn
offloading
performance
may
be
impacted
by
genetic
encapsulation
or
the
complexity
of
ovn's
programmable
sdn,
pipeline
routing,
net
security
groups
and
connection
tracking
are
all
parts
of
a
flexible
solution
in
OPN,
but
make
it
harder
to
apply
offloads
ovian.
B
A
B
A
We
appear
to
be
getting
benefit
from
the
stateless
offloads
of
the
connect
X6
network
cards
we
are
using.
In
particular,
the
network
card
can
do
TCP
segmentation
offload
to
a
vxlan
vni.
The
kind
of
networking
used
by
Calico,
enabling
the
Kernel's
networking
stack
to
process
network
data
in
much
bigger
chunks
in
a
Calico
configuration.
A
A
Choice
of
cni
for
high
performance
use
cases
is
to
use
sriv
in
which
pod
networking
has
direct
access
to
Nic
Hardware.
Unfortunately,
there
is
no
bonding
support
in
Lego
legacy
sriv
and
our
test
configuration
has
bonding
enabled.
So
this
restricts
us
to
only
one
port
of
our
dual
Port
network
interface
in
the
test
Hardware,
and
this
is
clear
from
the
results
where
the
bandwidth
quickly
hits
a
cap
of
100
gigabits.
A
second,
the
other
network
interface,
of
course,
can
be
used
for
Calico
or
another
cni,
providing
flexible
options
alongside
the
sriv
interface.
A
A
A
A
In
both
cases,
our
single
client
TCP
performers
is
over
50
gigabits,
a
second
somehow
exceeding
the
performance
of
the
host
networking
configuration
which
is
hard
to
explain,
but
maybe
due
to
Evolutions
in
our
lab
setup
tuning
with
accelerated
ovn,
we
achieve
very
high
performance
quickly,
saturating
the
200
gigabits,
a
second
available
I
should
say
the
bare
metal
accelerated
over
and
graph
on.
The
left
is
actually
the
best
result
of
five
runs.
A
A
Is
the
unfairness
apparent
in
the
ovn
configuration
the
different
stripe?
Widths
of
the
bare
metal
accelerated
over
the
end
result
implies
significant
unfairness
between
clients
under
contention,
which
requires
some
further
investigation
to
understand
a
little
better.
One
theory
is
that
a
reduced
number
of
Nick
receive
cues
used
in
the
ovn
virtual
functions
was
leading
to
uneven
performance.
A
A
A
B
A
A
Really
see
so
much
of
this
in
cloud
computing
and
it's
vitally
important
for
HBC
and
also
large-scale
AI
mpi's
design
also
fits
naturally
with
the
RDMA
Network
protocols
that
Ares
was
describing
earlier.
The
network
performance
of
MPI
can
be
measured
using
a
suite
of
benchmarks,
a
standard
one
of
which
is
called
IMB
or
the
Intel
MPI
benchmarks.
A
B
A
A
A
A
A
We
also
wanted
to
explore
the
effect
of
pod
organization.
What,
if
we
put
all
the
processes
in
our
parallel
workload
into
one
pot
versus
what
do
we
use
more
pods
with
fewer
processes
in
each
within
a
pod?
Parallel
processes
will
communicate
by
using
shared
memory
between
pods.
They
should
use
the
local
networking,
but
within
the
host
between
hosts,
they
will
use
the
ethernet
networking
and
the
cni.
A
A
A
In
all
cases,
we
used
48
parallel
processors,
a
low
lower
run
time
on
this
graph
means
better
performance.
Disappointingly,
there's
no
obvious
effect
to
the
different
pod
configurations.
Our
lines
are
pretty
flat.
Perhaps
MPI
was
smarter
than
we
thought
and
was
using
the
same
shared
memory
transport
both
within
pods
and
between
pods
on
the
same
node.
A
What
is
clear,
depending
on
the
cni
and
protocol
used,
our
simulation
took
between
125
seconds
and
407
seconds
to
complete
Network
performance
is
clearly
a
critical
factor
for
parallel
simulations
such
as
this
highest
performance
was
with
accelerated
ovn
and
Legacy.
Sriov
accelerated
ovn
yielded
about
125
seconds
of
runtime
once
again,
RDMA
protocol
data
points
are
marked
with
pluses
and
TCP
data
points
are
marked
with
crosses
one
interesting
thing.
A
We
can
see
the
hardware
accelerated
cnis
that
supported
RDMA
protocol
have
a
clear
advantage
over
everything
else,
but
when
the
same
cnis
run
with
TCP
instead
of
RDMA,
the
application
performance
drops
from
125
seconds
runtime
to
244
seconds.
That
is
almost
the
same
runtime
as
achieved
with
Calico
the.
A
This
application
harnesses
the
machine
learning
power
of
a
GPU
about
a
terabyte
of
signal
data
is
processed
for
each
genome
sequenced
using
open
source
code
from
Oxford
nanoport.
We
created
a
kubernetes
deployment
where
the
base
calling
process
is
presented
as
a
service.
We
wanted
to
see
what
effect
this
would
have
on
sequencing
performance.
A
A
Unfortunately,
for
lack
of
time
and
due
to
various
integration
issues,
we
only
managed
to
run
the
base
calling
Benchmark
on
host
networking
and
the
Calico
cni.
However,
what
we
can
see
from
this
single
comparison
is
encouraging.
There
was
no
obvious
impact
to
Performance
for
using
kubernetes
service
networking
for
base
calling,
indeed,
the
base
cooling
rate
was
actually
about
one
percent
higher
when
put
behind
a
service
IP
in
Calico,
which
is
hard
to
explain,
but
within
the
margin
of
variability
of
The
Benchmark
performance.
A
In
summary,
we
found
that
kubernetes
platforms
perform
well
for
HBC
and
AI
application.
Workloads
at
the
small
to
moderate
scale
of
our
test.
Lab
kubernetes
would
make
a
convenient
job
execution
framework
when
coupled
with
interactive
software
platforms
like
Jupiter,
and
this
doesn't
have
to
be
a
compromise
of
convenience
for
infrastructure
without
sdn
acceleration,
our
experiments
showed
Calico
performed,
particularly
well.
Although
it
is
hard
to
separate
out
the
uplift
of
all
the
different
offloads
and
accelerations,
a
modern
high-end
network
card
will
perform
further.
A
Our
results
also
showed
that
with
Hardware
accelerated,
networking
performance
of
kubernetes,
both
on
bare
metal
hosts
and
in
openstack
VMS,
can
be
almost
indistinguishable
from
performance
on
bare
metal.
With
the
right
networking
in
place,
we
can
harness
the
advantages
of
kubernetes,
either
on
bare
metal
or
nvms,
without
sacrificing
performance.
A
Finally,
I
would
like
to
say
a
huge
thanks
on
behalf
of
eres
and
myself
to
the
Nvidia
and
stack
HBC
engineering
teams
who
performed
This
research
and
made
this
presentation
possible.
These
people
are
heroes.
We
hope
you've
enjoyed
it
and
found
it
useful.
Please
do
feel
free
to
scan
the
QR
code
here
and
leave
us
some
feedback
thanks
very
much.