►
From YouTube: Calico/VPP: Using Calico’s Pluggable Dataplanes for Fun and Fast N... Aloys Augustin & Chris Tomkins
Description
Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Calico/VPP: Using Calico’s Pluggable Dataplanes for Fun and Fast Networking - Aloys Augustin, Cisco & Chris Tomkins, Tigera
The Calico VPP dataplane, introduced during last year’s KubeCon, adds a new networking option for Calico on Linux alongside iptables and eBPF. This takes advantage of Calico’s pluggable dataplane support, which offers futureproofing and agility for the user in dataplane choices. As it turns out, having VPP’s flexible and modular userspace networking stack in a Kubernetes environment enables all sorts of interesting things -- such as allowing network-intensive pods to send and receive data at ludicrous speeds while conserving the CNI features we love, thanks to kernel bypass. This is useful for any workload that spends most of its time doing network I/O. Think proxy, content delivery server, or network functions such as VPN endpoints. In this talk, we will show you how we achieve that, the level of performance that we can reach, and how it compares to other approaches.
A
A
I've
been
working
in
networking
since
about
2000
and
after
a
few
years
I
realized
that
a
per
device
cli
is
not
a
great
solution
for
a
large
environment.
So
I
took
an
early
interest
in
infrastructure
as
code
and
automation
and-
and
I
still
pursue
those
today
when
I'm
not
working
in
it.
I
love
reading
films,
music.
I
also
fly
radio
controlled
gliders,
which
is
a
strange
coincidence,
as
our
other
speaker
today
alois
augustine
who's.
A
software
engineer
at
cisco
also
does
like
flying
radio
control
planes.
A
A
A
But
calico
also
follows
that
model
software
modularity
is
the
concept
of
the
dividing
software
code
intentionally
into
blocks
that
work
together
to
create
a
larger
hole
and
software
loose.
Coupling
is
the
concept
of
making
these
blocks
opaque
to
each
other
so
that
they
can
only
talk
via
strictly
defined
interfaces
together.
These
concepts
make
the
software
easier
to
fix,
troubleshoot
and
replace,
and
in
the
case
of
calico,
these
benefits
extend
to
following
the
network
paradigm
of
of
a
control
plane
and
a
data
plane.
A
So
you
can
see
that
I've
highlighted
in
blue
here
the
control
plane
elements
of
the
four
tiers
that
calico
are
made
up
of.
So
the
control
plane
elements
are
the
data
model
which
defines
the
custom
resources
used
to
enable
calico
and
the
felix
calculation
graph,
which
is
the
control
plane.
That
makes
the
smart
decisions
and
then
the
data
plane
components
are
the
data
plane
driver,
which
is
a
simple
and
minimal
driver
for
programming,
the
data
plane
and
the
data
plane
itself,
which
is
responsible
for
actually
forwarding
packets.
A
A
A
It
manages
complex
device
and
network
configuration
and
state
and
if
you're
familiar
with
routing
protocols
such
as
bgp
ospf
and
isis
they're
implemented
in
the
control
plane.
So
it's
necessarily
sophisticated
software.
There's
a
lot
going
on
there
and
that's
why
it
runs
on
a
general
purpose
cpu,
which
is
appropriate
for
running
workloads
that
can
vary
in
software
complexity.
A
The
data
plane,
in
contrast,
is
a
different
beast.
The
data
plane's
job
is
actually
to
forward
the
traffic,
so
I've
put
cat
videos
in
the
diagram
because,
let's
be
honest,
that's
90
of
the
internet-
and
you
can
see
that
the
cat
videos
only
travel
through
the
data
plane.
They
don't
travel
through
the
control
plane
nodes.
A
B
A
Mpls
has
been
enormously
successful
in
service
provider
networks
and
that's
because
it
purges
unnecessary
functionality
from
the
data
plane
and
performs
minimal
packet
processing
per
packet
and
that
leads
to
fast,
affordable
devices.
So
mpls
the
designers
looked
at
what
was
strictly
necessary
and
some
of
the
functionality
that's
in
ip,
such
as
variable
length,
subnet,
masks
and
checksums,
which
adds
a
great
deal
of
complexity,
are
not
strictly
needed
in
in
the
mpls
network.
A
So
there
are
some
benefits
of
control,
plane
and
data
plane
separation.
The
first
is
that
you
get
specialized
minimal
data
plane
code,
so
you're
keeping
the
data
plane
code
as
small
as
possible.
It's
lean
and
that
carries
many
many
benefits.
For
example,
if
you
have
only
the
minimal
set
features,
then
each
line
of
code
can
be
groomed
in
more
detail
from
a
security
and
a
performance
standpoint
to
make
sure
that
it's
strictly
necessary
makes
the
code
easier
to
audit
and
easier
to
keep
performance
high.
A
Also,
by
having
a
targeted
data
plane
feature
set,
then
not
every
deployment
is
the
same.
You
might
find
that
one
deployment
needs
compatibility
with
older
kernels
or
another
deployment
needs.
High
cryptographic
throughput
and
having
a
targeted
data
plane
feature
set,
allows
you
to
use
a
different
data
plane
to
solve
each
use
case.
A
The
next
advantage
is
future
proofing.
We've
all
seen
technologies
that
are
going
to
be
the
next
big
thing,
but
it
doesn't
quite
work
out.
My
favorite
example
is
trill,
which
is
good
for
transparent
interconnection
of
lots
of
links
and
back
in
about
2014.
This
was
going
to
be
the
next
big
thing
in
networking,
but
it
it
never
came
to
fruition
by
supporting
multiple
data,
plane,
technologies,
calico.
A
If
a
new
data
plane
technology
appears
on
the
horizon,
then
the
calico
community
is
able
to
quickly
get
a
working
test
and
to
evaluate
whether
this
is
something
we
want
to
pursue
further
and
the
final
one
is
agility
for
the
user.
If
you
have
multiple
data
plane
choices,
then
you're
not
locked
to
a
particular
data
plane.
A
You
can
make
the
right
choice
today
and
maybe
the
right
choice
in
the
future
remains
the
same
or
maybe
it
changes,
and
you
can
continue
to
deploy
a
data
plane,
that's
appropriate
for
your
needs,
so
we
we're
nearly
ready
to
talk
about
vpp
and
what
vpp
can
offer
in
this
scenario.
But
before
we
do,
we
should
just
talk
about
the
three
other
data
pane
choices
that
we
have
today.
So
we
have
the
the
linux
ib
tables
data
plane,
which
is
our
original
data
plane,
it's
heavily
battle
tested.
A
The
windows
host
networking
service
data
plane,
which
offers
windows
container
deployment
and
securing
those
workloads
and
it
works
in
any
cloud
computing
provider
or
on-prem.
And
finally,
we
have
the
linux
eppf
data
plane,
which
scales
to
higher
throughput
uses.
Less
cpu
per
gigabit
reduces
first
packet
latency
to
services.
A
It
preserves
external
client
source
ip
addresses
all
the
way
to
the
pod,
and
it
supports
direct
server
return
for
better
efficiency.
If
you
want
to
hear
more
about
any
of
these
data
planes,
we've
had
pre-existing
talks
that
you
can
search
for
which
discuss
all
of
these.
If
you
look
for
my
name,
chris
tompkins
and
ebpf
you'll
see
those
talks,
but
today
we're
not
talking
about
these
data
planes.
We're
talking
about
this
fourth
option,
which
we're
really
excited
about
which
is
vpp
and
what
that
can
offer
with
calico.
B
Hi
everyone.
Thank
you.
Chris.
Before
we
dive
into
the
data
plane,
you
may
be
wondering
what
vpp
is
and
it's
been
presented
quite
a
few
times
already,
so
I
won't
spend
too
much
time
on
it,
but
in
short,
it's
a
user
space
network
data
plane.
So
it's
a
software
router
and
it's
highly
optimized
both
for
packet
processing
and
at
the
api
level
as
well.
It
supports
all
kinds
of
hardware
and
it's
easily
extensible
through
plugins
and
that's
something
we
used
in
the
in
the
calico
integration.
B
It
leverages
vectorization
to
maximize
the
cpu
instruction
and
data
cache
efficiency
and
that's
leading
to
some
pretty
impressive
performance
number
as
shown
here.
So
these
numbers
were
obtained
on
standard
two
socket
ice
lake
servers
and
doing
ipsec.
We
can
get
one
terabit
per
second
of
encryption
and
when
doing
simple
forwarding,
we
can
reach
1
billion
packets
per
second.
B
If
you'd
like
to
learn
more
about
it,
there
are
plenty
of
presentations
available
out
there
that
you
should
be
able
to
find
quite
easily
now
for
the
calico
vpp
integration.
What
we
do
is
that
we
use
vpp
as
a
data
plane
for
the
pod
traffic
in
calico
when
you
deploy
calico
vpp
on
the
kubernetes
cluster
you'll
get
one
vpp
instance
on
each
node
that
processes
the
container
traffic.
It
will
implement
the
cni
meaning
it
will
create
and
configure
the
interfaces
within
the
pods.
B
It
will
also
route
this
traffic,
of
course,
and
implement
all
the
specific
features
required
by
kubernetes,
so
that
includes
policies,
virtual
ap
load,
balancing
potentially
source
netting,
the
outgoing
traffic
and
all
the
other
calico
features.
There's
the
encapsulation
as
well.
Of
course
it
can
do
ipip,
vxlan
wire
guard
encapsulation,
and
all
this
logic
is
done
in
dedicated
vpp
plugins,
which
are
optimized
for
this
use
case.
B
So
if
you
want
to
deploy
calico
vpp
on
a
cluster
in
addition
to
the
calico
configuration,
the
only
thing
you
need
is
to
pass
an
interface
name
to
vpp,
that
it
will
use
as
the
onpc
interface
and
everything
else
will
be
taken
care
of
automatically.
You
should
have
a
working
configuration
out
of
the
box.
B
B
We
are
using
maglev
as
the
load
balancer
implementation
for
kubernetes
services.
Maglev
is
a
constituent,
consistent
layer,
4
load
balancer
and
basically
it
allows
connections
to
survive,
node
failures
and
it
also
does
dsr
direct
server
return,
which
will
reduce
the
global
networking
load
on
the
cluster.
If
you
enable
it,
a
third
area
where
vpp
is
very
helpful
is
if
you
have
custom
networking
requirements.
B
B
What
I
mean
by
network
intensive
applications
is
applications
that
spend
most
of
the
time
doing:
network
io
and
potentially
crypto
for
encrypted
protocols
such
as
tls
and
quick.
There
are
various
applications
that
fall
into
this
category
that
could
be,
for
instance,
proxies
distributed,
storage,
applications,
content,
servers,
vpn
endpoints
network
functions
and
so
on
with
these
applications.
At
some
point,
the
network
stacks
that
you
get
in
the
kubernetes
with
this
interface
to
the
pod
becomes
a
bottleneck
and
the
usual
solution
is
to
bring
a
physical
interface
directly
into
the
pod
with
sr
iov.
B
B
So
to
understand
where
these
optimization
opportunities
are
coming
from.
First,
let's
go
back
to
the
regular
calico
networking
analog.
So
this
diagram
here
is
what
you
would
get
with
the
ip
tables
or
the
ebpf
data
planes.
Basically,
applications
are
using
the
linux
kernel
syscalls
to
send
and
receive
data.
Then
this
data
is
processed
and
converted
into
packets
by
the
kernel
transport
stack,
and
then
it
goes
out
of
the
pod.
B
B
However,
if
we
have
very
high
network
requirements-
and
if
we
are
looking
for
performance
optimizations,
can
we
do
better
than
this?
The
answer
is
obviously
yes
and
in
theory,
it's
simple.
We
should
just
send
the
data
directly
from
the
app
to
vpp
and
bypass
the
kernel
entirely
and
save
at
least
two
data
copies.
B
In
addition
to
all
the
processing
that
the
kernel
does
in
practice,
though,
it's
going
to
require
a
bit
of
effort
to
do
so,
but
we've
tried
to
make
it
as
easy
as
possible,
so
we
have
actually
two
different
options:
targeted
at
different
kind
of
applications.
So
we'll
make
a
distinction
here
between
two
categories
of
network
intensive
applications,
the
first
one
will
be
packet
processing
applications
which
could
also
be
called
network
functions.
B
B
So
these
are
more
your
traditional
applications,
such
as
proxies
content,
server,
storage,
servers
and
so
on,
and
these
applications
don't
care
about
packets.
What
they're
interested
in
is
tcp
connections
or
maybe
even
tls,
are
quick
if
they
encrypt
their
data
and
for
these
applications
we
expose
the
vpp
host
stack
in
the
pods
before
we
dive
into
the
specifics
of
memoif
and
the
host
stack.
One
thing
I'd
like
to
mention
is
that
if
you
enable
either
in
a
pod,
then
the
pod
will
still
have
its
regular
tuned
app
interface
available.
B
This
is
useful
because
it
allows
to
have
some
containers
or
applications
in
the
pod
that
can
still
communicate
with
the
outside,
normally
without
requiring
any
changes.
So
this
is
useful,
for
instance,
for
the
supporting
processes
for
log
collection.
Maybe
configuration
things
like
that,
so,
let's
dive
first
into
the.
If
interfaces
what
are
mifs,
they
are
very
simple,
highly
optimized
software
interfaces.
B
They
relied
on
a
shared
memory
segment.
They
allow
to
exchange
packets
between
two
processes.
Just
a
very
simple
process,
just
sending
or
receiving
packets
on
the
mmif
interface
can
transfer
more
than
15
million
packets
per
second
on
a
single
queue
with
a
single
core
and
that's
around
one
order
of
magnitude
more
than
what
you
can
normally
get
in
a
kubernetes
pod,
calico
vpp
can
make
one
maf
interface
available
in
the
pod
and
then
the
application
running
the
pod
has
a
few
options
to
use
this
interface.
B
B
The
first
one
will
be
go
lib
mayf,
which
is
the
seagull
to
rob
the
c
library,
but
I
would
recommend
you
to
use
the
second
one,
which
is
called
go
maf,
which
is
a
pure
go
implementation,
but
there's
a
third
option
as
well,
which
does
not
require
writing
any
code
and
that's
to
run
vpp
in
the
pod.
Vpp,
of
course,
support
memif
and
it
can
be
configured
to
easily
implement
various
network
functions,
and
the
nice
thing
is
that
it
won't
need
elevated
privileges
to
connect
to
mmif
interface.
B
Here
is
an
example
of
a
trivial
application
using
gom-if
that
you
could
run
in
a
pod.
As
you
can
see,
the
api
to
send
and
receive
packets
on
an
interface
is
quite
simple:
there's
a
read
packet
and
right
packet
method
and
you
get
the
packets
in
byte
slices.
This
is
a
trigger
example
that
just
swaps
the
source
and
destination
ip
addresses
in
practice,
you'll
likely
want
to
do
more
elaborate
things
on
each
packet,
but
you
get
the
idea.
B
B
One
thing
to
note
is
that,
since
there
is
also
a
tune
interface
that
can
send
traffic
to
the
pod
vpp
when
it
receives
a
packet,
it
needs
to
know
where
to
send
the
incoming
packet
either
to
the
tuner
to
the
mmf,
and
this
is
done
based
on
destination
ports.
You
can
specify
in
the
annotations
which
interface
will
be
the
default
interface,
either
the
tune,
type
or
the
mif,
and
then
for
the
other
one.
You
can
specify
a
few
ports
or
ranges
of
parts
which
should
go
to
that
interface.
B
B
The
host
stack
has
an
api
for
each
of
these
protocols,
which
is
based
on
a
shared
memory
segment
just
like
memoif,
and
we
can
make
that
api
available
in
the
pod
in
order
to
use
it.
The
application
running
in
the
pod
should
use
the
vpp
communications,
library
or
vcl.
In
short,
the
vcl
is
a
c
library
that
provides
a
relatively
high
level
api
for
the
host
stack
and
it's
pretty
close
to
the
traditional
bsd
socket
api,
which
makes
it
quite
easy
to
use
here
is
a
summary
of
the
vcl
api.
It
should
look
fairly
familiar.
B
B
B
B
B
B
B
That's
it
for
the
host
stack.
There
are
a
few
other
things
to
know
that
apply
to
both
math
and
the
host
stack.
The
first
thing
is
that
for
all
the
traffic
that
goes
through
either
mma
for
the
whole
stack,
then
the
regular
kubernetes
networking
features
will
apply.
This
means
policies
will
work
as
you
expect.
This
load
balancing
will
work
as
well
if
you
connect
to
a
service,
ip
etc.
B
It's
split
into
two
sections
for
the
endpoint
connection,
oriented
applications
and
for
the
packet
processing
applications
for
endpoint
applications.
We
don't
have
definitive
numbers
yet
for
calico
vpp.
This
is
still
a
work
in
progress,
but
we
expect
the
whole
stack
to
provide
a
substantial
performance
boost.
B
B
That's
according
to
the
very
early
testing
that
we
did.
But
again,
this
is
still
a
work
in
progress.
It
will
require
more
testing
to
get
real
numbers
and
scanning
characteristics
and
without
calico
vpp.
If
you
would
like
to
get
the
same
features
today,
well,
then
the
application
could
use
an
af
packet
or
af
xdp
socket
under
this
interface
in
the
pod.
B
B
This
presentation
was
focused
on
high
performance
networking
apps
we've
shown
that
it's
possible
to
run
them
in
a
regular
kubernetes
cluster,
but
it's
not
just
about
performance.
Vvp,
really
unlocks
a
lot
of
possibilities
in
terms
of
what
it's
possible
to
do
in
kubernetes,
with
a
lot
of
velocity
as
well,
and
we
are
really
excited
for
all
the
things
we'll
be
able
to
do
with
it.
B
A
Thanks
a
louise,
it's
fantastic
tech.
So
in
summary,
this
is
a
new
vpp
based
user
space
data
plane
for
calico.
It
brings
networking
features
that
were
previously
only
used
by
service
providers
to
kubernetes
and
allows
the
managing
of
high
availability
and
high
performance
networking
apps
in
kubernetes.
A
So
that's
nearly
it
for
this
presentation.
We
have
a
number
of
new,
exciting
features
on
the
horizon,
including
maglev
load,
balancing
packet,
oriented
interfaces
and,
hopefully
soon
general
availability
in
calico.
Currently,
the
project
is
expected
to
move
from
tech
preview
to
beta
status
in
v321
or
v322
of
calico.
A
To
join
the
vpp
channel
in
the
calico
users
slack,
we
publish
our
releases
there
and
the
calico
uses.
Slack
is
a
really
active
community
for
calico
generally,
if
you'd
like
to
try
it
out
head
over
to
the
calico
documentation,
which
has
set
up
instructions
for
vbp
and
all
the
other
data
planes.
And
if
you
have
any
questions
at
any
point,
don't
hesitate
to
ping
us
on
the
slack
channel
as
well,
or
you
can
ask
them
straight
away
thanks
very
much
for
listening
and
taking
the
time.