►
From YouTube: Scalable and Multitenant Networking in XDP and Kubernetes Operators - Sherif Abdelwahab & Ying Xiong
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2021 Virtual from May 4–7, 2021. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Scalable and Multitenant Networking in XDP and Kubernetes Operators - Sherif Abdelwahab & Ying Xiong, Futurewei
Enabling multi-tenant and extensible networking in Kubernetes is of paramount importance for cloud providers requires traffic isolation across tenants. By introducing a new data-plane built using XDP and inter-working custom resource operators, we will discuss how to enable multi-tenant networking at scale without compromising packet processing performance. Sherif and Ying will discuss the use of cases of networking isolation and fast provisioning within the Kubernetes environment for large scale cloud. They will share their experience and design techniques building a networking solution for multi-tenant networks from the ground-up using XDP, CRDs, and Kubernetes Operators. They will conclude with a demo.
https://sched.co/ekCi
A
A
A
With
that,
let's
get
started.
This
is
the
agenda
for
today's
talk.
I
will
give
a
introduction
and
some
background
on
the
project
why
we
are
doing
this
and
what's
the
context,
I
will
also
talking
about
current
network
model
in
kubernetes
and
introduce
you
the
high
level
design
of
our
new
multi-tenant
model
for
kubernetes.
A
We
have
a
forked
version
of
kubernetes
called
actors.
You
can
consider
actors
as
a
multi-tenant
kubernetes.
I
will
introduce
actors
more
in
next
slides
after
that,
sheriff
will
give
you
how
will
implement
the
multi-tender
network
model
in
kubernetes
and
introduce
you.
The
misa
project
misa
is
a
virtual
network
solution
that
has
both
control
plane
and
a
data
plane
based
on
the
xdp
technology.
A
So
misa
project
is
a
part
of
an
umbrella
project
called
centaurus.
Centerra's
project
includes
two
independent
projects.
One
is
called
actors,
one
is
called
visa.
Misa
again
is
what
we
discussed
today
in
this
talk,
however,
I'd
like
to
briefly
introduce
actress
project
first,
so
that
you
understand
some
of
the
contents.
A
As
I
mentioned
earlier,
actors
project
is
a
focused
version
of
kubernetes
with
a
major
change
in
design.
There
are
three
goals
for
actors
project:
first,
we
want
to
unify
vmware
campaign
orchestration
and
the
runtime
for
that.
We
explain
product
path,
definition
and
then
make
the
runtime
agent,
which
is
kubernetes
to
be
a
unified
agent
that
supports
both
vm
and
containers.
A
A
A
So
by
default
there
is
no
water
tendon
for
network
perspective,
kubernetes
introduce
network
policy
to
isolate
container
apart
each
other.
However,
network
policy
is
not
secure
or
as
not
strong
isolated,
as
virtual
network
can
do.
For
example,
network
policy
does
not
prevent
packet
sleeping
put
in
somewhere
where
the
traffic
is
passing
by
and
extract
the
information
intensity
of
the
information
or
data
out
of
the
packet.
A
A
So
in
actors
we
introduce
a
network
object,
a
new
crd
object
that
represents
a
vpc
or
subnet.
Each
part
to
be
created
has
to
be
associated
with
a
network
object
and
each
never
object
has
its
own
ip
address
space,
so
paths
or
containers
created
in
a
different
network
are
naturally
isolated
with
the
network
boundary.
A
A
The
new
navigator
objects
will
introduce
is
the
abstraction
of
network
resources,
but
not
actual
implementation.
Someone
still
needs
to
actually
create
vpcs,
create
a
subnet
and
manage
the
api
address
spaces
and
the
road
network
traffic.
This
is
where
misa
comes
into
play.
Misa
is
one
of
the
network
implementation
for
multi-tenant
network
model
in
kubernetes.
A
B
Hi
everyone,
this
is
sharif.
I
led
the
development
of
the
mizar
project
and
I
am
now
a
software
engineer.
With
microsoft,
we
built
mizor
from
the
ground
up
to
accelerate
pulse
network
provisioning
at
scale
actually
to
rethink
cloud
networking
in
missouri
all
together
to
build
it
in
the
exact
same
way
as
we
build
distributed
systems
in
the
cloud
to
make
cloud
networks
simple
to
understand
and
simpler
to
operate
in
the
rest
of
this
talk,
I
will
walk
you
through
our
thought
process
and
how
mazar
works
from
a
high
level.
B
B
B
On
the
met
on
the
data
plan
side
mizar
consists
of
a
set
of
multiple
xdp
programs,
the
processed
nodes
packets.
I
will
detail
exactly
how
the
xdb
program
processes
the
packets
later
in
this
architecture.
We
rethink
the
data
plan
programming
model
to
scale
the
management
plan,
accelerate
pub
provisioning
and
develop
customized
logic
for
network
services,
as
a
result
in
mazar
enables
scalable
and
multi-tenant
kubernetes.
Networking.
B
Before
I
detail
how
misar
works,
I
would
like
to
discuss
the
limitations
of
flow-based
network
programming.
Load-Based
network
programming
is
the
de
facto
programming
model
in
virtual
switches,
including
open
v-switch.
I
will
take
ovn
and
ovs.
As
an
example,
ovn
uses
the
concept
of
logical
ports
to
create
a
large
logical,
switch
that
spans
multiple
hosts
with
this
model,
creating
10,
000
logical
ports
generates
more
than
40
000
support
bindings.
B
B
Moreover,
during
flow
programming,
it's
not
uncommon
to
observe
an
increase
in
the
cpu
utilization
during
flow
parsing
with
the
logical
switch
architecture.
The
time
to
provision
network
resources
for
each
new
container
depends
on
the
number
of
containers
that
already
exist
in
the
system
and
the
number
of
worker
nodes
of
the
cluster.
So
clearly
the
logical
switch
approach
restricts
scale
and
is
not
suitable
for
dynamic
cloud
applications
that
have
a
short
lifetime
span.
Like
serverless.
B
With
the
limitations
of
flow
programming
model
in
mind,
we
redesigned
the
host
networking
in
mizor
to
interconnect
containers
only
with
xdb
programs,
and
to
do
this,
we
attach
an
xdp
program
on
each
physical
interface
of
a
worker
node.
We
name
this
program,
transit,
xdp,
this
program
processes
all
english
packets
to
the
worker
nodes.
B
B
The
operator
programs
logical
functions
of
the
xdb
program.
Through
these
rpc
interfaces
to
understand
the
rule
of
virtual
function.
We
need
to
look
into
the
new
network
organization
of
kubernetes.
That
mizar
enables
we
extended
kubernetes
with
two
resources
that
we
typically
find
in
any
multicannon
cloud
system.
B
B
Unlike
logical
routers
or
switches,
the
bouncers
and
dividers
are
in-network,
distributed
hash
tables
and
I
will
detail
exactly
how
they
work
in
the
next
few.
Slides,
bouncers
and
dividers
are
the
logical
functions
that
make
up
the
vpcs
to
isolate
pulse
traffic
for
multi-tenant
and
allow
tenants
to
reuse
the
same
network
address
space.
B
The
user
creates
a
vpc
like
any
object
in
kubernetes
as
a
yaml
file.
The
user
specifies
the
side
range
of
the
vpc
and
the
number
of
vpc
dividers,
because
the
divider
is
a
distributed
hash
table.
We
can
provide
any
number
of
dividers
in
the
object
definition
with
one
being
the
default
when
the
vpc
operator
receives
the
vpc
object.
It
schedules
the
vpc
divider
on
one
of
the
worker
nodes
of
the
cluster
scheduling.
The
divider
does
not
mean
that
the
operator
runs
any
new
code
on
the
node.
B
B
We
also
provide
any
number
of
pouncers
to
create
for
each
network
with
one
being
the
default
when
the
network
operator
receives
the
subnet
object,
it
schedules
the
bouncers
on
some
of
the
worker
nodes
of
the
cluster
scheduling.
The
bouncer
involves
two
actions:
first,
the
operator
labels
the
host
as
bouncers
on
the
management
plan.
B
B
The
missile
operator
uses
these
annotations
to
provision
the
pod
within
the
requested
vpc
and
network
boundary.
The
missouri
operator
provisioned,
the
network
resources
with
a
constant
number
of
rpc
calls.
Typically
too.
The
number
of
rpc
calls
does
not
depend
on
the
number
of
worker
nodes
in
the
cluster
or
the
pulse
already
provisioned.
B
B
B
Compare
this
to
obvious,
which
does
not
scale
well,
as
we
add
more
nodes
in
the
cluster
or
if
the
number
of
poles
existing
already
increases
up.
Until
this
point,
I
describe
the
management
plan
operations
and
in
the
next
few
slides,
I
will
describe
in
detail
how
the
xdb
program
on
holds
process
packets,
consider
the
case
in
which
a
pod
with
ip
address
1001
on
host
a
sends
a
packet
to
a
pod
with
an
ip
address.
B
B
The
xdb
program
looks
up
the
inner
destination
address
of
an
eppf
map.
Then
it
rewrite
the
outer
destination
ip
address
to
host
c,
which
is
the
worker
node
running
the
destination
pod
1002,
when
the
packet
arrives
at
host
c,
the
xdp
program,
decapsulates
the
packet
and
redirected
to
the
vehicle
of
the
pod
to
receive
it.
B
B
I
will
now
describe
how
we
solve
this
entirely
in
xdp
overcome
the
extra
hub
problem.
We
modified
the
xdb
program
running
on
the
bouncer
host
to
respond
to
arp
queries,
since
we
already
have
the
pulse
ip
and
mac
address
configured
by
the
missouri
operator
when
we
provision
the
pulse
network.
So
it
makes
sense
to
respond
to
our
queries
at
this
stage
when
the
pod
at
host
a
sends
an
arp
query.
The
bouncer
responds
with
the
mac
address
of
1002,
but
it
does
not
only
respond
to
art
queries.
B
B
B
B
There
are
several
decisions
that
the
bouncer
can
make
right
now,
including
rewriting
the
inner
destination
ip
address
and
sending
the
packet
to
a
back-end
pod
like
any
conventional
net
or
load
balancer
device.
But
I
will
describe
a
different
approach.
The
bouncer
instead
has
the
geneve
option
of
its
decision.
B
It
instructs
the
pulse,
transit
agent,
how
the
service
should
modify
the
inner
packet
in
the
clone
example.
The
modification
option
says
to
rewrite
the
service
ip
address
to
1004
and
port
80..
Then
the
bouncer
returns
the
packet
to
the
sender
host,
which
is
host
a
the
transit
xdp
program
on
the
client's
host.
Add
this
information
option
in
any
in
an
eppf
map
entry,
then
it
rewrites
the
packet
according
to
the
accordingly
and
rescinds
the
packet
again,
but
this
time
not
to
the
service
ip,
but
at
the
back
end
pod
n004.