►
From YouTube: 100Gbit/S Clusters With Cilium: Building Tomorrows Networking- Daniel Borkmann & Nikolay Aleksandrov
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2023 in Amsterdam, The Netherlands from April 17-21. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
100Gbit/S Clusters With Cilium: Building Tomorrow’s Networking Data Plane - Daniel Borkmann & Nikolay Aleksandrov, Isovalent
Speakers: Daniel Borkmann, Nikolay Aleksandrov
The vast majority of Kubernetes users today are running IPv4-only clusters or transitioning to dual stack deployments as an interim step on the journey towards IPv6-only clusters. The latter are not only an enabler for more IPAM flexibility and larger cluster scale, but also unlock new Linux kernel innovations in networking and eBPF to cater for data intensive workloads. In this talk, we examine what's possible once we arrive at IPv6-only clusters. We present recent advancements in Cilium's networking data plane enabling a low-latency architecture suitable for "BIG TCP"-based workloads requiring IPv6 for 100Gbit/s transfers and beyond for a single socket. In addition, we deep dive our path into achieving host networking performance characteristics for Pods through a new veth driver replacement we specifically developed for the kernel in context of Cilium. With the resulting eBPF forwarding architecture, most unneeded parts of the stack are bypassed, drastically improving networking.
A
So
when
looking
at
data
center
networks,
what
are
the
the
large
challenges
and
essentially
I
think
you
can
put
them
into
three
big
pillars:
scale,
performance
and
operations.
Probably
all
of
them
would
deserve
like
a
talk
on
its
own,
so
scale
scaling
out
into
many
nodes,
many
parts
performance,
making
the
best
use
of
the
resources
and
operations
making
changes
to
your
clusters,
frequently
without
disruptions,
for
example.
So
my
question
is
what,
if
IPv6
could
address
both
scale
and
performance
requirements
from
those
three
pillars?
A
So
that's
what
I'm
going
to
elaborate
in
this
talk
before
going
and
looking
into
more
into
the
future
I
actually
want
to
take
a
step
back
and
look
into
2016,
because
then
we
actually
first
announced
the
psyllium
experiment.
You
know
we
started
out
with
IPv6
only
container
networking,
together
with
EE,
BPF
and
xtp.
For
those
of
you
who
don't
know
what
ebpf
is.
It's
kernel
technology
that
makes
makes
the
kernel
programmable
customizable
in
a
safe
way,
so
yeah.
A
So
until
you
know,
we
were
really
excited
about
IPv6,
because
the
way
how
we
can
implement
the
data
plane
with
that
until
of
course,
reality
kicked
in
and
back
in
2016
the
state
of
IPv6
and
in
kubernetes
and
Docker
in
particular-
wasn't
quite
there
yet
so
yeah
if
we
had
to
basically
go
back
and
Implement
ipv4
support
upon
popular
demand,
which
probably
a
lot
of
people
are,
are
running
sodium
with
so
yeah
fast
forward
2020.
How
does
the
situation
look
like
today?
A
Kubernetes
has
adopted
IPv6
single
stack
for
quite
a
while
and
dual
stack
also,
and
the
hyperscale
has
also
made
progress
integrating
IPv6
into
the
environment
over
most
of
it
rather
dual
stack.
So
if
you
look
at,
for
example,
the
manage
kubernetes
offerings,
AKs
eks
gke,
they
all
offer
you
know
various
level
of
IPv6
support,
most
of
them
dual
stack.
There's
one
single
stack
so,
but
that's
already
great
in
terms
of
IPv6
like.
Why
do
you
want
to?
A
So
in
general
there
was
a
actually.
It
was
a
really
interesting.
Dual
stack
adoption
panel
at
the
last
kubecon
I
can
recommend
watching
that
in
general,
it's
right
now
it's
regarded
as
a
as
a
transitional
state.
So,
of
course,
in
the
future.
Looking
further
out,
we
want
to
go
to
IPv6
end
to
end
in
general,
in
order
to
avoid
the
complexity
of
having
both
you
know,
of
having
both
ipv4
and
IPv6
in
your
cluster.
A
So
the
typical
approach
that
we've
seen
users
today
they
are
trying
to
build
out.
You
know
islands
of
IPv6
single
stack,
on-prem
clusters.
You
know
as
a
clean
slate
and
then
trying
to
successfully
move
applications
and
services
into
their
cluster
yeah.
So
then
you
go
and
deploy
an
IPv6
yeah
an
IPv6,
only
cluster,
so
no
ipv4.
But
then
then
you
face
again
reality
and
you
will
have
to
interface
somehow
with
ipv4.
So
that's
unless
you're,
air-gapped
or
you're.
A
You
know
you're
really
lucky
with
dependencies
and
if
you
look
at
the
Alexa
1000
sites,
it's
a
bit
of
a
sad
State
because
you
know
like
roughly
half
of
the
sites,
support
IPv6
and
the
other
ones.
Not
so
it's
it's
really
not
quite
there.
Yet
you
will
hit
a
couple
of
ecosystem
bumps
on
the
road,
for
example,
when
looking
at
that
I
saw
that
GitHub,
which
is
very
frequently
used
by
developers,
not
IPv6
ready
yet
but
their
support
in
in
terms
of
supporting
that,
so
there's
work
in
progress
and
yeah.
A
So
in
a
modern,
V6
only
environment.
How
do
you
deal
with
ipv4?
So
you
will
have
to
run
somehow
not
four,
six
or
six
four.
So,
in
order
to
translate
between
the
two
worlds,
so
looking
at
not
for
six
and
six
four
with
Linux,
it's
actually
not
possible.
So
there's
no
such
thing
in
net
filter
because
simply
dealing
with
kernel
is
really
complicated
and
it
has
a
lot
of
protocol
information
in
the
packet
itself.
But
we
added
a
support
for
this
in
ebpf,
so
ebpf
environment
is
a
bit
more.
A
Is
a
bit
more
constrained
so
they're,
you
know
it's
it's
less
complex
to
be
able
to
do
that
there
and
so
now
yeah.
So
how
is
now
the
question
is
how
do
you
Ingress
before,
like
if
you
have
to
before
clients
somewhere
on
the
internet,
and
you
want
to
connect
to
your
V6
only
cluster?
How
do
you
approach
that?
So
the
first
option
is
to
use
sodium
with
a
stateful,
not
four
six
Gateway
and
the
latest
typically
looks
so
on
the
left
side.
You
see
the
an
external
node
making
DNS
requests.
A
You
get
back
the
a
record
with
the
ipv4
address,
then
just
issues
the
syn
packet.
You
have
to
go
to
a
dual
stack
component,
so
that
will
be
like
you're,
not
for
six
Gateway,
which
will
map
the
ipv4
address
import
to
some
IPv6
address
import,
and
then
you
can
go
into
your
IPv6
single
stack,
kubernetes
cluster
and
there
you
maybe
have
a
kubernetes
slope,
balancer
service
with
that
front-end
IPv6
address,
and
then
you
can
talk
to
it
and
at
that
front-end
address
will
have
backends
that
we
can
manage
right
so
yeah.
A
So
this
has
a
couple
of
upsides,
so
the
ipv4,
Whip
and
Port
is
completely
decoupled
from
your
kubernetes
cluster
and
from
your
kubernetes
service
you
don't
need
a
special
load,
balancer
service,
you
don't
have
special
ipam
requirements
for
the
load
balancer
and
any
public
IP
IPv6
prefix
works
as
as
is
you
can
even
do
like
with
with
the
local,
like.
You
can
even
load
balance
through
multiple
kubernetes
clusters,
for
example
true
weighted
maglev
and
so
on.
So
that's
possible
they're.
A
bit
of
there
are
also
downsides
to
that.
A
So
you
need
to
have
like
a
control
plane
in
order
to
push
down
like
the
ipv4
to
IPv6
mappings
and
the
other
one
is
the
original
Source
IB
preservation
when
the
traffic
hits
the
kubernetes
cluster
is
lost
because
you
have
to
do
the
net
it's
stateful,
so
it
needs
to
do
dnat
and
s
net.
But
you
know
thanks
to
ebpf
and
you
can
push
it
into
the
xtp
layer
like
into
the
driver
layer.
You
can
actually
achieve
High
package
rates.
A
A
So
there's
like
a
special
prefix
when
you
translate
to
the
V6
and
then
you
need
to
have
this
like
a
load
balancer
service,
with
this
special
prefix
in
your
kubernetes
cluster
as
well
to
then
load
the
lens
to
the
back
ends,
and
then
you
have
the
advantage
to
not
have
state.
So
this
is
really
highly
scalable,
because
you
don't
hold
State
on
the
Node
on
the
on
the
Gateway
node.
A
It
allows
for
Source
address
preservation
because
you
know
the
original
client
Source
address
is
essentially
encoded
into
the
IPv6
address
and
you
can
use
things
like
Global
answer.
Source
ranges
to
also
filter
for
that
for
external
ipv4
clients
and
no
control
plane
is
needed
because
the
translation
Works
essentially
transparent,
but
the
downside
is,
of
course
you
need
to
have
like
a
like
a
local
answer.
Ipam,
which
understands
this
prefix
and
essentially
have
has
like
a
public
ipv4
accessible
address
in
there
and
yeah.
A
So
you
need
to
have
this
awareness
so
essentially
when
the
when
the
cluster
replies
back
to
the
client.
This,
of
course,
has
to
be
then
again
routable
in
the
internet,
so
it
just
needs
to
have
some
some
public
V4
address.
That
is
then
encoded
in
the
V6
so
now
like,
let's
bring
in
one
more
layer
of
difficulty.
So
this
was
maybe
like
more
of
this,
like
more
straightforward
part,
four
to
six
into
the
cluster.
The
other
layer
of
difficulty
is
like.
A
So
here,
DNS
624
plays
a
key
part
in
the
in
in
that,
for
example,
when
like
going
back
to
the
previous
example,
when
you
want,
when
you
want
to
do
a
DNS
lookup
for
for
GitHub,
you
get
the
ipv4
address,
but
when
you
try
a
quad,
a
record
it
it
doesn't
answer.
So
how
do
you
talk
to
it?
There's
a
six
like
dns-64
proxy,
for
example.
A
Here's
the
public
Google
one
and
it
will
return
you
an
address
where
it's
where
essentially
it
has
a
pv6
specific
prefix
and
the
remaining
bits
is
the
encoded
ipv4
address
in
there.
So
this
is
the
way
how
it
will
look.
So
now
you
have
on
the
left
side,
your
kubernetes
cluster,
on
the
right
side,
the
internet,
and
if
your
node
wants
to
talk
to
the
internet,
you
go,
you
have
to
go
through
this
proxy.
The
proxy
will
make
an
actual
a
request
through
DNS
to
the
internet.
A
It
will
receive
a
reply
and
it
will
then
encode
this
reply
in
a
quad,
a
record
back
to
your
cluster,
and
now
your
your
notes
in
the
cluster
can
do
a
send
request
to
to
this
address.
This
has
to
go
through
the
net
six
to
four
Gateway.
It
will
then
do
the
translation
and
it
will
then
go
back
to
ipv4
and
hit
the
actual
external
nodes
so
yeah.
So
this,
oh
okay,
yeah.
So
sorry,
so
yeah
yeah,
core
DNS,
for
example,
supports
this
kind
of
functionality.
A
If
you
want
to
use
this
in
your
in
your
environment,
so
the
upside
is
again
like.
If
you
have
a
stateless
translation,
this
is
again
highly
scalable
doesn't
hold
State
on
the
Node,
and
all
the
traffic
in
the
cluster
can
be
a
pure
V6
only
so
the
nodes
Parts,
even
the
Gateway
AP,
so
there's
no
dealing
with
ipv4.
A
The
downside
here
is
that
the
ipam
management
becomes
more
complex
because
the
parts
know
they
would
have
to
have
like
a
secondary
IP
address,
which
which
has
this
specific
prefix
like
the
not
for
well-known
prefix
in
there.
In
order
to
reply
back
so
in
order
to
overcome
this,
this
limitation,
one
at
one
thing
that
can
be
addressed
that
can
address
this
is
essentially
to
do
the
stateful
translation
on
the
way
out.
A
So
your
the
parts
reply
with
their
primary
address,
hit
the
Gateway
node,
do
the
translation
there
and
then
talk
to
the
talk
back
to
the
ipv4
client,
so
I
have
a
short
demo
for
the
for
the
four
to
six
six
to
four
Gateway.
Given
the
wireless
was
quite
bad
in
here,
I
did
some
pre-recording
the
other
day,
so
I'm
just
walking
you
through
that.
So
here
you
have
a
IPv6,
reachable
web
server
and
it
has
an
IPv6
address.
A
It
can
only
be
reached
there
and
now,
if
you
go
into
the
gateway
node
itself,
so
this
is
running
sodium
as
a
standalone.
You
will
see
now
that
it
has
an
ipv4
service
prefix
and
the
back
end
itself
is
the
actual.
You
know
like
the
the
web
browser
and
you
can
see
you
hit
the
you
hit
curl
on
ipv4
address
and
you
get
to
the
IPv6
in
here.
So
that's
the
one
case,
the
one
situation
and
now
the
other
the
other
way
around
they're,
not
six,
two,
four,
it's
quite
similar.
A
So
you
have
an
ipv4
reachable
address
and
it
does
not.
It
cannot
be
reached
to
IPv6.
Now
you
go
to
the
Standalone
Gateway
node.
You
have
an
IPv6
front
end
address
in
sodium
in
here,
and
this
is
all
done
through
ebpf
at
the
XDP
layer.
So
if
you
do
the
curl
it
will
do,
it
will
reach
the
ipv4
page,
so
yeah,
okay.
A
A
It
has
been
developed
by
by
Google
and
the
goal
from
from
Google
side,
but
also
you
know,
for
the
broader
Community
is
to
support
future
data
center
workloads,
for
you
know
single
sockets,
where
you
go
100
gigabit
per
second
200,
400
and
Beyond.
So
that
is
the
aim
for
that,
and
you
would
say
why
care
well,
if
you
have
a
big
data,
workloads,
AI
machine
learning
or
other
really
Network
intensive
workloads
that
have
bulk
transfers.
So
it's
really
useful
for
that.
But
even
if
you
don't
have
it,
it
will
free
up.
A
You
know
your
application
resources
so
that
you
can
use
that
instead
of
the
kernel
stack
when
you
have
to
spend
Cycles
there.
If
you
look
at
100
gigabit
per
second,
it's
really
constrained.
So
if
you
think
about
1500
MTU
you
would,
you
would
only
have
123
nanoseconds
per
packet
and
you
would
have
to
be
able
to
process
endpoint
around
8.15
million
packet
per
second,
so
that
itself
is
quite
unrealistic
for
the
kernel
stack.
A
So
what
you
can
see
here,
many
MTU
size
packets
are
coming
in
and
it
tries
to
aggregate
this
into
a
single
super
size
packet
up
to
64k,
and
it
will
push
it
up
to
the
stack
so
that
you
have
to
Traverse
the
stack
only
once
instead
of
multiple
times
and
the
same
way
also
works
on
the
way
out.
So
the
kernel
stack
typically
tries
to
aggregate
a
large
packet
and
then
on
your
Hardware
Nick.
It
will
typically
like
most
of
the
next
support
is
today.
It
will
then
chunk
them
up
into
individual
into
individual
packets.
A
There's
also
gso,
like
there's
the
software
equivalent
for
that
so
yeah.
So
once
you
like,
like
once,
the
Nick
chunks
them
up
into
packets,
usually
they
are
received
as
a
train
of
packets
like
for
TCP
streams,
and
then
the
go
will
will
pull
it
back
together.
So
there's
not
too
much
latency
where
go
would
have
to
wait,
but
there's
an
uppercise
limitation,
so
the
64k
is
really
the
upper
size
limitation
that
we
have
today.
A
The
reason
is
that
the
the
well,
the
the
the
packet
size
is
encoded
in
the
IP
address
total
length,
and
that
is
16
bit
and
therefore
it's
limited
to
64k
in
the
same
way,
of
course,
on
the
way
down.
So
can
we
make
bigger
batches
so
here
IPv6
is
to
the
rescue,
so
big
TCP
basically
overcomes
the
16-bit
limit,
we're
just
a
small
kernel
change,
so
it
will
insert
a
hop
by
hop
IPv6
extension
header,
and
this
is
just
local
to
the
node,
because
the
go
aggregation
is
it's.
A
A
It
will
have
a
jumbo
extension
there
and
there's
like
a
payload
length
field
for
32-bit,
so
it
can
theoretically
go
to
four
gigabit
as
a
packet
size,
which
is
of
course
unrealistic
today,
at
least,
but
that's
the
theoretical
limitation,
and
with
that
you
can
actually
bump
the
the
aggregation
size
for
go
and
also
TSO
like
for
the
way
in
and
for
the
way
out
right
now.
The
kernel
has
set
this
to
512k,
but
this
can
be
raised
in
the
future.
A
But
you
know
ipv4
is
stuck
essentially
with
64k
limit
and
we
implemented
support
for
this
feature
in
sodium
113.
That
is,
you
know,
released
approximately
like
towards
the
end
of
the
year,
so
there's
just
a
single
knob
and
then
all
your
parts
we're
going
to
use
it.
So
sodium
will
essentially
set
this
like
functionality
for
all
the
parts
for
all
the
devices
in
the
host
and
then
yeah
and-
and
there
may
be
less
obvious
part
of
this.
This
will
also
help
for
request
response
type
workloads.
A
So
we
did
some
latency
measurements
in
our
lab
and
what
we,
what
we
found
is
like
on
the
on
the
last
bar
here.
If
you
went
psyllium
with
with
big
TCP
disabled,
which
is
basically
just
a
you
know,
stock
psyllium,
and
if
you
then
enable
it,
you
will
see
like
a
or
at
least
we've
seen
in
our
tests
at
2.2
x,
lower
P99
latency,
which
is
quite
good,
and
it
also
helps
like
for
the
transactions
like.
A
We
have
the
like
request
response
transactions
and
it
improved
them
by
32
percent
in
looking
at
the
bulk
TCP
streams,
so
the
default
when
you
run,
for
example,
iperf
in
your
network
from
a
part
to
part
when
it
when
it
was
off,
we
were
able
to
reach
on
our
test
systems
52
gigabit
per
second,
so
the
so
those
test
systems
they
are
just
like
I
can
in
our
lab.
They
were
like
gaming
gaming
systems,
but
with
100
gigabit
nics
because
they
had
pcie
Express
4.
A
and
then,
if
you
turned
it
on,
we
just
got
so
so
we
got
A
plus
eight
gigabit
per
second
performance
Improvement,
so
we
were
able
to
get
to
60
gigabit
per
second
and
then,
if
you
look
at
what
else
would
be
missing.
So
if
you
look
in
into
actually
the
the
trace
profile,
you
see
like
a
lot
of
copy
to
and
from
user
that's
a
limitation,
because
iperf
does
not
yet
use
memory.
Map
TCP.
A
A
If
I
look
at
sodium
with
tomorrow's
data
plane
in
mind,
so
how
can
we
get
to
a
higher
scale,
opinionated
data
plane
and
if
I
look
at
psyllium
as
a
standalone
Gateway?
So
we
we
achieve
this
through
ebpf
on
the
xtp
layer
to
a
layer
for
load
balancing
which
has
a
programmable
API.
It
supports
Mark,
left,
DSR,
graceful,
back-end
termination
and
then
the
new
things
with
the
net
for
six
and
six
four
Gateway,
so
that
you
can
build
V6
only
clusters
and
then
the
other
side,
so
psyllium
inside
kubernetes
as
a
networking
platform.
A
So
there
are
like
a
bunch
of
pieces
of
the
parcel
to
get
it
really
to
high
scale.
So
one
of
the
things
is
the
evpf
based
Q
proxy
replacement,
which
uses
also
xtp
and
the
socket
lock
balancer
in
the
kernel.
Then
we
have
a
feature
which
is
called
ebpf
based
host
routing
for
lowering
the
latency
I
will
go
into
that
a
bit
we
have
a
bandwidth
manager
like
I,
have
to
talk
about
this
at
the
last
kubecon.
A
If
you're
interested,
so
this
essentially
allows
this
essentially
installs
fqq
disks
on
your
physical
devices,
and
this
allows
to
do
pacing.
This
enables
bbr
for
ports
which
would
otherwise
not
be
possible,
and
you
can
do
bandwidth
limitation,
which,
with
the
lower
latency
than
the
usual
means,
and
then
the
new
pieces
that
we
added
now
in
the
new
sodium
release,
is
IPv6.
Big
TCP
support
and,
last
but
not
least,
a
new
meta
driver
is
a
weave
device
replacement.
So
I
will
go
into
that
as
the
last
part
of
the
talk.
A
A
So,
typically,
you
go
to
the
upper
stack
for
packets
in
and
out
of
the
part,
you
have
to
go
to
IP
routing
net
filter
and
so
on,
and
we
added
an
extension
a
while
back
into
the
kernel
where
from
BPF
we
can
directly
push
traffic
into
the
Pod
just
in
one
go
so
without
having
to
Traverse
the
per
CPU
backlog
queues
in
the
kernel
for
weave
devices.
So
that
really
helps
to
deliver
latencies.
You
can
wake
up
the
application
right
away
without
having
to
go
to
a
rescheduling
point
on
the
way
out.
A
So
if
you
look
at
the
flame
graphs
for
the
for
the
kernel,
so
this
is
the
typical
Reef
case.
So
what
you
can
see
is
like
we've,
you
know
cleared
some
of
the
metadata.
It
pushes
it
into
a
pair
CPU
backlog
queue
also
on
the
way
out
and
then
in
the
really
in
the
worst
case,
like
the
kernel
software
IQ
Daemon
issue
is
another
thread,
so
you
have
this
like
rescheduling
point.
So
basically
the
like
the
call
graph
goes
here.
A
Then
it
does
an
enqueue
and
then
it
has
to
pick
up
the
packet
again
to
then
process
it
and
with
the
new
device
type
that
we
built
in
tailored
for
sodium.
You
just
do
everything
in
one
go.
You
don't
have
this
additional
queuing,
you
don't
have
this
this
rescheduling
and
that
really
helps
to
reduce
latency.
So
our
main
goal
with
like
with
the
ebpf
host
routing
this
was
all
the
way
for
the
for
packets
going
into
the
part
and
now
with
this
device
is.
A
This
will
basically
solve
it
for
packets
going
out
of
the
Pod.
So
this
in
combination,
the
goal
that
we
had
in
in
mind
with
that
is
basically
to
reduce
the
latency
and
to
increase
the
performance
in
the
same
way
as
the
application
would
run
inside
the
hostname
space.
And
if
you
look
at
the
latency
here,
you
can
see.
I
like
in
the
yellow
line
is
essentially
the
latency
for
applications
running
in
the
host,
and
it's
essentially
the
same
same
with
the
transaction
is
rate
like.
A
If
you
have
more
like
request
response
type
workloads,
they
will
also
be
on
par
with
the
host
and
throughput
as
well
so
yeah
that
will
basically
solve
it.
So
essentially,
the
pieces
of
the
puzzle
like
for
high
scale
data
plane
that
I
talked
about
that
are
really
critical
in
in
this
scenario,
is
like
the
ebpf
based
Q
proxy
replacement,
the
bandwidth
manager
that
I
covered
at
the
kubecon
in
in
April
the
ebpf
host
routing
the
new
IPv6
big
TCP
support
in
The
Meta
devices.
A
So
when
you
then
run
sodium,
you
will
get
all
of
that
yeah.
Last
but
not
least,
I
want
to
thank
a
couple
of
folks
in
the
in
the
Linux
kernel,
Community,
BPF
and
kubernetes
Community,
also
those
who,
from
from
Google,
who
created
initially
the
the
big
TCP
support
in
the
kernel,
because
I
think
it's
a
really
exciting
feature
which
has
a
lot
of
potential
and,
as
you
can
see
it's
it's,
maybe
not
like
an
obvious
thing.
A
A
A
A
A
A
All
right,
oh
one
thing
I
forgot
to
mention
well
I,
was
actually
preparing
the
talk.
I
noticed
that
there's
actual
progress,
so
GitHub
will
have
V6
support.
So
this
is
really
great,
okay,
so
yeah.
So
if
you
have
more
questions,
please
either
come
to
the
sodium
Booth
or
or
ask
me
directly
or
you
know,
yeah
thanks.