►
From YouTube: Making On-Prem Bare-Metal Kubernetes Network Stack Telco Ready - Christopher Dziomba & Marcel Fest
Description
Don’t miss out! Join us at our upcoming hybrid event: KubeCon + CloudNativeCon North America 2022 from October 24-28 in Detroit (and online!). Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Making On-Prem Bare-Metal Kubernetes Network Stack Telco Ready - Christopher Dziomba & Marcel Fest, Deutsche Telekom
If your task is to do Kubernetes at scale and at speed (read: 100s & 1000s of clusters instantly provisioned on-demand), in a complex on-prem environment, on bare metal (yes, no overlay networks, raw power) you will quickly find that the legacy network and network legacy are some of your biggest enemies. No matter how much Kubernetes gets easy on its own, making it usable in such context will require a lot of plumbing, workflows, host tuning etc. If you are a telco, like Deutsche Telekom with historically grown multitude of heavily segmented networks with 100s of historically grown overlapping VRFs it is practically mission impossible. In this talk Deutsche Telekom will share how they tackled that challenge and how they reimagined and implemented network fabric for on-prem bare metal Kubernetes cloud which pretty cloud natively supports their internal Cluster-as-a-Service offering. This cloud is hosting clusters where some of their most demanding applications like 5G core are running.
A
Yeah
hello
and
welcome
to
our
talk,
making
on-prem
bare
metal,
kubernetes,
ready
our
network
stack
tech
already
and
yeah
so
christopher,
and
I
will
guide
you
of
who
we
are
as
a
company
and
also
what
our
platform
is
and
in
the
end
there
will
be
a
tech
savvy
thing
which
we
open
sourced.
So
we
yeah
guide
you
through.
A
That
next
thing
would
be
just
our
introduction
where
we
are
so
we
are
located
under
it's,
not
working
yeah,
it's
okay,
so
yeah,
I'm
located
in
dusseldorf
germany
and
I'm
very
I'm
drinking
a
lot
of
coffee
and
yeah,
I'm
cycling
doing
stuff,
and
I'm
also
hacking
and
yeah.
I'm
also
now,
since
two
years,
I'm
amateur
snowboarder
and
if
you
want
to
check
me
out,
you
can
also
click
in
the
presentation.
A
A
A
A
Thank
you
for
that,
and
that
is
exactly
what
we
do.
So
everything
is
getups
managed
and
we
have
this
deck
here.
So
we
have
the
use
cluster
api,
as
some
of
you
know,
and
we
deploy
our
workload
clusters
also
like
diameter
showed
in
the
keynote
we
currently
don't
are
not
into
the
openstack
provider.
That
is
why
this
grayed
out
here
we're
currently
using
just
the
bare
metal
part
with
metal,
three
or
metacube,
and
we
use
the
copy
provider
vsphere.
A
B
Yeah
so
cloud
native
in
the
tackle
world,
so
I
think
when
you,
when
you
kind
of
start
thinking
about
telcos
cloud
native,
is
not
the
first
thing
that
comes
to
your
head.
To
be
honest,
we
try
to
change
that
or
trying
to
change
that,
especially
on
the
network
level
and
kind
of
trying
to
dig
free
telcos
of
like
being
heavy
tankers
or
container
ships
with
cloud
native
technology.
B
B
We
say
we
want
to
work
upstream.
First,
we
want
to
work
with
the
community
to
build
that
kind
of
thing
consume
upstream,
consume
open
source,
but
also
contribute
back,
but
we
also
want
to,
of
course
add
the
telco
view,
because
I
mean
telcos.
There
are
a
few
of
them,
but
we
are
kind
of
like
for
some
parts,
especially
in
the
networking
world
app
using
human
editors
a
bit,
I
would
say.
B
So
coming
over
the
parts
like
why
we
did
this
and
what
we
did
so
this
is
kind
of
how
it
looks
like
in
your
let's
say,
normal
vmware
kubernetes
spam
meta
cluster.
You
might
have
a
firewall
somewhere
that
firewall
might
connect
to
different
networks.
It
allows
connectivity
to
some
of
them.
It
does
egress
nothing.
It
does
some
other
nothing
whatever,
but
you
have
one
vlon.
B
Your
nodes
are
attached
to
it.
Your
kubernetes
nodes
and
you're
running,
for
example,
carliko
or
silium
on
top
of
them,
which
is
building
a
full
mesh
across
it,
but
you
still
have
that
kind
of
firewall
as
a
central
component
to
manage.
Of
course,
there
are
operators
that
do
that,
but
as
a
central
component
and
especially
in
high
traffic
cases,
firewalls
are
becoming
quite
expensive
now
what
people
did
is
there
is
a
rfc
which
is
like
okay,
please
run
bgp
into
in
large
data
center
networks.
B
If
you
don't
know,
bgp
bgp
is
kind
of
the
border
gateway
protocol
that
is
used
to
route
internet
traffic
or
to
steer
internet
traffic
over
the
all.
Like
word-
and
this
is
what
should
also
be
used
in
a
data
center
because
it
is
proven
to
work
in
the
internet,
so
why
shouldn't
we
use
it
there
now
with
ipfabrics?
B
Of
course,
you
want
to
have
somewhat
isolation
between
your
networks,
so
you
end
up
maybe
doing,
for
example,
bgp
evpn,
which
is
like
a
vpn
kind
of
solution
on
top
of
it
implementing
vxlan
across
your
switches
and
building
then
to
your
server.
You
might
have,
for
example,
connectivity
through
something
called
multi-chassis
link,
aggregation
on
your
leaf
level
and
or
evpn
mh
multi-homing,
which
is
quite
new,
which
I
think
other
than
cumulus
doesn't
really
support
it
anywhere.
B
But
the
traffic
flow
and
ip
fabric
kind
of
looks
like
this.
You
have
the
border
leaves
which
connect
to
the
outside
networks.
I
left
out
the
firewalls
in
that
pic
by
purpose
to
make
it
a
bit
easier.
You
have
the
spines,
which
are
your
like
your
spines
in
the
network.
Of
course,
like
the
name
is
saying,
and
you
have
the
leaves
which
your
servers
connect
to
and
from
a
traffic
flow
you
have
vlans
going
to
the
server
and
then
from
the
leaves.
You
have
it
kind
of
re-encapsulated
into
vxlans
and
then
yeah.
B
B
Now,
from
a
perspective
of
kind
of
a
layer,
two
I
mean
there's
no
firewall
anymore
in
here,
but
comparing
it
to
the
picture
before
you
now
replace
the
vlan
by
vxlan.
But
you
still
have
your
kubernetes
nodes
and
they
still
run
a
full
mesh
across
them
and
they
still
run
encapsulation
across
them,
which
is
under
standard.
I
think
ip
and
ip
vxlan
is
also
used
heavily
there.
So
now
of
encapsulation
and
encapsulation,
which
you
might
not
want.
B
So
what
we
did
is
we
made
the
host
actually
part
of
our
pgp
fabric,
so
it
is
directly
peering
with
the
leaves
with
bgp,
but
not
like
in
a
traditional
sense
with
we
just
do
ipv4
peering,
but
we
actually
do
the
full
bgp
evpn
peering,
so
the
full
address
family
evpn,
so
we
actually
participate
in
the
whole.
Oh,
I
learned
about
the
networks
that
are
there,
the
different
networks,
the
different
virtual
networks
and
so
on,
which
makes
it
like
from
a
traffic
perspective
like
this.
B
You
actually
encapsulate
at
the
node
level
directly
into
a
vxlan
that
you
want
to
talk
to,
for
example,
in
this
case
vxlan
orange,
which
is
your
cluster
vxlan.
You
can
yeah
talk
to
your
other
kubernetes
nodes.
You
can
talk
to,
for
example,
the
border
leaf
and
from
the
border
leaf
it's
getting
into
other
networks
from
vxlan
view.
This
also
brings
one
additional
thing:
everything
it
can
be
routed
now.
Nothing
needs
to
be
layer,
2
kind
of
vlan,
but
every
like
every
node
ip
address.
B
Every
pod
ip
address
can
be
a
so-called
host
route
or
evp
and
type
5
route.
However,
you
want
to
call
it
and
everything
is
really
routed:
no
layer,
2,
no
multicast,
broadcast
whatsoever.
What
you
have
there,
however,
this
brings
one
more
issue
on
the
border
leaf.
Actually
you
have
to
kind
of
select
okay.
Maybe
my
nodes
want
to
send
to
this
destination
to
this
network
or
my
for
for
this
ip
address
through
the
other
network,
so
you
have
to
do
some
kind
of
configuration
centrally
at
your
border
leaf.
B
So
what
we
proposed
as
a
solution
internally
and
also
kind
of
rolled
out
here
is
we
already
have
the
nodes
part
of
the
ip
fabric.
Why
should
they
participate
in
one
vxlan,
vrf
vni,
only
they
could
be
in
more
of
than
one
and
actually.
Why
should
we
then
do
actually
this
kind
of
steering
where
the
traffic
should
end
up
at
the
border
leaf
when
we
could,
for
example,
just
do
it
at
the
node?
B
So
now,
at
the
node
level,
you
directly
have
connectivity
to
your
like
orange
wielix
lan,
which
is
now
your
tenon
vxlan,
which
is
just
for
your
east-west
traffic
in
the
cluster,
but
also
your,
for
example,
blue
vxlan.
Here,
which
is
then
where
we
actually
directly
learn
all
the
routes
from
the
network,
one
on
our
kubernetes
node
and
can
say:
okay.
B
It
looks
a
bit
like
that.
For
example,
if
you
have
connectivity
to
both
networks,
you
have
the
three
vxlans
one
for
network,
one,
one
for
network
two
and
another
one
for
the
other
kubernetes
cluster
east
west
traffic,
pod
to
pod
node
to
node.
All
of
them
is
type
five,
so
everything
is
host
route.
Everything
is
a
direct
route,
no
layer,
2
network
engineers
are
happy.
B
B
We
use
linux
vrfs,
which
will
I
will
come
a
bit
later
on
marcel,
I
think.
Actually,
we
have
underlay
vrf,
but
we
have
one
issue.
There
are
containerized
network
functions
which
require
iov
a
nasa
iov.
I
can't
put
into
a
linux
vrf
I
mean
I
might
be
able
to,
but
in
the
end
it
will
be
slow
pathed
and
for
cnfs
that
have
chosen
sriv
they've
chosen
it
for
a
reason.
They
do
not
want
to
be
slow
path.
B
B
B
B
A
Netplan
doesn't
work
because
netplan
doesn't
know
about,
vxlan
doesn't
know
about
dummy
interfaces,
doesn't
know
about
any
vfs,
so
we
needed
something
so
that
we
can
connect
our
nodes
to
our
management
network
and
that's
where
we
built
a
new
tool
called
netplanner
and
that
one
is
consuming
exactly
also
a
netplan
compatible
configuration
and
then
templates
out
systemd
network,
the
files
and
yeah.
It
is
also
able
to
configure
sriv
on
the
cards
which
support
it.
A
It
has
a
normal
standard
linux
configuration
available
for
sriv,
as
I
already
mentioned,
and
it
looks
like
it
looks
very
familiar
to
netlin
configuration
yeah.
It
has
currently
no
wi-fi
support.
I
think
you
can
build
it
in
and
it
is
just
very
very
imperative,
so
you
can
have
a
single
file
configuration
which
holds
all
your
wiring
on
the
node
or
you
can
also
have
multiple
configuration
files
to
separate
your
configuration
a
little
bit
and
it
is
yeah
mostly
network
netplan,
compatible
yeah.
A
Vrfs
are
complex
and
also
a
pain
in
linux,
but
we
still
needed
them
because
we
have
a
lot
of
vrfs
in
our
network,
so
he
implemented
this
nice
way
of
having
an
ibtp
pairing
inside
the
node
to
do
not
do
any
route
leaking
there.
Yeah
we
put
most
of
those
things
into
a
network
operator
which
we
learned
there
and
kick
around,
recommend
it
yeah
and
then
one
of
the
downsides
or
from
an
operator
specific
perspective
or
a
platform
perspective.
It's
an
upside.
A
A
B
What
we've
shown
is
and
also
worked
on,
this
kubernetes
house
can
be
directly
participating
in
our
ip
evpn
fabric.
For
us,
it
has
flexibility
because
we
don't
need
to
heavily
automate
any
route.
Leaking
configuration
on
border
leaves,
in
addition
to
that,
it
kind
of
saves
costs
at
that
point
of
view,
because
if
you
do
have
any
route
leaking
you're
programming
them
multiple
times
into
your
hardware,
chip
which
at
some
scale
becomes
really
expensive.
B
Our
netplan
has
no
support
for
vxlan,
vrfs
and
vets,
so
that
is
what
we
implemented,
or
mainly
umarcel,
implemented
in
netplanner
and
network
operator,
serves
as
the
glue
between
kubernetes
and
fr,
so
customers
or
we
as
the
platform
team,
can
configure
fr
like
this
route
leaking
prefix
list
stuff
from
inside
kubernetes
and
we're
also
looking
a
bit
into
dpus
efs
iov
stays
around
to
offload
that
kind
of
stuff
down
to
a
network
card
like
all
this
abstract
away
from
the
cluster
again.
So
the
cluster
just
sees
one
default
route.
B
That's
it
for
our
talk.
Today-
and
I
will
take
a
selfie
with
all
you
in
a
couple
of
seconds,
because
I
can't
still
imagine
that
that
all
of
you
just
came
to
a
network
talk
on
friday
afternoon,
if
you
have
questions
either,
I'm
not
sure
if
you
will
get
a
microphone
or
just
come
to
the
front
hang
around
with
us
and
as
we
are
the
last
talk,
and
I'm
not
sure
if
anyone
is
saying
goodbye
to
you
thanks
seeing
you
all
in
valencia,
maybe
see
us
at
the
next
cubecon.