►
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2023 in Amsterdam, The Netherlands from April 17-21. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Making Kubernetes Networking Ready for Service Providers - Philip Klatte, F5, Inc
Service Providers are largely using Kubernetes for their 5G packet cores to deliver the capacity and performance for cutting-edge enterprise and consumer applications and services. But because Kubernetes was not designed for telco protocols or workloads there are networking challenges that must be solved. This talk will describe some of the unique requirements that telco workloads bring to Kubernetes and examine different solutions to address Kubernetes ingress/egress networking and security challenges. Phil will share lessons learned and explore how the different solutions align to Kubernetes design patterns. A similar version of this presentation was given at MPLS SD & AI Net World Congress 2022. The version for Cloud-Native Telco Day North America will be more technical and target people focused on Kubernetes networking architecture and deployment.
A
Not
meeting
its
promise
and
being
very
complex
and
fragmented,
but
here
I
think
a
lot
of
the
service
providers
are
really
looking
to
using
Cloud
native
principles
to
be
significantly
more
agile
to
leverage
a
lot
of
the
technology.
That's
happening
in
the
broader
sort
of
Technology
space
and
to
get
into
cicd
type
principles.
A
And
and
I
should
say
that
means
that
they're
moving
to
kubernetes
now
a
few
years
ago,
I
still
would
have
gotten
arguments.
People
would
have
said:
oh
5G
is
supposed
to
be
Cloud
native,
but
that
doesn't
mean
kubernetes
necessarily
yeah.
It
does
so
I
think
people
here
probably
recognize
that
kubernetes
has
become
the
de
facto
orchestrator
for
containerized
workloads,
and
it
is
really
key
to
anything
you're
doing
that
is
cloud
native
and
so
I'm
going
to
speak
specifically
about
the
networking
side
of
this.
A
So
again,
these
service
providers
are
rushing
Pell
Mel
into
kubernetes,
and
it
is
interesting
that
when
I
get
onto
calls
generally
with
the
customers,
I
find
that
there
are
a
couple
people
who
really
understand
5G.
Rarely
anyone
who
understands
kubernetes
or
I'm
in
an
alternate
reality
call
where
there
are
people
who
just
understand
kubernetes
and
nothing
about
5G
and
getting
them
both
on
the
same
call
is
virtually
impossible
and
they
don't
really
understand
each
other,
and
one
of
the
key
issues
is
that
kubernetes
was
not
created
for
this.
A
For
these
workloads,
right,
kubernetes
really
came
of
age
with
Enterprise
type,
workloads
and
web
type
workloads,
and
so
now
we're
trying
to
apply
Telco
workloads
to
kubernetes,
and
there
are
some
basic
fundamental
gaps
and
and
I'll
tell
you
it's
interesting.
We
had
one
of
the
original
guys
from
the
original
team
that
made
kubernetes
come
into
F5
about
four
years
ago
five
years
ago,
and
we
were
just
peppering
him
with
questions
about
networking
and
kubernetes
and
he
finally
held
up
his
hands
and
he
said,
look
you
know
the
guys
who
are
programming.
A
A
One
is
kubernetes
fitting
into
the
sort
of
broader
Telco
Network,
and
here
the
thing
is,
is
that
kubernetes
is
really
focused
on
Orchestra
orchestrating,
what's
happening
within
the
cluster,
and
that's
really
what
it's
good
at
is
what's
within
its
cluster.
The
problem
is
it's.
A
Telco
networks
are
extremely
complex
right.
You
have
a
lot
of
separated,
vlans
and
verfs
and
dmzs
and
firewalls
all
throughout
them,
and
just
just
the
IP
engineering
and
anybody
who's
worked.
A
There
knows
what
I
mean
when
I
say:
IP
engineering,
just
the
the
IP
engineering
to
get
an
IP
address
for
a
new
network
function
can
take
weeks
just
getting
the
tickets
through
to
get
through.
Firewalls
can
take
weeks
right.
These
are
very
complex
networks
and
and
dropping
a
cluster
in
there.
That
has
different
network
functions
is,
is
a
is
complex
effort,
and
so
you
need
to
be
able
to
integrate
with
the
routing
infrastructure,
with
the
different
kinds
of
networks
Etc.
A
Secondly,
again
when
the
5G
stuff,
even
service
based
architecture,
was
being
outlined,
it
wasn't
dictated
that
it'd
be
in
kubernetes
right
and
kubernetes
wasn't
designed
for
it.
So
the
the
concepts
in
kubernetes
do
not
necessarily
directly
match
to
the
concepts
that
were
defined
by
3gpp
and
so
forth.
A
For
example,
one
of
the
most
obvious
ones
is
protocols
right
so
and
I'm
going
to
set
aside
the
service-based
interfaces
for
a
second.
But
you
know,
diameter
is
just
not
a
friend
in
kubernetes
right
in
kubernetes,
the
whole
concept
of
Ingress,
which,
which
is
you
know,
exposing
services
to
the
outside
world
right
is
an
HTTP
function
period.
End
of
conversation
there
are
you
know,
and,
and
so
anything
that
falls
outside
of
that
sort
of
normal
Paradigm
is
problematic.
A
Also,
you
end
up
working
a
lot
in
terms
of
services,
things
that
are
being
exposed
to
the
outside
world,
which
are
essentially
just
saying
I'm,
going
to
expose
this
end
point
for
you
to
come
reach
me,
but
the
thing
is
is
that
Network
functions
are
not
just
things
you
reach
out
and
and
and
and
and
and
get
to.
There
are
also
things
to
call
out
right,
and
so
one
of
the
huge
things
is
the
sort
of
bastard
stepchild
egress.
A
There's
a
major
problem
with
egress,
and
this
ties
into
both
of
these
last
two
I
have
a
great
example.
If
you
have
a
packet
floating
around
in
your
network,
you
want
to
know
what
network
function
it
came
from,
because
if
it
came
from
your
AMF,
you
want
to
have
firewall
rules
to
allow
it
to
reach
your
RAM
Network.
A
A
So
kubernetes
has
a
fair
bit
of
security
again
within
the
cluster.
Kubernetes
feature
really
focuses
a
lot
on
what
happens
within
the
cluster,
but
in
terms
of
how
that
inter
integrates
with
the
broader
security
policies
of
the
service
provider,
that's
just
the
the
they
don't
want
me
talking
about
this
stuff.
A
So
if
kubernetes
is
not
really
fit
for
purpose
out
of
the
box,
if
vanilla,
kubernetes
and
the
sort
of
normal
tools
that
people
use
are
not
really
appropriate
for
for
these
Telco
use
cases,
what
do
we
do?
We
have
really
two
options:
right.
We
can
extend
kubernetes
and
ex
and
kubernetes
is
built
to
be
extensible
right.
This.
This
is
a
big
big
part
of
kubernetes
is
that
it
is
extensible
or
you
can
just
break
all
the
patterns
and
burn
burn
it
through
the
ground
and
I
I.
A
So
just
to
talk
about
a
couple
of
the
patterns,
one
of
the
ones
that
really
drives
me
a
little
nuts
personally
is
every
time
I
hear
the
word
maltes
now
multis
in
and
of
itself
is
not
a
bad
word
right.
Maltes
is
a
medicine
eye
that
allows
you
to
run
more
than
one
cni,
and
that
is
extremely
helpful.
It's
very
powerful,
but.
A
In
this
context,
it's
always
oh,
the
CNF
vendors
all
want
to
use
multis.
They
want
multis
interfaces,
they
want
multis
and
it
becomes
a
code
word
for
wanting
to
to
get
around
kubernetes
networking.
Basically,
the
cnfs
are
saying:
look
I
need
direct
network
access
to
the
outside
world.
I,
don't
want
kubernetes
involved
in
it
at
all.
Right
I
want
direct
network
access
off
an
sriv
access
and
I'm
going
to
manage
how
the
IP
addresses
are
presented.
I'm
going
to
manage,
you
know
how
it
connects
to
the
routers
I'm
going
to
manage.
All
of
that.
A
The
problem
is:
is
that
it?
It
then
spills
what
you're
essentially
doing
is
you're
taking
kubernetes,
which
is
this
thing
that
is
highly
Dynamic,
and
that
is
designed
to
to
orchestrate
a
highly
Dynamic
environment
and
you're
spilling
out
all
of
the
complexity
from
inside
your
cluster
to
the
outside
your
cluster,
and
it
means
that
the
the
owners
in
the
service
providers
now
have
to
worry
about
all
of
this
stuff
out
here.
How
do
they
deal
with
the
fact
that
these
containers
could
drop
at
any
time?
A
And
so
these
IP
addresses
could
come
and
go
at
any
time
right.
They
have
to
manage
all
of
this
complexity
that
is
now
Spilled
Out
into
the
network,
and
that
causes
obviously
operational
complexity,
and
it
makes
IP
address
management
very
complex
and
also
the
interconnection
between
Network
functions.
Very
complex
security
complexity,
because
now
each
of
those
cnfs
is
directly
exposed
to
the
outside,
there's
nothing
to
intermediate
there
right
and
then
networking
complexity,
because
you're
really
exposing
a
lot
of
the
the
the
networking
complexity
and-
and
you
need
to
have
these
suddenly.
A
We
see
this.
So
we
see
this
with
every
single
CNF
vendor
that
I
can
think
of
when
we
first
started
working
we're
in
production
right
now
with
a
large
tier
one
carrier
and
literally,
and
they
they
decided
to
build
a
basically
horizontal
stack
where
they
created
the
platform
and
they
invited
cnfs
in
every
single
CNF
vendor
they
had
asked
for
maltes
and
multis.
Was
this.
A
A
A
What
what
we?
The
approach
we've
taken
is:
we've
created
a
thing
called
a
service
proxy
and
the
name
came
from
essentially
one
of
the
first
rfps
we
responded
to,
and
it
really
replaces
the
the
built-in
service
proxy
and
kubernetes,
and
it
has
certain
characteristics
and
I
want
to
say
that
these
are
characteristics
that
we
should
be
replicating.
F5
should
not
be
the
only
ones
with
this
with
these
features.
A
But
at
least
then
it
is
wholly
managed
in
kubernetes
and
all
of
the
pieces
should
be
inside
of
kubernetes.
The
other
thing
that
a
lot
of
the
hacks
have
is
external
pieces.
Where
you
have,
you
know
an
external
load
balancer,
you
have
an
external
whatever,
and
then
you
have
to
do
all
this
coordination
of
the
configuration
of
that
thing
out
there
and
the
thing
in
here
right
with
our
product,
it's
all
entirely
within
kubernetes,
and
there
is
no
GUI.
A
A
Egress
Network
address
translations
so
that
when
packets
are
coming
out,
you
can
say:
oh
that's
an
AMF
packet.
That's
an
SMF
packet,
some
I
address
translations.
A
number
of
customers
have,
for
example,
there's
a
large
tier
one,
North
American
Carrier,
who
has
an
entirely
IPv6
network,
but
some
of
the
cnfs
couldn't
support
IPv6.
Yet
so
they
needed
ipv4
interfaces
and
they
had
a
single
stack
cluster
ipv4
cluster,
and
so
you
had
to
do
translations
with
every
single
packet
that
goes
in
and
out
between,
V4
and
V6.
A
There
are
a
lot
of
those
kinds
of
complexities
that
you
need
in
order
to
make
these
things
work.
You
also
need
to
link
Ingress
and
egress
so
that
a
CNF
presents
as
a
single
entity.
What
we
do,
the
sort
of
normal
pattern
with
our
software
and
what
our
customers
are
doing
is
trying
to
fit
a
CNF
within
a
namespace,
and
it
makes
it
easier
to
do
part
of
that
translation
between
the
kubernetes
concept
of
the
namespace
and
the
the
standards
concept
of
the
CNF.
A
Sometimes
they're
spread
across
a
couple
of
namespaces,
but
we
try
to
to
have
it
either
one
or
a
small
number
that
are
that
CNF
and
then
you
can
treat
the
traffic
that's
coming
in
and
out
of
those
namespaces
as
being
part
of
a
sing,
a
CNF
Additionally.
You
need
to
support
a
broad
number
of
protocols.
So
everybody
says:
oh
5G,
it's
it's
all.
You
know
service-based
interfaces.
It's
all
you
know
HTTP
too
you're,
you're,
you're
golden
well
you're,
not
100
golden.
A
Yeah.
You
know
the
connection
doesn't
last
for
months,
but
it's
lasts
for
a
good
long
time
and
a
lot
of
traffic
goes
over
it.
So
the
patterns
are
still
different
in
this
traffic
and
then
you
need
to
provide
a
security
layer,
and
so,
for
example,
we
we
have
a
layer,
four
firewall,
so
that
you
can
take
your
firewall
rules
that
are
in
your
external
firewall
and
and
condense
them
down.
A
A
A
Some
of
the
business
we've
won
is
that
service
providers
don't
want
to
have,
for
example,
if
there's
an
update
for
SSL
security
or
if
they're,
changing
the
way,
they're
rotating
their
certificates
or
doing
something
how
they're
what
you
need
to
integrate
with
for
security.
They
don't
want
to
go
to
all
the
different
cnfs
and
all
the
different
CNF
groups.
Even
inside
a
single
vendor.
There
are
multiple
groups,
usually
right,
but
but
especially
if
they
have
multiple
vendors,
they
want
to
go
to
one
place
to
say:
hey.
A
A
So
what
I
want
to
get
to
at
the
very
end
and
I
assume
I
am
close
to
the
end.
Okay
is
again
I,
don't
think
F5
should
be
the
only
people
doing
this
one
of
the
problems
that
I
have
again.
When
we
talk
to
customers
there
usually
are
people
who
understand
3G
or
3G
5G
or
people
who
understand
kubernetes
and
the
diagrams.
They
have
to
start
with.
Don't
have
these
functions
in
them
right,
I
I
want
to
start
having
those
diagrams
have
these
functions
and
I
want
to
have
it
be
standard.
A
So
one
of
one
of
the
other
parts
to
this
is
that
CNF
vendors
want
to
have
something
that
they
can
test
with
a
generic
interface
rather
than
a
bunch
of
crds,
because
if
they
are
testing
with
all
f5crds,
it's
a
separate
set
of
tests
versus
if
they're
testing,
with
a
standard
interface
just
to
dumb
that
down
a
little
bit.
A
lot
of
CNF
vendors
right
now
are
using
service
type
load.
A
A
A
We
have
other
groups
that
are
looking
at
adding
in
policies
around
security
Etc,
but
I'd
like
for
us
to
be
figuring
out
a
way
to
do
this
in
a
standard
way
using
a
sort
of
a
standard,
kubernetes
API,
because
otherwise
it's
it's
going
to
end
up.
Looking
like
nfe
again
a
guarantee.
So
are
there
questions.
A
C
I'm
just
going
to
ask
my
question
so
when
you
talk
about
mapping
Ingress
and
egress,
it
goes
back
to
your
original
problem
about
shifting
our
pieces,
pods
come
up
and
down.
One
of
the
reasons
why
we
wanted
to
find
egress
is
to
know
what
source
IP
traffic
is
originating
from
inside
the
cluster.
How
would
you
handle
that
with
your
scenario.
A
A
A
Okay,
so
the
two
questions
are:
does
this
sort
of
eliminate
cni
or
how
does
it
play
with
cni
and
then
do
we
Implement
API
Gateway?
At
the
moment,
we
Implement
API
Gateway
like
more
as
a
demo,
because
nobody
else
is
so
there's
no
point
in
our
fully
implementing
it
and
it
doesn't
have
the
extra
protocols
we
need
so
we're
in
the
early
stages
of
that
we're
sort
of
Road
mapping
late
next
year
to
have
full
API
Gateway
support.
A
We
could
have
it
sooner,
but
no
customer
wants
it
sooner,
but
I
want
to
see
things
like
egress
start
moving
into
the
API
Gateway
before
that
becomes
valuable.
The
second
part
how
it
plays
with
cni
is
a
fascinating
thing
right,
because
the
egress,
for
example,
egress,
is
controlled
by
the
cni
right.
It's
it's
the
thing
that
sees
the
packets
first
on
the
way
out.
So
we
we
we
are
working
on
what
we're
calling
cni
Independence
so
that
it
doesn't
matter
what
cni
there
is,
and
it's
a
little
bit
like
you
know.
A
Istio
has
a
cni
that
basically
puts
itself
in
the
data
flow
before
you
hit
the
the
primary
cni
we're
doing
something
like
that,
so
that
any
egressing
traffic,
any
traffic
that's
going
out
of
the
cluster,
is
sent
to
us
and
it's
essentially
invisible
to
the
other
cni.
But
while
we're
doing
that,
we
have
Integrations
with
a
couple
specific
cnis,
but
that
just
became
burdensome
like
if
you're
running,
ovn
kubernetes
on
openshift.
We
have
an
integration
with
it.
We
have
an
integration
with
Calico,
but
what
I
really
want
is
to
not
worry.
A
D
So
how
how
independent
or
open
source
this
service
proxy
is
because,
like
Cloud
native
should
be
like,
you
know,
like
you,
use
the
same
Concepts
and
you
can
run
on
any
Cloud.
So
if
this
is
something
like
in
the
service,
proxy
is
something
special
in
a
cloud.
Then
it's
like
doesn't
really
happen
or
problem.
Yeah.
A
Yeah
no
I
mean
at
the
moment
what
we
have
is
service
proxy
for
kubernetes,
F5
spk.
It
is
a
product,
but-
and
there
is
nobody
else
who
does
all
of
these
things-
you
can
band-aid
and
hack
and
script
and
do
a
bunch
of
things
to
get
some
of
it
done.
But
there
is
no,
no
other,
but
that's
what
I'm
saying
about
going
to
the
Gateway
API.
A
You
know
Alternatives
right
because
right
now
you
have
to
use
F5
crds,
which
is
a
valid
way
to
extend
and
kubernetes
and
their
crds
and
use
all
over
the
freaking
place.
But
I
would
really
like
to
see
it
be
more
standard
and
then,
and
then
each
vendor
would
add
their
special
sauce
right
to
try
and
you
know,
stand
out
from
the
crowd,
but
but
I
think
it
is
extremely
important
to
get
to
something
more
standard
and
use
a
more
standard
API
than
the
crds.