►
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Kubernetes Networking at Scale - Laurent Bernaille, Datadog & Bowei Du, Google
Running large Kubernetes clusters serving high volumes of traffic (thousands of nodes serving thousands of requests/second) requires tackling scaling challenges in both the control plane and data plane. This talk will present options that allow for performant networking when the number of nodes, services, endpoints and traffic grow in your Kubernetes cluster. Laurent and Manjot will cover how to use CNI plugins for efficient routing by not requiring overlays, how kube-proxy can be configured to handle clusters with thousands of services and endpoint and how ingress controllers can route traffic directly to pods without requiring nodeports. In addition, many of these solutions are at an early stage and the talk will dive into the issues faced and how they were addressed. Finally, the talk will discuss upcoming technologies that will allow Kubernetes to scale even further.
https://sched.co/MPch
A
Now,
in
terms
of
the
high-level
looking
at
scaling
challenges,
we
need
to
consider
a
couple
of
things.
First
is
in
the
data
plane,
you
really
want
to
remove
as
many
inefficiencies
as
possible
right,
ku
Bernays
offers
a
whole
bunch
of
abstractions,
but
we
really
want
to
reduce
the
cost
of
those
abstractions
in
the
control
plane.
When
you're
scaling
up
your
cluster,
you
will
hit
these
large
end
issues
where
n
is
number
of
nose.
Number
of
resources,
in
addition
to
the
large
n
issues,
you
will
also
want
to
simplify
your
architecture.
A
Simplifying
your
architecture
really
means
reducing
dependencies
and
making
it
easier
to
debug
and
in
general
we
have
found
that
the
really
the
strategy
to
do
this
is
to
sign
of
enable
more
native
integration
with
the
infrastructure
that
you're
running
on
now.
Let's
see
what
this
means
in
practice.
So
we'll
talk
about
pod
networking,
we'll
talk
about
service
load,
balancing,
we'll
talk
about
l7
load-balancing,
which
is
ingress
and
then
finally
we'll
discuss
DNS.
So
first,
let's
talk
about
pod
networking.
B
But
networking
is
the
way
to
give
interfaces
and
IP
addresses
to
two
pods.
There
are
many
different
ways
to
do
it.
The
most
simple
one
that
many
people
have
tried
when
they
get
started
is
to
use
static
routes.
So
the
way
it
works
is
you
give
every
node
on
your
cluster
a
cider,
and
then
you
use
the
cloud
provider
routing
table
and
you
add
static
routes
for
each
Siler
give
pointing
to
to
the
nodes.
This
is
very
simple
works.
Fine.
B
However,
there
are
limits
in
terms
of
number
of
static
routes,
for
instance
by
default
on
AWS.
You
can't
have
more
than
50
static
routes,
which
means
you
can
have
more
than
50
nodes
and
also
and
we'll
dive
into
this
later.
It's
not
very
efficient
in
term
of
address
space,
because
if
you
give
a
slash
24
to
all
the
nodes,
if
you
have
10
to
20
minutes
per
node,
it's
not
gonna
be
very
efficient
and
you're
going
to
consume
a
lot
of
address
space
and
it's
gonna
be
tricky
to
manage.
B
Two
nodes
are
actually
virtual
sizes,
which
means
the
underlying
network
doesn't
know
anything
about
it
and
what
the
way
it
works
is
you
actually
do
tunneling
using
either
the
X
LAN
or
IP
IP.
This
works
fine
and
it's
very
efficient
because
it
means
it
can.
Whirring
is
going
to
work
regardless
of
the
underlying
infrastructure,
which
makes
it
very
easy
to
set
up
and
that's
why
it's
very
popular
to
do
it.
B
So
here
is
a
typical
example
of
how
that
would
work.
Your
pod
will
get
an
interface,
so
a
VF
interface
connected
to
a
bridge,
and
then
you
would
have
a
victor,
an
interface
connected
to
this
bridge.
That's
actually
getting
traffic
out
of
the
node.
So
in
this
example,
here
I'm
getting
traffic
from
one
bird
with
IP
191
since
I
go
one
two
one,
nine
two,
six,
eight!
Oh,
why?
B
Another
issue
is
control,
plane,
I
mean
having
overlays
is
fine
when
you're
you
know,
number
of
node
is
not
that
big,
but
at
the
certain
scale
it
starts
to
be
tricky
because,
as
you
can
see
in
this
example,
it's
a
typical
deployment
of
an
overlay.
You
have
an
agent
on
the
node
that
is
going
to
configure
routes
and
channel
in,
for
instance,
vehicle
in
that
case,
and
this
agent
needs
to
get
information
on
the
cluster.
B
So
it's
going
to
connect
to
the
API
server
and
quite
often
it's
going
also
to
need
a
data
store
to
store
all
the
information.
It
needs
it's
quite
a
van
HDD
and
as
your
number
of
node
grows
the
load.
All
these
agents
are
going
to
cause
on
data
stalls
and
API
server
is
going
to
be
non
negligible
and
can
become
an
issue.
B
So,
in
terms
of
best
practices,
what
we
think
where
everybody
is
going
to
use
native
pod
routing,
which
means
avoid
avoid
overlays.
In
this
case,
we
want
to
have
to
open
to
a
gradable
IP
on
the
network
and
so
no
overlay,
it's
much
easier
to
debug
it's
simpler
and
also
it
allows
for
flat
network,
which
means
communication
between
two
clusters
put
in
different
clusters
and
even
VM
outside
of
your
cluster
to
your
pods
is
going
to
work,
because
everything
is
radical.
B
Another
optimization
that
we're
seeing
more
and
more
and
we're
relying
heavily
on
it.
A
data
dog
is
removed.
Bridging
like
typical
installation
is
the
one
I
showed
you
before,
where
all
pods
are
connecting
to
a
bridge
and
then
routed
outside
the
host.
But
when
you
think
about
it,
you
actually
don't
need
this
bridge.
Quite
often,
you
just
can
rely
on
the
house
itself,
and
this
is
what
what
we
do
most
of
the
time.
Another
optimization
that
is
that
we
see
quite
often,
is
IP
VLAN,
where
it's
been
faster
than
having
routing.
B
B
In
terms
of
so
this
is
the
principle
and
now
we're
going
to
talk
about
how
this
is
known
in
practice,
so
we're
gonna
very
quickly
talk
about
on
premise
on
premise,
sometimes
you're
very
lucky,
because
you
can
actually
control
the
network
and
you
can
establish
bgp
connection
between
your
nodes
and
the
network,
which
means
the
network
is
going
to
know
about
the
Sider
allocated
to
each
node
is
going
to
be
transparent.
It's
just
going
to
be
another
side
or
in
cloud
provider.
It's
slightly
different.
So
let's
first
talk
about
how
it's
done
at
Google,
okay,.
A
So
in
the
google
kubernetes
engine
or
if
you
run
kubernetes
directly
on
tcp,
it
used
to
be
in
the
old
way
that
we
create
these
static
routes.
Now,
as
we
discussed
before
static
routes,
the
static
route
functionality
is
actually
fairly
complex
and
you
can
easily
run
into
quota
issues
because
it's
somewhat
resource
intensive
to
sort
of
implement
all
the
different
features
there.
A
What
we
have
added
is
a
VPC
native
mode,
so
in
VPC
native
you
have
a
pre
allocated
range
for
the
pods
and
from
this
range
the
cloud
actually
allocates
the
IPS
as
alias
ranges
to
the
VMS,
and
the
cloud
actually
manages
this
IP
allocation
for
the
cluster.
Now
this
is
much
more
semantic
and
efficient
for
the
underlying
SDM
and
in
fact
you
are
telling
the
Sdn
about
the
pods.
So
the
Sdn
kind
of
knows
that
there
is
this
range
allocated
of
the
pods
and
the
stn
also
knows
about
this
allocation
to
the
pods
itself.
A
So,
let's
see
how
this
works,
so
in
classic
kubernetes,
you
have
a
range,
that's
devoted
to
the
VM
IPS.
You
have
this
pod
IP
allocation,
and
then
you
have
the
service
IPS
on
the
left
and
then
with
the
V
PC
native
mode.
The
the
cloud
will
allocate
these
slash
24s
to
the
various
nodes
that
come
up
in
your
cluster
and,
in
fact,
because
we
have
delegated
IP
allocation
to
the
cloud.
We
can
do
interesting.
A
Things
like
have
the
cloud
control
the
size
of
this
allocation
to
various
nodes,
and
actually
this
becomes
a
scalability
issue
as
well
that
kubernetes
I'm
sure
everyone
has
noticed
when
you
create
a
cluster
with
the
default
settings.
It's
actually
quite
greedy
in
terms
of
eating
up
your
IP
addresses,
for
instance,
with
the
default
settings
1k
nodes
with
a
slash
24.
Well,
that's
a
thousand
times
256,
which
is
actually
256
IPs
on
your
network,
and
we
find
that
a
lot
of
people
when
they
try
to
stitch
their
networks
together,
actually
run
out
of
space.
A
So
with
the
cloud
mediated
IP
allocation,
we
can
tell
it.
Oh
hey
I!
Don't
need
that
many
IPS!
This
is
a
node.
That's
gonna
run
a
very
specialized
workload.
We
can
give
it
a
fewer
I
P,
so
you
can
kind
of
right
size
it
for
your
node
footprint
and
in
general.
We
hope
that
sort
of
as
future
work
is
that
we
get
away
from
these
contiguous
ranges
and
basically
go
to
discontinuous
ranges
that
can
be
merged
together.
So
we
can
kind
of
have
clusters
that
fit
into
your
network
footprint.
A
Another
interesting
thing
about
VPC
native
sort
of
cloud
Sdn
integrated
plot
routing-
is
that
there's
an
opportunity
to
give
the
Sdn
full
visibility
of
your
traffic.
So
when
you
create
a
DPC
native
cluster,
you
can,
when
you
see
that,
when
a
pod
talks
to
another
pod
over
the
Sdn
from
node
to
node,
the
VPC
is
able
to
see
the
flow,
but
when
the
pod
talks
pod
to
pod.
Actually
this
flow
is
not
visible
to
the
Sdn.
A
Well,
you
can
play
some
tricks
with
the
PDP
to
basically
pin
that
traffic
out
to
the
Sdn,
and
what
this
means
is
that
all
of
your
pod
traffic,
whether
it's
between
nodes
or
within
the
node,
will
be
visible
by
the
Sdn
and
kind
of
use,
the
same
infrastructure,
for
example.
You
can
use
the
same
kind
of
tools
on
your
Sdn
to
observe
all
the
flows
in
your
cluster
and.
B
So
this
was
for
DCP
an
address
it's
slightly
different
because
they
no
equivalent
feature
in
terms
of
at
ple
assurance.
You
cannot
associate
an
additional
range
to
your
notes,
but
there
are
still
some
CNI
plugin
that
allow
you
to
give
to
give
radabaugh
IP
stupids,
but
the
first
one,
that's
pretty
well-known,
because
it's
the
one
provided
by
AWS
and
use
any
key
s
is
the
e
key
sen.
I
plug
in
the
way
this
plugin
works.
B
B
The
first
thing
they're
going
to
do
is
the
shehnai
plugin
is
going
to
call
a
demon
that
is
responsible
for
allocating
yen
eyes
and
providing
eyepiece,
and
then
this
runtime
is
going
to
create
the
interface
associate
this
IP
with
it,
and
do
some
clever
writing
on
the
host
to
make
sure
that
traffic
from
IP
one
on
part,
one
is
actually
going
to
throw
in
the
face.
Ethier
h1.
In
this
is
raid
zero.
B
An
alternative
plug-in
is
a
plug-in
developed
by
lift,
which
is
very
similar
in
terms
of
how
it's
designed,
because
it's
also
attaching
additionally
an
eyes
an
additional
IPS.
It's
a
bit
more
complicated
because
they
did
some
optimization.
So
the
first
difference
is
each
team.
Unless
so,
you
actually
have
a
binary
called
I
Pam,
something-
and
this
spanner
is
responsible
for
manipulating
eni
and
allocating
IPs
and
the
main
traffic
for
the
pod
is
going
to
use
an
IP,
VLAN
interface.
B
The
goal
for
this
one
is
to
be
much
more
efficient
than
during
just
routing,
because
I
believe
you'll
and
you
don't
have
to
cross
the
same
part
of
the
kernel,
so
it's
gonna
be
faster.
The
disk
was
pretty
fine,
but
sometimes
also
need
to
reach
to
reach
and
IP
addresses
on
the
node,
such
as
the
cubit
resistance.
In
that
case,
you
need
an
additional
interface
and
that's
why
there
is
this
plug-in
they're
called
PTP,
which
is
going
to
create
an
interface
to
the
house.
B
So
this
works,
pretty
fine,
we've
been
using
it
for
more
than
a
year
now,
and
we've
been
very
happy
with
it.
But
honestly,
it's
a
bit
complicated
to
debug,
sometimes
at
the
conclusion
on
this,
and
this
part
native
patrolling
is
very
efficient,
would
pretty
sure
that
everybody
is
going
to
go
this
way
in
the
future.
B
You
don't
pay
the
cost
of
the
overlay
CNI
plug
in
a
still
a
bit
young
but
they're.
Getting
there
I
mean
we've
seen
a
lot
of
progress
in
the
last
year
and
as
I
were
saying
before,
it's
allows
for
traffic
between
clusters
and
traffic
with
vm.
So
it's
a
lot.
It's
a
lot
easier
and
finally,
it
allows
ingress
traffic
to
be
managed
much
more
efficiently
and
we
discuss
this
later.
A
A
It's
going
to
watch
the
API
server
associated
with
this
whole
service
mechanism
is
a
set
of
resources
which
are
controlled
by
the
endpoints
control
and
the
service
controller,
and
the
endpoints
and
service
controller
basically
are
responsible
for
looking
at
the
service
resources
assembling
all
the
relevant
endpoints
and
writing
them
into
the
set
CD
that
then
can
be
consumed
by
the
queue
proxy.
Now
the
PUE
proxy
implements
a
client-side
load
balancer
on
every
node
right.
So
here
you
have
the
client
and
it's
sending
traffic
and
there's
this
proxy
err
box.
A
In
most
typical
implementations
with
IP
tables,
that's
actually
just
a
Linux
kernel
and
then
it's
going
to
load
balanced
traffic
to
the
various
pods.
Now
the
original
implementation
of
queue
proxy
was
in
user
space.
This
is
old
history.
Most
people
use
the
IP
tables
based
queue
proxy,
it's
default
since
1.2
and
it's
way
faster
than
the
user
space,
but
we'll
see
that
they
have
some
problems
and
then
finally,
there's
an
IPS
implementation
that
when
GaN
1.11
and
it
uses
a
different
kernel
mechanism,
but
it
still
relies
on
IP
tables
for
some
things.
A
A
The
outgoing
traffic
find
a
back-end
rewrite
the
back
end,
the
service
IP
to
the
back
end
and
then
also
handle
getting
the
traffic
back
in
the
reverse
path
and
the
way
this
works
I
won't
go
into
detail,
but
it's
a
lot
of
chains
of
iptables
right
and
generally
speaking.
Iptables
is
the
rules
are
basically
a
bunch
of
if
statements
chained
together
in
a
linear
list.
A
A
So
what
are
the
challenges?
Well,
a
couple
and
I
kind
of
hinted
on
it
in
the
previous
slide.
The
IB
tables
kernel
interface
requires
a
sink
of
all
the
rules
of
one
shot.
This
is
just
how
that
kernel
interface
works.
You
have
a
longer
sink
time
now
this
used
to
be
a
big
problem,
but
some
kernel
fixes
went
in
and
it
has
gone
down
to
seconds,
but
it's
still,
you
have
to
pay
that
cost
and
there's
significant
memory
usage,
because
you
have
to
generate
this
list
of
rules
it
has
to.
A
Even
if
you're
changing
one
thing
generate
the
whole
list,
which
may
be
in
a
large
cluster
over
100
megabytes,
sync
it
and
then
you
continue.
The
second
thing
is
IP
tables
uses
contract,
which
leads
to
a
lot
of
right
sizing
issue
right.
How
much
memory
do
you
have
to
reserve
for
contract?
And
we
have
seen
this
especially
challenging
when
you
have
DNS
traffic,
which
consumes
a
lot
of
contract
entries?
A
B
When
we
started
our
Ajani
in'
in
cuban
ideas,
we
knew
okay,
so
we're
going
to
be
big
in
terms
of
number
of
nodes
number
of
endpoints,
and
we
knew
we
were
gonna
have
issue
with
IP
tables.
So
from
the
get-go
we
decided
to
try
a
PDF,
because
we
we
assumed
it
was
going
to
be
a
lot
faster
and
a
lot
more
efficient
in
terms
of
design.
The
logical
design
is
the
same
as
the
same
one
as
we
used
before
and
in
terms
of
mapping
in
the
umber
of
components.
B
Accumulated
service
is
mapped
to
a
virtual
server
in
a
previous
and
but
as
a
real
real
server,
which
is
a
back-end
in
a
virtual
server.
So,
as
we
were
saying
before,
the
main
advantage
here
is
that
updates
are
atomic,
so
your
only
chance
I
mean
if
you
delete
a
pod,
you
own
two
days,
the
backend
and
in
a
virtual
server.
So
it's
much
more
efficient
in
terms
of
how
it
looks.
This
is
the
output
of
the
IPS
ADM
command,
which
is
giving
you
information
of.
B
B
Ipbs
has
been
working
pretty
great
for
us,
I
mean
we've
been
using
it
for
more
than
a
year
as
I
was
saying,
but
there
was
one
chain
that
was
very
important
and
that
made
things
a
bit
tricky
over
the
last
few
months.
I
PPS
used
to
not
support
graceful
termination
at
all,
which
meant
as
soon
as
you
delete
a
pod
or
and
big
a
pod
becomes
not
ready.
It's
remove
of
the
backend,
which
means
no
traffic
is
going
to
flow
to
this
pod
anymore,
which,
when
you
have
an
existing
connection
open.
B
You
would
want
some
time,
for
example,
file
HTTP.
You
would
want
for
the
end
of
the
communication
to
go
on,
which
is
a
lot
better,
so
graceful
termination
was
introduced
in
112
and
back
party
to
111
and
the
first,
the
first
version
of
it
add
a
few
bugs
and
but
we're
getting
there.
It's
a
lot
better
now.
So,
if
you
have
any
issues
with
that
PVS,
please
let
us
know
up
and
issues
and
queue
proxy
will
go
fix.
It
I
mean
we're
really
committed
to
fixing
it.
One
thing
that's
important
with
that
PVS
is.
B
We
have
double
contracting
IP
vs
as
its
own
connection
tracking
system,
and
you
also
rely
on
the
contracting
Bakula
and
it's
a
bit
tricky
because
then
I
mean
I'm.
Bobby
mentioned
issues
with
contract
sizing
and
timeout
before
so
now
we
have
to
do
it
twice,
based
on
IP
vs
and
in
the
in
the
normal
contract,
and
the
default
time
at
for
PBS
are
not
great.
The
figure
there
are
four
TCP
TCP
fee,
which
is
when
a
connection
in
tab
is
dead,
terminating
and
the
300.
B
So
five
minutes
is
UDP,
as
you
can
imagine
like
keeping
a
connection
the
contract
of
five
minutes
when
it's
a
DNS
query
is
very
bad
idea.
We
also
had
a
few
issues
with
shinai,
because
no
CNI
plugins
are
tested
with
IP
tables
and
routing
with
a
PPS
is
slightly
different,
so
we
noticed
some
movie
interaction.
So
if
you
want
write,
EVs
make
sure
that
you
test
everything
with
cyanide.
B
So
haproxy
is
one
of
the
implementation
of
clans
eyelid.
Balancing
there
are
several
others,
and
queuebrowser
is
one
of
them,
so
the
brother
is
doing
a
lot
of
things.
If
you,
if
you
know
it,
it
also
allows
you
to
control
routing
between
junior
birds,
and
it
comes
also
with
service
load
balance
using
IPPs
and
it's
a
different
limitation.
Another
option
would
be
to
use
CDM,
which
is
using
EBP
F
to
do
client-side,
load,
balancing
and,
in
addition
to
and
as
many
other
features
in
addition
to
that.
A
Okay,
so
just
to
wrap
up
on
the
service
load,
balancing
there's
actually
a
couple
of
other
exciting
things
going
on.
So
first
is
implementation
of
topology,
aware
service
routing.
So
what
this
means
is
like
you
can
express
things,
for
example,
low
load
balance
first
to
my
node
and
if
that's
not
available,
then
spread
it
out
across
the
cluster
and
that's
the
link
to
the
cap.
Please
take
a
look
at
it.
It's
pretty
much
implementable,
but
we
really
would
like
feedback.
The
second
is
I'm
sure.
A
If
you
run
in
a
big
cluster
and
run
big
services,
you'll
notice
that
eventually
your
endpoints
object
will
become
super
massive
and
no
longer
fit
in
your
at
CD
table.
So
this
project,
endpoint
slice,
is
a
project
to
take
that
endpoints
object
and
shard
it
among
several
endpoint
slices
to
get
around
this
scalability
issue.
So
please
take
a
look
at
that
cap
as
well.
That
is
an
ongoing
design.
B
B
B
B
B
There
are
several
implementation
of
this
design
on
AWS,
for
instance,
you
have
a
controller
called
the
alb
ingress
controller
and
which
can
be
configured
to
route
traffic
directly
to
pods
instead
of
instances.
So
it's
much
more
efficient.
The
way
it
works
is
it's:
creating
a
lb
and
the
target
groups
directly
directly
targeting
targeting
IPs
and
GCP
load
balancers
can
be
configured
to
use
the
network
endpoint
groups,
which
is
doing
going
to
do
exactly
the
same
and
write
traffic
directly
to
the
pods
proxy.
B
A
Okay,
so
now
you
notice
that
it's
those
cases
cover
some
of
the
workloads
surrounding
elf,
seven
in
ingress,
but
you
know
what
about
l4
and
ELB.
So
currently
there
doesn't
seem
to
be
a
solution
there
and
it's
also
limited
to
HTTP
traffic.
Now
we
want
to
consider
UDP
or
TCP
traffic,
and
so
this
is
kind
of
a
work
in
progress
to
get
container
native
load,
balancing
for
all
the
different
types
of
load,
balancing
that
we
need.
A
A
It
has
a
search
path,
which
means
that
the
number
of
queries
actually
gets
amplified
quite
a
bit
and
we
hit
the
contract
table
because
keep
DNS
is
going
to
be
a
service,
so
we
may
run
out
of
contract
entries
now.
Also
DNS
is
a
UDP
protocol
and
who
knows
where
that
UDP
packet
went
so
we
might
have
lost
it
going
to
the
query
and
finally,
because
of
this
search
path
amplification,
you
may
not
have
enough
instances
to
soak
up
all
this
load
so
there's
a
whole.
A
Our
problems
associated
with
the
volume
of
DNS
queries
that
we
see
in
kubernetes.
So
one
of
the
solutions
to
this
problem
is
to
actually
drop
a
no
local
cache
into
your
cluster.
So
the
no
local
cache
runs
on
every
single
node
and
it
can
first
of
all
as
a
cache,
so
it
can
absorb
some
of
the
queries
right
away.
The
second
thing
that
it
does
is
it
avoids
contract
it
actually
inserts
special
rules
into
your
IP
tables
chains
to
basically
skip
contract
and
go
directly
to
the
no
local
cache.
A
A
Another
interesting
thing-
and
this
is
a
ongoing
cap-
is
that
why
don't
we
take
care
of
that
search
path
issue?
So
no
local
cache
is
a
smart
agent.
We
can
put
some
logic
in
there
and
it
can
actually
coordinate
with
the
upstream
cube
DNS
infrastructure.
What
if
we
have
the
node
local
cache,
insert
that
search
path
somehow
into
the
DNS
query,
perhaps
as
a
a
DNS
field,
and
then
actually,
the
rural
search
path
mechanism
could
be
resolved
upstream
inside
the
server,
and
you
won't
have
to
send
all
those
packets
over
the
wire.
B
B
Node
are
all
the
nodes
running
your
Co
DNS
buds
and
then
the
query
is
going
to
come
from
there
and
if
the
Cordilleras
doesn't
know
about
this
service
is
going
to
fold
it
upstream,
as
we
were
saying
before
this.
This
led
to
this
leads
to
a
few
issue,
and
we
have
we
had
quite
a
few
outages
for
this
design
so
for
application
that
do
a
lot
of
queries
and
I'm
talking
about
title
of
query
per
seconds.
The
design
is
pretty
inefficient,
so
we
introduced
an
alternative.
B
That's
opted
in
so
people
can
just
add
an
annotation
on
their
part,
and
we
have
a
mutating
web
hook.
That's
going
to
modify
the
configuration,
the
DNS
configuration
of
the
pod,
and
you
can
see
here
the
configuration
of
the
pod,
so
we
do
two
important
things.
The
first
one
is.
We
have
a
single
search
domain,
which
is
going
to
be
SVC
that
cluster
dot
local,
which
means
apt,
even
in
the
single
in
the
same
name,
space
application
we'll
need
to
put
service
dot,
name
space,
they
can't
use
just
service.
B
So
that's
the
reason
it's
obtained
it's
because
it's
breaking
some
of
the
kubernetes
contracts,
but
it's
much
more
efficient
because,
most
of
the
time
it's
going
to
be
in
single
query.
In
addition
to
that,
because
we
are
very
opinionated
in
the
way
we
do
search
domains,
we
can
go
back
to
n
dot
equal
to
which
means,
if
you
do
a
standard
query
like
wwe.com.
B
B
We
modify
also
the
resolver
in
point
to
a
node
local
DNS,
cache,
okay,
so
it's
a
very
small
coordinates
instance
running
locally
and
this
instance
that's
to
think
it
does
caching.
So
if
you
hit
the
same
name,
a
lot,
you
can
a
bit
better
table.
Caching
is
gonna,
be
a
lot
more
efficient
and,
in
addition,
it
writes
traffic
that
is
not
resolvable
by
coordinates
natively
in
the
in
the
cluster
directly
to
the
upstream
and
locally.
A
Really
a
common
theme
is
you
know,
removing
the
cost
of
all
the
abstractions
right
native
integrations
with
the
infrastructure
is
key.
So
a
lot
of
these
things
that
we
described
it's
like.
How
do
we
get
things
to
route
directly?
How
do
we
remove
overlays
and
so
forth?
One
thing
to
observe
as
many
caps
in
flight
to
improve
scalability
and
then
the
third
thing
is
actually.
C
D
B
So
yes,
I
mean
we
don't
have
the
issue
that
much
because
we
don't
increase
the
cluster
size
very
fast,
but
yes,
a
way
of
boosting
work.
It
they
locally,
try
and
get
an
address,
and
so
you
can
get
rate
limited
by
the
other.
Escapee
is
so
yeah
I
think
this
can
be
improved
in
both
the
AWS
and
the
lift
plugin
but
I.
B
A
D
As
mentioned,
part
of
the
iptables
problem
is
like
the
N
squared
ever-present
networking
problem
and
a
linear
search
and
IP
tables
how's.
This
helped
by
moving
that
into
the
load,
balancer
or
or
the
neg
implementation
is
just
a
different
data
structure
that
does
it
require
the
linear,
search
or
you're
just
scale
it
out
a
different
way.
A
D
B
A
A
A
I
see
the
the
issue
that
was
illustrated
here,
so
the
question
is:
have
you
thought
about
using
a
separate
service
for
storing
the
rules?
The
issue
that
was
brought
up
was
more
that
the
kernel
interface
to
change,
IP
tables
is
actually
all
just
one
shot
like
you
have
to
generate
the
whole
thing
and
put
it
in,
even
though
your
change
might
be
very
small,
so
that
turned
out
to
be
very
inefficient,
because
if
you
have
a
lot
of
endpoints
a
lot
of
services,
you
kind
of
have
to
regenerate
that
thing.
F
In
the
on-prem
scenario,
it's
not
always
an
option
to
be
able
to
use
the
Royal
architecture,
IP
addresses
and
overlays.
It
seems
to
be
the
only
solution
unless
going
for
another
address
space
right
and
ipv6.
So,
given
what
you
know
about
communities
at
scale
is
ipv6
an
option
to
get
those
things.
I
cannot.
B
And
for
the
unprimed
squishin
most
of
the
people,
I
know,
but
I'm
really
not
a
specialist.
What
they
do
is
they
do
native
routing
they
have
BGP
so
either
a
browser
or
calico
running
on
the
node,
and
all
this
during
peering
I
mean
two
older,
BGP
routers
and
announcing
route
to
the
network,
in
which
case
you
don't
in
another
light.
If
you
can't
do
that
because
in
many
companies
you
can't
actually
do
that,
then
you
have
to
use
an
array
but
yeah.
A
B
Of
the
biggest
issue
in
the
cube
proxy
contracts
today
is
pretty
complicated,
as
we've
seen
on
the
I
mean
I've
worked
a
little
Mia
PBS
implementation
and
the
contract
established
by
the
de
facto
default,
which
is
IP.
Labels
is
very
complicated
and
getting
this
right
in
a
new
implementation
may
be
tricky.
It's
gonna
be
a
lot
of
work,
so
I
mean
if
someone
started
I'm
sure
the
community
would
love
to
try
and
use
it.
But
to
be
honest,
nf
table
is
not
a
load
balancing
solution
either.
B
B
H
B
Say
today
for
most
use
cases,
IPS
is
stable,
there's
a
few
issues
up
an
uptrend,
but
nothing
in
terms
of
features.
So
there's
one
where
this
communication
behavior
is
slightly
different
in
IP
vs
in
IP
tables,
for
instance,
you
can
do
localhost,
note
ports.
This
doesn't
work
all
that
most
of
the
teamwork.
B
The
only
remaining
issues
are
related
to
graceful
termination
and
we
decided
two
weeks
ago
to
remove
gradual
termination
for
UDP,
which
is
going
to
solve
most
of
the
ongoing
issues
and
since
the
IP
tables
implementation
is
not
supporting,
restful
termination
for
UDP
either
is
going
to
be
the
same
behavior.
So
we
had
a
few
people
that
wanted
graceful
termination
for
UDP
for
quick,
for
instance,
but
it's
very
hard
to
get
right.
B
Nobody
yes,
I
mean
most
of
the
new
issues
are
related
to
edge
cases
that
we
don't
know
about.
For
instance,
we
had
a
lot
of
discussion
with
people
from
metal
lb,
which
does
pretty
clever,
saying
that
no
one
in
the
IPV
Aseema
tried
before,
and
so
we
had
to
solve
this,
for
instance.
So
it's
like
specific
use
cases,
specific
CNI,
implementations,
so
I'm
sure
we're
gonna
get
it
fixed,
but
we
need
more
users
to
do
to
be
sure
that
we're
covering
everything.
H
B
H
B
We
started
with
the
abrasion
I
plug
in
because
it
seems
to
be
the
most
natural
thing
to
do.
The
first
implementation
was
very
young
when
we
started
it
had
quite
a
few
bug
and
we
had
to
thick
them
and
the
interaction
with
the
team.
Developing
the
plugin
was
bit
difficult
at
the
time,
so
we
decided
to
try
lifted
the
lid
implementation
and
it
turned
out
this
implementation
was
pretty
solid
and
it
was
very
easy
to
get
things
patched
upstream.
So
that's
why
we
went
this
way.