►
From YouTube: Adapting Kubernetes to Constrained IP Address Environments - Mahesh Narayanan & Satyadeep Musuvathy
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Adapting Kubernetes to Constrained IP Address Environments - Mahesh Narayanan & Satyadeep Musuvathy, Google
When it comes to IP addresses, Kubernetes has a demand and supply issue. On the Demand side, Kubernetes treats Pods as first class citizens with their own IPs. This makes port mappings and usage from a developer’s point of view much much simpler. But from an infrastructure perspective, this makes the whole cluster use IP addresses liberally. On the Supply side, Kubernetes deployments generally run alongside incumbent networks. Therefore there are not enough IPs to allocate and have a production grade deployment. Based on real world experience by our customers so far, we have found that there are a few ways to design your clusters to address these concerns: -- Optimize the per node allocation so that the overall consumption can be optimized -- re-use IP addresses for Infrastructure but have unique Services IPs. -- Leverage a new IP addressing scheme through non-RFC 1918 ranges
https://sched.co/UaVq
A
About
time
to
get
started
so
welcome
to
the
talk,
I'm
Cody
I'm
gonna
be
the
track
host
for
this
session.
So
if
you
need
anything,
come
get
me
step
out
of
the
projector.
So
today
we've
got
Satya
and
Mahesh
they're,
both
from
Google
working
on
gke
pretty
extensively,
and
it
talked
to
us
today
about
IP
constrained
spaces.
A
Administrative
note,
please
be
sure
to
rate
this
talk
and
all
the
talks
that
you
go
to
in
sched
afterwards
helps
for
future
planning
efforts
and
all
that
so
in
a
moment,
I'll
hand
it
off
and
then
five
to
ten
minutes
before
the
end,
we'll
do
Q&A
I'll
run
the
mic
and
we'll
go
from
there.
So
with
that
Satya
and
Mahesh
thanks.
Thank
you.
C
C
So
the
our
talk
is
on:
how
do
you
mitigate
IP
exhaustion
issues
by
running
kubernetes
right?
So
what
we're
seeing
is
there
is
a
demand
and
supply
issue
and
you're,
not
in
kubernetes.
On
the
demand
side,
there
are
different
resources
that
are
requesting
IPS
and
then
the
supply
side.
There
is
the
availability
of
free
IPS
with
the
customers
based
on
the
net
based
on
the
network
requirements
individually.
These
two
are
not
a
problem,
but
the
intersection
of
these
the
demand
side
and
the
supply
side
causes
an
issue.
C
The
demand
side
can
be
the
the
consumer,
and
the
supplier
side
is
the
producer
right.
So,
with
that
high-level
agenda
we
gonna
start
off
with
a
context
you
know.
Just
we
can
will
provide
you,
the
introduction
or
fundamentals
of
communities.
Networking
go
into
the
problems
scenarios
talk
about.
Why
and
when
we
see
the
scenarios
move
solution
space
and
then
at
the
end
there
is
a
you
know:
call-to-action
they
will
request
how
you
can.
You
know,
help
us
out
providing
your
inputs.
D
Spanish,
and
so
before
we
go
into
the
details
on
start
with
some
a
little
bit
of
context,
so
kubernetes
is
many
things
to
many
people.
It's
container
orchestration,
it's
a
lot
of
things
and
what
we
want
to
do
is
to
make
sure
that,
as
we
like
go
over
the
stock,
we
kind
of
take
an
abstraction
of
easier
for
us
to
reason
about
thanks
and
to
do
that,
we
essentially
start
with
something.
That's
very,
very
fundamental.
To
put
kubernetes
makes
the
choice
that
every
part
gets
its
own
IP.
D
This
makes
it
different
from
most
of
the
container
authorization
systems
and,
in
addition
to
each
part,
getting
its
own
IPS.
It
makes
kind
of
two
other
things
that
are
true:
the
kubernetes
network
model,
all
containers
coasters
within
a
part
share
the
same
address
space,
it's
the
same
IP
and,
in
addition,
the
part
IP
is
from
a
flight
address
space
in
the
cluster,
which
means
any
part
should
be
able
to
talk
to
any
other
part
just
based
on
its
IP
and
the
port
to
which
the
part
is
bound
or
expose.
D
In
addition,
service
IP
is
when
the
cluster
are
tracked.
In
terms
of
part,
I
base
for
endpoints
are
nothing
but
collections
of
particles.
Part
IP
is
are
ephemeral,
so
ports
come
and
go.
The
only
thing
that
stable
is
the
server's,
IP
and
and
subtype
is
like
a
bit
inside
the
cluster,
but
it
translates
basically
down
to
a
bunch
of
world
IPS
and
the
assumption.
D
The
third
thing
is,
and-
and
this
is
true
in
most
systems,
the
pod
IP
allocation
itself
happens
in
a
distributed,
uncoordinated
matter
like
what
that
means.
Is
you
take
the
full
address
space
that
you
are
saying
for
the
cluster
slice
it
up
into
smaller
ranges?
You
give
it
to
each
node
and
then
add
the
node
level
as
part
of
schedule
and
as
pods
come
and
go.
Ips
are
now
just
within
that
small
pool
right,
and
this
has
significant
advantages.
D
You
don't
have
to
have
a
global,
lock
you're,
guaranteed
that
as
long
as
each
node
is
having
some
listing
cider
any
allocation
that
you
do
will
be
globally
unique,
right
and,
and
it
does
lead
to
a
lot
of
benefits
in
the
scalability
of
how
you
can
do
these
IPR
segments,
but
then
kubernetes,
but
this
model
actually
has
certain
implications.
The
implications.
D
The
two
main
implications
are
as
follows:
right
kubernetes
is
IP
address
language
if
you're
going
to
give
every
part
an
IP
right
and
you
have
a
hundred
node
cluster
and
you
want
to
give
let's
say
a
slash
24
as
the
site
arrange
to
each
node.
You
need
at
least
a
slash
17
address
space
for
all
of
the
parts
to
be
able
to
fit
into
this
model.
That's
a
lot
of
addresses
and
from
if
you
come
from
a
traditional
world
of
each
VM
is
a
single
IP.
This
is
orders
of
magnitude
more
right.
D
So
that's
the
first
demand
problem
on
the
community
side.
Second
thing,
though-
and
this
is
not
as
obvious
is-
you
cannot
think
in
terms
of
just
the
number
of
pods
you
run
because
of
the
distributed
nature
of
allocation
of
IP
addresses.
We
need
to
have
a
buffer
of
IPs
at
each
node.
On
top
of
the
number
of
parts
you
expect
to
run.
This
is
because
kubernetes,
as
you
all
know,
is
eventually
consistent.
So
what
happens
when
a
part
is
rescheduled
right,
a
part
gets
rescheduled.
D
Then
you
go
update
the
server's
endpoints
to
basically
say
now:
I
have
a
new
IP
address,
so
that's
one
change
on
the
API
server,
then
all
the
queue
proxy
at
each
of
the
nodes,
watches
this
and
then
say:
okay
now
have
to
rewrite
the
IP
tabled
rules
to
say,
remove
the
old
IP.
Add
the
new
IP
to
that
list
before
the
IP
then
becomes
inaudible.
D
This
is
to
have
a
buffer
of
IP,
so
you
don't
run
into
IP
d
use
public,
and
so,
if
you
look
at
all
this
and
then
in
trying
to
think
about
this
problem,
we
actually
wanted
to
have
a
way
to
model
this
behavior
right
and
what
are
we
talking
about?
And
and-
and
we
found
this
model
a
little
bit
easy
to
reason
about.
If,
because
we
look
at
what
I
just
said
and
look
at
how
Google
DS
looks
at
this
from
a
deployment
and
runtime
perspective,
there's
three
things
at
play:
the
first
is
kubernetes.
D
Cluster
itself
exists
in
some
network
and
one
right,
the
machines,
it's
the
network
that
we
are
deploying
the
cluster.
All
the
nodes
form
the
boundaries
of
the
cluster
right,
so
the
ball
nodes
have
IPS
in
that
network,
Enma
all
the
parts
and
service
IP
that
kubernetes
talks
about
or
what
we
typically
call
cluster
IPS,
which
is
this
IP
space.
We've
called
n2
over
here
right
and
the
boundary
between
n1
and
n2.
That
kubernetes
says
must
word
is
the
node
boundary
right?
This
is
your
node
port.
D
This
is
load
balancer
type,
IP
e,
getting
read
out
being
added
back
into
the
parts,
so
those
are
the
things
that
kubernetes
expects
to
work,
and
so,
as
we
all
know,
Service
IPS
by
default
are
just
closed,
low,
Kalan.
The
way
you
export
a
service
outside
the
cluster
is
a
load
balancer
ingress
or
a
node
port
which
essentially
go
through
the
node
boundary.
D
What
kubernetes
does
not
specify
is
what
other
interactions
can
exist
between
n1
and
n2
and
where
we
see
the
challenges
is
how
n1
and
n2
itself
interact
and
the
way
we
can
think
of
that
is
in
terms
of
deployment
models
like
how
do
you
deploy
kubernetes
right
and,
as
we
all
know,
like
there's
only
two
hard
problems
in
computer
science?
Naming
things
invalidation
and
off-by-one
errors,
and
so
to
think
about
deployment.
What
we
wanted
to
do
was
figure
out.
How
do
we
reason
about?
D
How
do
we
talk
about
the
ways
in
which
n1
and
n2
interact?
Then
we
decided
to
name
these
things
and
the
first
mode,
which
is
what
we
call
flak
mode,
and
this
is
what
you
typically
see
in
a
lot
most
cloud
environments,
where
the
pod
ip's
form
a
flat
address
space
with
the
V
PC
or
the
network
that
you
belong
to,
which
means
part
IP,
is
are
directly
audible
from
outside
the
cluster
pods.
When
they
talk
to
an
IP
address
the
class,
not
by
the
note
boundary,
they
just
go
out
a
Spotify's.
D
This
is
on
top
of
everything
else,
and
kubernetes
does
for
note
port
and
all
of
that
stuff.
The
second
mode
is
what
we
are
calling
an
island,
an
island
mode.
Is
you
say
the
cluster
IPS
don't
necessarily
route
from
outside
the
cluster
right,
so
they
are
only
audible
inside
the
cluster.
But
all
services
that
are
accessed
or
created
in
the
cluster
can
be
exposed
through
one
of
these
node
boundaries,
load,
balancer,
node,
port
whatnot.
You
have,
and
this
braids
off
drag
quad
connectivity
with
just
service-oriented
exports
and
it
kind
of
seems
to
imply.
D
Oh,
we
now
have
a
different
set
of
IPs,
but
not
really.
It
also
has
its
own
challenges
and
constraints
that
might
not
be
obvious
and-
and
we
will
go
in
through
some
of
those
details.
So
this
is
the
context
around
which
we
are
talking
about
what
it
means
to
enact
kubernetes
to
a
constraint
IP
address
in
turn
on.
C
So
such
I
talked
about
the
the
demand
you
know
demand
problem
the
different
resources
that
are
requesting
IP
from
a
supplies.
Side
of.
Let
us
talk
about
the
supply
side
of
the
problem
as
well
and
look
at
you
know
what
are
some
of
the
consumption
and
you
know,
deployment
patterns
because
of
which
you
know
the
availability
of
my
pieces.
First
and
foremost,
you
know
the
communities
that
option
is
growing.
So,
as
you,
you
know,
start
increasing,
more
and
I,
see
my
great
more
and
more
were
close
to
kubernetes.
C
You
are,
you
know,
as
in
essence,
you
are.
You
are
asking
for
more
IPs,
hybrid
and
multi
cloud
adoption.
We
have,
you
know
talking
to
customers,
they
start
off
with
you
know
on
Prem
they
go
to
cloud
provider,
one
cloud
provider
to
and
by
the
time
you
know
they
spread
their
footprint
around
the
the
last
cloud.
By
the
time
they
go
to
the
last
clause.
They
don't
have
enough
IPS
to
to
kind
of
provision
on
the
on
that
particular
cloud
organizational
challenges.
So
you
know
this
is
you
know
who
is
provisioning?
C
The
infrastructure
who
is
creating
this
cluster
who's
running
the
application
and
what
we
are
seeing
is
sometimes
there
is
a
new
team
that
is
created
for
for
cloud
deployments
and
there
is
a
inefficiency
or
not
a
good
coordination
amongst
of
in
teams
when
it
comes
to
you
know,
finding
a
free
block.
So
this
leads
to
you
know
issues
as
well
in
terms
of
newer
technologies.
C
You
know
adoption
of
new
technologies
which
require
a
flat
surface
mesh
where
in
the
pod
needs
to
connect
directly
to
or
talk
to
directly
to
another
part,
whether
it
be
in
the
same
cluster
or
a
cross
cluster.
And
lastly,
there
are
certain
applications
that
that
want
direct
port
connectivity
for
for
stickiness
right.
So
these
are
some
of
the
consumption
and
deployment
patterns
that
that
asks
for
a
non-overlapping
IP.
C
Let's
look
at
some
of
the
solutions,
so
we
will
talk
about
different
categories
of
solution
to
solve
these
problems,
and
you
know
I'll
touch
upon
the
the
the
Y
or
the
ven.
The
solution
can
be
used
in
such
as
going
to
cover
the
the
the
how
of
the
solution.
So
we
talked
about
you
know.
Such
I
talked
about
how
every
part
requests
an
IP,
the
kresson.
C
The
problem
is,
we
need
to
make
certain
assumptions
about
the
the
the
pod
per
node
beforehand,
so
part
per
node
is
the
power
density
or
the
number
of
positive
running
per
node.
If
we
choose
it
as
each
pod
requires
an
IP,
if
we
give
a
small
block
to
a
node
for
the
board
assignment,
it
will
basically
means
that
your
cluster
is,
you
know,
convinces
is
going
to
be
pretty
small
and
you
can't
grow.
C
C
There
are
multiple
reasons
as
as
described,
you
want
to
make
sure
that
you
don't
have
a
lot
of
parts
when
it
comes
on
on
a
part
on
a
per
node
basis,
in
consideration
for
the
resources
and
CPU
utilization
edge,
compute
cases
where
you
are
looking
for
a
low
footprint
and
lastly,
from
a
high
availability
perspective.
What
we
see
is
a
lot
of
customers
are
also
saying
instead
of
a
big
large
cluster,
I
am
going
to
have
multiple,
smaller
cluster.
This
is
an
analysis.
C
It's
an
unscientific
sampling,
but
what
we
see
is
more
than
85%
of
the
clusters
are
running
with
more
than
16
watts
per
node
right.
What
it
translates
to
is
on
the
table
on
the
on
the
right
hand,
side
where
you
can
see
if
we
have
/
24
assignment,
irrespective
of
what
your
power
density
is
and
what
is
your
any
features
aside
or
based
on
your
power
density
right?
If
we
choose
a
site
or
based
on
the
power
density,
you
can
have
anywhere
from
50%
saving
to
about
99
percent
saving
on
a
per
node
basis
right.
D
How
this
works
is
actually
it's
a
little
interesting,
because
we
have
to
coordinate
to
two
variables
that
are
in
two
different
systems.
The
max
number
of
pods
per
node
is
a
cubelet
conflict
right
through
the
cubelet.
Basically
exports.
How
many
parts
it
can
it
can
support
at
the
max
number
of
pods,
which
the
scheduler
honors,
and
then
we
actually
have
the
node
ipam,
which
is
doing
the
IP
address
allocation
right,
and
what
we
have
to
do
is
to
to
make
sure
that
this
math
is
always
right.
D
D
The
buffer
is
equal
to
max
pods,
which
is
you
say
you
want
to
pick
a
node
part
cider
that
can
have
at
least
two
X
the
number
of
parts
you
the
math
gives
you
the
defaults
and
you
get
out
of
the
out
of
the
box.
In
clusters,
which
is
a
you,
get
a
slash
24
per
node,
which
is
to
put
few
addresses
and
max
pods
we
don't
set
at
cubelet,
is
set
to
one
time.
D
So
it's
essentially
2x
time
right
but
be
by
being
able
to
know
upfront
how
the
deployment
works
and
and
and
what
your
intent
is.
Anyone
can
go
and
actually
tune
these
two
variables,
and
you
can
also
do
it
but
note
pool,
for
example,
if
you
wanted
it,
and
not
just
globally
across
the
cluster
and
get
fair
amount
of
optimization
in
the
net
IP
allocation,
the
cluster,
without
changing
anything
else
about
how
kubernetes
clusters
are
working
right.
The
next
thing
that
we
run
into
is
yeah.
C
This
ranges
yeah,
so
you
know
this
is
about
the
the
solution
spaces.
How
do
you
dynamically
increase
your
cluster
and
how
do
you
use
discontinuous
this
contiguous
cider
in
your
network?
So
the
reasons
for
these
are
you
know
the
customers
are
migrating
a
few
book
loads
at
a
time.
And
what
happens
is
you
know?
Suddenly
the
gravity
on
the
cloud
increases
and
you
want
to
dynamically
increase
the
cluster.
You
don't
want
to
destroy
your
cluster
and
then
recreate
a
new
cluster
so
that
you
can
create
a
bigger
cluster.
C
The
other
scenario
is
customers
are
seeing
an
uptake
of
their
service
or
there
is
a
Black
Friday
event
coming
up
and
you
want
to
expand
your
cluster.
You
want
to
do
that
dynamically
without
creating
another
cluster.
You
don't
want
to
disturb
an
existing
cluster,
and
you
know
you
have
other
consideration
considerations
when
you're
running
Multi
cluster
right,
so
you
don't
want
to
kind
of
go
in
to
get
into
that.
So
again,
how
do
you
dynamically
increase
your
class
area?
And,
lastly,
I
think
we
alluded
to
this
before?
How
do
you
use
a
fragmented
range?
C
D
And
the
solution
to
this
again,
interestingly,
involves
something
that's
deep
inside
kubernetes.
So
if
you
look
at
at
kubernetes
IP
table
rules
in
even
ipbs
as
a
bunch
of
rules
that
try
to
determine
whether
the
cluster
traffic
or
other
traffic,
a
packet
originated
from
inside
the
cluster
or
not
right
and
kubernetes
makes
assumptions
because
of
the
n1
and
n2
that
we
talked
about
that.
If
your
traffic
did
not
originate
from
inside
the
cluster,
you
want
to
s.
D
At
that
point,
you
can
just
ask
the
cluster
grows,
keep
adding
fragmented
sizes
of
ranges
and
and
keep
making
a
cluster
order.
Whatever
addresses
you
have,
instead
of
having
to
find
a
large
contiguous
block
of
that,
and
and
in
this,
we
believe
it
will
be
a
fundamental
driver
to
actually
taking
a
lot
of
the
stress
about
how
do
I
predict
what
size
cider
do
I
ever
need
on
the
cluster
like
with
this,
we
don't
have
to
worry.
D
You
can
start
with
what
you
have
and
add
later,
if
you
think
it's
wrong,
which
then
leads
to
the
next
one,
which
is
this
little
transition
so
far,
because
everything
that
we've
did
so
far
assumed
the
flat
model
of
addressing
where
we
said
the
pod
IPS
are
the
same
as
the
network
IPS,
and
now
we
are
trying
to
transition
into
the
second
models
where
we
have
islands
and
different
kinds
of
islands.
So.
C
In
terms
of
you
know,
running
clusters
and
islands,
the
biggest
advantage
is
the
ability
to
use
reuse
IPS
across
multiple
clusters
right.
Why
would
someone
want
to
do
that
or
when
somebody
would
want
to
do
that?
Is
you
know?
Customer
wants
to
emulate
their
existing
network?
You
know
based
on
VLAN
segmentation
or
routing
segmentation,
so
they
want
to
run
an
island
mode.
C
They
want
to
do
Network
segmentation
on
purpose,
because
you
know
having
one
flat
network
and
running
multiple
clusters
had
some
has
some
fate
sharing
issues,
so
from
that
perspective
they
want
to
run
multiple
clusters
and
the
lastly.
Lastly,
one
of
the
other
reasons
is
clusters
are
self-serving
in
that
there
is
not
there's
no
need
for
inter
cluster
communication,
and
you
know
anything
that
a
cluster
wants.
It's
it's,
it's
an
outbound
communication
out
to
internet
or
somewhere.
So
these
are
some
of
the
drivers
for
creating
an
island
more
clustered
again.
D
And
now,
when
you
say
Cambridge,
it
depends
on
what
your
context
is
right.
So
in
here
are
you,
as
you
could
say,
Cambridge
any
good
third
Massachusetts
or
you
could
say,
Cambridge
and
actually
mean
England
right,
and
there
are
two
different
Cambridge's
and
way
where
you
end
up
going
to
is
a
function
of
where
you
start
from,
which
is
what
you
run
into
with
Island
more
because
any
address.
D
D
But
it
cannot
overlap
with
any
address
space
that
actually
needs
to
be
rounded
outside
the
cluster
from
inside
the
cluster
right
and,
as
we
already
talked
about
all
communication
happens
through
the
road
boundary,
either
through
node,
port
or
load
balancers,
which
again
either
do
doing
under
forwarding
or
node
ports,
but
essentially
hit
the
note
boundary.
So
the
only
cost
it's
visible
from
outside
the
cluster,
in
addition
to
the
the
network
being
a
hole,
is
essentially
the
IPS
for
the
doors.
C
C
C
Your
communication
that
that
you
would
do
is
based
on
some
kind
of
a
proxy
that
you
set
up,
or
you
know
something
that
you
can
leverage
from
the
cloud
provider
and
I
will
talk
about
the
the.
How
of
that
and
lastly,
there
are
certain
you
might
have
certain
concerns
about
ranges
that
are
being
used.
So
we
have
customers
who
are
saying
you
know:
I,
have
this
non
RFC
1918
that
I'm
using
I
have
to
keep
it
in
an
island
or
in
a
in
a
contained
environment,
because
the
cloud
doesn't
support
it.
Other
whale
orders.
C
D
Essentially
leads
to
the
next
pattern,
which
is
to
use
IP
mask
agent
to
have
certain
networks
behave
like
islands.
Remember,
the
only
thing
that
that
makes
something
an
island
mode
is
all.
Communication
is
through
the
node
boundary
right.
It
doesn't
actually
say
anything
beyond
that
in
terms
of
IP
address
overlap,
and
then-
and
this
is
useful
because
with
IP
mask
agent,
you
can
then
basically
masquerade
ranges
of
traffic
that
leave
the
cluster
by
the
node
boundary,
which
allows
us
to
essentially
have
our
cake
and
eat
it
too,
which
is
in
a
cloud
environment.
D
For
example,
you
could
have
flat
IP
within
the
V
PC,
so
you
can
use
features
like
Meg's,
for
example,
on
GCP
that
do
direct
load
balancing
to
the
part
IPs
and
things
like
that
using
the
flat
address
space.
But
then
the
cluster
is
used
to
host
services
that
are
essentially
accessed
from
on-premise
environment,
say,
for
example,
with
IP
mask
agent.
We
can
make
sure
that
all
of
that
access
essentially
acts
like
an
island,
more
access,
so
essentially
the
node
IP
is
then
become
a
boundary
for
all
on-prem
access,
but
then
for
the
cluster
cider.
D
D
So
this
is
a
way
to
actually
have
the
ability
to
trade
off
for
certain
use
cases
ability
to
behave
like
a
plan
type
flight,
I
P,
address
space
and
for
other
cases
where
actually
the
IP
address
is
a
constraint
to
build
a
notion
of
a
mile
and
mode
like
axis
using
existing,
which
then
goes
leads
us
into.
What
do
we
do
these
patterns
like?
Where
do
we
go
next?
So
we
have
some
couple
of
ideas,
and
this
is
where
we
want
to
invite
community
discussion
and
Signet
working.
Please
come
party
vintage
mark.
D
If
you
have
other
ideas,
please
come
join
us
and
the
first
thing
is
what
I
is
a
little
too
before,
which
is
the
cap
that
we
have
out
there
to
actually
find
a
different
way
to
identify
a
cluster
networking
traffic?
The
key
insight
here
was
this:
the
way
communities
networking
works
is
then,
and
a
packet
leaves
a
node
before
it
leaves
a
load.
You
have
to
determine
whether
you
have
to
masquerade
or
not
right
once
you've
left
the
road.
D
D
It
doesn't
have
to
be
contiguous,
can
be
disjoint
and
you
can
grow
the
cluster
with
different
kind
of
IPs.
We
kind
of
so
this
idea
is
out
there.
We
would
like
participation
to
see
there
is
problems,
sympathy
with
this
or
or
we
would
like
to
see
if
this
makes
sense,
all
other
kinds
of
I,
Pam
or
other
implementations
of
cluster
networking
cube
procs
implementations,
the
future
kind
of
try
and
honor
this
kind
of
a
pattern
right,
so
it
just
becomes,
it
doesn't
become.
That
is
this
model
only
works
if
using
IP
tables
or
ipbs
right.
D
So
if
there
is
two
parts
with
two
different
namespaces
of
node
n1
and
they
both
egress
out,
they
will
both
come
as
node
n1
IP.
When
you
see
from
outside
the
cluster.
With
your
IP
address
perspective,
you
have
lost
essentially
valuable
information.
What
the
source
range
of
does
it
make
sense,
and
this
is
the
open
question
for
debate
and
we
would
like
to
I
will
join
the
Sigma
network
America
to
have
this
is
egress
a
missing
concept
in
kubernetes.
D
In
addition
to
English,
ingress
is
very
well
defined,
egress
is
not
other
than
just
saying:
masquerade
be
no
boundary
and,
and
the
the
problem
is,
is
even
more
interesting
because
since
I
associated
between
services
and
pods
is
dynamic,
just
on
label
selection,
it
can
be
possible
that
a
part
is
participating
in
two
different
service
definitions.
So
you
could
have
two
different
increases
to
the
same
part
and
when
you
aggress
from
the
prod,
you
cannot
go
by
what
suppose
I
belong
to.
You
actually
have
to
have
a
different
association
between
a
what
you
want.
C
So
this
is
something
you
would
be.
You
know,
no.
You
should
have
noticed
that
we
haven't
touched
upon
yet.
But
this
is
the
you
know,
elephant
in
the
room.
So
to
say
you
know
be
there.
What
do
we
do
about
ipv6?
There
is.
There
is
dual
stack,
but
there
is
dual
Sakai
pv6
support,
but
what
that
requires
is
end-to-end,
ipv6
connectivity
right.
If
you
have
an
ipv6
environment
and
ipv4
environment,
they
cannot
talk
to
each
other
by
default.
So
going
back
to
such
a
point,
you
know
there
is.
C
There
are
two
approaches
right:
either
we
make
a
dual
stack
work
end-to-end.
We
have
some
kind
of
a
model
where
or
we
have
some
kind
of
a
model,
but
there
is
a
there's,
a
protocol,
translation
v6
to
P
for
translation
going
on.
So
again
the
request
would
be
to
come
and
come
to
the
Signet
working
and
you
know
kind
of
help.
Us
you
know,
drive
the
solution
towards
this.
There's
also
a
talk
by
Tim
tomorrow
to
mark,
in
his
keynote
he's
going
to
talk
about
dual
dual
stack:
support
for
kubernetes
for
the
pods.
C
Thank
you.
What
I
the
take
away
for
us
is
based
on
all
this
different
diplomas,
different
cuts
talking
to
different
customers,
different
different.
You
know,
network
types
that
is
not
a
single
there's
known
as
this
is.
This
issues
cannot
be
solved
with
a
single
or
a
silver
bullet
solution.
We
require
multiple
solutions
to
kind
of
help,
different
kinds
of
customers,
so
so,
if
you
have
missed
out
on
any
other
use
cases,
any
solutions,
please
do
let
us
know.
E
My
name
is
Anil
I'm
from
visa,
so
we're
exactly
running
into
the
same
problem,
but
the
extreme
use
case.
What
we
have
is
we
allocate
IP
pools
our
IP
subnets
at
namespace
level,
not
just
beyond
even
node
right.
That
makes,
if
you're
not
using
anti
sub
subnet
or
I
people
you're
already
wasting
quite
a
bed
right,
but
for
the
for
the
auto-scaling
you
need
some
kind
of
buffer.
So
what
we
are
thinking
is
most
of
the
network,
we're
on
frame
we
use
calico.
Most
of
the
security
tools
are
not
label
based.
E
We
were
thinking
to
go
view,
Network
policy
flat
and
everything
that
were
policies
and
all
of
that,
but
unfortunately,
security
to
us
are
still
IP
based
to
understand
source
and
destination.
What
not
right!
So
one
of
the
thing
differently.
We
are
looking
at
the
egress
using
NYE
get
away
as
the
egress
I
know.
You
touched
up
a
little
bit,
but
can
you
shell
it
more
ideas?
Why
do
you
think
the
egress
is
not
a
right
way
or
what
are
the
limitations
with
that.
D
Maybe
III
didn't
come
across
Scalia,
I,
didn't
say
egress
or
I
didn't
mean
to
say:
egress
is
not
the
right
way.
I.
The
only
thing
I
was
pointing
out
is
that
egress
I
think
is
independent
of
the
ingress.
There
are
a
lot
of
use
cases
where
just
people
assume,
because
we
have
an
ignorance
IP.
We
could
just
use
that
as
an
eClass,
IP
and
I
think
that
is
not
necessarily
the
most
portable
way
of
thinking
about
it.
D
I
completely
agree
with
you
that
having
an
egress
as
a
first
class
concept-
and
maybe
it's
equal
Esprit
namespace
here
use
case
right.
So
you
know
when
it
comes
from
that
ingress
IPS
from
that
namespace
and
then
security
dudes
can
then
make
that
assumption
to
say,
because
it
came
from
that
egress
IP
I
know
it's
a
namespace
and
have
rules
around
and
and
and
all
that,
while
I
was
calling
out,
which
is,
we
don't
have
a
definition
of
ingress.
C
To
your
other
point,
I
know:
you
talked
about
on-prem,
but
the
baby
are
solving
this.
This
visibility
or
visualization
problem
is
visualization.
Problem
is
by
doing
VPC
flow
logs
and
offering
annotations.
You
know,
GK
specific
annotations,
so
you
know
you
have
more
richer
information
about
who
is
talking
to
over
here.
F
D
Neither
actually
the
the
service
side
are
actually
think
it
was
an
interesting
question
and,
yes,
we
didn't
touch
base
upon
it
because
maybe
I
briefly
walk
through
it.
When
we
look
at
what's
driving
the
IP
address
constrain,
it
is
actually
part
I
piece.
The
service
IP
is
just
defined.
The
number
of
services
that
you
can
have
it
is
doesn't
depend
on
how
many
nodes
you
have.
It
doesn't
depend
on
how
many
parts
you
have
wait.
Do
you
have
a
nice
24
first
service
cider?
It
means
you
can
have
up
to
255
services.
D
Obviously
you
lose
a
few
IPS.
By
default,
you
lose
a
dot
dot,
one
for
the
master
axis.
You
use
dot,
ten
for
the
DNS
axis,
and
things
like
that.
But
essentially
the
only
things
that
was
IP
determines
is
the
number
of
services
given
that,
in
most
cases,
service
number
of
services
and
orders
of
magnitude
lower
than
the
number
of
parts
that
has
never
been
the
constraint.
That's
driving
this.
B
This
is
not
just
a
use
case
for
well-behaved
applications.
We
had
nodes
that
were
on
top
of
Debian.
There
was
a
Debian
bug
where,
when
the
node
restarted,
it
didn't
release
the
IP
address,
the
net
effect
of
it
is
is
sucked
up
every
IP
in
this
in
the
subnet
for
the
DHCP
and
took
the
whole
thing
down.
B
So
there
needs
to
be
a
more
rigorous
scheme
about
provisioning
and
de-provisioning
where,
for
instance,
if
you
have
pods
of
a
certain
service
type,
you
can
claim
those
to
be
within
maybe
a
certain
site
arranged
within
a
very
certain
VLAN.
So
you
have
to
do
some
very
structured
training
of
these
things
to
manage
IPs,
especially
in
a
constrained
a
network
space
yep.