►
Description
Tim Hockin presents a webinar where he looks at different models for integrating Kubernetes into your network in both single-cluster and multi-cluster environments. He looks at IPs, gateway configurations, and how to navigate security boundaries, describing pros and cons of each solution, so that developers can make the best choice for their particular environment.
Presenter:
Tim Hockin, Principal Software Engineer @Google
A
Okay,
let's
get
started.
Thank
you.
Everyone
for
joining
us
today
welcome
to
today's
cncf
webinar
kubernetes
network
modules.
Why
is
this
so
dang
hard,
I'm
jerry,
fallon
and
I'll
be
moderating.
Today's
webinar
I'd
like
to
welcome
our
presenter.
Today,
tim
hawkin
principal
software
engineer
at
google
just
a
few
housekeeping
items
before
we
get
started
during
the
webinar.
You
are
not
able
to
talk
as
an
attendee.
There
is
a
q,
a
box
at
the
bottom
of
your
screen.
A
Please
feel
free
to
drop
in
your
your
questions
in
there
and
we'll
get
to
as
many
as
we
can.
At
the
end.
This
is
an
official
webinar
of
the
cncf
and,
as
such
is
subject
to
the
cncf
code
of
conduct.
Please
do
not
add
anything
to
the
chat
or
questions
that
would
be
in
violation
of
the
code
of
conduct.
In
short,
please
be
respectful
of
your
fellow
participants
and
presenters.
A
B
Thanks
so
much
for
the
introduction
I
am
excited
to
be
here
today.
I
am
talking
about
one
of
my
favorite
topics,
which
is
networking
and
kubernetes.
B
This
is
this
is
a
problem
that
has
plagued
a
lot
of
people
for
a
long
time
and
people
still
five
years
after
the
interaction
kubernetes
are
still
wrestling
with
the
model
of
how
it
works
and
how
to
integrate
it.
So
I,
a
few
weeks
ago,
I
prepared
some
slides
just
for
a
reading
and
they
got
tens
of
thousands
of
views,
and
I
thought
it
was
really
interesting,
so
I
would
present
them
here
today.
B
So
before
I
jump
into
that,
let
me
introduce
myself.
My
name
is
tim.
I
work
at
google
on
kubernetes
and
gke
and
related
projects.
I
was
part
of
the
original
kubernetes
team.
I've
been
working
on
it
since
before
it
was
open
sourced,
as
tends
to
happen
with
projects
like
that
somebody
had
to
to
do
the
networking
and
I
drew
the
short
straw,
so
I
mostly
pay
attention
to
things
like
networking,
storage,
nodes,
multi-cluster,
lower
level
topics
within
the
system,
but
today
we're
going
to
be
focusing
on
the
network
model.
B
So
what
do
I?
What
do
I
mean
by
the
network
model?
So,
let's
go
back
to
basics
a
little
bit
when
you
have
a
kubernetes
cluster.
It
is
a
bunch
of
machines
and
we
tend
to
think
of
it
in
terms
of
virtual
machines.
But
the
truth
is
it
doesn't
really
matter?
I
don't
care
if
they're,
virtual
or
physical,
those
machines
are
plugged
into
some
some
network
right.
This
makes
sense.
This
shouldn't
be
surprising
to
anybody.
B
This
is
standard
machine
stuff,
the
those
machines
we
call
nodes
and
we
run
our
workloads
pods
on
those
nodes.
Kubernetes
is
interesting
because
pods
get
ip
addresses
so
pods
10.
The
question
that
we
have
here
is
how
do
pods
get
those
ip
addresses
and
how
do
they
integrate
with
the
rest
of
the
network?
B
There
are
basically
two
rules
for
kubernetes
integrations
one
all
pods
on
a
given
node
can
communicate
with
all
other
pods
on
all
nodes
without
nat.
Don't
worry
if
that
doesn't
resonate
for
you
yet
we'll
walk
through
it
and
rule
two
agents
on
a
node.
For
example,
system
demons
can
communicate
with
all
the
pods
on
that
node
and
that's
so
that
we
can
do
things
like
health
checks
and
and
system
management
agents.
I'm
going
to
focus
less
on
that
today.
B
B
Now,
in
this
case,
I'm
carving
off
a
slash
16,
which
is
64
thousand.
I
p
164
k
ip
addresses.
It's
important
to
note.
It
is
not
required
that
a
cluster
be
a
single
ip
range
and,
in
fact,
we're
seeing
more
and
more
users
today
who
are
not
doing
it
this
way,
but
it's
very
common
and
historically,
that
is
how
people
have
done
it,
and
it
makes
my
pictures
a
whole
lot
easier.
B
Here's
the
fun
part
each
node
also
gets
a
carve
out
from
the
cluster's
space
for
the
pods
that
will
run
on
each
node.
In
this
example,
I've
shown
that
two
nodes
are
each
taking
a
slash
24,
which
is
256
ip
addresses,
which
is
a
lot
and
again.
You
don't
have
to
give
every
node
24,
but
it's
easy
for
the
pictures
and
for
people
to
think
about
in
whole
whole
octets.
So
we're
going
to
run
with
this
for
this
example
as
before.
B
It's
not
required
that
nodes
have
predefined
ip
ranges,
but
it
is
typically
how
it
is
done,
and
it's
it's
going
to
make
my
drawings
a
lot
easier.
So
we're
going
to
assume
that
every
node
has
a
pre-allocated
space.
This
does
put
a
bound
on
the
number
of
pods
that
you
can
run
concurrently
on
a
given
node.
If
every
pod
gets
an
ip
address,
it's
going
to
be
able
to
only
run
256.
B
The
good
news
is
that's
a
lot,
so
when
those
pods
run,
each
of
them
gets
an
ip
address,
and
again
I
mentioned
it's.
They
get
it
from
the
node's
ip
range.
Not
always.
There
are
some
implementations
that
get
ip
addresses
via
more
dynamic
mechanisms,
but
this
is
how
it's
usually
done
and
how
we
usually
represent
it.
So
again,
each
pod
gets
an
ip
address.
You
can
see
in
this
case
the
ip
address
for
each
for
each
pod
comes
from
the
range
allocated
to
each
node.
B
B
In
short,
we
put
up
a
wall
around
the
cluster
and
we
pretend
that
things
outside
the
cluster
don't
really
exist,
or
rather
they're,
not
part
of
the
fundamental
networking
model.
So
if
I
have
a
client
outside
of
my
cluster-
and
I
want
to
talk
to
a
pod
or
a
service
inside
my
cluster,
can
I
do
it?
How
do
I
do
it?
If
I
have
a
pod
in
my
cluster,
and
I
want
to
reach
some
other
service
outside
my
cluster
say
an
auth
server
or
a
database?
B
B
So,
let's
start
with
what
I
call
fully
integrated
mode
or
flat
mode,
I'm
terrible
at
naming
things.
So
I
apologize
in
advance
they're
just
going
to
get
worse
from
here,
so
in
a
flat
mode.
You
can
see
that
each
node,
like
I
showed
before,
has
its
own
ip
address,
but
it
gets
an
ip
range
from
the
network.
B
Everyone
who's
on
that
broader
network,
so
those
clients
and
those
servers,
the
things
that
I've
labeled
other
here
they
know
how
to
deal
with
the
fact
that
each
node
has
more
than
one
ip
address,
and
they
they
understand
that.
That
might
be
that
the
routing
is
configured
statically
in
the
network,
devices,
they're
top
of
rack
switches
or
even
on
each
node,
or
you
might
use
bgp
for
advertisement
or
it
might
be
configured
deep
in
the
fabric
like
in
a
cloud
environment.
B
The
point
is
people
don't
have
to
think
about
where
those
like
addresses
are.
They
can
just
talk
to
those
ip
addresses.
Now
I
called
it
flat
mode
and
anybody
who's.
A
networking
expert
is
immediately
raising
their
hand.
This
might
be
l2
flat,
it
might
be
l3
flat.
That
is
a
different
discussion
and
I
don't
really
want
to
get
into
that
today.
There's
trade-offs,
I
don't
want
to
talk
about
cni
drivers
today
either.
Really
it's
it's
somewhat
orthogonal
to
the
discussion.
B
The
main
point
here
is
that
pods
and
everything
else
share
an
ip
space.
This
makes
communications
really
easy
to
reason
about
everything's
on
the
same
network.
There's
no
overlapping
ip
addresses.
There's.
C
B
B
This
gives
you
full
integration,
so
this
is
really
great
when
you
have
a
lot
of
ip
space.
Kubernetes
loves
ips,
and
we've
shown
here
that
we've
carved
off
a
slash
16
right.
That
is
a
lot
of
ip
space.
It's
good
when
the
network
is
very
programmable
or
dynamic.
When
you
have
control
over
the
routing
it's
great,
when
you
need
high
integration
and
performance,
there's
no
translations,
there's
no
proxies
in
between
the
clients
and
the
servers,
regardless
of
where
they
are,
and
it's
also
really
good.
B
When
kubernetes
makes
up
a
large
part
of
your
footprint
right
when
you
can
afford
vip
space-
and
you
can
justify
it
because
you're
spending
a
lot
of
your
your
budget
on
kubernetes,
not
so
good
when
you
have
ip
fragmentation
or
scarcity
of
ip
addresses,
you
can
read
that
as
brown
fields,
lots
of
established
companies
have
already
taken
the
10
8
space
and
carved
it
up
and
routed
it
around
their
infrastructure.
B
B
It's
also
not
really
good.
When
your
network
infrastructure
is
very
difficult
to
configure
or
not
very
dynamic
or
if
kubernetes
is
just
a
tiny
piece
of
your
overall
footprint,
then
this
may
not
be
the
best
model
for
you
at
the
complete
other
end
of
the
spectrum.
We
have
a
model
fully
isolated
or
air
gapped
mode.
B
In
this
model,
their
networks
are
completely
disconnected.
There
is
no
connectivity
from
inside
or
outside
the
clients.
There's
no
such
thing
as
a
client
outside
of
your
cluster
reaching
into
your
cluster.
It
just
doesn't
happen
and
within
your
cluster
you
can
still
maintain
the
kubernetes
requirements,
but
between
clusters
or
two
other
things
it
doesn't
exist.
B
In
fact,
this
is
an
interesting
model
because
you
can
reuse
all
of
the
ip,
because
everything
is
completely
disconnected.
There's
no
reason
that
you
can't
allocate
the
same
ip
addresses
in
every
cluster,
in
fact
they're,
basically
on
different
networks,
so
you
can
see
sort
of
in
this
model.
There
is
no
connectivity,
they
are,
as
the
expression
goes.
They
are
air
gap.
There
is
nothing
connecting
them.
B
This
is
really
good
when
you
don't
need
any
integration,
when
you
have
a
workload
that
needs
to
run
in
isolation,
it's
you
know,
it
applies
when
ipspace
is
scarce,
we
network
is
not
programmable.
One
of
the
reasons
people
look
at
this
model
is
because
they
care
a
lot
about
security,
and
this
makes
it
much
easier
to
reason
about
security
boundaries.
B
B
B
Some
people
have
claimed
that
this
is
the
only
way,
or
this
is
the
default
way
for
kubernetes,
that
I've
heard
people
say
quote
you
have
to
choose
which
overlay
you're
going
to
use.
Those
sorts
of
statements
are
false.
Obviously,
we've
looked
at
two
different
models
already.
That
said,
this
is
a
very,
very
common
model.
B
B
This
can
be
implemented
via
overlays.
It's
very
common
to
use
things
like
vxlan
there's
a
lot
of
products
out
there
that
will
help
you
set
up
vxlan
overlays,
but
it
doesn't
have
to
be.
It
can
also
be
set
up
via
private
routing
rules
right.
So
another
common
model
is
to
use
not
to
use
an
overlay
encapsulation,
but
to
use
straight
ip
addressing,
but
to
use
private
vgp
advertisement.
So
they're
limited
and
only
those
nodes
in
the
cluster
know
about
each
other.
B
In
fact,
in
this
model
you
can
reuse
pod
ips
in
each
cluster,
which
is
a
major
motivation
for
this
model,
so,
depending
on
what
gateway
means
for
you,
you
may
or
may
not
be
able
to
reuse
node
ip
addresses,
but
node
ips
are
very
low
count
compared
to
pot
ips
right
in
this
example,
we're
looking
at
256
times
more
ip
addresses
for
pods.
So
here
you
can
see.
I
can
use
the
same
pod
range
for
every
cluster,
because
pods
never
directly
talk
to
other
pods.
B
B
One
of
the
things
that's
happening
in
upstream
kubernetes
right
now
is
making
it
more
possible
to
run
your
clusters
with
less
monolithic
chunks,
which
will
take
away
some
of
the
value
of
this
model,
which
I
think
is
a
good
thing.
B
B
It's
not
so
good
when
you
have
to
debug
connectivity
when
you're
bringing
traffic
in
through
a
gateway,
traffic
gets
necessarily
more
complicated.
There's
gonna
be
some
form
of
translation
and
you
need
to
figure
out
what
is
happening
there.
It
becomes
much
harder
to
reason
about
what
exactly
is
happening.
It's
also
not
very
good.
If
you
need
direct
to
endpoint
communications,
there
are
some
systems
out
there
that
assume
that
clients
can
talk
directly
to
their
endpoints.
B
They
don't
use
things
like
load
balancers,
and
if
you
need
those
direct
communications,
then
this
gateway
model
starts
to
break
down.
It's
also
problematic.
If
you
need
a
lot
of
services
exposed,
the
gateways
can
be
complicated
or
sometimes
expensive,
especially
if
you're
not
using
http,
where
you
can
reuse
ip
addresses.
If
you're
using
layer,
4
services
like
tcp
or
udp,
but
not
http,
you
can
end
up
requiring
a
lot
of
infrastructure
to
bring
traffic
in
and
out.
B
B
Well,
those
gateways
become
more
complicated
now
because
you
have
a
limited
number
of
them
and
it
can
also
be
problematic
if
you
have
a
large
number
of
nodes,
because
things
like
overlays
and
route
advertisement
tend
to
scale
poorly
and
you
need
to
get
more
and
more
infrastructure.
As
you
get
larger.
B
So
I
seem
to
have
painted
that
model
in
a
bad
light,
but
it
is
a
very
common
model,
so
I
want
to
dig
in
more
into
what
gateway
really
means,
so
here's
the
simplest
form
of
gateway.
You
use
your
nodes
as
the
gateways
in
this
model.
All
the
traffic
into
or
out
of
the
cluster
is
done
explicitly
by
the
nodes
in
the
cluster,
there's
no
extra
load,
balancers
or
or
routers,
or
anything
else.
Here
the
traffic
comes
through
the
node.
B
So
this
is
like
I
said,
the
nodes
have
one
leg
in
the
main
network,
which
would
be
their
node
ip
address
here,
the
the
10
to
40
network,
and
they
have
another
interface
in
the
pod
network,
which
is
the
10
0
network.
B
So
in
in
this
case,
I'm
going
to
take
this
apart
even
further
and
talk
about
how
exactly
ingressing
risk
works
here.
So
let's
talk
about
ingress
first,
so
service
node
ports
is
something
anybody
who's
played
with.
Kubernetes
is
probably
run
up
against
and
try
to
figure
out
if
they
can
make
do
what
they
want
and
it's
a
confusing
mechanism.
So
I'm
gonna
dig
into
it
even
a
little
bit
further.
B
In
the
case
of
a
service,
node
port,
you
have
a
client
here,
128.1.1
it's
sending
a
packet
that
packet
as
it
transits
is
targeted
at
a
node
ip
address.
Remember
the
the
client
only
knows
node
ip
addresses
and
not
pod
id
addresses.
So
hopefully
you
have
a
service
that
is
load
balanced
or
or
is
only
a
single
instance,
because
I'm
sending
it
to
a
port
on
the
node.
It
doesn't
really
matter
which
node,
because
node
ports
exist
on
all
of
the
nodes.
B
So
you
need
some
mechanism
to
figure
out
where
you're
going
to
send
it
into
those
nodes
which
is
a
slightly
different
conversation.
But,
let's
assume
you
have
a
dns
or
something,
and
it's
going
to
tell
you
that
the
service
you're
looking
for
is
available
on
these
nodes
at
this
port
and
the
port
in
a
service.
Node
port
isn't
arbitrary.
You
don't
get
to
choose
port
80.
kubernetes
generally
manages
those
ports
for
you
and
it
uses
them
uses
a
high
port
range
to
generally
avoid
conflict.
So
you'll
see
ports
in
the
30
000
range.
B
When
it
arrives
at
the
node,
the
node
is
going
to
use
the
destination
port
of
the
ip
packet
to
figure
out
which
service
you're
talking
to
so.
If
you're
talking
to
port
3001,
you
might
be
going
to
service
foo
and
if
you're
talking
to
port
3002
you're
going
to
service
bar
and
it's
going
to
do
a
destination
network,
address,
translation
or
dnat,
and
it's
going
to
pick
one
of
the
back
ends
that
it
understands
as
being
part
of
service
foo
or
service
bar
depending
on
which
court
it
came
in.
B
And
it's
going
to
rewrite
that
ip
packet,
the
most
common
implementation
is
iptables,
but
we
also
have
implementations
now
in
ipvs
and
nf
tables
and
evpf,
who
are
all
able
to
perform
the
same
basic
logic.
B
We're
going
to
forward
that
packet,
and
linux
is
actually
quite
good
at
this,
so
it'll
forward
the
packet
onto
the
destination
pod
that
we
wanted
to
it
to
arrive
at
note
here.
The
client
didn't
choose
which
pod
it's
talking
to.
It
just
talked
to
the
port
and
I'm
sorry,
I'm
seeing
a
chat
in
there
an.
B
The
packet
arrives
at
the
destination
pod
now,
so
one
of
the
common
things
that
people
do
is
they
will
ingress
traffic
into
an
l7
proxy.
What
what
does
that
mean?
They'll
run
something
like
envoy
or
nginx
or
aj
proxy
inside
their
cluster.
They
will
route
all
the
traffic
through
a
node
port
into
that
and
use
that
to
do
further.
Forwarding
things
like
http
are
wonderful
because
they
have
headers
that
you
can
use
to
tell
what
did
you
actually
mean?
What
host
were
you
actually
looking
for?
B
This
is,
for
example,
the
in
cluster
ingress
controllers
that
many
people
run
so
again
going
back
to
this
diagram,
you
can
see
once
it's
arrived
at
that
pod.
I
can
do
anything
I
want
with
it
on
the
reverse
path.
The
host
is
going
to
do
the
opposite
of
that
dnat
translation
and
it's
going
to
convert
the
packet
back.
So
when
your
original
client
gets
the
the
response
when
the
packet
gets
through
when
the
client
gets
the
response,
it
seems
to
come
from
the
nodes,
node
port,
that
it
was
talking
to
originally.
B
So
the
client
is
happy
and
the
pods
are
happy
and
everybody's
happy,
except
if
you
cared
about
some
of
the
finer
details
which
we'll
get
into
in
a
bit.
So
looking
at
the
egress
side
right,
if
I
want
to
send
traffic
from
my
cluster
to
something
else,
we
often
call
this
snap
source
nat.
The
typical
model
here
is
to
use
what
linux
calls
ip
masquerade.
B
What
ip
masquerade
does
is
it
says
that
traffic
leaving
this
node,
this
machine
is
going
to
look
like
it
came
from
this
machine.
So
when
I've
got
my
client
inside
the
cluster
and
I
want
to
send
traffic
out
of
my
cluster,
we
send
the
packet
out
when
it
reaches
the
edge
of
the
node
it's
going
to
translate
into
the
node's
ip
address.
So
all
the
traffic
from
that
node
appears
to
come
from
that
nodes
ip.
B
If
you
were
tcp
dumping
it,
you
would
see
the
source
address
as
the
the
node
2
in
the
first
cluster
now,
because
we're
doing
ingress
and
egress
through
nodes
when
it
arrives
at
the
destination
cluster.
What
we
saw
happening
with
node
port
is
also
going
to
happen,
and
so
now
you
have
a
packet
whose
source
address
and
destination
address
have
both
been
modified.
It's
got
two
translations
along
the
way.
B
The
network
fabric
happened
handled
it
all
for
you
there's
a
different
model,
though,
which
uses
a
virtual
ip
address,
and
this
is
used
in
in
some
cloud
providers
to
provide
a
slightly
different
model
instead
of
having
to
know
about
a
node
which
node
and
which
node
port
to
use
which
is
kind
of
a
clunky
human
interface.
B
It's
very
similar
to
node
port,
but
instead
of
using
the
destination
port,
we're
going
to
use
the
destination
ip
to
make
those
same
routing
decisions,
so
I'm
not
going
to
re-animate
the
same
drawing,
but
the
packet
will
flow
through
basically
the
same
path.
It
will
go
through
the
same
sorts
of
translations,
but
instead
of
a
human
having
addressed
a
30
000
port,
it
can
address
a
virtual
ip
address.
This
is
more
compatible
with
things
like
typical
dns
and
open
source
software,
which
isn't
always
easily
configured
for
random
arbitrary
ports.
B
With
the
egress
path,
you
still
need
something
like
ip
masquerade
as
snat
to
to
egress
and
there's.
B
Otherwise,
it's
sort
of
behaviorally
very
similar.
Again
the
packet
will
arrive
at
the
proxy.
The
proxy
will
choose
which
back
end
it's
going
to
go
to
and
will
forward
the
packet
one
of
the
interesting
things
about
proxy
model
is
you
can
either
route
to
a
node
port,
which
is
how
kubernetes
started
with
support,
for
example,
amazon,
elbs
or
it
can
route
directly
to
pod
ip
addresses,
if
that
proxy
is
smart
enough
to
know
how
to
get
onto
the
island?
B
The
problem
here
and
in
many
of
these
models
is
the
mechanism
here
will
obscure
the
client
ip
so
in
this
case
the
proxy,
because
it
terminated
the
original
session
and
opened
a
new
session
traffic
looks
like
it's
coming
from
that
proxy's
ip
address.
So
if
you
have
a
client
assuming
if
you
have
a
server
that
needs
to
understand
the
client
ip
address,
you
have
to
pass
it
through
in
some
other
mechanism.
B
Again,
some
proxies,
like
elb,
support
things
like
the
proxy
protocol,
which
will
include
a
header
on
a
tcp
stream.
If
you're
using
http
proxies,
then
you've
got
ways
of
encoding
it
in
http
headers.
If
you
don't
have
those
things,
you
have
a
much
more
difficult
time
getting
the
true
client
ip
address
and
like
the
vip
model,
you
still
need
something
like
s
nat.
In
order
to
be
able
to
leave
the
cluster
now
I
could
probably
talk
for
another
hour
just
about
ingress.
B
In
fact,
I
have
another
slide
deck
which
I
tried
to
merge
in
here
and
then
realized.
I
was
way
over
time
that
goes
into
more
detail
about
how
these
models
work
and
the
various
trade-offs
of
them
and
and
the
the
actual
details
of
how
kubernetes
implements
them.
But
I'll
have
to
do
that
at
another
webinar.
B
B
So
we
looked
at
island
mode
and
one
of
the
variants
of
island
mode
that
we're
seeing
in
more
usage
now
is
what
I
call
archipelago
mode,
which
is
roughly
bigger
islands
or
groups
of
islands.
B
Within
the
archipelago,
the
the
model
is
effectively
the
flat
model,
which
means
that
your
multiple
clusters
can
talk
to
each
other
without
translation,
it
means
you
have
a
flat
space
and
you
carve
off
a
lot
of
ip
within
that
archipelago.
But
when
you
integrate
it
with
the
rest
of
network,
it
becomes
island
mode
again
like
island
mode.
This
can
be
implemented
as
an
overlay
or
not
as
an
overlay,
and
it
still
needs
gateways
to
come
in
and
out
of
this
again,
this
could
be
gateways
using
the
nodes.
B
B
Well,
sorry,
backing
up,
you
can't
reuse
ip
addresses
between
clusters,
but
you
can
between
archipelagos,
so
we've
seen,
for
example,
customers
who
set
up
an
archipelago
per
cloud
region
or
per
data
center,
and
they
can
economize
on
ip
space
between
those
but
still
have
high
levels
of
connectivity
within
a
particular
region.
B
Like
island
mode
itself,
if
you
need
to
debug
connectivity
between
clusters,
this
can
get
complicated.
If
you
need
those
direct-to-end
point
communications
from
outside
of
the
archipelago,
it
can
be
complicated
if
you
need
to
expose
a
lot
of
services
to
their
non-kubernetes
environment.
B
B
So
if
you
have
large
numbers
of
services,
it
can
be
a
problem
like
plane,
island
mode,
if
you're,
relying
on
client
ips
for
firewalls,
it
can
be
problematic
or
if
you
have
large
numbers
of
nodes
across
all
of
your
clusters.
Now
your
scale
limit
is
the
number
of
nodes
in
the
archipelago
instead
of
the
number
of
nodes
in
each
cluster,
so
you
can
see
there's
a
real
trade-off
to
be
made
in
this
model.
B
The
gateway
options
are
very
similar
to
plane,
island
mode,
and
so
I
won't
go
back
into
them
again,
but
I
will
look
for
an
opportunity
to
present
more
on
on
ingress
and
the
ingress
modes
later
and
I'm
hoping
that
we
will
soon
have
the
ability
to
talk
more
about
egress
gateways
and
more
egress
modes.
B
So,
of
course
you
want
to
know
which,
which
one
should
I
use
and
the
sad
truth
is.
There
is
no
right
answer.
Unfortunately,
I
don't
know
your
environment
in
the
abstract.
There's
there's
no
way
that
I
can
tell
you
which
one
is
best.
They
have
real
trade-offs
and
I've
tried
to
elucidate
some
of
those
trade-offs
here,
but
you
have
to
make
some
value
decisions
yourself.
B
If
I
could,
I
would
wave
my
magic
wand
and
give
everybody
ipv6
and
then
flat
mode
would
not
be
such
a
big
deal,
but
that
isn't
possible
yet,
and
there
are
other
considerations
beyond
just
ip
efficiency.
So
I'm
happy
to
talk
with
people
about
which
modes
they
think
make
sense
for
them,
but
there
isn't
a
magical
answer
so
I'd
like
now.
We're
we've
got
plenty
of
time
for
questions.
I'd
like
to
switch
to
questions,
because
I
think
this
is
such
a
nuanced
topic
that
the
details
will
be
fun
to
talk.
C
A
Okay
tim.
Thank
you
very
much
for
the
presentation.
We
have
plenty
of
time
for
questions,
as
tim
said.
So,
if
anyone
has
anything
they'd
like
to
ask,
please
drop
it
into
the
q,
a
box
and
we'll
get
to
as
many
as
we
can
before
the
end
of
the.
B
I
believe
they're
recording
yes,
the
question
is,
will
ever
record
be
sure
there
will
be
a
shared
recording
of
this.
I
apologize
for
the
audio.
I
will,
I
see
in
the
chat,
a
question
ways
to
look
over
the
slides.
Yes,
I
have
posted
the
slides
already
on
my
speaker
deck.
So
if
you
go
to
speakerdeck.com
t-h-o-c-k-I-n,
then
you
can
find
the
slides
there,
and
I
imagine
that
the
cncf.
B
Question
can
you
explain
island
mode
sure,
so,
let's,
let's
flip
back
in
there
to
island
mode,
since
this
is
the
the
most
common
model
that
I
see
for
larger
enterprises.
B
All
right
so
plain
old
island
mode
in
this
model,
for
example,
let
me
just
pick
on
one
particular
model.
If
you
look
at
the
two
different
clusters,
you
can
set
up
a
vxlan
overlay,
for
example.
Again
this
is
not
the
only
way
to
do
it.
B
Just
one
example:
instead
of
a
vxlan
overlay,
each
of
the
nodes
in
that
cluster
have
an
agent
running
that
understands
how
to
route
vxlan
packets
and
they
might
get
that
information
by
gossip
by
talking
to
each
other
or
they
might
get
it
through
the
kubernetes
api
or
through
some
other
configuration
mechanism,
but
they
all
know
how
to
route
to
each
other.
B
So
node,
one
in
the
top
cluster
knows
that
if
it
needs
to
talk
to
the
pod
c,
then
it's
going
to
encapsulate
the
packet
and
forward
it
on
to
node
two
right,
but
it's
only
between
the
nodes
within
that
cluster
that
that
information
is
shared.
So
if
pod,
a
in
the
top
cluster
wants
to
talk
to
pod
a
in
the
bottom
cluster,
it
doesn't
know
how
to
get
there
and
that's
why
it's
called
an
island,
because
there
isn't
there
isn't
a
bridge
between
these.
B
Two
things
they're
not
connected
to
each
other
and
that's
why
I
drew
the
edge
of
the
cluster
in
a
darker
shade
to
emphasize
that
it's
kind
of
a
barrier
within
the
cluster.
They
all
know
how
to
reach
each
other,
but
outside
the
cluster.
It
doesn't
work.
Now
I
picked
on
the
xlan,
but
there
are
other
models
you
can
use,
for
example,
bgp,
to
share
routing
information
between
those
nodes.
C
B
Go
through
the,
I
can't
find
the
follow-up,
so
nat
scalability
issues,
so
yeah,
nat
scalability
is
a
fun
question.
When
you're
doing
all
these
translations,
the
kernel
has
to
keep
track
of
which
translations,
it's
doing
so,
there's
a
tool
called
contract
connection
tracking
which
the
linux
kernel
offers
and
which
we
use
heavily
in
the
default
implementations.
Now
there
are
many
implementations.
Services
are
an
abstraction
connection
tracking
we
set
up
by
default.
B
When
you
run
cube
proxy,
we
set
up
a
large
number
of
connection
tracking
records,
so
we
make
a
lot
of
space
for
the
kernel
to
be
able
to
track
connections.
This
is
generally
fine
for
most
users
it's
fine
and
for
tcp
in
particular.
It
tends
to
not
be
a
huge
problem
because
tcp
is
connection
oriented
and
when
the
connection
closes,
we
can
immediately
clean
up
the
connection
tracking.
B
If
you
use
a
lot
of
udp
services,
it
can
be
problematic.
Udp
doesn't
have
a
connection,
so
we
have
to
time
out
udp
contract
records.
So
what
we
have
seen
occasionally
is
customers
who
have
a
high
number
of
udp
based
services
and
with
connection
tracking
records
that
are
just
sitting
on
waiting
to
be
tied
out
and
so
for
some
of
those
customers
we've
given
them
some
flags.
B
You
can
set
that
key
proxy
to
change
the
scaling
for
comedy
connection
tracking
records
that
you
create
some
other
models
like
psyllium,
as
a
replacement
for
proxy
doesn't
use,
I
believe,
to
not
use
the
kernel's
connection.
Tracking
mechanism
uses
its
own
connection.
B
C
B
Is
going
to
look
at
the
destination?
That's
the
first
decision,
which
service
did
you
need
to
talk
to
and
given
that
it's
going
to
use
I'll
talk
about
the
model
I'm
most
familiar
with,
which
is
like
tables
it's
going
to
enter
any
logic
that
says:
okay,
I've
determined
that
you
are
heading
for
service.
B
So
determine
that
you're
you're
aiming
for
service
food,
it
will
then
figure
out.
Okay
service
foo
has
some
number
of
pods
behind
it,
and
it's
going
to
try
to
choose
a
pod
for
you
to
route
to
now.
I
don't
get
into
it
very
deep.
B
B
It's
going
to
actually
route
it
to
another
node,
which
adds
some
complexity
to
the
logic
and
makes
it
that
much
more
difficult
for
humans
to
reason
about,
because
their
con
other
tcp
dump,
for
example,
will
show
multiple
connections
with
multiple
address
translations
happening
across
multiple
nodes.
This
is
the
part.
That's
really
unfortunate
about
that
model.
B
Kubernetes
has
introduced
some
parameters
like
external
traffic
policy,
which
allow
you
to
control
that
a
little
bit
more
and
limit
it
to
only
sending
traffic
to
pods
on
the
same
node,
which
is
great,
except
if
you
happen
to
have
routed
to
a
node
that
doesn't
have
any
back
ends,
in
which
case
it
will
end
up
being
a
black
hole.
B
Something
we
do
in
the
cloud
load
balancers
is
when
we
implement
the
vip.
We
have
a
health
check
on
each
node,
which
tells
us
whether
or
not
each
service
has
a
back
end
for
that
service.
On
that
node.
That
was
a
long
sentence
given
node
one
does
it
have
back
ends
for
foo?
If
the
answer
is
yes,
the
vip
will
potentially
route
to
node
one.
If
the
answer
is
no,
the
zip
will
not
route
to
node
one.
It
will
only
route
to
node
two
or
whichever
nodes
actually
have
back
ends.
B
How
do
you
increase
the
default
number
of
pods
from
100
to
254?
Like
you
mentioned,
I
know
the
ips
needs
to
be
there,
so
the
number
of
pods
you
run
on
a
node
are
governed
by
two
different
flags.
One
is
the
number
of
ip
addresses
you
allocate
to
that
node,
and
the
other
is
how
many
pods
cubelet
is
willing
to
run.
Cubelet
generally
sets
a
limit
of.
I
think
the
default
is
110
for
the
number
of
pods
on
a
node.
B
There
are
some
people,
some
use
cases
where
you
want
to
raise
that
what
we
don't
want
to
do,
what
we
discourage
people
from
doing
is
making
the
number
of
pods
and
the
number
of
ip
addresses
very
close
to
each
other.
What
happens
in
that
case
is
you
can
potentially
reuse
ip
addresses
very
quickly
and
kubernetes
being
a
distributed
system.
There's
generally
something
out
there
that
has
cached
an
ip
address,
whether
it's
dns,
which
has
you
know,
time
time
to
live
measured
in
seconds,
but
still
not
instantaneous.
B
So
we
we
generally
say.
However
many
pods
you
want
to
run
on
a
node.
Your
ip
addresses
should
be
about
twice
that
big.
In
fact,
what
we
do
by
default
in
gke
is,
when
you
tell
us
how
many
pods
you
want
to
run
on
a
node
we
round
up
to
a
power
of
two
and
then
double
it.
So
if
you
want
to
run
110
pods,
that's
going
to
round
up
to
128,
which
double
this
256.
B
many
users,
most
users,
in
fact,
don't
need
to
run
100
pods
per
node,
and
they
can
actually
dial
that
in
the
other
direction
and
set
it
lower.
So
if
they
say,
I
only
need
to
run
32
pods
per
node,
then
we
round
that
up
to
32
and
we
double
it
to
64
and
they
get
four
times
as
many.
C
B
C
B
B
B
So
if
you
look
at
a
model
like
island
mode,
where
you
might
potentially
reuse
those
ip
addresses,
there
is
simply
no
way
that
you
can
have
a
pod
directly
speak
to
another
pod,
because
it
can't
tell
whether
it's
talking
to
its
own
private
version
of
that
ip
address
or
to
the
other
clusters
version
of
that
ip
address.
B
Would
ipip
overlays
work?
Any
overlay
should
work.
If
you
can
route
the
traffic
into
the
other
cluster
deterministically
vxlan
is
the
one
that
has
sort
of
risen
as
very
popular
right
now,
because
it's
a
fairly
simple
overlay,
but
people
are
certainly
doing
it
with
gre
or
other
encapsulation
mechanisms.
B
Could
you
give
some
idea
on
node
pool
what
is
the
use
case
for
more
than
one
node
pool?
Sure
node
pools
is
not
an
official
kubernetes
concept,
but
it's
implemented
by
many
of
the
kubernetes
providers.
It's
just
a
group
of
machines
that
have
something
in
common,
so
I
can
speak
to
the
way
gke
implements
it.
B
B
B
Questions,
how
do
you
increase
the
default
number
of
plot?
Oh,
that
we
did
that
one?
What
are
some
of
the
challenges
trade-offs?
You've
observed
when
you
correlate
these
models
with
low
latency
packet
processing.
B
The
short
answer
for
low
latency
packet
processing
is
the
more
things
that
have
to
touch
the
packet,
the
worse
off
you're,
going
to
be
right.
That
seems
pretty
self-evident
if
you're
bouncing
through
proxies
you're,
going
to
have
generally
higher
latency
than
if
you're,
not
and
so
for,
customers
who
need
who
really
focus
and
care
about
low
latency
packet
processing,
we
encourage
them
to
go,
have
fully
integrated
flat
mode
as
much
as
they
can,
and
sometimes
that
means
having
different
mixed
models.
B
Question
are
gateways
managed
by
kubernetes.
Only
no,
absolutely
you
can
do
whatever
you
want
to
make
those
gateways.
Work.
Kubernetes
happens
to
have
some
abstractions
that
make
it
relatively
easy
to
set
up
load
balancers
in
various
cloud
environments,
but
you
kubernetes
is
wonderful
in
this
way.
There's
an
api
and
there's
the
implementation
and
they
are
distinct
and
you
can
implement
your
own
again
quote
cloud
provider
via
whatever
mechanism
you
want.
B
If
you
have
some
load
balancing
infrastructure
that
you
like
or
some
creative
traffic
routing
that
or
network
fabric
that
you
can
leverage,
you
are
welcome
to
do
your
own
thing.
Kubernetes
gives
you
lots
of
interesting
api
mechanics,
so
you
can
watch
for
updates
and
implement
your
own
controllers
and
in
fact
all
of
our
controllers
are
open
source.
So
you
can
embrace
and
extend
the
controllers
that
we've
already
written
to
work
in
your
own
environments.
B
How
would
flannel
versus
calico
work,
which
pattern
are
they
flannel
and
calico?
Are
island
modes
generally
calico
being
a
bgp
model?
Well,
actually,
a
mixed
model
now,
but
primarily
was
focused
on
bgp.
B
It
can
integrate
with
the
larger
network,
but
it
also
can
not
depending
on
where
you're
going
to
route
those
advertisements,
flannel
or
canal,
are
vxlan
implementations.
So
those
are
generally
island
mode.
Implementations
do
you
know
any
way
to
simplify
this
model
and
remove
the
dependency
on
that?
Do
you
see
service
mesh
solutions,
adding
value
to
the
network
layer
indicates
or
the
opposite.
B
So
nat
is
one
of
the
implementations
that
we
use
for
cube
proxy
right.
If
you
wanted
to
use
something
like
a
service
mesh
instead
of
a
cube
proxy,
that
is
a
completely
valid
implementation
and
it
wouldn't
need
nat
in
the
same
way
like
it
doesn't
rely
on
the
kernel's
nat,
but
it's
going
to
be
doing
effectively
translation
in
the
proxy,
so
you
still
have
some
amount
of
state
that
is
stored
as
your
routing
pockets
from
around
I'm
a
big
believer
in
funny.
I
was
just
talking
about
this
this
morning
on
twitter.
B
I'm
a
big
believer
in
service
mesh
and
the
idea
that
eventually
most
users
will
want
some
of
what
service
mesh
offers.
Probably
most
users
will
not
want
all
of
it,
but
most
users
will
want
some
subset
of
it
and
so,
as
service
meshes
become
more
powerful
and
more
easy
to
use.
B
C
C
B
B
Virtualization,
sorry,
that
wasn't
the
question
as
a
follow-up,
I
think.
Okay,
do
you
have
references
for
setting
up
these
different
modes,
technical,
configs
and
tools.
A
B
The
references
are
generally
in
the
implementation,
so
there
is
there's
documentation
on
setting
up
flannel
and
canal
for
island
mode
and
calico
has
lots
of
great
docks
for
setting
up
calico
the
default
kubernetes
implementation
kind
of
assumes
flat
mode,
but
it
doesn't
assume
it
very
hard.
It
just
doesn't
assume
anything
about
networking
other
than
the
kubernetes
networking
model,
which
means
pods
can
talk
to
other
pods
yeah.
The
kubernetes.
B
B
Suggestions
on
local
platforms
to
explore
experiment
with
case
networking
options.
I
am
a
big
fan
of
kind
myself,
but
kind
builds
unknocker
and
there's
only
so
much
you
can
do
with
it
before
you
end
up
running
into
things
that
the
kernel
lets
you
do,
which
you
might
not
be
able
to
do
in
a
real
network
model.
So
I
would
I
personally
use
kind
for
testing
stuff
out
when
I
would
play
with
like
l2
space.
B
I
use
google
cloud
when
I
want
to
play
with
a
non-l2
space
is
cube
proxy
based
on
iptables
qproxy
has
several
modes
built
into
it.
One
of
them
is
iptables.
That
was
the
default,
and
maybe
still
is
the
default
and
most
common
model.
B
Increasingly,
people
are
using
ipvs
as
the
implementation,
which
is
a
somewhat
optimized
path
through
the
kernel,
but
very
similar
in
internal
mechanisms.
Very
different
in
configuration
there's
also
a
user
space
mode
which
hardly
anybody
uses
anymore,
and
there
are
now
there's
been
proposals
to
add
nf
tables
directly,
although
I
think
we're
going
to
see
a
standalone
replacement
for
cube
proxy
that
uses
nf
tables
and
psyllium,
for
example,
uses
evpf
and
rather
than
folding
all
that
into
one
giant
q
proxy
binary.
I've
been
encouraging
people
to
build
their
own
cube
proxy
replacements.
B
In
most
implementations,
you
can
use
the
nodes
ip
and
node
port,
but
I'm
not
sure
why
you
would
want
to.
Unless
you
have
some
weird
consistency
requirement.
Node
ports,
very
frankly
were
designed
to
build
higher
level
load
balancers.
So
when
people
are
using
node
ports
directly,
it's
for
me
it's
a
smell
like
I
feel
like
you're
missing
a
piece.
C
B
B
They
do
not
work
very
well
in
island
mode,
because
you,
if
you
have
five
replicas,
you
need
five
ip
addresses
to
address
them
or
five
ports
to
address
them,
and
the
problem
is
the
number
of
replicas
can
change
dynamically
and
it's
much
slower
to
program
load,
balancing
infrastructure
than
it
is
to
program
within
the
cluster,
and
so
in
general,
if
you
have
headless
services.
This
is
what
I
said
about
direct
connectivity
direct
to
end
point
connectivity.
B
It
does
not
map
very
well
into
highland
modes
and
I'm
told
I
have
no
minutes
left,
I'm
happy
to
do
more
questions
via
whatever
mechanism
people
can
find
me
I'm
available
on
twitter
on
slack
on
github,
on
whatever
medium,
I
want
to
also
throw
a
quick
shout
out.
We
have
another
ambassador
webinar
coming
september.
25Th
caitlyn
is
doing
and
bowie
we'll
be
talking
about
the
evolution
of
the
ingress
api,
which
we
just
touched
on
barely
today
and
it's
new,
hopefully
replacement
called
gateway.
A
Okay,
well,
thank
you
very
much
tim
for
a
wonderful
presentation
and
thank
you
all
so
much
for
your
participation
as
well.
As
I
said
before,
today's
presentation
and
slides
will
be
available
on
the
cncf
website
later
today.
That
is
all
the
time
we
have
for
today.
Thank
you
very
much
for
joining
us
today.
Everyone
and
stay.