►
Description
Don’t miss out! Join us at our upcoming events: EnvoyCon Virtual on October 15 and KubeCon + CloudNativeCon North America 2020 Virtual from November 17-20. Learn more at https://kubecon.io. The conferences feature presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Kubernetes Networking Intro and Deep-Dive - Bowei Du & Tim Hockin, Google
Networking is less complicated than you think! This session is a combined intro and deep dive. This talk will start with some background on Kubernetes networking. Attendees who are not already comfortable with the "hows and whys" of basic networking in Kubernetes can get a bit of a primer before we dive deep on a few of the more recent developments and efforts in the networking space.
https://sched.co/ZewF
A
B
B
B
This
is
for
you,
the
second
half
of
this
presentation
will
be
a
deep
dive
and
really
we're
going
to
look
at
sort
of
the
new
developments
that
are
happening
in
the
sig,
so
this
will
be
a
deeper
look
at
some
of
the
newest
work
that
the
sig
has
been
doing
if
you're
comfortable
with
kubernetes
networking
concepts
and
want
to
see
what's
next.
This
is
for
you,
and
especially
if
you
want
to
influence
the
direction
of
the
project,
we
highly
recommend
you
listen
to
this
presentation
now.
B
So
kubernetes
project
is
split
into
multiple
special
interest
groups,
which
focus
on
various
aspects
of
the
system
and
we
break
the
project
problem
into
aspects
that
are
covered
by
these
special
interest.
Groups
and
sig
network
covers
basically
networking
for
the
cooper's
netize
project.
This
is
our
charter.
We're
responsible
for
kubernetes
networking
between
various
components,
including
pod,
networking
within
and
between
nodes
service
abstractions,
both
for
ingress
egress
and
within
the
cluster,
and
then
also
network
policies
or
security
policies
and
access
control
below
is
a
link
to
our
zoom
meeting
and
our
slack
and
don't
worry.
B
B
Now,
to
start
off
with,
there's
also
the
infrastructure
of
how
these
apis
are
implemented.
So
at
the
lowest
level,
the
low
level
network
drivers
and
how
pod
networking
and
connectivity
is
wired
up
is
covered
by
what
is
known
as
the
container
network
interface,
which
is
the
cubelet
implementation
also
included
in
kubernetes.
Networking
is
an
implementation
of
the
service
api,
and
this
is
known
as
cube
proxy.
B
Also,
there
are
many
controllers
that
act
on
the
api.
For
example,
they
implement
endpoints
and
endpoint
slice.
The
various
service
load
balancer
integrations,
for
example,
the
things
that
create
load,
balancers
or
configure
load
balancers
in
response
to
services
and
also
aspects
of
network
management
such
as
ip
address
allocation
or
ipam
for
pods.
B
B
So,
let's
begin
at
the
beginning:
what
is
the
kubernetes
networking
model
so
before
we
get
too
far
into
detail?
It's
worth
revisiting
briefly.
The
basics
and
the
kubernetes
api
networking
model
is
that
all
pods
can
reach
all
other
pods.
Now
this
might
sound
simple,
and
the
key
here
is
that
pods
are
not
special
from
a
networking
standpoint.
B
In
many
implementations
they
will
appear
as
yet
another
network
endpoints
think
about
the
way
your
vms
and
other
network
devices
work.
Now,
while
the
concept
is
simple,
there
are
actually
many
implementations.
For
example,
there
are
flat
where
the
paws
exist
on
the
network
directly,
there
are
overlays
such
as
vxlan
and
their
routing-based
configs
such
as
bgp.
A
So
I
think
it's
worth
digging
into
one
of
the
more
fundamental
abstractions
in
kubernetes
services.
We
have
a
fairly
common
situation.
I
have
a
client
and
the
client
wants
to
talk
to
one
of
my
servers.
I
have
a
bunch
of
server
replicas,
because
availability
scalability,
how
does
my
client
find
and
reach
those
servers?
A
A
A
Sorry
and
it
has
to
pick
a
new
server
now
this
sounds
easy,
but
the
truth
is
that
most
software
out
there
does
not
handle
this.
Well,
especially
if
the
server's
ip
address
is
going
to
change.
Pod
ips
are
ephemeral,
ips
and
clients
can't
rely
on
them.
One
of
our
main
goals
in
kubernetes
was
to
work
with
open
source
software
as
it
exists.
A
The
statement
I
have
a
group
of
servers-
and
I
want
my
clients
to
find
them-
is
a
pretty
typical
thing.
Services
are
how
you
expose
a
group
of
pods
to
your
clients,
mostly
that
involves
having
a
durable
virtual
ip,
but
you
don't
have
to
that
includes
a
virtual
port
and
a
tcp
protocol
or
ip
protocol.
A
This
is
how
we
build
service
discovery.
It
has
built-in
load
balancing,
but
it
doesn't
have
to.
You
can
opt
out
of
that
thing.
We're
not
going
to
go
into
all
the
details
of
all
the
facets
of
service
today.
We
just
simply
don't
have
time
so
coming
back
to
the
problem
statement.
Let's
look
at
what
really
happens.
A
The
node
runs
a
proxy
which
intercepts
the
traffic.
Now
this
api
is
pretty
abstract
and
the
default
implementation
is
in
cube
proxy,
which
isn't
actually
a
proxy
usually,
but
it
does
turn
your
node
into
a
proxy.
For
example,
we
use
iptables
or
ipvs
to
make
the
node
become
your
sidecar
proxy,
we'll
get
into
that
in
a
little
bit
more
detail.
A
A
A
A
A
Digging
into
services
a
little
bit
more
I'd
like
to
show
you
yaml,
because
everybody
loves
yammer.
What
we
have
here
is
a
service
definition.
This
is
what
you
would
specify
when
you're
writing
a
service.
You
can
see
that
we
have
metadata.
The
name
is
my
service
and
the
namespace
is
default.
We're
going
to
use
that
as
part
of
our
service
discovery
system.
A
The
specification
says
which
pods
are
mapped
to
this
service
in
this
case
I'm
taking
all
pods
running
that
are
labeled
app,
my
app
and
lastly,
we
define
the
ports.
These
are
logical
ports
for
your
clients
to
use
that's
the
port,
and
then
the
target
port
is
the
actual
port
on
the
backend
pod.
The
service
allows
you
to
remap
these
things,
which
makes
a
bunch
of
things
easier.
A
Once
you
submit
that
to
the
api
server,
it's
going
to
fill
in
a
bunch
of
other
fields,
for
you,
it's
going
to
default
the
type
to
a
cluster
ip,
and
it's
going
to
allocate
you
a
virtual
ip
address.
It's
also
going
to
define
the
protocol
by
default,
but
you
can
specify
that
manually.
If
you
really
wanted
to
now
I'll
hand
it
to
bowie
to
talk
about
the
endpoints
part
of
this.
B
A
service
is
virtual,
so
how
does
the
proxy
know
what
pods
to
send
traffic
to
like
most
everything
in
kubernetes?
It's
yet
another
controller.
Basically,
the
controller
will
convert
the
end
pods
using
the
label
selector
to
a
smaller
set
of
endpoints.
So
how
endpoints
are
represented
is
a
list
of
ips
behind
a
service.
Now,
usually
these
are
pods,
but
actually
endpoints.
Api
also
gives
you
a
way
to
put
your
own
ips
and
have
custom
controllers
that
do
this
now
remember
that
ports
services
had
a
port
and
a
target
port
field.
B
So
in
fact
the
kubernetes
cube
proxy
mechanism
can
actually
remap
these
ports.
Given
your
configuration
and
generally
these
endpoints,
most
people
use
them
are
managed
by
the
system.
So
there's
an
endpoints
controller
that
comes
out
of
the
box
with
kubernetes
but,
as
I
said
before,
you
can
use
them
to
generate
custom
endpoints
as
well.
Now,
let's
look
in
detail
about
how
this
works.
B
So
here
we
have
a
pod,
a
service,
sorry
and
it's
the
name
is
foo,
and
it
has
that
app
for
selector.
As
we
said
before,
and
along
with
the
service,
we
have
a
bunch
of
pods
who
have
app
selectors.
Now,
not
all
of
them
are
foo.
Only
some
of
them
are
who,
in
this
diagram
and
the
ones
that
are
foo
sort
of
comprise
the
service
foo.
B
Now
what
happens
with
the
endpoints
controller?
Is
it
will
look
at
the
selector
in
service?
Look
at
all
the
pods
that
have
the
appropriate
labels
and
create
an
endpoints
object
that
comprises
the
endpoints
of
the
pods
that
match
the
service,
and
this
endpoints
object
is
then
something
that
will
be
fed
into
the
rest
of
the
system.
For
example,
load
balancers,
cube
proxy
dns
to
tell
the
rest
of
the
system
that
this
service
has
these
endpoints
as
part
of
the
service.
B
B
So
dns
starts
with
a
specification,
there's
a
specification
online
of
sort
of
the
set
of
records
that
are
generated
in
response
to
various
kubernetes
apis
and
generally
there's
a
default
implementation
that
runs
as
a
pod
in
the
cluster.
But
of
course,
this
is
not
necessary.
Any
implementation
that
meets
the
specification
will
do
just
fine
and
the
dns
is
exposed
itself
as
a
service
vip.
But
again,
this
is
not
required.
B
B
I
guess
a
token
svc,
that's
just
a
constant
that
denotes
as
a
service
and
then
cluster.local
is
the
cluster's
dns
node
zone.
Now
we
note
that
because
of
the
alias
mechanism,
you
can
actually
say
my
service
and
expect
to
be
scoped
within
your
namespace
or
myservice.default
to
be
scoped
within
a
kubernetes
cluster,
and
these
things
are
automatically
handled
for
you
through
the
cubelet
dns
configuration
injection.
A
So
we
promised
that
we
would
tell
you
more
about
how
the
proxy
system
works
at
first.
It's
worth,
bearing
in
mind
that
cube
proxy
is
the
default
implementation,
but
it
is
not
the
only
implementation
it
can
be
replaced
and
in
fact
we
encourage
people.
If
you
have
special
use
cases
to
go
ahead
and
replace
it,
it
is
an
implementation
of
the
apis.
So
a
cube
proxy
by
default
runs
on
every
node
in
your
cluster.
A
It
uses
that
node
as
a
proxy
for
traffic
from
pods
on
that
node.
The
mechanisms
that
we
have
built
into
it
right
now
are
iptables
ipvs
user
space
windows
has
also
a
user
space
and
an
internal
options
in
linux,
iptables
and
ipvs
are
the
more
modern
versions.
The
user
space
is
less
scalable
and
less
behaviorally
correct
in
general.
Cube
proxy
is
completely
transparent
to
consumers.
All
those
pods
that
are
running
on
the
same
node
don't
know
that
cube
proxy.
Is
there
intercepting
traffic?
That's
part
of
the
api.
A
So
the
way
cubeproxy
works
on
the
control
path
is
it's
going
to
watch
all
of
the
services
and
endpoints
in
the
system
which
we
just
looked
at
it's
going
to
apply
some
filters.
For
example,
it's
going
to
ignore
some
services
that
don't
have
virtual
ip
addresses.
We
call
those
headless
it's
going
to
link
those
endpoints.
A
The
back
ends
with
the
services
which
are
the
front
ends,
and
it's
going
to
accumulate
the
changes
to
all
of
those
and
it's
going
to
then
update
rules
on
the
node
to
represent
those
changes,
for
example
in
iptables
it
will
capture
traffic
destined
for
the
virtual
ip
and
port,
and
it
will
redirect
it
into
the
data
path
to
handle
the
data
path
on
the
data
path,
it's
going
to,
like,
I
said,
recognize
the
destination
ip
and
port.
It's
going
to
choose
a
back
end
pod.
Generally,
it's
going
to
consider
things
like
client
affinity.
A
Have
there
been
other
sessions
from
this
client
to
this
virtual
ip?
If
the
service
has
requested
those?
And
then
it's
going
to
rewrite
those
packets
to
a
new
destination,
that's
destination
nat.
On
response,
it's
going
to
apply
the
opposite
of
that
and
undeniab
it
and
return
it
back
to
the
original
client.
Again,
the
clients
are
unaware
of
what's
happening
here.
This
is
all
happening
on
the
node
as
a
service
to
the
pods
on
the
node.
A
A
One
of
the
fundamental
things
we
learned
about
kubernetes
very
early
on
is
that
most
dns
clients
are
broken
and
they
don't
handle
things
like
dns
records.
Changing
very
well
many
applications,
never
resolve
re-resolve
the
names
they've
looked
up.
This
abstraction
provides
a
stable
ip
address
so
that
clients
can
work
as
they
existed
before.
Kubernetes
in
a
world
where
durable
ip
addresses
were
more
prevalent
for
those
clients
that
are
enlightened
and
can
actually
handle
this
properly.
Can
you
opt
out
of
this
stuff?
Of
course
you
can.
This
is
kubernetes.
You
have
a
million
options.
A
Alongside
services
is
the
load
balancer
system
services
are
how
you
expose
load
balancers
in
kubernetes.
We
have
different
load,
balancer
implementations
and
they
work
in
very
different
ways
across
cloud
providers
and
across
mechanisms
with
your
networking.
It's
a
little
too
broad
for
us
to
go
into
and
into
this
talk.
B
Thanks
tim,
so
we
saw
that
services
describe
l4
protocol
based
services,
but
what
about
l7-
and
this
is
where
the
ingress
resource
comes
in,
so
what
an
ingress
resource
does?
Is
it's
an
api
to
describe
an
http,
proxy
and
routing
rules?
So
this
is
a
very
simple
api
to
match
hostnames
and
url
paths.
It
might
be
too
simple,
so
we'll
see
more
on
this
later
in
the
deep
dive
and
what
it
does
is
it
targets
a
service
for
each
http,
url
path.
B
B
So
here
we
have
a
ingress
configuration
along
with
the
services
and
pods
and
you'll
see
at
the
top
there's
an
ingress
with
a
hostname
and
a
bunch
of
paths,
and
each
path
corresponds
to
a
rule
that
then
maps
to
a
specific
service.
So
you
can
imagine,
there's
a
path
for
slash
store
and
that
will
be
your
store
service
and
there's
a
path
for
slash
search
and
that
will
go
to
your
search
service
and
then
the
services
are
used
as
a
way
to
select
which
server
pods.
B
B
Now
a
couple
of
questions
come
up,
which
is
one:
how
is
this
different
from
service
type
load
balancer?
So
I
think
the
difference
is
that
service
type
load
balancer
is
constrained
at
the
l4.
It
doesn't
talk
about
anything
higher
in
terms
of
load.
Balancing
and
ingress
provides
a
way
to
describe
these
things,
and
the
other
thing
to
note
is
that
when
you
go
to
l7,
it
is
very
common
that
you
will
be
composing
services,
multiple
l4
services,
together
to
create
a
single
l7
service.
B
Second
question
is:
why
isn't
there
a
controller
in
the
box?
So
there
were
actually
many.
If
you
think
about
the
overall
l7
proxy
landscape,
it
is
a
pretty
mature
landscape.
There
are
many
implementations,
many
different
proxies
and
we
didn't
want
to
pick
a
winner
among
the
software
lbs.
However,
you
know
it
is
an
interesting
point
to
have
a
system
that
has
batteries
included.
A
So
you
can't
really
have
a
network
without
some
way
of
describing
what's
allowed
and
disallowed
on
the
network
network
policy
is
the
kubernetes
api
that
describes
the
graph
of
your
application.
It
is
allows
your
app
developers
to
express
what
do
you
expect
to
be
happening
here
and
disallow
everything
that
isn't
expected.
So,
for
example,
you
could
have
all
your
front
ends
which
talk
to
your
back
ends
and
your
back
ends
which
talk
to
your
databases
but
you'd,
never
want
your
front
end
to
talk
directly
to
your
database.
A
A
It's
very
likely
that
the
implementations
of
network
policy
are
very
coupled
to
the
way
your
network
is
configured,
and
so
it
didn't
really
make
sense
to
have
an
in
the
box
version
of
this
when
we
don't
have
an
in
the
box
version
of
pod.
Networking
network
policy
is
a
very
simple
api.
It's
very
focused
on
application
owners
rather
than
network
administrators,
and
you
know
we
may
yet
need
another
api
at
a
higher
level
for
network
administrators,
but
network
policy
is
really
supposed
to
be
part
of
your
application's
description.
A
A
B
B
So
as
part
of
the
deep
dive
we're
going
to
look
at
a
closer
look
at
the
work,
that's
happening
in
sig
network.
So
some
of
this
is
a
ga
and
others
are
in
alpha
and
beta
stages,
and
I'd
like
to
repeat
that,
if
you're
looking
to
influence
the
direction
of
kubernetes
networking,
we
really
value
feedback
and
participation
from
the
community
here
and
the
ongoing
work
in
the
sig
that
we're
going
to
highlight
for
the
deep
dive
are
endpoint
slice,
which
is
work
in
scalability
and
extensibility.
B
B
B
B
So,
as
dns
is
the
canonical
example
of
achieving
global
scalability
through
caching,
the
answer
is
simple:
run
a
cache
on
every
node.
However,
we
have
to
be
very
careful
because,
as
it
turns
out,
if
you
run
something
per
node,
actually
the
overhead
can
dominate
in
large
clusters.
So
we
have
to
be
very
careful
and
in
fact
the
node
local
dns
is
an
extremely
slimmed
down
very
small
local
cache,
that's
based
on
core
dns,
but
basically
without
many
of
the
plugins
compiled
in
as
a
system
critical
component
which
is
in
a
daemon
set.
B
We
also
have
to
be
quite
careful
about
high
ability
during
upgrades
and
failures,
and
this
was
one
of
the
gating
criteria
for
the
ga
of
no
local
dns.
So,
let's
go
into
how
no
local
dns
looks
on
your
system,
so
here
we
have
sort
of
how
dns
works
without
no
local
dns.
So
you
see
in
green,
you
have
a
set
of
pods
and
cubelet
has
configured
for
your
pod.
B
B
The
secondary
effect
of
the
interface
and
the
binding
of
the
vip
is
that
actually,
this
allows
node
local
dns
to
capture
the
vip
traffic
and
sort
of
live
alongside
cube
proxy
you'll
notice
that
we
still
have
to
go
upstream
to
cube
dns.
So
how
do
we
do
that?
We
actually
create
a
another
service
that
tells
the
no
local
dns
about
the
upstream,
which
is
this
cube
dns
upstream
here
this
is
the
10
0,
and
then
the
cluster
vip
will
be
allocated
from
your
cluster
vip
range
now
you'll
notice.
B
What
a
very
interesting
thing
here
is
that
these
two
mechanisms
live
side
by
side
and
in
fact,
for
and
upgrades,
for
example,
if
the
no
local
dns
goes
down,
it
has
a
watcher
process
that
will
remove
the
node
track
and
once
the
node
track
entries
are
removed.
Actually
the
standard
q
proxy
service
mechanism
will
happen.
B
B
B
The
next
topic
for
our
deep
dive
is
also
partially
scalability
and
partially
extensibility.
So,
as
we
talked
about
before,
there
is
an
endpoints
api
that
serves
as
a
service
discovery
mechanism
for
kubernetes
at
the
kubernetes
api
level.
Now
in
larger
clusters,
think
15k
nodes,
for
instance,
and
very
large
services,
will
lead
to
api
scalability
issue.
So,
if
you
think
about
it,
an
endpoints
object
is
a
single
object
and
let's
say
you
have
a
thousand
endpoints:
all
thousand
endpoints
will
have
to
go
into
that
single
object.
B
If
you
keep
expanding
services
and
the
number
of
backends
in
a
service
eventually,
this
will
exceed
the
size
of
a
single
object
in
fcd.
Now,
of
course,
you
can
reconfigure
fcd
to
support
larger
objects,
but
there
is
a
limit
to
that.
You
can
conceivably
double
the
size
of
the
object,
maybe
triple
it,
but
at
some
point
you're
going
to
have
to
live
with
the
limits
of
sed.
B
Also,
a
huge
issue
is
the
number
of
watchers
for
endpoint
objects.
So,
as
you
recall,
cube
proxy
watches
the
endpoints
objects
in
the
cluster
and
there
is
a
copy
of
q
proxy
running
on
every
single
node.
In
addition,
other
controllers
may
be
watching
endpoints
as
well,
and
what
this
does
is.
It
creates
a
large
amount
of
data
that
needs
to
be
sent
to
the
watchers.
B
So
in
this
example
below-
and
we
have
a
5
000
nodes-
let's
say
the
size
of
each
endpoint
object
is
one
megabyte,
and
if
you
have
an
update
to
the
endpoints
object,
it
will
actually
send
five
gigabytes
of
data.
Now.
This
is
basically
a
dvd
and
then,
if
you
do
a
rolling
update
where
you
are
slowly
changing
the
endpoints
one
by
one
and
you
imagine
these
updates
propagating
through
the
system
by
the
end
of
your
rolling
updates,
this
will
be
25
terabytes,
which
is
a
lot
of
data.
B
Here
we
have
a
diagram
of
what
the
endpoints
objects
update,
looks
like
for
an
endpoints
object,
so
here
we
have
at
the
top
the
resource
living
in
the
api
server.
So
there's
like
a
single
resource
and
we
perform
a
single
update,
which
is
the
yellow,
highlighted
endpoint
here,
but
for
each
cube
proxy
in
the
cluster
we
actually
end
up
having
to
send
the
entire
endpoints
object,
even
though
that
little
piece
changed,
we
have
to
send
the
entire
thing
with
endpoint
slices.
B
What
happens
is
that
that
service
is
in
fact
instead
represented
by
a
set
of
endpoint
slices.
So
there
are
actually
many
endpoint
slices
for
a
given
service
and
if
you
see-
and
they
are
each
of
a
maximum
size
by
default,
they
will
have
a
hundred
endpoints.
Each
and
a
single
update
to
an
endpoint
actually
only
needs
to
send
the
slice.
So
you'll
see
here
that
highlighted
in
red
that
endpoint
slice
object.
Abbreviated
eps
will
be
sent
to
these
cube
proxies,
but
the
ones
that
did
not
change
will
not
be
sent.
B
B
So
sort
of
more
detail
on
how
endpoint
slices
work,
so
it
turns
out
the
update
algorithm
is
actually
an
interesting,
optimization
problem.
So
at
the
same
time
you
want
to
keep
the
number
of
slices
low
because
you
don't
want
to
have
create
too
many
resources
and
start
running
into
fcd
right
issues.
In
terms
of
number
of
resources.
B
You
need
to
write
to
the
database
while
minimizing
the
number
of
changes
to
the
slices
per
update,
because
we
don't
remember,
every
single
change
in
every
single
resource
needs
to
be
sent
out
to
a
watcher
and,
of
course,
keep
the
amount
of
data
sent
low.
So
these
are
kind
of
somewhat
intersecting
and
you
can
imagine
that
to
keep
the
number
slices
low,
you
might
need
to
make
more
changes
to
slices
and
so
forth.
B
Its
rollout
is
going
to
take
a
couple
of
releases,
so
in
117
the
endpoint
slice
controller
and
api
was
available
in
118
we
made
the
controller
enabled
so
that
the
resources
will
be
created,
but
q
proxy
will
not
be
defaulted
in
and
then
119.
We
expect
endpoint
slice,
controller,
endpoint
slice,
mirror
windows
and
q
proxy
enabled,
and
finally
in
120
it
will
be
ga.
A
Thanks
bowie,
as
kubernetes
users
become
more
prevalent
in
more
market
segments
and
their
installations
get
bigger.
Their
needs
get
bigger,
it's
becoming
very
common
for
users
to
have
more
than
one
cluster
and
that
we
have
a
lot
of
reasons
why
users
have
more
than
one
clusters.
Some
are
some
are
blast:
radius,
geography,
legal
restrictions,
dev
test,
prod
environments.
A
A
This
comes
up
as
one
of
the
most
common
pain
points
that
we
hear
from
users.
For
example,
let's
look
at
this
situation.
I
have
two
clusters
a
and
b
and
I
have
my
front-end
service
in
cluster,
a
and
my
back-end
service
in
cluster
b.
How
do
I
get
my
front-ends
to
be
able
to
reach
my
back
ends?
This
seems
like
it
should
be
easy.
A
There
are
ways
to
do
this
generally
involving
load
balancers
and
translating
from
one
ip
space
to
another
or
or
rewriting
packets,
but
I
think
we
can
do
better
than
that.
So
one
of
the
things
that
we're
starting
to
look
at
is
what,
if
we
could
expose
services
between
these
clusters
so,
for
example,
given
the
service
on
the
left,
the
backend
service.
What
if
I
had
something
like
the
right
that
simply
declares
that
this
service
is
exported
right,
but
what
does
it
mean
exported
to
what.
A
So,
first,
let's
assert
that
a
group
exists,
I'm
not
defining
the
group,
I'm
just
saying
that
it
exists.
This
leaves
a
lot
of
room
for
interesting
implementations
for
vendors
and
for
people
to
try
different
things
with
their
grouping
mechanisms.
So,
let's
just
assume
that
it
exists
and
within
that
group
these
clusters
are
related.
A
A
A
So
we
looked
at
dns
before
so
we'll
look
at
dns
again
here,
similar
to
the
main
dns.
We
have
backend
service,
which
is
the
name
of
my
service.
We
have
backend,
which
is
the
name
of
my
namespace,
and
we
have
a
new
suffix.
This
suffix
here
is
to
be
decided
on
the
name,
but
right
now
we're
running
with
super
cluster,
we'll
give
you
a
different
zone
and
by
simply
changing
the
zone
that
you
use
from
cluster.local
to
supercluster.local.
A
To
make
it
even
more
fun
that
back-end
service
might
exist
in
both
clusters.
In
this
case,
accessing
the
super
cluster
service
will
use
back
ends
in
either
cluster.
You
can
see
there's
a
lot
of
lines
in
this
drawing
here,
but
both
clusters
will
have
endpoints
and
endpoint
slices
that
represent
the
union
of
all
of
the
back
ends
across
both
clusters.
A
Now,
to
be
fair,
this
is
mostly
kepware
right
now,
there's
some
proof
of
concepts
that
exist.
We're
still
hammering
out
a
lot
of
details,
things
like
api
and
names.
We
have
a
poll
running
on
what
the
proper
suffix
should
be,
we're
still
working
out
some
of
the
semantics,
for
example,
conflict
resolution
and
propagation
time
and
scalability
we're
still
looking
at
things
like
cross-cluster
network
policy,
which
doesn't
quite
exist
yet,
but
we're
excited
about
this
work.
That's
coming
again.
This
is
a
collaboration
with
sig
multi-cluster.
A
So
I
also
want
to
talk
about
some
of
the
exciting
work.
That's
being
worked
on
right
now,
ipv4
and
ipv6.
Dual
stack,
the
reality
of
the
world
is
that
some
users
need
both
ipv4
and
ipv6.
At
the
same
time,
kubernetes
has
gotten
this
far
by
assuming
that
there's
a
single
ip
address
for
pods
we've
also
gotten
this
far
by
assuming
there's
a
single
ip
address
for
a
service.
A
If
we're
going
to
exist
in
a
dual
stack
world,
we
have
to
fix
this.
This
is
a
small
change
to
the
api,
but
it's
actually
very
profound
and
the
impact
of
it
is
pretty
large.
Now
some
of
you
may
be
asking
wait.
Wasn't
this
done
already?
Didn't
she
talk
about
this
before?
Yes,
we
did.
We
got
this
to
alpha
and
we
found
a
bunch
of
significant
problems
with
it,
so
we're
doing
a
major
reboot
of
the
api.
A
So
I
thought
it
was
worthwhile
to
throw
out
the
new
api
here
so
starting
with
a
plain
old
pod.
We
had
this
now
we're
going
to
have
this
so
we're
going
to
take
the
pod
ip
field
and
we're
going
to
pluralize
it
and
we've
worked
out
a
protocol
for
what
pluralization
means
within
the
api,
and
it
involves
synchronizing
the
legacy
singular
variable
into
the
plural
field.
A
A
A
A
A
A
A
So
this
work
was
alpha
and
it
is
alpha
in
kubernetes,
1.18
and
1.19
we're
shooting
for
a
second
alpha
and
1.20.
This
is
a
breaking
change
from
the
api
point
of
view,
because
we're
still
in
alpha,
we
can
technically
do
that.
It's
unfortunate
for
people
who
have
started
using
it
and
to
all
those
people
I
apologize.
B
Thanks
tim,
so
as
part
of
thinking
about
services,
we
titled
this
services
v
plus
one
is.
We
are
looking
at
sort
of
what
the
evolution
of
services
is
and,
if
you
think
about
it
currently,
kubernetes
service
is
a
number
of
things.
One
is
it's
a
method
of
exposure,
so,
on
the
side
you
have
here
the
hierarchy
of
different
service
types
such
as
headless
cluster,
ip
node,
port
and
load
balancer.
In
fact,
they
imply,
for
example,
a
node
load.
Bouncer
implies
a
node
port,
implies
a
cluster
ip
and
so
forth.
B
The
second
thing
that
a
service
does
is
sort
of
talk
about
groupings
of
pods
into
a
workload
which
is
a
service
eg,
the
selector
and
finally,
the
service
resource
contains
things
which
is
attributes
about
the
service
about
various
policies
such
as
external
traffic
policy,
session
affinity
and
so
forth.
Now
what
we're
finding
is
that
evolving
and
extending
this
resource
has
become
harder
and
harder,
because
all
of
these
fields
they
sort
of
serve
different
purposes,
but
they
as
part
of
the
same
api,
they
will
interact
in
strange
ways.
B
At
the
same
time,
there's
been
an
evolution,
a
desire
to
evolve
the
l7
ingress
api,
as
I
said
before,
it
was
fairly
bare
bones
and
simplistic,
and
a
lot
of
the
proxy
vendors
today
can
support.
You
know
much
more
expressibility
and
we
also
want
to
incorporate
as
part
of
this
division
of
responsibility
and
resources
that
allows
for
modeling
of
different
roles.
So
it's
both.
What
do
we
do
about
extending
service
further
and
also,
how
do
we
describe
some
of
these
things
that
we
know
exist
and
can
be
done.
B
So
the
idea
of
services
v,
plus
one
which
is
also
known
as
the
gateway
api,
is
to
sort
of
decouple
these
things
along
a
role
in
concept
access.
So
we
have
a
bunch
of
roles
which
is
and
the
apparently
the
emojis
did
not
translate
well
into
powerpoint.
So
there
is
supposed
to
be
some.
You
see.
The
female
sign
is
actually
supposed
to
be
an
emoji,
so
there
are
a
couple
of
roles
that
we're
looking
at
in
terms
of
modeling.
B
These
resources
there's
an
infrastructure
provider
which
is
sort
of
the
set
of
people
that
have
set
up
your
cloud
environment.
Your
cloud
platform
have
created
your
cluster
there's
a
cluster
operator
who
are
sort
of
the
super
user
of
an
individual
cluster
and
or
network
operator
who
control
say
who
can
have
access
to
internet
access
and
egress
proxies
and
so
forth,
and
then,
finally,
there's
the
application
developer.
B
So
what
we
want
to
do
with
the
services
api
is
to
basically
allow
a
user
to
express
these
roles
and
sort
of
grant
different
permissions
based
on
these
different
roles.
The
second
thing
that
we
wanted
to
do
in
sort
of
service
refactoring
is
to
separate
these
concepts
into
different
resources
so
that
the
evolution
of
each
of
the
resources
can
be
much
easier
to
deal
with.
B
So
the
three
concepts
I
said
before
are
grouping
and
selection,
so
this
is
sort
of
what
workloads
comprise
a
service.
The
second
piece
is
how
routing
and
protocol
specific
attributes
will
work
so
for
l7
http.
This
will
be,
for
example,
that
a
given
hostname
and
a
given
path
will
go
to
a
particular
service
and
then
finally,
the
last
part
is
how
the
service
will
be
exposed
and
how
it
will
be
accessed
so
I'll
kind
of
go
over
a
high
level
of
what
this
looks
like
in
the
api
proposal.
B
So
at
the
top
we
have
and
you'll
see
on
the
right.
We
have
the
various
resources
that
are
being
developed.
So
at
the
top
we
have
gateway
class,
so
gateway
class
is
a
resource
intended
for
the
infrastructure
provider
and
what
a
gateway
class
describes
is
it
defines
a
kind
of
service
access
to
a
cluster,
for
example?
B
Example,
gateway
classes
would
be,
for
example,
an
internal
proxy
like
a
proxy,
let's
say:
nginx
ingress
that
runs
on
the
cluster
or
an
internet
lb,
for
example,
a
kind
of
proxy
that
runs
in
the
cloud
provider
infrastructure
so
similar
to
storage
class.
This
abstracts
the
implementation
of
the
mechanism,
for
example,
an
internal
proxy,
may
use
a
specific
version
or
deployment
of
al7
proxy
from
the
consumer.
B
So
from
the
consumer
standpoint
they
would
just
say
I
want
to
have
an
internal
proxy
or
I
want
to
have
an
internet
lb
and
then
the
cluster
operator,
the
infrastructure
provider,
will
create
this
class
sort
of
say:
hey.
You
wanted
an
internet
lb.
This
is
how
we're
going
to
actually
do
it
and
below.
You
can
see
an
example
of
the
configuration
for
a
gateway
class
you'll
see
that
it
has
a
controller.
B
So,
for
example,
if
acme
io
was
a
proxy
implementation,
and
this
would
be
the
controller
for
acme
io
and
a
way
to
pass
controller
specific
parameters
to
it.
But
from
the
user
standpoint,
for
example,
people
who
create
gateways
and
routes
and
so
forth,
they
will
only
know
that
this
gateway
class
corresponds
to
a
in
cluster
gateway,
which
is
sort
of
the
name.
Here.
They
wouldn't
have
to
know,
for
example,
that
in
actuality
that
in-cluster
gateway
is
implemented
using
an
acme.
I
o
gateway
controller.
B
Oh
next,
we
have
the
gateway
resource,
so
the
gateway
resource
is
what
we
would
typically
think
of
controlled
by
the
role
of
a
cluster
operator
or
network
operator,
and
this
governs
how
the
service
is
accessed
by
the
user
eg,
what
port
it's
on?
What
protocol
it's
on?
What
addresses
it
uses?
This
is
a
keystone
resource.
B
What
I
mean
by
that
is
this
will
be
one
to
one
with
the
lifecycle
of
some
infrastructure
action,
so
the
configuration
of
instruction
will
be
triggered
by
the
association
of
a
valid
gateway
with
a
class
and
a
route,
and
this
could
have
many
implementations
in
terms
of
deployment.
It
could
spawn
a
software
lb,
it
could
add
a
configuration
stanza
to
an
lb.
It
could
program
the
sdn
and
so
forth,
but
basically
the
gateway
sort
of
hooks
all
of
the
resources
together
and
provides
that
life
cycle
relationship.
B
Now.
One
thing
to
note
is
that
a
gateway
may
be
underspecified
quote
unquote.
For
example,
there
may
be
many
defaults
that
are
sort
of
provided
by
a
gateway
class,
so
the
user
can
only
needs
to
concern
themselves
with
aspects
of
the
gateway
that
they
care
about
and
the
gateway
class
will
provide
most
of
the
other
things.
B
So
it's
a
family
resources
group
by
protocol
and
use
case
to
solve
the
issue
of,
for
example,
in
the
api,
defining
a
single
closed
union
type,
basically,
a
single
type
that
has
all
the
possible
types
enclosed
in
it.
That
probably
will
make
it
very
hard
to
evolve
it
quickly.
For
example,
the
single
api
type
needs
to
talk
about,
tcp
needs
to
be
talked
about,
sni
needs
to
talk
about
http
and
so
forth.
B
B
So
here
we
have
an
example
of
sort
of
how
all
these
things
are
hooked
up.
We
have
a
gateway
class
that
is
referenced
by
a
gateway
and
the
gateway
references,
other
routes
in
green
and
then
finally,
the
routes
route.
Each
of
the
interesting
paths
and
protocol
specific
matches
to
their
respective
services.
B
So
we're
still
working
on
the
v1
alpha
one,
but
this
is
a
cut
list
of
the
things
that
we
really
want
to
get
done.
We
want
to
show
demonstrate
basic
applications
and
the
data
types,
which
is
the
resources
we
want
to
have
show
that
gateway
class
can
be
used
for
interoperation
between
controllers.
So
we
want
to
have
two
controllers
or
end
controllers
and
then
sort
of
switch
the
classes
between
them
and
see
that
working
support.
Basic
support
for
http
and
tcp
has
two
examples.
A
A
A
A
This
is
something
we
do
on
a
bi-weekly
basis
with
our
sig
calls,
which
again
we'll
post
the
zoom
link
in
just
a
minute
come
and
help.
The
issues
are
a
great
place
for
people
to
get
started
and
get
involved
come
and
look
for
the
problems
you've
had
or
help
other
users
with
the
problems
that
they're
having.
A
A
We
welcome
anybody,
who's
watching
to
participate
in
our
enhancement,
conversations
and
the
planning.
We
always
welcome
more
eyeballs
help
us
find
new
problems,
ask
great
questions
or
come
and
submit
an
enhancement
proposal
of
your
own.
If
you've
got
a
problem
with
kubernetes,
we
want
to
hear
about
it.
A
Lastly,
to
get
involved
with
our
sig,
you
can
find
all
of
our
information
at
our
sig
network
community
repo.
We
do
meet
every
other
thursday
at
2100
utc.
We
have
a
slack
channel
sig
network
on
the
slack.kates.ioslac
server
and
we
have
a
mailing
list.
So
there's
got
to
be
something
in
here
that
works
for
you
to
get
a
hold
of
us
and
come
have
a
conversation
with
that.
I
want
to
say
thank
you
to
everybody
who
came
today
and
thanks
to
bowie
for
all
of
your
hard
work
on.
A
There
we
go,
they
we've
done
some
rough
categorization
of
them,
while
people
were
asking
them
so
I'll,
just
go
through
and
I'll
read
the
questions
out
and
we'll
answer
them
and
feel
free
to
submit
new
questions,
and
there
are
people
prioritizing
those
two
we'll
try
to
get
to
as
many
as
we
can
and
afterwards,
we'll
jump
to
the
slack
channel
question.
Our
team
is
using
gke
and
we
had
multiple
problems
running
out
of
contract.
A
We
had
to
deploy
a
demon
set
to
customize
it
for
our
needs,
and
there
was
some
fallout
questions
that
the
usual
answer
of
node
local
dns
isn't
solving
the
problem.
So
we've
seen
it
with
a
small
number
of
users,
not
a
large
number
of
people
actually
run
out
of
the
contract
records.
We
set
up
a
lot
of
records.
There
isn't
a
general
answer.
What
you're
doing
with
the
demon
set
is
is
a
good
way
of
customizing
nodes.
A
A
Yeah,
unfortunately,
it's
hard
to
do
contract
very
dynamically.
The
kernel
does
a
lot
of
work
when
you
change
the
size
of
the
contract
tables
next
question:
when
I
deploy
a
load
balancer
ip,
if
I
don't
specify
an
external
ip
will,
I
still
be
able
to
access
it
outside
of
the
cluster
load.
Balancer
services
are
quote
external
to
the
cluster,
so
the
idea
is
yes,
you'd
be
able
to
access
that
load
balancer
ip
from
outside
the
cluster.
What
exactly
that
means
really
depends
on
the
implementation.
A
I'm
just
seeing,
if
you
have
anything
to
add
next
question:
if
my
service
has
three
pods
and
q
proxy
runs
in
ip
tables
mode,
I.e,
33
randomness,
how
does
it
cope
if
one
pod
crashes?
How
long
is
the
delay
before
it
figures
out
one
service
pod
is
down,
so
the
delay
is
generally
as
long
as
you've
set
your
readiness
probe
on
your
pod.
So
if
your
pot
is
down
but
alive
like
it's
running,
but
it's
not
serving
for
some
reason,
your
readiness
probe
will
trigger
the
update
to
the
pod.
A
Generally,
readiness
probes
are
on
the
order
of
single
digit
to
two
digit
seconds.
Most
people
can
figure
them
that
way,
so
it
will
be
a
few
seconds
before
it's
removed
from
the
service.
If
you
don't
have
a
readiness
probe,
a
you
should
have
one
and
b
well,
we
won't
know
if
your
pod
actually
crashes,
we'll
know
pretty
quickly,
because
it
will
no
longer
be
something
that
the
service
controller
is
selecting
so
that
that
is
the
quickest
way
of
getting
it
removed
from
the
set.
B
A
B
A
B
A
Will
service
meshes
be
integrated
into
kubernetes
networking
sig
in
the
future?
So
I'm
on
record
saying
I
think
kubernetes
is
already
a
service
mesh
is
becoming
a
service.
Mesh
is
taking
on
more
service,
mesh-like
properties.
If
you
look
at
the
gateway
work
which
I'll
let
bowie
speak
to,
you
start
to
see
a
lot
more
affordances
for
things
like
service
meshes,
I
don't
think
we're
going
to
like
absorb
istio
or
linker
d
or
or
any
of
the
service
meshes.
I
do
think
kubernetes
apis
will
start
to
cover
more
service
mesh
like
functionality
right.
B
You
know
universally
accessible,
specific
service
mesh
implementations
on
the
implementation
side
right
then,
hopefully,
the
service
meshes
will
be
able
to
support
these
apis
along
with
their
own
and
then
the
idea
is
that
gateway
is
extensible,
so
you
might
be
able
to
get
away
with
using
the
portable
api
rather
than
having
to
have
a
custom.
One.
A
Next
question:
have
you
considered
evpf
instead
of
iptables
for
the
default
we've
thought
about
it?
Evpf
is
more
complicated
and
honestly,
my
personal
preference
is
to
decouple
more
cube
proxy,
as
a
monolith
has
sort
of
become
a
problem
from
a
software
management
point
of
view.
So
I'm
actually
a
fan
of
the
fact
that
projects
like
psyllium
are
decoupled
from
the
core
project.
I
think
it
forces
us
to
be
a
little
bit
more
honest
about
compatibility,
so
thought
about
it.
I
don't
think
we
make
ebpf
the
default
in
the
project.
B
It
depends
on
your
platform,
so
I
think
someone
just
checked.
It
looks
like
it's
enabled
on
some
platforms
like
eks.
I
think,
but
it
is
not
it
really.
It
does
take
up
some
resources
so
for
very
small
clusters.
It
probably
doesn't
make
sense.
They
won't
hit
that
scale
anyways,
but
the
way
it's
implemented.
It
is
a
good,
especially
with
the
aha
and
transparency,
is
a
good
easy
drop
in.
If
you
hit
dns
issues.
A
Next
question:
some
network
control
plane
implementations.
Allow
is
based
enforcement
of
network
flows
like
nsxt.
How
do
you
feel
about
that
way
of
doing
traffic,
security
versus
internal
policy
systems?
I
don't
know
what
internal
here
means.
What
I
care
about
is
consistent,
apis
and,
for
example,
network
policy.
A
B
A
B
A
B
Protocol
and
it's
something
that
we
expect
to
be
the
sort
of
thing
we
expect
to
be
covered
by
gateway,
so
I
know
that
it
load
bounces
on
hostname,
as
I
understand
it,
low
bounces
on
hostname
and
path,
sort
of
like
htp
and
that's
sort
of
where
the
future
will
go
right.
Now
you
can
use
some
ingress
implementations.
I
think,
may
support
it,
but
it
will
be
an
extension.
A
And
there's
some
work
now
happening
for
proxy
list
grpc,
where
you
can
use
a
more
mesh
like
control
signal
to
run
your
grpc
services
without
proxies
next
question:
how
will
services
v,
plus
one
interact
with
cloud
lbs
or
customer
lbs
like
metal
lb?
Will
there
be
improvements
to
them
if
they
integrate
with
the
new
api
boy?
That's
you.
B
Yeah,
at
least
for
the
cloud
lbs,
we
definitely
it's
in
this
target
of
what
will
be
supporting
the
apis,
I'm
less
familiar
with
metal
lb
and
how
it
fits
in
the
picture,
mostly
because
it
will
be
relevant
from
the
l4.
I
just
don't
know
what
the
metal
lb
l7
story
is,
so
it
to.
It
would
be
great
if
you
ping
me
on
slack,
we
can
follow
up
there,
I'm
just
less
familiar
with
metal
lb.
A
Services
across
clusters
can
it
be
via
wan?
Is
there
any
latency
limitation,
there's
no
intrinsic
limitation
right
now.
Kubernetes
does
not
really
respect
topology.
So
if
you
have
two
clusters
in
two
different
cloud
regions,
for
example-
or
if
you
have
them
linked
across
a
lan,
it
will
happily
spread
traffic
across
that
link,
which
is
probably
not
what
you
really
wanted
so
right
now,
the
the
implementations
that
we're
looking
at
are
assuming
same
latency
zone.
A
B
A
Next
question:
do
you
know
how
the
network
plug-ins
calico
psyllium
stand
with
dual
stack
calico,
I
believe,
already
supports
dual
stack.
It
just
can't
expose
it
through
kubernetes
because
we
lack
the
api,
but
you
can
create
pods
with
two
ip
addresses
psyllium.
I
don't
know
for
sure,
but
I'd
be
surprised
if
it
doesn't
have
some
form
of
that.
Also-
and
I
imagine
most,
the
others
will
be
relatively
straightforward
to
integrate
favorite
question:
what's
the
time
frame
for
the
new
service
api
when.
B
Can
I
actually
actually
we
have
a
goal
of
v1
alpha
one
and
if
you
attend
the
meetings
or
take
a
look
at
the
notes,
we're
basically
burning
down
the
features.
The
idea
is
to
sort
of
get
the
out
something
out
in
the
user's
hands
to
get
feedback.
I
think
that's
the
most
valuable
at
this
point.
Some
things
you'll,
like
some
things,
you'll
hate,
and
we
would
love
to
hear
all
that
feedback.
B
Yeah
I
had
a
answer
to
that.
Historically,
observability
has
been
orthogonal,
so
ping
me
on
slack,
because
I
kind
of
understand
understand
which
part
of
observatory
you're
talking
about.
There
are
some
aspects
where
we
are
adding
sort
of
extra
hooks
to
put
to
describe
observability
such
as
request
mirroring,
but
the
other
aspect
which
is
like
what
metrics
would
be
published.
That's
a
separate
thing
that
I
think
we
haven't
looked
deeply
into
but
be
interesting
to
sort
of
get
some
ideas.
B
B
A
B
There's
two
one
of
them
is
that,
if
you
are,
I
think
I
might
misread
the
question
when
I
answer
this.
When
you're,
if
you're
using
a
feature
like
a
wildcard
domain,
it
should
be
the
gateway.
The
new
api
should
be
a
superset,
so
those
features
should
continue
to
be
like
you
could
be
able
to
express
them.
B
The
other
question
is:
how
will
it
interact
potentially
with
something
like
external
dns
or
actually
search
management
as
well,
and
that
is
an
area
that
we're
explicitly
looking
at
is
that
gateway
and
service
api
is
not
just
as
a
way
to
express
things,
but
as
also
a
layer
that
people
build
on
top.
I
know
that
for
ingress
api,
for
example,
cert
manager
will
build,
will
look
at
your
ingress
resource
and
then
do
cert
management
stuff,
and
that's
that's
the
kind
of
use
case
that
we
are
also
thinking
of
as
part
of
its
design.
A
A
B
Yeah,
I
do
have
to
just
reiterate
that
right,
there's
the
implementation,
there's
the
apis
and
it
seems
like
many
of
the
service
mesh
concepts.
It's
great,
so
then
they're
sort
of
being
absorbed
into
sort
of
the
base
layer.
That's
a
natural
progression
of
the
system
in
terms
of
implementation,
then
you
know
it
will
be
like
your
favorite
thing
will
remain
relevant.
B
A
So
the
short
answer
is,
you
can
tell
the
kernel
at
the
beginning
of
a
packet's
lifetime.
Don't
track
this
packet
any
further
and
it
won't
go
through
the
rest
of
the
connection
tracking
subsystem,
which
means
it
isn't
subject
to
the
normal
kubernetes
load,
balancing
the
service
load,
balancing,
which
means
you
don't
create
contract
records
for
it,
which
is
fine
if
you
know
that
the
back
end
is
always
a
stable
ip
address
like
the
same
host.
A
And
contract
for
dns
is
one
of
the
prime
culprits
that
we've
seen
for
people
who
are
having
contract
troubles
because
dns
is
udp,
so
there's
no
connection
to
close,
so
you
just
have
to
time
out
those
connections
and
because
they
tend
to
be
one
request,
one
response
and
then
the
client
opens
a
new
socket
we're
getting.
We
have
just
one
minute
left
so
we're
at
almost
at
the
end
of
our
questions.