►
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Intro + Deep Dive: Kubernetes (Network) SIG - Tim Hockin, Google
This presentation will include an introduction to Kubernetes networking, covering the high level challenges the Kubernetes network model solves, Kubernetes specific concepts such as Services, Ingress, and NetworkPolicy, and how they interact and are typically used. We'll then dive deeper into topics currently being discussed in SIG network, opening the floor for discussion and debate on a range of topics based on attendee interest and moderated by members of the SIG. In this latter half, we hope to bring contributors and users together to discuss the problems they are currently facing and how we can collaborate to solve them.
https://sched.co/MPky
A
So
welcome
everyone
to
Sikh,
Network,
intro
I
see
a
few
familiar
faces.
Who
here
is
already
involved
with
zigge
Network
all
right,
you're.
Probably
watch
me
embarrass
myself
to
everyone
else.
Welcome
so
sig
network
is
responsible
for
more
or
less
anything
to
do
with
networking
in
kubernetes.
So
we
own
some
broad
areas
of
responsibility.
Like
can
pause,
talk
to
other
pods.
A
Can
we
have
traffic
going
into
the
cluster
and
so
on,
and
this
translates
into
a
number
of
specific
components
that
we
own
as
well
and
maintain
so
I'll
get
like
at
the
end
more
into
how
to
get
involved
to
the
sig,
but
we
meet
every
two
weeks.
We
have
like
a
video
call,
we
talk
or
argue
around
a
document,
and
then
we
also
have
like
our
mailing
lists,
our
slack
lists,
like
the
other
SIG's,
if
you've
interacted
with
them,
where
we
have
more
like
day
to
day
coordination,
a
discussion.
A
So
some
of
the
components
we
maintain
is
the
CMI
implementation
so
being
able
to
have
different
network
drivers,
we're
responsible
for
the
base
layer
of
that
we
maintain
services
and
endpoints
to
be
able
to
have
higher
level
abstractions
towards
having
instance,
groups
that
we
load
balanced.
We
own
the
coop
proxy,
which
is
a
fun
little
low-level
mechanic
for
routing
traffic
internally
around
our
cluster
DNS.
So
we
can
actually
have
discovery
of
our
services
as
well
as
some
functionality
towards
like
outside
world
DNS.
A
We
have
the
ingress,
which
is
a
fun
and
somewhat
controversial
way
of
routing
traffic
into
our
clusters.
We're
currently
looking
at
GAR
existing
ingress
spec
with
some
tweaks
and
looking
at
ingress
v2,
but
Tim
we'll
get
more
into
that
with
the
deep
dive
and
then.
Lastly,
we
also
own
the
network
policy
spec,
which
is
just
a
spec.
We
don't
have
a
controller
for
it
currently
in
core.
A
So
we
get
questions
like
this
a
lot
lately.
Last
couple
months,
we've
been
doing
a
triage
at
the
beginning
of
each
session,
and
almost
always
we
get
a
question
like
this
help.
My
pods
can't
reach
google.com
help.
My
service
doesn't
run
so
in
that
spirit,
I
thought
I
would
go
over
kind
of
the
components
and
steps
involved
in
service
to
service
networking
in
kubernetes.
A
So
this
is
like
how
we,
how
we
like
to
whiteboard
it
or
think
about
it
more
first
introducing
the
concept
we
have
some
pod.
It
goes
to
something
goes
to
another
pod
and
it
just
works,
but
there's
many
many
moving
parts
involved
in
there's
many
ways
that
you're
add-ons
can
interfere.
Your
configuration
can
be
wrong
or
we
might
have
a
bug
in
the
base
stack.
It
happens.
A
So,
as
I
was
saying,
this
is
like
our
nice
super
high-level
way
that
we'd
like
to
plan
this
the
stock
works.
This
is
even
still
a
very
high
level
view
of
what's
actually
going
on,
so
we
have
DNS.
We've
got
everything
going
through
a
proxy
we
have
Services
was
not
even
traffic
going
through
services
they're
like
an
abstract
management
concept
that
creates
and
manages
a
lot
of
other
resources.
So,
let's
kind
of
step
through
this
in
a
conceptual
order.
A
Firstly,
I'm
going
to
look
at
what
the
service
controller
actually
does.
So
when
you
create
some
service
object
in
your
cluster,
this
is
what
it's
doing.
Services
are
a
high
level.
Constructor
applaud
networking,
so
you
have
some
pool
of
pods
and
you
want
able
to
route
traffic
to
them
without
having
to
manually
track
every
pod
and
load
balance
and
wrote
and
discover
it
all
yourself.
That
would
be
very
messy.
A
This
is
one
of
the
like
distinguishing
characteristics
of
kubernetes
evolving
at
a
borg,
because
borg,
who
mostly
relied
on
just
publishing
that
endpoint
data
around
pods
and
then
clients
would
kind
of
decide
how
to
handle
it
on
their
own.
So
we
expose
a
group
of
pods
with
a
set
of
ports
and
protocols
and
so
on.
So
this
might
be
just
like
exposing
a
tea.
A
So
this
is
a
snippet
out
of
a
service
spec,
it's
a
core
object
and
we
give
it
a
name.
We
give
it
a
set
of
ports
and
those
ports
map
onto
something
on
the
container
side.
So
here
in
this
example,
we
have
a
tea
as
the
port
that
we're
exposing
to
the
world,
that's
pretty
typical
and
then
like
we
often
do.
A
We
run
our
web
server
on
some
arbitrary
port
on
a
container,
in
this
case
we're
using
TCP
there's
another
very
important
part
in
the
spec
called
the
selector
and
the
selector
is
what
makes
sure
that
we
actually
hit
some
pods.
So
it's
often
a
frequent
complaint
of
like
initially
users
when
they're
looking
at
the
docs
they're
confused.
Why
is
services
not
built
into
deployments,
and
the
answer
is
that
services?
Don't
necessarily
map
onto
a
deployment,
so
selectors
can
be
changed
and
they
can
correspond
to
multiple
things.
A
For
example,
you
might
choose
to
do
a
blue
green
deploy
of
your
application
using
labels,
so
you
have
deployment
a
deployment
be
same
label.
The
same
service
hits
them
even
though
the
different
deployment
objects.
You
can
also
play
around
with
like
evicting
and
adding
pods,
but
that's
another
story.
A
So
this
is
what
it
looks
like
in
practice.
We
have
our
service
and
it
the
service
controller,
finds
and
selects
every
single
pod
with
that
matching
selector
set.
So
this
is
watching
every
single
pod,
but
that's
not
necessarily
helpful
on
its
own,
because,
for
example,
what
if
a
pod
is
just
starting
up
what,
if
it's
still
creating
what,
if,
like
the
web
server,
hasn't
loaded?
Yet
we're
still
missing
sign
to
make
sure
that
we
have
proper
service
discovery
because
that's
an
integral
part.
A
So
another
useful
thing
that
services
do
is
having
DNS
records,
so
it
creates
a
DNS
record
corresponding
to
the
service
name.
This
means
that
we
have
song
it's
very
intuitive
to
access
and
we
don't
have
to
use
something:
that's
not
a
terminus
t'k,
for
example,
if
it
created
an
IP
we'd
need
to
create
an
object,
then
check
what
the
IP
is
and
then
try
to
work
with
that.
That's
not
very
kubernetes.
A
So
instead
we
have
the
service
DNS
record,
there's
a
particular
path:
that's
globally
accessible,
where
we
create
records,
it's
the
service
name
space
and
then
SBC
cost
or
local,
as
well
as
within
a
namespace.
We
typically
just
access
it
by
its
name
and
it's
a
standard
DNS
record.
So
we
just
access
it
reports
protocols.
However,
we
desire
the
code
for
the
controller
lives
in
Korea's
core.
Most
of
it
is
in
package
controller
service,
and
it's
run
by
something
called
coop
controller
manager.
A
A
So
next
up
we
have
endpoints
I
said
earlier
that
services
aren't
directly
resolved
for
handle.
They
aren't
directly
involved
in
handling
any
traffic,
they
create
DNS
records
and
they
create
endpoints.
What
endpoints
are
our
routes
to
individual
pods?
So
it's
a
one-to-one
relationship
of
a
pod
and
an
IP.
A
A
So
the
endpoint
object
is
what
gets
consumed
internally
for
doing
load,
balancing
and
serves
discovery,
and
it's
also
something
that
you
can
custom
use.
For
example,
one
thing
that
I've
been
working
on
is
doing
endpoint
level,
load,
balancing
and
I
know.
A
lot
of
companies
like
Google
also
have
something
like
that,
where
you
wrote
out
a
load
balancer
level
directly
to
pods
versus
going
through
the
standard
mechanisms.
Later
I'll
talk
more
about
how
those
are
implemented.
A
An
interesting
thing
about
endpoints
is
recently.
We
had
an
enhancement
proposal,
come
up
around,
adding
a
concept
of
an
endpoint
slice,
so
incriminate
is
when
we
have
some
kind
of
object
list.
It's
one
massive
list
of
everything,
so
this
might
be
a
list
of
many
many
many
hundreds
of
pause
there
in
a
deployed-
and
we
certain
scalability
concerns
with
this
at
some
point,
because
we
either
have
too
many
objects
and
@cd
has
a
scaling
problem
for
the
key
or
perhaps
it's
just
too
much
churn
too
much
data
going
around
the
cluster
at
once.
A
For
example,
if
you
have
a
list
of
endpoints
in
this
case,
every
time
endpoints
are
coming
or
going,
you
have
a
massive
list
of
changes
to
sync
out
to
your
controllers.
This
one's
I
having
performance
impacts,
not
great
I,
think
Tim
will
get
more
into
the
endpoint
slices
proposal
and
his
deep
dive,
but
there's
a
debate
to
be
had
about
that
model
because
it's
new
to
kubernetes,
we
don't
really
have
slices
or
partitioning
of
any
other
resources
yet,
and
this
proposal
could
get
messy.
A
So,
there's
a
discussion
to
be
had
around
what
compromises
we
make
and
how
much
we
try
to
eat
performance
or
change.
Some
of
our
other
models
around
having
a
singular
endpoint
list,
so
endpoint
controller
again
is
another
a
bit
of
coop
controller
manager,
magic
and
it's
in
package
controller,
endpoint.
A
So
next
up
DNS
a
lot
of
the
DNS
stuff.
We
actually
don't
own
anymore
because
it
runs
as
core
DNS
rather
than
our
own
implementation.
So
we
manage
a
fairly
simple
system
where
we
deployed
DNS
to
the
cluster
via
just
a
deployment
and
then
use
both
to
help.
My
head
I
think
it's
the
couplet
to
inject
information
around
pointing
to
that
DNS
server,
but
we
don't
actually
maintain
the
core
DNS
server
code
anymore.
A
So
I
believe
we
do
still
have
the
old
DNS
entry,
but
using
it
as
highly
ill-advised
unless
they
have
a
very
specific
reason.
We're
kind
of
going
for
this
pattern
more
and
more
in
communities
of,
rather
than
providing
a
core
implementation
of
something,
provide
an
interface
or
provide
a
standard
way
and
then
outsourcing
either
to
like
it
kind
of
for
lack
of
a
better
way
of
putting
it
singular
winner
with
coordinates,
because
it's
just
the
standard
or
something
like
Network
policy,
where
there's
many
different
options
with
their
own
trade-offs.
A
So
last
component
that
I'm
going
to
get
into
is
my
favorite
one,
and
by
that
I
mean
the
one
that
I've
spent
the
most
time
poking
at
it's
the
cube
proxy.
So
the
cube
proxy
is
responsible
for
a
lot
of
magic
behind
the
scenes,
kind
of
sticking
some
iptables
glue
in
here
and
there
and
just
making
sure
that
everything
Ravitz.
A
A
Submits
that
signal
on
the
pod
status.
Now
the
endpoint
controller
is
watching
the
pods
and
when
it
sees
pods
become
ready,
it
adds
the
corresponding
pod
IP
to
its
own
list.
So
if
you're
not
very
familiar
with
kubernetes
architecture,
there's
a
lot
of
this
kind
of
back
and
forth
pattern.
That
happens
between
controllers,
where
we
have
massive
lists
of
things
and
we
essentially
watching
maintain
independent
states.
A
So
we
have
the
endpoint
ads
the
list,
that's
something
that
route
proxy
is
constantly
pulling
for.
When
Clube
proxy
sees
the
endpoint
that's
available,
it
then
adds
some
Network
rules
in
the
host,
so
the
network
rules
are
things
like
IP,
IP
tables
or
IP
vs,
the
several
different
modes
that
I'll
talk
about
briefly
later
that
then
provide
on
host
networking.
So,
let's
see
what
that
looks
like
ku
proxy
is
a
daemon
set
on
every
single
node.
That
specifically
is
used
to
configure
the
host
network.
A
A
A
So
this
is
meant
to
be
a
stable
interface
for
connecting
to
pods
and
then
ku
proxy
is
also
responsible
for
not
just
owning
this,
but
specifically
doing
the
load
balancing.
Although
behavioral
around
load
balancing
really
varies
depending
on
the
way
they
have
KU
proxy
configured,
so
ku
proxy
right
now
has
four
modes.
There's
the
older
user
space,
one,
which
is
a
legacy
mode.
It
essentially
ran
as
a
ghost
server
that
would
terminate
connections
and
open
new
connections.
A
This
one
I
am
too
young
in
the
project
who
I've
worked
with,
but
it
was
much
more
feature-rich,
especially
than
IP
tables,
because
working
in
user
space
and
working
programmatically
with
everything
you
could
do
a
lot
of
complex
things,
so
it
had
major
scalability
issues.
If
you
look
back
and
github,
you
can
see
many
complaints
about
this
around
throughput
issues
and
cluster
size,
but
IP
tables
in
some
ways
was
a
regression
around
load.
Balancing
functionality.
Piping
tables
is
now
our
default
mode.
So
if
memory
serves,
we
don't
have
much
loading
balancing
functionality.
A
It's
a
bit
done
with
her
notes
things,
but
it
works
especially
with
IP
tables
speed
ups
under
the
hood
going
on.
But
that
said,
it
can
still
be
a
performance
constraint
just
because
once
you
have
so
many
hundreds
and
hundreds
of
IP
tables
and
rules
in
any
system,
we
start
to
have
issues
IP,
vs,
I'm,
not
100%
sure,
if
it's
actually
beta
anymore,
but
it's
our
okay,
so
IP
V
SSgA.
A
We
started
talking
about
ways
to
split
these
modes
out
more
and
maybe
deprecated
user
space,
just
because
there's
so
many
bugs
that
have
come
up
in
the
interplay
of
these
various
modes,
like
one
particular
thing
that
I
did
recently
was
deprecating
automatic
cross
mode
cleanup
because
we
found
that
there
was
some
there's
some
weird
intersection
between
all
of
these,
for
example,
user
space
and
ipbs
actually
create
some
iptables
rules.
So
this
means
that
we
can't
just
wipe
the
IP
table
stuff
clean
because
there
might
be
some
other
implications.
A
So
there
are
several
different
kinds
of
service
types
and
cube.
Proxy
is
responsible
for
the
implementation
details
of
how
they
actually
work.
First,
one
I'm
talking
about
is
just
a
very
simple
one:
it's
external
name,
so
that's
just
a
cname
to
some
other
system.
So
there's
no
cluster
IPA
doesn't
map
to
any
pods.
It's
just
a
way
of
like
taking
some
external
service
of
yours
and
making
it
look
kind
of
like
a
kubernetes
service.
A
A
That's
a
cluster
IP
iptables
rules
on
the
host
hijacks
that
the
IP
tables
rules
contains
a
list
of
50,
pods
or,
however
many
ready
pods
you
have,
and
that
picks
which
IP
within
the
cluster
to
send
that
request
to
a
question
that
we
get
a
lot
is
why
don't
we
use
dns
for
this
kind
of
thing?
Why
do
we
have
this
IP
table
set
up?
The
short
answer
is
DNS
sucks,
it's
more
complicated
than
that,
but
as
anyone
who's
tried
to
load
balance
stuff
with
DNS
is
found.
A
There's
a
lot
of
pitfalls
to
that
there's
often
unpredictable
behavior
on
client.
Caching,
where
things
can
happen
if
your
connectivity
gets
cut
off
and
a
lot
of
clients,
don't
respect
that
kind
of
load.
Balancing
it's
kind
of
up
to
whatever
library
is
using
it.
So
you'll
often
find
that
you
get
like
your
first
index
and
through
nice
record,
running
very
hot,
because
everything
is
just
picking
the
first
record
out
of
many
rather
than
picking
a
random
one.
A
Next
time
we
have
is
the
node
ports.
The
note
port
can
be
thought
of
as
like
a
very
simple
building
block
for
doing
external
connectivity.
It
just
opens
a
port
on
the
host
and
that
maps
to
any
local
containers.
If
I
remember
correctly,
it
actually
fails.
If
you
don't
have
any
containers
with
that
port
open
and
running.
A
The
more
advanced
combination
of
a
cluster
IP
and
a
node
port
is
called
the
service
type
load
balancer.
So
this
is
a
combination
of
creating
cluster
IPS,
opening
node
ports,
pointing
to
them
and
then
running
some
kind
of
cloud
provider
code
to
spin
up
an
external
load
balancer.
So
the
result
of
this
is,
if
you
hit
any
class,
any
node
in
your
cluster
it'll
contain
a
node
port
that
you
can
access
and
that
node
port
will
route
to
a
cluster
IP,
which
then
wrote
somewhere
in
your
cluster.
A
This
is
where
potta,
where
load
balancing,
which
I
mentioned
earlier,
starts
to
become
desirable,
because
you
have
a
high
amount
of
abstraction
here.
You
have
things
just
kind
of
bouncing
around,
and
this
requires
going
through
some
like
iptables
rules.
That
might
not
be
strictly
necessary,
and
it
also
means
that
you
might
be
taking
an
extra
hop
of
going
one
host
into
your
cluster
and
that
host
redirects
you
to
a
different
host.
It's
not
something!
That's
critical,
but
if
you're
very
constrained
about
performance-
or
you
have
very
high
scale
systems
that
really
starts
to
matter.
A
So
that's
the
stack
involved
in
service
service.
What
else
didn't
recover
well
is
the
ingress,
which
is
our
layer,
seven
way
of
writing
traffic
to
services
under
those.
Although
low-level
routing
concerns,
so,
for
example,
you
could
get
almost
infinitely
deep
when
talking
about
coop
proxy,
all
the
way
down
the
IP
stack
as
to
how
exactly
things
work
and
what
kind
of
fan
tangling
goes
on
and
there's
also
aspects
like
CNI
that
we
can
get
into
as
well.
A
Somewhat,
notably,
is
we
don't
really
work
with
network
overlays
in
our
sake
ourselves,
so
we
get
many
many
questions.
In
fact,
I
would
say.
Probably
one
person
a
day
bouncing
into
the
channel
saying,
hey
I,
ran
something
I
installed.
Flannel
now
my
notes,
don't
work,
fix
it
and
there's
some
very
lovely
people
who
try
to
help
if
they
can.
But
it's
not
something
it's
our
purview
and
it's
not
something
that
we
make.
A
So
I'm,
guessing
that
most
of
you
are
these
curious
about
getting
involved
since
you're.
Here,
how
do
we
do
that?
The
easiest,
open
lowest
friction
way
is
just
to
get
involved
with
github
issues,
starting
with
adding
your
own.
If
you
have
a
problem
or
if
you
want
something
so
easiest
ones
are
just
file,
feature
requests
or
file
a
bug
you
can
also,
if
you
are
more
familiar
with
like
how
urban
edits
works
or
you're
familiar
with
the
code
base
itself,
you
can
file
cleanup,
cleanup
recommendations.
A
We
have
in
many
areas
of
kubernetes,
but
it
feels
like
a
lot
in
Signet
work.
We
have
a
lot
of
deliberately
incurred
tech
debt.
The
project
tried
to
grow
very
quickly.
It's
how
we
want
the
cluster
orchestration
battle,
but
now
there's
always
stuff
to
kind
of
clean
up
and
there's
a
slow
tale
of
making
sure
that
we
don't
break
things
in
the
process
of
that.
A
So
we
do
have
I'm.
Gonna
say
this
on
camera:
to
hold
myself
accountable.
We
do
need
to
get
better
with
tagging
things
as
it
first
issue
or
Help
Wanted,
but
we
do
have
issues
that
are
meant
to
be
things
that
aren't
too
time
critical
and
are
fairly
approachable
without
a
whole
lot
of
in-depth
knowledge
to
fix.
For
example,
the
very
first
ticket
I
did
was
just
moving
some
test
files
around
to
be
more
consistent
with
our
code
based
conventions.
A
Triaging
issues
is
another
good
one.
We
thanks
to
Tim
went
through
from
hundreds
and
hundreds
of
open
question
mark
issues
to
having
only
about
50
right
now,
ountry
ozt
and
most
of
that
massive
backlog
closed.
But
that
said,
there's
always
more
work
to
do.
People
are
still
using,
kubernetes
tickets
are
still
being
filed.
A
A
Enhancements
are
basically
non-trivial
features
or
things
that
require
coordination
across
things
or
things
that
require
like
multiple
lifecycle
stages
as
we
go
across
releases.
So
there
are
things
that
are
notable
and
user
facing,
as
opposed
to
like
individual
bugs
or
just
small
bits
of
cleanup,
so
enhancements
are
kind
of
where
all
the
cool
ideas
come.
For
example,
the
ingress
GA
changes,
the
endpoint
slice
lots
of
other
stuff,
so
you
can
go
through
the
communities
enhancements,
repo,
not
sure.
A
If
there's
a
shortcut
for
it,
but
github
comm,
career
daddies
enhancements
will
take
you
there
and
you
can
look
at
the
proposals
that
we
have
merged,
which
means
that
we've
decided
that
we
agree
upon
that
phase
of
it,
and
then
you
can
also
see
the
pull
request
of
the
stuff
that
we're
still
bickering
about.
So
enhancement
proposals
go
through
a
couple,
different
phases,
starting
with
just
provisional,
which
means
we
want
to
do
this.
We
think
it's
reasonable.
We
kind
of
have
an
idea
of
how
to
do
it,
but
we're
not
necessarily
ready
to
do
it.
A
We
don't
have
a
specific
plan
to
implementable,
which
is
we
have
a
solid
plan
would
agreed
on
this.
Api
team
doesn't
have
a
problem,
there's
no
big
logistical
issues
and
so
on,
as
we
get
sawing
into
the
codebase
and
then
eventually
all
the
way
to
GA
when
it's
done
and
you
can
submit
an
some
proposals
of
your
own.
If
you
want
I
suggest,
starting
with
an
issue,
that's
like
our
first
way
to
get
traction
and
decide
if
something
is
worth
pursuing,
but
there's
nothing
stopping
you.
A
As
for
ongoing
areas
of
work,
there's
always
tons
to
do.
The
biggest
thing
that
comes
to
mind
for
me
is
just
a
cleanup
of
existing
components
and
tech.
Debt
Tim
often
likes
to
say
that
if
you
spend
15
20
minutes
looking
at
any
part
of
the
codebase
you'll
find
something
that
looks
unpleasant
and
it's
fixing.
It
took
me
a
while
to
become
familiar
enough
with
a
lot
of
things,
but
I
definitely
agree
with
that
statement.
It's
a
big
codebase.
That's
lost
its
old
its.
How
programming
works?
A
A
There's
a
couple
things
around
Louisville
networking
features
like
yes,
it's
GA,
but
still
could
do
with
some
tuning
was
work
around
dual
stock
IP
before
ipv6
and
so
on.
There's
always
work
to
do
so.
If
you're
interested
in
joining
in
I
have
links
here
to
our
main
page
on
github,
it's
kind
of
like
our
man
readme
of
a
reminder
of
what
we
do
as
well
as
where
we
talk.
So
it's
got
the
mailing
list
links
lock,
link
things
like
that,
it's
linked
to
from
the
main
community,
SIG's
page
and
honestly.
A
If
you,
google,
kubernetes
sig
network,
it
should
be
the
top
result.
I
admit
that's
shamefully
often
how
I
find
the
video
link
for
meetings
so
there's
also
our
slack
and
our
mailing
list.
Communications
are
kind
of
split
between
slack
and
the
mailing
list,
almost
based
on
preference
and
communication
style,
and
then
every
two
weeks
we
have
our
zoom
call
where
we
have
a
specific
agenda
of
what
we're
going
to
deal
with,
as
well
as
doing
some
just
cleanup
of
our
backlog
going
through
some
triage
and
most
of
those
should
be
posted
on
youtube.
A
If
you
want
to
go
through
the
archive
and
kind
of
see
what's
going
on,
if
you're
planning
on
getting
involved,
I
strongly
recommend
either
just
mostly
silently
attending
some
meetings
are
watching
some
YouTube
videos
to
kind
of
get
an
idea
of
like
our
interactions.
What's
going
on
what
we're
working
on
and
try
to
get
up
to
speed,
they're
getting
a
news.
Feed
on
github
can
only
get
you
so
far.
In
my
opinion,
that's
it
for
the
main
content.
A
A
C
There's
been
a
bunch
of
bugs
in
ipbs
that
we're
finding
like
all
good
software,
nobody
uses
it
until
it's
released.
So
now
that
it's
GA
people
are
starting
to
find
the
bugs
with
it
and
so
we're
working
through
those
bugs
still
they're.
Not
none
of
them
have
been
super
critical
bugs,
but
there
there
are
nonzero
bugs.
B
C
B
One
of
the
things
I
will
not
ask,
is
I.
Keep
finding
like
a
lot
of
the
ingress
is
just
one
layer,
seven,
what
what
layer?
Four
like
I
have
use
cases
where
I
want
to
go
into
the
system.
Looking
to
the
cluster
like
using
layer,
four
but
node
port
doesn't
really
suit
me,
because
I
just
wanna
have
more
control
of
where
the
traffic
goes.
To
so
is
any
plans
to
do
is
a
load
balancing
for
layer,
four
traffic.
C
Are
there
more
plans
to
layer
four
load
balancing
is
strictly
outside
the
bounds
of
what
committees
does
so
we
have
the
internal
cluster
load
balancer
right,
which
is
intentionally
more
of
a
load
sprayer
than
a
load
balancer.
We
when
we
talk
about
bringing
stuff
in
from
the
outside,
which
is
what
I
think
you're
getting
to.
We
have
to
talk
about
which
load
balancer
are
you
using,
and
how
does
that
work?
And
what
does
it
understand
if
you
hold
that
question
I'm
happy
to
I?
C
B
B
C
C
A
I'm
the
question
it
was
like:
are
we
splitting
up
coup
proxy
to
its
own
repository,
and
the
answer
is
almost
certainly
one
day,
hopefully
not
in
the
very
distant
future?
There's
many
many
mechanics
involved
with
that,
because
we
have
an
extreme
number
of
dependencies,
including
kind
of
circular
dependencies
in
the
project,
so
crude
proxy
should
be
theoretically
easy
in
practice
like
Lucas
and
I
were
stepping
over
it.
This
afternoon,
oh
there's
some
fun
imports,
we're
importing
all
kinds
of
stuff
from
like
API
server,
and
so
on
that
we're
going
to
need
to
detangle.
C
You
know
weep
if
I
can
pontificate
for
a
moment.
Kubernetes
we
talk
about
controllers
in
the
controller
pattern
and
how
that
really
drives
everything.
All
cue
proxy
is
is
a
controller
for
services
and
endpoints.
It
watches
services
endpoints,
and
it
makes
them
appear
on
a
machine.
So
if
you
can
replace
that
with
a
different
controller,
cool
yeah.
A
I'll
coup
proxy
is
doing
is
creating
a
cluster
IP
when
it
sees
a
service
that
should
have
a
cluster
IP
and
then
populating
end
points
from
the
endpoints
object
when
it
sees
endpoints.
So
there's
a
lot
of
fan
dangling
around
in
the
plumbing
of
like
using
IP,
vs
or
using
IP
tables.
But
fundamentally,
if
you
do
like
a
flow
diagram
of
it,
it's
doing
a
very,
very
simple
task.
A
C
The
more
abstract
answer
is:
we
have
an
interface
between
cubelet
on
each
node
and
a
plug-in
and
there's
configuration
that
you
feed
to
cubelet.
That
tells
it
how
to
which
plug-in
to
run.
There's
an
interface
called
CNI,
which
is
defined.
It's
an
exec
based
interface
right
now,
I
will
exec
your
plug-in
with
a
set
of
comparable.
C
So,
there's,
of
course,
somebody
who's
going
to
get
more
pedantic
than
me.
It
is
in
fact,
in
the
container
runtime
interface,
which
is
called
by
cubelet
and
that
execs
your
plugin
sets
up
the
container
interface
and
then
returns
the
IP
address
that
it
allocated
for
you
just
way
down
at
the
bottom
of
the
stack
I'm.
Sorry.
C
A
C
C
Thank
you
all
for
coming
here.
I
honestly,
end-of-day
last
day
of
the
conference
I
expected
about
five
people
to
show
up.
So
it's
amazing
they
were
probably
all
gonna,
be
like
people
I
know.
So
thank
you.
All
for
coming,
I
have
like
I,
said,
sort
of
way
more
material
than
I
do
time.
I
picked
a
few
topics
from
this
stream
of
things
that
are
happening
and
I
thought
I'd
go
into
them,
but
if
these
are
not
really
interesting,
we
can
also
talk
about
other
things.
So
you
know
obviously
there's
a
lot
of
questions
here.
C
So
you
know
I
love,
questions,
stop
me
just
wave
your
hand
and
scream
if
you
have
a
question
as
I'm
going
through
and
we'll
do
it
that
so
landmark
you
guys
might
have
seen
this
week.
We
love
to
think
about
the
network
kubernetes
as
like
this,
like
it's
almost
done,
but
really
I.
Think
of
it
more
like
this
there's
we
got
a
lot
of
work
to
do.
The
general
scaffolding
is
in
place,
but
there's
a
lot
of
finishing
work.
C
So
I
picked
five
topics
for
today,
I'm
happy
to
talk
about
these
or
not
talk
about
these.
As
you
see
fit,
I
could
probably
do
this
in
an
hour
and
a
half
and
I've
got
40
ish
minutes
so
and
I
don't
know
if
they're
gonna
kick
us
out
at
5:20,
but
we'll
see
I
was
going
to
talk
about
the
ingress
version
one.
So
anybody
who's
used
ingress
in
kubernetes
maybe
has
wondered
why
it's
a
v1
beta
one
for
the
last
three
and
a
half
years.
C
C
I
was
going
to
talk
about
service
topology,
the
idea
of
having
control
over
which
endpoints
of
a
service
you
choose
to
talk
to,
based
on
where
you
exist
like
same
node,
I
was
going
to
give
a
quick
update
on
IP,
VI
ipv6
and
dual
stack.
Support
and
I
was
going
to
talk
about
the
endpoints
API
reboot,
that
is
in
flight.
The
Valerie
mentioned
earlier.
I,
don't
have
any
particular
preference
order.
Oh,
apparently
we
have
people
waiting.
Can
we
again
scooch
in
there's
been
a
bunch
of
empty
people
who
have
left
so?
C
C
C
So
there's
there's
two
main
reasons:
one
there's
a
big
press
to
get
rid
of
the
extensions
API
group,
which
is
where
ingress
lives.
We
thought
that
was
a
good
idea
turns
out.
It
was
not
a
good
idea
to
put
things
all
in
this
extensions
group.
We
want
to
move
it
out
of
that
group,
so
we've
started
the
process
of
moving
out
we're
basically
the
last
API
in
the
group,
and
they
want
to
turn
the
lights
off,
but
we're
holding
them
back
from
doing
that.
C
We
could
replace
it
with
an
entirely
new
API,
which
we
actually
really
want
to
do,
and
you
could
see
this
in
some
of
the
other
talks
earlier
this
week.
But
that's
gonna
take
a
really
long
time
and
like
let's
be
realistic,
you
know
to
get
an
API
design
and
approved
by
all
the
implementations
and
then
implemented
and
then
through
alpha
beta
GA
is
going
to
take
a
year
or
more.
The
extensions
a
peg
group
has
to
go
away,
so
we
figured
now
is
a
really
good
time
to
push
it
through.
C
C
Ingress
has
more
users
than
there
are
people
on
earth
and
about
a
dozen
different
implementations
that
I
know
of
people
are
using
it
like.
It
is
a
de
facto
ga
API.
So
we're
gonna
just
admit
that
right.
But
you
know,
if
we're
gonna
change,
the
API
version
we're
gonna
change
the
API
a
little
bit
along
the
way
because
hey
why?
Why
not
so
in
the
cap
that
Bowie
wrote
I,
don't
always
hear
but
I
the
kept
that
we
wrote
for
this.
C
We've
proposed
a
half-dozen
ish
things
that
we
want
to
fix
along
the
way
for
the
API,
some
of
which
are
really
sort
of
obvious
and
some
of
which
are
actually
really
complicated
and
I'm
gonna
just
dig
into
these
a
little
bit
if
anybody's
used
ingress
before
maybe
you'll
be
nodding.
Your
head
going
yeah
I
needed
that.
C
C
C
So
if
I
define
it
as
a
prefix
API
and
you
round-trip
it
into
the
beta
API
and
then
back,
you
could
lose
information
or
if
you
defined
it
in
the
beta
API
and
you
bring
it
into
the
v1
API,
you
could
have
a
field
that
doesn't
work
right.
It
could
fail
validation.
So
we've
got
to
define
it
a
little
bit
carefully
and
how
we
can
round-trip
that
the
exact
details
of
the
API
are
still
sort
of
being
decided,
but
this
is
a
pretty
straightforward
one.
C
Also
in
the
ingress
specification,
we
have
a
host
field
and
people
have
asked
us
time
immemorial.
Does
it
support
wild
cards
and
we
didn't
specify
it
to
support
wild
cards?
I,
don't
know
if
any
of
the
implementations
support
wild
cards,
but
it's
come
up
often
enough
and
it
seems
to
be
supported.
All
of
the
implementations.
We're
gonna
make
sure
that
it
does
support
simple
wildcards
for
host
names.
C
C
So
anybody
here
runs
ingress
in
like
a
cloud
provider
and
ends
up
running
to
ingresses,
yeah
yeah,
and
now
you
have
to
specify
this
annotation.
That
tells
it
which
ingress
you
want
to
use
on
a
particular
ingress
instance
yeah.
So
we're
gonna
formalize
that
we
have
this
pattern
within
kubernetes
of
a
something
class
which
is
a
way
for
your
cluster
administrator
to
advertise
to
users.
What
types
of
a
thing
are
available.
C
We
see
this
in
like
storage
class,
so
we're
gonna,
follow
that
same
pattern
and
implement
an
ingress
class,
so
you'll
be
able
to
specify
I
want
to
ingress
and
I
want
the
cheap
one
or
I
want
an
ingress
and
I
want
the
good
one
and
you
don't
have
to
actually
know
what
that
means.
You
don't
even
know
whether
that's
Google
or
nginx,
and
you
can
decide
which
one
of
those
is
good,
which
one
of
those
is
cheap
and
as
an
administrator,
you
can
actually
change
this.
C
Basically,
it's
this:
the
users
can
now
go
shopping,
they
can
say
hey
what
ingress
classes
are
actually
available
in
my
cluster,
we
give
them
back
a
list
of
ingress
classes.
I
feel
like
I,
had
to
have
pictures
in
the
slides,
and
then
they
can
go
create
their
ingress
with
an
abstract
class,
and
now
the
controllers
aren't
gonna
fight
over
it
anymore,
ingress
status.
This
is
a
basically
non-existent.
We've
had
a
lot
of
requests
for
people
to
get
feedback
from
their
ingress
implementation
back
to
the
users
about
what's
happening
or
why
or
why
it's
not
happening.
C
Most
of
the
controllers
will
throw
events
into
kubernetes
and
you
can
sort
of
see
that
way
what's
happening.
But
events
are
not
a
programmatic,
API
they're,
more
of
a
human
API.
So
then
that
wouldn't
be
an
API
there,
a
human
interface
so
so
we're
gonna,
add
a
status
field,
we're
working
through
the
details
of
whether
we're
going
to
implement
it
as
a
CR
D,
with
a
type
specific
details
or
whether
it'll
be
sort
of
a
string
to
string
map.
C
B
C
Health
checks,
so
every
cloud
API
out
there
if
you're
like
Google
cloud
Amazon,
whatever
you're,
setting
up
your
load
balancers
and
the
API-
requires
you
to
establish
a
health
check
right
and
ingress,
doesn't
have
any
way
for
you
to
specify
what
that
health
check
is.
So
in
most
cases
we
just
assume
that
it'll
be
the
/url
and
that'll.
C
Be
fine,
we'll
hit
your
/url
and
if
you
return
200,
you
must
be
alive
right
and
that
works
for
a
lot
of
people,
but
also
not
for
a
lot
of
people,
and
so
some
implications
go
even
farther
and
say
well.
I'll,
look
at
the
service
and
I'll
look
at
the
pods
behind
the
service
and
I'll
look
and
see
if
there's
a
readiness
probe
and
if
there's
a
readiness
probe
on
the
pod,
then
I'll
assume
when
it's
type
HTTP
then
I'll
take
that
path
and
I'll
use
it
in
my
load
balancer.
C
Instead,
if
any
of
you
followed
that
I'm
impressed
it's
kind
of
a
mess,
it
explodes
far
more
often
than
I'm
happy
about
so
I
think
we're
going
to
try
to
formalize
a
just
a
very
simple
concept
of
if
this
is
an
ingress.
What
is
the
path
to
the
health
check
for
this?
For
this
back-end
service
and
protocol?
You
mean
scheme
like
HTTP,
HTTPS,
okay,.
C
And
an
expected
error
code
I
think
Bowie
Bowie.
Are
you
taking
notes
so
interesting,
we'll
have
to
we'll
have
to
work
through
the
details?
We
don't
act
expected
error
code
is
something
that's
been
asked
for
before
on
readiness,
probes
and
we've
so
far
declined
to
implement
that
mostly
because
if
you
have
control
of
it,
why
don't
you
just
convert
it
like
we're
at
a
new
URL
path,
but
I
would
love
to
hear
the
the
use
cases
for
why
we
should
add
every
new
field
we
add
to
the
API
I'm
gonna
fight
over
so.
C
C
C
So
another
thing
that
comes
up
a
lot
is
the
ability
to
have
a
load
balancer
that
fronts,
things
that
are
not
services.
There's
an
assumption
today
that
everything
behind
and
ingress
is
a
kubernetes
service
and
we've
had
requests
for
well
I'm
gonna
target.
An
external
host
like
I,
want
to
add,
as
act
as
a
proxy
to
another
host
or
I,
want
to
target
a
storage
back-end
like
in
Google
cloud.
C
You
can
set
up
a
GCS
bucket,
I
think
in
Amazon
you
can
set
up
an
s3
bucket
I
want
to
be
able
to
target
those
things
from
my
from
my
ingress
and
because
of
the
lack
of
capability.
There
people
end
up
going
behind
the
scenes
into
the
the
underlying
cloud
and
changing
things,
and
then
they
get
really
mad
when
we
undo
it
through
the
ingress
controller
and
they
call
us
and
they
yell
at
my
boss,
so
I'm
proposing
that
we
fix
this
again.
C
C
C
C
So
there's
a
list
of
things
that
we
can
fix
at
a
later
point.
Things
like
I
want
my
backends
to
all
be
HTTPS
right.
A
bunch
of
the
implementations
have
annotations
that
they
support
for
this.
It
seems
like
a
relatively
simple
thing,
but
it
didn't
come
up
in
the
most
critical
list
of
things
to
fix,
and
it
also
seems
like
it's
relatively
easy
to
fix
later.
It's
strictly
additive
right,
we're
not
changing
things.
We've
had
a
lot
of
questions
about
restricting
host
names
like
I.
C
Want
you
to
be
able
to
set
up
an
ingress,
but
I
don't
want
you
to
be
able
to
set
up
an
ingress
for
google.com
right
and
so
maybe
a
way
to
sort
of
grant
you
access
to
some
host
names
to
delegate
down
to
a
namespace
per
back-end
timeouts.
This
comes
up
a
lot.
There's
a
lot
of
annotations
for
this
I
think
this
is
one
of
the
most
popular
nginx
annotations
that
people
use
and,
oh
sorry,
I
messed
up.
C
The
last
one
is
the
backend
protocol
HTTP
the
first
one
is
only
HTTPS
like
don't
take
HTTP
traffic
right.
This
comes
up
surprisingly
often
too,
and
then
we
have
a
short
list
of
things
that
are
interesting
and
exciting,
but
really
hard
to
fix
without
fundamental
changes
to
the
API.
So
automatic
protocol
upgrades
like
the
way
ingress,
is
currently
implemented
with
all
the
cloud
providers.
It's
very
difficult
to
guarantee
that
this
is
going
to
work
on
all
the
implementations.
C
We
may
just
punt
that
one
to
the
next
version
of
the
API
cross,
namespace
secrets,
there's
a
massive
potential
security
hole
there,
but
I
also
understand
the
use
case
for
people
having
shared
secret.
Excuse
me
shared
secrets
that
they
want
to
use
their
organization
without
having
to
copy
them
into
every
namespace.
C
Is
there
a
question?
No
back
end
affinity.
There
are
so
many
different
implementations
of
back
end
affinity
that
it's
hard
to
describe
an
abstract
API
for
that
there's.
So
many
things
that
we'd
love
to
add
like
those
other
things,
but
that
there
would
be
optional.
This
was
an
idea
was
why
don't
we
add
them
and
just
say
that
they're,
optional,
right
and
I'm
really
concerned
about
the
portability
of
this
I,
really
don't
want
to
add
features
that
people
use.
C
That
would
couple
them
to
their
implementation
without
them
being
aware
that
they're
choosing
to
get
couple
to
their
implementation,
so
we've
discussed
a
little
bit
of
how.
How
would
we
make
like
rings
of
conformance
like
there's
a
core
set,
that
everybody
implements?
There's
an
optional
set
that
you
can
choose
to
implement
and
then
there's
other
stuff
that
the
implementations
are
specific
I,
don't
know
how
to
do
that
yet
and
I.
C
Don't
think
it
fits
really
well
into
the
API
that
we've
got
and
I
don't
want
to
do
a
fundamentally
new
API
today
explicit,
sharing
or
non
sharing
in
my
piece.
This
has
come
up
a
lot
like
Google
Cloud
ingress
will
Caray
will
create
a
new
IP
address
for
every
ingress
and
the
engine
s
engine
X
implementation
will
share
an
IP
address
for
every
ingress
and
what,
if
I,
want
the
opposite
of
that
right
on
a
GCP
instance,
I
could
share
an
IP
address
across
ingress
is
and
save
some
money
on
nginx.
C
Maybe
I
want
a
dedicated
IP
address,
so
I
can
set
up
separate
firewall
rules
or
other
things.
It's
right
now
it's
not
possible
to
specify
that
that's
not
gonna
fit
within
this
API
and
then
TCP
support.
There's
really
not
a
ton
of
demand
for
this.
So
far
we
tried
to
leave
room
for
it,
but
we
didn't
spend
a
lot
of
time.
Thinking
about
it,
so
I
just
want
to
sort
of
put
on
the
record
I,
don't
think
we're
gonna.
C
C
Implement
eight,
so
there
is
an
imp
TCP
support
at
the
nginx
ingress
I'm,
saying
I'm,
not
gonna
formalize,
it
I'm
not
gonna,
take
it
away
from
the
implementations
but
I'm,
just
not
gonna
formalize
it
as
part
of
the
API
oops.
It
is
at
the
kept
stage
the
enhancement
proposal
stage
you
can
find
it
I
made
some
short
links
because
I
don't
like
pasting
three
lines
of
link,
but
it
is
at
a
kept
stage.
You
can
go.
Take
a
look
at
the
proposal.
C
There's
some
debate
about
the
exact
shape
of
the
API
is:
if
you're
an
ingress
user.
I
saw
a
lot
of
hands
in
here
and
you're
interested
in
these
things.
Please
come
and
weigh
in
and
tell
us
what
you
think
about
these
requirements.
For
example,
protocol
we'd
love
to
get
those
on
the
record,
because
honestly,
I'll
probably
forget
it
between
now
and
the
time
I
get
back
to
California.
C
Remember
that
the
round-trip
requirement
is
maybe
the
the
hardest
part
of
all
of
this,
so
anything
we
add
has
to
be
able
to.
Round-Trip
definitely
could
use
help,
especially
if
for
people
who
are
implementing
ingress
is
if
you're
hearing
you're
an
implementer
I
would
love
to
hear
your
thoughts
about
how
this
will
map
to
you.
B
C
So
the
question
is
nginx.
Implementation
does
support
regex
as
a
path,
so
I'm
not
going
to
take
away
the
implementation
specific
option,
but
I'm
gonna
make
the
default
be
more
consistently
portable,
so
the
new
API,
for
example,
I'm
not
saying
this
is
the
API,
but
it
might
look
like
imagine.
There
were
three
separate
fields.
You
have
to
pick
one
path:
prefix,
implementation,
specific
and
in
the
implementation,
specific
you're
saying
well,
I
know
nginx.
I
know
this
can
be
interpreted
as
a
regex.
C
So
that's
fine
right
and
then
I
can
round-trip
it
through
the
from
the
beta.
If
you
specified
path
in
the
beta,
that's
gonna
go
straight
to
the
implementation
specific
field
and
in
the
new
API.
If
you
specified
the
the
proper
path,
that'll
go
to
a
new
field
in
the
beta,
so
we'll
have
to
map
and
convert
them,
but
I'm
not
gonna.
Take
it
I'm,
not
gonna.
Take
it
away.
The
intention
here
is
not
to
break
anybody
right.
It's
actually
the
opposite
of
the
intention.
The
intention
is
to
not
break
people.
C
C
C
So
the
the
question
was:
do
I.
Some
of
the
implementations
are
already
starting
to
do
their
own
CR
DS,
because
we
haven't
had
a
lot
of
this
and
do
I
have
a
comparison
of
this
versus
the
CR
DS.
As
we
prepared
for
this.
We
we
asked
Alejandro
the
nginx
maintainer.
What
are
the
top
most
used?
Nginx
annotations,
the
sort
of
crazy
things
that
people
are
doing
that
are
outside
of
the
ingress
scope
and
we
got
a
list.
He
actually
was
great.
C
He
gave
us
a
fold
like
sorted
list
and
we
went
through
those
and
that's
what
informed
these
these
things.
So
I
would
imagine
that
these
are
the
same
things
that
are
going
into
those
CRTs
I
haven't
looked
at
the
specific
CRTs.
If
you
have
a
link
to
one
of
the
CR
DS,
that's
being
developed,
I'd
love
to
see
it
traffic
splitting
right,
so
traffic
splitting
I
think
would
fall
into
the
heart
enough
to
implement
in
the
existing
API
that
it
probably
wouldn't
be
in
this
API.
C
C
One
of
the
advantages
of
having
two
generic
ingress
API
over
a
bunch
of
implementation.
Specific
API
is,
is
you
can
then
very
simply
iterate
over
all
the
ingress
--is
in
your
cluster,
if
I
have
an
engine,
X
ingress
and
a
traffic
ingress
and
a
Google
ingress
and
an
Amazon
ingress,
it's
very
hard
for
me
to
generically
say
show
me
all
the
ingress
--is.
So
this
is
the
reason
that
I'm
trying
to
keep
the
generic
one
usable
question.
C
C
So
I
said
behind
every
ingress
as
a
service,
but
a
lot
of
the
ingresses
actually
go
to
the
backend
pods.
How
do
they
find
those
pods
via
the
endpoints
which
are
related
to
the
service?
So
that's
what
I
don't
mean
they
actually
go
through
the
service
IP.
They
go
through
the
service
concept
right
through
the
grouping
concert
services
like
three
api's
and
one
the
you
know
the
there's
the
the
grouping
concept.
They
goes
into
concept
and
the
policy
concept.
There's
another
question
over
here:
no
all
right.
C
Let's
talk
about
dns
anybody's
ever
had
a
problem
with
dns
and
their
kubernetes
clusters.
Oh
ouch,
so
we've
we've
heard
you
there's
some
problems.
They
tell
me
if
you've
heard
these
before
UDP
can
drop,
packets
connection
tracking
for
UDP
tends
to
fill
up
connection
tracking
tables,
because
all
we
can
do
is
time
them
out.
There's
no
connection
to
track
and
it
turns
out.
Most
dns
connections
are
open,
a
connection.
Send
a
request,
get
a
respond
or
like
open
a
socket,
send
a
connection,
get
a
response.
C
C
So
if
you
experience
these
latency
graphs,
where
your
latency
is
really
good,
and
you
got
this
outlier
five
seconds,
they
were
5.000
one
right,
that's
me
and
you
know
DNS.
We
run
DNS
in
the
cluster
for
kubernetes
services
and,
unfortunately,
even
though
we
try
to
auto
scale
it
you
have
to
at
some
point
be
aware
of
it
right,
because
we
can't
get
all
the
coefficients
right
for
your
particular
use
case.
So
we
try
to
tune
it
for
the
average
use
case
and
it
turns
out
for
a
lot
of
our
larger
customers.
C
C
We
want
to
lower
the
latency.
We
want
to
get
a
high
cache
hit
rate.
We
want
to
get
rid
of
connection
tracking
for
DNS
altogether,
avoid
the
bugs
avoid
the
timeouts
I
want
to
use
TCP
to
the
upstream
DNS
server.
So
we
don't
have
to
think
about
whether
there
was
a
lost
back
in
and
I
want
to
distribute
all
the
load
among
those
those
thousand
VMs.
So
there's
a
design
and
alpha
implementation,
that's
ready
to
go.
Beta
I
gave
the
buried
the
lede.
C
C
When
you
create
your
pods,
you
pass
that
IP
address
and
we
create
a
dummy
IP
address
on
the
machine.
You
pass
that
to
cubelet
which
passes
it
to
all
the
pods.
So
when
your
pods
start
up,
it's
going
to
talk
to
the
local
cache
first,
if
we
don't
find
a
hit
in
the
cache
we'll
upstream
to
the
cluster
DNS
server,
just
like
we
always
did,
except
we'll
upstream
over
TCP
and
we'll
configure
IP
Able's
to
ignore
any
connections
to
that
local
cache.
C
So
now,
all
of
the
UDP
connections
are
not
going
to
be
connection
tracked
and
we
don't
have
to
deal
with
almost
all
the
bugs
we
want,
and
just
like
that,
we've
seen
a
really
great
response
from
the
people
who
are
using
this
that
it's
made
a
ton
of
their
bugs
go
away.
It's
made
their
latency
more
predictable,
they're
no
longer
experiencing
these
five-second
timeouts
everybody's
happy
as
I,
so
it
is
alpha
now
we're
moving
to
beta
with
very
minimal
changes.
The
results
look
really
fantastic.
C
So
if
you've
had
DNS
problems,
you
might
take
a
look
at
this.
It
should
go
beta
in
115.
What's
next
115
we're
at
15
releases
already
and
take
a
look
at
this
Pavitra
I.
Don't
know
if
she's
here
today,
she's
here
at
the
condo
he's
in
the
back
she's
doing
all
the
work
there.
If
you
have
questions
I'm
sure
she'll
be
happy
to
talk
to
you
afterwards
in
the
future.
The
things
we're
looking
at
you
know
we're
thinking
about
H
a
once.
You
put
a
node
agent
in
the
data
path,
things
get
squirrely.
C
If
it
goes
down,
you
can
affect
everybody
on
the
machine,
so
we're
looking
at
how
to
make
that
a
little
more
reliable
and
we've
got
some
future
work
that
we're
looking
at
once.
We
have
this
that
could
we
could
enable
things
like
automatic
path,
expansion
on
the
DNS
server,
because
we
could
use
the
cache
as
an
intermediary
for
a
more
customized
upstream
protocol
service,
topology
other
questions
so
far
before
I
get
in
that
all
right.
How
are
we
for
time
time?
Is
it
all
right?
We
got
it.
We
got
about
20
minutes,
awesome
service,
topology.
C
Many
times
we
have
people
who
want
to
access
this
service,
but
they
want
to
access
the
one,
that's
on
the
same
machine
that
they
are
right,
whether
that's
logging
or
monitoring
or
whatever
it's
important
that
they
access
the
one.
That's
on
the
same
machine
there's
another
use
case,
though,
like
we
have
clusters
within
Google
and
and
other
cloud
providers
that
are
spread
across
different
zones
within
a
region
right,
so
you
get
better
failure
modes
if
one's
whole
zone
goes
out,
you're
going
to
be
survive
that
outed,
but
traffic
between
zones
is
charged.
C
Traffic
within
zone
is
free,
and
so
people
say
well.
I
would
like
to
access
the
service.
That's
in
my
zone,
as
long
as
there
is
one
right
seems
like
a
totally
reasonable
thing
to
do,
and
it
also
helps
you
to
maintain
failure,
domains
right.
So
if
you
set
up
topology
and
say,
look
I'm,
always
gonna
access,
the
one,
that's
in
my
rack
right
or
the
top
of
rack
switch
or
whatever
your
whatever
topology
you
have,
because
topology
is
really
an
arbitrary
thing.
C
If
that
racks,
which
goes
out
you'll,
be
affected
but
you're
back
into
your
back
and
will
be
dead
but
you'll
be
dead
too.
So
who
cares
so
the
design
of
this?
We?
Actually
we
had
a
kept
on
like
a
year
ago
and
then
we
said
hold
on.
We
got
to
go
and
investigate
this.
It
might
be
bigger
than
this
and
we
went
off
and
we
investigated
it.
I
don't
know
if
John
is
here,
but
John
did
a
bunch
of
investigation
and
he
came
back
and
said
good
news.
C
The
simplest
possible
API
seems
like
the
best
one,
which
is
always
my
favorite
sentence
in
the
world.
It's
one
field,
it's
an
array
of
strings
and
the
idea
is
that
you
specify
a
list
of
topology
keys
that
you
care
about
what
do
I
mean
by
topology
keys?
They
are
label
keys
on
the
node
that
you
are
running
on.
So
all
nodes
are
labeled
with
kubernetes
io
/
hostname
I
can
now
specify
my
topology
keys
to
polymerase
at
io
/
hostname.
Now
it
will
always
try
to
find
me
a
back-end
on
my
same
hostname
yeah.
C
You
can
also
specify
create
topology,
dr.
Bernie's,
that
I
ozone,
and
it
will
always
try
to
find
you
one
in
your
same
zone
and
if,
at
the
end
of
the
list,
you
know
you
decide
well
I,
don't
care
pick
me
one!
Then
you
can
put
a
star
and
it'll
pick
you
one
from
any
from
any
topology
that
you
can
get
to,
but
it's
nice
because
you
can
actually
choose
to
fail
well,
rather
than
pick
something
in
a
different
region
or
different
rack
or
whatever
right.
So
you
have
your
choice
on
this.
C
So
here's
just
a
quick
example
where
you
can
throw
your
topology
keys
here
and
say
always
pick
me
one
on
the
same
zone
on
the
same
node
right,
I'm,
gonna,
look
at
your
hostname
I'm
gonna!
Look
at
the
target
hostname
in
this
case.
Its
hostname
is
really
easy
or
pick
me
one.
If
there's
none
available
on
my
host
name,
then
pick
me
one
in
the
same
zone
and
if
there's
none
available
in
my
zone
then
pick
any
end.
Point.
C
So
how
will
services
know
the
client
requests,
so
the
good
thing
about
services
being
implemented
on
every
node?
Is
they
know
what
node
you
came
from
because
you're
coming
from
the
same
node,
so
I'm
going
to
assume
if
I'm
looking
for
the
same
hostname
I'm
looking
for
an
endpoint
that
has
a
hostname
that
matches
me
right
or
if
I'm
looking
for
the
same
zone,
I'm
looking
for
an
endpoint
that
has
a
zone
that
matches
me,
it
gets
interesting
when
we
talk
about
DNS
here
to
get
through
all
this
yeah.
C
So
it's
interesting
when
you
get
into
the
design
because
things
like
DNS
with
a
headless
service,
you
want
to
give
you
a
response
that
only
includes
the
IPS
that
you
would
have
gotten
if
you
had
accessed
the
virtual
IP,
but
DNS
doesn't
know
who
you
are
right.
So
we
have
to
build
a
reverse
mapping
thing.
So
we've
got
a
couple
of
different
ideas
that
are
open
here,
one
of
the
ideas.
C
Well,
so
one
thing
is,
you
can
have
every
cube
proxy
watch
every
node,
and
now
it's
gonna
get
a
mapping
of
all
the
nodes
and
all
their
labels
right.
That
sounds
great,
but
it's
really
expensive.
Nodes
node
is
a
really
big
object
like
several
tens
of
kilobytes,
and
it
turns
fairly
frequently
today
because
it
gets
updated
whenever
a
container
image
is
polled
or
whenever
the
resources
change
or
whenever
the
heartbeat
changes.
So
we're
moving
the
heartbeat
out
we're
reducing
the
churn
there.
C
But
it's
still
a
very
big
object
when
all
you
really
want
is
the
labels.
So
we've
got
a
couple
of
different
ideas.
One
idea
is
to
introduce
a
new
resource
called
the
pod
locator,
which
is
essentially
a
reverse
mapping
for
your
pod.
Given
a
pods
name,
I
can
look
at
the
locator
with
the
same
name
and
I
can
take
it
back
and
say
well,
what
is
your
node
name
and
what
is
your
IP
address
right
and
then,
given
that
I
can
either
selectively
watch
nodes
or
I
can
build?
C
Another
option
that
we
looked
at
is
whether
we
could
teach
the
API
machinery
to
be
smarter
and
only
watch
the
metadata,
and
so
we
would
only
get
updated
on
the
watch
Channel
with
the
metadata
right,
API
machinery
team
said:
oh,
my
god,
you're
crazy,
we're
so
busy
leave
us
alone,
and
then
Justin
went
often
like
in
about
30
minutes
prototype
tit.
So
maybe
we'll
get
this.
Maybe
we
won't
we'll
see
we'll
keep
working
through
this.
C
This
design
was
sort
of
co-designed
with
the
scalability
sig,
so
that
we
would
make
sure
we
don't
reinvent
some
of
the
problems
of
the
past.
So
now
two
headless
services
in
DNS
or
any
arbitrary
other
thing
that
wants
to
comprehend
services.
I
can
now
watch
pod,
locators
and
I
can
tell,
based
on
your
pods
IP
address
what
node
you're
coming
from
and
that's
actually
a
relatively
low
overhead
operation.
C
Now,
interestingly,
it'll
also
give
us
the
ability
to
do
reverse
dns
for
pods,
so
the
status
on
this
one
while
I
have
all
the
details
in
my
head,
we're
at
a
camp,
we're
pre
or
we
are
pre
alpha
kept.
Nobody
is
currently
implementing
this,
so
we
have
the
design
mostly
worked
out,
but,
like
all
good
things
in
open-source
it
needs
a
contributor,
it's
gotten
some
movement,
then
it
got
stalled
and
you
know
this
is
how
open-source
works.
C
C
C
Well,
so
I'm
not
going
to
there's
an
order
to
keep
repeating
the
question.
There's
an
order
to
them
I'm,
not
even
going
to
put
the
ones
that
are
out
of
consideration.
So
if
there's
any
IP
address
on
the
same
node,
I'll
just
use
that
one
and
none
of
the
other
ones
I'm
not
gonna,
try
to
dynamically
balance,
it
that's
a
problem
for
like
a
service
mesh
or
something
right.
That's
way
too
hard.
I
can't
I
can't
teach
the
kernel
to
do
that.
C
C
C
That's
not
expressible
today,
but
when
we
talked
to
the
workloads,
people
they
say
yeah.
We
want
that.
We
were
going
to
do
that
at
some
point.
Then
you
could
say:
well,
you
can
already
Express
I
want
a
daemon
set
that
runs
on
every
node,
but
why
couldn't
you
say?
I
want
a
daemon
said
that
runs
in
every
zone.
Right
there
was
another
question
No.
C
Is
that
gonna
work
with
multi
cluster?
What
an
awesome
question
I,
don't
know:
I
it's
it's
a
hot
topic
today,
whether
we
should
introduce
multi
cluster
concepts
into
kubernetes
proper
or
whether
we
should
be
building
those
strictly
on
top.
It's
literally
I,
don't
know
how
I
feel
about
this.
On
the
one
hand,
I
want
kubernetes
to
stay
small
and
get
boring,
so
we
stop
adding
new
concepts.
C
At
the
same
time,
I
want
to
make
sure
that
multi
cluster
is
a
viable
thing,
because
I
think
everybody
is
going
to
be
in
that
situation,
so
we're
trying
to
figure
out
the
exact
boundaries
there.
We
you
know
at
Google
we're
pushing
on
a
couple
things
that
within
the
sig
we're
pushing
on
a
couple
things.
We've
got
some
multi
cluster
ingress
capabilities
at
Google
that
we're
happy
with
a
little
bit
but
not
happy
with
a
little
bit.
So
we're
gonna
keep
evolving
those
ideas.
There's
another
question
in
the
back.
C
C
C
C
C
Anybody
here,
ipv6
nice,
so
kubernetes
has
had
ipv6
support
for
a
while,
like
a
year
and
a
half
or
something,
but
it's
stuck
in
alpha
state,
because
we
need
continuous
integration
for
it
and
the
team
that
was
working
on
it.
Like
all
good
things,
an
open-source
got
sort
of
repurposed
and
we
need
help.
So
we
took
a
step
back
and
we're
looking
at
it,
and
you
know,
v6
single
stack
is
interesting,
but
it's
not
really
the
goal.
C
Do
Oh
Bob!
You
work
me
right
into
my
next
couple
of
slides.
The
question
was:
can
you
explain
what
I
mean
by
dual-stack?
I
will
do
exactly
that
in
30
seconds.
There's
a
cap,
it's
a
huge
cap.
It
is
incredibly
complicated
cap.
It
touches
tons
of
the
system,
it
has
forced
us
to
go
out
and
define
new
API
conventions
for
how
to
do
things
and
how
to
evolve.
This
is
part
of
why
it's
taken
so
long
to
get
here.
It
touches
almost
every
binary
component
of
the
system
and
it
touches
a
half-dozen
different
API
resources.
C
So
we
have
broken
up
the
development
into
two
stages.
The
first
stage
is:
let's
go
through
all
those
fields
in
our
API
that
talk
about
an
IP
address
in
a
singular
and
make
them
plural
and
we'll
make
them
ready
for
two
families
of
IP
address.
Right
so
this
is
there's
probably
six
or
eight
fields
throughout
the
different
API
resources
that
say
address
now
they
have
to
say
address
is
so
we
had
to
invent
a
convention
in
kubernetes
to
pluralize
an
existing
field
and
still
be
compatible
question.
C
So
the
question
is
why
I
only
have
two
addresses
so
yeah,
so
we're
not
stopping
at
two
interfaces
but
we're
starting
at
two
interfaces.
So
I'm,
not
gonna,
say
there's
now
address
1
and
address
it's
gonna
be
an
addresses
list.
Those
it'll
all
be
lists
right,
so
we're
making
room
for
multiple
addresses
later,
but
we're
not
focusing
on
that
use
case
right
now.
Right,
we're
right
now
we're
assuming
in
that
list
is
going
to
be
at
most
one
v4
address
in
one
v6
address
yeah
question
I.
C
Am
NOT
a
v6
I'm?
Not
so
the
question
is:
do
I
expect
that
people
will
use
unique
link
locals
or
public
addresses,
I,
don't
know,
I'm,
not
a
v6
expert,
honestly
and
so
I'm.
Looking
to
the
people
who
are
v6
experts
to
tell
me
how
this
works,
how
it
works
best
for
for
these
situations,
from
what
I've
seen
so
far,
I
think
they'll
be
using
the
public
addresses,
not
the
link
locals.
So
this
touches
a
dozen
a
half
dozen
ish
API
resources
it
touches.
C
Almost
every
binary
has
at
least
one
flag
that
has
something
to
do
with
an
IP
address,
whether
it's
a
listen
address
or
publishing,
address,
or
something
like
that.
So
these
things
all
need
to
become
pluralized.
We
need
to
fix
all
the
validation
that
checks
for
v4
addresses
and
make
sure
that
we
can
handle
these
six
addresses,
and
this
will
make
all
the
pods
be
dual
stack.
So
now
your
CNI
plugins,
which
can
already
actually
return
two
IP
addresses,
will
actually
be
able
to
take
those
addresses
and
publish
them
all
the
way
up
into
pods.
C
And
when
you
go
look
at
your
pods
through
the
API
server
you'll
be
able
to
see
that
there
are
two
different
addresses.
There's
also
the
hope
to
make
host
ports
be
dual
stack,
so
host
ports
are
implemented
today
by
mostly
by
cubelet,
and
that
would
set
up
forwarding
rules
between
the
host
and
the
pods.
And
that's
it
that's
phase
one
right:
I
want
to
get
that
in
debugged
and
established
phase.
Two,
let's
make
endpoints
support,
multiply,
Peas
and
multiple
families
will
add
DNS
support,
so
you'll
have
a
and
quad-a
records.
C
Now
some
of
these
things
might
flip
between
phases,
depending
on
how
hard
they
are
to
actually
implement
we'll
talk
about
making
node
ports
and
ingress
controllers
and
probes
all
dual-stack
there's
some
talk
about
whether
we
need
two
options
to
control
the
dual
stack:
nough
Slyke.
If
my
application
only
understands
before,
should
I
be
able
to
say
that
so
that
nobody
tries
to
reach
me
on
v6
or
vice
versa,
and
then
do
we
make
services
dual
stack.
Are
we
gonna
actually
take
two
different
service
IP
range
and
allocate
different
ciders
for
v4
and
v6?
C
That
is
a
to
be
decided.
Question
that
is
essentially
gravy
once
you've
done
all
the
rest
of
the
work
that
that
becomes
a
relatively
simple
thing
to
do.
If
we
want
to
do
it,
I
anticipate
that
this
work
will
take
the
better
part
of
the
rest
of
this
calendar
year,
apparently
there's
a
pull
request
in
my
mailbox
right
now
that
locky
was
yelling
at
me
to
go
review
before
code
freeze
next
week,
so
great
we're
seeing
real
progress
here,
which
is
fantastic
questions.
C
C
So
how
do
I
expect
DNS
to
work?
I
will
expect
it
to
work,
however,
that
whatever
the
client
is
going
to
ask
for,
like
we,
don't
really
have
control
over
whether
they
ask
for
a
and
quad-a
in
parallel
or
if
they
ask
for
them
in
sequence.
Right
we've
been
bitten
before
by
trying
to
assume
too
much
about
what
in
client
implementations
of
dns
are
going
to
do.
There
is
zero
consistency
across
clients,
so
I
don't
think
we
have
a
whole
lot
of
control
over
that.
C
My
pv6
will
probably
work
like
before
yeah
I,
guess,
I
assume
so
IP
management
becomes
easier.
Already.
Many
hosts
already
get
a
easier
slash,
64
/
96
assigned
to
them,
so
the
IP
management,
the
stuff
that
we
have
to
do
with
cloud
providers
today,
gets
a
little
bit
easier
for
the
v6
side.
I
think
we're
gonna
feel
this
out
as
we
go.
C
Yes,
yes,
so
we
didn't
talk
yesterday
on
alias
IPS
in
Google
cloud.
Google
cloud
does
not
support
ipv6,
yet
so
I
don't
know
exactly
how
our
implementation
will
look
when
and
if
we
get
there.
Microsoft
does
support
ipv6,
so
they're
they're
driving
this
work
right
now.
This
is
the
beauty
of
open
source
communities.
C
The
last
topic
that
I
prepared
for
today,
though
I'm
happy
to
talk
about
other
things
that
people
want
to
talk
about.
Other
things
was
a
rethink
of
the
endpoints
API.
So
let's
look
real
quickly
at
the
problem.
Today,
our
endpoints
API
is
a
monolithic
resource.
All
of
the
endpoints
for
a
single
service
are
in
a
single
object.
This
can
get
very
large.
We
have
customers
who
are
trying
to
run
thousands
or
tens
of
thousands
of
endpoints
behind
a
single
service.
Each
endpoint
takes
up
a
few
hundred
bytes
of
that
resource.
C
You
end
up
with
research
with
endpoint
resources
that
are
now
literally
mega.
Almost
a
megabyte
big,
which
causes
pain
for
SCD,
maybe
has
a
limit
in
the
amount
of
memory
that
it's
going
to
store
resources
and
it
causes
pain
for
API
servers
which
have
to
serialize
and
deserialize
the
resources
to
to
and
from
proto
or
JSON,
as
an
example
of
the
math.
C
So
we
work
together
with
sig,
Network
and
sig
scalability
to
try
to
find
a
new
endpoint
API.
That
would
sort
of
trade-off
between
this
and
some
of
the
other
scalability
issues
that
API
server
and
sed
will
have.
The
result.
Is
a
dock
I've
linked
it
here,
min
I,
don't
know
if
you're
in
the
room
but
min
hun
row
at
the
dock
and
we'll
be
turning
it
into
a
kept
any
minute.
Now
the
short
story
is
we're.
C
Gonna
take
this
model
at
the
endpoint
and
break
it
up
into
chunks,
and
the
actual
size
of
the
chunks
is
now
a
parameter.
So
we
can
actually
experiment
with
this
and
say
we
can
set
this
parameter
all
the
way
down
to
1.
We
could
set
the
parameter
up
to
a
hundred
or
something
like
that.
We've
chosen
100
right
now
as
a
default,
because
most
services
are
less
than
a
hundred
endpoints.
C
You
know
looking
at
data
talking
to
people
which
means
for
most
services,
the
actual
behavior
isn't
going
to
change
and
for
those
large
services
you're
going
to
get
this
reduction
in
throughput
and
SCD
rights
and
and
these
other
operations,
the
dock
is
great.
It
has
a
nice
table
comparing
different
trade-off
values
and
there's
basically
an
elbow
in
the
graph,
and
this
will
replace
the
endpoints
API
for
entry
users,
so
specifically
cube
proxy,
but
we're
not
going
to
get
rid
of
the
endpoints
API,
because
that's
a
ga
supported
API.
C
C
If
you're
building
higher-level
systems
that
use
endpoints
for
a
service
discovery
or
mesh
or
whatever,
I
would
also
love
to
talk
to
you
about
how
this
fits
with
your
higher-level
systems
that
use
those
api's,
Wow
I
actually
got
through
all
that
material.
That's
amazing,
I
know
this
is
really
dense.
I
didn't
spend
too
much
time
trying
to
make
it
fluffy
I
figured
the
people
who
are
here
on
Thursday
afternoon
at
4
o'clock.
5
o'clock
have
chosen
to
be
here
so
blame
yourself.
C
If
you
take
away
just
a
short
message
from
this,
we
have
a
lot
going
on
and
signal
work.
This
is
not
the
full
stream
of
projects
that
are
going
on
or
the
full
gamut
of
ideas
that
we've
got
on
the
table.
We
have
a
million
or
not
a
million,
so
I
guess
a
couple
hundred
now
bugs
that
are
open.
We've
done
the
triage.
Thank
thank
the
Lord,
but
there
are
still
dozens
and
dozens
of
real
bugs
that
are
open.
C
So,
if
you're
interested
in
getting
involved
in
Signet
work,
you
can
jump
in
and
try
to
help
reproduce
or
fix
bugs
if
you're
experiencing
bugs
please
file
them.
We
have
lots
of
cleanups
that
need
to
be
doing
code,
refactoring
or
new
new,
smaller
features
that
we
can
use
help
with
if
you're
interested
in
getting
involved
in
Signet
work.
The
first
thing
to
do
is
join
get
on
the
mailing
list,
jump
on
the
slack
channel
jump
on
our
zoom
calls
they're,
not
a
really
euro
friendly
time
right
now,
but
I'm
open
to
moving
that
time.
C
C
We're
at
about
time
so
I'm
happy
to
take
questions.
I,
don't
think
they're
gonna
kick
us
out,
because
I
think
we're
the
last
session.
So
I'll
just
hang
out
here.
Sorry,
you
know:
okay!
Well,
we
get
kicked
out
the
hardest.
I
guess,
but
I'm
gonna
sit
here
until
they
drag
me
away.
So
if
anybody's
got
questions
or
wants
to
talk
about
stuff
or
go
into
things
that
we
didn't
talk
about
here,
now's
your
chance.