►
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Co-Evolution of Kubernetes and GCP Networking - Purvi Desai & Tim Hockin, Google
This talk will explore the trials and tribulations behind the evolution of GCP Networking for Kubernetes and evolution of Kubernetes to meet provider requirements. Our key guiding principle is to provide managed solution while maintaining portability and consistency. Question of consistency with OSS vs provider’s solution is not just on providers’ minds but also on our users’ minds as they decide whether to use OSS or managed offering. We have gone through the journey from supporting subset of features to making Kubernetes a first-class platform for networking. But there are still open questions. With ecosystem extending to hybrid, multi-cluster and service mesh, these questions become even more relevant. Talk will cover learnings on how we see GCP networking evolving in offering Kubernetes solution and in evolving Kubernetes to provide better extensibility for providers.
https://sched.co/MPb9
A
Good
afternoon
everybody,
my
name
is
pervy.
In
about
few
years,
back,
I
was
given
an
honor
to
run
kubernetes
networking
on
TCP
and
I
was
literally
new
I'm
still
little
II
knew
it
was
out
of
selfish
interest
that
I
wanted
to
understand
the
evolution
of
copán.
It
isn't
working
and
GCP
networking
and
extract
the
principles
that
we
could
use
to
shape
at
our
future,
and
a
thought
occurred
to
share
this
with
larger
community.
Hence
this
stock.
A
B
Thank
You
Purvi,
so
I
want
to
talk
first,
just
a
little
bit
of
background
like
why?
Why
communities?
Why
is
it
exciting
to
people?
And
you
know
why
did
it
catch
hold?
Why
are
we
here
in
Spain,
with
8,000
of
my
best
friends
talking
about
this
this
thing
and
so,
from
my
perspective,
I
think,
there's
there's
a
few
really
main
points.
B
I
think
kubernetes
is
is
successful
because
we
are
ruthlessly
focused
on
the
application
and
we
didn't
really
want
to
conflate
infrastructure
with
applications,
and
so
you
can
see
this
in
a
bunch
of
different
places
in
our
API
and
I.
Think
this
gives
us
an
opportunity
to
talk
to
application
owners
and
application
developers
in
a
way
that
maybe
other
systems
couldn't.
At
the
same
time,
we
had
to
be
opinionated
enough.
We
had
to
take
certain
opinions
and
positions
on
how
we
wanted
things
to
work
in
order
to
derive
the
user
experiences
that
we
really
wanted.
B
B
We've
designed
this
system,
obviously
to
work
with
popular
open
source,
apps
and
I
think
this
is
really
the
driving
force
that
makes
adoption
of
communities
possible.
We
wanted
to
focus
on
patterns
that
people
already
know
how
to
do
right.
How
to
run
my
sequel.
We
didn't
require
you
to
change
your
applications,
but
we
wanted
to
open
new
doors
and
and
change
the
way
people
think
about
how
these
sorts
of
applications
would
be
run.
B
You
can't
build,
you
can't
build
a
distributed
cloud
native
system
without
networking
right.
Networking
is
really
at
the
heart
of
communities,
but
it
can
be
really
hard
like
I'm,
not
gonna,
do
a
show
of
hands,
but,
like
everybody,
struggles
with
networking
at
some
point
or
another
and
kubernetes
doesn't
make
it
any
easier.
If
you
don't
know
what
you're
doing
at
the
beginning,
so
we
will
really
wanted
to
to
try
to
break
through
that,
and
networking
varies
between
environments.
B
The
idea
that
if
you
want
to
build
a
full
stack
app,
you
need
to
understand
layer,
3
networking,
so
let's
take
a
step
back
and
maybe
do
a
quick
survey
of
where
we
came
from
where
we
were
at
the
beginning
of
kubernetes
and
what
was
going
on
so
kubernetes
started
with
this
system
called
borg
internally
and
borg
successor
as
omega.
We
took
the
ideas
from
Borgan
omega.
B
We
really
wanted
to
to
build
on
this
idea
of
containerization
and
as
we
started,
we
had
to
take
a
look
around
what
else
was
going
on
in
the
world
when
it
came
to
networking,
because
you
know
borg
had
some
opinions,
but
maybe
they
weren't
the
best
opinions.
So
let
me
start
and
I'll
tell
you
just
a
little
bit
about
how
borg
works
with
networking
borg
is
all
about
shared
machines
and
they
have
a
shared
nic.
Everybody
shares
the
the
resources
on
the
machine
we
didn't
have.
B
B
One
of
you
would
get
it
one
of
you
would
not
get
it.
One
of
you
would
crash
and
then
the
other
person
would
get
it.
It
would
be.
It
was
a
nightmare
and
so
the
end
result
was
all
ports
were
allocated
by
the
Borg
system
right
so
part
of
the
scheduling
pass.
Was
you
tell
us
how
many
ports
you
need?
We
would
decide
that
the
ports
for
you
and
then
we
would
do
variable
expansion
into
your
config
language.
B
So
you
could
pass
a
dash
dash
port
flag
to
your
application,
and
then
you
had
to
use
the
port
in
the
URL
of
your
monitoring
server
so
that
you
could
find
it
and
the
ports
were
going
to
change
all
the
time
because
they
were
dynamic,
and
so,
if
you
took
a
URL
and
you
sent
it
over
Hangouts
to
your
friends,
so
they
could
go,
look
at
a
bug
report
or
something
well.
The
port
might
have
changed
by
the
time
it
got
there.
It
was.
It
was
a
mess
at
the
end
of
it.
B
We
needed
our
own
naming
system
because
DNS
wasn't
gonna
work
for
this
anymore,
and
so
you
can
sort
of
see
how
this
design
point
propagates
through
the
system,
even
to
the
extent
that
we
wrote
a
kernel
namespace
to
govern
which
ports
are
allowed
to
open,
so
that
we
could
stop
people
from
stepping
on
each
other's
toes
so
2013
what
happened?
Docker
docker
happened.
We
took
a
look
at
at
what
doctor
was
doing.
Docker
took
this
idea
of
sharing
a
machine
and
they
added
namespaces
and
man.
Namespaces
Liffe
I
had
a
ton
of
things
right.
B
It
gave
you
this
really
nice
user
experience.
You've
got
a
pretend
machine
with
your
own
interface,
but
fundamentally
it
was
still
a
machine
based
model.
Every
machine
is
its
own
Island.
They
get
the
same.
Ips
they're
gonna
allocate
IPS
to
the
containers
running
on
that
machine
and
if
you
want
to
talk
between
two
different
machines,
you've
got
to
come
up
with
some
sort
of
port
mapping
on
the
host
which
will
forward
to
your
containers
and
the
packet
gets
translated
at
the
edges.
B
So
you
actually
lose
the
source
IP
and
you
can't
figure
out
the
destination
or
so
the
source,
port
and-
and
yes,
report,
end
IP,
sorry
and
it
gets
it's
complicated
and
entirely
different
way,
and
we
didn't
really
like
it
at
the
end
of
the
day,
how
an
app
sees
itself
was
not
at
all
how
other
apps
see
it
right.
I
couldn't
get
my
own
IP
address
and
tell
you
about
it
and
expect
you
to
be
able
to
reach
me.
It
was.
B
I
want
to
listen
on
port
80,
I,
don't
care
if
somebody
else
wants
to
listen
on
port
80
I
want
to
listen
on
port
80.
It
shouldn't
be
my
problem
if
somebody
else
wants
to,
and
so
this
is
evident
in
the
design
of
kubernetes
in
a
bunch
of
places,
but
probably
nowhere
more
than
networking.
So
we
started
with
the
kubernetes
model:
let's
not
share
ports.
Let's
start
with
this
assumption.
B
B
Networking
is
circa.
2013
was
not
very
friendly,
not
very
programmable,
but
we
took
another
principle.
Here
is
kubernetes
willing
to
be
limited
by
what
we
can
achieve
today.
No,
we
decided
early
on
that.
We
were
not
going
to
be
limited
by
that
we
were
gonna,
take
some
strategic
bets
on
future
technology.
So
we
bet
on
Sdn
and
programmable
networks,
and
we
said
we're
going
to
figure
out
the
UX
that
we
want
our
developers
to
have
so
we
took
up
this
model.
B
Everything
is
direct
flat
network,
no
need
for
NAT
no
need
to
manage
port
collisions,
it
works
the
way
you
would
expect
it
to
work.
That
was
the
idea
now.
Can
I
actually
make
this
work
right,
so
we
started
to
do
some
prototyping
some
experimenting
on
Google
Cloud.
We
did
this
proof-of-concept
out
to
our
networking
team,
a
wonderful
GCP
networking
team
and
they
said
who
are
you
go
away?
We
were
nobody.
Kubernetes
was
not
a
thing
yet,
and
so
we
didn't
have
any
leverage.
We
didn't
have
any
ability
to
ask
for
changes
to
the
networking
stack.
B
They
said
this
is
what
we
have
live
with
it.
Okay,
so
we
we
did.
We
took
a
step
back.
I
said
the
networking.
Api's
are
very
VM,
centric,
not
surprising,
given
that
it
was
an
infrastructure
platform.
What
we
were
asking
for
is
not
part
of
the
design
space.
So
what
are
we
going
to
do
with
that?
Well,
we
found
this
little
crack.
It's
called
a
routes
API
in
Google
cloud,
and
it
led
us
coax
the
network
into
carrying
IP
addresses
to
a
VM
that
weren't
specifically
assigned
to
that
VM.
B
In
order
to
do
that,
we
had
to
disable
things
like
IP
spoofing
protections,
which
is,
you
know,
probably
not
a
great
idea,
and
we
had
to
go
out
and
configure
we
kind
of
hear
one
static
route
per
kubernetes
node.
We
had
to
do
IP
address
management.
This
really
did
not
scale
well.
At
the
initial
limit
that
we
ran
into
was
about
50
nodes.
We
can
only
get
to
about
50
all.
A
Right
so
the
fact
that
this
world
basically
was
encouraging
in
the
RUS
to
press
this
issue
forward
and
the
UX
was
really
good.
The
IP
/
part
actually
worked
and
not
only
did
work.
It
looks
simple
and
all
the
things
look
buttoned
up,
but
the
50
routes
were
not
going
to
scale
that
well,
we
knew
it
somehow.
Last
long,
so
there
was
a
lot
of
persuasion
which
hippie
networking
team
and
the
route
limits
were
posted
200,
and
that
is
kubernetes
1.0,
which
was
shipped
now
a
question.
A
Does
anybody
know
how
many
nodes
kubernetes
1.2
support
it
can
I
give
the
answer.
100
so
fact,
basically,
is
that
here
is
where
the
coevolution
of
kubernetes
networking
and
cheese
peanut
working
starts
and
in
the
next
few
slides
in
the
next
few
minutes.
We
will
go
over
how
this
evolution
has
gone
through
in
terms
of
cluster
networking
in
terms
of
services
and
the
Lord
balancers.
A
They
had
followed
balancers
and
the
l7
or
balancers
beginning
with
the
cluster
networking
the
cluster
and
working
work
based
on
model,
and
it
worked
very
well
and
this
drove
the
GCP
networking
team
to
scale
this
scale.
The
routes
even
more,
we
went
from
50
to
5,000
in
less
than
two
years
and
that
drove
major
architectural
changes
within
in
within
the
g-sync
networking
team,
and
it
worked
well,
but
there's
a
catch
and
what's
the
catch
here,
there
are
few
issues
that
we
just
talked
about.
Like
IP
spoofing
disable.
A
We
don't
want
that,
but
then
there
was
there
was
something
more
than
that
which
is
the
IP
for
the
pod
ranges
were
not
visible
on
the
Sdn
of
network
fabric,
so
she's
the
Google
clouds
Sdn
had
no
way
to
reason.
What
are
the
part
I
piece
now
if
I
want
to
do
deeper
integration
with
load,
balancer
or
telemetry
I
have
to
infer
from
routes
which
is
not
really
the
right
way
of
going
forward.
It's
not
going
to
work.
A
Also,
since
this
ranges
were
not
this
libanon
PSTN,
you
could
imagine
there
could
be
future
use
of
the
same
range
and
that
causes
the
collision
and
last,
but
not
the
least
they.
Even
if
nothing
of
this
was
a
problem,
the
fact
that
could
be
overlapping
ranges
causes
confusion
and
it
was
hard
to
debug
all
in
all.
We
knew
we
can
do
better
so
why
we
were
scaling
the
routes.
A
There
was
another
angle
which
was
going
on
in
terms
of
how
do
we
really
get
a
better
integration
going
in
terms
of
different
TCP
networking
products
and
kubernetes,
and
the
second,
which
you
will
see,
a
pattern
that
will
reoccur?
How
do
we
make
it
easy
to
reason,
intuitive
and
easy
to
debug,
so
the
alias
IPS
actually
were
designed
to
begin
some
to
address
some
of
these
issues,
deeper
integration
and
easy
to
debug
and
easy
to
maintain
and
intuitive
okay.
A
So
here
is
a
GK
cluster.
In
this
illustration,
there
are
two
nodes:
Nords
get
their
range
from
VPC,
just
like
before,
even
they're
out
more
it
caught
the
same
thing
and
then
now
the
next
thing,
which
is
the
pod
Rangers,
the
pod
Rangers
and
a
service
ranger,
is
now
I
allocated
from
VP
Singh.
So
ABC
knows
that
this
is
going
to
be
a
special
use
of
this
rangers
by
GK
when
the
VM
gets
created.
At
that
point,
the
per
node
pod
range
is
also
allocated
in
form
of
alias
IPs
by
the
SDM.
A
A
Another,
the
simplicity
of
this
is
that
perd
nord
I
Pam,
went
to
Sdn
so
earlier.
If
the
node
had
to
node
was
started,
the
kubernetes
had
to
stop
is
scheduling.
Get
the
per
node
range.
Give
the
note
that
range,
then
travel
would
be
programmed
after
which
the
scheduling
would
be
unblocked.
All
of
this
is
gone,
and
now
they
are
exactly
the
same
as
any
other
instance,
groups
in
cloud
networking
and
the
purport
the
purple
dye
Pam
is
still
within
the
node,
just
like
in
the
kubernetes
using
CNI.
A
B
So
we've
now
got
pod
networking
and
pod.
Networking
is
now
deeply
integrated
with
the
rest
of
the
GCP
network,
which
is
fantastic.
Pods
are
not
the
only
networking
construct
that
we
need
to
deal
with.
Kubernetes
services
is
a
very
fundamental
part
of
the
surface
area
and
part
of
the
model.
I
want
to
take
a
moment
to
look
at
the
design
of
services,
so
services,
specifically
service
type,
equals
load.
B
Balancer
was
designed
from
the
beginning
to
be
centered
on
clouds
to
make
sure
that
we
understood
and
adapted
to
clouds
before
I
get
too
deep
in
that
I
won't
take
a
step
back
and
highlight
a
key
difference
that
sort
of
drove
some
of
the
design.
There's
two
basic
types
of
load:
balancers
that
you'll
find
in
the
various
cloud
products.
There's
virtual
IP
VIP
like
proxy
VIP
like
load,
balancers
and
proxy,
like
load,
balancers
and
I'll
talk
a
little
bit
about
the
difference
between
the
two
and
and
hopefully
you'll
see
why
it's
important.
B
So
a
VIP
like
load
balancer
is
always
going
to
deliver
packets
to
the
end
machine
as
if
they
came
directly
to
the
machine
and
not
through
the
load
balancer.
The
packet
contains
the
destination
IP
of
the
VIP
and
the
source
IP
of
the
original
client
that
the
load
bouncer
has
effectively
transparent,
and
the
assumption
here
is
that
the
VM
or
the
machine
itself
can
respond
directly
to
that
client
IP
and
you
don't
need
to
go
back
through
the
load.
Bouncer
I
call
this
direct
server
return
or
direct
server
response,
and
it's
a
very
powerful
system.
B
Now,
when
you
look
at
this,
is
how
Google
clouds
Network
load
balancer
is
implemented.
When
you
look
at
this,
the
load
balancer
again,
it
only
understands
VMs,
just
like
the
network
in
the
Sdn,
only
understood
vm's,
the
load
balancer
only
understands
VMs,
so
we
have
to
convert
between
a
VM,
zip
and
a
pod
endpoint
by
running
it
through
IP
tables.
So
we
have
a
set
of
IP
tables
rules.
B
Anybody
who's
dealt
with
kubernetes
long
enough
has
bumped
their
head
against
IP
tables,
I'm
sure,
and
we
use
those
IP
tables
to
translate
from
a
virtual
IP
into
a
particular
service
and
then
into
a
particular
pod.
A
nice
thing
is
it's
implanted
with
you.
Basically,
the
same
set
of
rules
that
you
use
to
implement
the
in
cluster
Service
IPS,
so
it
wasn't
particularly
onerous
to
implement
this,
but
this
is
how
the
load
balancers
work.
B
So
now
we
talk
to
our
friends
at
other
cloud
and
other
clouds
don't
have
a
fit
like
load,
bounce
or
didn't
at
the
time.
Anyway,
they
had
a
proxy
like
load,
balancer
and
a
proxy
like
load.
Balancer
is
very
different.
It
terminates
that
TCP
connection
and
it
opens
a
new
connection
on
the
back.
So
what
you
see
at
the
end
is
a
packet,
that's
from
the
load
balancer
to
the
the
VM
right.
B
B
Node
port
allocate
support
from
our
reserve
pool
in
kubernetes
and
opens
that
port
on
all
the
machines
in
your
cluster,
all
of
them
and
whenever
you
access
that
port
on
any
machine,
it's
going
to
forward
it
into
your
service,
and
now
you
can
tell
based
on
that
port
number
that
you're
destined
for
a
specific
service,
and
this
is
a
simple,
simple,
primitive
which
we
can
then
use.
Then
we
can
use
to
to
build
these
proxy
like
load
balancers.
B
We
can
now
target
the
VMS
on
a
specific
port,
and
now
we
have
enough
information
to
figure
out
which
service
you
are
talking
to.
Now.
You
can
sort
of
see
the
difference
between
these
two
models.
Right.
They
require
very
different.
Ip
tables
rules
are
very
different
on
node
processing
to
set
them
up
fit
like
load
balancers
there,
the
handles
the
same
as
in
cluster
VIPs
proxy,
like
load
balancers,
handle
by
node
ports.
I,
don't
want
users
to
experience
that
that
is
not
the
UX
that
I
wanted
I.
B
If
you
have
to
write
different
yamo
for
different
clouds,
I
have
failed.
So
how
do
we
get
past
that
so
the
design
of
the
communities
service
model?
Is
we
call
it
concentric
when
you
allocate
a
service,
a
plain
old
service,
you're
going
to
get
a
cluster
IP?
If
you
want
a
node
port
allocated
to
your
service,
you're
going
to
get
a
cluster
IP
and
a
node
port,
and
if
you
want
a
load,
balancer
you're
going
to
get
a
cluster
IP
and
a
node
port
and
a
and
s
Aria
and
a
load
balancer?
B
What
this
lets
us
do
is
to
translate
your
llamo
in
to
a
consistent
form.
So
now,
if
you're,
using
a
VIP,
if
you're
using
a
VP
load,
balancer
I
can
tell
from
the
status
load
balancer
ingress,
IP
I
can
inspect
any
packet
until
you're
going
for
the
particular
service.
In
this
case,
front-end
I
can
also
tell
if
I'm
receiving
traffic
on
a
local
IP
address,
and
it's
aimed
at
port
306
69,
which
is
allocated
by
the
server.
So
you
don't
have
to
step
on
each
other's
toes
again.
B
I
can
also
tell
that
you're
going
to
the
front
end
service,
so
as
an
average
user,
I
can
write
that
really
simple.
Amyl
and
I
don't
need
to
worry
about
the
difference
between
the
different
cloud
providers
and
different
environments,
I'm
going
to
get
something
that
looks
like
this.
After
all,
the
defaulting
has
been
applied
and
it's
gonna
work
on
the
different
providers.
All.
A
Right
so
we
just
went
over
the
services
and
how
services
abstraction
evolved
to
meet
the
load.
Balancers.
Now
we'll
go
over
the
l7
load
balancers
the
kubernetes
introduced
the
ingress
resource
to
solve
the
Lord
Balon
else
to
actually
support
l7
load
balancers
and
in
those
days
in
the
early
days
of
companies,
the
most
of
the
road
is
targeted,
virtual
machines
or
bare
metal,
but
not
the
containers
and
I
would
say
most
of
the
road
bands
even
today
are
proxy
are
based.
A
There
were
many
implementation
that
had
shown
up,
and
it's
pretty
much
in
the
Covenant
is
there
it's
just
an
API
and
implementations
are
different
for
each
proxies
in
Google's,
kubernetes
engine
be
back
hit
by
GCL
B,
which
is
cloud
load
balancer,
and
the
way
it
works
is
that
conceptually
the
traffic
and
hates
the
ingress
layer
which
is
not
balancer
once
it
hits.
The
road
balancer
then,
is
unfortunately
kicks
in
after
that
services
under
defied.
That
is
the
community
service
to
send
the
traffic
which
is
type
Northport.
A
If
you
actually
tie
back
in
to
previous
discussion
and
then
the
surface
type
Northport
on
the
node
will
forward
it
to
the
right
part,
things
look
pretty
simple
and
intuitive.
However,
when
you
start
looking
at
the
data
path,
underneath
it's
pretty
complex,
the
most
of
the
implementation
on
the
node
for
for
all
this
reports
and
all
are,
is
the
IP
table
based
or
they
PVS?
We
have
it
based
on
IP
tables.
We
had
it
based
on
IP
tables
and
we
still
have
it
for
this
mode.
A
In
this
case,
you
can
see
that
the
traffic,
which
is
a
blue
dot,
it
goes
from
decline
to
load
balancer
from
load
balancer
to
the
one
of
the
nodes
and
antinodes
the
IP
table
based
random
lookup
happens.
One
of
the
pod
gets
picked
up,
which
could
be
any
of
the
node
in
the
cluster,
and
then
traffic
hits
support.
So
in
order
to
make
this
happen,
the
wait
wait
it
is
done
is
gke
when
it
creates
the
when
ingress
resources
created.
A
It
actually
creates
an
instance
of
GCL
B,
which
is
Google
cloud
load
balancer
and
takes
all
the
instances
all
the
nodes
in
the
cluster
puts
them
into
the
instance
group
behind
that
load
balancer.
It
is
how
conceptually
it
gets
done,
as
you
can
see
here
that
you
could
argue
the
balances
balancing
is
good,
because
what
if
the
board
distribution
was
asymmetrical,
some
nodes
have
more
pod
support
some
nodes,
a
less
so
you
kind
of
Chiva,
better
or
balancing
by
having
this
IP
tables
spraying
the
traffic.
But
then
what
happened?
It
came
at
a
cost.
A
The
cost
is
performance,
as
you
can
see
here,
there's
a
double
hop.
You
can
assume
latency,
you
can
expect
latency
to
go
up.
There
is
a
network
hop
that
comes
in
between
and
it's
again
going
back
to
the
theme,
easy
to
debug
and
easy
to
understand,
eventually
for
the
customers
and
then
the
last,
but
not
the
least.
Some
of
the
elbe
features
are
just
incompatible
with
this
mode
double-click
on
one
of
this
feature,
which
is
not
compatible
example,
cookie
base,
definitely
load.
Balancer
GCL
me
supports
cookie
based
or
cookie
based
affinity.
A
That's
a
very
powerful
feature
for
you
developers
and
users
who
would
like
to
maximize
the
caching
in
the
application,
or
they
want
to
make
sure
that
for
the
stateful
application,
client
connection
end
up.
On
the
same
back
end
to
make
that
happen,
let's
see
how
it
how
the
packet
flow
goes
through.
The
client
sends
a
request
to
lb
lb,
does
his
load
balancing
sensor
to
a
node
and
then
iptables
on
the
node
wooden
dummy
pick.
One
part
you
standard
response
back
lb
is
going
to
insert
the
cookie
in
the
response
which
is
going
to
mention.
A
We
just
gonna
specify
the
node
next
connection
when
a
client
which
is
back
to
the
lb
that
cookie
is
there
lb,
looks
up
the
cookie
and
says:
oh,
this
is
for
node
a
and
the
traffic.
Then
the
request
is
forwarded
to
note
a
the
IP
tables
or
node
a
is
not
cookie
aware,
so
it
does
a
random
look
up
again
and
in
the
random
look
up.
What
it
comes
is
a
pod
or
node
a
so
that
feature
is
practically
broken
at
this
point
right.
A
A
Next
were
introduced
exactly
for
this.
Next
is
network
and
one
groups.
Basically
it's
a
grouping
of
the
parts
that
can
be
then
inserted
into
different
products,
for
example,
in
this
case
it's
or
balancer.
Actually,
this
the
illustration
is
so
boring.
They
don't
even
talk
about
it.
They,
the
Lord
balancer
Center
traffic,
and
it
goes
directly
to
the
pods.
If
I
have
to
make
it
interesting.
I
can
say
now
see.
There
is
no
that
there
is
no
confusion
there.
A
The
load
very
well
because
we're
sending
to
the
nodes
and
the
nodes
would
do
something
else
under
on
the
second
half,
so
it
was
not
very
accurate
in
load
balancing
now
the
road
balancing
is
also
accurate,
so
traffic
shipping
is
right
and
all
the
features
work
see
in
the
summary
in
what
we
are
seeing
from
the
early
kubernetes
to
now,
TCP
networking
evolved.
It's
a
it's
support
to
make
containers
a
first-class
endpoint
on
the
Sdn
and
also
on
many
other
products
like
telemetry
and
load
balancers.
B
So
that
sounds
all
amazing
right.
This
is
this
is
a
wonderful
improvement
and
for
all
of
us
sort
of
infrastructure.
Nerds.
It's
a
wonderful
thing,
friend
users.
It's
a
wonderful
thing,
because
now
your
features
just
work,
but
of
course
it
doesn't
actually
just
work.
It
turns
out
that
programming
external
load
balancers
is
a
little
bit
a
lot
slower
than
programming
iptables.
What
I
can
do
in
sub
one
second
and
IP
tables
might
take
ten
or
fifteen
seconds
to
program
an
external
load
balancer.
B
We
didn't
really
think
of
that
when
we
were
pushing
pods
and
doing
things
in
this
super
fast
in
cluster
mode.
We
need
some
way
for
the
load
balancer
to
check
in
with
kubernetes.
So
we're
talking
about
the
coevolution
here
right.
That
means
both
of
us
are
going
to
evolve.
So
what
do
we
need
to
do
to
make
kubernetes
aware
of
these
load
balancers?
So
let's
look
at
a
real
quick
example
of
just
sort
of
what
could
go
wrong.
B
Imagine
now
I've
got
my
my
read
deployment
up
and
I've
got
all
my
load
balancer
point
into
my
read:
deployment
pods
and
everything's,
going
great
and
I
decide.
I'm
gonna
do
a
rolling
update
of
my
deployment
and
I
bring
up
a
new
pod
right,
so
we've
got
probes
on
our
pods
as
I'm
sure
all
of
you
well
behaved.
Kubernetes
users
are
using,
the
pod
is
alive
and
the
pod
is
in
fact
ready.
It's
ready
to
receive
traffic
right.
B
What
do
we
know
about
the
infrastructure
at
this
point?
What
do
we
know
about
the
load
balancer?
We
know
nothing.
So
if
I'm,
a
poorly
behaved
application
controller,
what
I'm
gonna
do
is
delete
the
old
pod
and
I'm
gonna
start
the
process
again
and
I'm
gonna
do
the
next
pod
and
then
I'm
gonna
do
the
next
pod.
All
of
those
read
load
balancer
links
are
gonna,
go
down
before
any
of
the
green
ones
are
going
to
come
up.
You've
just
experienced
the
total
outage
of
your
application
by
pushing
a
rolling
update.
B
This
is
exactly
what
kubernetes
is
supposed
to
not
do
so.
We
have
now
a
feature
in
kubernetes
that
we
call
readiness
gates
or
pod
ready,
plus
plus.
It
allows
arbitrary
infrastructure
to
extend
the
definition
of
readiness
so
that
you
can
feed
back
into
these
application
controllers.
The
workload
controllers
like
deployment
and
tell
them
hey
hold
on
a
second.
Let
me
wait
for
the
load
balancer
to
check
in
and
I'm
not
gonna
get
into
the
details
of
the
API
right
now.
B
But
if
anybody
wants
to
talk
about
it,
we'd
be
happy
to
talk
about
the
details
and
the
whys
and
wherefores.
But
now
I
can
actually
say
hey.
This
infrastructure
is
not
ready,
wait
and
now
all
of
my
work
load
controllers,
without
actually
being
aware
of
the
fact
that
there's
a
load
balancer
now
right,
I've
totally
decoupled
this
they
are
not--.
They
know
that
they
have
to
wait
until
these
load.
Balancers
are
up
to
date,
and
it's
funny.
We
put
this
in
front
of
our
network
sig
and
we
said:
we've
got
this
problem.
B
A
Right
so
we
went
or
now
have
TCP
networking
evolved
to
meet
some
of
many
of
the
abstractions
and
then
have
kubernetes
abstraction
has
evolved
to
meet
the
cloud
providers.
Let's
talk
about
one
of
the
very
key
problem:
how
do
we
extend
kubernetes
resources
to
express
something
that
is
very
cloud
provider
centric,
so
most
of
the
load
balancers?
A
They
have
features
right
now,
which
are
above
and
beyond
what
kubernetes
synchronous
can
specify
and,
as
you
go
back
to
principle
of
portability,
the
main
thing
is:
if
most
implementation
cannot
support,
the
or
most
implementation
should
be
able
to
support
the
most
features.
So
with
that
we
were
struggling
in
the
beginning
on
how
do
we
really
specify
the
features
of
GCL
B,
which
is
I
the
identity
of
a
proxy
CDN
for
caching
security,
and
earlier
we
began
with
the
notations?
It
very
soon
became
bulky
and
hard
to
understand.
A
Going
back
to
the
theme,
how
do
we
make
it
easy
to
understand
and
easy
to
manage?
Cid
came
to
the
rescue
CR
ds1
introduced
in
1.7,
and
that
was
a
great
way
to
extend
an
API.
The
users
can
now
add
a
custom
object
and
make
it
behave
like
a
native
resource
and
that's
exactly
what
we
did.
We
introduced
back
in
conflict
as
a
CR
D.
The
services
were
now
point
to
back-end
conflict
in
the
backend
config.
A
Then
you
use
the
customers
can
go
ahead
and
expand,
extend
and
Express
the
features
of
GCL
B,
which
were
poor
services,
and
now
they
can
be
referenced
by
egress.
We
have
a
load
balancer
features
now
expressed
in
in
the
CR
D,
without
affecting
the
portability
of
the
ingress.
So
that
is
one
of
the
key
theme.
How
do
we
really
extend
without
affecting
the
portability?
How
can
customer
take
the
same
yeah
more
poor,
to
put
it
somewhere
else,
and
things
should
just
work
so
we
achieved
it.
Society
has
been
great
in
that
phase
all
right.
A
So
now
we
went
over
abstractions
on
both
sides,
but
there
were
few
things
that
we
really
found
it
difficult
to
overcome
or
maybe
fail.
There
may
be
something
done
better
here.
So
let's
go
over
some
of
the
abstractions
where
we
found
either
they
were
too
flexible
or
not
flexible
enough
in
terms
of
dope
flexible.
A
The
services
has
extremely
powerful
features
of
poetry,
mapping
you
can
have
target
port
and
the
name
port
and
then
the
name
port
would
then
allow
the
ports
to
have
their
own
ports,
which
would
now
give
you
enough
lot
of
power
to
change
the
ports
and
stuff
with
each
Rev.
But
then
that
really
puts
it
odd.
With
some
of
the
network
fabrics
see
many
of
the
load
balancers,
they
would
use
TSR,
which
is
direct
server
server
return
mcpadden.
This
pattern
is
specifically
made
so
that
the
high
performance
application
sticks.
They
go
through
lease
latency.
A
The
response
goes
direct
to
the
server.
Well,
the
request
is
going
through
the
proxy,
so
there
is
no
space
for
port
remapping
in
that
with
port
remapping
and
the
dsr
there
is
incompatibility
vile.
This
port
remapping
wood
is
a
powerful
feature
inspired
from
docker
in
the
early
days.
If
we
go
back
and
apply
the
filter
off
of
our
portability
principle,
that
most
implementations
should
be
able
to
support
the
features,
then
this
probably
would
not
cut
it
in
the
main
construct.
A
Now,
what
are
the
things?
They
were
not
flexible
enough.
We
have
seen
use
cases
where
customers
wanted
to
want
to
forward
traffic
directly
to
the
IP
addresses,
or
maybe
they
have
use
cases
where
the
rangers
are
wide
and
they
want
to
express
the
range
in
their
services
or
the
range
is
a
dynamic
web
applications
are
great
here
as
an
example
and
service.
Abstraction
today
doesn't
have
the
flexibility
to
describe
this
and
then
service
abstraction
is
very
intuitive.
When
you
are
working
with
cluster
IP
and
load
balancer
and
moment,
you
start
going
into
okay.
A
How
do
I
do
the
headless?
Well,
you
have
cluster
IPA
type
and
then
you
may
cluster
IP.
None
things
start
becoming
interesting
when
the
same
type
starts
behaving
differently
depending
upon
how
different
things
are,
there's
a
room
for
a
refactoring,
and
then
we
should
have
done
the
elbe
check
in
a
bit
back.
There'll
be
primitives
are
not
that
defined
right
now
and
the
quadratic
8
pod
readiness
who
have
been
done
long
back.
B
Looking
looking
at
what
we've
talked
about
here,
you
can
see
sort
of
how
kubernetes
has
changed
over
time,
how
the
design
of
kubernetes
was
influenced
by
what
the
cloud
providers
could
provide,
but
but
maybe
not
so
limited
by
that
and
how
we
at
Google
cloud
have
tried
to
adapt
to
the
things
that
kubernetes
offers
to
people
to
give
better
experiences
with
kubernetes
I
hope.
This
gives
you
a
little
bit
of
insight
into
sort
of
where
we
were
and
where
we
are.
B
B
We'll
hang
out
here
and
do
some
questions.
If
people
want
to
do
questions
you
can
we
have
do
we
have
time
to
do
questions
on
mics,
let's
do
a
few
and
then
we'll
take
it
out
in
the
hallway
afterwards.
Let
me
just
pick
up
a
couple
from
the
front
and
then
we'll
move
back.
So,
let's
start
with
you,
what
about
ipv6
come
to
the
deep
dive
tomorrow?
B
Tls
ot
LSE's
is
a
feature
of
the
ingress
resource,
so
you
can
ask
ingress
whatever
load-balanced
are
using
for
ingress
to
terminate
TLS.
One
of
the
we
talked
about
here
is
backside
TLS
or
backside
protocol.
We
actually
have
a
some
annotation
right
now
to
do
backside.
Protocol,
that's
the
sort
of
thing
that
should
be
moving
into
back-end
config
and
do
a
you
know,
CRD.
If
it's
cloud
specific
that
one
in
particular
we're
looking
at,
maybe
making
a
first-class
feature
of
the
API.
B
I'm,
sorry
yeah,
so
there's
a
unfortunate
reality
that
the
networking
infrastructure
from
old
gke
clusters
and
the
VPC
native
networking
infrastructure
are
very
difficult
to
automatically
migrate
between
and
we've
looked
at,
how
to
do
the
automatic
migration
it's
incredibly
thorny
and
dangerous,
without
understanding
what's
going
on
in
your
cluster.
So
so
far,
the
best
recommendation
we
have
for
you
is
learn
how
to
bring
up
a
new
cluster
and
move
your
workloads
between
them.
It's
actually
a
good
skill,
and
you
know
if
you
get.
B
If
you
get
too
crystallized
into
one
cluster,
it
becomes
very
difficult
to
recover
from
a
disaster.
So
it's
probably
actually
a
really
good
thing
for
every
organization
to
understand
what
happens
if
my
cluster
goes
away,
how
do
I
bring
up
a
new
one
right,
so
I
don't
have
a
better
answer
than
that.
Also.
B
So
service
mash
is
coo.
Brandis
gonna
deal
with
it.
I
do
not
want
kubernetes
to
become
a
service
mesh
I
want
to
understand
what
service
meshes
need
so
that
we
can.
They
can
use
kubernetes
better.
So,
like
I
talked
with
the
sto
team
fairly
frequently
and
understanding
what
primitives
they're
building
and
why
it
turns
out
they're
working
around
things
in
kubernetes,
so
we're
gonna
see
if
we
can
find
some
colors
and
four
primitives
between
sto
and
linker
D
and
all
the
other
options
and
service
mesh,
so
that
we
can
make
those
things
easier
to
build.
B
B
So
is
there
planning
to
release
this
stuff,
so
if
using
kubernetes
on
GCP
but
not
gke
right,
this
already
works,
you
can.
You
can
use
VPC
native
already,
if
you,
if
you,
when
you
set
up
your
BBC
subnet,
you
can
turn
on.
You
can
use
IP
aliases.
You
can
create
your
instance
group
with
the
alias
configuration
it'll
automatically.
There's
nothing
we're
doing
here.
That's
private
ap
is
or
secret.
This
is
all
building
on
public
GC.
Pap
is
that's
a
it's
a
core
principle
of
the
way
we
build
kubernetes
at
GC
p.
B
B
Are
there
any
CNI
plug-ins
for
vanilla,
kubernetes
doing
the
same
I'm,
not
sure
that
C&I
has
the
scope
to
handle
this?
You
need
your
load
balancers
to
be
aware
of
endpoints
directly,
which
not
all
load
balancer
products
are
happy
about.
For
example,
we
have
to
program
the
network
end
point
group
like
during
a
rolling
update
and
for
every
new
pod
we
go
out
and
we
change
the
network
endpoint
group
and
we
resynchronize
some
load.
Balancers
are
fine.
With
this
and
I
know.