►
Description
Want to view more sessions and keep the conversations going? Join us for KubeCon + CloudNativeCon North America in Seattle, December 11 - 13, 2018 (http://bit.ly/KCCNCNA18) or in Shanghai, November 14-15 (http://bit.ly/kccncchina18).
Global Container Networks on Kubernetes at DigitalOcean - Andrew Sy Kim, DigitalOcean (Intermediate Skill Level)
Building a container network that is reliable, fast and easy to operate has become increasingly important in DigitalOcean’s distributed systems running on Kubernetes. Today’s container networking technologies can be restrictive as Pod and Service IPs are not reachable externally which forces cluster administrators to operate load balancers. The addition of load balancers introduces new points of failure in a cluster and hinders observability since source IPs are either NAT’d or masqueraded. This talk will be a deep dive of how DigitalOcean uses BGP, Anycast and a variety of open source technologies (kube-router, CNI, etc) to achieve a fast and reliable container network where Pod and Service IPs are reachable from anywhere on DigitalOcean’s global network. Design considerations for scalability, lessons learned in production and advanced use cases will also be discussed.
About Andrew
Andrew is a Software Engineer at DigitalOcean and an active member of the Kubernetes community. At DigitalOcean, his team provides a robust and comprehensive set of tools for delivering services to production.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
All
right,
hi,
so
I'm,
Andrew,
Kim
I'm,
a
software
engineer
based
in
Toronto
Canada,
where
I
work
remotely
forwarded
ocean.
So
my
talk
is
going
to
be
about
our
globally
distributed
container
networks
and
how
we
integrated
kubernetes
directly
into
our
data
center
networking
to
offer
what
we
think
is
a
very
effective
networking
model
for
our
engineers.
A
So
my
talk
is
going
to
be
a
technical,
deep
dive,
and
my
hope
is
that
this
talk
gives
you
enough
context
that
you
can
actually
build
a
similar
container
networking
for
your
own
clusters
and
if,
if
I
didn't
give
enough
context,
I'd
love
to
chat
with
you
about
how
to
do
that.
For
you
for
your
clusters.
So
before
going
there,
though
I
do
want
to
provide
some
background
on
kubernetes
at
digitalocean,
so
our
first
production
deployment
of
kubernetes
was
way
back
when
kubernetes
version
1.1
was
released.
A
So
that
was
roughly
two
years
ago,
early
2016,
and
at
that
time
we
were
experimenting
with
kubernetes,
because
we
wanted
to
reduce
the
operational
overhead
that
a
lot
of
engineers
were
struggling
with.
So
we
want
to
create
a
consistent
platform
that
made
it
really
easy
for
developers
to
deploy
applications
into
production,
which
I'm
sure
everyone
here
kind
of
had
that
similar
story.
A
So
as
we're
looking
into
adopting
kubernetes,
we
faced
a
few
challenges,
one
of
them
being
that
kubernetes
wasn't
moving
target
and
it
still
is
today
so
best.
Practices
and
features
are
constantly
changing
or
sometimes
just
being
deprecated,
and
we
also
found
that,
because
kubernetes
offered
so
many
features
and
knobs
that
you
can
tune
a
lot
of
develop
a
lot
of
developers
just
didn't
know
where
to
start.
A
So
on
top
of
that,
the
general
impression
of
the
time
was
that
kubernetes
had
too
much
ml
and
and
no-one
really
wanted
to
learn
what
the
metadata,
the
api
groups
and
all
that
stuff
was
so
out
of
that
we
built
in
an
internal
platform
called
GOCC
which
stands
for
digital
ocean
control
center.
So
d
LCC
is
a
multi
tenant
internal
path
built
on
top
of
kubernetes.
It's
very
simple,
API
layer
that
translates
a
specification
that
we
maintain
into
kubernetes
resources.
A
What's
really
powerful
about
this
is
that
it's
built
with
simplicity
in
mind,
while
also
promoting
and
sometimes
even
enforcing,
some
of
the
best
practices
that
that
we
think
that
you
should
follow
when
you
deploy
applications,
containers
and
with
kubernetes.
So
some
examples
are,
if
a
engineer
deploy
something
with
the
latest
tab.
They
just
drop,
will
just
block
it
and
be
like
no,
no
latest
tack
and
if
you
deploy
an
application
and
into
production
and
does
have
an
in
metric
standpoint,
we'll
just
drop
that
thun.
A
It
also
meant
that
there
were
no
huge
files
of
yeah
more
for
developers
to
onboard
them
on
our
platform.
Today,
it's
a
little
bit
different,
where
we
have
some
engineers
who
really
want
that
cubes
ETL
access
and
and
is
okay
with
managing
ya
know,
and
there
are
some
engineers
that
don't,
but
that's
kind
of
a
different
story
for
another
talk
and
as
much
as
I
do
want
to
talk
about
the
internals
of
DCC,
more
and
kind
of
the
motivation
behind.
A
Why
we
did
this
I
don't
have
time,
but
we
did
have
previous
talks
that
previous
cube
cons,
that
you
can
look
at
so
two
years
later
and
we're
managing
DL
CC
at
a
pretty
large
scale
and
do
CC
manages
a
lot
of
applications
across
all
our
data
centers
and
across
all
our
environments.
What
we
found
was
really
good
at
this
large
scale
was
that
kubernetes
provided
an
excellent
multi-tenant
environment
where
resources
were
utilized
more
efficiently,
where
we
didn't
have
like
a
single
machine
running.
A
You
know
this
really
small
micro
service
and
it
did
an
excellent
job
of
making
sure
containers
were
running
where
they
should
be,
and
we
find
that
kubernetes
provided
all
the
great
tools
that
may
building
a
platform
really
easy.
So
it
was
great
platform
to
build
a
platform
on
the
flip
side
of
things
once
we're
running
at
a
larger
scale.
A
We
faced
into
a
few
challenges
and
it
was
mostly
around
our
networking,
so
we
found
that
our
initial
design
of
our
pod
networking
was
fast
enough
for
a
majority
of
applications
that
are
on
GCC,
but
it
wasn't
fast
enough
for
the
applications
that
were
kind
of
latency
sensitive
and
had
really
high
throughput.
We
also
found
that
our
container
network
wasn't
intuitive,
so
we
had
source
and
destination.
A
Adding
and
developers
has
never
really
knew
like
which
IPS
were
private
and
which
IPS
were
like
the
external
ingress
facing
or
public
facing
IPs,
and
so
we
have
to
explain
to
developers
a
lot
what
those
different
components
were
so
I
do
want
to
dig
a
little
deeper
into
why
this
was
the
case.
So
since
we
were
pretty
early
adopters
of
kubernetes,
when
we
built
our
first
production
cluster,
we
use
what
was
kind
of
well
known
at
the
time,
which
was
an
overlaid
pod
network
with
sanam.
A
So
this
diagram
kind
of
shows
the
path
of
a
packet
between
two
pods
using
this
overlay
and
I'm
not
going
to
go
into
too
much
detail
into
how
this
works.
But
the
idea
was
that
the
packets
went
through
a
lot
of
different
abstractions
and
and
different
layers
on
the
kernel
for
that
pod
pod
communication
to
work
so
some
challenges
with
this
top
this
pod
network
topology,
was
that
the
packet
overhead
became
more
and
more
impactful
to
performance
with
higher
throughput.
A
We
also
didn't
like
how
every
node
in
the
cluster
had
to
have
a
route
for
every
other
node
in
order
for
the
cluster
networking
to
work,
because
that
made
operating
the
individual
nodes
extremely
difficult.
We
also
didn't
like
how
I
FEMA's
grading
was
essentially
mandatory
for
any
cluster
network
to
work.
So
on
top
of
this,
because
we
had
this
overlay
and
is
confined
to
the
cluster
only
if
we
wanted
to
ingress
traffic
into
the
cluster,
we
have
to
set
up
as
I'm
sure
many
of
you
know,
ingress
controllers
and
if
you've
ever
administered
ingress
controllers.
A
You
know
the
network
ends
up.
Looking
something
like
this,
where
the
ingress
controller
pods
are
configured
usually
with
a
host
port
binding
or
there
they
run
on
host
network
so
that
they
can
actually
receive
external
traffic
for
the
cluster.
So
I
just
want
to
a
quick
poll,
raise
your
hand
if
you've
used
ingress
controllers
in
production.
A
Okay,
a
lot
of
you:
okay,
raise
your
hand
if
you
have
a
love/hate
relationship
with
ingress
controllers.
Okay,
it's
like
half
of
you
and
I'll.
Tell
you
why?
Okay
well
I
I,
definitely
have
a
love-hate
relationship
and
I'll.
Tell
you
why.
So
what
we
really
liked
about
ingress
controllers
was
that
we
were
able
to
standardize
how
traffic
came
into
the
cluster,
so
we
can
set
connection
timeouts
that
are
global
to
the
cluster
and
all
those
things,
and
we
were
able
to
use
a
lot
of
the
kubernetes
primitives
to
build
mechanisms
to
kind
of
automatically
failover.
A
So,
as
you
can
tell
by
now,
this
was
really
complicated
and
difficult
to
operate.
It
worked
but
nonetheless
very
complicated
and
no
one
really
wanted
to
kind
of
explain
how
this
works.
Anyone.
So
on
top
of
that,
we
had
IPS
being
added
both
for
incoming
and
outgoing
traffic.
So
when
an
outside
client
needs
to
request
and
the
ingress
controller
proxies
the
request
to
a
pod,
the
pod
will
see
the
source
IP
as
the
ingress
controller
instead
of
the
actual
client
IP.
A
Likewise,
if
pods
talk
to
services
that
are
outside
of
a
cluster,
the
source
IP
would
be
n--
added
to
whatever
the
node
IP
was,
which
is
kind
of
a
requirement,
because
if
the
source
IP
was
the
pot
IP
and
the
outside
application
doesn't
know
how
to
route
to
that
pod,
the
networking
would
work
so
I
had
to
be
the
node
IP
for
this
set
up
to
work.
So
with
all
this
in
mind,
what
was
lacking
was
that
there
was
too
much
overhead
from
packet
encapsulation.
It
was
extremely
operationally
complex
to
manage
this.
A
A
So
what
we
wanted
in
this
new
network
was
something
that
was
simple
to
operate:
had
better
visibility
for
source
and
destination,
IP
s,
something
that
skills
with
the
cluster.
So,
as
you
add
capacity
into
the
cluster,
you're
automatically
scaling
that
ingress
capability
and
we
wanted
something
that
was
just
generally
faster
and
then
it
had
so
much
overhead
on
the
actual
node.
So
we
started
doing
some
discovery
around
what
this,
what
this
networking
might
look
like,
and
it
was
important
that
what
we
designed
initially
and
put
into
production
was
done
right.
A
The
first
time,
because
we
know
from
experience
that
is
extremely
difficult
to
actually
change
the
container
networking
on
a
cluster
that
is
actually
already
serving
production
traffic
and
if
you
did
build
it
incorrectly,
you
always
had
to
build
a
new
cluster
and
then
set
the
networking
crack
there.
And
then
you
have
to
move
all
your
pods
over
and
then
you
have
to
deprecate
that
cluster
and
I've
done
it
once
and
it
just
it's
a
big
hassle.
No
one
wants
to
so.
A
This
complex
overlay,
an
abstraction
that
developers
had
a
hard
time
understanding,
and
it
was
hard
to
figure
out
what
this
might
look
like
without
grasping
a
better
understanding
of
what
our
data
center
networking
looked
like
and
understanding.
The
internal
internals
of
our
data
center
network
is
also
important
to
understand
how
we
efficiently
built
this
globally
distributed
network.
A
So
I
want
to
be
starting
out
with
what
we
call
a
pod
and
it's
not
a
kubernetes
pod,
but
it's
a
pod
kind
of
in
the
networking
world
which,
in
this
context,
represents
a
single
component
of
compute
and
network
in
our
data
centers,
you
can
think
of
a
pod
as
a
building
block
for
our
data
centers,
where
we
host
physical
machines
on
racks
that
are
connected
over
an
aggregation
of
switches.
So
you
can
see
in
this
diagram
at
the
top
of
each
rack.
A
But
even
then
you
may
want
to
scale
out
even
further,
but
you
can
do
by
adding
new
pods
connected
to
this
new
layer.
Of
course,
witches
which
we
call
zones
and
then
the
core
switches
of
each
zone
would
be
connected,
so
we
and
still
have
networking
across
all
the
different
pods
and
zones,
and
so
I'm
sure
some
of
you
are
asking.
Why
do
we
need
this?
Because
it
seems
really
complicated
and
the
reason
why
so
many
data
centers
are
designed?
A
This
way
is
because
there
are
limitations
with
how
many
physical
connections
you
could
have
for
per
switch,
and
even
in
theory,
if
you
can
create
a
magical
switch
that
had
thousands
of
physical
ports
that
you
can
actually
connect.
Every
single
machine
in
the
data
center
to
those
switches
would
actually
just
end
up
being
single
points
of
failure
for
your
data
center
network.
So
with
data
center
is
hosting
thousands
of
machines.
A
So
how
do
we
do
this?
How
do
we
integrate
our
systems
with
this
data
center
network?
So
it
turns
out
there's
actually
protocol,
that's
older
than
probably
most
of
us
in
this
room
called
BGP
and
it's
a
protocol
that
lets
us
actually
communicate
with
with
the
switches
to
actually
configure
routes
in
our
data
center.
A
So
in
short,
BGP
stands
for
the
border
border
gateway
protocol.
It's
in
a
inter
autonomous
system,
routing
protocol
so
and
really
BGP
is
kind
of
the
the
protocol.
That's
that's
kind
of
keeping
the
public
Internet
together,
and
the
way
it
works
is
that
the
internet
really
is
this
interconnection
of
autonomous
systems
that
are
built
upon
a
system
of
trust.
A
So
these
autonomous
systems
can
be
digital,
O'shane,
Google
cloud
or
you're,
probably
at
your
internet
service
provider,
and
these
autonomous
systems
are
uniquely
identified
using
autonomous
system
number
or
an
ASN,
and
they
appear
amongst
each
other
to
kind
of
share
routing
information,
either
about
itself
or
about
another
system
that
it
peers
with.
So
I
am
skipping
a
lot
of
important
details
on
how
this
really
works,
but
this
is
kind
of
the
gist
you
need
to
know
for
this
talk
and
just
like
how
there
are
public
and
private
IPs.
A
A
A
So,
with
this
in
mind,
we
start
to
think
about
how
we
can
fit
kubernetes
clusters
as
these
bgp
autonomous
systems.
If
we
can
do
that,
it
meant
that
we
can
use
BGP
to
dynamically
advertise
routes
in
our
data
centers
and
propagate
IPs
and
our
clusters
into
our
entire
backbone
network.
And
luckily,
due
to
the
the
nature
of
this
Kloss
topology,
we
don't
necessarily
need
to
peer
directly
to
all
the
different
tiers
in
earth.
You
know
and
different
tiers
of
switches
in
our
data
centers.
A
We
can
just
peer
with
the
the
top
of
rack
above
each
server
and
the
top
of
racks
would
then
peer
to
all
its
known
peers
and
then
so
it'd
be
it
would
peer
with
spines,
and
then
it
appeared
to
the
core
of
the
network,
so
that
meant
from
from
a
node
per
node
configuration
standpoint
and
kind
of
all
we
have
to
do
from.
There
was
to
know
what
is
the
internal
autonomous
system
number
for
the
cluster
and
how
do
we
actually
peer
with
this
top-of-rack
switch.
A
So
what
is
the
the
IP
of
the
switch
and
what
is
the
autonomous
system
number
that
the
switches,
the
top
of
rack
switches?
So
to
accomplish
this?
We
used
a
really
cool
open
source
project
called
cube
router,
so
it
supports
in
cluster
internal
BGP
and
external
BGP
peirong,
which
is
what
we
use
to
peer
with
our
top
of
rack
switches,
and
you
can
also
configure
it
to
advertise
the
pod
and
the
service
IPS
to
all
of
its
beech
Pearse.
A
And
lastly,
you
came
with
a
lot
of
other
features
that
we
didn't
anticipate
of
meeting,
but
it
ended
up
being
pretty
helpful
for
some
of
our
current
and
future
use
cases
which
I'm
going
to
talk
about
more
year
at
the
end
of
the
slide
at
the
top.
So
to
go
back
to
the
top
of
our
configuration
really
all
we
had
to
do
on
cube
router.
Let's
tell
it.
What
is
the
cluster
autonomous
system
number,
which
again,
you
can
think
of
as
this
unique
identifier
for
a
clusters,
but
then
this
this
larger
bottom
system?
A
That
is
our
data
center
and
then,
how?
Do
you
peer
with
this
top-of-rack
switch,
which
I
mentioned
earlier,
which
is
how
what
is
the
top
of
racks
IP
and
what
is
the
autonomous
system
that
it's
part
of
and
from
there
we
can
fully
rely
on
cube
router
to
advertise
the
IPS.
We
need
to
accomplish
what
we
wanted,
which
was
to
advertise
the
pod
and
the
service
IPS
in
all
our
clusters.
A
So,
by
doing
this,
what
we
actually
accomplished
was
that
our
service
IPS
were
not
only
these
virtual
IPS
that
are
only
routable
to
applications
inside
the
same
cluster,
but
they're.
Now
these
anycast
IPS
that
are
available
in
on
our
entire
backbone
network
and
I,
say
any
cast
IPS
because
on
the
balance
system,
we
now
have
this
one
too
many
Association,
where
we
route
packets
to
one
receiver
out
of
many
where
each
kubernetes
node
that's
running.
A
So
this
made
a
huge
difference
when
it
came
to
monitoring
our
district,
our
distributed
systems,
because
we
can
correlate
force
IPS
directly
back
to
services,
whereas
before
we
have
to
correlate
source
IPS
back
to
note
IPs
and
then
we'd
have
to
check
all
the
pods
running
on
that
know
what
IP
to
figure
out
what
service
could
possibly
be
talking
to
another
service.
So
this
was
really
helpful
when
it
came
to
kind
of
monitoring
things
that
pods
talk
to
outside
the
cluster.
A
This
model
also
made
configuring
and
operating
the
nodes
in
the
cluster
significantly
easier.
So
we
no
longer
had
issue
in
production,
because
this
one
node,
for
whatever
reason,
couldn't
converge
a
route
to
another
node,
and
these
are
the
kind
of
the
worst
problems
to
have
because,
like
you'd,
have
connectivity
from
one
pod
to
another,
but
not
other
pods.
A
But
here
we
can
kind
of
just
rely
on
the
top
of
rack
switch
to
route
traffic
for
the
pod
Network,
which
meant
that,
on
up
on
the
nodes,
we
just
needed
the
route
that
goes
to
the
containers,
local
to
the
node.
And
then
we
just
need
everything
else.
We
can
just
you
know,
send
it
to
the
top
of
rack,
and
we
can
rely
on
the
top
of
rack
to
manage
that
pond
networking
for
us.
A
So
this
was
this
is
an
interesting
one.
So
DNS
was
something
that
also
kind
of
improved
a
lot
with
this
topology.
So
one
day
we
remembered
that
cube.
Dns
has
a
service
IP,
which
is
the
service
IP,
that's
injected
into
the
resolver
file
for
pods,
which
is
kind
of
how
the
in
cluster
DNS
resolution
works,
which
meant
that
this
IP
that
we
use
is
also
routable
in
our
data
centers,
which
meant
that
developers
could
easily
use
cube
DNS
as
a
utility
for
applications
that
are
not
even
in
this
in
the
kubernetes
cluster.
A
But
instead
what
we
did
was
we
actually
set
an
NS
record
in
all
our
regional
DNS
servers
and
we
would
configure
them
so
that,
if
a
query
matches
a
cluster
domain
for
any
of
our
clusters,
we
could
then
have
the
the
regional
DNS
server
query.
The
regional
cube,
DNS
service
and
then
cube
dns
would
return
either
a
service,
IP
or
party.
If
you're,
using
a
hello
service
back
to
the
regional
DNS
server,
and
then
the
regional
DNS
server
was
would
kind
of
give
back
to
whatever
queried
it.
A
So
we
still
kept
them
around,
because
many
applications
depended
on
it
from
our
previous
generation
of
clusters.
But
we
were
able
to
make
significant
architectural
changes
that
made
it
so
much
easier
to
operate
ingress
controllers.
So,
instead
of
binding
pods
to
the
host
network
using
daemon
sets,
we
can
just
create
a
service
for
the
English
controller
and
then
use
the
corresponding
anycast
IP
for
the
DNS,
and
this
meant
that
we
don't
have
to
do
this
magic
with
daemon
sets
and
node
labels.
A
So,
like
any
big
change
in
our
networking
or
kind
of
any
architectural
change,
we
definitely
faced
some
challenges
and
we
there
was
definitely
some
trade-offs
along
the
way.
So
for
one
we
were
seeing
this.
This
is
what
I
call
the
proxy
mesh.
It's,
where
you
route
traffic,
for
a
service
to
node
that
isn't
actually
running
any
of
the
pods
for
the
service.
So
then
it
reroutes
traffic
back
out
to
another
node
which
causes
like
which
causes
unnecessary
bandwidth
between
your
cluster.
A
So
luckily
cube
router
does
offer
a
solution
to
this
problem,
which
I'm
going
to
talk
more
about
shortly.
Next
thing
we
noticed
was
that
we
now
had
this
uniqueness
requirement
for
the
pod
and
service
IPS.
We
were
using
across
our
clusters,
so
in
our
previous
overlay
networking
this
was
never
a
problem
because
we
can
easily
just
use
the
same
subnet
for
all
our
clusters
and
since
those
IPS
were
contained
in
the
same
pod
network,
we
didn't
care
if
they
overlapped.
A
So
when
creating
clusters,
we
now
had
to
be
more
resourceful
with
how
large
the
IP
prefixes
were,
and
this
became
very
important
in
capacity
planning,
because
there's
only
so
many
private
ipv4
IPS
that
you
can
actually
use
in
your
in
your
entire
network
globally
and
lastly,
BGP
route,
conversions
and
rebalancing
is
was
difficult
to
monitor
it's.
It
still
is
difficult
to
Margaret.
So
as
routes
are
added
and
removed
Inter
clusters,
it
was
difficult
for
us
to
gain
visibility
into
how
quickly
those
routes
were
converging
over
our
backbone
Network.
A
So
in
general
we
found
it
fast
enough
to
handle
pretty
much
most
of
our
failure
scenarios,
but
having
more
visibility
to
the
path
of
a
packet
outside
of
our
clusters
is
something
we
definitely
want
to
work
on.
So
luckily,
we
do
have
very
experienced
network
engineers
that
we
can
page
during
incidents
who
kind
of
do
have
better
visibility
there.
A
So,
with
these
challenges
in
mind,
there
are
definitely
future
improvements
that
that
we
want
to
make
as
well
so
going
back
to
the
proxy
mesh.
The
reason
we
have
this
proxy
mesh
is
because
all
the
nodes
in
your
clusters
are
advertising.
All
the
service
IDs-
and
this
is
okay,
since
we
know
that
any
node
can
receive
traffic
for
a
service
IP,
since
its
proxy
can
route
it
to
the
right
node
eventually.
A
But
there
are
cases
where
you
really
want
to
reduce
that
proxy
mesh,
especially
for
applications
that
ingress
a
lot
of
bandwidth
and
by
following
this
model,
you're,
actually
doubling
the
amount
of
bandwidth
you
actually
need.
So
a
cube
router
does
provide
a
simple
solution
to
this
problem
by
specifying
the
services
external
external
traffic
policy
to
local.
When
you
do
this,
you
configure
the
proxy
to
only
route
to
pods
local
to
the
node
and
then
cube
router
would
only
advertise
routes
if
the
pot
is
at
the
pod
for
that
service
actually
exists
on
the
node.
A
So
doing
this
would
actually
eliminate
the
proxy
meshes
since
the
top
of
racks
only
route
to
nodes
advertising
that
IP-
and
it's
worth
noting
that
by
doing
this,
we
would
also
preserve
the
source
IP
for
ingress
traffic.
Whereas
with
our
current
model,
we
still
have
matting
for
the
for
the
ingress
causing
the,
because
we
have
the
source
netting
that
happens
from
queue
proxy.
A
So
something
else
we
were
experimenting
with
to
reduce
the
effects
of
that
proxy
mesh
with
something
called
DSR
which
stands
for
direct
server
return.
So
this
is
a
feature
supported
also
supported
in
cube
router.
So,
in
short,
the
way
it
works
is
you
you
end
up
modifying
the
packets
return
address,
as
it
goes
through
the
proxy
so
that
the
receiving
pot
could
actually
so
that
the
receiving
pot
ends
up
returning
packets
directly
back
to
the
client,
instead
of
going
back
out
through
the
proxy.
A
We
also
want
to
see
what
it
looks
like
to
expose
kubernetes
services
directly
to
the
public
Internet
by
advertising,
the
public
IP
straight
from
cue
batter.
So
there
are
there
definitely
a
lot
of
considerations
here,
such
as
what
does
BGP
rep
convergence,
look
like
when
you're
appearing
with
all
these
publicly
available
autonomous
systems
and
and
of
course,
digitalocean.
A
Since
we
are
a
public
cloud,
we
already
do
that
at
a
pretty
large
scale,
but
if
we
wanted
to
know
how
we
could
actually,
but
but
we
want
to
know
how
it
could
actually
change
the
way
developers
actually
manage
their
applications
when
they
have
kind
of
that
control.
Also
for
a
kubernetes
cluster,
that's
receiving
public
traffic.
What
sort
of
pod
security
best
practices
should
be
in
place
that
so
that
this
is
actually
doable.
A
A
Okay,
so
to
recap,
we
found
that
there
are
a
lot
of
interesting
ways
to
design
networks,
especially
when
you
have
that
context
like
when
you're
in
the
context
of
a
data
center.
So
we
found
that
the
combination
of
using
protocol
like
BGP,
that
is
widely
used
in
data
centers
and
and
new
technologies
like
kubernetes,
can
be
extremely
powerful.
A
More
importantly,
though,
we
found
that
designing
a
container
network,
that
is
that
is
simple
and
intuitive
to
understand,
goes
a
really
long
way.
There
was
definitely
a
journey
for
my
team
trying
to
understand
the
data
center
networking
and
adopting
new
technologies
to
make
this
work,
but
what
we
actually
ended
up
offering
to
our
engineers-
which
was
this
extremely
simple-
to
understand-
that
working
model
made
a
big
difference
when
it
came
to
adopting
containers
and
making
it
easy
for
engineers
to
actually
build
reliable
applications.
A
A
Because,
in
my
context,
we
actually
have
access
to
the
data
center
networking
and
you
can
do
this
on
the
cloud
and
there
is
definitely
some
more
architectural
and
security
considerations
that
that
I
didn't
get
to
cover,
but
I,
don't
think
I
have
time,
but
if
any
of
these
topics
do
interest,
you
please
approach
me
I'd
love
to
chat
all
these
things
kind
of
really
trust
me.
So
if
you
have
any
opinions-
or
you
want
to
chat
more
about
this,
please
approach
me
at
any
point
during
the
conference
and
with
that.
A
A
A
Yes,
so
the
question
was
how
we
considered
using
route
reflectors,
essentially
route,
reflectors
that
you,
if
you
have
a
if
you
have
a
node
with,
if
you
have
a
cluster
with
100
nodes,
maybe
only
five
of
them
are
actually
appearing
with
the
top
of
racks
and
then
the
other.
Ninety
five
would
peer
with
the
the
nodes
that's
kind
of
what
reflectors
do
so
we
actually
did
not
have
to
use
route
reflectors
yet,
and
that's
mostly
because
kind
of
the
switches
we
use
are
actually
really
beefy
and
powerful.
A
A
A
A
Yeah,
so
we
don't
do
network
policy
enforcement's
yet,
but
cube
router
does
support
it
and
the
way
it
works.
I'm,
sorry,
I'll,
repeat
the
question.
So
how
are
we
enforcing
networking
policies
between
our
pods
and
between
our
clusters
so
cube
routers?
So
there's
an
actual
API
in
kubernetes
that
actually
has
a
spec
for
what
a
network
network
and
policy
looks
like
and
cube.
Router
is
just
a
consumer
of
that
network
policy
type
and
then
it
will
create
kind
of
IP
table
rules
on
the
actual
nodes
to
enforce
that
policy.
A
So
that
out
be
up
to
the
developers
because
of
the
developer
knows
like
what
something?
That's
the
so,
if
a
service
a
and
cluster
a
oxes
service
B
in
cluster
B,
the
developer
should
know
what
the
source
and
destination
IP
is
between
the
clusters
should
be
so
they
can
set
their
network
policies
themselves.
We
haven't
hashed
that
out,
though,
because
if
you
haven't
used
network
policies
yet
because
this
is
all
in
kind
of
our
internal
network,
so
we
don't
have
to
worry
too
much
about
that
were
policies,
but
we
definitely
want
to
dig
into
that.
A
Yes,
so
the
question
was:
do
we
plan
to
use
calicoes
BGP
feature
so
the
when
we
were
initially
kind
of
looking
at
technologies
to
use
for
this?
We
were
debating
between
calico,
key
router
or
to
just
build
our
own.
The
reason
why
we
didn't
stick
with
calcoast,
because
it
didn't
actually
support
advertising
the
service
at
Pease
and
we'd
have
to
build
that
in,
and
so
we
and
cue
powder
already
at
the
pod
and
the
service
and
external
IPS,
and
it
also
supported
Network
policy.
So
we
can't
just
run
with
that.
A
Yeah,
so
the
question
was
around
why
we
didn't
use
metal
lp's
out
yeah.
Essentially,
we
didn't
use
a
metal
lb
because
we
really
want
that.
The
pod
IPS
to
be
advertised
and
metal
LP
does
not
support
the
advertising
of
parties
and
the
reason
why
we
actually
want
to
use
parties.
Is
this
really
interesting
use
case
where
we
actually
want
to
run
from
ETS
outside
of
the
actual
cluster?
A
Because
we
don't
want
this
dependency
loop
between
your
clusters
and
Prometheus
and
with
all
the
parties
kind
of
being
where
you
can
actually
scrape
the
parties
outside
the
cluster?
You
can
actually
have
managed
to
burn
a
managed
prometheus
outside
of
your
cluster,
which
is
why
we
wanted
the
valuable
polities.
A
Yeah,
so
the
question
was:
do
we
allow
automatic,
like
dynamically
assigned
service
ip's
and
for
most
services
we
actually
let
the
control
plane,
the
kubernetes
control
plane,
assign
the
dynamic
IP
out
of
the
subnet.
We
tell
it
it
can
use
and
then
from
there.
As
long
as
you
don't
delete
your
service,
then
that's
pretty
much
a
static
IP
for
you,
and
even
if
you
deleted
it,
if
you
knew
what
the
IP
was,
you
can
recreate
the
service
and
specify
exactly
which
cluster
IP
that
you
want
it.