►
From YouTube: The Easy--Don't Drive Yourself Crazy--Way to Kubernetes Networking [B] - Gerard Hickey
Description
The Easy--Don't Drive Yourself Crazy--Way to Kubernetes Networking [B] - Gerard Hickey, Smartsheet
Implementing Kubernetes is not technically difficult, but the networking layer continues to confuse and cause implementation problems for those new to Kubernetes. Not everyone is capable of using GKE and may need to implement Kubernetes in an on-prem facility. Certainly there is a wealth of online documentation to assist new users but some of this documentation is contradictory due when the documentation was written and the multitude of network stacks available.
This presentation attempts to provide clarity for new implementers and those wishing to understand Kubernetes networking better. The content covers how networking is accomplished in the Kubernetes environment and the reasons why it is implemented differently than traditional network environments. In addition, several of the popular network stacks will be reviewed to provide attendees with knowledge to make a better informed decision when choosing between network stacks.
About Gerard Hickey
Gerard Hickey is a Principal Systems Engineer at Smartsheet where he is building the next generation data center for the world's leading collaboration solution. He has spent the past decade working with and implementing the latest technologies in an effort to provide better automation. In addition, he is active with the open source community providing bug fixes and patches to several projects and in his spare time he works on a number of Raspberry Pi and ESP8266 home automation projects.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
My
name
is
jarred
hickey
I'm
principal
systems,
engineer
at
smartsheet
we're
up
in
Seattle.
We
do
a
lot
of
interesting,
crazy
things
and
have
lots
of
fun
doing
it.
Today.
I
want
to
talk
to
you
about
kubernetes
networking,
talk
to
you
about
some
of
the
pitfalls.
I've
encountered
and
hopefully
get
you,
so
you
don't
see
them
yourself.
A
First
I
would
think
lots
and
lots
of
people.
There's
many
people,
I'd
like
to
thank--,
there's
probably
way
too
many.
They
can
think
the
one
person
I
do
want
to.
Thank,
though,
is
Eric's
it
him
from
ty
Guerra
he's
the
one
person
it
really
helped
me
through
got
my
first
network
stack
up
and
running
under
kubernetes.
He
was
patient
enough
to
sort
of
guide
me
through
and
show
me
the
errors
that
I
was
having
definitely
definitely
good
good
good
resource
out
there.
A
I
also
would
like
to
say
during
the
presentation:
if
you
have
questions
I'd
like
you
to
hold
on
to
them
to
the
end,
there's
a
lot
of
information
here,
especially
with
a
30-minute
time
line.
It's
gonna
be
awfully
hard
to
fit
it
all
in
and
hopefully
we'll
fit
in
and
this
time
at
the
end
for
questions
we'll
do
that,
if
not
I'll
be
happy
to
sit
out
in
the
hallway
and
talk
to
people
and
answer
questions
there
so
that
let's
go
ahead
and
get
started.
A
So
this
is
a
fairly
typical
kubernetes
setup,
cuckoo-bananas
network
setup.
Principally
there
are
three
networks
involved.
There's
your
physical
network,
the
one
that
everybody's
aware
of
it's
a
switch
and
computers
are
plugged
into,
but
there's
two
virtual
networks
that
occur
ones.
The
pod
network
and
the
other
is
the
service
network.
A
The
on
the
pod
network,
each
pod
receives
a
virtual
interface
on
the
host,
a
virtual
Ethernet
interface.
As
a
result,
it
sits
out
on
the
pod
network,
addressed
separately,
we'll
see
more
of
that
here.
In
a
moment,
I've
tried
to
annotate
a
few
other
things
that
go
on
in
here,
such
as
you
have
the
qubit
doing
health
checks
to
each
pod,
the
Cuba
that
actually
sits
in
the
OS
and
talks
to
the
pod.
Through
the
virtual
interface
or
virtual
Ethernet
interface,
we
have
the
server,
which
is
the
abstraction.
A
A
Real
quick
I
wanted
to
touch
on
what
network
rages.
Could
you
can
use
I
actually
personally
like
to
use
the
private
address?
As
the
RFC
1918
addresses
on
the
pod
and
the
service
networks?
It's
it
solves
a
lot
of
problems,
a
lot
of
networks.
A
lot
of
corporate
networks
tend
to
use
the
10
network
for
their
backbone.
A
The
standard
kubernetes
install
tends
to
use
those
address
spaces
too.
That's
great.
If
you're
not
conflicting,
but
long
cases,
it
can
either
conflict
or
cause
other
problems.
So
that's
why
a
lot
of
times
I
move
them.
That
does
cause
issues.
When
you
do
your
install
we'll
talk
about
that
here
in
a
bit
too,
the
one
thing,
I
would
say
is
no
matter
which
way
you
go.
Keep
your
networking
simple.
Don't
don't
try
to
be
creative,
don't
try
to
take
your
say
like
192,
168
break
it
up,
make
part
service
and
part
pod.
A
You'll
drive
yourself
crazy
and
you
just
caused
all
sorts
of
problems
down
the
road.
So
just
keep
us
keep.
Those
address
ranges
separated
you'll,
be
much
much
better.
So
I
have
a
couple
key
understandings
that
I've
tried
to
incorporate
through
this
and
I
hope.
Everybody
will
start
to
understand.
The
big
one
is
that
every
pod
can
communicate
to
every
other
pod
on
the
pod
network.
So
we
don't
have
any
funny
things
we
have
to
do.
A
It's
just
straight
open
up
a
socket
and
talk,
so
it's
just
like
having
a
computer
and
talking
to
another
computer
through
a
typical
TCP
socket.
So
there's
no
port
mapping
schemes
like
with
you're
dealing
with
docker
on
two
separate
hosts.
You
have
to
go.
Do
your
port
mapping
to
be
able
to
get
one
container
on
one
host
talk
to
another
container?
That's
not
so
in
kubernetes
and
that's
actually
a
benefit
make
some
things
a
lot
easier.
So
if
we.
A
Again,
the
are
connected
through
a
virtual
Ethernet.
It
actually
connects
into
a
bridge
that
then
eventually
gets
out
through
the
network
stack
onto
the
Ethernet.
We
also
have
a
cube
proxy.
That
runs
will
talk
more
about
that
here
in
a
bit
when
we
get
towards
services.
The
cube
proxy
ends
up
controlling
a
lot
of
the
IP
tables
for
moving
traffic
around
sometimes.
A
Looking
a
little
bit
closer
at
some
of
the
network,
substrate
the
the
network
substrate
itself,
the
the
stack
itself
is
responsible
for
maintaining
all
IP
information.
That's
maintained
the
subnet
information
for
all
the
nodes
on
the
Cabernets
cluster,
along
with
what
IPS
are
in
use,
so,
whether
you're
using
flannel,
calico
or
anything
else
like
that.
That's
the
responsibility
of
that
stack
to
Maine
to
maintain
those
IPs.
A
Since
we're
given
out
eyepiece
to
every
pod,
heavy
pod
gets
a
unique
IP.
That's
that's
where
we're
just
like
on
a
regular
Network,
even
though
it's
a
virtual
network,
it's
just
like
on
a
virtual
note
of
real
network,
where
each
pod
is
now
able
to
communicate
directly
by
just
a
straight
IP
address.
Now
here's
one
of
the
critical
things.
A
This
is
probably
what
my
biggest
failure
is
when
I
installed
koobideh
is
the
first
time
I
was
I
actually
and
I
actually
had
many
installs
and
I
actually
was
doing
a
lot
by
hand,
but
eventually
by
talked
with
Joe
beta
at
hep.
Do
he
convinced
me
to
use
cube?
Atm
and
I
will
tell
you
a
cube
ATMs
like
magic,
I,
love
it
to
death.
Now
the
one
failure
is
that
I
had
not
not
really
understanding
is
when
I
ran,
cube.
Atm
I
did
not
know.
A
A
If,
when
you
run
your
cubes,
your
cubed
a
set
up
whatever
you
use
when
you
use
cops
or
cue
ATM
or
any
other
software
out
there,
almost
all
of
them
will
require
you
to
tell
tell
what
network
ranges
you're
going
to
use
and
they're
just
command
line
switches
or
if
you
usually
config
file,
you'll,
listen
to
the
config
file,
and
that
will
allow
you
to
move
forward.
So
once
you
specify
that
things
get
a
whole
lot
easier.
A
So
again
understand
where
your
network
traffic's
going
to
be,
will
help
you
in
troubleshooting,
but
also
keep
you
from
being
frustrated,
because
you
will
get
frustrated
once
in
a
while
I
sort
of
added
this
in
I
originally
had
this
in
and
I
pulled
it
out
for
awhile
and
with
all
my
many
rewrites
I've
added
it
back
in
because
it
caused
me
some
grief
earlier
this
week
in
preparing
some
of
the
demos
here
for
you.
So
I
wanted
to
give
you
a
little
heads
up.
Cni
the
container
I
think
its
container
yeah
container
network
interface.
A
The
container
network
interface
is
a
wonderful
standard,
make
it
tends
to
unify
and
make
things
a
lot
easier
for
breaking
up
your
stacks.
So
you
don't
have
to
configure
each
stack
separately
used
to
be
a
pain
in
the
rear
end
for
getting
stack
up
and
running,
and
if
anybody
has
ever
done,
OpenStack
stuff,
you'll
know
what
I'm
talking
about
that
was
very
much
a
each
stack
was
unique
and
how
you
brought
it
up
so
see
and
I
certainly
solves
a
lot
of
those
things.
The
interesting
thing
is
see
and
I
will
also
accept
multiple
backends.
A
A
You
may
end
up
having
to
go
clear
out
all
your
CNI
configuration
files
as
you
bring
your
stack
up
to
just
keep
it
clean
they're
located
under
/f
c
/c,
NE
c
ni
many
times,
they're
put
in
a
net
d
directory,
so
you'll
go
down
in
there.
Clear
them
out
reads
up
your
stack
and
should
be
hopefully
much
better.
A
This
is
a
typical
some
well,
it's
actually
an
older
ones
for
an
old
old
old
sack,
the
the
host
local
stuff,
which
you
would
have
to
maintain
the
subnet
and
everything
so
I
mean
every
machine,
would
have
a
separate
CNI
configuration
but
most
network
stacks.
Today,
your
CNI
configuration
will
be
the
same
on
all
machines
which
does
help
a
lot
and
lend
itself
to
configuration
management
and
things
like
that.
So
get
a
little
demo
here.
A
A
A
So
there
we
go,
we
get
into
a
busy
box.
Now
we
can
do
a
interface
look
at
the
interface
on
it.
Then
we
see
just
a
single
each
zero.
So
these
are
the
sort
of
things
you
see,
and
this
is
one
of
the
things
that
always
drove
me
crazy.
Trying
to
bring
stuff
up
is
everybody
says,
run
these
commands
and
everything's
magical,
and
you
never
know
what
just
what
to
expect
so
I'm
hoping
this
will
give
people
an
idea
of
what
to
expect.
A
A
A
All
right
so,
hopefully
by
now
you're
starting
to
get
a
feel
of
what
networking
looks
like
in
a
pod
and
how
it
traverses
back
and
forth
and
so
forth.
Let's,
let's
move
on
and
talk
about
services,
so
key
number
key
understand:
number
three:
the
services
are
crucial
for
not
only
service
discovery,
but
also
moving
traffic
between
skimming
distributing
the
traffic
into
your
pod.
So
if
you
have
a
service
that
you
run
and
say
like
five
pods
against
the
wall,
five
services
are
running.
A
It's
sort
of
a
simple
load
balancer,
so
as
being
a
load
balancer,
it
does
suit
traffic.
The
one
thing
it
doesn't
do
like
typical
load
balancer,
it
doesn't
allow
you
access
control.
It
doesn't
say
that
I'll
accept
traffic
from
pod
this
part,
but
not
that
pod
or
this
address
range.
Not
this
address
range.
There
are
no
real
traffic
controls,
anything
that
the
service
receives.
It
will
rebroadcast
to
your
pods
or
one
of
the
pods.
A
So
the
one
of
the
secrets
here
is
that
the
service
VIP
know
the
virtual
IP
is
actually
a
magic
of
IP
tables,
so
you'll
never
see
the
the
IP
on
an
interface,
so
you'll
never
see
an
interface
with
that
IP
on
it,
on
a
machine
or
in
a
pod
or
anything
else,
and
you
will
never
see
a
route
to
the
services
network
anywhere.
So
it's
all
IP
tray
tables
magic.
The
magic
of
the
service
discovery
piece
is
cubes
DNS,
so
cubes
DNS
will
create
an
a
record
for
every
one
of
your
services.
A
I'll
give
you
an
example
here
about
midway
down
on
the
fourth
third
bullet:
nginx
default,
SVC,
dot,
cluster
dot
local,
so
nginx
is
the
service
name.
The
default
is
your
namespace.
So
if
you
put
something
in
another,
namespace
expect
that
to
be
different,
SVC
is
always
going
to
be
there
and
then
cluster
dot.
Local
is
the
domain
that
you've.
Given
your
cluster,
like
I,
say
by
default
cluster
dot.
Local
is
your.
What
kubernetes
normally
has
what
it
expects
that
can
be
changed
if
you
wanted
to
add
it
into
a
full
DNS
architecture?
A
If
you
want
to
so
and
that's
about
it,
there
I
talked
a
little
bit
about
cube
proxy
I.
Just
want
to
touch
on
that
for
a
moment
of
it's
a
couple.
Little
things
to
know
about
cubes
proxy.
The
IP
tables
should
be
the
default
for
cube
proxy
now
on
any
installation.
I,
don't
know
when
that
changed
in
Cuba
Nettie's
world
I
know
at
least
1.6.
It
was
IP
tables.
Someone
may
know
it's
been
earlier
than
that,
but
no
one
should
ever
use
it
user
space.
Unless
you
need
to
do
some
real
debugging.
So.
A
Now
the
thing
about
cube
proxy
is
all
it
does.
Is
it
it's
principally
they
have
to
set
up
node
port
and
we'll
talk
about
node
port
in
a
second
here,
I
believe
actually
I
think
I
might
fight.
You
talked
about
on
last
slide.
Node
port
allows
you
to
distribute
the
traffic
to
your
pods
from
the
outside
interface,
so
it's
a
way
of
ingesting
traffic
into
your
cluster.
A
It
does
this
in
a
not
even
in
the
round
robin
fashion.
It
does
this
randomly
so
I
got
a
little
diagram
here.
I
actually
have
them
better
diagram,
one
point,
but
unfortunately
I
lost
that
image.
The
the
way
it
does
that
with
the
IP
tables.
The
new
connection
comes
in
hits
the
first
IP
tables
and
there's
a
basically
one
out
of
three
chance
of
hitting
that
node.
If
it
its
hits
that
node
we're
done,
if
it
doesn't,
if
it
decides
not
to
hit
the
node,
it
moves
a
second
rule
which
then
has
a
50%
chance.
A
So
if
you
haven't
figured
out
one
over
three
went
over
to
so
ever
how
many
knows
you've
got
its
1
over
whatever
node
number.
It
is
percentage-wise
its
if
that
rule
is
going
to
activate
or
not
if
it
doesn't
decide
to
hit
node
to
the
catch-all
rule,
which
is
the
last
rule,
we'll
send
all
the
traffic
through.
That
sounds
great.
It's
a
simple
approach,
but
the
simple
approach
is
not
always
the
best.
There
is
a
fallacy
here
that
the
distributing
traffic
randomly
you
can
get
Hut
spots.
A
So
if
you've
got
high
traffic
loads,
lots
and
lots
of
traffic
coming
in
you'll
start
to
see
errors
appear
and
then
just
disappear,
and
it
just
be
instantaneous
like
that.
You
won't
even
easily
be
able
to
track
it.
So
just
be
aware,
if
you're
using
the
node
port
for
adjusting
traffic,
if
you've
got
lots
of
lost
traffic
coming
in
this
could
be
an
issue
so.
B
A
So
it's
a
little
little
thing
that
I
found
again,
which
is
fairly
common,
a
lot
of
people.
Don't
really
think
about
this.
Initially,
it's
actually
a
DNS
problem.
If
we
look
at
our
resolve
cough
inside
the
pod,
we
see
that
our
IP
range
is
out
of
spec.
It's
not
what
our
service
range
is
so
I'm
going
to
go
ahead
and
correct
it
here
for
a
second
just
continue.
The
demo
and
I'll
show
you.
A
I
can't
even
spell
sight
that's
a
typical
response.
If
you
hit
something
it's
not
there,
you'll
get
a
was
a
307,
yeah,
307
and
then
I
hit
it
correctly
on
the
correct
port
and
I
get
a
response
from
the
nginx.
That's
running
there,
so
I
exit
back
out
and
I'm.
If
you
go
look
at
how
your
cubelet
gets
configured
in
this
case,
mostly
cubelets,
probably
run
under
system
D,
so
they'll
be
under
the
Etsy
system.
D
directory
there's
actually
a
parameter
that
sets
your
DNS
server.
A
A
A
That's
about
all.
It
really
is
route
there.
There
is
a
little
bit
there's
a
bit
of
a
hack
to
do
layer
4
with
nginx
there's
also,
if
you're
I
didn't
put
it
on
here,
but
the
like
the
f5
container
connector
also
allows
you
to
do
layer
4
through
some
special
magic.
They
do
it's
not
a
standard
ingress
resource.
A
A
That
way,
one
thing
that
should
be
noted:
two
of
the
three
types
of
services
are
out
there:
the
cluster
IP,
the
load
balancer
and
the
node
port
cluster
IP
is
the
type
that
cannot
be
sent
to
an
ingress.
Cluster
IP
is
really
geared
just
for
talking
within
the
cluster.
So
if
you
have
two
micro-services
one
that
depends
on
the
other
and
one
that
talks
as
another
micro
service,
that's
where
you
would
use
something
like
cluster
IP
to
route
the
traffic
from
one
micro
service
to
another.
A
Yeah
and
now
I
just
have
to
note
about
sort
of
if
you're
going
to
use
a
layer
4
protocol,
if
you
don't
do
it
through
the
nginx
in
ingress,
the
other
way
you
can
sort
of
do
it
is
export
that
service
as
a
node
port
and
then
on
an
external
load.
Balancer
manually,
configure
it
to
forward
traffic
to
that
node
port.
A
A
Alright,
sorry,
to
sum
up
here
a
little
bit,
but
some
of
the
network
stack
choices.
You
have
did
this
by
all
mean
by
no
means
exhaustive.
This
is
just
for
the
most
common
ones
or
popular
ones.
There's
a
whole
lot
more
out.
There
flannel,
which
is
what
I've
been
using
here
for
the
demonstrations,
is
a
very
simple
stack
to
bring
up.
It
manages
itself
it's
it's
turned
it
on
it.
Just
pretty
much
goes
weave
the
weave
net
stuff
I've
actually
never
used
that.
A
But
from
what
I
understand
it's
pretty
straightforward
to
set
up,
it
doesn't
scale
better
than
flannel.
It's
a
it's
easier
to
manage
than
flannel
when
you
go
to
scale
it
Project
calico.
This
is
actually
my
favorite
just
because
of
some
of
the
other
features
it
has,
but
it's
very
similar
to
weave
and
scaling.
A
There's
indications
specifically
when
you
go
to
BGP
mode,
they
will
scale
much
larger
than
weave,
but
I
don't
have
anything
to
fully
document
that
and
anything,
but
what
it
does
do
one
of
the
advantages
it
does
have
it's
one
of
the
few
and
I've
only
found
like
two
or
three
Network
stacks.
To
do
this,
allow
you
to
specify
the
egress
rules,
so
all
the
sacks
all
the
stacks
do
adhere
to
the
network
policy
API.
So
you
can
have
the
ingre
rulz,
but
the
egress
rules
on
what
you
can
to
make
yourself
more
secure.
A
The
egress
rules
of
what
you
want
to
use
and
then
Ramona,
which
is
something
I,
was
looking
at
for
a
while
because
of
the
security
aspect,
but
I
had
to
rule
it
out,
but
it
does
look
have
some
interesting
aspects,
it's
much
like
the
others,
but
one
of
the
interesting
things
is.
It
will
actually
allow
you
to
take
a
service
and
expose
it
as
a
real
real
world
dip.