►
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Inside Kubernetes Services - Dominik Tornow, Cisco & Andrew Chen, Google
Kubernetes Services are a core abstraction of Kubernetes. In combination with Kubernetes Pods, Kubernetes Services provide the very foundation for scalable and reliable applications hosted on Kubernetes. However, even experienced Kubernetes users struggle to reason about Services end-to-end. Using a systems modeling approach, this talk will explore the mechanics of Kubernetes Services, connect the dots between K8s Services, K8s Endpoints, and the KubeProxy, all the way to IP Tables and Network Filters. You will leave with a concise and accurate understanding how K8s Services enable scalable and reliable communication in the Kubernetes cluster. In addition, you will leave with a detailed understanding under what circumstances K8s Services DON’T work and how to mitigate the situation.
https://sched.co/Uabq
A
Alright,
let
me
quickly
calibrate
the
room
who
here
can
raise
their
hands.
Please
raise
your
hand.
Alright,
thank
you
who
cannot
raise
their
hand.
Please
raise
your
hand,
there's
always
one
there's,
always
one
alright,
but
more
more
seriously,
who
is
relatively
new
to
kubernetes,
alright,
who
is
relatively
experienced
and
who
confidently
uses
kubernetes
services
today
and
who
confidently
understands
kubernetes
services
today,
oh
wow,
okay,
I
did
not
expect
that.
Actually
I
did
not
expect
that.
But
so,
when
we
talk
about
cloud
native
systems,
we
talk
a
lot
about
micro
services
or
services.
A
For
short
right,
and
in
that
conversation,
a
service
is
an
active
component
of
the
overall
application.
An
active
component
that
provides
part
of
the
functionality
so
simply
put
a
service
is
something
that
runs.
But
then,
sooner
or
later
we
realize
a
kubernetes
service
is
not
something
that
runs
well.
What
is
it
over
time?
We
build
an
intuition.
After
all,
we
have
other
things
to
worry
about,
and
there
is
probably
a
hem
chart
somewhere
that
does
citric
anyways
so
well
today.
A
A
Again,
a
distributed
system
is
composed
of
a
set
of
concurrent
processes
and
again
processes
communicate
by
exchanging
messages.
However,
here
indirectly,
via
a
network,
in
the
context
of
this
presentation,
we
are
interested
in
two
different
types
of
events.
A
process
may
send
a
message
and
a
process
may
receive
a
message.
A
message
identifies
carries
both
the
source
process
and
the
destination
process.
A
So
if
we
observe
a
send
event
that
process
one
and
the
destination
of
the
message
is
processed
for,
we
will
subsequently
observe
a
receive
event
of
that
messages.
Process
for
and
similarly,
if
we
observe
a
send
event
the
process,
one
and
the
destination
of
the
messages
process,
five,
we
will
observe
a
receive
event
of
that
message
at
process.
Five,
this
model
a
model
of
concurrent
indirectly
communicating
processes
is
not
capable
to
express
the
idea
of
services
to
do
so.
We
have
to
make
a
few
alterations.
A
First,
we
need
to
introduce
a
new
concept.
Unsurprisingly,
the
concept
of
services
here
service,
one.
Second,
we
have
to
introduce
a
new
relationship.
Membership
in
this
example
process.
Four
five
and
six
are
all
members
of
service.
One
and
third,
the
destination
of
a
message
may
now
either
be
a
process
or
a
service.
A
If
we
observe
a
send
event
and
the
destination
of
a
message
is
a
service,
the
contract
now
states
that
we
will
observe
a
receive
event
at
some
process.
That
is
a
member
of
this
service
here,
if
we
observe
a
send
event
that
process
one
and
the
destination
of
the
message
is
service
one,
we
may
observe
a
receive
event
the
process
for
or
if
you
observe,
another
event
at
process,
one
and
the
destination
of
the
messages
once
again
service
one.
A
So
at
this
point
we
identify
it
and
address
a
major
source
of
confusion
and
contention
is
a
service,
something
that
runs
or
something
that
doesn't
run,
and
the
answer
is
well.
It
depends.
It
depends
on
how
we
look
at
it.
From
the
point
of
view
of
the
application
developer
depicted
on
the
depicted
on
the
left.
A
service
is
something
that
runs
it's
a
process
from
the
point
of
view
of
the
application
operator.
A
service
is
something
that
doesn't
run
it's
an
any
cost
domain.
A
A
We
need
a
way
to
state
the
existence
of
a
service,
spoiler
alert,
kubernetes
service
objects.
We
need
a
way
to
specify
the
members
of
a
service.
Spoiler
alert
label
select
us.
We
need
a
way
to
design
a
message
to
a
service
again
spoiler
alert.
This
gets
a
unique
IP
address
and
we
need
a
way
to
deliver
a
message
to
a
service
that
is
to
deliver
the
message
to
a
member
of
that
service.
Last
spoiler
alert
network
address
translation,
but
before
we
dive
into
the
specifics,
will
briefly
review
kubernetes
architecture
and
kubernetes
network
architecture.
A
This
diagram
depicts
kubernetes
architecture.
The
kubernetes
api
server
processes
read
as
well
as
create,
update
and
delete
requests
and
persists.
Kubernetes
object
in
the
kubernetes
object
store.
A
core
controller
depicted
on
the
left
is
any
controller
that
interacts
only
with
the
API
server
reading
and
writing
kubernetes
objects.
We
are
the
kubernetes
api
typical
example.
For
a
core
controller
is
a
replica
set
controller,
it
reads:
replica
set
objects
and
right
spot
objects.
An
edge
controller
depicted
on
the
right
is
any
controller
that
interacts
with
the
API
server
and
with
other
components.
An
example
is
a
qubit.
A
Now
to
my
least
favorite
part:
networking
no
wait
advocate
Cisco
now
to
my
favorite
part
networking.
This
diagram
depicts
kubernetes
network
architecture,
kubernetes
defines
three
distinct
networks:
the
node
network,
the
pod
network
and
the
service
network.
The
node
network
consists
of
one
address
space.
Each
node
of
the
cluster
has
an
IP
address
from
this
address
space.
A
The
port
network
consists
of
multiple
address
spaces.
One
per
node.
Each
part
of
the
cluster
has
an
IP
address
from
the
address
space
of
the
node.
The
part
is
running
on
the
service
network
again
consists
of
one
address
space
and
each
kubernetes
service
of
the
cluster
has
an
IP
address
from
this
address
space.
Okay.
Finally,
we
get
to
deep
dive
into
kubernetes
services
themselves.
A
A
The
service
type
is
either
cluster
IP
or
node
port.
A
service
of
type
cluster
IP
is
reachable
from
within
the
kubernetes
cluster.
A
service
of
type
node
port
is
reachable
from
within
and
from
the
outside
of
kubernetes
cluster.
So
therefore,
cluster,
IP
services
and
node
port
IP
services
are
not
in
fact
disjoint
sets,
but
every
node
port
service
is
also
cluster
IP
service.
So,
to
put
it
simple,
node
port
adds
a
little
bit
to
cluster
IP
on
creation,
the
API
server.
A
A
A
If
the
services
selector
matches
the
parts
Labour's
and
the
pods
readiness
probe
indicates
that
the
part
is
ready
to
receive
traffic,
if
the
part
does
not
specify
a
readiness
probe,
the
part
is
considered
ready
if
it
is
a
life
and
if
the
bot
does
not
specify
a
lifeless
probe,
it
is
considered
a
life
if
at
least
one
of
the
pods
main
containers
is
up
and
running.
The
endpoints
object
is
a
materialized
view
of
the
members
of
the
service
conceptually.
A
If
the
service
is
of
type
cluster,
IP
or
node
IP
conceptually
then
points
objects
acts
as
a
lookup
table.
Just
like
this.
In
this
example,
the
service
cluster
IP
and
the
service
port
80
is
mapped
to
any
of
these
pod
IPS
and
the
port
port
8080
and
similar
the
service
port
443
is
mapped
to
the
port
port
8
443.
A
In
addition,
if
the
services
of
type
node
port
conceptually
the
endpoints
object,
acts
as
an
additional
lookup
table,
just
like
this
in
this
example
on
each
of
on
each
node
of
the
cluster,
the
node
IP
and
port
3
501
is
mapped
to
any
part.
I
P
and
the
port
port
8080
and
similar
the
node
port
3
to
502
is
mapped
to
a
pod
port
8
443,
the
node
port
values,
are
either
user
specified
or
Benares
assigned.
A
A
A
A
So,
to
reiterate,
the
queue
proxy
is
an
edge
controller
that
runs
on
every
node
of
the
cluster.
Read
service
and
endpoints
objects
and
installs
network
address
translation
rules
and
network
address.
Translation
rules
are
capable
to
change,
source
address
and
source
port
or
destination
address
and
destination
ports
of
an
IP
packet.
A
This
diagram
depicts
a
Linux,
Network
stick
or
more
specifically,
Linux
netfilter
netfilter
is
a
set
of
hooks
inside
the
Linux
kernel
that
allows
kernel
modules
to
register
callback
functions
with
the
network.
Stick
a
registered
callback
function
is
then
called
back
for
every
packet
that
traverses
the
respective
hook
within
the
network
stack.
A
Sorry,
pre
routing
output
and
post
routing
or
hooks
that
are
relevant
for
network
address
translation,
but
only
pre,
routing
and
output.
Hooks
are
relevant
for
kubernetes
network
service.
Kubernetes
service
network
address
translation
when
a
pod.
Here
the
process
on
the
right
sends
a
message
with
a
destination
address
of
a
services.
A
Cluster
IP
the
output
chain
will
rewrite
the
destination
address
to
a
member
part
of
the
service
before
the
packet
even
leaves
the
node,
and
when
a
node
receives
a
message
via
its
interface
depicted
on
the
left,
with
its
node
IP
and
the
message
has
a
destination
port
that
matches
a
node
port,
the
routing
chain,
the
pre-rounding
chain,
I'm.
Sorry,
the
pre
routing
chain
will
rewrite
the
destination
address
to
a
member
part
of
the
service
and
either
deliver
the
message
locally.
A
When
does
kubernetes
service
not
work
even
so,
kubernetes
is
working
correctly.
So
no
single
component
of
kubernetes
is
in
a
failure
State
for
this
example.
For
this
example,
we
assume
a
kubernetes
cluster
with
two
nodes,
n1
and
n2.
We
assume
one
service
and
two
parts
as
its
members,
p1
and
p2
initially
part
1
part
2
are
alive,
healthy
and
ready.
Both
p1
and
p2
are
listed
as
endpoints
in
the
endpoints
object
and
network
address
translation
will
route
requests
to
either
p1
or
p2
on
either
node
n1
or
n2.
A
Kubernetes
is
in
a
steady
state,
requests
reach
their
destination,
but
next
p1
fails
its
readiness
probe.
It
is
not
ready
to
serve
traffic
anymore,
however,
both
p1
and
p2
are
still
listed
as
endpoints
in
the
endpoint
object
and
network
address,
translation
will
still
route
requests
to
either
p1
or
p2
on
either
node
n1
or
n2.
A
Now
per
request.
There
is
a
50%
chance
of
failure
just
because
kubernetes
hasn't
caught
up
yet
next,
the
endpoints
controller
removes
p1
from
the
endpoints
object.
However,
network
address
translation
will
still
are
out,
requests
to
either
p1
or
p2
on
either
node
and
one
or
ln2
still
per
request.
There
is
a
50%
chance
of
failure,
kubernetes
simply
still
didn't
caught
up
yet
didn't
catch
up.
It
didn't
catch
up
it.
A
Next,
the
pew
proxy
on
either
n1
or
n2,
will
remove
its
network
address
translation
to
p1
now
per
request.
There
is
only
a
25%
chance
of
failure.
Kubernetes
still
didn't
fully
catch
up,
but
finally,
the
queue
proxy
on
the
other
node
will
remove.
Its
network.
Address
translation
to
p1
kubernetes
is
in
a
steady
state.
It
is
caught
up.
There
is
no
failure
happening
now.
Fun
question:
last
but
not
least,
how
can
we
possibly
mitigate
this
situation
if
you
take
anything
away
from
this
presentation?
Well,
this
one
sentence
in
the
era
of
desktop
computing.
A
B
A
C
D
So
I
do
realize
one
just
come
into
my
mind
when
you,
when
I
saw
swine.
So
when
you
create
a
service
right,
you
specifies
the
server's
IP
address
and
the
port
number
and
you
deploy
this
service
with
multiple
and
the
point.
What
about
the
service
port
is
company
with
a
with
a
particular
load?
Do
we
take
care
of
their
one?
Did
you
get
itself.
A
So
generally,
one
additional
point
we
skipped
is
that
it
is
recommended
that
you
use
named
ports
so
for
the
parts
you
would
typically
not
specify
the
the
port
you
use
in
imports
and
with
named
ports
you
let
kubernetes
choose
ports
and
then
also
the
port
mappings
as
much
as
possible,
so
in
order
to
reduce
conflicts.
But
of
course
you
are,
you
are
correct.
Conflicts
are
possible.
E
A
Could
know
the
semantics
of
kubernetes
services
is
fixed,
any
member
will
will
respond
to
a
request
with
equal
probability.
You
can,
of
course,
slice
and
dice
if
you're.
If
you
want
to
target
a
pod,
specifically,
you
can
slice
and
dice
a
pod
can
be
a
member
of
multiple
services
or
if
you
want
to
get
more
more
capabilities
into
the
hand
of
the
application
operator
and
application
developer,
you
can
resort
to
one
of
the
service
measures
like,
for
example,
linker
tea,
where
you
have
more
capability
of
actually
influencing
the
routing,
but
kubernetes
routing
is
fixed.
F
So,
as
I
was
watching
the
presentation
I
started
to
get
lost
around
the
endpoint
part
part
of
it.
It
seemed
like
the
end
point
was
what
was
doing
the
service
job
like
it
went
and
talked
to
all
the
pods
individually.
Could
you
just
talk
a
little
bit
more
about
that?
Park
I
really
got
lost
in
that
area.
Oh.
A
Yeah
absolutely
so.
The
endpoints
object
is
really
just
the
materialization
of
the
list
of
pods
that
are
the
members
for
for
the
service,
so
you
could
think
of
it
as
a
database
materialized
view,
because
otherwise
it
is
just
too
costly
for
for
kubernetes
to
actually
recalculate
which
parts
match
the
services
selector
each
and
every
time.
So
there
is
the
endpoints
controller
that
does
this
specific
part
centrally
and
uses
the
endpoints
object
to
materialize
that
information.
It
is
simply
a
list
of
pods
that
match
the
services
selector.
C
A
Sense,
yes,
that
is
now
this
is
far
out
of
my
comfort
zone,
but
network
address
translation
actually
has
something
that
is
contract
contract
tracking,
so
it
does
one
on
a
level
four.
It
actually
does
track
the
the
session
of
the
of
the
TCP
connection,
but
that's
actually
far
out
of
my
comfort
zone.
So.
B
D
G
My
understanding
there
is
no
additional
network
hope
from
what
you
explained.
Is
there
sometimes
a
circumstance
when
service
represents
an
additional
network?
Hobb.
Sorry,
just
my
understanding,
there
is
no
additional
network
hop.
So
basically
you
inept
from
the
client
directly
to
the
port
in
just
a
lookup
table.
Is
there
I,
don't
know
some
setup
when
a
service
is
representing
an
additional
network,
hop
and
its
associated
overhead.
A
With
with
IP
tables,
I
do
believe
know
where,
as
the
specifics
are,
the
Devils
are
in
the
detail.
In
that
case
the
detail
is
container
network
interface,
but
the
cni
yes,
but
in
in
in
any
case,
no,
it
should
not.
If
the
queue
proxy
mode
is
actually
user
space,
then
the
queue
proxy
is
indeed
a
reverse
proxy.
Then
you
have
an
additional
network
hop
and
if
the
packets
come
from
the
outside,
they
may
be
routed
to
any
so
node
port.
They
may
be
routed
to
any
node,
typically
in
a
round
robin
fashion.