►
From YouTube: Failure Stories From the On-premise Bare-metal World - Stephan Fudeus & David Meder-Marouelli, 1&1
Description
Don’t miss out! Join us at our upcoming events: EnvoyCon Virtual on October 15 and KubeCon + CloudNativeCon North America 2020 Virtual from November 17-20. Learn more at https://kubecon.io. The conferences feature presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Failure Stories From the On-premise Bare-metal World - Stephan Fudeus & David Meder-Marouelli, 1&1 Mail & Media Development & Technology GmbH
Setting up a Kubernetes cluster is not trivial in general and doing that on-premise with bare-metal machines has very specific challenges. In addition to "just" defining the pod overlay network and embedding the host network into an existing topology, ingesting and directing traffic into the cluster is an essential task. Regardless of the chosen solution, compromises need to be made and eventually, failures will occur. In this session David and Stephan will show their real world setup of reasonably large clusters (up to 150 nodes currently), the reasoning behind the network design and the failure stories they experienced and had to deal with. The topics are ranging from ingress controllers via BGP, network routing, DNS and iptables to hardware load balancer appliances - and how those technologies are not as cloud native as one might wish.
https://sched.co/Zemo
A
B
A
To
give
you
a
bit
of
context
where
kubernetes
is
running,
of
course
we
are
developing
in-house
software
and
one
of
the
key
factors
that
we
strive
for
is
time
to
market
and
for
optimizing
time
to
market
there's
three
main
components
which
are
tightly
coupled
to
each
other
one.
We
need
a
strong
devops
culture
which
only
can
work
if
we
have
a
reliable
and
a
good
cicd
platform,
which
in
turn
requires
a
robust
and
scalable
quantitative
runtime
platform,
and
this
is
where
kubernetes
is
being
used.
B
B
Those
operations
and
devops
teams
are
tenants
for
our
platform.
We
focus
on
soft
multi-tenancy,
so
we
do
not
try
to
strictly
isolate
the
tenants
from
each
other
because
they
are
considered
friendly
users,
but
we
still
try
to
do
our
best
to
help
them
isolate
their
their
compute
workloads
and
their
network
traffic
from
each
other
by
using
network
policies
and
limits
and
requests,
and
we
try
to
provide
them
with
tenant.
Specific
isolated
infrastructure
like
dedicated
ingress
controllers,
pretend
our
main
focus
is
stateless
microservices.
But
by
now
we
have
some
stateful
services
as
well.
B
B
We
serve
multiple
clusters.
Currently
it's
15
clusters.
We
are
operating
mostly
separated
on
network
dimensions,
so
we
have
dedicated
clusters
in
the
front
end
layer.
We
have
dedicated
backend
clusters
and
dedicated
clusters
on
the
infrastructure
layer.
We
separate
as
well
on
the
data
center
level.
We
have
three
separated
worldwide
distributed
data
centers
and
we
isolate
clusters
regarding
live
usage
and
non-live
usage.
B
All
our
clusters
use
bare
metal
nodes
on
premise
in
our
own
data
centers.
We
do
not
use
any
cloud
providers
at
all
for
our
services
and
one
thing
which
is
relevant
for
some
failure
stories
later
on.
We
use
cg
nut
pod,
siders
and
service
sliders,
so
they
are
isolated
and
reused
within
each
clusters
and
there
is
no
direct
network-wise
interaction
between
our
clusters.
B
B
B
Of
course,
that's
not
trivial,
so
let's
have
a
bit
of
a
look.
How
our
setup
is
looking
like
this
problem,
you
described
is
usually
occurring
or
can
usually
occur
in
our
frontend
clusters.
Our
frontend
clusters
do
use
a
hardware
load
balancers.
We
use
the
f5
big-ip
appliances
to
expose
our
clusters
to
the
public
internet
and
we
use
them
as
kind
of
an
edge
gateway
to
protect
from
malicious
public
traffic
and,
as
you
all
might
know,
services
in
kubernetes
can
be
can
have
two
different
external
traffic
policies.
B
There
is
external
traffic
policy
cluster
and
external
traffic
policy.
Local
and
the
difference
between
both
of
them
is
that,
with
external
traffic
policy
cluster,
each
traffic
entering
a
kubernetes
node
is
distributed
across
all
pots
within
the
cluster
q.
Proxy
takes
care
of
programming
iv
tables
in
that
way,
so
that
everything
is
distributed
for
the
integration
of
the
hardware
load.
Balancer
services
are
of
type
noteport,
so
it's
a
port-based
addressing
scheme,
and
so
the
f5
forwards
traffic
to
one
of
the
host
ips
on
a
specific
service.
B
B
So
but
let's
have
a
look
at
the
state
in
between
what
happens
when
the
health
status
unknown,
and
now
we
stumble
upon
our
problem.
F5
has
a
logic.
When
traffic
is
unknown,
then
it's
still
routed
there.
Only
if
the
health
state
is
red,
then
the
traffic
will
not
be
routed
and
if
you
use
external
traffic
policy,
local
and
as
just
mentioned,
a
new
node
does
not
contain
any
workload
in
the
beginning.
You
have
a
risk
that
the
traffic
is
routed
to
a
node
which
cannot
serve
the
workload.
B
B
In
addition,
you
can
tune
your
health
checks
so
that
you
notice
earlier
that
the
health
state
is
red
and
not
green.
The
standard
settings
would
take
16
seconds
until
the
health
state
would
be
established
and,
of
course,
you
can
tune
that
to
notice
that
earlier
you
do
the
health
checks
more
often,
but
of
course
that
puts
more
health
check
load
on
the
cluster.
B
So
we
use
this
as
mitigations,
but
the
real
thing
we
did
is
we
patched
the
f5
operator
and
we
opened
the
issue
and
provided
a
merge
request
to
to
fix
it
or
to
at
least
ease
that
and
we
explicitly
activate
nodes
now
after
the
health
state
has
been
established.
So
first
we
join
a
node
to
the
cluster
and
it
is
still
inactive,
but
it's
already
known
to
f5.
The
health
state
can
be
established,
and
only
if
it's
established,
we
enable
the
node
in
the
pool
member
table
so
that
it
gets
its
share
of
traffic.
B
Another
option
is
to
reduce
the
number
of
nodes
where
a
five
forwards
traffic
turn
by
default.
Every
kubernetes
node
would
be
eligible
to
receive
traffic
directly
from
f5,
but
you
could
mark
only
a
specific
subset
of
your
nodes
as
ingress
nodes
like
a
onenote
per
rack,
for
example,
and
only
forward
traffic
there.
So
you
have
a
lower
amount,
a
smaller
amount
of
notes
which
are
registered
in
f5,
and
you
have
less
fluctuation
on
the
pool
members
and
the
last
thing,
which
is
the
deepest
integration.
B
You
can
do
a
direct
vxlan
integration
with
f5.
Then
you
would
not
use
the
node
port
mechanism
anymore,
but
the
f5
balancers
would
be
considered
pot
in
your
cluster.
They
would
receive
ips
from
the
pot
cider
range
and
by
that
the
f5
instances
directly
address
pots
and
have
pot
ips
in
their
pool
member
tables
and
not
node
ips
anymore,
and
the
problem
with
the
unknown
health
state
for
a
node
would
not
occur
anymore.
A
Okay,
great,
but
not
all,
of
the
problems
and
issues
we
encounter
are
located
in
the
control
plane.
Some
also
occur
on
the
data
plane.
Let's
have
a
look
at
these
so
now
that
my
application's
running
fine,
I'm
looking
at
the
data
and
processing-
and
I
notice
that
for
some
use
cases
where
in
the
old
world
I
could
use
the
source
ip
address.
I
only
see
internal
ip
addresses
like
for
geo
location
services
or
for
classifying
traffic
or
for
for
identifying
botnets.
A
B
So
thanks
for
that
question,
that's
a
tricky
one
as
well,
so
in
kubernetes,
a
bunch
of
network
address,
translation
is
involved
and
the
standard
thing
is
when
a
user
connects
to
a
service
within
kubernetes
traffic
is
directed
to
one
of
the
kubernetes
nodes
and
then
the
ip
tables
rules
kick
in,
which
are
programmed
by
q,
proxy
and
usually
two
layers
of
network
address
translation
take
place
there.
The
one
is
that
the
destination
address
is
changed.
B
This
is
the
load
balancing
mechanism
of
kubernetes
so
that
the
service
ip
destination
address
is
changed
into
a
pots.
Ip
address
for
routing
the
the
traffic
to
one
of
the
pods.
But
the
second
thing
which
happens
when
traffic
is
sent
from
one
kubernetes
node
to
another
one
is
that
sourcenet
is
applied
as
well
to
allow
for
the
traffic
returning
on
the
same
path
as
it
entered
the
pod.
So
then
a
pod
can
respond
to
the
original
host
there.
B
Imagine
you
have
a
client
and
you
have
some
workload
pot
and
the
first
thing
the
client
does
is
usually
the
request
is
sent
to
some
load
balancer
already
part
of
your
corporate
network
and
in
you
remember
in
our
case
these
are
f5,
big-ip
load
balancers
and
they
are
for
full
isolation.
We
apply
already
source
network
address
translation
on
the
load
balancer
so
already
when
the
traffic
enters
the
kubernetes
cluster
at
the
cube
proxy.
B
It's
not
the
original
client
ip
anymore,
but
it's
one
of
the
source
net
addresses
of
the
f5,
and
now
we
have
the
one
from
the
slide
before
cue
proxy.
Does
a
source
net
and
destination
net
to
forward
the
packet,
but
now
there's
ingress
controllers
as
well.
In
our
example,
here
it's
traffic,
so
the
packet
is
not
only
forwarded
to
the
workload
directly.
It's
for
what
it
took:
traffic
and
now
traffic
acts
as
a
layer,
7,
reverse
proxy
and
then
for
what's
the
traffic
to
your
workload.
So
what
the
workload
part
sees
as
the
physical
source?
B
B
On
the
first
point,
where
the
packet
is
entering
your
system,
in
our
case
it
would
be
our
load
balancer
and
you
would
preserve
and
use
a
forwarded
or
x,
forwarded
four
or
x
forwarded
host
headers
for
what
it
is
based
on
an
rfc
and
x4
x4.4
and
x4.
What
it
hosts
are
best
practices
which
are
not
really
rfc
backed.
B
If
you
really
want
to
make
sure
that
nobody
injected
wrong
ip
addresses,
then
you
would
even
reset
potentially
existing
x,
forwarded
four
headers
on
the
load
balancer
and
only
add
the
physical
client
ips.
The
first
hop
when
you
within
your
x
forwarded
for
list
for
traffic,
which
is
not
capable
on
the
on
the
application
protocol
level
to
to
add
some
headers
with
original
ip
information.
B
A
B
Was
a
tricky
one?
We
observed
that
with
our
calico
network.
We
are
using
calico
with
the
ipip
encapsulation
and
since
some
recent
calico
version
calico
by
default,
assigns
smaller
ip
blocks.
Originally,
we
use
24
ip
blocks,
so
the
the
block
where
the
pot
ips
come
from,
which
is
attached
to
a
specific
when
it
is
node
since
a
more
recent
version,
the
default
is
slash
26,
and
that
means,
if
on
your
nodes,
are
running
more
than
64
pots.
B
You
need
more
ip
addresses
than
fit
into
a
single
slash,
26
block,
so
you
have
multiple
blocks
attached
to
a
host.
That
in
itself
is
not
really
a
problem,
so
for
each
of
the
ip
blocks
a
dedicated
ip
address
is
used
for
the
ipip
tunneling
and
we
observed
that
an
ip
address
is
used
twice
once
for
a
pot
and
once
for
the
ipip
tunnel
interface
and,
of
course,
the
tunnel
interface
won
at
that
point.
B
B
A
B
So
what
you
hit
now
is
even
one
layer
deeper.
This
now
is
a
buck
in
the
linux
kernel,
so
the
linux
kernel
has
a
completely
fair
scheduler
and
that
has
a
bandwidth,
control
management
and
that's
responsible
for
the
pod
cpu
limits.
There
is
a
different
mechanism
for
managing
cpu
requests
and
if
you
have
now
a
multi-threaded
workload
which
is
not
really
cpu
bound,
but
does
a
lot
of
I
o
weights
or
network
weights,
then
you
might
get
hit
by
this
bug.
B
B
A
Sometimes
I
lose
tcp
connections
which
already
have
been
established,
but
all
of
a
sudden
they
are
dropped
again.
This
is
happening
when
you
do
cluster
redeployments,
but
it
also
happens
when
my
application
is
scaling
up
and
down,
because
spikes
cause
the
autoscaler
to
create
more
replicas
or
remove
replicas.
B
So
now
we
come
back
a
bit.
We
return
a
bit
from
the
low
level
and
that's
more
of
a
configuration
and
design
thing
in
our
bare
metal
setups.
So
I
mentioned
earlier
that
we
use
the
five
load
balancers
as
gateways
for
our
fronted
networks,
but
in
our
backend
networks
and
in
our
infrastructure
clusters.
B
We
do
not
use
hardware
load
balancers,
but
we
use
direct
bgp
announcements
of
service
ips
or
in
fact
of
load
balancer
ips
to
our
routing
infrastructure,
and
we
use
metal
lb
for
that
and
when
those
service
ips
are
announced
via
bgp,
then
we
use
equal
cost,
multi-path
routing.
So
the
routers
then
would
know
multiple
paths
on
how
to
reach
that
load.
Balancer
ip
address
when
using
when
knowing
multiple
paths
and,
of
course
the
router
has
to
decide
which
path
to
use
and
the
router
in
general
is
stateless.
B
So
it
needs
to
be
some
reliable
algorithm
on
where
to
send
traffic,
and
that
means
that
anytime,
the
routing
tables
are
changing.
Then,
potentially
the
association
where
to
route
connections
are
sent
because
there's
no
real
connection
tracking
in
place.
So,
every
time
the
routing
tables
are
changing,
it
can
cause
resets
of
your
tcp
connections,
because
the
follow-up
packets
are
sent
to
a
different
destination
host,
which
does
not
know
of
the
tcp
connection
and
first
sends
a
reset
reply.
So
only
when
the
tcp
connection
is
newly
established,
everything
works
again.
Fine.
B
The
solution
to
that
one
is
to
reduce
the
number
of
speakers
to
stabilize
the
changes.
The
less
speakers
you
have,
the
less
changes
to
your
routing
tables
you
have,
but
the
speakers
you
have
the
less
ingress
paths
you
have
as
well
to
ingest
your
traffic,
so
you
cannot
reduce
them
the
speakers
to
just
a
single
one
or
two,
because
you
need
to
have
sufficient
amount
of
speakers
to
process
all
your
inbound
traffic.
B
The
second
thing
you
mentioned
was
when
you
have
load
spikes
and
you
are
using
auto
scaling.
We
observed
that
when
auto
scaling
our
traffic
ingress
controller
and
for
being
able
to
keep
our
source
ip
addresses,
we
use
traffic
with
external
traffic
policy.
Local
and
metal
lb
is
aware
of
external
traffic
policy
local
and
is
then
only
announcing
the
service
ip
address
of
traffic.
B
B
Then
a
new
traffic
pod
comes
up
on
a
new
speaker,
so
a
new
route
is
established
or
a
new
route
is
announced
to
the
to
the
routing
infrastructure
and
the
routers
change
the
routing
tables,
because
they
now
have
one
routing
entry
more
in
their
tables.
And
so
you
have
an
ecmp
change
and
tcp
connections
are
disrupted
and
yeah.
B
If
you
have
spiky
workloads,
so
scaling
up
means
an
ecmp
change,
scaling
down
means
again
an
ic,
an
ecmp
change,
and
that,
of
course,
is
then
a
problem
because
you
have
a
lot
of
changes
to
the
routing
tables.
Our
solution
in
that
case
was
that
we
have
as
many
traffic
instances
always
as
we
have
bgp
speakers.
B
The
real
traffic
balancing
or
the
distribution
occurs
on
the
routers.
So
if
you
have
one
node
with
the
bgp
speaker
and
five
traffic
pods-
and
you
have
one
node
with
the
bgp
speaker
and
just
a
single
traffic
pod,
then
the
node
with
a
single
traffic
pod
will
get
the
same
amount
of
traffic
into
your
pot.
As
the
other
note,
so
it
will
have
five
times
as
much
traffic
as
the
other
traffic.
A
Okay,
as
the
last
takeaway
there's
still
a
few
more
remaining
open
issues.
Recently,
we
observed
again
network
ingress
issues
where
actually
we
lost
a
traffic
and
it
certainly
has
got
something
to
do
with
the
load,
balancing
mechanisms
and
or
egp.
The
analysis
is
still
ongoing,
so
we
weren't
actually
able
to
show
that
to
you
yet.
B
We
stumbled
upon
pot
affinities
and
anti-affinities,
which
did
not
work
as
expected
to
to
guarantee
the
highest
amount
of
redundancy
in
our
deployment
pipelines.
We
add
standard
affinities
to
distribute
the
application
first
in
different
rooms,
then
in
different
top
of
rack
switches,
then
to
different
racks
and
then
to
different
hosts,
and
we
noticed
at
multiple
occasions
that
these
affinities
did
not
work
as
expected.
So
setting
up
affinities
is
not
as
trivial
as
one
might
expect,
and
maybe
there
are
even
bugs
lurking
in
there,
which
we
did
not
really
pinpoint
so
far.
A
One
last
thing
we
haven't
been
able
to
clarify
yet
is
an
issue
with
q
proxy
that
was
completely
overloaded
through
a
single
application
with
just
20
parts
and
these
pods
for
some
reason
kept
crashing
and
went
in
ultimately
into
a
crash
loop
back
off,
but
q
proxy
was
completely
overloaded,
did
consume
lots
of
cpu
and
wasn't
able
to
update
the
ip
table's
rules
quickly
enough,
so
that
sort
of
screwed
up
the
entire
nodes.
Where
that
happened.
A
Design
through
the
entire
stack
the
platform,
but
also
the
applications
should
take
it
seriously
to
design
for
failure.
Failures
will
happen
once
in
a
while
and
over
and
over
again,
there's
no
perfectly
safe
and
perfectly
reliable
system.
So
if
your
design
respects
that
the
impact
is
probably
way
lower
than
it
could
be,
what
helped
us
in
many
places
was
having
metrics.
So
we
could
do
post
mortems,
but
of
course
we
didn't
know
initially
what
all
the
metrics
where
we
needed.
A
A
Also,
whenever
something
goes
wrong,
it's
not
unlikely
that
it's
not
only
in
either
the
infrastructure
or
the
application
more
often
than
not.
It
shows
that
there's
an
interaction
or
interference
between
both,
and
so
whenever
a
failure
occurs.
You
should
keep
in
mind
to
look
at
both
at
both
sides
and
see
if
there's
a
an
interconnection
between
them
and
what
helped
us
in
many
places
is
to
have
really
well
designed
proper
liveness
and
readiness
checks.
A
So
if
these
are
well
designed,
they
can
quickly
point
to
problems
and
quickly
help
you
to
find
the
root
cause
of
whatever
goes
wrong
so
and
at
the
very
end,
of
course,
we
are
not
the
only
people
experienced
saying
failures
and
I
would
like
to
point
you
for
more
failure
stories
to
the
collection
maintained
by
hanging
jakobs
at
the
address
shown
here.
So
thanks
for
attending
this
presentation,.