►
From YouTube: K8s in the Datacenter: Integrating with Preexisting Bare Metal Environments - Max Stritzinger
Description
Don’t miss out! Join us at our upcoming events: EnvoyCon Virtual on October 15 and KubeCon + CloudNativeCon North America 2020 Virtual from November 17-20. Learn more at https://kubecon.io. The conferences feature presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
K8s in the Datacenter: Integrating with Preexisting Bare Metal Environments - Max Stritzinger, Bloomberg
As Kubernetes adoption grows, it will need to support the ability to run in a variety of host networking configurations to conform to the wide variety of datacenter architectures that exist in bare metal environments. This talk discusses our path of adapting Kubernetes components such as kube-proxy and the Calico networking project to operate in the pure L3 ECMP fabric our hosts live in. We will discuss our network configuration, how Calico (and other commonly used CNI providers) provide overlay networks to clusters, the steps we took and tools we used (ebpf, ftrace, tcpdump, conntrack) to diagnose some uncommon Linux networking issues, and how we can provide network connectivity to our clusters both with and without an overlay.
https://sched.co/ZeqL
A
Hi
everyone,
my
name,
is
max
schritzinger,
I'm
an
sre
at
bloomberg
and
today
I'm
going
to
talk
about
integrating
kubernetes
networking
into
bare
metal
or
on-premise
environments.
So
just
kind
of
an
overview
of
this
talk.
The
first
thing
I'm
going
to
go
over
is
kind
of
the
the
basics
of
kubernetes
networking.
I'm
gonna
talk
about
kind
of
what
the
options
are
for
implementing
that
network
model
on-prem
or
on
bare
metal.
I'm
gonna
talk
about
our
specific
networking
environment.
So
what
our
hosts
like?
A
What
our
hosts
look
like,
what
our
network
looks
like
and
kind
of
how
that
creates
some
challenges
in
implementing
some
kubernetes
networking
and
then
hopefully,
if
I
have
time,
I'm
gonna
talk
about
some
tips
and
some
tools
that
you
might
be
able
to
use
to
debug
your
own
environments
if
you're
doing
this.
So
the
first
thing
I
want
to
do
is
I
want
to
get
on
the
same
page
around
what
the
kubernetes
networking
model
is
so
pods
are
our
schedule.
A
Units
work
they
all
get
an
ip
address
and
the
model
says,
among
other
things,
that
pods
need
to
be
able
to
communicate
with
each
other.
So
there's
no
real
spec
on
how
you
have
to
implement
that,
but
in
general
there's
kind
of
two
categories
of
implementations,
so
there
are
overlay
networks
and
then
there's
kind
of
more
of
a
flat
network
model,
and
the
distinction
between
these
two
is:
does
the
rest
of
the
network
have
knowledge
of
the
pod
id
space
of
the
pod
cider?
A
If
the
rest
rest
of
the
network
does
then
you're,
probably
running
with
a
flat
network
if
the
rest
of
the
network
doesn't
really
have
any
knowledge
of
the
pod
ip
space,
then
you're,
probably
using
an
overlay-
and
I
guess
really
the
obvious
difference
here-
is
that
you
know
if
you're
off
cluster.
If
you
can
talk
to
a
pod
ip,
then
you're
probably
running
the
flat
network,
and
if
you
can't,
then
it's
probably
an
overlay.
So
in
our
in
our
environments,
we
have
to
support
both.
A
A
We
need
encapsulation
if
we
ever
want
to
talk
to
a
pod
ip
from
a
pod
or
a
host
in
the
cluster.
So
if
we
drop
packets
onto
the
wire
with
a
destination
address
being
a
pod
ip
that
network's
just
or
the
network's
just
gonna
drop
that
traffic
on
the
ground.
Because,
again,
in
an
overlay
scenario,
the
network
doesn't
have
any
knowledge
of
the
body
b
space.
A
But
if
we
want
to
have
this
going,
we
need
kind
of
an
agent
that's
running
on
all
of
our
nodes.
That's
programming
these
routes.
That
knows,
if
a
pod
gets
launched
on
a
certain
host,
if
any
of
the
other
hosts
want
to
talk
to
that
pod
ip,
then
they
need
to
encapsulate
traffic
to
that
host.
So
that's
one
of
the
things
that
we
need
for
implementing
an
overlay
network.
A
Conversely,
flat
networks,
we
don't
need
encapsulation,
we
don't
need
source
snapping
right,
because
you
can
just
talk
directly
to
pod
ips
they're,
totally
routable
on
the
network.
What
you
do
need,
though,
is
you
still
kind
of
need,
an
agent
that
is
sharing
pod
routes?
So
again,
if
pod
gets
launched,
we
need
some
way
of
informing
the
rest
of
the
network
that
this
pod
lives
on
this
host,
and
this
is
where
the
traffic
should
go.
A
You
don't
actually
need
to
maybe
engage
your
network
team
for
these,
so
all
you
really
need
to
implement.
An
overlay
is
udp
connectivity
or
ipconnectivity,
but
I'll
put
a
little
asterisk
by
that
and
come
back
to
that
later,
you
can
reuse
the
pod
ip
space
between
clusters,
because
again
this
is
not.
These
are
not
real
ips.
You
don't
have
to
worry
about
allocating
out
of
your
organization's
range
range
or
something,
and
something
else
that
I
think
is
really
nice
about.
A
These
is
that
the
ingress
points
your
cluster
are
really
well
defined,
so
you've
got
you've,
got
node
port
services,
you've
got
load,
balancer
services,
you've
got
host
network
pods,
but
it's
it's
really
kind
of
obvious
where
traffic
can
come
into
your
cluster
and
that's
not
the
case
with
flat
networks.
Some
downsides,
though,
are
that
you
are
going
to
pay
a
little
bit
of
overhead
in
terms
of
computation
in
terms
of
space
on
the
packet
in
terms
of
encapsulation.
So
that's
not
so
great.
A
Also
if
you,
if
you
have
on
the
rest
of
your
network
source,
address
space
firewalls,
all
of
your
pod
traffic
is
going
to
have
the
ips
of
your
host,
so
all
of
your
pods
are
going
to
be
as
privileged
or
as
unprivileged
as
your
hosts
are
against
those
firewalls.
So
that
may
be
something
that
you
don't
want
or
something
that
you
do
want,
but
something
to
consider,
and
also
this
does
kind
of
complicate
debugging.
A
A
So
that's
kind
of
nice
also
pods
can
be
talked
to
you
directly
so
because
they're
routable
on
the
rest
of
the
network,
you
could
imagine
maybe
a
service
discovery
mechanism
where
we
talked
directly
to
pods.
I'm
not
sure
if
I
would
do
that,
but
something
you
could
do,
there's
also
lots
of
options
for
sort
of
stress
phase
filtering.
So
I
mentioned
before
that
with
source
snatting
everything
is
as
privileged
as
your
host,
but
in
a
flat
network
we
can
kind
of
choose.
A
A
We
also
have
this
nice
property,
where
it's
really
obvious,
where
traffic
is
coming
into
the
cluster,
so
you
might
be
exposing
certain
services
to
the
broader
network
in
a
flat
network,
or
it's
just
something
that
you
have
to
be
aware
of
and
kind
of
handle,
maybe
with
network
policy,
and
I
think
something
that
that
is
really
overlooked
is
that
ip
address
management
is
is
a
huge
issue
in
flat
networks.
So
I
I
talked
a
little
bit
about
that
here.
A
I
think
a
really
common
scenario
is
that
in
a
data
center
you
may
have
multiple
availability
zones
and
each
of
these
availability
zones
may
correspond
to
a
different
ip
range,
and
so
a
requirement
on
your
cluster
is
going
to
be
that
if
you
launch
a
pod
in
one
availability
zone,
you
need
to
make
sure
that
it
gets
an
ip
from
that
zone's
range.
And,
conversely,
you
know
if
it's
launched
in
the
other
zone,
it
needs
an
ip
from
the
other
zones
range,
and
so
this
is
something
that
you
know.
A
You
really
need
to
be
thinking
about
if
you're
going
to
go
down
the
flat
network
path,
and
it's
kind
of
for
that
reason
that
I
wouldn't
suggest
advertising
service
fips
to
the
rest
of
your
network.
So
I
know
that's
something
that
some
network
providers
will
do,
but
there's
really,
as
far
as
I
know,
no
great
way
of
getting
good
ipam
management
around
these
service
hips.
So
something
to
keep
in
mind
I
mentioned
previously
with
overlays.
A
You
may
not
have
to
talk
document
that
we're
saying
what
should
be
good,
but
I
think
you
should
also
consider
that
I,
if
you
don't,
then
your
options
for
layer,
3
or
layer,
4
ingress
are
just
node
boards,
and
that
may
not
be
enough
for
you,
so
you
might
not
be
out
of
the
woods
in
that
regard,
and
even
with
node
ports,
there's
some
downsides
right,
so
you
have
to
come.
You
have
to
grab
a
port
out
of
this
high
port
range.
Clients
still
have
to
find
out
about
your
node
ips.
A
So
something
to
consider
the
last
thing
I'll
say
is
that
by
default,
the
kubernetes
controller
manager
assumes
that
you
have
one
pod
slider,
that's
gonna
cover
your
entire
cluster
range
and
out
of
this
cider,
it's
gonna
kind
of
hand
out
a
node
sider
to
every
single
node
that
joins
your
cluster
on
a
flat
network.
I
think
this
is
probably
too
heavy-handed
to
work,
especially
if
you
ever
want
to
like
grow
your
ip
range.
If
you
want
to
just
joint
ip
range,
that's
probably
not
something!
A
So
for
us
we
use
calco
for
implementing
both
these
models.
It
does
overlays
those
flat
networks
and
it
has
really
good
support
for
ipam.
So
with
calco,
I
can
say
if
I
launch
a
host
on
a
certain
node
or
actually,
if
I
launch
a
pod
on
a
node,
I
can
look
at
the
node
label
and
use
that
node
label
to
determine
which
ipool
that
pod
gets
an
ip
from.
A
A
A
What
that
kind
of
looks
like
is
shown
in
this
diagram,
so
each
host
has
a
service
ip,
what
we
call
a
service
ip-
and
that
is
the
real
address
of
the
host-
that
that
is
the
address
that
other
hosts
or
pods
are
going
to
be
able
to
talk
to
that
host
on.
But
it's
also
got
on
the
actual
interfaces.
It's
got
two.
A
What
we
call
transit
addresses
and
these
transit
addresses
you
can
think
of
them
as
like
link
local
addresses,
they're
not
going
to
be
routable
on
the
rest
of
the
network,
and
the
only
thing
that
they're
really
used
for
is
for
advertising.
This
service
address
using
bgp
to
the
top
rack
switch.
So
we
can
talk
to
the
service
addresses
we
can't
talk
to
the
transit
addresses.
So
how
does
that
cause
issues
for
us?
Well,
one
one
thing
that
really
stands
out
is
that
you
know
when
we
look
at
github
issues
or
prs.
A
We
don't
see
that
many
people
that
are
running
into
the
same
problems
that
we've
run
into
with
our
setup,
so
we're
thinking
that
there
maybe
aren't
that
many
people
doing
this,
and
so
that's
kind
of
a
challenge.
Another
challenge
is
that,
typically,
your
networking
stack
is
going
to
be
made
up
of
these
open
source
components,
so
you
may
have
to
make
changes
to
those
open
source
components.
A
You
may
have
to
post-hoc
kind
of
figure
out
what
certain
implementation
implementation
details
are
supposed
to
do
so
why
certain
iptables
rule
exists
are
why
routing
is
set
up
in
a
certain
way,
and
maybe
that's
in
the
code.
Maybe
that's
nowhere
and
you
just
kind
of
have
to
figure
out
on
your
own,
why
that
is,
and
sometimes
people
don't
understand
your
use
case.
A
A
A
Calico
also
makes
use
of
bird,
which
turns
out
to
be
something.
That's
that's
kind
of
hard
to
deal
with,
so
they
both
are
running
in
the
root
network
name,
space
they're,
both
binding
the
same
ports
and
addresses,
and
also
because
the
bgp
spec,
the
top
rack
switch,
is
only
going
to
accept
one
peering
connection
from
our
host.
So
how
do
we
deal
with
this
for
the
same
port
and
same
address
issue?
A
You,
and
also,
of
course,
you
know,
you
can
change
the
the
configure.
The
the
bird
instance
that's
running
for
the
host
and
for
the
service
id
the
tour.
Only
accepting
one
peering
connection
is
a
little
bit
tougher
to
deal
with,
so
we
were
trying
to
figure
out.
You
know.
Should
we
even
run
two
bird
instances?
A
Maybe
we
could
you
know
bootstrap
our
host
and
then
once
kubernetes
comes
up
we
could
have
the
calico
bird
be
responsible
for
advertising,
both
kubernetes
pod
routes
and
the
service
address,
but
we
didn't
want.
We
didn't
really
want
to
do
that,
because
you
know
we
don't
really
want
to
rely
on
our
cni
provider
to
be
up
in
order
for
our
hosts
to
be
reachable
on
the
network.
A
You
can
imagine
you
know
that
that
would
be
a
tough
dependency
to
have
so,
and
you
know
in
the
other
direction
you
know
maybe
there's
some
way
we
could
have
figured
out
how
to
get
the
host
bird
to
advertise
kubernetes
routes,
but
I
think
that
was
something
that
was
going
to
be
a
little
tricky
to
figure
out.
So
what
we
ended
up
doing
is
we
ended
up
pairing
them
together,
so
calico
bird
and,
as
you
can
see
in
this
this
slide.
A
Calico
bird
is
sharing
these
kubernetes
routes
to
the
host
bird
and
the
host
birds,
kind
of
acting
as
a
route
aggregator,
combining
them
all
up
and
advertising
all
of
them
to
the
top
rack
switch.
So
that's
how
we
dealt
with
that,
but
after
after
dealing
with
that
now
we
see
that
our
hosts
kind
of
drop
on
and
off
the
network
intermittently,
and
so
we
actually
weren't
really
sure
what
was
going
on
here
and
and
to
have
an
idea.
You
kind
of
have
to
understand
how
bird
works
so
bird.
A
I
think,
like
a
lot
of
routing.
Daemons
works
like
a
similarly
to
a
kubernetes
operator
in
that
it
has
a
desired
state,
which
is
all
the
routes
that
it
knows
about,
and
it's
constantly
sinking
and
diffing
that
desired
state
against
an
actual
state.
In
this
case,
that's
going
to
be
your
routing
table,
so
it's
going
to
look
at
all
the
bird
routes
in
the
routing
table
do
a
diff
against
the
ones
that
it
expects
and
then
either
delete
or
add
routes
based
on
that.
A
So
we
ended
up
syncing
them
to
two
separate
routing
tables,
and
now
we
can
run
two
bird
instances
at
the
same
time.
So
that's
those
were
kind
of
the
bird
issues
we
faced.
The
second
really
big
issue
is
that
iptables
masquerade
is
tough
to
deal
with
in
our
environment,
so
iptables
masquerade
is
an
iptables
target.
A
That's
used
for
source
netting,
which
I
mentioned,
we
need
for
overlay
networks,
but
it's
used
in
some
other
places
as
well
and
what
it
does
is
it
determines
the
address
to
use
for
that
source
and
adding
kind
of
automatically
it
what
it
does
is.
It
looks
at
the
address
on
the
interface
that
that
traffic
is
leaving
out
of
and
it
uses
that
primary
ip
address
on
that
interface
as
the
source
address
for
the
source
netting.
A
So
this
presents
some
challenges
because
masquerade
is
used
in
a
lot
of
different
places.
It's
used
in
q,
proxy
calco,
it's
used
for
short
standing
of
pods.
This
is
for
hairpin
traffic,
and
in
most
of
these
cases
we
need
an
iptables
snap
rule
instead
and
an
snap
rule
allows
you
to
specify
a
specific
ip
address
to
use
for
that
that
source
and
adding,
but
it's
not
all
that
we
don't
always
need
an
sr.
A
So
the
tricky
part
here
is
that
you
know
we
can't
just
find
and
replace
all
the
mask
grades
and
replace
them
with
snaps,
because
there
are
certain
situations
in
which
we
do
need
that
masquerade
to
happen
typically
with
an
overlay,
so
in
calico
addressing
mask
grade
is
somewhat
easy.
So
what
we
did
was
we
modified
calico
node
to
accept
as
a
parameter
to
its
config
and
ip
address
to
be
used
for
an
snap
rule
and
we
upstream
those
changes.
A
So
if
anyone
else
is
is
running
in
a
similar
setup,
that's
something
that
you
can
use
in
coup
proxy.
The
issue
is
a
little
more
nuanced,
so
coup
proxy
uses
masquerade
for
more
more.
In
more
nuanced
cases.
Right
so
it's
used
for
hairpin
traffic
for
traffic,
that's
coming
off
cluster,
but
we
do
sometimes
need
the
masquerade,
typically
with
an
overlay
network
and
a
tunnel
device,
because
the
address
on
the
tunnel
device
is
actually
the
address
we
want
to
use.
A
So
we
kind
of
have
to
look
at
you
know
each
of
these
situations
in
coup
proxy
kind
of
on
a
case-by-case
basis.
So,
in
the
course
of
this
we
ran
into
something
that
seemed
like
a
kernel
bug.
We
were
running
in
an
overlay
environment
using
ip
and
ip
tunnels,
and
some
hosts
that
were
talking
to
a
kubernetes
service
bip
were
doing
masquerade
on
the
encapsulating
traffic,
but
only
to
some
bips
only
on
some
hosts.
A
So
it
kind
of
looks
like
a
kernel
bug
at
this
point,
but
let's
dig
in
so
in
addressing
pretty
much
any
kubernetes
networking
issue.
The
the
first
tools
I
go
to
are
ping,
dig
medcat
and
nip
route
pretty
much
so
ping
for
checking
out
routability
to
pod
ips
dig
if
there's
seems
like
there's
something
related
to
dns
netcat
for
testing
up
your
tcp
connectivity
and
then
ip
route
2..
A
So
looking
at
how
your
routing's
configured
looking
at
how
the
sockets
are,
are
listening
or
not
listening
and
we
we
ran
this
across
both
these
hosts
and
actually
we
were
reproducing
the
issue
with
netcat,
so
the
netcat
was
not
able
to
netcat
was
demonstrating
the
issue,
but
but
all
these
other
tools,
kind
of
it
appeared
that
both
hosts,
that
you
know
the
ones
that
were
working,
the
ones
that
were
not
working
kind
of
had
the
same
setup.
So
going
a
little
deeper.
You
can
use
tcv
dump,
so
tcp
dump
kind
of
lets.
A
You
treat
the
kernel
as
a
black
box.
You
can
kind
of
look
at
how
traffic
looks
like
when
it
goes
in.
You
can
look
at
how
traffic
looks
like
when
it
goes
out
and
then
from
there
you
can
kind
of
guess
what
all
these
kubernetes
components
in
the
middle
are
doing.
So
it's
really
useful
for
seeing
you
know
what
does
the
network
see
when
we
drop
traffic
on
it?
A
It's
also
really
useful
for
even
seeing
if
traffic
is
on
the
wire.
So
a
lot
of
times.
If
you
check
that
first,
if
you
check,
if
the
post
host
is
even
passing
traffic,
you
can
solve
the
issue
right
there
or
have
a
better
idea
of
where
to
start.
Looking
the
second
tool,
I'll
mention
is
iptables,
so
I
think
I
think
it
has
a
little
bit
of
a
a
bad
wrap
or
maybe
like
a
scary,
wrap,
but
I
think
it's
you
know
once
you
run
through
a
couple
of
scenarios
with
it.
A
I'll
mention
is
contract,
so
I
didn't
know
about
this
tool
when
I
started
doing
this
work
and
if
I
had
it
would
have
really
helped
me
out
which
I'll
talk
about
in
a
sec.
But
contract
is
something
that
will
show
you
all
the
connections
that
the
kernel
knows
about
and
all
the
address
translation.
That's
done
on
those
connections.
So
essentially
it's
just
a
big
table
of
connections
with
the
address
translation
that
that's
been
done.
A
So
all
those
things
were
kind
of
showing
us
the
same
behavior
or
they
were
kind
of
showing
us
what
we
expected
on
the
good
meadows.
So
again
we
don't.
We
don't
really
know
what's
happening
and
what's
causing
this
issue,
so
we
have
to
go
a
little
bit
deeper
in
this
case.
We're
not
even
debugging
the
network
anymore,
we're
kind
of
debugging
the
kernel,
and
so
the
tools
I
recommend
for
that
at
least
starting
out
are
like
perf
trace.
A
So
perf
trace
is
an
s-trace
kind
of
drop-in
replacement,
but
it
also
gives
you
in
addition
to
the
system,
call
boundary.
You
can
also
get
events
that
are
happening
inside
the
kernel.
So
there's
all
these
trace
points
within
the
kernel
that
perf
trace
is
able
to
show
you
happening.
So
we
ran
perf
trace
on
the
working
and
non-working
hosts
and
you
know,
replicate
the
issue
and
we
didn't
actually
see
a
difference
there,
but
what
it
was
really
useful
for
was
with
these
tracepoint
events.
A
That
gives
us
a
place
to
go
back
into
kernel
code
and
start
digging
into
kernel
code
and
kernel
functions.
So
I'm
obviously
not
a
kernel
expert.
But
using
these
trace
points
I
can
get
an
idea
of
you
know
what
are
the
functions
involved
in
this,
and
and
where
can
I
start
looking
in
using
that?
I
can
start
using
evpf,
so
evpf
is
really
hot,
really
cool
right
now
for
cni
implementations,
but
you
can
also
use
it
to
inspect
functions
or
inspect
arguments
to
kernel
functions
on
basically
stock
systems.
A
The
last
one
I'll
mention
is
f
trace.
So,
like
evpf,
you
kind
of
need
an
idea
of
where
you
want
to
start
looking
first,
but
it's
going
to
show
you
the
entire
call
stack
for
a
process
that
is
using
kernel
code
and
it'll.
Also,
you
know,
if
you
need
it
can
show
you
every
kernel
function
that
runs
on
a
given
cpu
core.
So
again,
you
kind
of
have
to
have
an
idea
of
where
you
want
to
be
looking,
but
if
you
do
then
ebpf
and
f
trace
are
really
useful
too.
A
So
what
was
our
issue
so
to
understand
kind
of?
What's
going
on
here?
I
need
to
mention
that
something
that
may
not
know
is
that
in
ip
tables
the
nat
table
is
only
processed.
Once
per
connection,
so
I
mentioned
contract
is
the
table
that
has
all
these
connections
in
it,
and
that
is
only
done
once
so.
Every
other
connection
that
or
every
other
packet
that
is
in
that
connection,
is
going
to
use
that
same
entry
in
the
contract
table.
A
So
coup
proxy
is
using
iptable's
mark
to
select
packets
for
masquerading.
So
when
it
does
destination
that,
if
that
packet
is
going
to
need
a
source
out
as
well
it'll
mark
that
packet
and
using
evpf,
we
would
able
to
see
that
this
mark
persists
through
encapsulation.
So
if,
if
we
had
an
encapsulated
connection
that
is
marked
for
masquerade,
then
the
encapsulating
connection
is
also
going
to
be
marked
for
masquerade.
So
this
kind
of
explains
why,
on
encapsulating
packet,
we
were
seeing
the
masquerade
as
well.
A
What
it
doesn't
explain
is
why
some
hosts
were
good,
why
the
problem
would
kind
of
come
and
go
and
why
you
know
why
would
you
see
this
on
some
hosts
and
not
others?
So
the
reason
for
that
is
that
the
encapsulating
protocol
is
ipip,
which
is
a
stateless
protocol.
So
every
subsequent
encapsulated
connection
that
is
going
to
be
used
over
this
tunnel
is
going
to
use
the
same
encapsulating
connection
in
contract.
A
Conversely,
if
the
first
encapsulated
traffic
that
passes
through
that
tunnel
is
not
marked
for
masquerade,
then,
when
that
encapsulating
connection
gets
created,
it's
not
going
to
have
the
mask
rate
on
it,
and
so
any
subsequent
encapsulated
traffic
that
goes
to
that
tunnel
again,
even
if
it's
on
a
separate
connection
is
not
going
to
be
marked
for
masquerade,
so
that
ended
up
being
our
big
issue
and
yeah.
That's
that's
pretty
much
my
talk,
I
thank
you
for
listening
and-
and
I
hope
that
you
got
something
useful
out
of
this.
Bloomberg-
is
hiring
sres.