►
From YouTube: Network Machinery: A United-Front for Network Troubleshooting with CRDs - Adel Zaalouk, SAP
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Network Machinery: A United-Front for Network Troubleshooting with CRDs - Adel Zaalouk, SAP
The current state of network troubleshooting in Kubernetes is complicated. The knowledge of how to troubleshoot Kubernetes networking is scattered all over the place either in the heads of highly skilled network crafts-men or as a tool that has its own learning curve and usually forged at a time of disaster in a fire-and-forget fashion. The goal of this talk is to propose and introduce a rather simpler way of pooling this common knowledge and tooling together into a well-defined, consistent and community-accepted set of resources that are available to the average Kubernetes user. These set of resources can be divided in NetworkingPerformanceTest, NetworkingDebuggingTest, NetworkSimulationTest all under the NetworkMachinery group. Finally, the resources might utilize common network knowledge and even some dark-art tooling (e.g. OpenFlow, OVS, OVN) to provide more insight on the network
https://sched.co/MPZw
A
Good
afternoon
everyone,
my
name,
is
Alan
zaloog
and
I'm.
A
senior
software
engineer,
an
essay
P.
Welcome
to
my
talk.
So
the
title
for
the
talk
today
is
network
machinery
a
united
front
for
an
airport
troubleshooting
with
CR
these
okay.
So
let
me
go
through
the
agenda
for
today.
So
basically,
what
we're
gonna
do
today
is
we're
gonna
discuss
the
state
of
network
troubleshooting
in
kubernetes,
we're
gonna
also
go
through
CRTs
and
why
they
are
useful
or
what
can
they
be
used
for?
A
Can
they
be
used
for
networking
and
we're
going
to
go
through
also
a
couple
of
use
cases
that
the
first
use
case
is
network
for
each
ability
and
traffic
shaping
and
the
second
one
is
that
communities
Sdn
and
we're
also
going
to
give
two
demos
so
what's
happening.
So
the
network
landscape
of
committees
is
quite
complex.
There's
so
many
things
happening
so
many
puzzle
pieces.
So
there
is
DNS
core
DNS
was
lots
of
plugins,
the
kubernetes
plug-in
cache
cube.
Dns
was
DNS
mass
DNS,
metrics
and
cubed
DNS,
and
there's
also
node
local
caching.
A
I
mean
there's
also
no
local
caching,
agent
debugging,
there's
ingress.
There's
services
there's
Q
proxy,
with
IP
v
s,
IP
tables,
there's
also
a
lot
of
components
that
install
IP
table
that
we
need
to
debug.
There's
Network
policies,
the
cni
each
of
the
cni
needs
a
different
debugging
mentality
and
there's
also
clustered
for
a
cloud
provider.
Networking,
that's
that's
just
the
present,
and
then
there
comes
the
future
and
in
the
future,
you're
you're
dealing
with
ipv6,
actually
not
that
far
of
a
future.
A
If
it's
not
pretty
here
already
and
there's
winter
windows
networking-
and
there
is
a
multi
cluster
networking
and
there's
service
meshes,
there
is
East
here,
there's
the
network
service
mesh
and
so
much
things
are
going
on.
So,
if
you
get
some
some
part
wrong,
you
basically
have
a
functioning
cluster
cool.
So
I
would
like
to
go
through
what
two
of
the
previous
cube
contacts.
They
called
my
attention.
The
first
one
was
titled
black
holes
and
wormholes,
basically
understanding
and
troubleshooting
the
magic
of
kübra
neatest
networking.
A
Basically,
where
you
have
conflicting
IP
tables,
because
you
have
many
components
and
communities
that
does
install
IP
tables
and
if
two
of
components
and
installed
an
IP
table
rule
that
matches
the
same
packet,
then
the
packet
might
go
to
where
it
should
not
go
and
you
might
have
back
packet
loss
and
they
also
recommended
that
you
should
take
a
TCP
dump
on
the
source,
pod
source
node,
this
nation
pot
and
destination
node.
And
this
is
all
nice.
A
And
this
is
not
just
the
two
talks.
There
is
a
lot
of
other
blog
posts
and
talks
about
troubleshooting,
core
Anita's
networking.
But
what's
the
problem,
the
problem
is
that
there
is
no
common
place.
There
is
no
common
api's,
so
there's
the
great
tools
that
is
being
implemented
for
troubleshooting
networking
just
died
out,
so
that
question
is
how
to
pull
in
this
knowledge
in
kubernetes
in
a
standard
way
enter
CR
DS.
So
if
you
were
in
a
previous
coop
con
in
North
America,
there
was
a
great
talk
given
by
Tim,
Hawking
and
I
hope.
A
He
doesn't
get
angry.
That
I
showed
his
his
github
logo
here,
but
he
talks
about
how
see
are
these
basically
went
through
an
evolutional
cycle
from
being
just
third-party
resources
and
kubernetes
to
going
through
CR
these
which
are
more
mature?
So
you
have
you,
get
the
API
server
and
storage
for
free.
You
also.
He
also
talks
about
the
birth
of
the
operator
pattern,
which
is
basically
a
declarative
way
of
specifying
an
operator
pattern
which
is
basically
a
way
to
encapsulate
domain
knowledge,
whether
it
is
databases,
storage
or
basically
anything.
A
It
could
be
also
networking
right
and
then
there's
also
the
the
native
feeling
that
you
get
as
if
you're
defining
the
declarative,
the
ePub.
You
also
get
with
the
CR
D,
there's
also
add
mesh
and
web
hooks
and
mutation
web
hooks
right
so
but
CR
these
are
not
just
for
add-ons.
They
are
also
for
networking.
So
there's
a
lot
of
networking
plugins
already
that
make
use
of
the
concept
of
custom
resource
definitions
such
as
Malta's.
They
define
their
own
network,
attachment
definition
and
CNI
genie
and
there's
also
calico.
A
That
comes
also
was
a
pretty
much
large
set
of
CR
DS
and
there's
also
network
policy,
which
is
not
itself
in
a
CD,
but
it's
also
falling
into
the
context
of
a
networking,
and
there
is
also
a
CEO
that
comes
with
a
bunch
of
other
CDs
and
there
comes
network
service
meshes
and
much
more,
but
there's
none
for
for
troubleshooting
networking.
So
if
you
want
to
troubleshoot
networking,
you
have
to
do
it
manually,
so
why
not
introduce
another
set
that
would
make
it
easier.
A
The
life
of
people
or
developers
easier
to
troubleshoot,
right
and
here
comes
network
machinery.
So
the
idea
of
network
machinery
is
basically
to
utilize
the
cost,
the
the
the
idea
of
customer
resource
definition
to
build
network
troubleshooting
operators,
it's
very
familiar
and
widely
accepted
by
the
community.
There
are
many
frameworks
available
out.
There
think
controller,
runtime
things
I
think
of
razor
framework.
A
These
are
some
of
them
as
it's
currently
in
POC,
so
we
have
network
traffic
shaping
CRT,
which
allows
you
to
shape
the
traffic
by
specifying
a
pod
and
an
interface.
So
you
can
inject
loss
for
delay.
We're
gonna
also
talk
about
it
later
and
there's
also
the
network
module
validator.
So
we
talked
about
the
in
the
previous
cube
contacts
where
there
were
the
problem
where
certain
modules
were
disabled.
A
You
can
also
do
so
and
also,
if
you
want
to
generate
traffic
across
the
deep
parts
to
also
measure
the
performance
of
the
cni
that
you
using
underneath
of
all
these.
So
we're
going
to
talk
about
network
traffic,
shaping
we're
going
to
talk
about
network
connectivity,
testing
and
we're
gonna.
Also
talk
about
the
network,
monitor,
control
and
notification,
so
the
first,
the
first
use
case
is
basically
reach
ability
and
traffic
shaping.
So
you
have.
A
It
would
be
nice
right
if
you,
if
you
would
specify
that
you
have
these
sources
in
your
cluster
and
you
would
like
to
a
layer,
three
check
a
certain
destination.
The
destination
can
be
an
external
IP,
it
can
be
a
pod
IP,
it
can
be
a
pod
name
and
it
can
be
a
service.
Now
you
might
be
thinking.
The
service
does
not
have
an
actual
IP.
It
has
a
virtual
IP,
that's
correct,
but
the
service
also
has
endpoints.
A
So
if
I
am
to
ping
a
service
by
specifying
the
name,
then
I
can
just
say:
hey
ping,
this
service,
and
then
the
operator
will
just
fetch
out
the
service
end
points
and
then
ping,
the
endpoint
and
the
second
use
case.
If
you
say
that
you
have
a
you
want
to
layer
for
connectivity,
checked
right.
So
in
case
you
have
network
policies,
it
would
be
nice
to
say
to
see
if
your
network
policy
is
actually
functioning
or
after
you
install
the
network
policy.
A
There's
also
the
case
where
you
want
to
troubleshoot
services,
so
queue
proxy
uses,
IP
tables
to
map
to
do
the
netting
between
the
virtual
IP
of
the
service
to
the
destination
pubs
right,
and
it
would
be
nice
to
to
be
able
to
layer
for
check
the
service,
the
server's
IP
and
the
VIP,
and
then
check
also
the
a
back-end
endpoints.
So
if
the
service
IP
and
the
web
and
the
port
doesn't
work,
this
means
that
something
is
really
wrong
with
the
mapping
in
the
IP
tables.
A
It
will
be
also
nice
to
do
traffic
shaping.
So
sometimes
you
want
to
were
test
or
chaos
test
your
network.
So
let's
say
you
have
an
application
that
runs
HTTP
that
has
some
timeout
I,
won't
how
this
application
will
behave
after
injects,
a
certain
loss
or
a
certain
delay
right.
So
with
this
traffic
shaper
I
I
can
just
choose
to
inject
loss
or
delay
on
specific
interfaces.
A
All
right,
I
think
this
is
the
right
cluster.
So
we
have
here.
We
have
a
three
pots
in
a
deployment
and
we
also
have
two
separate
pots,
and
then
we
have
a
service
called
demo
cube
con,
which
we
will
target
and
let's
look
also
the
CR
DS.
So
we
define
network
connectivity
tests
here
these
network
monitor
and
network
notifications
and
network
traffic
shaper
CR.
These
will
forget
what
we're
gonna
make
use
today
off.
Is
the
network
connectivity
testing?
So
what
resources
do
we
have?
We
have
the
network
connectivity
test
resource
and
I'm
gonna.
A
Look
at
the
layer,
three
one
first,
so
here
what
I'm
just
saying
is
I'm
saying
here's
my
source
I
want
to
be
able
from
the
API
server
to
reach
these
destinations.
Basically,
so
I
have
the
specified
the
kind
of
the
destination
that
I
have
and
I
say.
The
kind
of
spot
and
I
give
the
name
in
the
namespace
and
I
also
have
a
kind
of
IP,
so
I
give
it.
The
IP
is
in
this
case
external,
it
could
be.
A
And
I
can
see,
there
is
also
let's
let's
go
through
the
result
quickly,
so,
as
I
showed
already
in
the
source,
this
is
clear
and
then
in
the
destination
I
said
the
result
is
first
of
all,
there's
the
IP
endpoint
result,
which
is
just
an
IP
endpoint.
It
just
told
you
that
it
was
able
to
successfully
paying
eight
eight
eight,
which
is
a
Google
DNS,
and
it
gave
you
also
some
statistics
about
the
the
ping,
and
then
it
goes
and
fetches
the
pods.
A
It
shows
you
I
didn't
specify
the
IP
of
the
pot,
but
the
operator
fetched
the
IP
of
the
pod
and
pink
all
the
pod
that
had
a
certain
service
selector,
so
I
was
able
to
to
ping
these
these
pots
successfully,
and
then
the
service
endpoint
was
also
working
so
either
the
previous
ones
are
were
the
ones
that
the
separate
parts
and
the
the
service
endpoints
are
the
the
actual
posit
that
are
backed
backing
up
this
service.
This.
This
worked.
Fine,
if
I
want
to,
let's
say.
A
It
won't
work
because
there
isn't
a
web
hook
installed
to
this.
Allow
me
because
layer
4
needs
to
have
4
it's
defined,
but
I
don't
have
pores
defined
so
the
web
hook.
This
allowed
me
from
really
modifying
this
object.
It
would
be
the
same
if
I
create
the
object.
The
same
would
happen.
It
would
disallow
me
from
creating
the
object
say
if
I
look
at
the
object
again.
A
A
What
happened
is
I
get
the
result
for
the
direct
service
endpoint,
so
I
made
the
connectivity
check
or
the
port
check
on
the
virtual
IP
of
the
service,
and
that
worked
and
then
I
made
also
the
connectivity
check
on
the
back-end
endpoints
and
on
the
endpoints
that
backing
I
that
are
backing
up
this
service
and
they
also
work
and
the
nice
thing
that
I
can
also
do
is
I
can
inject
delay
or
I
can
inject
loss.
So,
let's
say
I
want
to
inject
some
loss
in
certain
parts.
A
A
A
A
So
that
was
fine
for
just
the
basic
testing
right.
This
is
the
first
line
of
defense
layer
for
layer,
3
connectivity
checks,
fine,
but
what,
if
I
need
more
details
of
the
network?
I
want
to
network
monitor
my
network
and
I
also
want
to
be
able
to
control
my
network.
So
basically,
I
want
to
see
what's
happening
and
take
actions
to
do
right
enters
that's
the
end,
so
Sdn
open
flow
and
s
flow
right,
Sdn.
Basically,
we
use
this
term
currently
for
anything
that
is
making
use
of
networking
and
software
at
the
same
time.
A
But
the
original
definition
of
Sdn
came
by
making
the
control
plane
and
the
data
plane
of
a
network
device
set
two
separate
things.
So
the
separation
of
the
control
plane
and
data
plane,
basically
allowing
the
logic
to
run
somewhere
else,
not
in
the
device
itself,
and
this
gives
the
user
to
program
their
own
logics
right
and
then
I
could
program
routing
protocol
on
my
own
and
then
basically
programmed
into
the
network,
and
then
the
network
would
basically
just
use
my
protocol
throughout
the
packets.
A
This
was
an
early
effort
for
programmable
networks
and
to
make
this
happen,
basically
a
protocol
called
open
flow
was
introduced
and
open
flow
is
nothing
but
a
way
to
specify
how
the
traffic
will
will
will
be
redirected
and
gives
you
a
customization
on
top
of
what
what
the
actual
networking
does
how
it
works
is.
Basically,
you
have
a
flow
Keys.
A
You
can
match
traffic
based
on
source
destination,
source,
port
source,
IP
destination,
IP,
sorry,
source,
IP
destination,
IP
source
port,
this
port
destination
port,
and
you
can
also
do
stuff
like
matching
on
Ethernet
type
QoS,
and
then
once
the
traffic
is
matched
you
can
take
actions.
You
can
program
the
actions
that
can
be
taken.
You
can
take
action
to
forward
traffic
to
a
controller
where,
if
the
traffic
is
not
known
and
the
controller
can
then
program,
other
rules
to
either
drop
the
packet.
A
So
what
is
our
goal?
Our
goal
is
to
again
monitor
and
control
the
network
right
so
for
controlling
the
network.
We
already
talks
about
open
flow,
but
we
also
want
the
monitoring
side
and
the
monitoring
side
can
be.
It
can
be
achieved
by
many
ways.
For
example,
you
could
use
s
flow
for
sampling
traffic.
You
could
use
net
flow,
there's
many
other
tools,
but
for
this
presentation,
we're
going
to
give
an
example
based
on
s
flow.
What
we're
trying
to
build
is
basically
a
control
system.
A
You
have
the
desired
and
the
actual
result
and
the
desired
result
is,
is
having
to
monitor
or
having
a
way
to
monitor
and
control
your
traffic,
and
you
want
to
apply.
For
example,
if
I
see
something
wrong
in
the
network,
I
want
to
apply
some
rules,
and
these
rules
can
be
basically
scaling
up
a
pod,
creating
new
services
or
completely
blocking
via
open
flow
and
the
traffic
coming
to
this
or
from
this
pod
or
to
this
one.
How
are
we
going
to
do
this
in
the
context
of
containers?
A
There
is
a
project
called
open,
V
switch
and
open
V
switch
is
basically
Linux
bridges,
plus
plus
it
allows
you
to
do
stuff
like
expose
your
interfaces
for
monitoring
via
s
flow
or
span
ports.
It
also
allows
you
to
program
your
your
your
Linux
bridges
or
use
open
flow
to
set
flow
rules,
so
it
gives
you
a
lot
of
customizability
how
it
works
in
details
just
for
a
quick
overview.
The
first
packet
comes
in
I,
don't
know
how
to
forward
it.
The
kernel
data
pass
doesn't
know
how
to
forward
the
packet.
A
It
sends
it
to
the
user
space
where
it
gets
sent
to
the
controller.
The
controller
has
certain
logic
of
how
this
package
should
be
forwarded
then,
and
flow
rules
programmed
on
the
on
the
V
switch
or
the
kernel
data
pass,
and
then
the
traffic,
the
second
packet
basically
comes
in
and
B
gets
forwarded
right
away
for
network
machinery.
These
are
the
ingredients
that
we
require
to
be
able
to
build
a
prototype.
First
of
all,
you
need
a
CNI
plugin
that
is
based
on
open,
V
switch
right.
A
Second,
you
need
to
be
able
to
configure
and
install
S
+
OS
flow
needs
to
be
running
and
you
need
to
configure
OVS
or
often
be
switch
with
s
flow
parameters.
And
to
do
this
we
have
a
component
called
the
S
flow
of
es
installer
and
basically,
a
config
map
that
has
the
configuration
parameters
for
s
flow
and
the
diamond
set.
That
runs
s
flow
on
all
of
the
nodes
that
have
obvious
and
the
third
thing
or-
and
the
final
thing
we
need
are
our
network
operators.
A
A
Alright,
so
let's
have
a
look
at
the
environment
quite
fast
sobieszczyk.
Basically,
this
is
this
is
all
the
interfaces
you
can
see
that
there
is
OBS
installed,
and
these
are
the
bridges
that
this
is
the
bridge
that
is
on
the
obvious
system.
So
if
I
would
do
the
SVC
TL
show,
it
would
show
me
that
this
OBS
has
these
bridges.
It
has
only
one
bridge
now
and
all
these
interfaces
are
basically
my
pots.
A
Let's
apply
first,
the
config,
the
coming
man
before
we
apply.
Maybe
we
can
have
a
look
at
it
yeah.
So
basically,
what
I'm
configuring
here
or
specifying
in
the
config
map
is
basically
the
bridge
name,
the
collector
IP,
and
in
that
case
it's
s
flow.
So
I'm,
basically
saying
s
flow
lives
in
on
that
IP
and
we
have
the
collector
port
and
it
listens
on
this
port
and
we
also
have
the
agent
IP.
This
is
basically
the
bridge
that
will
forward
the
traffic
to
the
collector,
which
is
s
flow.
A
A
A
A
A
Let's
apply
our
network
monitor
before
we
apply
probably
a
good
idea
to
have
a
look
at
it
as
well
all
right.
So
what
we're
doing
here
is
we're
specifying
we're
saying
our
collector
lived
the
lives
in
this
IP
on
this
port
and
we're
defining
certain
flows,
we're
defining
a
TCP
flow
on
an
ICP
flow
and
the
TCP
flow
basically
defines
the
IP
source,
IP
destinations,
TCP,
source
port
and
please
be
destination
port,
and
this
is
basically
the
flow
keys
that
I
want
to
see.
A
Traffic
from
in
this
case,
I
will
see
these
these
keys
the
traffic
from
these
keys.
What,
if
I,
want
to
see
ICMP
flow
I
want
to
see
to
see
the
just
IP
traffic,
then
I
just
say:
I
wouldn't
need
to
define
another
flow
which
maps
to
the
IP
source
and
IP
destination
and
I
also
have
the
chance
to
define
or
customize
what
I
want
to
see.
What
I
want
to
event.
A
Basically,
so
I'm
saying
that
if
there
is
more
than
100
frames
per
second
I
would
like
to
get
notified,
and
this
is
basically
the
threshold
section.
The
last
section
is
the
events
config
where
I
say:
okay
I
only
want
maximum
of
five
max
five
maximum
of
five
events
and
a
timeout
of
60
seconds.
If
the
event
is
generated,
so
it
will
be
deleted
after
60
seconds.
I
apply
this
all.
A
A
I
can
judge
it
by
the
ports,
4
4,
3
and
there's
also
other
things
that
are
going
on,
and
I
also
can
see
that
the
thresholds
that
I
defined
are
there
so
I
want
if
I
know
that
there's
100
frames
per
se
in
my
cluster
hundred
frames
per
second
I
would
like
to
get
notified
by
an
event.
So
the
next
thing
that
I
want
to
see
is
I
want
to
get
Network
notifications
for
things
that
are
happening.
A
A
Okay,
things
are
happening
now
and
I
can
see
that
the
traffic
is
being
generated
like
crazy
from
my
source
to
destination
and
there's
already
events
for
that,
because
I
set
the
threshold
to
30
sister.
She
is
a
very
low
value,
but
I
can
also
set
slower
thresholds
for
that
and
I
can
also
see
that
there
is.
A
A
Okay,
so
basically,
the
network
notification
is
telling
me
there
is
an
event
that
has
happened
from
a
certain
port
right
and
there
was
a
threshold
that
was
defined
of
30,
and
basically
it
tells
me
where
exactly
the
traffic
is
coming
from.
So
what
I
can
do
now
is
I
can
just
block
traffic.
Okay,
I
can
do
anything,
I
want
and
in
fact,
I
have
a
controller.
This
is
the
network
control
controller.
That
is
basically
telling
me.
Hey.
A
A
A
Can
see
that
number
one
was
my
host
local
or
my
moment
yeah,
so
this
is
basically
the
mapping
between
the
interfaces
end
and
the
interface
index
that
comes
from
SNMP
cool,
so
we
managed
so
in
this
demo
we
managed
to
be
able
to
build
the
whole
control
loop.
We
have
network
monitoring,
we
have
network
control
in
a
kubernetes
context,
so
you
can
take
these
events.
A
Also,
if
I
want
to
network
performance
test,
my
my
cluster
I
can
specify
here's.
My
clients
here
are
my
targets.
Please
run
an
I
breath
test
or
any
other
kind
of
test,
and
this
is
basically
configurable.
So
the
status
of
this
restore
is
basically
like
a
provider
spec
where
you
can
it's
a
pulley
for
polymorphic
status,
where
it
can
change,
so
you
can
define
your
own
status
in
a
very
dynamic
fashion
or
even
define
more
more,
more
CR
DS,
which
are
more
specialized
to
the
use
case.
What
is
the
summary?
A
So
there
are
many
tools
and
patterns,
but
there's
no
Oh,
API
or
common
access
point
for
troubleshooting
see.
Are
these
basically
are
very
nice,
well-known
community
standard
that
enable
us
to
describe
and
harmonize
our
api's
and
network
machinery?
Basically
utilizes
c
are
used
for
network
troubleshooting.
It's
the
first
line
of
defense
to
be
able
to
do
reach
ability
and
performance
checks,
even
inject
loss
or
delay
by
a
traffic
shaping
series,
there's
also
the
second
line
of
defense,
which
gives
you
more
network
visibility
and
with
management
and
control
over
a
cluster.
A
A
If
you
feel
like
it,
if
you
need
to
and
what
this
is,
this
is
basically
my
my
Twitter
account
and
my
github
handle
check
out
network
machinery
on
github,
and
if
you
have
any
questions,
just
let
me
know
also
check
out
Gardner,
which
is
basically
a
tool
that
allows
you
to
deploy
communities,
cluster
and
different
cloud
providers
in
a
homogeneous
and
consistent
manner.
Thank
you.
Everyone.