►
From YouTube: k8snetlook – Root-Causing k8s Network Problems in an Automated Way - Arun Sriraman, VMware
Description
Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
k8snetlook – Root-Causing k8s Network Problems in an Automated Way - Arun Sriraman, VMware
More and more applications in production call Kubernetes their home. As the density of workloads on a Kubernetes cluster increases, so does the probability of downtime due to an underlying network issue. Some of the most common quibbles we hear from users: I can’t connect to my service A running within a K8s cluster or my service A seems to not be responding some % of the time. What do you do in these situations; Do you call the network gurus to help out, or kubectl delete the application and let Kubernetes self heal? What if you could identify an issue without needing to master the internals of K8s Networking? Arun will go over the various issues seen in the data plane, from dns, external traffic to internal app-to-app communication, and then discuss open source tools available to identify these issues in real time. We will look at k8snetlook - a simple open source tool that empowers every Kubenretes user; expert or otherwise, to root cause these issues in an automated way.
A
Hello,
everyone
good
afternoon
welcome
to
this
talk
on
root,
causing
kubernetes
networking
problems
in
an
automated
way.
I
wish
this
session
was
in
person
and
interactive.
I
still
would
like
to
have
this
as
interactive
as
possible,
so
keep
your
questions
coming
in
on
chat
and
I'll
answer
them
as
we
go.
A
I
will
also
locate
some
time
at
the
end
as
much
as
possible
for
q,
a
I've
debugged
network
problems
in
over
100
customer
cost
clusters
in
production.
So
whatever
you
hear
today,
as
well
as
what
you'll
see
from
a
tool
kds
network
perspective,
it
is
from
my
that
experience
all
right.
So
what
is
this
talk
all
about
right?
A
Have
you
been
in
a
situation
where
everything
looks?
Okay,
all
of
the
points
are
running,
but
then
your
app
just
won't
communicate
or
it
works
most
of
the
times,
but
does
not
work
sometime
now.
If
you
look
at
the
the
cartoons
there,
that
could
turn
from
an
exciting
happy
person
running
kubernetes
and
happen
within
kubernetes
to
a
very
frustrated
person,
especially
when
it
comes
to
a
network
networking
problem.
A
In
that
sense,
I
will
be
talking
about
kubernetes
networking.
In
brief,
it
is
a
big
topic
in
itself
and
a
whole
talk,
so
think
of
it
as
a
crash
course.
We'll
then
look
at
the
problems
that
you
normally
see
with
kubernetes
networking
and
running
pods
within
kubernetes,
and
how
to
classify
these
issues
and
then
we'll
look
at
what
people
do
to
resolve
these
issues.
A
It's
very
interesting
what
people
do
and
that
slide
is
exciting
after
that,
the
important
part
we'll
look
at
some
open
source
tools
and
how
we
can
use
them
to
debug.
Any
networking
issue
and
how
to
go
about
that
debugging
process
and
then
we'll
look
at
kts
network
in
detail,
give
you
an
inflection
on
it
and
a
quick
demo
on
how
to
work
with
it
all
right
without
further
ado.
Let's
get
started
so
kubernetes
networking
to
keep
it
simple
right.
A
The
address
that
you
see
is
the
same
address
that
another
container
outside
of
within
that
cluster
would
use.
The
other
is
that
all
nodes
are
able
to
reach
these
parts
or
containers
without
any
net.
Nyc
version
right
so
without
nad
is
a,
and
it
is
an
important
construct
here
now
we'll
to
understand
network
problems
and
how
to
debug
it's
beneficial
to
know
what
the
components
are
that
play
into
kubernetes
networking.
So,
let's
quick
quickly
take
a
look
at
what
these
components
are,
which
are
responsible
for
everything.
A
The
first
one
is
the
cni
back
end.
Most
of
you
are
would
be
familiar
with
it.
You
either
run
calico
or
flannel
or
celium
and
shia.
There
are
so
many
other
cni
backends
out
there.
That's
the
component
that
provides
pod
connectivity.
It
connects
multiple
parts
together,
either
on
the
same
host
or
across
hosts.
It
also
does
nothing
so
if
a
part
needs
to
reach
the
internet,
it
kind
of
connects
that
part
over
to
the
internet.
A
A
To
give
you
that
functionality
of
internal
load,
balancing
or
e-stress
load
balancing,
and
then
there
is
the
load
balancing
implementation.
If
you're
running
a
cluster
cloud,
you
have
aws
efb,
for
example,
metal
lb
is
an
open
source
project
that
is
used
in
production
to
deploy
a
software
headload
balancer
within
the
kubernetes
cluster.
A
Then
there
is
ingress.
If
you
want
to
be
able
to
connect
multiple
services
and
route
through
a
single
point,
single
entry
point
either
by
using
url
paths
or
ports,
you
can
do
l4
english,
l7
english.
There
are
various
examples
there
again,
as
you
can
see
as
we
go
down,
this
list
complexity
increases.
So
that
is
service
mesh,
which
is
a
lot
more
complex.
You
have
hto
link
rd
console,
connect
and
the
whole
kubernetes
networking
stack
right,
ip
tables,
ipvs
linux,
networking
stack
and
yeah.
A
So
let's
look
at
how
communication
works
across
the
cluster
say,
for
example,
two
parts
in
an
east-west
manner
and
say:
let's
say
you
have
two
parts:
one
is
the
blog
app.
The
other
is
an
index
app.
You
have
two
copies
of
these
they're
running
on
two
different
hosts
in
your
kubernetes
cluster.
Now,
if
pod,
a
or
the
blog
app
wants
to
talk
to
part
b
or
the
index
app
through
the
service,
assuming
there's
a
service,
that's
backing
this
app
or
index
b
or
part
b.
A
How
would
it
look
like
right
so
it
for
the
first
call
would
be
to
core
dns
to
get
the
dns
resolution.
If
you're
doing
a
http
get
using
the
service
discovery
dns
address,
then
you'd
need
a
cluster
id
once
you
get
that
cluster
ip
you
reach
out
to
or
on
the
packet
goes
to
iptables
to
look
up
which
part
it
needs
to
get
to
so
it
it
hits
the
ip
tables
with
the
service
id.
A
It
translates
over
to
a
pod
ip
destination
port
ip,
and
then
it
goes
to
the
particular
point
and
on
on
on
the
way
back
contract
and
keeps
track
of
these
connections
and
replaces
the
source
destination
ip
accordingly
and
when
it
comes
to
north
south.
You
have
end
points:
node,
ports
and
load
balancers.
A
A
So
when
a
client
tries
to
connect
your
app,
they
hit
the
load
balancer
ip,
and
then
it
translates
over
selects
one
of
the
hosts.
That's
part
of
your
kubernetes
cluster,
either
where
the
port
is
running
or
any
of
the
other
worker
hosts.
It
goes
to
that
host
through
the
node
port
or
a
large
high
number
port
and
from
there
on
goes
through
cluster
ip
and
then
to
the
pond.
A
A
So
the
way
I
see
it
when
you
have
a
networking
issue,
it's
good
to
think
about
it
and
do
two
terms
or
two
ways.
One
is
think
about
the
the
impact
right.
So
what
kind
of
an
outage
is
it?
Is
it
a
complete
service
impact
or
is
it
partial
outage
or
is
it
negated
performance
so
plays
the
problem
that
you
see
in
one
of
these
three
buckets?
A
A
It's
a
larger
problem,
whereas
if
it's
a
partial
outage,
that
means
maybe
few
parts
in
your
of
your
service
isn't
working.
That's
that's
a
lot
more
localized,
that's
mainly
your
app
or
even
if
it's
the
subsystem,
it
has
something
to
do
with
specificity
of
that
particular
one.
App
degradation
performance
is
the
most
difficult
problem
to
debug
and
fix,
as
you
can
see,
the
difficulty
of
debugging
kind
of
increases
when
you
start
moving
towards
your
right
and
debugging
performance
is
difficult.
A
We'll
see
what
tools
are
there
and
how
to
do
it,
but
it's
always
good
to
classify
your
problem
in
these
two
aspects.
One
is
the
impact,
severity
or
class
type.
I
call
it
the
other
one
is
where
this
problem
is
occurring
right
or
what
type
of
problem
it
is.
It
could
be
an
application
problem
which
means
that
your
application
isn't
working
and
what
you
could
do
there
right.
So
some
of
the
things
that
people
end
up
doing
is
look
at
logs
to
see
if
there
are
errors
in
your
app.
A
Maybe
there
was
a
code
problem,
the
process
crashed
file,
descriptors
run
out
or
if
the
app
was
just
doing
so
much
and
running
out
of
cpu
or
memory.
Now,
let's
look
at
the
other.
One
could
be
platform
components
right.
So
this
is
where
a
problem
within
the
kubernetes
subsystem
happens:
dns
load,
balancer
q,
proxy
service.
There
could
be
some
kind
of
a
problem
happening
somewhere
there,
you're
not
able
to
get
dns
access
or
load
balancer.
A
There
could
be
a
configuration
problem,
which
is
the
third
type
or
third
bucket,
which
is
very
common.
In
my
experience,
there
is
a
problem
in
mtu
settings
somewhere
across
from
your
source
to
a
destination.
That's
causing
an
issue,
or
there
are
route
routing
issues
or
there
is
an
ip
overflow.
Just
simply
that
your
part
is
not
able
to
get
an
ip
address
and
it
just
fails
coming
up
now.
How
do
people
resolve
these
right.
A
This
is
a
good
segue
to
what
do
you
do
today
without
any
of
the
debugging
tools
right?
The
first
one
is
the
big
hammer
approach.
What
that
means
is
there
are
two
ways
of
how
people
fix
it.
One
is
by
just
hammering
the
part
or
the
deployment
of
the
service
or
the
english
just
kill
it
right
and
let
kubernetes
do
the
job
we
spin
it
up
or
reschedule
it
on
a
different
node
and
everything
starts.
A
I've
also
seen
people
do
this
where
they
go
ahead
and
delete
the
node.
If
there's
a
problem
in
the
node,
with
multiple
apps
and
rather
than
fixing
it,
it
might
just
be
easier
and
quicker
to
delete
it,
especially
if
the
kubernetes
cluster
is
running
in
the
cloud
just
delete
it
and
let
aws
or
google
or
azure
kind
of
recreate
that
vm
for
you.
If
it's
in
the
scale
set
the
other
one
that
actually
works,
most
cases
is
asking
for
help.
A
So
that
happens
quite
a
bit
too
for
people
who
are
a
lot
more
excited
and
a
lot
more
exploring.
They
try
to
troubleshoot
live
setups
and
manually,
and
that's
where
today's
talk
should
help
now
troubleshooting
workflow
right.
This
is
an
interesting
topic.
So
what
I
wanted
to
do
here
was
to
put
whatever
I've
seen
out
in
the
field
with
respect
to
problems
and
how
I've
gone
about
solving
a
problem
and
how
I
built
cades
network
to
start
looking
at
it
in
an
automated
way.
A
So
the
first
thing
that
I
say
is:
if
there's
a
problem,
then
see
what
the
problem
is
right.
If,
if
a
part
is
crashing,
it's
it's
a
lot
more
obvious.
But
then,
if
everything
is
working,
but
you
still
don't
have
communication,
it's
a
lot
more
harder
to
do
so
see
if
there
are
other
apps
down
more
often
than
not.
There
are
other
apps
that
aren't
working,
which
means
that
it's
not
localized
to
your
app.
It
could
be
more
than
one
app.
A
If
it
isn't,
then,
if
it's
an
app
specific
problem,
first
would
be
to
rule
out
your
app
itself.
Look
for
logs
look
for
help
metrics
in
all
of
these
cases.
Right,
if
you
have
prometheus
the
fauna
set
up,
have
time
series
data
being
pulled
in
it's.
It's
a
lot
better.
There's
a
cube
metrics
service,
which
gives
you
node
statistics,
there's
a
node
exporter.
There
are
a
bunch
of
tools
out
there
that
allow
you
to
monitor
your
kubernetes
cluster
again.
Let
me
put
a
warning
on
that.
A
In
a
lot
of
cases,
I've
seen
network
policy
within
the
kubernetes
cluster
cause
issues.
I've
seen
firewall,
causing
issues
outside
in
your
environment,
which
means
external
traffic
does
not
come
in
or
go
out
and
then
search
or
tls,
causing
issues
which
could
be
either
mtu
or
again
firewall,
or
just
your
app
right
anyway.
Now
coming
back
to
that
flowchart,
if
other
apps
are
down,
then
look
at
your
subsystem
right
either
case.
What
we
need
to
do
is
to
identify
the
problem.
A
Type
of
the
class,
like
I
said,
is
it
partial,
complete
or
a
performance
degradation
problem
in
case
of
both
partial
and
complete
outage?
What
we
do
next
is
to
see
what
type
of
traffic
path
the
problem
lies
on,
and
here
again
we
classified
into
three
things:
one
is
east
west,
and
what
I
mean
by
east
west
is
applications
within
the
communities
cluster
talking
to
each
other
somewhere,
there's
a
problem
there
and
that's
normally
through
a
kubernetes
service.
A
You
would
first
run
something
like
thing
to
see
if
pod,
a
of
your
app
a
is
able
to
reach
part
b
of
your
b
right
through
ip
through
a
specific
container.
If
that
doesn't
work,
there's
definitely
a
problem
either
in
the
overlay
or
the
overlay,
which
is
your
cni
backend,
you're,
now
kind
of
localizing.
The
problem,
which
is
a
very
important
step
and
then
you'll
be
able
to
get
help.
A
The
second
one
is:
if
that
works,
then
look
at
dns,
try,
pinging
or
reaching
the
destination
service
through
dns
you,
you
can
use
pod
dns.
If
that
doesn't
work,
there's
probably
a
coordinate
issue
or
again
that
could
be
an
issue
or
an
overlay
issue.
If
that
works
too,
then
look
at
cluster
ip.
So
if
there's
a
service,
that's
backing
your
destination
app
then
pick
up
the
cluster
ip
for
that
particular
service
and
see,
if
you're
able
to
connect
to
that.
If
that
works
again,
if
that
doesn't
work,
it's
q
proxy.
A
If
that
works,
then
you'd,
look
at
np
tables,
ipvs
admin,
if
you're
using
ipvs
for
q
proxy
and
then,
of
course,
look
at
pathmt.
If
there's
an
if,
if
the
amount
the
size
of
the
packet
that's
allowed
on,
the
network
is
some
size
defined
and
if
you're
trying
to
send
larger
packets,
it's
not
going
to
go
coming
to
north
south.
You
do
similar
to
add
it
to
whatever
you
do
in
east
west.
A
You
also
do
auto
gateway
right,
so
leaving
the
part
it
snaps
out
of
the
node
and
gets
to
the
gateway
on
your
physical
network
and
are
you
able
to
reach
your
gateway
first?
If,
yes,
then,
are
you
able
to
is?
Is
the
path
mtu?
Fine
again,
if
that's
a
s2,
then
maybe
there's
a
problem
between
your
gateway
and
your
internet,
which
means
it
could
be
a
firewall.
A
If
that
works
too,
then
check
net
rules,
iptable
rules.
If
there's,
if
there's
any
problem
there
more
often
than
not
I've
seen
flannel
or
calico.
If
there
isn't
a
rule
present
in
in
iptables
in
that
table,
that
that
tells
that
the
packet
that's
leaving
pod
busting
to
external
to
the
node
needs
to
be
matted
out.
A
So
in
interest
of
time
quickly
going
forward
open
source
tools
right,
so
there
are
a
bunch
of
open
source
tools
that
you
can
use
to
figure
out.
What
where
the
problem
lies,
some
of
them
that
I've
used
or
found
helpful
is
net
shoot.
Net
shoot
is
a
docker
image
that
essentially
bundles
all
of
the
binaries
and
programs
that
are
useful
to
debug
tcp
dump,
ipping
hyper.
A
A
But
then
you,
you
need
to
be
able
to
log
into
that
pod
to
try
it
out
right.
So
let
you
can
what
you
can
do
is
run.
It
shoot,
attach
it
to
that
particular
pod
and
then
run
commands
as
if
you're
running
within
the
network
namespace,
and
what
k
does
netlook
does
or
the
way
I've
built.
It
is
to
do
somewhat
similar
right.
So
it's
a
binary.
A
A
So
it's
like
a
synthetic
probe,
so
it
gives
you
information
of
whether
the
kubernetes
cluster
is
healthy,
whether
the
pod
networking
is
healthy,
but
it
doesn't
tell
you
if
why
your
part
is
not
working,
because
it
could
be
that
there
are
other
apps
running
on
the
same
note,
that's
working
and
you
would
still
get
a
healthy
result,
but
your
app
might
still
be
crashing.
So
that's
where
k
does
not
look
and
net
should
to
some
extent
comes
into
play.
Okay,
checker
is
another
probe
based
tool.
A
It's
not
been
updated
for
a
long
long
time,
so
I'll
leave
it,
as
is
what
to
use
when
right.
This
is
important
so
again,
going
back
to
the
slide
where
we
looked
at
problem
classification
and
common
problems.
What
I
did
was,
I
put
it
put
down
what
you
would
normally
do
when
you
saw
such
problems
and
we've
discussed
this
in
the
flowchart.
So
I'm
going
to
skip
this
slide
very
quickly,
but,
for
example,
if
it's
a
complete
outage,
then
you're,
probably
better
off,
looking
at
continuous
monitoring
and
looking
at
probes
yeah.
A
Each
of
these
links
I'll
share
the
five
slides,
and
each
of
these
links
will
take
you
to
github
repos,
where
you
get
the
app
you'll,
be
able
to
install
it
and
play
it
play
for
yourself
now
for
the
automated
way.
How
do
you
do
things
in
an
automated
way
right?
So
that's
where
k
test
comes
in
so
why
k
does
net
look?
What
does
it
do?
A
A
A
Time
is
of
the
essence
for
debugging
things.
The
problem
might
correct
itself.
So,
knowing
that
there's
a
problem
is
important
because
kubernetes
is
built
to
fix
itself,
but
then,
if
there's
a
problem,
then
it's
good
to
know
about
it,
because
if
it
keeps
compounding,
there
might
be
a
large
scale,
downtime
automating
a
lot
of
those
steps
that
you
saw
in
on
the
flowchart
one
it's
hard.
You
might
probably
need
to
know
about
some
networking
technology
and
tools.
A
The
idea
is,
with
kds
network
you're
able
to
automate
that
it
is
automated,
so
you're
able
to
run
it
easily
and
then
get
some
kind
of
feedback
essentially
give
you
that
self-service
debugging,
and
it's
inspired
by
that
note
problem
detector
that
that's
there
in
the
github.
You
should
check
that
out.
What
can
it
do
today?
Yeah?
It
works
with
ipv6
ipv4
stacks.
A
There
are
a
bunch
of
checks
that
it
runs
on
the
host
side.
When
it's
running
on
the
host
network
name
space,
it
runs
connectivity
check
for
the
gateway
api
server.
It
runs
if
there
are
multiple
api
servers
in
a
highly
available
cluster.
It
checks
all
of
them.
When
it
comes
to
pods,
it
again
looks
at
gateway.
It
looks
like
if
it's
able
to
talk
to
the
api
server
it
does
dns
by
looking
at
dns
lookup.
A
It
also
does
pmto
discovery,
which
is
a
big
one,
that
I've
seen
causing
issues,
and
it
also
looks
like
various
ip
endpoints
for
a
service.
So
if
a
service
has
20
points,
that's
backed
by
it,
it's
going
to
reach
each
of
these
20
parts
and
see
if
everything's
working,
because
if
there's
a
partial
degradation
where
you
have
18
parts
working
out
of
20,
you
want
a
tool
to
tell
you
that,
so
this
tool
will
do
that,
for
you
we'll
quickly
go
to
the
demo.
A
I'm
hoping
I'll
be
able
to
give
you
five
minutes
in
the
demo.
But
what
you
will
see
in
the
demo
is
that
there
are
three
nodes:
two
workers
and
and
a
control
pin
node
kt
netlook
binary
is
installed
on
all
of
the
nodes
and
there
is
a
busy
box
part
that
I've
deployed,
as
well
as
the
hostnames
part,
which
is
my
app
backed
by
a
service
printed
by
a
service.
A
A
I
have
a
service,
that's
backing,
hostnames,
it's
a
cluster
ip
service,
not
load
balancer
and
there's
also
traffic,
which
you
can
ignore.
So
we'll
see
two
cases
here,
we'll
mainly
see
east
west
communication
and
how
kts
network
helps
in
in
that,
and
the
first
case,
we'll
see,
is
where
the
communication
stops
working
due
to
maybe
say
a
network
policy
right.
A
One
of
the
parts
running
that
if
you
run
the
same
command
again
you're
going
to
get
a
reply
from
another
one,
so
it
seems
to
be
working
fine
right
and
if
you
look
at
it
busy
box
is
running
on
worker
one
so
on
worker
one.
I
have
gds
network
installed
here.
So
if
I
do
a
grand
demo,
this
is
where
my
k
doesn't
network
binary
is
if
I
do
a
host
and
run
it,
I'm
going
to
get
a
bunch
of
checks,
and
it's
what
it's
saying
is
my
cluster
looks
healthy.
A
A
A
Happening
yesterday
and
let's
run
the
same
test
again
looks
like
things
are
back
back
to
good,
so
this
tool
kind
of
gives
you
or
helps
you
figure
out.
What's
going
on
right
and
you
can
see
the
debug,
it
also
tries
various.
It
does
a
binary
search
to
get
to
the
right
mtu
and
it
tells
you
that
there
is
a
the
path.
Mtu
is
1480
and
things
are
working.
So
let's
look
at
an
mq
problem.
Next,
what
I'm
going
to
do
is
I'm
going
to
do
a
cubicle,
exact
minus.
A
A
Connectivity
check
works,
but
something
failed
right.
So
let's
look
at
what
failed
pmtu
check
for
destination
ip
actually
failed,
because
it's
not
able
to
get
things
from
there
and
if
you
look
at
the
pmt
discovery,
it
got
a
reflection
for
one
four
to
four
bytes,
but
it
didn't
get
something
for
1500.
A
A
A
A
With
that,
I
hope
you've
had
a
good
session
and
learned
something
about
kubernetes
networking
looked
at
various
problems
that
can
happen
within
kubernetes
when
you're,
deploying
pods
or
apps,
as
well
as
hopefully
kds
network
will
help
you
to
get
to
that
first
level
debugging
and
at
least
finding
out
where
the
problem
could
could
lie
and
then
getting
to
the
right
people
for
fixing
it.
As
always.
Looking
for
contributors
there's
a
lot
of
things
that
needs
to
be
done,
the
tool
doesn't
do
any
automation
for
the
external
report.