►
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Understanding Windows Container Network in Kubernetes Using a Real Story - Cindy Xing, Huawei & Dinesh Kumar Govindasamy , Microsoft
For enterprise companies, there are tons of existing Windows application assets in their IT environment. Windows container technology is new and the kernel is quite different from Linux. Running Windows applications natively in a Kubernetes Windows container cluster is very challenging especially in the network area. In this presentation, Cindy and Dinesh will do a deep dive into the Windows container network technology. They will compare Windows container network mechanisms with Linux ones and make visual analogies. Using a real world case study, a sample implementation of a Kubernetes Windows CNI plugin along with network troubleshooting tips will also be walked through.
To learn more click here: https://sched.co/FuKI
B
A
Today,
Denisha
and
I
are
gonna
use.
A
real
story
will
help.
You
understand
Windows
container
network
for
rent.
It
is
in
the
next
35
minutes.
We
are
going
to
talk
about
windows,
container
architecture,
walk
through
windows,
container
network
in
a
kubernetes
cluster,
then
from
a
cloud
providers
perspective,
help
explain
what
is
CI
plug-in
and
how
to
write
one.
A
A
B
Chosen
in
this
session
of
the
presentation,
we
are
going
to
be
looking
at
the
high-level
architecture
of
kubernetes
for
Windows
the
net,
the
building
blocks
of
the
Windows
container
networking
and
the
kubernetes
network
connectivity
and
how
it
works
for
Windows.
The
Windows
operating
system
has
two
mains
inbox
services.
The
host
computer
service
and
host
network
service
that
provide
the
required
container
function
on
the
host
computer
service
is
responsible
for
providing
the
computer
related
functionalities,
such
as
job
object,
creation
and
management
and
general
lifecycle
management
of
containers.
B
Then
a
set
of
Linux
fundamental
networking
building
blocks
that
communities
architecture
is
built
upon,
let's
look
at
them
and
how
they
compare
with
Windows
and
how
we
have
developed
the
Windows
networking
drivers,
the
network
namespace
in
Windows,
the
network
namespaces
Network
compartments,
the
compartments
are
logical
container
in
the
tcp/ip
stack
and
the
network
layer
or
the
inner
layer
in
the
tcp/ip
stack
is
making
sure
that
each
compartment
is
isolated
and
packet.
Forwarding
between
the
compartments
is
prevented.
B
All
I
P
related
objects
such
as
IP
addresses
IP,
prefixes
routes.
Everything
stay
for
that
unique
to
that
compartment
in
linux,
the
layer,
2,
switching
and
lately,
functionality
is
provided
by
linux
bridge
and
the
IP
routing
in
Windows.
A
v2v
switch
provides
the
layer,
2,
switching
and
layer
3
functionality.
B
You
can
have
many
instances
of
V
switch
if
you
are
familiar
with
open,
V
switch,
it's
very
analogous
to
that
of
Windows
and
you
can
dynamically
add
and
remove
switch
ports
to
this
V
switch
and
each
instance
of
a/b
switch
has
its
own
forwarding
table
and
forwards
packets
based
on
the
MAC
address
and
the
VLAN
tagging
of
the
packets,
hyperlynx
The
Container
network
interfaces.
It
could
be
host
Phoenix
or
VM
NICs
that
are
added
to
the
network
name
space
and
then
bound
to
the
corresponding
switch
port
in
the
B
switch.
B
I,
PBS
and
IP
table
provide
rich
packet
filtering
mechanism
in
Linux,
whereas
in
Windows
Windows,
Firewall
and
vfp
provides
you
the
similar
packet
from
packet
filtering
functionality.
Vfp
is
virtual
filtering
platform.
It's
a
programmable
match.
Action
based
filtering
engine
that
helps
you
to
apply,
trim
it
like
very
rich
data.
B
From
a
data
plane,
primitives
such
as
a
clean
metering,
netting
load,
balancing
and
n
cap
D
cap
and
much
more
VFP
is
the
same
data
plane
that
is
used
to
power,
Microsoft,
Azure
and
Azure
stack
and
we
are
bringing
the
same
data
plane,
basically
reusing
the
same
data
plane
and
providing
you,
the
windows.
Networking
functionality
we'll
be
looking
at
vfp
in
detail
in
the
next
few
slides.
B
So,
let's
put
all
of
this
together
and
look
at
the
internals
of
windows
container
networking.
Let's
take
a
look
at
how
bare-metal
networking
works
in
Windows
right,
so
you
have
a
physical
host
and
you
have
a
physical
NIC
connected
to
this
host
and
the
physical
NIC
has
an
IP
and
a
Mac.
By
default,
this
physical
NIC
is
exposed
to
the
default
route.
Namespace
of
the
host
and
host
uses
this
to
communicate
to
the
external
world.
B
Let's
say
you
are
adding
a
virtual
network
to
this
host.
A
new
instance
of
virtual
switch
is
created
on
the
host
and
the
switch
is
bound
to
this
physical
leg.
Now
for
the
host,
we
are
creating
a
new
virtual
leg
and
then
attach
it
to
the
root
namespace
and
then
bind
it
to
the
corresponding
switch
port
on
the
switch.
The
IP
and
the
Mac
that
was
in
the
physical
neck
is
now
moved
to
this
virtual
leg.
B
For
adding
a
container
namespace
is
just
trading
in
a
compartment
in
the
host.
Now
adding
a
container
endpoint
is
basically
adding
a
new
virtual
knit
to
this
namespace
and
then
binding
it
to
the
corresponding
switch
port
in
the
V
switch.
The
container
has
its
own
IP
and
the
Mac,
the
NAT
and
load
balancing
functionality
is
provided
by
basically
using
the
vfp
rules.
The
vfp
rules
are
present
in
four
container
port,
the
management
port
and
the
external
port.
Combining
all
of
this,
we
will
be
able
to
shape
the
diff
traffic.
B
Let's
take
a
look
at
the
virtual
filtering
platform,
so
the
that
is
the
brain
behind
the
entire
container.
Networking
that
we
are
providing
right.
So
virtual
filtering
platform
is
an
extension
added
to
the
V
switch,
so
any
traffic
that's
flowing
through
the
V
switch,
will
also
flow
through
the
vfp
and
the
VSP
kept.
The
rules
can
be
applied
both
on
the
inbound
and
outbound
direction
of
the
packet.
B
So
this
on
the
right
side
is
a
example
of
a
vfp
rule.
So
VAP
has
different
layers.
The
layers
can
be
again
a
cooling
metering,
we
need
SLV,
each
layer
has
different
groups
and
the
groups
are
in
Direction
out
direction
or
ipv4
ipv6
or
it
could
be
user-defined
and
then
inside
the
group
you
have
a
rule
again.
Remember
I
said
that
the
vfp
is
a
match.
B
Action
based
rich
packet
filtering
engine
right,
so
the
rule
is
having
a
filtering
condition
and
you
can
filter
based
on
the
IP,
addresses
the
source
or
destination
IP
address
or
protocol,
and
then
filter
that
particular
packet.
And
then
you
can
apply
an
action
on
that
packet
and
action
can
be
in
kappa,
kappa
or
lb
nod
or
any
such
action.
So
it's
it's
very
rich
right
now,
H&S
is
basically
providing
the
controller
plane
and
vfp
is
providing
in
the
data
plane
a
channel
H&S
programs,
the
corresponding
rules
into
vfp
and
then
shapes
the
traffic.
B
However,
we
want
I
I
hope.
I
gave
you
enough
information
about
the
windows
container,
networking
how
it
works
in
both
control,
plane
and
data
plane.
Now,
I'm
gonna
put
all
of
this
together
and
walk
you
through
some
examples
of
how
kubernetes
network
connectivity
works
for
Windows
I'm
going
to
go
over
three
examples.
The
first
example
is
pod
pod
connectivity.
Let's
say
you
have
two
parts
in
this
host:
pod,
1
and
part
2
and
pod.
One
wants
to
communicate
to
pod
to
write
one
of
the
kubernetes
requirement.
B
Is
that
part
of
what
connectivity
should
not
be
Natick
right?
Basically,
it
has
to
be
provided
through
an
l-3
connectivity.
So
the
path
the
packet
reaches,
the
port
port
3
of
the
vfp
right.
There
are
a
bunch
of
vfp
rules,
so
by
default
in
kubernetes
we
apply
our
outbound
natural.
So
what
it
means
is
every
traffic
coming
out
of
that
container
will
be
Natick,
but
we
we
have
some
special
exceptions
where
you
can
apply
some
exceptions,
rules
in
vfp
and
say
that
I
don't
want
to
nad
this
these
set
of
traffic's.
B
So
there
are
some
some
parts
under
the
service
leader,
though,
are
those
exceptions
where
we
apply
on
the
vfp
so
that
we
can.
We
don't
need
to
nad
them
right,
so
the
packets
will
not
been
added
and
will
get
forwarded
to
the
forwarding
plane.
Vsp
has
its
own
routing
table
based
on
the
destination
IP.
It
looks
up
the
destination
MAC
and
forwards,
the
packet
to
the
corresponding
destination
port
and
the
packet
reaches
the
destination
pod.
B
The
second
example
I'm
going
to
talk
about
is
a
pod
trying
to
reach
an
Internet
traffic,
so
the
by
by
default,
as
I
said
earlier,
every
traffic
coming
out
of
the
pod
will
be
basically
out
born
added
right,
so
the
packet,
the
pod
one,
is
trying
to
reach
Internet
the
packets
reach
port,
three
no
Acula
supply,
so
it
goes.
It
goes
and
hits
the
esnatt
layer
and
esnatt
will
be
applied
on
that
packet.
So
esnatt
is
a
stateful.
Let's
not
is
created
I'll
come
to
what
this
stateful
mean.
B
Basically,
the
source
IP
of
the
container
and
the
source
port
of
the
container
is
replaced
with
the
host
IP
and
the
port
and
the
traffic
is
gets
forwarded
to
the
external
port
and
the
traffic
goes
out
to
the
Internet.
Now,
when
the
response
is
coming
back,
you
need
to
base
that's
why
we
create
a
stateful
match
so
that
when
response
is
coming
back,
we
need
to
reverse
it.
So
we
need
to
rewrite
the
host
IP
:
port
to
the
container
I
peep
:
port.
B
The
next
example
is
part
2
load-balancing.
Basically
the
cue
proxy
flow
right.
So
let's
say
you
have
a
load
balancer
created
on
this
kubernetes
cluster
and
the
bottom
bottom.
One
shows
the
load
balancer
right,
so
you
have
a
whip,
:
port
and
you
have
a
set
of
back-end
dips.
Now
the
pod
is
trying
to
reach
the
service
whip,
so
the
packet
reaches
the
port
port
3
now
in
Windows.
What
we
do
is
especially
for
the
non
DSR
flow.
We,
the
management
port,
is
the
load
balancer
for
us.
B
So
it's
basically
a
little
earmarks
for
us
and
it's
distributed.
So
every
node
has
its
own
load
balancer
right.
So
the
traffic
from
the
pod
a
has
to
go
to
the.
If
it's,
if
it's
hitting
the
service
whip,
it
has
to
go
to
the
management
port.
So
what
we
do
is
we
in
cap
the
packet
to
the
management
port
and
send
it
to
the
management
port
in
on
the
management
port.
We
decap
the
packet
again
and
then
apply
this
lb.
B
Natural
LD
natural
is
basically
a
rule
where,
given
a
set
of
dips,
it
selects
a
dip
based
on
the
fight
apples
like
basically
the
source,
IP
destination,
IP
source
port
destination,
port
and
protocol.
It
applies
that
hashing
and
then
selects
a
dip
once
it
has
selected
the
dip,
it
rewrites
the
destination
IP
and
the
destination
MAC
and
sends
it
to
the
corresponding
dip.
So
this
is
how
load
balancing
works
in
Windows.
B
A
So
first
of
all,
couplet
starts
dr.
shim,
which
is
a
G
RPC
server
in
it.
There
are
some
components
and
bunch
of
API
functions
and
then
secondly,
Hulett
will
call
the
dark
regime
GRP
C,
API,
Grampa's
sandbox
through
this
API
call,
dr.
shim
will
call
dr.
engine
and
the
dr.
engine
will
start
a
passed
continue.
Sometimes
people
call
it
infrastructure
container,
as
well
as
a
result
of
this
container
creation.
Apart
in
our
name,
space
will
be
created,
then
dr.
sham
will
call
dr.
engine
to
grab
this
part
network
space.
A
After
that,
dark
regime
will
cause
the
internal
CNI
plugin
manager,
and
then
the
ceiling,
c9
Network,
plug-in
manager
will
run
CNI
plug-in
with
network
configuration
file
and
environment
variables.
So
internally,
the
CLI
plug-in
will
create
first,
we'll
create
a
virtual
network
and
then
create
a
network.
Endpoint
then
plug
in
the
network,
endpoint
one
side
to
the
past
container
and
the
other
side
to
the
virtual
network,
and
then,
after
that,
apply
the
routing
and
then
the
host
level
policies.
A
As
you
can
see,
CLI
is
the
interface
between
CI,
between
container
rantin
and
CI
plug
in
and
then
CI
plugin
is
a
executable
which
can
help
you
to
create
virtual
network,
continue,
endpoint,
apply
ratings
and
policies.
As
what
Denisha
has
explained
earlier
for
Windows
container,
there
will
be
continual
networks
which
container
network
endpoint
like
the
virtual
switch
port
and
then
the
policy
mechanism
of
vfp.
A
A
I
have
two
executables
that
I
am
executable
and
the
C&I
plug-in
executable
in
the
middle
and
then
I
need
to
author.
My
network
configuration
file,
as
you
can
see
the
right
bottom
at
the
right
bottom
block.
You
can
see
the
configuration
file
basically
I
specify
the
name
for
my
Upham
and
see
my
plugin
and
then
the
subnet
prefix
and
then
on
the
top
left,
a
right.
A
Sorry,
you
can
see
the
environment
variable
in
it,
I
specify
the
add
action,
the
pod
Network
space
and
then
the
path
for
where
to
find
executables
for
sing
I
plug
in
an
item
after
that,
once
I
run
the
I
plug-in
with
the
configuration
file
and
the
environment
variables.
First,
the
CI
plug-in
will
call
the
functions
to
create
the
virtual
network,
then.
Secondly,
it
will
call
I
Pam
to
assign
IP
address
to
the
pod
based
on
the
subnet
prefix.
A
So
at
this
point,
I'm
pretty
much
ready,
I
am
very
clear
about
seeing
I
plug
in
how
it
works.
I'm,
ready
to
write
my
sing
I
plug
here,
I'd
like
to
share
you
a
bunch
of
api's.
As
what
inish
explained
earlier,
the
HTS
shim
exposed
by
Microsoft,
you
can
find
it
on
github,
let's
walk
through
all
the
api's
so
for
network
they
are
api's
to
create
and
delete
the
virtual
network
specifically
for
the
there
is
a
data
structure
called
HC,
HC
n
network
config
in
it.
A
A
A
What
special
thing
I
want
to
call
out
is
on
the
each
Windows
node.
We
specially
created
a
CBR
0
and
the
point
I
use
it
as
the
continue
Network
a
default
gateway
and
also
for
the
CBR
0.
We
enabled
the
IP
forwarding
so
that
for
the
continued
Network
egress,
it
will
be
dotted
to
CBR
0.
Then
after
that
forwarded
to
the
host
Nick
and
then
for
the
API
side,
I
shared
earlier,
you
can
find
it
on
those
two
things.
A
A
A
Besides
the
actually
this
one,
so
there
are
two
notes
and
then,
besides
the
web
UI,
you
can
also
use
the
Kouga
kado
to
directly
manage
your
machines
before
we
drill
down
to
detail
scenarios
I
like
to
point
out
a
bunch
of
tools
which
are
really
useful.
When
you
try
to
troubleshoot
your
network
issues,
the
first
one
is
the
clock
blog.
A
This
would
help
you
to
collect,
run
on
your
Windows
host
and
collect
all
your
continued
network
settings,
for
example
like
what
a
Danish
call
out
earlier,
your
what
kind
of
virtual
network
you
have,
how
many
endpoints
you
created,
what
kind
of
vfp
policies
you
created
and
apply
to
the
endpoints,
and
then
this
to
commands
a
start
and
stop
packet
capture
habit
commence.
You
can
also
run
on
your
Windows
node
to
collect
all
the
container
Network
tracings
and
later
on.
A
You
can
analyze
and
troubleshooting
issues,
so
let's
get
to
the
master,
Linux
master
node
and
see
what's
going
on
for
the
for
the
cluster.
As
you
can
see,
I
have
two
nodes
on
the
machine
and
then
I
have
a
bunch
of
paths
deployed,
especially
I
want
to
show
you
some
of
the
puzzle,
keep
in
mind
all
the
IPS.
So,
for
example,
like
the
1.13,
six
and
1.39
are
hosted
on
one
Windows
node
and
then
the
0.10
0.7
are
on
the
room.
A
Other
node
and
then
here
I
have
all
the
the
two
Windows
machines
available
and
then
I
want
to
show
you
the
config
file.
So
here
you
see
the
subnet
proof,
subnet
prefix
and
then
the
type
and
IPAM
type
are
the
executable
names
for
my
cluster,
seeing
I
plug-in
and
then
especially
here
are
the
the
nod.
Policies
finished
just
colored
earlier.
A
A
A
B
B
A
So
I
downloaded
this
is
oh
and
then
you
can
run
it
and
then
go
to
the
folder.
Then
you
can
see
bunch
of
files
created.
Those
are
the
log
files
and
then
later
on,
Tinesha
gonna
go
through
those
files
and
explain
in
detail.
After
that.
Let
me
like
running
into
the
containers.
Let
me
get
to
the
container.
A
And
then,
let's
first
see
the
IP
address
for
this
container,
as
you
can
see,
it's
the
1.1
39.
How
about
it
like?
Let
me
ping
the
part
on
the
same
host,
but
it's
the
other
part.
So,
as
you
can
see
it
can
heart-to-heart.
Communication
is
working.
How
about
the
other
part
like
the
pod
on
the
other
machine,
canopy
unit,
it's
working
as
well,
so
for
this
wise
pod
through
the
routing
table
and
then
to
the
other
host,
then
how
about
I
want
to
go
to
the
internet?
I
want
to
go
to
I.
A
You
so,
as
you
can
see,
it's
working
as
well,
so
on
this
container
inside
the
container,
it
can
also
go
to
the
Internet
and
then
the
other
thing
we'll
be
able
to
try
to
try
out
is
basically
on
my
other
machine.
I
have
a
real
customer.
Is
web
server
installed
how
about
I
get
get
to
there
and
see
what
weather
is
working
or
not?.
A
A
A
A
B
Okay,
so
before
I
go
swimming
into
the
logs,
I
would
like
to
just
show
you
guys
the
debugging
flow,
so
basically
in
general,
how
to
debug
contain
a
Windows
container
networking
issues.
So
you
basically
have
so
look
at
the
cubelet
and
Q
proxy
locks
and
make
sure
that
the
configuration
from
the
kubernetes
side
is
propagated
through
cubelet
and
q
proxy
to
the
underlying
system,
and
then
look
at
the
cni
spec
and
the
cni
locks
to
make
sure
that
the
configuration
what
Kuban
it
is
in
q
proxy
was
telling
is
communicated
to
CNI.
B
B
Are
there
and
policies
are
there
on
the
vfp
side
and
then
eventually,
if
control
plane,
looks
good,
then
go
to
the
data
plane
and
make
sure
that
you
can
do
a
packet
capture
and
then
look
at
packet
drops
and
see
what
has
happened
now
in
the
scenario
that
silliest
walk-through
she
had
to
host
and
on
host
1,
she
was
trying
to
curl
a
application,
that's
running
on
another
host.
So
what
is
the
expected
flow
here
right?
B
B
Now
the
packet
reaches
the
external
host
and
then
it
goes
to
the
container
port,
and
then
it
eventually
is
delivered
to
the
tcp/ip
stack
and
inside
the
container
and
tcp/ip
stack
is
going
to
drop
the
packet
because
she
did
stop
the
website
inside
the
other
container,
so
basically
that
that
packet
will
be
dropped
with
saying
that
there
is
no
endpoints.
Now
let
me
quickly
walk
through
the
set
of
locks
that
we
collected
right.
So
here
is
a
network
dump.
B
So
this
network
dump
shows
that
she
created
an
l2
bridge
network,
and
this
is
the
JSON
output
of
the
l2
bridge
network
and
it
will
show
you
the
set
of
IP
addresses
prefixes
and
the
MAC
address,
pool
and
the
state
of
the
network
right
and
then
the
end
point
or
text
will
show
you
the
list
of
endpoints
on
this
host.
We
have
three
local
endpoints
and
the
one
remote
end
point.
The
remote
endpoint
are
used
for
Q
proxy.
B
When
you
are
trying
to
play
a
load
balancer
and
whatever
the
policy
she
applied
on
the
cni
config.
You
can
see
them
here.
So
there's
a
set
of
exception
list
out
born
ad
and
the
set
of
route
policies,
and
here
is
a
IP
config
output
that
is
also
collected.
Part
of
the
collect
locks.
You
can
see
the
set
of
compartments
created
for
those
two
containers.
B
She
was
running
on
this
host
and
their
corresponding
IP
addresses,
and
here
is
the
vfp
policy
that
it,
which
is
also
collected
by
the
collect
log
scrip,
and
you
can
see
that
this
is
the
port.
As
I
said
earlier,
vfp
has
individual
policies
per
port
like
container
port,
has
its
own
policies,
management,
port
and
external
port.
So
here
is
an
example
of
a
container
port
policy.
You
can
see
that
there
is
an
aquiline
point
layer
and
a
bunch
of
fat
old
rules
right
and
then
you
have
the
NAT
layer.
B
This
is
a
NAT
layer
and
the
NAT
layer
has
a
bunch
of
groups
and
inside
each
NAT
layer
you
have
the
corresponding
exceptions
program.
The
same
policies
that
were
applied
in
its
own
eyes
will
H&S
will
be
translated
to
vfp
rules
right.
So
now,
let's
take
a
look
at
the
packet
capture.
So
here
is
a
set
of
packet
capture
that
cindy
captured
on
host
1
and
host
2
right.
So
here
is
the
packet
that
said
that
okay
I
am
trying
to
reach
from
39
1.39
and
I'm,
going
to
0.7
packet
right,
the
the
container
right.
B
So,
as
you
can
see,
the
the
port
that
we
are
interested
in
is
port
5.
So
whatever
the
policies
I
showed
in
vfp,
all
of
those
will
be
processed.
So
basically
the
packet
will
flow
through
the
endpoint
laical
pool
layer,
the
NAT
layer
and
then
the
customer
out
layer
and
everything
right.
Eventually,
the
packet
gets
forwarded
from
45
to
port
3,
which
is
the
CBR
0.
B
So
this
is
the
statement
that
says
that
the
packet
was
forwarded
from
code
5
to
port
3,
then
on
port
3,
the
packet
goes
to
the
basically
the
tcp/ip
stack,
and
you
can
see
that
the
TCP
stack
pipe
without
lookup
happens
and
looks
for
the
next
hop
and
then
sends
the
packet
to
the
other
host.
Now,
on
the
other
host,
you
can
see
clearly
that
the
packet
reached
the
container
port
and
it
was
eventually
dropped
because
the
end
point
was
not
fallen.
So
basically,
I
would
follow
the
same
flow.
B
Saying:
okay,
look
up
the
data
plane
only
at
the
last
resort,
right
and
first
look
at
the
configure
issues
and
then
eventually
go
to
the
data
plane.
I
think
we
are
already
running
out
of
time.
So
sorry
for
this
condense
session,
I
hope
you
guys
had
a
good
information
on
windows.
Container
networking.