►
From YouTube: High Performance Networking for Distributed DL Training in Production K8s - Nivedita Viswanath
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2021 Virtual from May 4–7, 2021. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
High Performance Networking for Distributed DL Training in Production K8s - Nivedita Viswanath & Vatsan Kasturi, NVIDIA
Distributed DL training requires high performance networks connecting tens, hundreds, or for certain natural language processing models, even thousands of GPUs. Running these workloads on Kubernetes clusters of GPU enhanced servers requires careful engineering to avoid bottlenecks at NIC and switching fabric that act as interconnect between nodes. In this presentation we will describe the design and architecture of a 800 GPU cluster interconnected over RoCE fabric to achieve line rate performance between communicating containers in a multi-node job. Some of the topics we will cover are scalable cookie-cutter POD design for DC, low latency one hop network design that enables NCCL rings to avoid output port congestion and K8s integration with a multi-homed network for optimal GPU utilization. We will share performance numbers for training workloads from our production clusters.
https://sched.co/ekBq
A
A
The
multi-node
data
center
consideration
has
to
be
you
know,
involved
here
for
the
design
of
the
cluster
itself
and
depend
depends
on
the
heating
and
the
cooling.
The
rack
layout,
along
with
the
fine
networks
which
are
listed
here,
what
we
call
as
the
east
west,
not
south,
ipmi,
storage
and
internet
network,
the
edi
hdr,
ndr
and
pci-
are
bandwidth
related.
The
edr
is
100k,
hdr
is
200,
gig
and
ndr
and
400
links,
and
how
do
they
actually
communicate
gpu
to
gpu
in
the
east-west
direction?
A
A
A
A
Faster
model.
Convergence
for
the
data
center
means
faster
performance
numbers
coming
down
into
increasing
productivity
and
also
converging
faster
into
the
accurate
states.
When
you
have
8
billion
parameters
to
operate
with
the
network
needs
to
be
match
the
multi-node
cluster
gpu
targets,
so
they
don't
become
a
bottleneck.
A
A
We
call
it
a
power
side
where
the
heat
will
come
out
and
then
leave
the
data
center.
There
are
basically
six
switches
here,
each
of
them
128
ports,
100
gig
switches-
and
there
are
four
rings.
We'll
talk
about
in
the
subsequent
slides
and
two
of
them
are
used
for
leaves
to
go
up.
The
north
south
traffic.
A
A
In
multi-node
cluster,
these
are
the
building
blocks,
moving
from
left
to
right.
There's
a
data
center.
There's
power,
cooling
racks.
How
do
you
do
the
placement
of
the
dgx
and
the
cpu
nodes,
and
this
is
typically
done
by
the
side,
ops
and
the
networking
team,
and
then
there
is
perimeter
network
and
then
the
pi
network
is
right
here
which
has
a
k.
It
has
a
bgp
layout
in
the
switching
along
with
the
vlan
on
the
lower
end,
and
you
do
have
monitoring
metrics
and
logging
associated
with
both
the
networks,
as
well
as
the
user
jobs.
A
This
particular
green
thing:
gpu
gpu
network,
is
a
short
span
in
cluster
bandwidth,
heavy,
which
is
where
all
the
ndr
edr
hdr
traffic
actually
flows
and
they're
very
sensitive
to
rtt
between
the
nodes,
and
you
want
them
to
be
non-blocking
and
ensure
that
there's
performance
in
this
particular
network
and
that
needs
to
interact
with
storage
network.
It
can
be
in
the
leaf
of
the
spine
to
fetch
the
data
and
also
write
the
results
and
the
logs,
and
obviously
the
data
center
needs
access
to
the
internet
for
pulling
images
or
data
sets
or
for
other
purposes.
A
A
Each
of
the
ring
connects
to
one
of
the
nicks
in
each
of
the
dj
axes
and
they
form
a
ring,
and
so
each
node
is
only
one
half
away.
There
is
uplink
associated
with
this,
which
is
about
6.4
terabytes.
If
you
need
to
go
not
south
to
go
to
the
storage
or
other
needs,
but
when
the
gpus
are
communicating
with
each
other,
they
are
only
one
half
of
it
with
low
latent
round
trip
time.
A
Many
of
you
are
aware
of
this
core
spine
leaf
model.
This
is
basically
the
car
switching
for
not
south
and
100.
Nodes
are
stuck
here
in
the
in
the
racks,
with
six
nicks,
four
of
them
used
by
the
east
west
and
two
of
them
for
the
north
south
and,
as
I
indicated
earlier,
gp
to
gpu
traffic
is
significant
and
you
want
this
to
be
high
performance,
low,
latent.
A
A
A
This
is
a
layer,
7
stack
which
most
of
you
are
familiar
with,
and
this
talks
about
the
infiniband
and
the
rocky
and,
as
you
can
see,
only
the
lower
layers
are
a
little
bit
different
from
the
from
the
l4
onwards.
It's
identical
and
one
can
use
either
rocky
ethernet
and
infiniband,
and
you
can
reconfigure
the
mix.
A
We
discussed
this
particular
slide
earlier
about
edr
hdr,
mdr,
100,
gauge
200
and
400
gig,
and
how
do
they
relate
to
the
pci
bus?
Do
they
saturate
the
pci
bus,
as
you
can
see
as
they
move
up
the
the
bandwidth
and
the
line
rate
increases
they're
going
to
saturate
this
pci
bus
theo
and
you
have
to
move
up
to
the
forum
network?
A
A
This
is
another
view
of
their
network
from
seeing
inside
the
box,
there's
a
motherboard
and
lots
of
memory
and
then
the
cubelet
and
the
container
runs
here.
There
is
a
lots
of
traffic
going
through
this
particular
nic,
and
the
kubernetes
actually
operates
that,
along
with
the
metrics
being
collected
here.
A
A
A
So
this
is
a
traditional
switching
structure
which
most
of
you
are
familiar
with,
so
edge
spine,
leaf
and
torque.
The
storage
can
decide
beside
very
close
to
the
leaf
if
you
want
to
have
a
local
traffic
or
move
it
up
a
little
bit
into
the
spine
and
the
ip
method
actually
connects
it
to
the
bmc
on
the
switches
to
managed.
A
B
Thank
you,
watson,
hello,
everyone,
I'm
nivedita
and
I'm
glad
to
be
co-presenting.
This
session
now
watson
talked
about
the
best
practices
and
network
protocols
to
be
considered
while
designing
a
high
performance
multi-node
cluster
in
a
data
center.
I
would
like
to
talk
about
how
we
leveraged
those
design
decisions
to
enable
high-speed
networking
in
kubernetes.
B
B
B
B
B
The
rdma
interfaces
can
then
be
managed
by
cubelet
as
allocatable
resources,
so
once
a
pod
that
is
requesting
these
rdma
interfaces
in
its
resource,
spec
gets
scheduled
onto
a
node.
Cubelet
starts
moving
the
network
resources
from
the
host
namespace
to
the
port
namespace
to
the
registered
cni
plugin.
B
B
Now
we
often
see
a
sawtooth-like
pattern,
and
this
is
a
result
of
the
two
phases
that
generally
occur
in
every
epoch,
of
an
ml,
app
the
two
phases
being
the
compute
phase
and
the
communicate
phase
in
the
compute
phase.
The
gpu
workers
are
fetching
from
storage
performing
computation,
creating
mpi
barriers,
and
there
is
no
communication
with
other
gpu
workers,
hence
the
dip
or
the
valley
in
the
traffic.
B
In
the
communicate
phase,
the
gpu
workers
share
the
result
of
the
previous
phase
with
other
workers,
and
this
results
in
the
peak.
In
the
pattern
that
you
see
now,
we
ideally
want
to
maximize
the
compute
time
and
minimize
the
communicate
time
in
an
algorithm
to
improve
epoch,
training
efficiency,
in
addition
to
monitoring
the
rdma
traffic.
We
also
run
nickel
all
reduce
routine
and
other
collective
primitives
as
part
of
a
pre-flight
check
prior
to
running
multi-node
workloads.
B
Nickel.
All
reduce
test
captures
the
average
bus
bandwidth
and
it's
run
with
different
configurations
where
we
vary,
the
mpi
rank
per
node,
the
number
of
threads
per
rank,
the
number
of
gpus
per
thread,
and
so
on
from
the
table
and
graph.
We
see
that
even
with
increasing
number
of
nodes,
we
get
a
pretty
consistent
average
bus
bandwidth
of
about
45
gbps,
where
the
theoretical
light
rate
is
50.
Gbps.
B
Where
bandwidth
is
concerned,
comprehensive
understanding
of
the
network
flows
will
provide
guidance
for
designing
multi-node
clusters
without
congestion
at
edr
and
future
hdr.
Ndr
rates,
what's
to
be
kept
in
mind
is
hdr.
Ndr
rates
are
applicable
for
tor
switches
as
well,
and
not
for
dc
to
dc
connectivity
alone.
B
Regarding
the
kubernetes
control
plane,
we
found
that
the
upstream
components
and
plugins
in
the
community
are
easy
to
adopt
and
customize
and
by
implementing
features
like
gang
scheduling,
starvation
handling
quotas.
In
our
custom
batch
scheduler,
we
were
able
to
achieve
seamless
scheduling
of
multi-node
workloads
while
maintaining
fairness
for
end
users.