►
From YouTube: Supercharge Kubeflow Performance on GPU Clusters - Meenakshi Kaushik & Neelima Mukiri, Cisco
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Supercharge Kubeflow Performance on GPU Clusters - Meenakshi Kaushik & Neelima Mukiri, Cisco
AI/ML applications on Kubernetes can be optimized for performance at many levels. This presentation provides an overview of the optimizations such as: - Distributed training on multiple GPUs with optimal selection of interconnects between the GPUs and CPUs. - Utilizing different types of GPUs/Servers for different workloads like training and inference. - OS level optimizations to get optimal performance on the hardware. - Usage of GPU Passthrough for optimal utilization and performance. This presentation will also cover how the selection of machine learning framework, like Kubeflow, can impact performance and hardware utilization.
https://sched.co/Uada
A
Laughter
noon.
Everyone
thank
you
for
coming
to
the
last
session
of
the
conference,
making
it.
Thank
you
so
much.
My
name
is
Vinod
Shekar,
Sheikh
and
I
would
like
to
introduce
the
co-presenter
in
elementary.
We
both
work
at
Cisco.
We
are
part
of
the
Cisco
container
platform
team
and
we
also
enable
to
flow
on
our
platform
and
many
of
our
customers,
our
machine
learning
customers
and
they
have
started
looking
at
distributed
training.
So
what
we
are
going
to
do
is
share
our
learnings
as
we
are
helping
customers
through
their
journey.
A
So
to
start
by
looking
at
what
is
the
opportunity.
So,
as
I
mentioned,
the
opportunity
is
to
compress
end-to-end
machine
learning,
so
make
the
pre-processing
training
and
serving
faster,
and
there
are
many
ways
to
achieve
that
and
I'm
going
to
give
an
example
of
three
ways.
So,
for
example,
you
can
reduce
the
input/output
stall
so
that
your
data
is
always
available
when
you're
training
loop
leads
it.
The
second
is
you
reduce
all
your
overhead
so
that
you
are
utilizing
your
memory
efficiently
and
then?
A
The
third
is
that
you
distribute
your
training
by
scaling
up
within
the
node
through
the
GPUs
and
then
scaling
out
across
the
nodes
and
make
your
machine
a
machine
running
faster.
So
what
we
are
going
to
look
at
is
just
the
last
piece
of
how
you
can
do
that
scaling
using
GPUs
and
we're
going
to
look
at
in
the
context
of
cue
flow.
So
before
we
look
at
what
is
a
cue
flow
wanted
to
show
a
result.
So
this
is
the
result
which
we
did
in
our
lab.
A
This
is
on
UCS
480
ml,
which
is
a
Cisco
UCS
box,
which
has
eight
GPUs
and,
as
you
can
see,
that
we
are
doing
this
tensor
flow
benchmarking
using
the
resonate
model
and
as
we
are
scaling
up
the
number
of
GPUs,
you
can
see
the
number
of
images
per
second
that
we
can
use
in
the
training
has
improved,
and
this
we
are
doing
using
env
link.
Let's
say
we
were
not
using
the
NB
link,
a
faster
link
if
you
would
see
this
same
scale-up,
except
maybe
the
reduction
of
like
40
to
50
percent.
A
So
this
is
the
opportunity
which
you
have
with
distributive
training.
So
what
is
Q
flow
I
know
by
now,
you
have
might
have
listened
to
it.
So
many
times
that
I'm
going
to
say
this
once
more.
It
is
a
collection
of
tool
which
makes
the
end-to-end
machine
learning
lifecycle
easier,
and
the
thing
is
that
it
is
built
on
top
of
kubernetes,
so
it
is
as
portable
as
kubernetes
is,
which
is
super
portable,
so
you
can
actually
run
it
in
public
cloud.
You
can
run
it
in
a
in
a
private
cloud.
A
It's
it's
justa,
kubernetes
application
and
the
other
thing
about
the
queue
flow
is
that
the
job
of
queue
flow
is.
It
gives
you
a
very
easy
interface
and
it
hides
the
complexity
of
orchestrating
whatever
the
tooling
which
the
queue
flow
provides,
and
it
does
the
underlying
orchestration
of
kubernetes
pod,
the
secrets
and
everything.
So
it
manages
for
you
and
you
get
this
CRD
interface.
So
let's
look
into
the
queue
flow
and
the
tools
that
the
queue
flow
provides.
A
So
the
first
is-
and
this
is
where
many
of
our
customers
are-
you
know
you
they
have
the
data
and
they
are
trying
to
do
is
just
exploration
and
many
of
them
are
maintaining
their
own
notebooks
and
so
the
first
thing
they
like
about
queue
flow.
Is
it
standardizes
and
you
have
a
single
place
where
you
know
you
are
not
security
and
compliance
across
each
of
these
individual
magic
machine
learning
teams.
But
you
are
in
the
central
theme
and
you
are
managing
the
notebook
and
you're
using
the
notebook
server.
A
A
So
if
you
look
look
here
and
I
go
to
a
notebook
server,
so
I
can
start,
for
example,
with
a
if
I
want
to
scale
from
single
GPU
to
a
single
CPU
to
a
GPU
and
want
to
tell
scale-up
I
would
use
I
choose
a
GPO
image
which
has
all
the
CUDA
driver
libraries
all
installed
into
that
image,
and
then
there
is
just
a
simple
place
where
you
can
add
the
number
of
GPUs
you
can
use.
So
there
is
a
simple
command
for
that.
So
this
is
the
command.
A
So
let
me
just
copy
this
and
show,
and
then,
if
I
just
launched
this,
then
now
I
I
have
one
CPU
and
you
can
scale
it.
So,
within
even
an
exploration
phase,
you
can
go
from
CPUs
to
GPUs
and
start
scaling
up,
but
what
happens
underneath
is
that
this
notebook
eventually
brings
up
just
a
single
pod,
so
you're
scaling
up
that
for
number
of
GPUs
and
that
pod,
but
you're
not
you
are
not
able
to
scale
out.
So
if
you
have
to
scale
out,
then
you
can
use.
A
Then
then
you
go
ahead
and
use
the
model.
Training
model,
training
operators
which
are
available
within
the
queue
flow,
so
tensorflow
supports
variety
of
model
training
framework.
Sorry,
the
two
floor
supports
variety
of
model.
Training
frameworks
such
as
tensorflow
PI,
Taj
XE
boost
MPI
char,
and
it
also
supports
this
pre-processing
library
framework,
which
is
SPARC.
So
many
of
our
customers,
for
example,
initially
have
been
just
doing
data
analysis
and
on
using
spark
and
there's
just
started,
looking
at
doing
some
kind
of
a
deep
learning.
A
So
this
they
are
at
this
phase
and
the
tensorflow
provides
these
operators
to
do
do
just
that.
And
then,
once
you
have
trained
the
model,
then
you
want
to
optimize
the
model
to
be
as
performant
as
possible,
and
you
can
do
that
using
hyper
parameter,
training
and
in
queue
flow.
There
is
a
and,
of
course,
if
you
wanted
to
even
figure
out
what
model
it
is,
you
could
use
the
neural
architecture
search,
and
so
both
of
these
are
part
of
this
tool
which
is
or
the
library
which
is
called
catty.
A
A
It's
called
as
accelerators
in
Kiev
serving
I,
encourage
you
to
check
out
their
talk
where
they
are
just.
They
described
the
exact
syntax
of
doing
that
so
and
so
the
way
the
cue
flow
implements
all
the
tools
in
the
cue
flow
implement.
Their
logic
is
through
extending
kubernetes
using
cue
flow
operators
so
basically
have
it
has
custom
resource
definition
and,
as
I
mentioned,
that
it
has
these
different
layers,
the
pre-processing,
the
training,
hyper
parameter
tuning
and
serving
and
talked
about
spark
in
the
training.
A
So,
for
example,
what
happens
with
these
operators
is:
let's
take
an
example
of
a
denser
fluid
thr
operator.
What
it
does
is
that
when
you,
when
you
are
looking
at
distributed,
training
in
your
training
code,
what
you
would
specify
is
a
cluster
spec
and
clusters.
Spec
is
how
you
want
to
distribute
your
training
workload
and
the
tensorflow
job
makes
it
very
easy
to
describe
that.
A
As
you
will
see
in
the
CML
file,
it
makes
it
very
easy
to
describe
your
cluster
spec
and
then
underneath
it
spins
all
the
pods
necessary
to
maintain
that
cluster
definition
and
because
the
part
of
the
tensorflow
job
specification,
you
specify
the
container
image
that
you
want
to
run,
or
your
training
code
image
that
you
want
to
run.
It
injects
the
environment,
variable
TF
config
inside
that
pod,
so
that
the
cluster
spec
is
now
available
to
your
training
code.
A
Why
we
need
that
distributed
training
communication
so
maybe
like
50
percent,
so
I'll
just
quickly
walk
through
the
deep
neural
network
and
just
give
a
high-level
view
of
what
communication
is
needed
and
how
it
can
be
optimized
across
different
layers.
So
this
is,
this
is
a
view
of
a
model
and
the
blue
dots
represents
the
different
features
of
the
input
and
the
green
dot
represents
the
output.
So,
for
example,
the
blue
dots
could
be
the
characteristics
of
a
flower,
it
could
be
the
flower
color.
A
The
types
of
the
petals
of
the
flower
and
the
stem
of
the
flower
and
the
green
dot
would
be.
What
you
would
see
is
your
output,
which
describes
the
species
of
the
flower
so,
for
example,
it's
hibiscus
and
the
the
whole
idea
of
finding
the
model
is
to
find
these
layers.
These
hidden
layers,
which
are
these
orange
layers,
these
layers,
which,
with
weights
and
activation
functions,
which,
when
multiplied
with
the
input,
can
produce
the
right
output
that
it
can
say
that,
oh
this
is
hibiscus.
A
So
in
this
example,
the
weight
is
supposed
to
is
called
the
model
parameters
and
the
activation
function,
which
is
user
supplied,
is
called
as
a
hyper
parameter,
and
what
happens
is
that
the
input
is
multiplied
by
this
weight
and
activation
function,
depending
on
the
way
the
arrows
are
attached
to
these
neurons.
And,
finally,
you
evaluate
the
weight.
So
that's
the
deep.
A
So
in
distributed
training,
there
are
two
ways
to
distribute
the
training.
One
is
called
data
parallelism
and
the
other
is
called
as
the
model
parallelism.
So
in
data
parallelism,
what
happens
is
that
in
each
of
the
worker,
in
your
distributed
training,
you
add
the
same
model
to
each
of
the
workers
and
you
split
the
data.
A
So
your
overall
data
is
the
sum
of
the
three
data,
but
you
have
split
the
data
the
and
then
each
worker
is
independently
working
on
its
set
of
data,
and
then
there
has
to
be
some
communication
and
which
I'm
going
to
talk
about
to
make
this
data
parallelism
work.
So
what
we
have
seen
so
far
in
tensor
flow
is
that
all
the
model
frameworks
support
data
parallelism
so
with
the
and
the
model.
A
Parallelism
is
where
you
split
the
model
and
have
a
portion
of
the
model
run
in
one
worker
and
a
second
portion
of
the
model
run
in
the
second
worker.
While
you
are
training
or
the
entire
set
of
the
data
on
that
worker.
So,
as
you
can
see
here
so
although
this
is
not
supported
in
queue
flow
in
what
you
can
do
this
manually
by
pinning
the
each
of
the
sub
sections
of
the
model
to
a
different
device
and
simulate
that
model
framer.
A
But
it's
not
of
the
box
currently
supported
so
within
the
data
parallelism
to
again
dig
a
little
bit
deeper
as
to
figure
out
what
communication
happens
in
context
of
a
data
parallelism.
So
let's
look
at
one
worker
view
and
it
has
this
snapshot
of
data
and
what
what
happens
is
that
the
way
it
works?
Is
it's
a
series
of
forward
and
backward
propagations?
So
in
the
forward
propagation
phase,
what
happens
is
that
the
inputs
get
multiplied
by
as
I
set
up
with
the
weights
and
the
activation
function
to
produce
the
output?
A
So
there
is
no
the
new
communication
needed
across
the
workers
in
the
forward
pass
in
the
reverse,
pass
or
in
the
backward
backward
propagation.
What
happens
is
that
the
output
is
then
compared
with
the
actual
output
and
the
last
function
you
want
to
minimize,
so
you
can
calculate
the
gradient
and
with
that
gradient,
you
try
to
find
the
next
weight.
You
find
a
better
way
than
the
old
order
way
to
minimize.
A
So,
as
you
can
see,
when
we're
finding
the
weight,
we
are
only
finding
the
weight
for
this
section
of
the
data
in
this
worker,
but
then
the
other
other
workers
are
finding
on
a
different
section,
so
you
have
to
combine
or
you
have
to
agree
gate
the
weight.
So
this
aggregation
of
the
weight
is
the
communication
that
happens
in
during
the
distributed
training.
So
now
that
we've
looked
at
that
now,
let's
look
at
how
the
different
layers
can
impact
this,
so
so
to
look
at
the
different
layers.
A
We
want
to
start
with
the
first
layer,
which
is
the
infrastructure
layer,
so
in
the
infrastructure
layer,
let's
start
by
assuming
that
the
model
is
not
distributed.
So
it
is
running
on
one
GPU
in
one
node.
If
we
want
to
distribute
to
a
second
GPU
on
the
same
node,
then
the
the
bandwidth
between
the
GPUs
becomes
more
important
and
so
the
way
the
GPUs
are
connected.
A
That
link
impacts
the
performance,
and
so
it
can
be
a
PCIe
link
or
it
could
be
a
faster
env
link
so
that
that
would
impact
as
you're
scaling
up
within
the
node
and
then,
if
you
are
scaling
from
one
node
scaling
out
from
one
node
to
the
second
node,
then
it's
the
communication
between
the
notes
or
the
links.
There's
this
link
speed
between
the
node,
which
becomes
important,
and
then
it
could
be
anything
like
a
high-speed
Ethernet
playing
or
take,
for
example,
InfiniBand
or
any
of
the
other
links,
and
that
would
impact
the
performance.
A
So
in
the
next
layer,
which
is
the
OS
layer,
this
is
the
layer
which
exposes
it.
D
thinks
a
functionality
to
into
the
kubernetes.
So
in
the
OS
layer,
you
would
want
to
expose
your
the
GPUs.
The
drivers
eye
drivers
to
expose
to
kubernetes,
and
the
second
thing
is,
it
should
be
also
be
able
to
expose
the
env
link,
for
example,
if
you
had
a
faster
link
to
the
upper
layers
and
when
the
selection
is
made.
A
So,
for
example,
if
your
workload
wants
three
GPUs,
it
should
be
able
to
select
the
three
GPUs
which
are
connected
with
the
fastest
link
rather
than
with
some
slow
links,
because
that
would
impact
the
performance
and
then,
as
we
talked
like
as
I
mentioned
about
to
flow
it,
it
is.
It
makes
the
orchestration
of
the
distributed
machine
learning
framework
easier,
and
so
it
provides
so
so.
For
example,
I
talked
about
tensorflow
job
and
the
hyper
parameter
tuning
with
khatib
it.
It
provides
a
simple
interface
and
then
underlying
orchestrate
spot
and
other
resources
for
you.
A
And
finally,
you
run
the
as
part
of
that
tensorflow
job
or
MPI
job.
You
would
run
your
training
container
and
within
the
training
container,
that
you
have
a
training
code.
In
addition
to
that,
you
would
have
a
collective
communication
layer
which
is
basically
used
to
communicate,
and
this
is
what
is
communicating
your
weights.
So
these
weights
could
consist
these
weights
which
are
getting
communicated.
You
can
communicate
through
any
number
of
libraries.
For
example,
there
is
nikkor,
there
is
Qi
RPC
and
each
of
the
model
frameworks
support
many
other
utilities.
A
B
So
you
actually
walked
us
through
the
different
types
of
parallelism.
So
how
do
the
different
training,
pods
or
jobs,
communicate
with
each
other?
One
of
the
common
patterns
that
we
see
is
using
a
parameter
server.
This
can
be
a
part
that
is
separate
from
the
training
pods
or
it
can
be
a
part
of
the
training
part
itself.
So
we
will
see
an
example
of
both
using
chip
flow,
so
this
is
an
example
of
specifying
a
tensorflow
job
using
as
an
explicit
parameter
server.
B
So
you
have
here:
I
have
two
replicas,
so
there
are
two
parameter
servers
and
there
four
replicas
for
the
workers,
so
there
for
training
parts
that
are
running
in
parallel.
The
pods
can
talk
to
the
parameters
of
the
server
that
is
most
closest
to
them
to
be
optimal
in
terms
of
the
bandwidth
and
the
communication
links
the
parameter
servers
then
aggregate
the
weights
within
each
other
and
pass
the
updated
weights
back
to
the
training
pods.
B
If
you
want
to
change
from
this
model
of
a
synchronous,
parameter
service
to
a
synchronous,
parameter
server,
it's
as
simple
as
removing
the
the
service
pack
in
your
TF
job
specification.
So
it's
almost
identical
to
the
previous
specification
that
we
saw
you
just
remove
the
peer.
The
PS
section
and
I
have
increased
the
number
of
replicas
here
just
to
make
the
picture
look
nice.
So
we
have
four
replicas.
There
is
no
explicit
parameter
server
here,
but
all
the
pods
are
communicating
with
each
other
all
the
time.
When
would
you
use
one
versus
the
other?
B
So
if
you
have
fast
communication
links
between
the
pods,
this
model
is
preferred
more
and
more.
This
model
is
preferred
because
you
have
all
the
pods,
always
in
sync:
they
all
see
the
same
weights
and
there's
no
drift
between
them.
But
if
you
don't
have
a
fast
communication
link
between
the
pods,
it
gets
very
slow
to
do
this.
So
you
want
to
use
an
asynchronous
parameter
server.
We
will
look
at
distributed
training
using
an
MPI
job
spec
in
cube
flow.
The
reason
being
MPI
allows
you
to
use
multiple
frameworks,
so
this
is
a
TF
job.
B
Specification
tensorflow,
with
2.0
allows
you
to
do
fast,
communication
across
nodes
and
within
a
node,
but
that's
only
available
with
intensive
flow.
So
if
you
want
to
use
the
same
functionality
with
PI,
torch
or
other
frameworks,
it's
easier
to
use
the
MPI
model,
so
how
does
cube
flow
implement
the
MPI
job?
This
is
a
very
high
level
architecture
diagram.
So
you
have
a
custom
resource
width,
which
is
the
MPI
job.
B
It
it
composes
of
two
primary
components:
one
is
the
launcher
and
the
second
one
is
the
worker,
the
launcher
acts
as
the
orchestrator
of
the
workers,
so
it
creates
multiple
worker
pods
and
handles
all
the
coordination
between
them.
How
does
this
specification
look?
So
here
we
have
the
launcher.
I'm
saying:
there's
a
replica
of
one
and
I
have
slots
per
worker
is
one
so
there's
a
little
bit
of
tweaking
required
here.
B
So
if
you
have,
the
slots
per
worker
specifies
how
many
tasks
you
want
to
run
in
one
pod,
so
I
have
one
GPU,
so
I'm
running
one
slot
in
one
worker
and
the
MPI
run
command
here
says
in
total.
I
have
to
process
only
two
roses,
because
I'm
running
only
two
workers
and
each
of
them
has
a
single
slot.
The
worker
definition
here
is
very
similar.
Have
two
replicas
and
each
of
them
has
one
GPU,
so
they
have
to
request
one
GPU
from
kubernetes.
B
This
is
similar
to
any
kubernetes
resource
request,
definition,
the
only
difference
being
with
GPUs.
You
can't
really
have
a
high
and
a
low
limit.
You
just
specify
what
you
want
and
that's
all
you
get
if
you
want
to
scale
the
MPI
jobs,
it's
just
as
simple
as
changing
the
number
of
workers
that
you
have
and
the
number
of
GPUs
that
you
have
so
the
same
specification
that
we
saw
before
I
have
updated
it
to
have
three
workers,
and
each
of
them
have
three
replicas
right.
B
So
there's
nine
processes
that
are
running
in
parallel
and
all
of
them
are
running
the
tensorflow
benchmark,
ResNet
model
and
they
are
communicating
to
each
other
using
MPI.
So
that
is
why
the
variable
update
of
the
parameter
server
update
mechanism
is
specified
as,
however,
if
you
want
to
scale
up
that
has
increased
the
number
of
slots
in
your
worker
and
increase
the
number
of
GPUs
that
are
available.
You
basically
update
the
resources
that
is
specified
here
and
again.
This
is
the
scaling
out
part
where
I've
increased.
B
B
We
just
ran
it
and
deleted
it.
So
so
this
is
a
tensorflow
benchmark
that
I'm
running
it's
an
MPI
job.
We
look
at
the
specification
while
the
pods
are
starting
very
similar
to
what
we
talked
about.
We
have
one
pod,
one
function
that
is
running
in
one
worker.
You
have
one
one
launcher:
two
workers
and
the
workers
have
one
GPU
that
is
available.
We
can
look
at
the
log
launcher
pod
logs
and
it
tells
you
that
it
is.
B
B
This
is
better
okay,
so
yeah,
so
we
have
yeah
the
first
part
that
is
starting
here.
The
second
part
here
you
will
see
each
of
the
devices
being
identified,
so
we
will
have
a
Tesla
v
100
that
comes
up
here
so
and
then
you
will
see
the
actual
job
running
and
the
training
happening
takes
about
a
minute
because
I'm
just
running
a
sample
job
at
the
end
of
which
it
will
get.
It
accumulate
the
communication
between
the
worker,
zero
worker
one
and
give
you
the
end
result
of
your
model.
B
We'll
skip
the
slides
because
we
already
saw
that
so
that's
how
you
implement
a
distributed,
training
using
cube
flow.
We
just
saw
one
example
of
MPI
job,
but
it's
very
similar,
irrespective
of
what
operator
you
choose,
TF
job
is
very
similar.
Any
model
is
very
similar
right,
so
we
trained
a
model.
But
how
do
you
start?
How?
Where
did
that
model
come
from
right?
So
you
have
a
user
who's.
B
Looking
at
that
data
and
saying,
okay,
this
is
the
sort
of
architecture
that
will
fit
my
data
right,
so
they
take
an
architecture,
see
run,
train
it
and
see
how
well
it
is
performing
it
doesn't
perform.
Well,
they
change
it,
tweak
it
a
little
bit,
get
a
next
model
and
then
keep
repeating
the
process
until
they
finally
find
a
model
that
works
for
them,
and
you
this
process
is
very
laborious
and
manuals.
B
So
you
want
to
reduce
the
amount
of
human
effort
involved
there
and
the
way
cube
flow
allows
you
to
do
that
is
using
something
called
Kathy.
Kathy
is
a
hyper
parameter,
tuner
and
a
neural
architecture.
Such
this,
we
can
argue
about
whether
neural
architecture
search
is
also
high.
Povera
major
tuning,
but
it
basically
does
both
it
lets.
You
identify
the
best
hyper
parameters
and
the
best
models
for
your
data.
B
It
does
that,
but
it
does
that
in
a
way
using
multimodal
parallelism.
So
your
you
have
multiple
models
or
a
same
model
with
different
hyper
parameters
and
you
training
them
on
multiple
devices.
Multiple
nodes
using
multiple
GPUs,
so
you're
distributing
each
of
your
model.
Training
but
you're
also
running
it
in
parallel
with
multiple
models
again
doing
this
in
cube
flow
is
very,
very
simple,
so
we
look
at
how
it
act.
How
Cathy
actually
achieves
this?
B
There
is
something
called
a
model
manager
which
handles
this
job
of
creating
the
models
and
sending
it
down
to
a
trial
controller.
The
triangle
controller
takes
each
of
the
models,
takes
your
data
and
trains
them.
It
then
evaluates
how
well
they're
performing
and
creates
this
the
feedback
to
the
suggestion,
controller,
which
creates
which
tells
the
model
manager
what
type
of
model
or
what
type
of
modification
it
needs
to
make,
or
it
gives
new
hyper
parameters
that
you
want
to
try
out.
B
This
process
continues
until
either
khatib
finds
the
model
that
fits
your
requirements
or
it
exhausts
the
budget
that
you
have
specified
for
it.
So
this
is
an
example
of
how
you
specify
a
job
in
Khateeb,
so
we
here
we
have
you're
creating
an
experiment,
so
you're
saying
here
that
create
three
create
maximum
twelve
trials,
run
at
most
three
trials
in
parallel.
B
So
I'm
running
this
on
my
laptop
and
I,
don't
really
want
to
run
thousands
of
trials
in
parallel
that
have
a
small
spec,
and
then
you
say
what
is
your
objective,
so
you
want
to
maximize
the
validation
accuracy
here.
You
can
set
any
objective
that
you
want
any
data,
that's
output
by
your
model
and
here
I'm
asking
it
to
identify
the
best
neural
architecture
so
that
layer
them
name
is
neural.
Architecture
search.
Khateeb
supports
a
lot
of
hype,
a
parameter
tuning
model,
so
you
can
do
random
search.
B
The
job
specification
for
khateeb
is
very
similar
to
the
job
specification
that
we
saw
for
the
MPI
job.
You
just
say
what
type
of
a
job
it
is
either
a
built
in
kubernetes
resource
like
a
pod
or
a
job,
or
it
can
be
a
cube
flow
custom
resource
like
a
TF
job,
and
here
again
I'm
saying
this
is
how
you
specify
a
GP
you
to
be
used.
B
This
is
an
example
of
a
cue
flow,
neural
architecture,
search
so
not
very
clear,
but
you
can
see
that
every
suggestion
generates
the
number
of
layers,
the
type
of
activation
functions
that
you
want
to
use,
how
many
neurons
are
activated
in
each
layer
and
runs
them
too
good
through
the
trials.
We
didn't
want
to
do
that
life
because
it
takes
quite
some
time.
B
This
is
an
example
of
that
running
and
with
the
view
of
the
different
nodes
on
which
it
is
distributed,
so
you
can
very
clearly
see
that
it's
run
across
multiple
nodes,
so
very
easy
to
do.
Distributed
hyper
parameter
tuning
using
Kathy
alright,
so
we
have
looked
at
model
training.
How
do
we
distribute
it?
We've
looked
at
hyper
parameter
tuning
now.
If
Kathy
makes
it
all
so
super
easy,
do
we
need
to
do
anything
else?
B
B
What
do
I
use
right
and
the
answer
is
it
depends
it
depends
on
what
you
are
trying
to
do,
what
kind
of
resources
you
have
and
how
much
you
want
to
share
if
you
have
a
single
dedicated
user
or
team,
to
whom
you
want
to
give
all
your
GPU
resources,
then,
where
metal
may
be
the
best
option
to
go,
because
you
can
get
the
last
bit
of
performance
out
of
your
GPUs.
There
is
no
interference,
there's
no
sharing
right,
but
then
you're
effectively,
locking
down
GPUs,
which
are
potentially
a
very
costly
resource
right.
B
So
if
you
want
to
avoid
that-
and
if
you
have
multiple
users
with
whom
you
want
to
share
using
virtualization,
is
a
very
easy
mechanism,
if
you
combine
virtualization
with
PCI
pass-through,
you
get
near
native
bare-metal
performance,
so
there's
absolutely
no
difference
in
the
performance
that
you
see
with
bare
metal
or
PCI
pass
through
using
GPUs.
If
you
combine
that
with
auto
scaling
now,
you
are
able
to
share
your
GPUs
across
different
clusters.
Somebody
creates
a
pod
or
a
workload.
B
They
ask
for
two
GPUs,
the
entire
workload
runs
and
the
workload
is
done
so
your
autoscaler.
Does
it
says:
okay,
I,
don't
need
this
GPO
Nord
anymore.
It
removes
it
and
that's
now
available
for
another
part
to
use
in
another
cluster.
Potentially
what
about
V
GP
is
when
de
you
use
V
GPUs.
If
you
have
diverse
types
of
workloads
that
you
are
trying
to
run
so
you're
not
doing
all
model
training,
so
it
may
be.
Some
people
in
your
team
are
doing
training.
Some
people
want
to
do
inference.
B
Some
people
want
to
just
run
a
Jupiter
notebook
and
do
data
exploration.
All
of
these
have
different
compute
and
GPU
resource
requirements.
So
in
such
cases
it's
easier
to
divide
and
partition
the
GPU
into
pieces
by
using
V,
GPUs
or
virtual
GPUs.
So
there's
a
component
that
goes
into
your
hypervisor
layer,
which
takes
a
physical
GPU
and
splits
it
up
into
fractional
DPS.
So
you
can
say:
I
have
an
inference
serving
part.
B
It
needs
one
tenth
of
a
GPA
and
when
that
part
is
done,
that
one
tenth
is
freed
up,
you
could
again
combine
the
entire
GPU,
give
it
to
someone
else.
So
it's
easy
to
share
your
resources
across
a
diverse
set
of
TAS.
If
you
have
different
kinds
of
workloads
that
are
running
in
your
clusters
and
let's
go
a
little
bit
deeper,
then
how
you
set
up
your
service?
How
does
the
communication
actually
happen
within
the
server?
So
this
is
a
topology
of
the
GPUs
on
one
of
our
servers
see
480
ml
from
Cisco.
B
It
has
an
8
GPUs.
There
are
many
such
servers
that
are
available
with
a
GPUs
which
you
can
see
which
have
similar
topology
restrictions.
So
what
this
gives
you
is
a
6
env
link
lanes
connectivity
between
multiple
GPUs.
It
also
has
multiple
pcs,
which
is
to
connect
the
GPUs
to
the
CPUs,
so
you
can
have
fast
communication
paths.
Why
does
it
matter
right?
So
each
of
the
env
link
lanes
gives
you
25
gigabits
of
bandwidth
together.
All
of
them
give
you
the
six
of
them,
give
you
300
gigabytes
of
bandwidth.
B
If
you
were
to
take,
compare
it
to
just
a
PCIe
link
that
just
gives
you
30
GB,
it's
like
a
10x
improvement
in
performance.
If
you
have
time,
we
will
do
a
demo
that
will
actually
show
how
much
of
a
difference
it
makes.
So
if
you
have
multiple
GPUs
that
are
really
connected
using
fast
interconnects
and
you're,
doing
things
like
distributed,
training
which
have
a
lot
of
which
are
using
the
synchronous
parameter
server
model
where
all
the
quads
are
talking
to
each
other
all
the
time.
It
makes
a
huge
difference.
What
GPUs
you
pick
right.
B
If
you
pick
the
GPUs
that
are
really
tightly
connected
it,
you
can
be
really
fast.
If
you
pick
GPU
0
here
and
GPU
8
here
as
the
two
GPUs
you
give
to
your
pod,
then
it's
basically
not
it's
just
the
same
as
running
them
on
two
different
nodes
right,
so
you're
missing
out
on
the
fast
connect.
So
it
helps
to
be
cognizant
of
the
resources
underneath
as
much
as
cube
flow
tries
to
abstract
it
away
from
us.
We
also
need
to
have
little
bit
of
awareness.
B
There
are
improvements
going
on
in
the
community
in
the
form
of
the
changes
to
the
topology
manager,
which
allows
you
to
do
placement
which
automatically
do
placement
within
a
cluster
by
picking
the
optimal
GPUs.
But
again,
if
you're
dividing
it
out
across
different
clusters,
then
you
still
need
to
understand
this.
I,
don't
think
we
have
time
for
a
demo.
So
let
me
conclude
quickly,
so
we
basically
we
saw
how
cube
flow
lets.
B
You
take
data,
be
processing,
training,
hyper-v,
remaining
model
serving
any
of
these
tasks
in
your
machine
learning,
workflow
and
easily
distributed
across
multiple
nodes
across
multiple
GPUs
right,
so
one
how
many
of
you
have
used
something
like
this
right,
so
I
was
thinking
of
what.
How
do
we,
what
do
I
save
cube
flow
is
right,
I've
heard
of
it
referred
to
as
a
buffet
of
different
things
right,
because
when
you
come
to
cube
flow,
it
looks
like
you
have
everything
thrown
in
there's
just
so
many
operators
there
they're
doing
so
many
different
things.
B
But
it's
it's
exactly
like
this.
It's
like
a
drill,
electron
electric
drill,
so
you
have
lots
of
bits
there.
What
it
does
is
gives
you
a
standardized
interface
right,
so
you
have
cube
flow
and
you
take
any
of
these
different
frameworks.
You
have
the
same
knobs.
You
increase
the
speed
on
your
drill.
It
doesn't
matter
what
bit
you're
plugged
in
it's
the
same
way
to
do
it.
B
It
makes
it
easy
for
you
to
switch
from
one
to
another,
to
give
a
standardized
environment
to
all
your
consumers
of
your
cluster
and
to
take
advantage
of
new
things
that
are
coming
in
just
as
easily
with
that.
I
would
like
to
thank
everyone
who
was
contributed
to
cube
flow
because
without
you,
we
wouldn't
be
here
huge
thanks
to
the
Cisco
cube
flow
team
that
helps
us
with
our
customer
engagements
answers,
questions
and
helps
us
anytime.
We
need,
and
also
to
the
Cisco
UCS
team,
who
gives
us
hardware
and
again
they're
available
anytime
they're
awesome.