►
From YouTube: Democratizing Machine Learning on Kubernetes [I] - Joy Qiao & Lachlan Evenson, Microsoft
Description
Democratizing Machine Learning on Kubernetes [I] - Joy Qiao & Lachlan Evenson, Microsoft
One of the largest challenges facing the machine learning community today is understanding how to build a platform to run common open-source machine learning libraries such as Tensorflow. Both Joy and Lachie are both passionate about making machine learning accessible to the masses using Kubernetes. In this session they'll share how to deploy a distributed Tensorflow training cluster complete with GPU scheduling on Kubernetes. We’ll also share how distributed Tensorflow training works, various options for distributed training, and when to choose what option. We’ll also share some best practices on using distributed Tensorflow on top of Kubernetes, based on our latest performance tests performed on public cloud providers. All work presented in this session will be accessible via a public Github repository.
About Lachlan Evenson
Lachlan Evenson is a cloud native evangelist and mercenary. Lachlan has spent the last two and a half years working with Kubernetes and enabling cloud native journeys. He is a believer in open source and is an active community member. Lachlan spends his days helping make cloud native projects run great on Azure.
About Joy Qiao
Joy Qiao is a senior solution architect in the AI & Research Group at Microsoft, where she is responsible for driving end-to-end AI/ML solutions on Azure among the partner eco-system. Joy has over 15 years of IT industry experience including 11 years at Microsoft working as technical lead/architect roles at various Azure & AI engineering teams, as well as senior consultant/architect in the Microsoft services team. Joy has mainly been focusing on Microsoft Azure, Big Data and Machine Learning technologies, leading and delivering Machine Learning, Big Data and Cloud-based solutions for both internal and external MS enterprise customers and partners.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
Hello,
hello,
hello,
how's,
everybody
doing
good
good
Friday
afternoon
Friday
afternoon,
I
just
spoke
next
door.
I
had
to
run
over
I'm
sorry
for
being
a
minute
late.
So
let's
get
started,
come
take
a
seat
all
right,
I'm,
talking
about
a
topic
here
with
joy
today
that
I'm
really
excited
about
I
know.
Joy
is
as
well
so
democratizing
machine
learning
on
kubernetes
and
I.
Don't
know
what
you
want!
A
What
you
expect
when
you
see
that
title,
but
we're
going
to
talk
about
a
journey
that
we've
both
had
speaking
from
the
heart
and
some
different
things
that
we've
seen
and
then
give
you
some
best
practices
for
what
we've
learnt
and
how
we
propose.
We
might
move
together
as
a
community.
So
with
that
I
will
hand
it
over
to
my
lovely
colleague,
joy
to
introduce
yourself
hi.
B
A
A
Can
you
help
me
I,
hear
kubernetes
a
great
place
for
me
to
run
machine
learning,
workloads
she'd
already
taken
a
great
look
around
and
had
a
lot
of
things
running,
but
we
took
a
look
together
and
what
we
found
is
what
we're
going
to
present
and
see
see
what
you
all
think,
but
I
represent
the
SRA,
so
I
build
and
maintain
the
infrastructure.
I
know
how
to
build.
Kubernetes
I
know
how
it
should
look,
but
I
have
no
machine
learning
experience
right.
A
What
I
would
like
to
propose
here
that
this
is
a
quite
a
common
occurrence
in
this
space
you
have
highly
talented
people
at
the
infrastructure
level
who
know
how
to
build
kubernetes
with
their
eyes,
sharp
and
run
it.
And
then
you
have
a
team
of
more
than
capable
wonderful
data
scientists
chomping
at
the
bit
to
use
this
infrastructure
that
you've
built.
The
problem
is
that
in
the
middle
of
that,
then
Venn
diagram
lives,
no
one
right,
so
nobody
lives
there.
So
what
we're
looking
to
do
with
this
talk
is
start
conversation
around.
A
How
do
we
get
people
in
that
middle
middle
point
so
that
the
data
scientists
can
get
the
best
out
of
kubernetes
as
a
platform
to
run
the
machine?
Learning
deeply
Newell,
Network
workloads
and
me
have
confidence
that
I'm
actually
building
the
most
efficient
platform
for
them
to
utilize,
and
this
is
where
the
revelation
is
I
won't
chomp
at
the
bit
here.
So
what
I'm,
seeing
in
the
environment
and
working
with
joy
is
the
right
tools
are
out
there
to
run
machine
learning.
A
The
tooling
is
already
there
and
kubernetes
is
the
right
platform,
but
we
have
two
distinct
set
of
people
that
know
two
different
knowledge
pillars
with
not
nobody
in
the
middle
right.
So
how
do
we
actually
start?
This
conversation
between
infrastructure
engineers,
who
can
lay
down
kubernetes
and
the
data
scientists
who
want
to
do
want
to
put
kubernetes
to
the
test,
which
is
what
I
was
really
excited
about.
A
A
A
So
thinking
thinking
about
that
just
digging
down
into
what
we
do
here,
how
do
we
lower
the
barrier
so
when
we
say
democratizing
I
think
there's
a
whole
world
and
I
believe
working
with
joy
that
there's
a
whole
world
of
folks
chomping
up
a
bit
to
get
out
this?
They
have
neural
networks
to
build
and
there's
great
minds
out
there
to
build
them,
but
the
barrier
to
entry
is
still
very
high
for
them
to
be
able
to
get
this
knowledge
out
of
their
head
and
actually
start
putting
it
to
the
test.
A
So
how
do
we
get
some
best
practices
and
create
some
content
out
there?
So
people
don't
stumble
around
as
as
much
as
I
did
I,
don't
know
if
anybody
else
did
but
to
get
there.
So,
let's
get
started,
but
we're
going
to
take
you
on
a
tour
of
deep
learning,
because
I
think
one
of
the
things
that
trips
me
up
at
least
is
naming
and
nomenclature.
A
lot
of
these
things
are
very
deeply
related,
but
they
have
different
names
in
the
ml
world
and
different
names
in
the
infrastructure
world.
A
B
Thank
you
Lackey,
so,
before
I
dive
into
the
nitty-gritty
details
about
doing
deep
learning,
especially
distributed
deep
learning,
which
is
actually
much
harder
than
a
single
node
deep
learning,
training
on
kubernetes
I
would
like
to
just
to
provide
a
quick
context
on
the
typical
distributed
training
architecture
on
multiple
nodes.
So
there
are
mainly
in
a
deep
learning
space.
Specifically,
there
are
mainly
two
ways
of
doing
distributed:
training
if
you
want
to
scale
out
to
multiple
nodes,
one
most
popular
way
is
actually
data
parallelism.
B
Basically,
if
you
have
multiple
GPUs
sitting
across
multiple
nodes,
the
way
you
do
deep
distributed
training
for
deep
learning
is
basically
you
place
a
copy
of
your
deep
neural
net
model
within
each
every
single
GPU
that
you
have,
and
then
you
feed
a
mini
batch
of
the
entire
training
data
once
one
batch
at
a
time
into
each
every
single
GPU
and
each
GPU.
Basically
just
works
on
that
mini
batch
of
the
data
is
being
fed
to
and
train
a
model
one
iteration
at
a
time
and
at
the
end
of
each
iteration.
B
All
of
all
of
those
gradients
there
are
sent
across
all
those
GPUs
and
do
a
central
elevation
and
then
update
a
model
globally
and
then
push
back
the
updated
version
of
the
model
to
each
GPU
again
and
then
the
whole
process,
the
whole
iteration
starts
again.
So
this
is
the
way
a
typical
data.
Parallel
parallelism,
distributed,
training
architecture
works.
B
The
second
way
is
called
model
or
parallelism,
and
this
happens
when
you
have
a
model
that
is
too
big
to
fit
into
one
single
GPU,
and
this
normally
happens
when
you
have
a
big
large
Miron,
and
that
is
doing
some
natural
language
processing,
big
LSP
hermanos
or
those
big
neural
machine
translation
models.
Those
are
typically
pretty
big
and
cannot
fit
into
one
GPU.
B
So
this
picture
I'm
sure
if
you
have
used
tensorflow
distributive
training,
you
must
have
seen
a
architecture,
diagram
alot,
and
this
is
a
typical
setting.
When
you
do
tensile
flaw
and
distributive
training,
you
can
have
one
or
more
parameters
servers
sitting
at
the
top,
and
then
you
have
multiple
GPUs
across
multiple
nodes.
One
thing
to
call
out
is
tensile
flow
has
done
some
basic
optimization
in
terms
of
the
gradient
that
are
being
sent
from
each
GPU,
each
worker
node.
That
is
doing
the
work
with
multiple
GPUs.
A
Fantastic,
thank
you.
Joy,
I
think
one
of
the
things
just
in
hearing
those
three
slides.
That
was
a
revelation
to
me
as
an
engineer
knowing
where
the
bits
get
pushed
and
crunched
and
knowing
I
can
already
mentally
map
how
that
lays
down
on
a
kubernetes.
So
what
joy
actually
did
for
me
and
open-source?
A
That
was
just
a
bunch
of
steps
to
actually
help
me
build
out
and
map
what
she
was
doing
so
she's
created
a
very
lightweight
kind
of
in
the
weeds
set
up
for
us,
so
that
we
can
see
a
very
low
level
of
how
this
thing
hangs
together
without
making
it
overly
complex,
so
bring
any
kubernetes
cluster
on
your
provider
of
choice.
This
will
work,
but
what
we're
doing
here
is
with
kubernetes.
We
make
sure
that
we
have
availability
to
the
GPUs.
So
if
you've
ever
used
GPUs
in
kubernetes,
they
are
scheduled
as
a
resource.
A
You
see
them
there,
but
just
because
they
show
up
doesn't
mean
you
have
all
your
house
in
order
on
the
nodes
right,
so
you
need
the
libraries
to
access
the
GPU
hardware,
whatever
that
be
a
common
common
one
is
Nvidia,
so
you
need
to
make
sure
that
you
contain
a
runtimes,
actually
have
access
to
utilize
that
hardware
right.
So
just
because
you
can
see
that
you
have
four
GPUs
doesn't
mean
your
workloads
actually
have
access
to
those
GPUs.
A
So
there's
a
script
inside
that
repo
that
will
actually
go
and
on
your
nodes,
you
can
and
make
sure
that
your
GPUs
are
in
the
right
place
and
shape,
and
then
you
can
run
this
or
file.
So
there's
a
I'll
quickly,
just
pop
over
to
this
one
yeah,
okay,
fantastic,
thank
you
and
we'll
have
we'll
have
the
links
here
at
the
end,
but
for
me,
joy,
putting
this
together
and
actually
I
could
get
in
the
weeds.
I
know.
A
Lots
of
folks
are
like
this,
but
knowing
what
a
PS
0
and
a
PS
1
wasn't
a
worker
0
and
a
worker
1
in
the
context
of
what
we
just
learnt
was
incredibly
I.
Was
incredibly
grateful
that
I
had
that
knowledge,
because
I
could
start
to
peel
back
how
I
could
design
an
infrastructure
for
what
her
workload
was.
So
you
can
work
through
this
at
your
own
pace,
but
what
it
ends
up
is.
Is
you
have
a
parameter
server
in
several
workers
and
number
of
workers?
A
We
run
a
Python
script
manually
inside
them
all
and
describe
where
the
parameter
servers
and
the
workers
are,
and
this
kicks
off
the
process
that
joy,
just
detailed.
But
what
you
can
see
in
there
is
you
can
see
the
number
of
batches
we
want
to
serve
up
and
the
different
models
which
is
going
to
be
great
joy,
is
going
to
take
us
into
what
we
actually
saw
when
we
did
a
lot
of
benchmarking
and
testing.
So
this
is
available
to
you
and
we'll
have
the
links
there
at
the
end.
A
So,
finally,
if
you
do
a
describe
on
your
nodes,
you
will
see
GPUs
if
they're
available,
that
is
that
features
tool
and
alpha
I
believe,
but
you
can
do
that,
and
most
some
providers
gke
have
access
a
CS
to
GPU
nodes.
You
can
run
this
up
and
start
playing
with
it
today.
Now
I'm
going
to
hand
it
back
over
to
joy
and
she's
going
to
take
us
through
what
we
actually
found
together
as
we
built
this
infrastructure.
B
Yeah
so
artists
to
share
with
you
some
of
the
key
learnings
we
observed
when
we
were
doing
a
lot
of
training
on
kubernetes,
especially
doing
a
lot
of
the
benchmarking
test,
tuning
various
different
knobs
and
auctions
and
and
what
we've
learned
so
far.
So
just
to
let
you
know
quickly
the
training
environment
we
used
for
the
benchmark
testing.
We
did
a
couple
of
months
ago.
I
was
using
basically
two
worker
nodes.
B
Each
worker
has
four
Tesla
ki
t
GPUs
and
I
was
using
one
just
GPU
VM
for
my
parameter
server
since
parameter
server,
as
you
have
seen.
Basically,
its
job
is
to
do
a
lot
of
the
gradient,
aggregation
and
update.
It
actually
doesn't
do
the
amount
of
training
itself,
so
the
parameter
server
doesn't
does
not
really
need
the
GPU.
So
I
was
just
using
a
CPU
server
for
that
and
we
were
using
the
rail
image
all
the
whole
image
net
data
set
and.
B
Correct
so
before
we
go
into
the
distributed
training
just
to
show
you
what
we
observed
on
a
single
part
with
multiple
GPUs,
as
you
can
see
mr.
chart
showing
here
you
pretty
much,
you
can
get
a
pretty
good
linear,
scalability
here
and
I'm,
showing
a
res
not
here
just
for
an
example
but
actually
the
same.
Pretty
predictable.
Linear
scalability
applies
to
a
lot
of
different
models.
We
ran
and
tested,
so
vgg,
60
and
Inception
v3
Google
met
different
versions
of
Rasmus.
B
We
try
vanilla,
we
try
them
all
and
they
they
all
scale
from
one
GPU
to
two
to
four
on
a
single
node
pretty
nicely
and
all
the
GPUs
are
fully
saturated,
which
is
actually
good.
That
means
you're
not
wasting
your
GPU
resources
and
for
the
distributed
training
testing.
Just
a
quick
settings,
sortie
I
was
using
just
one
parameter
server
and
two
workers
and
I
was
doing
the
async.
B
A
gradient
update
between
because
of
the
communication
between
the
parameter
server
and
the
worker
node
can
happen
in
a
synchronous
fashion
or
in
an
asynchronous
fashion.
I
think
sometimes
can
help
you
if
you,
especially
if
you
have
slow
network
bandwidth.
That
means,
if
you're,
using
a
think
the
worker
each
worker
does
not
have
to
wait
for
the
rest
of
the
workers
to
all
finish.
Their
batch
of
the
data
before
the
worker
can
move
on
to
the
next
batch.
B
The
communication,
the
synchronizer
it
does,
it
can
happen
asynchronously,
but
sometimes
using
async
can
also
hurt
your
speed
to
accuracy,
meaning
the
primary
server.
Sometimes
you
can
get
a
stale
model
coming
from
a
slower
worker
node,
which
might
be
having
some
network
issues
so
the
the
model,
your
training,
the
performance,
might
go
up
and
down
a
little
bit
if
you're,
using
async
and
and
in
this
particular
testing,
the
parameter
server,
each
parameter
server
and
each
worker
part
has
their
own
dedicated
host.
B
So
they're
not
competing
for
resources
in
this
particular
and
I'm,
not
using
anything
that
Network
just
pure
Ethernet,
our
GPR
CD
protocol.
So
here
is
the
distributed.
Testing
results,
as
you
can
see.
Actually
we
have
tested
different
models
here
and
you
can
see
different
models
actually
have
very
different
behaviour
when
you
scale
out.
So
it's
not
necessarily
that
you
will
always
get
the
linear
scalability
in
this
case,
as
you
can
see,
Google
net,
which
is
the
very
left
here.
B
A
I
think
this
was
when
I
first
saw
these
results.
This
is
a
bit
of
a
revelation
because
I
was
under
the
impression
that
you
could
just
throw
more
hardware
at
the
problem,
but
if
these
models
aren't
built
to
take
advantage
of
that
distributed,
hardware
you're
just
wasting
your
time.
So
a
lot
of
effort
on
my
part
was
wasted.
Building
larger
clusters
with
bigger
GPUs
and
putting
more
of
them
out
there
when
the
model
couldn't
actually
saturate
them.
A
So
you
can
see
there
that
first
diagram,
Google
node,
actually
linear
scales
in
the
distributed
fashion,
but
over
to
the
right
there's
a
model
there,
that's
actually
worse.
When
you
run
it
distributed
right
and
some
of
1.6
the
average
their
Google
nets,
the
outlier
there.
The
rest
is
not
linearly
scaling.
So
we're
going
to
talk
a
little
bit
more
about
that.
B
So,
just
to
dive
a
little
deeper
on
why
those
models
are
behaving
very
differently
when
you
scale
out.
Basically,
it
all
comes
down
to
in
one
single
sentence:
a
critical
characteristic
of
your
actual
model
architecture,
which
is
the
compute
over
communication
ratio
of
the
model.
Basically,
those
two
factors:
how
much
GP,
how
much
GPU
resources
your
MOU
takes
to
get
trained
and
also
how
many
parameters
your?
How
many
layers
your
Mountaineer
Annette
model
architecture
has
it
has
a
dictating
factor
or
how
much
network
bandwidth
it
needs
to
consume.
B
So
the
compute,
our
communication
ratio
of
the
model
really
decides
how
well
your
model
can
scale
when
you
try
to
scale
out
when
you're
doing
the
training
and
as
you
can
see,
Google
net
really
has
the
highest
ratio
here.
So
that's
why?
Obviously,
we
we
saw
during
our
real
training
as
well.
It
does
scale
pretty
well,
whereas
the
other
models
they
have
lower
ratio
and
they
don't
scale
as
as
well
and.
A
A
There
is
traditional
ml
things
like
spark
which
are
disk
intensive
and
then
there's
distributed
deep
learning,
neural
networks,
which
is
the
more
modern,
some
more
modern
approaches
which
are
actually
moving.
These
data
sets
around
the
network.
Crunch
move
crunch
move,
so
it's
actually
the
move
of
that
data
after
the
models
been
trained,
that
it
actually
causes
that
the
whole
model
to
slow
down.
How
fast
can
I
get
the
train
model
these
layers
pushed
around,
so
everybody
else
knows
about
them.
A
So
again,
a
revelation
to
me
here
we
are
with
40
gig
NICs
right
so
to
actually
move
this
around.
Otherwise,
the
hardware
also
I
think
you
know
in
the
parameter
server
model,
as
you
scale
out,
more
things
get
go
to
the
parameter
server,
so
your
parameter
server
actually
becomes
your
bottleneck
because
everybody's
trying
to
shoot
them,
40
gigabits
per
sec
and
and
he's
only
got
a
40
kid
gigabit
per
second
Nick.
So
again,
I
thought
you
could
throw
more
hardware
at
the
problem
and
actually
made
the
problem
worse
right.
B
So
some
other
observations
are
that
we
wanted
to
call
out
when
we
were
doing
the
training,
in
addition
to
the
network
bandwidth
and
obviously
the
compute
our
communication
ratio
of
your
actual
model.
There
were
so
as
a
obviously
Cooper
natty
admin
when
you're
supporting
your
data
scientist
with
your
kubernetes
cluster,
they
are
doing
their
training
on
your
kubernetes.
The
one
big
important
thing
to
monitor
is
whether
they
are
saturating.
B
You
are
their
GPUs
or
not,
because
if
they
their
GPUs
are
basically
sitting
there
idle,
not
getting
saturated,
then
you
want
to
think
twice
before
you
throw
more
hardware
and
the
problem
and
also
another
thing
that
I
was
having
to
basically
try
different
options
and
wasn't
really
very
intuitive.
For
me
to
come
up
with
an
optimal
design
at
the
beginning.
Was
the
correct
ratio
between
the
price
of
the
worker
to
the
parameter
servers?
B
B
It
is
important
to
note
that
it
is
not
replacing
tensile
flow
or
just
yet
another
fourth
of
canceled
flow.
It
is
actually
a
standalone
Python
package
that
works
on
top
of
tensile
flow,
so
you
don't
have
to
worry
about
getting
distracted
or
getting
detached
from
the
mainstream
kinds
of
flour.
This
is
just
another
nice
standalone
package
that
works
with
tensile
flour
and
it
solves
a
lot
of
the
the
network,
bottleneck
issues
and
also
the
hard
decision,
or
was
the
correct
ratio
between
your
parameter
server
and
worker.
B
It
solves
a
lot
of
those
problems
by
using
the
ring
or
reduced
algorithms,
so
that
all
of
those
workers,
instead
of
sinking
and
top
standing
their
parameters,
those
gradients
to
a
centralized
primary
server.
They
can
all
just
talk
directly
among
each
other.
So
now
with
hora
world,
you
don't
need
even
to
specify
how
many
parameters
servers
you
need
anymore.
There
is
no
parameter
server.
All
the
workers
can
just
talk
directly
with
each
other
among
themselves,
using
this
ring
or
reduced
algorithm.
B
That
is
highly
efficient
at
communic
note,
internal
or
internal
GPU
communication,
and
it
is
using
this
nickel
and
nickel
2
library
from
nvidia.
Nickel
1
is
for
internal
GPU
communication
inter
nico
to
is
for
internal
a
GPU
communication,
and
they
all
uses
ring
all
radios
to
talk
to
each
other
directly
and
also.
B
It
basically
because
all
the
workers
now
can
talk
directly
to
each
other
from
the
network
bandwidth
perspective.
You
also
basically
avoid
a
lot
of
the
other
network
bandwidth
bottleneck.
We
were
running
into
earlier,
so,
as
you
can
see
from
the
chart
here,
if
you're
running
horrible
on
top
of
tensorflow,
you
get
a
much
better,
a
much
more
predictable
linear
scalability
when
you
scale
out,
and
it
doesn't
matter
what
model,
what
your
specific
model
architecture
is
anymore,
as
you
can
see
whether
you
using
instruction
v3,
ResNet
or
v2
g16.
B
Your
model
can
all
pretty
much
scale
pretty
nicely,
and
in
this
way,
since
you
get
better
predictability
of
the
scalability,
you
then
can
design
in
a
much
easier
way
how
exactly
how
many
workers
we
need,
how
many
GPUs
you
need
instead
of
guessing
and
testing,
and
another
thing
is
this
deep
gradient
in
compression
technology
there's
a
page.
This
is
a
paper
that
literally
just
came
out
about
two
weeks
ago.
It
also
directly
targets
on
addressing
the
the
huge
network
communication
bandwidth
issue
here
when
you're
doing
deep
learning
training
across
multiple
nodes.
B
Basically,
it
turned
out
that
99.9%
of
the
gradients
exchanged
could
be
to
actually
tiny
to
be
significantly
significant
enough
to
be
transferred
in
the
first
place,
so
it
does
a
lot
of
compression
technique
without
losing
the
accuracy.
If
you're
interested
in
this
check
out,
this
paper
is
actually
it
can
really
democratize
distributive
training
over
community,
just
one
Gigabit
Ethernet
yeah.
A
B
B
So
if
I,
for
example,
if
I
want
to
just
spin
up
a
Jupiter
notebook,
I
can
go
and
select
a
tensorflow
iPad
for
CPU
template
and
I
can
just
click
on
a
something
it
button
and
I'll
get
a
jupiter
notebook
really
quickly,
and
but
what
I
really
want
to
show
you
now
is
how
I
can
do
a
distributed.
Tensorflow
training
just
from
this
simple
GUI
interface.
B
So
I
can
now
just
basically
select
a
job
type
to
be
a
distributed
training
and
I
can
give
it
a
job
name.
I
can
call
it
joy,
IDF
test
in
this
case,
for
example,
I
can
have
just
one
parameter
server
and
two
workers,
and
each
worker
should
have
just
one
GPU
and
for
the
darker
version.
I
want
to
use,
say,
for
example,
tensorflow
1.2
and
in
the
containing
command.
I
can
just
simply
tell
it.
This
is
the
Python.
B
Obviously,
the
part
of
my
talk,
my
pencil
flow
Python
script
here,
so
I
can
enable
tensile
board.
So
obviously
people
love
tensile
board
for
debugging
purposes
or
in
order
to
do
your
training.
Often
you
can't
just
say
enable
tensile
board
and
you
will
get
an
endpoint
and
you
can
just
quickly
specify
a
model
path
for
saving
your
model
checkpoint.
And
then
you
just
click
on
submit
and
now
you,
as
you
can
see
a
job.
B
B
So
here
I
was
given
an
end
point
and
by
just
simply
clicking
on
a
tiempo
and
Ike
there's
my
Jupiter
notebook
and
then
back
to
my
previous
distributed
training
job.
As
you
can
see,
the
job
is
already
running.
So
if
I
click
into
the
Java
code,
you
can
see
actually
there's
a
real
time.
Training
going
on
here,
I
specified
5,000
stats
across
to
worker
nose
and
you
can
see
there
are
like
there
are
two
workers,
each
working
on
some
steps
here.
So.
B
So,
just
very
quickly,
two
quick
things:
I
want
to
call
out.
There
is
a
free
flow,
CNI
plug-in
that
comes
with
deep
learning.
Workspace
we've
just
shown
you
and
the
plugin
is
from
Microsoft
reserves
it.
Basically
it
automatically
leveraged
shared
memory.
If
you
happen
to
have
part
of
the
arsenio
same
host
and
also,
if
you
do
have
our
DMA
Network,
whether
it's
walkie
or
if
in
event,
it
automatically
detects
that
you
have
our
DMA
network
and
it
will
use
the
our
DMA
network
automatically
and
it's
completely
transparent
to
your
path
and
your
apps.
B
So
with
that
scheduler
as
a
custom
plugin
that
you
can
plug
into
your
core
in
any
cluster,
your
data
scientist
can
or
your
DevOps
can
specify
apart,
not
only
with
just
how
many
GPUs
you
need
for
the
path,
but
also
how
many
GPUs,
with
how
much
memory,
and
also
how
many
GPUs
with
that
are,
inter
inter
connected
through
an
MV
link.
Those
are
all
very
important
when
you're
running
deep
learning,
GPU
related
our
clothes
on
kubernetes
thanks.
A
Joey
and
just
to
just
a
roundup
there's
a
bunch
of
other
resources
that
we
placed
on
here.
There's
a
lot
of
community
engagement
already
going
on.
I
know
that
the
folks
at
Google
announced
cube
flow
a
couple
of
days
and
that
we
folks
rallying
around
that.
So
maybe
go
take
a
look
at
that
as
well,
but
everything
up
here
today,
but
what
we're
hoping
for
is
that
we
can
all
work
together
and
build
a
commune
around
making.
Ml
workloads
run
great
on
kubernetes
and
get
some
knowledge
gaps
that
existed
in
between
me.