►
Description
Amazon EKS has quickly emerged as a leading choice for machine learning workloads. In this session, we’ll walk through some of the recent ML related enhancements the Kubernetes team at AWS has released. We will then dive deep with a walkthrough of the TorchElastic Controller for Kubernetes, a new open source collaboration between the Kubernetes team at AWS and the PyTorch team at Facebook, the TorchElastic Controller for Kubernetes, which addresses these limitations and unlocks new capabilities with PyTorch built models and Kubernetes distributed training, including the ability to train on EC2 Spot instances, run jobs that are resilient to hardware failures, and dynamically scale jobs based on cluster resource availability. For more information, read the launch blog: https://aws.amazon.com/blogs/containers/fault-tolerant-distributed-machine-learning-training-with-the-torchelastic-controller-for-kubernetes/
A
Welcome
everyone
really
glad
you
can
join
us
for
this
session
and
today
we're
going
to
be
talking
about
machine
learning
on
amazon
eks.
My
name
is
mike
stefaniak,
I'm
a
product
manager
on
the
eks
team
and
I've
been
focusing
on
our
machine
learning
efforts
in
the
kubernetes
space
before
we
get
into
the
demos
just
real
quickly.
Why
you
know
why
containers,
why
machine
learning?
A
It
it
solved
some
of
the
change
same
challenges
for
machine
learning
where
you
might
be
packaging.
Your
training
code,
your
dependencies,
your
configurations,
maybe
even
some
of
your
data
together
and
run
it
in
multiple
places
and
so
on.
Kubernetes,
there's
really
three
things
why
machine
learning
has
become
so
popular
composability
portability
and
scalability
composability
is
is
really
what
we
just
talked
about.
A
Where
you
can,
you
can
package
up
all
of
your
dependencies
into
a
container
and
that
really
allows
you
to
separate
the
separation
of
duties
between
the
data
scientists,
where
they
can
run
whatever
library
they
want,
and
then
the
machine
learning
operations
team
who
can
handle
the
kubernetes
cluster
underneath
portability,
meaning
you
can
run
kubernetes
in
an
open
source
framework
like
kubeflow
on
your
laptop.
You
can
run
it
on
premises
and
you
run
it
on
the
cloud
and
it
makes
it
really
easy
to
move
your
training
jobs
in
in
inferencing
endpoints
from
on-premise
to
the
cloud.
A
For
example,
when
you
need
a
lot
more
machines,
you,
you
might
not
have
them
on
premises.
But
if
you
go
to
aws,
you
know
you
can
scale
to
many
gpus
in
a
cluster
and
then
scalability.
That's
that's!
Really!
Where
kubernetes
and
eks
excels,
you
can
run
hundreds
even
thousands
of
worker
nodes
in
a
cluster,
and
you
can
iterate
really
quickly.
A
We're
excited
to
talk
about
a
new
project
that
we've
collaborated
with
the
pi
torch
team
at
facebook
on
called
the
torch
elastic
controller
for
kubernetes.
One
of
the
challenges
with
with
machine
learning
training
is
compared
to
say
a
normal
micro
service.
Application
you're
running
kubernetes
is
that,
with
with
the
normal
application,
when
a
pod
goes
away,
typically,
another
replica
comes
up
and
your
application
continues
serving
traffic
with
machine
learning
training
if
you
lose
a
worker
node,
for
example.
A
So
I
have
another
cluster
I've
created
ahead
of
time
and
in
this
case
I
used
eks
managed
node
groups
to
create
two
different
node
groups.
One
is
just
a
a
node
group
with
one
instance
with
a
small.
I
think
it's
a
t3
instance
in
it
and
then
another
one
with
node
group
of
two
amazon,
ec2
p3
instances
which
are
designed
for
training
and
I'll
get
to
why
we
need
both
of
these
in
a
minute.
A
So
I'm
going
to
go
back
to
my
to
my
ide
here
and
the
first
thing
I'm
going
to
do
is
install
the
controller
in
the
custom
resource
into
my
cluster,
so
how
the
torch
elastic
for
control
for
controller
for
kubernetes
works
is
the
controller
runs
in
your
cluster
and
it
watches
a
custom
resource
definition.
That's
also
installed,
and
in
that
custom
resource
definition,
you
define
the
minimum
and
maximum
number
of
workers
that
you
can
have
as
part
of
your
training
job
and
the
controller
manages
the
pods
that
you
define
in
there.
A
So
I've
downloaded
the
torch,
elastic
github
repository
locally
here
and
to
install
the
controller
you
just
run
I'll
run
this
command
and
you'll
see
it'll,
create
a
new
namespace
role,
rule
bindings,
the
custom
resource
definition
and
then
the
controller
itself.
So
we
should
see
here
the
controller
got
installed
in
this
namespace.
Let's
filter
this
down.
A
A
A
A
That
should
be
able
to
run
in
this
training
job
so
in
this
case,
minimum
of
one
maximum
of
three
and
we're
going
to
start
with
two
replicas
and
each
replica
is
requesting
a
gpu,
and
I
currently
have
two
gpu
instances
in
my
cluster.
So
this
should
start-
and
in
this
case,
for
the
training
container,
we're
just
using
the
built-in
torch
elastic
example,
which
I
believe
uses
an
imagenet
model
to
train.
A
And
we
should
see
two
workers
start
here,
two
pods,
and
so
in
this
case
the
controller
was
watching
the
custom
resources
called
elastic
job.
It
saw
one
was
added
to
the
cluster
and
it's
go,
went
ahead
and
created
the
pods
needed
for
training
for
us
and
if
we
look
into
the
logs
here,
we
can
see
that
the
training
job
is
starting.
A
You
can
see
that
the
the
logs
here
are
mentioning
the
rendezvous
state,
so
rendezvous
is
keeping
track
of
the
number
of
workers
in
my
training
job,
and
so
now
our
training
jog
is
chugging
happily
along
and
let's
simulate.
If
we
were
running
these
on
spot
instances
and
one
of
them
got
taken
away
what
would
happen.
A
A
Cluster
and
you
can
see
that
one
of
the
nodes
is
already
it's
scheduling,
disabled
and
it
eventually
it'll
be
removed
from
the
cluster.
Now
in
a
normal
training
job,
this
would
mean
the
job
would
fail
and
if
you
had
a
job
running
for
three
hours,
you'd
have
to
restart
the
entire
thing,
but
let's
go
back
and
check
and
you
can
see
that
one
of
the
workers
went
away
it's
now
pending
because
there's
no
node
in
the
cluster
for
it
to
run
on.