►
From YouTube: Building GPU-Accelerated Workflows with TensorFlow and Kubernetes [I] - Daniel Whitenack
Description
Building GPU-Accelerated Workflows with TensorFlow and Kubernetes [I] - Daniel Whitenack, Pachyderm
GPUs are critical to some artificial intelligence workflows. In particular, workflows that utilize TensorFlow, or other deep learning frameworks, need GPUs to efficiently train models on image data. These same workflows typically also involve mutli-stage data pre-processing and post-processing. Thus, a unified framework is needed for scheduling multi-stage workflows, managing data, and offloading certain workloads to GPUs.
In this talk, we will introduce a stack of open source tooling, built around Kubernetes, that is powering these types of GPU-accelerated workflows in production. We will do a live demonstration of a GPU enabled pipeline, illustrating how easy it is to trigger, update, and manage multi-node, accelerated machine learning at scale. The pipeline will be fully containerized, will be deployed on Kubernetes via Pachyderm, and will utilize TensorFlow for model training and inference.
About Daniel Whitenack
Daniel (@dwhitena) is a Ph.D. trained data scientist working with Pachyderm (@pachydermIO). Daniel develops innovative, distributed data pipelines which include predictive models, data visualizations, statistical analyses, and more. He has spoken at conferences around the world (ODSC, Spark Summit, PyCon, GopherCon, JuliaCon, and more), teaches data science/engineering with Purdue University (@LifeAtPurdue) and Ardan Labs (@ardanlabs), maintains the Go kernel for Jupyter, and is actively helping to organize contributions to various open source data science projects.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
Okay,
it's
time
to
get
started.
Thank
you
guys
for
coming
so
I'm
Daniel
white,
knack
I'm
work
for
a
company
called
pachyderm
you'll
hear
more
about
that
project
a
little
bit
later,
but
today,
I'm
gonna
be
talking
to
you
about
building
GPU
accelerated
workflows
with
tensor
flow
and
kubernetes,
which
is
this
really
long.
Title
I
should
have
made
it
more
shorter
and
more
exciting,
but
but
hopefully
the
talk
will
be
exciting.
So
it's
a
great
set
up
in
the
last
talk
talking
about.
A
You
know
some
of
the
challenges
around
using
GPUs
on
a
data
science
team
and
how
you
could
actually
offload
like
model
training
on
to
a
GPU
on
kubernetes.
This
talk
is
going
to
be
related,
but
you
know
it's
it's
going
to
be
have
a
slightly
different
spin.
So
really
what
I'm
gonna
be
talking
about?
The
key
word
here
is
workflows,
so
oftentimes
running
model
training
isn't
the
only
piece
of
the
puzzle
right.
Actually,
it's
only
a
very
small
piece
of
a
very
much
larger
puzzle,
which
includes
a
lot
of
pre-processing
of
data.
A
It
includes
training,
it
includes
inference,
it
includes
post-processing
and
includes
visualization,
and
what
I'm
going
to
try
to
get
across
to
you
guys
today
is
how
we
run
all
of
that
together
on
kubernetes,
while
still
being
able
to
offload
those
important
pieces
to
the
GPUs
when
we
need
to
utilize
them,
and
so
we've
worked
with
a
bunch
of
different
users
and
clients
to
do
this.
So
what
I'm
gonna
describe
is
kind
of
how
how
we
do
that.
A
To
that
end,
I'm
gonna
start
by
talking
about,
like
my
my
picture
of
this
kind
of
bigger,
bigger
data
pipeline
scenario,
where
a
piece
of
that
is
model
training
on
a
GPU,
but
it's
it's
more
and
then
we're
gonna.
You
know,
obviously,
as
part
of
that
talk
about
where
the
GPU
comes
into
play,
then
we're
gonna
talk
about
why
kubernetes
is
so
good
at
managing
this
sort
of
thing
and
what?
A
If
anything,
we
need
to
add
on,
on
top
of
kubernetes,
to
be
to
support
this
sort
of
workflow
and
then
all
of
you
guys
are
a
technical
crowd.
So
I
know
you
won't
believe
any
of
that.
So
then
I'm
going
to
go
right
in
and
do
a
live
demo
and
we're
going
to
try
to
deploy
a
bunch
of
tensorflow
and
data
processing
stuff
on
kubernetes
and
switch
between
CPU
and
GPU
nodes,
and
you
know
fingers
crossed
that
goes
ok.
So
it's
that
sound,
ok,
everybody
good
I
mean
that's.
A
The
only
talk
I
have
so
I
mean
you
can
like
exit
now.
If
you
want
to
hear
something
different
all
right.
Well,
let's
go
ahead
and
get
started.
So
in
my
mind
again
like
the
the
typical
workflow
for
a
data
science
team
or
an
analytics
team
is
very
much.
You
know
broader
than
just
model
training
that
that
gets
a
lot
of
attention
because
that's
kind
of
I
guess
the
cool
part
right,
but
actually
there's
really
really
a
struggle
around
managing
the
pipeline
as
a
whole
and
actually
I
think
this.
A
This
problem
is
really
crucial
because,
if
you
can't
handle
like
all
these
pre-processing
things
and
then
get
the
data
to
the
GPU
and
you
need
to
and
handle
all
the
post-processing
things
and
be
able
to
share
those
resources
and
also
be
able
to
reproduce
certain
analyses
and
hand
them
off
to
other
parts
of
the
team,
then
you're
really
gonna
struggle
to
create
value
in
a
business,
so
I'm
going
to
share
like
kind
of
an
example.
So
this
is
by
this
is
very
much
an
example.
A
It's
not
you
know
the
the
things
that
we'll
talk
about
today
will
apply
in
many
more
examples,
but
one
of
the
example
of
this
is
like
processing
images
and
maybe
doing
some
object
detection.
So
in
like
this
sort
of
workflow,
we
might
start
with
like
a
raw
data
set
of
images
from
somewhere.
Let's
say
it:
let's
not
worry
about
where
they
come
from,
but
we
have
access
to
this
raw
data
set
of
images.
A
The
first
thing
that
we
probably
need
to
do
is
pre
process
those
somehow
so
maybe
our
like
model
needs
to
take
those
in
in
a
certain
format
or
a
certain
size,
or
maybe
we
need
to
like
pair
the
with
other
images
or
label
them
or
format
them
or
tag
them
in
some
way.
Anyway,
there's
a
lot
of
different
pre-processing
that
we
might
want
to
do
and
out
of
that
pre-processing
then
we
might
get
a
set
of
of
nice
images
that
we
want
to
feed
into
our
model.
A
A
A
But
again,
this
is
an
example,
so
this
would
apply
to
any
framework
that
you're
wanting
to
use,
whether
that
be
tensorflow
or
cafe
or
whatever,
and
so
we're
gonna
train
our
model
using
that
framework
on
that
input,
pre
process
to
input
we're
gonna,
maybe
serialize
a
model
or
export
it
in
in
some
way
such
that
we
can
use
that
model
for
inference.
So
we're
not
gonna
retrain
our
model.
Every
time
an
image
comes
in
that
we
need
to
do
object,
detection
on
we
need
to
like
serve
that
model,
somehow
like
tensorflow,
serving
or
other
things.
A
So
we,
this
is
kind
of
a
separate
stage
and
I
hate
I
haven't
even
added
in
after
this
like
if
there's
post,
processing
right,
so
here,
I've
already
kind
of
got
these
three
distinct
phases:
I've
left
out
post
processing
and
these
could
be
expanded
into
multiple
other
stages
right.
So
you
can
start
to
see
that
this
can
be
a
little
bit
of
an
orchestration
nightmare,
and
this
is
why
oftentimes
me
as
a
data
scientist
and
I
start
talking
to
you,
know
DevOps
people
and
infrastructure
people
and
they
say
I'm
a
data
scientist.
A
Okay,
so
we
again
here
we
have
the
these
kind
of
three
distinct
stages.
So,
let's,
let's
talk
about
like
where,
where
GPUs
come
into
that,
so
actually
most
of
the
time,
so
I'm
not
like
saying
a
general
statement
here,
but
most
of
the
time
people
utilize
GPUs
for
model
training,
as
was
mentioned
in
the
previous
talk,
so
here
we
actually
have
two
stages
that
will
run
just
fine
on
CPU
nodes
so,
like
basically
our
whole
workflow.
A
A
We
can
utilize
GPUs
in
kubernetes,
but
really
like
I,
don't
think,
there's
a
lot
of
content
out
there
and
tooling
around
actually
managing
this
sort
of
workflow
on
top
of
on
top
of
kubernetes
outside
of
scheduling
the
individual
pieces.
So
that's
that's
really
what
I
want
to
focus
on
is
like
enabling
this
workflow
and
then
also
being
able
to
get
those
necessary
stages
that
need
some
sort
of
acceleration
on
two
on
two
GPUs
when
we
need
them.
A
Okay,
so
again,
let's
say
that
we
have
these
these
few
stages
just
generally
well,
I
mean
one
of
the
things
that
we
need
to
do
is
need
to
make
these
stages
portable,
right
and
I'm
kind
of
preaching
to
the
choir
here.
But
you
guys
understand,
like
I:
can
I
can
docker
eyes
these
these
different
stages
and
run
them
in
any
sort
of
environment?
A
And
that's
that's
great
and
I
like
not
having
to
convince
you
of
that
in
this
conference,
because
a
lot
of
times
I
have
to
convince
people
that
in
data
science
conferences,
but
you
guys
understand
that
so
this
this
is
a
way
that
we
can
package
things
up
and
and
get
them
running
with.
You
know:
reproducible
behavior
in
another
environment.
But
again
we
don't
want
to
you
know
we
don't
want
to
be
like
SSA
Qing
and
the
machines
and
deploying
these
things
manually,
all
right
so
kubernetes.
You
know,
among
other
things,
I
can't
you
know.
A
I
don't
have
time
to
go
over
all
the
benefits,
but
again
I'm
preaching
to
the
choir
somewhat
here.
Right.
Kubernetes
gives
us
this
great
and
awesome
framework
for
being
able
to
take
these
stages
and
deploy
them
not
only
on
the
cpu
nodes,
but
on
GPU
nodes
in
a
very
descriptive
way.
Where
I
can
say
you
know,
I
I
want
these
certain
workloads
defined
by
these
containers
to
run
on
these
types
of
nodes
and
then
guess
what
it
happens,
and
that's
that's
really
great.
A
So
we
have
up
to
this
point
in
in
this
picture,
we've
made
our
individual
processing
stages,
portable
right
and
we've
actually
made
deploying
all
of
them
together
portable
because
we
can
run
kubernetes
anywhere
right.
Your
data
scientists
can
develop
these
things.
You
can
deploy
kubernetes
where
you
want
on
whatever
infrastructure
you
want
and
then
deploy
the
set
of
things
on
kubernetes,
but
that's
actually
not
that's,
not
the
the
only
key.
So
so,
what
am
I
missing
someone
someone
tell
me
so
so.
Let's
say
that.
A
Lifecycle
management,
updating
that
sort
of
thing?
What
else
data
yeah
so
so
the
first,
the
first
one
that
comes
to
my
mind
right-
is
I.
Deploy
these
containers
here
and
I
I
am
a
data
scientist,
so
I
want
to
process
data.
Where
is
the
data
right?
I
have
to
somehow
get
the
right
data
to
the
right
code
right
and
maybe,
like
let's
say,
that's
stored
in
an
object
store,
that's
what
was
talked
about
in
the
last
talk
as
well.
A
Let's
say
our
data
isn't
an
object
store,
so
somehow
I
need
to
get
the
right
pieces
of
data
which
aren't
everything
that's
stored
in
the
objects.
All
right,
I
need
to
get
the
right
pieces
of
data
to
the
right,
pods
to
be
processed
okay
and
then
there's
the
element
of
the
linkage
that
was
mentioned
right.
Actually,
that's,
that's!
Not
all
right.
A
I
need
to
get
the
right
data
to
the
right
code
and
I
need
to
run
those
steps
of
processing
in
a
very
specific
predefined
sequence
right
for
things
to
go,
right
and,
and
so
like
kubernetes
provides
this
really
great
framework
and
foundation,
but
similar
to
like
you
know
how,
like
Borg,
is
different
in
Google
than
what
kubernetes
is
and
Borg
includes
a
bunch
of
these
pieces
that
fills
gaps
in
the
context
of
what
Google
is
doing
and
then
there's
kubernetes
and
in
industry
it
you
know
it.
Doesn't
it
doesn't
offer
us
everything
we
need.
A
You
know
we
we
might
want
to
use
like
vault
for
secret
management
or
sto
for
for
service
mesh
right.
So
somehow
we
need
to
like
fill
this,
this
gap
of
like
getting
the
right
data
to
the
right
code
in
the
right
order
on
the
right
notes
right,
so
so
really
what
I'm?
What
I'm
saying
is
like
kubernetes
is
great
for
ml,
because
you
know
we
can
have
this
portability.
We
can
have
scalability,
we
can
have.
You
know
Auto
scale,
all
of
these
great
things
that
you
guys.
A
That's
why
you're
here
and
those
things
are
directly
applicable
to
machine
learning,
but
we
need
a
little
extra
sugar,
okay
and
this
extra
sugar
is
actually
really
important
and
and
not
that
trivial.
So
we
need
to
get
the
right
data
to
the
right
code.
We
need
to
process
the
right
data
with
the
right
code
on
the
right
nodes
right,
so,
whether
that
be
a
CPU
work,
work
load
or
a
GPU
workload,
and
we
need
to
trigger
the
right
code
at
the
right
time
with
the
right
data
on
the
right
nodes.
A
You
guys,
you
guys
get
the
idea,
so
the
this
is
really
this.
This
is
really
what
I
believe
you
know
we
we
need
and
what
our
team
believes
we
need,
but,
as
was
mentioned
up
front
here,
so
this
is
like
operationally
kind
of
what
we
need
to
enable
this.
It
would
be
nice
as
a
bonus
as
well
oops
to
be
able
to
actually
have
some
concept
of
maintaining
this
overtime
and
making
sure
that
we
do
it
in
a
sustainable
way,
which
also
means
that
we
need
to
be
somehow
tracking.
A
So
all
this
together
is
what
what
we
put
together
in
the
open
source
project
pachyderm.
So
pachyderm
is
the
open
source
data
pipelining
and
data
management
layer
on
top
of
kubernetes.
So
what
I
really
mean
by
that
is:
there's
data
pipelining
and
there's
data
management.
So
we
need
to
get
the
right
data
to
the
right
code,
which
is
related
to
this
sequence,
sequence
of
things
and
and
data
pipelining.
We
also
need
to
somehow
manage
that
data.
We
need
to
shim
the
right
data
to
the
right.
A
A
So
all
data-
that's
processed
in
pachyderm,
is
version
controlled,
so
kind
of
think
like
get
get
for
data,
you
can
set
up
collections
of
data
and
commit
data
in
there
make
changes
and
will
track
all
of
those
all
of
those
changes
which
which
both
lets
us
have
reproducibility.
But
it
also
lets
us
know
when
there's
new
data,
so
we
can
trigger
the
right
things
at
the
right
time.
Right.
A
A
The
containers
provide
this
unified
layer,
but
they
struggle
so
much
and
I
struggled
so
much
in
my
past,
with
all
of
this
diverse
set
of
tooling
and
being
able
to
string
it
all
together,
so
being
able
to
use
tensor
flow
when
I
need
it
and
then
connect
that
output
to
R
and
do
some
visualizations
and
then
someone
else
builds
some.
You
know
weird
thing
with
Julia
or
something,
and
we
do
that
so
that
gives
us
like
a
unified
framework
for
saying
our
basic
units
of
data
processing
or
containers.
A
We
run
opinionated
about
what
you
run
in
those
run.
Tensorflow
run
our
run,
Python
run
Julia
run
a
bash
command.
We
don't
we
don't
care
next.
We
kind
of
combine
the
containers
for
analyses
with
the
data
versioning
to
build
up
pipeline
or
build
up
distributed
pipelines
or
or
DAGs
of
processing,
where
these
containerized
processing
stages
subscribe
to
version
collections
of
data
descriptively.
So
you
say:
I
want
to
process
this
data
with
this
image
and
you
build
up
this
dag
of
processing,
stuff
steps
with
which
is
also
scalable
and
parallelizable.
A
And
finally,
because
we're
versioning
all
the
data-
and
we
know
what
docker
images
we're
using
for
each
stage.
We
actually
have
complete
quote/unquote
provenance
for
any
data
anywhere
and
what
I
mean
by
that
is.
We
can
produce
a
result
and
then,
if
we
want
to
know
all
of
the
other
pieces
of
data
and
the
states
of
the
pose
pieces
of
data
and
the
states
of
our
docker
images.
A
All
of
that,
when
we
produce
that
result,
we
can
get
all
of
that
information
very
easily
which
helps
both
in
terms
of
compliance
and
maintainability
and
debugging
and
and
all
of
that
stuff.
Just
to
again
I
know,
I
know
we're
in
a
with
the
with
the
technical
crowd
here,
so
I
I
want
to
definitely
don't
want
to.
You
know,
leave
without
kind
of
giving
you
a
few
more
of
details.
Before
we
jump
into
the
demo
so
pachyderm
again,
it's
a
it's
a
layer
that
runs
on
kubernetes,
so
kubernetes
forms
the
base.
A
This
gives
us
most
of
what
we
need
right.
Pachyderm
runs
as
a
pod
on
top
of
kubernetes,
and
then
it
talks
to
an
object
store,
which
is
where
all
the
data
is
backed,
and
we
talked
to
that
pachyderm
pod
and
tell
it
you
know
we
want
to
process
this
data
with
this
code
and
then
pachyderm
talks
to
kubernetes
under
the
hood
and
then
spends
up
whatever
pods
are
needed
to
do
that
processing.
So
if
I
have
an
inference
stage
or
a
training
stage
running
tensorflow,
then
I
can
spin
up.
A
Okay,
so
enough
of
my
enough
of
my
blabbing,
let's
get
to
the
let's
get
to
the
good
stuff.
Okay,
so
we've
got
a
demo
here.
This
is
sorry
about
the
fuzzy
the
fuzzy
text.
I
think
the
terminal
will
be
a
little
bit
better,
but
just
to
give
you
an
idea.
So
this
is
like
a
dashboard
that
you
can
have
that
you
can
look
and
see
what's
running
as
pachyderm
pipelines
and
I
have
this
pipeline
running
here.
I'll
show
you
on
the
backend.
A
What
that
looks
like
here
in
a
second
but
just
to
kind
of
illustrate
here,
I
here,
I'm,
doing
an
image
to
image
translation
with
tensorflow,
which
means
like
an
image
comes
in
in
one
style
and
I
want
to
transfer
it
to
another
style.
In
this
particular
case,
I
want
to
bring
satellite
images
in
and
kind
of
automatically
transfer
their
style
to
like
Google
Maps
images,
okay,
so
here
in
training,
so
each
of
these
blue
dots
here
represents
one
of
these
version
collections
of
data.
A
Remember
this
is
a
kind
of
our
first
piece
of
the
puzzle
and
in
this,
in
this
version,
collection
of
data
I
have
a
bunch
of
of
images
that
I
can
use
for
training.
So
I
want
to
be
able
to
translate
images
like
what's
on
the
left,
two
images
like
what's
on
the
right,
also
in
my
in
my
input
here:
I
have
two
input,
images
and
I
want
to
say:
okay,
I
want
to
take
this
image
and
I
want
to
translate
it.
So
this
is
my.
A
This
is
my
input
that
I
want
to
utilize
my
trained
model
to
transform,
and
so
my
inputs
are
those
that
training
data
and
then
and
then
that
input
data
or
input
images,
then
over
here
on
the
Left
I,
do
the
training
so
this
this
next
stage
here
is
the
training.
Then
I
do
just
some
model
export
so
just
by
the
way
that
the
the
scripts
were
set
up.
A
I
just
changed
the
format
of
the
the
model
and
then
that
model
is
used
in
a
generate
stage
along
with
the
pre
processed
images,
so
I
have
a
stage
of
pre
processing
and
then
I
feed
those
together
into
the
generate
stage
which
generates
output
images.
So
remember,
each
of
these
collections
is
one
of
those
version
collections
of
data
and
each
of
my
pipeline
stages
or
containerized
analyses
right.
So,
if
I
click
on
one
of
these,
this
is
my
model
training
stage
and
I
go
down
here.
A
I
can
see
how
this
how
this
pipeline
stage
is
defined
and
it's
defined
via
a
docker
image
and
a
command.
That's
run
in
that
docker
image,
so
I'm,
basically
telling
pachyderm,
hey
I,
want
you
to
process
this
data
using
this
docker
image
and
when
you
do
it
run
this
command,
which
is
just
my
Python
script.
That's
that's
using
tensor
flow
okay
and
each
of
these
is
defined
and
in
a
similar
way.
A
So,
in
this
case,
all
all
of
my
pipeline
stages,
except
for
this
checkpoint,
the
the
model
training
stage-
all
of
those
are
just
fine
running
on
CPUs
and
actually
I
can
paralyze
them
very
very
easily
across
CPU
node
instances.
So
no
worries
like
I,
don't
have
to
worry
about
using
a
GPU
for
for
those
stages.
A
So
ideally,
what
I
would
want
to
have
is
I
would
want
to
have
those
pods
that
are
running
those
processing
stages
run
on
CPU
nodes
and
then,
when
I
need
to
run
checkpoint
I
want
that
to
be
scheduled
on
a
GPU
node
such
that
I
can
do
my
model
training
very
quickly,
okay.
So
so,
let's
move
over
and
connect
some
of
the
dots.
A
So
we
have,
if
I
look
at
what's
running
in
this
cluster
now
I
can
see,
there's
pack
D
running,
which
again
manages
all
of
this
pipelining
stuff
and
data
management,
things
and
then
I
have
all
of
these
pipeline
workers.
So
I
have,
in
this
case
I
have
a
single
worker
for
each
stage
of
my
pipeline.
Although
that's
by
no
means
not
the
only
thing
you
can
do,
you
could
spin
up
a
hundred
workers
to
process
a
stage
in
parallel.
A
That's
that's
fine
and
I
can
talk
about
that
in
the
question
and
answer
if,
if
that's
something
you
want
to
talk
about,
but
I
have
each
of
these
pods
scheduled
and
I've
already
actually
put
some
example
data.
So
if
I
look
at
what
jobs
have
run,
which
jobs
just
mean,
how
many
times
have
these
stages
run?
Sorry
for
the
the
wrapping
there,
but
I
can
see.
Like
checkpoints
run
once
I've
pre-processed
a
couple
times,
I've
run
my
model
export
and
I've
generated
images
once
and
that's
reflected
over
here
in
the
in
the
dashboard
as
well.
A
A
A
If
we
look
at
the
output
here,
it's
actually
not
super
impressed
because
I
ran
so
it
kind
of
looks
like
a
Google
Maps
drawing,
but
I
only
ran
the
training
for
like
one
epoch,
if
you're
familiar
with
training,
that's
not
sufficient
in
general,
but
I
did
it.
So
it
ran
for
about
eleven
minutes
on
a
CPU
and
the
reason
why
I
did
that
is
to
show
you
something
that
would
actually
execute
in
the
time
of
this
talk.
A
A
A
The
piece
that
I
haven't
covered,
which
is
really
the
the
punchline
of
this
talk,
I
guess,
is
that
that's
actually
all
not
not
the
complete
story,
because
I
still
need
to
make
sure
that
at
least
one
of
these
stages
runs
on
a
GPU
right.
I
want
to
run
my
training
on
a
GPU
and
actually
what
what's
happening
under
the
hood.
Let
me
show
you
here
the
way
that
I
told
pachyderms
to
spend
these
up
is
with
a
specification
like
this.
That
says,
you
know,
create
a
pipeline
called
checkpoint
use.
A
This
docker
image
and
command
run
it
on
this
input.
Data
here,
I'm
just
going
to
run
one
one
instance
of
that,
but
I
didn't
say
anything
about
a
GPU
right.
So
what
happened
is
pachyderms
said:
okay,
that's!
That's!
All
cool
I'll
run
that
whenever
the
data
comes
into
training,
then
I'm
gonna
train
your
model
on
that
on
that
new
data.
A
But,
like
I
said,
if
we
look
at
this,
so
our
our
original
training
here
took
11
minutes.
So
I
just
ran
it
earlier
this
morning
and
we're
I
mean
11
minutes.
Isn't
that
much
in
the
training
world.
But
let's
say
that
we
want
to
do
better
and
we
want
to
sure
that
that
runs
on
a
GPU
first,
let's
just
see
if
it
did
run
on
a
jeep
on
a
GPU
node.
So
to
kind
of
show
you
how
I
have
this
cluster
set
up
I
have.
This
is
the
the
instances
in
my
cluster.
A
A
Let's,
let's
check
so
if
I
I
list
job
and
I
look
here
at
this
training
and
let's
go
ahead
and
get
the
logs
for
that
job
and
just
grep
for
cuda,
okay,
oh
crap,
I,
didn't
I,
didn't
find
any
I
didn't
find
any
GPU
right.
So
this
this
ran
just
says
it
as
it
normally
would
on
a
on
a
CPU
node.
Now
all
I
have
to
do
so.
Remember
going
back
to
the
last
talk.
Remember
what
the
standard
workflow
these
days
is
for
data
scientists,
usually
GPUs.
A
You
do
a
bunch
of
stuff
on
your
local
machine.
Maybe
you
do
some
stuff
in
the
cloud
or
on
a
cluster
with
CPU
nodes
and
then,
when
you
need
to
do
something
on
a
GPU
node,
you
turn
around
you're
like
hey,
Frank
you're,
using
the
the
GPU
node,
and
then
you
know
he
says
no,
but
then
maybe
Susie's
using
it.
So
you
schedule
a
job
on
it
and
then,
like
all
all,
goes
to
crap
and
everybody's
angry
and
it's
not
harmonious
at
all.
A
So
we
want
to
strive
for
something
more
harmonious,
so
I
want
to
just
make
sure
that
that
runs
on
on
a
GPU.
So
all
I
have
to
do,
is
go
in
here
and
modify
my
my
pipeline
spec
and
just
set
some
resource
limits
and
say:
hey
I
want
to
run
I
want
to
run
with
a
GPU,
okay
and
then
here's
where
you
can
cross
your
fingers.
A
Let
me
just
update
that
and
I'm
gonna
reprocess,
which
means
you
know:
I
want
to
reprocess
with
the
updated
spec
oops.
You
actually
do
this
sorry
with
the
GPU.
Let
me
actually
because
I
just
a
second,
you
didn't
cross
your
fingers
quite
good
enough,
because
I
forgot
the
GPU
tag
there.
So
let
me
delete
this
other
job.
A
A
There's
my
GPU,
and
so
all
I
had
to
do,
was
just
set
that
resource
limit
and
now
pachyderm
knew
hey.
You
don't
want
me
to
run
this
on
a
CPU
anymore.
You
want
me
to
run
it
on
a
GPU
and
then
it
talked
to
kubernetes
under
the
hood
and
it
said
hey.
This
needs
to
run
on
a
GPU
and
it
would
schedule
on
a
GPU
node
and
then
it
it
recognizes
the
drivers
boom
we're
off
to
the
races.
A
The
rest
of
the
things
are
still
going
to
run
on
the
CPU
nodes,
everything's
still
wired
together.
So
when
I
output
this
model
from
the
GPU
training,
it's
still
gonna
be
supplied
to
the
other
stages
that
are
running
on
CPU
nodes
and
then
I'm
and
then
I'm
golden
I'm.
Okay.
So
let's
I'll
just
keep
this
up
in
case
your
this
should
finish.
I
think
it
finishes
in
like
one
minute
so
one
minute
compared
to
11
minutes,
it's
pretty
pretty
good
improvement.
A
A
It's
lunchtime
now
happy
to
have
lunch
with
you
guys,
there's
stickers
up
here,
so
if
you're
into
that
sort
of
thing
grab
some
of
those
I'll
also
post
these
slides
on
the
sked
schedule
site
the
PDF
there's
a
few
so
don't
feel
like
you
have
to
the
picture
unless
you
want
it,
but
this
will
be
posted
on
the
schedule,
so
you
can
grab
the
PDF
of
this
and
then
follow
these
links
to
all
the
docs.
Let's
just
see,
if
oh
we
got
it
okay,
so
we
ran
about
a
minute.
Okay.
So
all
right,
thanks
guys.