►
From YouTube: The Path to GPU as a Service in Kubernetes - Renaud Gaubert, NVIDIA (Intermediate Skill Level)
Description
Want to view more sessions and keep the conversations going? Join us for KubeCon + CloudNativeCon North America in Seattle, December 11 - 13, 2018 (http://bit.ly/KCCNCNA18) or in Shanghai, November 14-15 (http://bit.ly/kccncchina18).
The Path to GPU as a Service in Kubernetes - Renaud Gaubert, NVIDIA (Intermediate Skill Level)
TLDR: Kubernetes modern production patterns for Deep Learning applications and a deep dive into the Kubernetes GPU and Hardware accelerators subsystem and it’s challenges (performance, scheduling, monitoring). Autonomous vehicles, face recognition, High Performance Computing, Virtual Reality, NVIDIA GPUs are enabling a new computer era with cloud computing at its center. With kubernetes being the next iteration in cloud technologies, the NVIDIA container team is driving with the community the advances in GPU integration. During this talk we will take a look at the possibilities and limitations provided by the kubernetes infrastructure for hardware accelerators. We will also be discussing the latest improvements in the device plugin subsystem of Kubernetes, powering GPU support, and the challenges ahead of it.
About Renaud
Renaud Gaubert has been working for a year at NVIDIA on making GPU applications easier to deploy and manage in data centers. He focuses on supporting GPU-accelerated machine learning frameworks in container orchestration systems such as Kubernetes and Docker swarm. He is an active member of the Kubernetes Resource Management workgroup and helps pushing forward GPU integration in the Container ecosystem. He holds an M.S. in computer science from the French engineering school EPITA.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
Join us for KubeCon + CloudNativeCon in San Diego November 18 - 21. Learn more at https://bit.ly/2XTN3ho. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
Hello:
everyone,
my
name
is
Renault
Qbert
I'm,
an
engineer
at
Nvidia
I've
been
contributing
to
the
and
working
in
the
communities
community.
For
the
past
year,
with
my
team,
a
container
team
and
today
I'm
going
to
talk
mostly
about
GPU
as
a
service.
How
do
you
actually
deploy
your
GPU
cluster
and
where
are
the
challenges
that
you
face
when
you
operate
a
GPU
cluster
or
as
a
user?
The
challenges
that
you
face
when
deploying
your
GP
applications
will
also
look
a
bit
into
how
we
actually
built
them.
A
The
GPU
and
device
system
in
kubernetes
and
less
part
will
be
about
where
the
challenges
and
the
different
solutions
that
we
want
to
see
in
kubernetes
and
in
general
that
we
want
to
solve
for
GPUs,
and
so,
if
we
go
ahead
and
look
at
why
GPUs
and
containers
makes
a
lot
of
sense.
You'll
find
out
that
actually
containers
solve
a
lot
of
issues
that
mostly
deep
learning
scientists
or
HPC
applications
or
in
general
GPU
applications
have
a
problem
with,
and
so,
if
you're
deploying
scientists,
for
example,
you're
going
to
have
a
lot
of
issues.
A
A
So
so,
if
you
solve
a
lot
of
these
challenges,
we've
we
built
our
container
tools
from
the
ground
up
to
and
will
help
us
support
a
lot
of
different
use
cases.
And
if
you,
if
you
were
familiar
with
Nvidia
dr.
before
what
we
were
actually
doing,
is
wrapping
docker
specifically
and
you
couldn't
really
use
a
video
docker
for
like
any
other
runtime
and
what
we've
what
we
did
is.
Actually
we
integrate
now
at
the
runtime
level.
That
means
that
we
support
a
lot
more
runtimes.
A
So,
for
example,
Dockers
here
are
your
singularity,
LXE
and
if
you're
actually
not
familiar
with
this,
the
challenges
that
we're
trying
to
face
with
GPUs
in
containers
there's
a
lot
of
small
things,
the
biggest
one
being
that
the
cooler
libraries
that
a
lot
of
these
frameworks
are
deploying
frameworks
depend
on.
They
have
this
very
strict
requirement
that
you,
your
your
library
version,
must
exactly
match
your
kernel,
your
kernel
module
version.
A
That
means
that
you
can't
actually
ship
a
container
with
the
cooler
libraries
or
just
in
general,
the
user
libraries
that
you
would
like
that,
like
you
would
do
for
any
other
general
application.
We
need
to
actually
inject
these
at
at
one
time
level
and
or
at
the
run,
sorry
step.
So
that's
why
we
basically
integrated
at
the
runtime
level.
We
have
a
really
in
general,
the
the
way
we
are
better
integrated
for
OCI
runtimes,
because
the
way
we
do
it
is
that
we
get
called
at
the
pre-start
level.
A
And
that
means
that
for
a
lot
of
these
runtimes,
we
don't
need
to
ship
a
specific,
a
specific
sorry
for
a
lot
of
these
container
runtimes.
We
don't
need
to
ship
a
specific
runtime,
but
for
docker,
for
example,
we
actually
have
to
just
wrapper
NC
and
add
a
single
line
of
code
that
says
well
add
restart
hook,
and
so
that
allowed
us
actually
to
support
not
a
lot
of
new
use
cases,
and
you
actually
see
that
on
the
docker
hub.
Is
that
we're
slowly
actually
adding,
for
example,
support
for
graphics?
A
So
that's
that's
pretty
interesting
and
now
that
we
actually
have
these
support
with
these
tools
to
actually
support
more
runtimes,
we're
really
looking
at
kubernetes
and
we've
actually
been
involved
in
the
community
for
the
past
year
at
the
creation
of
the
well
during
the
part
where
the
resource
management
workgroup
was
being
created,
the
resource
management
work
group
had
actually
to
face-to-face.
We
had
one
recently
at
Nvidia
and
March,
and
there
was
actually
there's
a
lot
of
engagement
in
that
workgroup
and
it's.
A
We
are
solving
a
lot
of
different
challenges
and
one
of
the
things
we
actually
managed
it
to
do.
Is
we
actually
added
this
new
device
system,
and
in
a
year
we
graduated
from
alpha
to
beer,
and
if
you
look
a
bit
at
kubernetes
now,
you
know
that
upstream
GPU
is
now
in
beta,
but
it's
still
a
bit
challenging
to
actually
deployed
provision
of
GPU
cluster.
On
the
other
hand,
when
you
look
at
what
we
went
through
last
year
were
a
bit
before
in
one
six.
A
Doing
that
your?
What
we
did
is
we
added
this
new
plugin
system
that
allows
not
only
to
support
GPU
but
in
a
more
generic
way
more
devices
and
we're
engaging
in
the
community
and
especially
in
the
resource
management
workgroup
with
a
lot
of
different
companies
to
handle
things
like
FPGAs,
like
nicks,
and
a
lot
of
different
devices
that
allowed
us,
in
our
case,
to
use
our
tools
so
the
Nvidia
container
runtime
and
in
110,
since
we
grabbed
it
to
be
done,
we
actually
have
now
he'll
stay
well,
we
had
health
check
before
we
are.
A
We
have
some
monitoring
and
we're
looking
at
solving
a
lot
of
different
problems
like
heterogeneous
clusters,
support
for
GPU,
sharing,
support
for
topology,
except
a
lot
of
features
that
all
go
a
bit
later
into.
In
my
talk,
if
you
look
at
it
from
a
user
perspective,
what
that
means
to
you
is,
as
I
was
mentioning
in
one
six
one:
seven,
you
support
one
GPU
per
node.
You
need
to
manually
mounted
volumes.
A
So
as
I
was
mentioning
the
different
libraries,
you
actually
have
to
make
sure
that
they
are
all
in
the
same
directory
for
all
the
GPUs
in
your
node.
If
you
are
actually
upgrading
the
driver,
then
you
have
to
make
sure
that
you
are
placing
the
libraries
back
in
that
same
directory
and
at
the
at
that
time
you
had
very
fragmented
ecosystem,
so
everyone
was
doing
their
own
thing.
We
were
doing
Nvidia,
docker
kubernetes
was
doing
its
own
implementation
of
Nvidia.
A
A
A
So
what
that
means
is
as
a
cluster
and
then
you
have
to.
If
you
want
your
GPU
cluster
to
be
your
cluster
to
be
GPU
aware,
you
only
have
to
run
one
simple
command,
cube,
CT,
I'll,
create,
and
then
this
URL,
which
are
that
bravery
at
the
two,
because
it
didn't
fit
on
the
side,
and
this
allows
you
to
just
create
a
basic
pod
that
requests
two
GPUs
and,
in
the
end,
as
a
user.
That's
all
you
care
about.
A
You
just
want
to
create
a
parts
back
that
requests
that
has
a
specific
docker
image
and
just
request
a
number
of
GPUs.
So
that's
where
we
are
right
now.
What
that
means
is,
however,
operating
GPUs
in
a
cluster
has
a
lot
of
challenges
either
from
a
cluster
admin
perspective
or
from
a
user
perspective.
The
first
one
you're
actually
going
to
run
in,
is
well
you're,
going
to
try
to
do
your
blue-green
updates,
and
you
have
two
GPUs
left
in
your
cluster
and
what
that
means
for.
A
You
is
you're
going
to
have
to
deal
with
resource
with
the
fact
that
you
resource
contention,
you
don't
have
enough
GPUs
to
actually
get
your
version,
two
on
your
cluster,
and
so
your
GP
or
your
version.
The
version
2
of
your
apology's
actually
just
going
to
wait
indefinitely,
so
a
few
solutions
to
fix
that
is,
you
could
pre
provision
some
notes
or
you
could
try
to
scale,
but
if
you
try
to
scale
you're
going
to
hit.
The
second
second
issue
is
that,
if
you're
using
a
double
yes,
for
example,
you
are
limited.
A
A
A
Just
we'll.
Have
your
data
pod,
just
pre-scheduled
on
other
on
other
nodes,
because
it
doesn't
request,
it
doesn't
require
you
to
have
a
GPU
driver,
sorry,
a
GPU,
and
then
you
would
have
your
data
in
your
inference.
Container
talk,
for
example,
through
IPC.
So
that
would
allow
you
to.
Basically,
if
you,
for
example,
want
to
make
sure
that
you
have
some
resiliency.
You
have
a
liveness
probe
on
your
container
inference
container
and
once
kubernetes
the
text
that
your
inference
container
is
down.
A
The
other
one
is
about
scaling
GPU
services,
that's
one
of
the
requests
that
we
actually
get
a
lot
is,
or
just
in
general
people
tend
to
get
this
idea
that
it's
probably
better
to
scale
in
GPU
load.
Now,
usually,
when
you
try
to
scale
services,
it's
mostly
inference
services,
and
what
that
means
is
that
your
bottleneck
is
probably
not
going
to
be
GPU
load
and
you're
better.
Our
factory
scaling
based
on
QPS.
A
You
can
actually
use
entrepot
affinity
to
line
up
nodes
that
have
your
data
power,
so
you
pre
provision
some
notes,
make
sure
your
data
pod
runs
on
these
nodes
and
use
this
use
the
entrepot
affinity
feature
of
kubernetes
to
make
sure
your
inference
service
scales
correctly.
As
usual.
You're,
probably
you
probably
want
to
keep
some
spare
nodes,
and
so
this
is
also
one
of
the
interesting
things.
A
Is
that
when
you're
actually
building
a
GPU
cluster-
and
you
have
a
lot
of,
for
example,
for
your
data
scientists
you're
actually
going
to
face
a
lot
of
issues
in
terms
of
how
do
I
actually
integrate
some
of
the
business
logic
inside?
That's
one
of
the
things
you
might
want,
for
example,
is
you
might
want
one
queue
per
user
and
you
might
want
to
be
able
to
process
the
jobs
one
by
one
or
sequentially
for
the
user,
and
the
issue
that
you
have
right
now
is
its
business
logics.
A
So
you
should
use
an
operator,
but
if
you
go
down
the
operator
way,
you
have
all
these
issues
where
it
it
makes
some
sense
to
be
in
an
operator,
but
it
would
make
more
sense
to
be
in
a
custom
schedule,
for
example,
and
in
your
operator
say
you
have
you,
have
these
three
cues
and
all
these
spots
and
well,
you
want
to
schedule
the
first
part,
and
so
you
would
basically
take
that
pod.
We
post
it
to
the
won't
post
it
to
the
kubernetes
api
is
and
well
subscribe
to
the
events
of
that
pod.
A
So
you
would
look
around
and
make
sure
that
if
that
part
has
been
scheduled,
then
you
can
remove
it
from
the
queue,
but
if
it
hasn't
scheduled,
if
it
hasn't
been
scheduled,
where
it
fell
scheduling,
then
you
have
to
delete
that
part
and
at
that
point,
maybe
between
the
moment
that
you
got
the
event
that
your
part
hasn't
but
has
failed
scheduling
and
the
moment
that
you
actually
delete
that
part.
It
could
have
gone
through
the
scheduler
again
and
being
scheduled.
A
So
there's
a
lot
of
different
there's,
actually
a
lot
of
business
logic
and
that's
more
advanced
than
that
yeah
you
as
a
cluster
admin,
might
want
to
actually
get
into
and
there's
no
actually
good
solution
for
that
there
are
some
things
that
are
being
discussed
in
the
community.
I
think
it's
called
the
scheduler
framework
that
six
scheduling
is
working
on,
but
right
now
it
seems
like
it's
custom.
Scheduler
is
the
best
way
you
can
go
through,
though
you
still,
you
can
still
see
that
it's
probably
not
the
best
solution.
A
So
as
I
was
mentioning
for
me
to
confirm
you
from
your
perspective
as
a
cluster,
then
you
just
deploy
device
login
and
what
happens
is
that
since
it's
a
demon
said
it's
going
to
be
scheduled
on
the
nodes?
The
nodes
are
going
to
create
the
containers.
You
probably
have
a
driver
install
if
you're
on
GK,
for
example,
they
have
their
own
driver
and
so
on.
A
If
you,
if
the,
if
you've
managed
to
fix
the
GPU
crash
or
the
device
will
get
manages
to
recover
some
of
it,
then
it
bubbles
that
up
to
the
node,
which
bubbles
that
up
to
the
API
server
and
goes
to
a
schedule.
I
was
also
mentioning
that
we
have
a
continuing
life
cycle
too.
So,
when
your
power
is
created
from
a
user's
perspective,
your
Prada
gets
assigned
its
node.
A
A
The
last
thing
that
I
want
to
talk
about
is
zero,
downtime
registration
and
currently
the
way
you
did
register
your
device
for
again
well,
the
way
either
device
plug-in
registers
itself
with
qubit
is
really
do
or
your
pc
protocol
or
mechanism
we're
looking
at
making
it
very
your
pc
serve
your
device.
Plugin
just
puts
its
socket
into
an
agreed-upon
path,
with
qubit
qubit
watches
that
pass
and
then
calls
the
G
RPC
server
that
allows
you
to
actually
do
zero,
downtime
registration.
A
So
if
you
want
to,
for
example,
upgrade
your
device
plug-in,
it
gets
scheduled
on
the
node
places
its
new
socket
inside
that
path.
Qubit
will
ping
that
G
RPC
server
once
it's
sure
that
the
G
RPC
server
is
up
and
is
ready,
it
will
switch
over
to
the
1.9
and
that
allows
a
pre
needs
pattern
and
sorry
that
we're
looking
into
so,
for
example,
you
might
want
to
have
in
your
cluster
some
nodes
that
have
ECC
enabled
and
what
we're
looking
at
is
your
device
plug-in
watches
the
annotations
on
your
node.
A
You
as
a
cluster
admin.
Add
the
annotation-
and
you
know
that
says:
I
want
ECC
enabled
so
Nvidia
comm
slash
device
plug-in,
for
example,
and
ECC,
enabled
your
G
on
PC
your
Nvidia
device.
Plugin
looks
well
watches
that
gets
that
event
enables
in
CC
on
the
GPUs
and
me
registers
itself
against
qubits
once
again
zero
downtime.
So
that
means
that
you
won't
get
any
pods
that
won't
get
rejected
from
your
qubits,
and
so
all
of
this
is
the
reason
I
presented.
A
All
of
this
is
so
that
you
can
see
now
that
the
lifecycle
of
the
device
fuggin
is
pretty
much
figured
out
and
right
now
we're
really
focusing
on
solving
a
lot
of
challenges
that
you,
as
a
user,
will
be
able
to
that.
You,
as
a
user,
will
be
able
to
enable
you
so
the
first
one
we
are
really
want
to
push
is
GPU
monitoring.
A
We
want
to
be
able
to
expose
a
lot
of
these
different
metrics
to
you
as
the
end
user,
because
you
you'll
be
able
then
to
identify,
for
example,
GPU
tests
that
are
trying
to
that
that
are
just,
for
example,
faulting
or
detect
power
in
efficiencies.
That's
something
that
might
be
interesting
for
you
or
in
general.
You
might
want
to
be
able
to
tell
your
deploying
scientist
that
you've
seen
some
bottlenecks
and
some
throttling
on
the
GPU
other
challenges
that
we're
looking
at
is
supporting
heterogeneous
clusters.
A
A
A
This
annoying
bug
that,
or
just
in
general
implementation
shortcoming
that
with
darker,
your
only
option
is
to
actually
set
the
default
runtime
to
Nvidia,
and
that
means
that
all
your
in
all
the
docker
images
or
all
the
container
images
that
are
actually
run
on
your
node
will
use
the
Nvidia
runtime.
That
leads
to
this
little
unintended
consequence
that
if
you
have
Nvidia
images
that
don't
request,
GPUs
they'll
see
all
the
GPUs.
A
This
is
something
we've
actually
fixed
in
cRIO,
because
you
don't
have
to
set
the
default
runtime
to
Nvidia,
but
we're
looking
at
pushing
that
in
the
community
really
what
what
are
the
things
that
are
really
interesting
and
we
want
to
solve
for
the
users,
our
GPU
sharing
and
GPU
topology
and
GPU
sharing
is
an
interesting
features.
That
is
an
interesting
feature
that
you,
as
a
user,
will
be
able
to
well
you'll,
be
able
to
save
some
money.
A
The
question
we're
still
figuring
out
is:
how
do
we
actually
expose
that
to
the
user?
Because
GPU
sharing
is
something
that's
mostly
going
to
be
about
scheduling
in
terms
of
how
its
implemented
in
the
runtime?
It's
still
something
that
we
need
to
figure
out,
because
we
don't
have,
for
example,
a
specific
way
to
actually
limit
the
amount
of
memory,
a
specific
process
or
container
use
and
GPU
topology
is
well.
If
you,
if
you're,
if
your
container
actually
request
Q
GPUs,
you
can't
give
it
any
Q
GPUs
on
your
node.
A
If
you
have
a
numerous
10,
for
example,
you
might
want
to
be.
You
might
want
to
be
able
to
have
these
two
GPUs
communicates.
As
you
env
link
and
not
through
QP
I,
usually
if
you
actually
just
take
any
two
GPUs
the
issue
you're
facing-
is
that
it's
probably
not
as
sometimes
it's
probably
not
worth
to
even
run.
The
task
we
had
tree
had
the
same
problem
is:
how
do
we
actually
expose
that
to
the
user
in
general?
These
are
the
challenges
ahead
in
kubernetes,
but
even
in
containers.
We
still
have
a
lot
of
challenges.
A
For
example,
we
want
to
be
able
to
attach
and
detach
GPUs
dynamically.
We
want
to
be
able
to
support
a
lot
of
different
architectures,
so
currently
we
we
we
have
a
supporting
in
cost,
but
there's
a
lot
of
other
container
OS
out
there
that
we
want
to
be
able
to
support
or
virtualization
so,
for
example,
Quetta
containers
as
I
was
mentioning
we're
enabling
graphics
and
video
encoding,
but
there's
a
lot
of
things
that
we
still
need
to
actually
expose
with
EPS.
A
B
Hi
you
mentioned
NWS
a
couple
of
times
there
and
I
I
recently
found
out
that
there's
actually
two
ways
to
get
GP
using.
In
the
rest,
you
can
get
GPU
instances,
but
you
can
also
attach
standalone
GPUs
to
I.
Guess
any
instance
type
are
both
supported
on
whatever
setup
we
have
today.
That
setup
that
you
mentioned
are
both
ways
supported
both
a
proper
GPU
instance
and
just
taking
any
AWS
instance
and
attaching
a
GPU
to
it.
So.