►
From YouTube: A Deep Dive on Supporting Multi-Instance GPUs in Containers and Kubernetes - Kevin Klues, NVIDIA
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon North America 2021 in Los Angeles, CA from October 12-15. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
A Deep Dive on Supporting Multi-Instance GPUs in Containers and Kubernetes - Kevin Klues, NVIDIA
MIG (short for Multi-Instance GPU) is a mode of operation in the newest generation of NVIDIA Ampere GPUs. It allows one to partition a GPU into a set of "MIG Devices", each of which appears to the software consuming it as a mini-GPU, with a fixed partition of memory and compute resources. In this talk, we take a deep dive into the details of how we built support for MIG in containers and Kubernetes. You will learn how MIG is made available to containers, what challenges we faced building MIG support for Kubernetes, and how you can use it today. Everything we built is 100% open-source and part of the NVIDIA container toolkit stack and NVIDIA k8s-device-plugin. This talk will conclude with a discussion on best practices around how to distribute MIG devices throughout a Kubernetes cluster, including how to handle the lifecycle of MIG devices on a node.
A
Hello:
everyone,
my
name
is
kevin
cluse
and
I'm
a
principal
software
engineer
at
nvidia.
Today,
I'm
going
to
be
talking
about
the
support
I
built
for
multi-instance
gpus
in
containers
and
kubernetes,
multi-instance,
gpus
or
mig
for
short,
is
a
new
hardware
feature
of
the
latest
nvidia
ampere
gpus.
So
everything
I
talk
about
today
is
specific
to
this
type
of
gpu.
A
I
imagine
if
you're
at
this
talk,
you
probably
already
understand
the
benefits
of
supporting
gpus
and
kubernetes,
so
I'm
only
going
to
spend
a
few
minutes
highlighting
some
of
their
more
important
use.
Cases
first
and
foremost
supporting
gpus
in
kubernetes
lets
us
scale
up
ai
training
and
inference
jobs
to
a
cluster
of
gpu
machines.
Using
the
same
underlying
infrastructure
we
use
to
deploy
all
of
our
cpu-based
workloads
such
as
web
servers,
databases,
streaming,
data
servers,
etc.
A
A
A
typical
gpu
cluster
then
looks
something
like
this,
where
you
have
a
bunch
of
servers,
each
hosting
some
number
of
possibly
heterogeneous,
gpus,
all
being
managed
under
a
single
kubernetes
instance,
where
training
jobs
tend
to
require
multiple,
more
powerful
gpus
and
inference
or
development.
Jobs
tend
to
only
require
a
single
instance
of
a
less
powerful
gpu.
A
Or,
as
is
often
the
case,
you
have
a
small
cluster
with
only
the
most
powerful
gpus.
You
can
get
your
hands
on
and
want
to
be
able
to
efficiently
use
them
for
all
types
of
workloads,
and
even
if
you
do
have
a
bunch
of
nodes
at
your
disposal,
wouldn't
it
be
better
to
have
your
cluster
full
of
the
most
powerful
gpus
possible
so
that
you
have
them
when
you
need
them,
but
then
have
a
way
of
sharing
an
individual
gpu
for
jobs
that
only
require
a
fraction
of
its
full
power.
A
A
Additionally,
each
gpu
instance
gets
a
dedicated
percentage
of
the
overall
memory
bandwidth
and
any
faults
that
occur
are
isolated
to
a
single
instance
without
disrupting
the
others
from
the
perspective
of
the
software
consuming
it.
Each
of
these
gpu
instances
really
looks
like
its
own
individual
gpu
and
can
be
treated
as
such
by
the
end
user.
A
Okay,
so,
as
I
said
before,
mig
is
a
way
of
taking
a
full
gpu
and
dividing
it
into
a
smaller
set
of
what
we
call
gpu
instances,
each
of
which
has
some
amount
of
guaranteed
memory
and
compute
resources
available
to
it.
But
what
are
some
typical
use?
Cases
for
multi-instance
gpus
well
in
general,
mig
is
useful
anytime.
You
have
an
application
that
doesn't
require
the
full
power
of
an
entire
gpu,
but
does
want
similar
isolation
guarantees
provided
by
full
gpus.
A
This
might
be
a
single
user
who
wants
to
run
multiple
inference,
jobs
or
some
sort
of
experimentation.
He
is
doing,
or
it
might
be,
several
trusted
users
from
the
same
organization,
each
running
model
exploration
in
a
jupiter
notebook
on
their
own
dedicated
gpu
instance,
and,
given
the
hardware
isolation
that
multi-instance
gpus
provide,
you
really
can
treat
them
as
separate
gpus,
allowing
untrusted
users
to
run
on
their
own
dedicated
instances,
as
would
be
the
case
when
running
on
a
managed
cloud
service
such
as
google
cloud,
ai
or
amazon
sagemaker.
A
A
A
A
However,
the
yields
in
the
actual
silicon
make
it
so
we
can
only
reliably
get
seven
compute
slices
instead
of
eight,
so
we
are
stuck
with
a
somewhat
awkward
combination
in
any
case,
to
create
a
gpu
instance.
All
we
need
to
do
is
combine
some
number
of
compute
slices
with
some
number
of
memory
slices
and
merge
them
together.
A
A
So
anyway,
jumping
back
to
our
compute
instance.
Example,
the
compute
instance
may
be
created
to
consume
the
first
compute
slice
or
the
second
or
the
third
or
the
fourth,
or
it
might
take
up
two
compute
slices
or
three
or
all
four,
and
in
this
special
case,
where
a
single
compute
instance
consumes
all
of
the
compute
slices
of
its
wrapping
gpu
instance.
A
Dropping
this
prefix
is
unambiguous,
because
these
names
are
meant
to
refer
to
the
actual
mig
devices
they
represent,
and
not
any
individual
gpu
instance
or
compute
instance,
and
in
the
context
of
kubernetes.
These
are
actually
the
only
type
of
mig
devices.
We
currently
support
due
to
the
fact
that
all
compute
instances
share
access
to
the
gpu
memory
of
the
wrapping
gpu
instance.
A
Subdividing
into
multiple
compute
instances
does
not
fit
well
into
the
isolation
guarantees.
One
normally
expects
from
the
kubernetes
resource
model
as
such.
For
the
remainder
of
this
talk,
I'll
only
be
referring
to
make
devices
of
this
type.
We
may
expand
on
this
support
in
the
future.
As
things
like
pod
level,
resources
start
to
take
shape,
but
for
now
this
is
what
we
have
and
I'll
go
into
the
details
of
how
all
of
this
works
a
little
bit
later
on.
A
You
can
also
have
some
combination
like
this
or
like
this
or
like
this,
but
you
can't
have
something
like
this
or
like
this
in
general,
this
diagram
represents
the
physical
layout
of
how
these
make
devices
are
created
on
the
actual
hardware.
So
finding
a
valid
combination
consists
of
walking
from
left
to
right
and
adding
devices
into
the
configuration
such
that
no
two
devices
overlap
vertically.
A
For
example,
the
container
shown
here
will
run
just
fine
on
a
host
with
the
v1
kernel
module
installed,
but
fail
on
a
host
with
the
v2
kernel
module
installed
to
solve.
For
this
we
provide
a
package
called
the
nvidia
container
toolkit,
which
takes
care
to
ensure
that
compatible
nvidia
driver
libraries
are
injected
into
a
container
at
runtime,
as
well
as
give
an
application
access
to
any
required.
Gpu
hardware.
A
Many
of
you
have
likely
interacted
with
the
nvidia
container
toolkit
through
a
docker
command,
like
the
one
shown
here.
This
basically
says,
launch
a
docker
container
with
gpus
0
and
1
injected
into
it,
and
run
nvidia
smi
over
them
to
make
this
possible.
This
command
hooks
into
a
component
called
nvidia
docker,
which
is
just
one
small
piece
of
the
overall
nvidia
container
toolkit
that
I've
been
talking
about.
A
The
toolkit
itself
consists
of
a
stack
of
components
allowing
gpus
to
be
used
by
many
different
container
runtimes
different
runtimes
hook
into
different
layers
of
this
stack,
depending
on
their
integration
points,
for
example,
container
d
hooks
in
here,
cryo
hooks
in
here
and
lexi
hooks
in
here
and
of
all
of
these
components.
This
bottom
one
is
the
most
important
because
it
does
all
of
the
heavy
lifting
for
injecting
gpu
support
into
a
container.
A
A
For
example,
docker
can
be
configured
like
this
container
d
can
be
configured
like
this,
and
cryo
can
be
configured
like
this,
and
once
you
have
that
set
up
a
component
called
the
nvidia
kh
device.
Plugin
can
be
installed
to
allow
gpu
resources
to
be
requested,
as
shown
here
when
invoked.
This
plugin
sets
things
up
so
that
the
nvidia
container
toolkit
will
be
triggered
under
the
hood
to
inject
any
necessary
libraries
and
gpu
hardware
into
a
container
at
runtime.
A
A
Of
course,
you
can
always
specify
the
full
uuid
of
the
mig
device
you
would
like
to
inject
as
well
likewise
in
kubernetes.
If
this
is
how
you
request
gpus
in
a
pod
spec,
then
requesting
access
to
a
mic
device
can
be
done.
Similarly,
instead
of
asking
for
a
resource
type,
nvidia.com
gpu,
you
instead
ask
for
a
resource
type
of
nvidia.com,
slash
make
dash
whatever
make
device
type
you
want
in
this
case
1g.5gb.
A
A
A
A
A
A
Well,
it
boils
down
to
a
new
abstraction
provided
by
the
nvidia
kernel
driver
called
nvidia
capabilities
whenever
a
gpu
instance,
or
compute
instance
is
created.
A
set
of
capability
files
are
created
under
slash
proc,
which
point
to
the
set
of
device
nodes
under
slash
dev,
which
will
give
a
user
access
to
them.
A
A
A
A
Well,
unfortunately,
it's
not
quite
as
straightforward
as
one
might
hope.
There
are
actually
two
distinct
workflows.
You
need
to
consider
when
configuring
a
gpu
for
use
with
mig,
enabling
mig
mode
on
a
gpu
in
the
first
place
and
then
taking
a
mig-enabled
gpu
and
configuring
it
with
a
set
of
mic
devices.
A
The
commands
seen
here
can
be
used
to
perform
both
of
these
steps
in
the
case
of
creating
7
1g
5gb
devices
across
all
gpus
on
a
node,
but
what
steps
are
required
to
enable
mig
mode
on
the
gpu?
Well,
unfortunately,
it's
not
as
simple
as
just
running
the
command
I
showed
on
the
previous
slide
under
the
hood.
A
mig
mode
switch
actually
requires
a
full
gpu
reset,
which
not
only
requires
that
all
gpu
workloads
are
complete,
but
also
that
all
support
clients
are
disconnected
from
the
underlying
nvidia
driver,
including
components
like
the
kx
device.
A
A
And
finally,
these
last
two
challenges
shown
here
are
the
ones
most
complained
about
when
talking
with
users
of
make
mig
configurations
do
not
persist
across
a
node
reboot
and
managing
make
device
state
across
a
cluster
of
machines
is
not
well
supported
to
address
each
of
these
challenges.
We've
created
a
tool
called
the
nvidia
mig
partition
editor
or
make
parted
for
short.
A
A
A
And
with
that,
I'm
now
going
to
show
a
demo
of
all
of
these
pieces
in
action.
I'll
first
show
a
kubernetes
setup
capable
of
advertising
eight
full
gpus
on
a
dgx
a100
box
and
run
a
pod
to
consume.
One
of
these
gpus
I'll
then
reconfigure
the
box
to
advertise
56
1g.5
gb
devices
using
the
mixed
mig
strategy
and
run
a
pod
to
consume
one
of
these
mic
devices.
A
A
A
A
A
A
Now,
I'm
going
to
show
you
the
contents
of
a
mig
parted
configuration
file,
where
the
main
thing
to
note
here
is
the
all
1g
5gb
configuration
that
I
have
highlighted
this
configuration
says
to
take
all
gpus
on
the
node
and
divide
them
into
seven
1g.5gb
devices
each
and
when
I
then
run
this
next
command.
This
new
configuration
will
be
applied.
A
A
Where
we
now
see
that
there
are
56
nvidia.com
gpu
resource
types
instead
of
eight,
likewise,
our
labels
have
all
been
updated
to
put
the
mig
properties
directly
on
the
gpu
resource
type
rather
than
in
the
individual
named
mig
device,
where
the
product
label
now
encodes,
which
mig
device
this
resource
corresponds
to.
So
user
can
set
their
node
selector
appropriately.