►
From YouTube: State of the GPU(nion) by Rudi Chiarito, Clarifai
Description
State of the GPU(nion) - Rudi Chiarito, Clarifai
As Kubernetes becomes more popular, the number of use cases for it grow as well, bringing with them more specialized requirements. Scheduling and launch of containers that interact with GPUs is one of such new frontiers. We'll talk about the current state of affairs, as well as ongoing work in the area. Learn also how an end user of Kubernetes can get involved and become a code contributor.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
B
B
A
Hi
I'm
Rudy
kirito,
our
Conor
infrastructure
at
clarify,
welcome.
So
today's
talk
is
going
to
be
covering
two
main
areas.
First
of
all,
we're
going
to
be
talking
about
scheduling
and
the
second
smaller
part
is
going
to
be
about
how
to
become
or
be
a
contributor
by
accident,
so
Who
am
I
and
how
involve
never
been
with
open
source,
so
I
mean
in
what
we'd
open
source
for
about
16
years
now,
mostly
minor
contributions.
For
example,
when
I
was
at
amiga,
we
had
a
developer
system
that
Rena
linux
distribution
I.
A
To
maintain
that
data
Mayo
Clinic
I
did
more
work
more
related
work.
We
had
one
project
that
was
brand
new
code.
Actually
most
of
what
I
wrote
you
cannot
see
because
it
was
about
binary
file
formats
that
I
reverse-engineered.
So
we
can
open
source
that
then
I
went
to
Google
work,
necessary
and
I
was
pretty
much
I
disappeared
from
the
planets
fun.
As
far
as
open
source
is
concerned,
then,
last
year
I
joined
clarifying
I've
been
working.
A
lot
with
Kobernus.
A
Let's
see
I
should
give
you
some
background
about
clarify
because
somehow
pretty
much
the
work
that
has
been
done
so
far
for
GPU
resources
has
been
influenced
by
our
needs,
because
we
were
the
first
to
push
for
for
this.
So
we
are
a
start-up
based
on
based
in
New.
York
was
funded
by
our
CEO
after
he
finished
his
PhD
and
won
the
image
net
contest
in
2013
we
have
an
API
offering
machine
learning
as
a
service.
A
You
give
us
images
and
videos,
we
give
you
tags,
we
use
pre
built
models,
custom
models
that
you
can
build
yourself
and
then
we
also
allow
image
search.
We're
a
number
of
investors.
I
mentioned
them
because
India
is
going
to
be
mentioned
throughout
the
talk
and
I
want
to
have
the
disclaimer
that
they're
one
of
our
investors.
But
it's
just
a
coincidence.
A
Let's
see,
let's
talk
about
workloads
when
the
crowd
here
is
using
gpus,
okay,
fair
amount,
how
many
on
Cooper
Nerys
Wow
cool,
so
I
like
to
classify
workloads
in
two
broad
categories.
First
batch-
and
this
is
probably
the
vast
majority
of
workloads
stuff,
like
runner
farms,
let's
say
you're,
rendering
the
next
Pixar
movie.
This
is
going
to
be
running
for
hours
days
weeks,
whatever
or
your
training
machine
learning
models,
and
then
there
are
more
real-time
workloads
where
you
have
stuff
like
machine
learning.
A
Predictions
we'll
we'll
talk
a
little
bit
more
about
that,
more
specifically
for
clarify
we
trained
neural
networks.
First,
this
is
a
very
intensive
process.
We
have
millions
and
millions
of
images
and
then,
after
hours
days,
we
get
a
small,
much
smaller
model
out
of
this
whole
process.
It's
a
bunch
of
parameters
that
can
recognize
images.
Basically,
then,
out
of
that
we
we
feed
that
into
the
next
kind
of
work
on
which
is
making
predictions,
so
we
feed
the
model
with
a
much
smaller
set
of
images.
A
C
A
Cooper
natives
on
AWS,
the
first
kind
of
work,
was
widowed
run
on
coronaries.
Yet
when
I
joined
clarify,
we
had
an
old
infrastructure
that
was
starting
to
show
its
age.
It
was
running
on
virtual
machines,
which
created
when
the
company
had
maybe
just
three
or
four
employees.
It
was
very
awkward,
even
if
you
had
to
add
a
new
model
year
to
rebuild
a
whole
new
set
of
virtual
machines,
transfer
traffic
over
and
then
kill
the
ill
ones.
A
Well,
that
sounds
good
on
paper.
The
problem
is
that
in
practice
coordinate
is
they
don't
have
all
the
features
that
we
really
needed?
For
example
around
this
time
last
year
there
was
no
GPU
support.
If
you
use
a
lbs,
you
couldn't
easily
do
ssl
encryption.
There
was
no
support
for
ECR
and
there
were
a
bunch
of
bugs.
You
could
easily
get
your
masters
or
your
nose
to
make
too
many
calls
to
AWS
and
your
whole
account
will
get
throttled.
So
we
would
see
I've
a
run
stuff
on
our
test.
A
Cluster
and
the
whole
production
environment
would
get
throttled
that
wasn't
fun
in
wasn't
even
easy
to
to
debug.
So
how
did
we
tackle
GP
support?
As
I
said,
that
was
a
Kinetoscope
con
last
year,
I
approached
Eric,
TuneIn,
David,
Oppenheimer
and
I
asked
them
about.
You
know
how
about
we
support
GPS
and
magic
happened.
Well,
not
really.
A
It
took
a
few
months,
many
discussions
and
ideas,
and
it
turns
out
that
you
cannot
just
extend
the
governess
API
with
you
know,
with
the
new
objects
and
interfaces
and
then
later
risk
in
to
undo
that
so
I
had
several
proposal,
but
initially
we
just
settle
on
something
that
we
call
be
0,
which
was
very
stripped
down
very
basic.
This
has
been
available
since
1.3
in
July,
so,
as
I
said,
this
has
been
stripped
down
to
the
bare
minimum,
so
it
has
many
limitations.
A
For
example,
you
can
only
use
in
video
cards,
there's
only
support
for
one
device
per
machine.
It
doesn't
get
auto
detected.
You
cannot
tell
if
you
have,
you
know
the
brand
new
latest
Nvidia
card
or
something
from
four
years
ago.
You
can't
even
find
out
easily
how
much
GPU
ram
you're
using
or
if
you,
if
you're,
using
something
like
the
kubernetes
dashboard
data
dog,
you
can't
even
say
hey,
I'm
using
maybe
eighty
percent
of
my
my
GPU
devices
and
last
but
not
least,
the
resource
name,
is
kind
of
awkward.
A
A
Also,
there
is
a
problem
if
you
build,
for
example,
a
version
of
if
you
build
your
software
against
the
version
of
cuda,
that
you
need
to
have
a
matching
driver
that
supports
that
at
least
at
dub
version
and
the
real
problem
there.
It's
not
just
you
have
this
limitation
that
if
you
mess
up
it's
going
to
be
very
difficult
to
even
find
out,
because
you're
going
to
get
a
very
unhelpful
error
message
and
there's
no
check
in
the
queue
bullet
that
you
know
even
reject
the
even
refuses
to
start
the
container.
A
A
Second,
you
need
to
expose
the
libraries
and,
if
you
use
in
BDSM
I,
which
is
a
diagnostic
binary
you
to
expose
that
into
your
pod.
You
don't
want
to
package
that
into
your
images,
because,
as
I
said,
the
low-level
libraries
are
need
to
match
the
kernel
driver.
So
putting
the
library
in
the
image
is
a
big
no-no
and
then
last
but
not
least,
you
have
to
request
the
resource.
So
this
is
what
it
looks
like
in
a
yellow
file:
I
split
them
over
two
columns,
but
basically
on
the
left
side.
A
So
why
would
you
do
this?
It's
a
lot
of
work.
You
know,
as
I
said,
there's
a
you
need
to
expose
libraries,
a
lot
of
note
set
up
that
is
necessary
for
us.
Even
with
all
this
work,
it's
still
a
big
plus
because
we
went
from
three
hours
to
maybe
mean
it's
to
deploy
a
simple
code
change
and,
if
I
just
need
to
add
a
new
model.
I
just
click
on
the
dashboard
and
I
have
a
new
replica
up
in
a
few
seconds.
So
this
was
worth
it.
A
Still
we
we
know
there
is
a
lot
to
do.
There
is
a
PR
from
we.
She
works
at
HP,
which
is
going
to
discover
devices
for
you,
so
you
don't
need
to
pass
the
magic
flag
anymore.
You
can
now
support
more
than
one
GPU
/
machine.
Not
only
that
if
you
have
a
container
that
needs
more
than
one
GPU,
you
can
request
that
as
well.
There
is
a
GPU
plug-in
abstraction,
but
it's
still
using
the
nvidia
GPU
resource.
A
So
it's
we
know
there's
a
lot
to
do
after
after
this
as
an
alternative,
so
the
PR
from
ug
just
scans
entries
in
the
deaf
directory
and
it
finds
the
devices.
So
you
notice,
for
example,
there
are
two
devices,
so
in
knows
there
are
two
in
media
cards.
Hopefully
there
is
an
alternative,
which
is
the
MV
MV
ml
library,
which
not
only
tells
you
how
many
devices
are
on
the
machine.
You
can
also
find
out
all
sorts
of
details
like
vintage
memory,
core
account
even
affinity.
A
For
example,
there
might
be
four
cards
on
the
system,
but
two
of
them
are
faster
when
talking
to
each
other
compared
to
the
other
two
and
so
on.
The
problem
with
MV
ml
is
that
it's
native
code?
So
you
need
to
use
sigo
and
I,
don't
believe,
there's
any
Kobernus
binary
that
uses
sigo
yet
so
stability
and
other
concerns
would
be
getting
the
way
of
merging
nvm
L
support.
A
What
else
do
we
need?
We
definitely
need
to,
as
I
said,
expose
more
details
about
the
hardware.
In
our
case,
for
example,
we
have,
we
call
it
the
general
model,
it's
the
largest
model
that
can
bear
is
trained
to
detect
tens
of
thousands
of
concepts
of
tags,
and
that
requires
the
largest
hardware
that
we
have.
But
then
we
have
many
smaller,
almost
one
of
models
for
customers
for
special
that
uses
and
those
don't
need
a
very
beefy
card
or
machine
so
being
able
to
tell
one
device
from
the
other
would
be
helpful.
A
To
reduce
waste
then
also
exposing
affinity
would
be
would
be
ideal.
And
last
but
not
least,
we
would
need
some
kind
of
reschedule
er.
This
is
something
that
has
been
known
to
to
the
coronaries
project
for
a
while.
There
is
there's
an
issue
that
has
been
open
for
maybe
you're
too.
Basically,
for
example,
let's
say
you
have
four
machines
with
two
cards:
each
and
all
of
them
have
one
card
in
use.
A
Now
someone
comes
in
and
they
they
request
two
cards
and
they
cannot
do
anything
because
there
is
capacity
left,
but
it's
all
stranded
across
multiple
machines.
So
a
reschedule
would
come
in
and
evict
one
random
pod
free
up
two
cards
and
then
let
the
new
one
let
the
new
pod
schedule
on
on
that
machine.
This
is
not
really
GPU
specific.
You
can
get
into
that
kind
of
scenario
with,
even
just
with
cpu
and
ram.
If
you're
very
unlucky.
A
What
else
is
needed?
I
show
the
earlier
the
yamo
for
setting
up
the
pod
and
that's
very
ugly.
It's
a
lot
of
work.
You
can
make
easily
mistakes
and
you
can
easily
share
this
with
others.
You
have
to.
You
would
have
to
agree,
for
example,
where
in
which
host
site
directory
your
libraries
are.
We
have
our
own
set
up
with
because
we
use
Korres.
A
Also,
there
is
no,
as
I
said,
when
the
low
level
libraries
don't
support
the
cuda
version
that
your
your
image
is
built,
for
there
is
no
helpful
message.
The
container
will
just
get
started
over
and
over
and
fail
every
time.
There
is
something
that
there
is
a
project
that
immediate
created.
It's
called
a
media
docker,
it's
a
volume
plugin.
The
idea
is
that,
basically,
you
start
a
demon
in
advance
and
then
the
the
doctor
demon
will
talk
to
it
when
it
needs
to
create
a
new
container
and
it
will
get
information
about.
A
It
will
be
given
a
directory
where
all
your,
where
all
your
libraries
are
so,
for
example,
let's
say
you
need
cuda
7.5
by
adding
a
label
to
your
docker
image.
The
the
media
docket
volume.
Plugin
knows
if
those
libraries,
if
that
version
of
code
is
supported,
if
it
supported
the
director,
will
be
exposed.
If
it's
not,
it
will
return
with
an
error.
So
the
container
wouldn't
start
I
have
a
PR
to
support
darker
volume,
plugins
and
the
BL
dr.
one
would
be
one
of
them.
A
A
Also
resource
monitoring,
as
I
said
earlier,
is
a
is
an
issue.
We
need
to
make
sure
that
you
don't
have
to
go
to
the
shell
and
run
command
line
scripts
like
we
do
to
show
resources
allocated
on
the
all
the
nodes
and
also
having
more
details
on
what's
happening
inside
the
container.
You
know
once
I
have
given
a
pod
of
GPU,
I
would
like
to
know
how
much
of
the
ram
its
beam
is
being
used
and
also
multiple
vendor
support.
A
This
is
all
invidious
specific
because
for
machine
learning
purposes
the
de
facto
standard
is
nvidia,
but
we
want
to
be
able
to
support
multiple
vendors
and
then
this
is
a
feature
that
might
not
apply
to
everyone.
If
you
have
more
than
one
pod
that
doesn't
use
the
GPU
the
whole
time,
you
will
be
straight,
you
will
be
wasting
resources
if
you
have
a
model,
for
example,
that
is
used
ten
twenty
percent
of
the
time.
Eighty
percent
of
time
you're
just
buying
a
very
expensive
GPU
and
doing
nothing
with
it.
A
So
right
now
you
can
actually
share
one
GPU
device
between
among
several
several
pods.
They
will
just
compete
for
each
other
for
access
and
one
of
when
one
of
them
is
done.
The
other
one
will
will
continue
and
undo
its
own
its
own
work.
This
is
currently
supporting
corners
with
a
weird
trick,
but
it's
probably
gone.
It's
definitely
going
to
break
when
the
new
discovery
code
lines.
So
we
need
to
find
a
better
solution.
A
Also,
sharing
a
device
means
that
you
will
have
to
ask
for,
for
example,
1
gigabyte
of
GPU
ram
out
of
4
or
6.
Whatever
is
on
the
card.
This
is
mostly,
I
want
to
say,
accounting
fiction,
but
it's
just
something
that
you
need
to
plan
in
advance
for
because
there
is
no
actually
enforcement.
If
I
asked
for
one
GPU,
one
gigabyte
of
GPU
ram,
there's
no
way
for
the
runtime
to
enforce
that.
So,
if
I
go
above
that
the
request,
I
I
will
cause
trouble
to
the
other
parts.
A
A
Even
if
you
have
a
use
case-
and
you
know
you
kind
of
contribute
code-
that's
very
helpful.
Many
times
I've
been
in
discussions
where
I
brought
up
something
that
needed
to
be
done,
but
I
was
the
only
person
talking
about
it.
So
your
voice
will
help
make
the
case
for
a
forum
or
working
in
this
area
and
also
for
prioritizing
the
work.
If
you
can
make
code
changes
even
better
now
how
to
become
an
incidental
contribute,
remember
the
features
I
mentioned
earlier
that
we're
missing.
A
These
are
not
all
the
features
that
we
needed,
but
these
are
the
ones
that
were
the
most
urgent,
so
they're,
not
missing
anymore.
How
did
that
happen?
So
nobody
was
working
on
those
I
open
a
bunch
and
nobody
really
was
working
on
them.
So
I
said:
let's
do
it.
I
started
with
something
very
small
bugging,
the
way
with
the
traffic
shaping
and
also
we
needed
easier
support.
Nobody
was
working
on
it.
I
ended
up
doing
it.
A
How
do
you
do
it?
My
first
most
important
advice
is
to
start
with
something
very
simple:
you
find
a
bug,
usually
they're,
very
restricted
in
the
in
the
amount
of
work
that
day
they
need
what
they
involve,
and
then
this
way
you
get
familiar
with
the
whole
process.
You
know
the
code
review
the
how
to
build
on
tesco,
Burnett
is
and
how
to
rebate
more
about
that
later
then,
once
you've
fixed
a
bug
or
two,
you
can
start,
maybe
exploring
how
to
add
features.
You
don't
have
to
implement
something
crazy.
A
Just
you
know
maybe
a
cube.
Let
cook
control
flag,
that's
something
that
will
give
you
more
experience
with
with
a
code
base
and
then
maybe
from
the
area.
Wherever
you
start,
you
can
move
into
other
areas
of
the
project,
but
my
advice
there
is
not
to
over.
Do
it
c'mon?
It
is
a
very
large
and
complex
project.
A
If
you
want
to
learn
about
I,
don't
I,
don't
know
if
there's
any
one
person
that
knows
all
the
parts
of
the
project
very
well
I
would
advise
to
maybe
focus
on
three
four
five
areas
where
what
do
you
want
to
contribute
most
be
patient?
As
I
said
earlier,
rebasing
as
being
the
pain
of
my
existence,
especially
when
you
make
changes
to
the
API,
but
some
of
this
is
going
to
go
away
in
1.6,
we
won't
be
checking
in
auto-generated
code.
A
So
many
of
the
conflicts
that
happen
now
we'll
pick
on
you
don't
have
to
worry
about
rebasing
your
auto-generated
code,
also
when
you
are
really
when
you're
in
your
code
reviews,
it's
very
helpful
to
take
a
step
step
back
and
understand
what
your
reviewer
is
commenting
about,
because
sometimes
there's
a
lot
of
context
that
the
reviewer
doesn't
give
you.
But
if
you
think
about
it,
maybe
you
can
see
behind
between
the
lines
what
what
they're
trying
to
get
it,
and
maybe
some
objections
that
they
have
ever
reason.
It's
just
not
very
clear
in
the
way.
A
The
comment
I
like
to
sometime
pick
on
one
or
two
reviewers,
but
it's
just
a
joke:
I'm
not
enough
serious
about
it.
I
understand
that
they're
very
overloaded
with
work,
so
we
want
to
make
sure
that
we
don't
waste
their
time
until
free
to
ask
questions
if
you're,
rather
than
going
down
a
rabbit
hole.
Maybe
ask
one
good
question
before
you
try
to
change
your
PR
in
a
way
that
you
will
later
have
to
undo
either
improve
tests.
A
I,
don't
know
one
person
who
likes
writing
tests,
but
we've
been
bitten
by
a
few
cases
where
the
tests
were
missing
or
incomplete.
So
please
write
more
tests
and
also
documentation.
I
believe
I
had
one
or
two
features
where
I
wanted
to
what
I
was
about
to
file
or
even
file
one
and
then
sorry
filing
a
request,
and
then
I
found
out
that
the
feature
not
only
existed.
It
was
documented,
but
in
a
very
remote
place
in
the
documentation,
so
better
documentation
helps
a
lot
and
it
prevents
this
from
happening
tool
to
others.
A
Let
me
show
you:
the
I'll
show
you
where
it
happens
this
flag,
so
you
can
tell
that
flag.
There's
no
check,
there's
no
validation
check.
Somebody
brought
it
up
and
one
of
the
viewer
said
no,
let's
not
complicate
this
because
it's
going
out
and
it's
gonna
be
removed
eventually
and
I
also
didn't
want
to
put
the
check,
because
I
had
this
record
in
mind.
You
can
say,
for
example,
I
have
a
hundred
GPUs
and
then
you
request
20.
That
way
you
get
twenty
percent
of
one
GPU.
A
A
B
B
D
A
So
our
cola
machines
are
run
by
a
separate
team,
the
data
team
and
the
research
team.
They
don't
like
having
to
run
their
own
machines
and
they've
seen
they
run
some
stuff
on
coordinates
as
well,
so
they
like
how
flexible
it
is
and
how
much
time
it
saves
them.
So
we
want
to
move
everything
to
burn
areas
as
well.
But,
as
I
said,
there
are
features
are
missing
and
until
we
have
toes
we
cannot
do
the
migration
yet
yeah.
A
D
A
The
questions
about
sharing
GPU
ram
between
multiple
parts
yeah.
This
is
what
we
do
right
now,
as
I
said
with
this
trick.
Basically,
so
we
have
some
models
that
are
very
large
and
they
can
take
most
of
the
memory
and
then
we
can
give
the
rest
to
you
know
tiny
models
that
just
use
a
few
like
few
tens
of
megabytes
and
that
works
because
we
know
in
advance
we
do
some
kind
of
profiling.
We
know
how
much
Maori
each
other
and
use
it.
So
we
know
roughly,
but
this
is
all
it's
sold
yeah.
A
If
it's
not
automated,
it
should
be,
and
also
the
other
thing
is
that
we,
our
workloads,
are
fairly
static
in
terms
of
memory
location.
They
allocate
all
the
memory
at
the
beginning.
They
don't
get
anything
else
afterwards,
so
we
can
do
that,
but,
as
I
said,
you
cannot
enforce
that.
There's
a
this
is
just
there's
no
Mme
you
on
the
GPU,
basically,
otherwise.
Yeah
would
be
great
any
more
questions.