►
From YouTube: GPU-Accelerated AI and ML with OpenShift Container Platform | OpenShift Commons Gathering KubeCon NA
Description
GPU-Accelerated AI and ML with OpenShift Container Platform
Diane Feddema (Red Hat) | Michael Bennett (Dell)
OpenShift Commons Gathering KubeCon NA
November 17, 2020
This presentation by Dell Technologies and Red Hat will go over OpenShift Container Platform 4.3, and show how we have proven the performance of containerized machine learning workloads accelerated with NVIDIA GPUs and how they can match baremetal systems. You will see the results of our development effort including MLPerf benchmark results that show performance of training and inference on both the Dell EMC PowerEdge R740 and C4140 NVLink platforms using NVIDIA T4 and V100 GPUs. This session will also cover monitoring dashboards built with Grafana and Prometheus that enable IT operators to track metrics relevant to this validated design. Finally, there will be discussion of the essential software tools that enable machine learning on OpenShift such as NGC containers and the NVIDIA GPU operator.
466770 GPU Enabling OpenShift for AI and ML workloads
A
Hey
my
name
is
michael
bennett
and
I'm
here
to
talk
to
you
about
the
gpu
accelerated
machine
learning
with
the
openshift
container
platform.
I
have
diane
federna
here
who's,
a
principal
software
engineer
at
red
hat,
who
will
be
presenting
the
performance
results
from
our
ml
ml
perf
benchmark
on
this
platform.
A
So,
to
start
out
talking
about
ai
and
machine
learning,
artificial
intelligence
is
a
big
subject
that
encompasses
many
things,
of
which
machine
learning
is
only
a
subset
of
what
enables
artificial
intelligence.
Artificial
intelligence
is
also,
you
know,
driven
with
automation
and
being
able
to
act
on
the
predictions
or
the
classifications
that
are
made
by
these
machine
learning.
A
A
And
so
what
we
like
to
do
in
order
to
break
down
the
process
into
a
journey
that
can
have
smaller
steps
is
to
first
discover
the
problem
that
we're
trying
to
solve
with
artificial
intelligence,
and
usually
that's
a
business
problem
that
we
want
to
solve,
not
an
infrastructure
problem.
The
infrastructure
will
just
be
used
to
run
the
solution,
and
so
after
we
have
discovered
our
problem,
we
will
begin
to
explore
what
data
we
have.
A
And
so
bringing
all
this
back
to
kubernetes,
we
enable
it
with
this
stack
of
what
we
call
ngc
containers
which
run
on
this
red
hat
openshift,
environment
and
then
on.
Finally,
underlying
dell
emc
data
center
infrastructure,
and
so
that
delivers
you
your
containers,
your
gpu
acceleration,
as
well
as
red
hat
openshift
for
enterprise
kubernetes,
and
then
the
ngc
software
stack
from
nvidia.
That
is
optimized
for
ai
model
development
and
training
and
includes
all
of
the
latest
sdks
like
tensorflow,
pi
torch
and
cafe.
Inside
of
those
different
ngcs
software
registry.
A
A
A
And
so
we've
talked
about
artificial
intelligence
and
why
gpus
and
containers
are
a
good
choice
for
developing
and
hosting
your
artificial
intelligence
application,
and
now
I'm
going
to
touch
on
the
specific
benefits
of
red
hat
openshift
for
your
enterprise,
kubernetes
environment.
That's
going
to
be
running
ai
workloads.
A
A
We
still
have
to
gather
data
deploy,
implement
in
apps,
but
of
course,
in
order
to
do
the
ml
model,
development
and
deployment
we'll
need
the
ml
frameworks
like
tensorflow,
as
well
as
devops
tooling,
like
selden,
to
enable
us
to
deploy
those
models
into
an
environment
where
they
can
be
reached
at
as
an
api.
Endpoint.
A
We'll
also
need
to
tie
in
data
sources
that
the
model
trains
on
or
we'll
be
running
inference
operations
against.
So
we
have
to
plumb
in
those
data
sources
and
then,
ideally,
you
know
we
would
want
a
platform
that
is
self-service
so
that
our
developers
can
request
new
containers
or
pods
on
demand
and
move
things
between
development
environments
and
production
with
minimal
need
to
interact
with
operations
teams.
A
It
is
a
operator
that
red
hat
has
for
data
science
tools
and
deploying
them
on
top
of
the
open
shift
environment,
and
so
it
if
you're
familiar
with
kubeflow,
you
know
it.
It
does
services
like
that
for
data
science
workbench
functionality,
but
it
also
enables
spark
and
tensorflow
deployments
and
then
logging
data
to
prometheus
and
grafana.
So
it
allows
us
to
deploy
all
those
different
components
we
saw
in
the
earlier
slide
where
the
ml
code
was
only
a
small
part
of
it.
A
Open
data
hub
ensures
that
we
can
not
only
run
the
ml
code,
but
that
we
can
have
all
the
supporting
services
around
it,
and
so
openshift
is
a
it's.
A
trusted.
Enterprise
kubernetes
red
hat,
puts
a
lot
of
effort
into
both
upstream
kubernetes
release,
but
then
also
taking
the
kubernetes
releases
and
hardening
them
and
then
making
them
production
ready
for
openshift
and
dell.
Emc
works
with
red
hat
to
validate
these
openshift
releases
on
dell
emc
hardware
to
ensure
that
our
customers
have
the
best
experience.
A
A
They
provide
ansible
scripts
as
well
as
installer
tooling,
for
to
enable
deployment
on
hci
infrastructure
like
vsphere
or
public
cloud
instances,
and
then
they
support
auto
scaling
of
the
resources
like
worker
nodes,
inside
of
your
kubernetes
environments,
and
it's
very
easy
to
do.
Updates
with
one
click
on
the
home
page
of
your
kubernetes
cluster
manager.
A
Red
hat
also
has
certified
operators
which
are
built
with
the
operator
framework
and
then
certified
through
the
open
shift
operator
certification.
At
that
point,
then
they
can
go
into
the
operator
hub.
That
is
integrated
into
openshift
4,
which
you
can
see
on.
The
right
is
a
portal
right
inside
of
the
cluster
manager
to
make
it
easy
to
deploy
these
operators,
and
they
all
have
different
levels
of
automation
that
when
you
drill
into
them,
they
show
you
if
it
supports,
for
example,
automated
lifecycle
management
or
just
automated
deployment
and
scaling
things
like
that.
A
So
it's
really
cool
I
enjoy
knowing
my
operators
are
certified
and
going
to
work
when
I
install
them
and
then
another
benefit
of
red
hat.
Open
shift
for
kubernetes
is
right
at
core
os
that
which
is
an
immutable
operating
system
with
core
os.
All
of
the
operating
system.
Images
are
managed
by
the
machine
operator,
and
so
the
systems
are
and
operating
system
configurations
get
handled
with
apis
and
def
configuration
definitions,
and
so
you
don't
have
to
worry
about
mismatched
settings
inside
of
the
hypervisor
os
that
are
running
your
containers
and
things
like
that.
A
The
design
is
pre-validated
in
in
the
lab,
but
we
create
a
lot
of
collateral
that
enables
our
systems
engineers
to
customize
and
right-size,
the
ready
stack
for
customer
usage
and
you
can
find
on
infohub.delltechnologies.com
a
lot
of
information
like,
for
example,
the
reference
architecture,
design
and
deployment
guides.
Some
solution
briefs
and
some
videos
and
blogs
talking
about
the
benefits
of
openshift.
A
In
the
center.
We
have
the
dell
emc
ready
stack
hardware,
where
we
used
three
powerjar
640s
as
manager
nodes,
as
well
as
different
gpu
accelerated
server
options
that
we
have,
with
the
dss
8440
being
a
10
gpu
server
for
dense
configurations
and
the
c4140
supporting
four
servers
with
nv
link
or
for
gpus
within
vlink,
I'm
sorry
and
then
to
manage
the
identification
of
the
appropriate
workers
with
gpu
resources
and
install
the
nvidia
drivers
stack.
We
have
the
nvidia
gpu
operator
inside
of
openshift
and
then
finally
on
the
right.
We
have
those
ngc
containers.
B
Michael
gave
us
an
introduction
to
the
benefits
of
running
aiml
models
in
openshift,
using
the
dell
ready
stack
now
we'll
talk
about
the
benchmarks
we
ran
in
the
dell,
ai
innovation
lab
and
how
we
monitored
them.
We
chose
the
mlperf
benchmarks
to
validate
our
remnants
architecture
because
they
are
all
open
source
they're,
created
and
supported
by
more
than
40
industry
and
university
research
organizations
and
they're,
designed
to
help
people
choose
the
right
ml
infrastructure
for
their
applications.
B
The
models
in
mlperv
are
neural
networks
that
train
on
publicly
available
data
sets.
You
can
go
to
mlperf.org
and
download
the
benchmarks
yourself
and
try
them
out
now.
We'll
take
a
look
at
the
lab
setup.
We
had
in
the
dell
technologies
ai
innovation
lab
in
austin.
On
the
left.
You
see
the
cluster
system
admin
host,
which
serves
as
a
single
entry
point
into
the
openshift
cluster.
B
B
And
now
we'll
talk
a
bit
about
monitoring,
so
this
is
how
we
monitor
our
monitored
our
benchmarks
that
we
were
running
in
the
lab.
You
can
see
the
dell
cluster
on
the
upper
left
running
openshift.
Next,
you
see
the
nvidia
dcgm
exporter
that
is
installed
by
the
nvidia
gpu
operator
and
exports.
The
metrics
from
the
worker
nodes
that
have
gpus
the
dcgm
exporter
is
continuously
exporting
metrics
from
the
gpu
worker
nodes.
B
Then
the
prometheus
time
series
database
scrapes
those
metrics
and
puts
them
into
a
time
series
database
and
you
can
roll
back
and
look
at
those
metrics
later.
If
you
have
a
reason
to
because
they
are
stored
in
a
database
and
then
we
are
continuously
displaying
those
metrics
on
grafana
dashboards
that
we
created.
So
in
this
case
I
created
this
dashboard
and
created
the
queries
that
we
see,
but
you
can
also
download
from
grafana.org.
B
B
B
Now
we'll
look
at
the
system
details
for
the
cluster
that
we
for
the
the
worker
node
that
we
ran
the
training
benchmarks
on,
and
so
you
see
on
the
left
column,
we
have
the
dell
c4140,
and
this
is
the
hardware
and
software
stack
that
we
had
for
that
worker
node
on
the
right.
We
have
the
nvidia
dgx1
comparator
system
that
we
used.
The
published
results
to
you
know
measure
how
well
we
were
doing
on
on
our
openshift
cluster,
so
both
of
these
systems
have
v100
gpus.
B
B
It
is
running
openshift
and
gives
you
all
these
added
benefits,
and
what
we'll
see
in
our
results
is
that,
even
though
you've
got
this
extra
layer
of
software,
there's
virtually
no
performance
penalty
that
we
paid
because
of
that
we
ran
a
slightly
different
operating
system.
We
ran
red
hat
core
os
4.3
on
the
dell
in
the
dell
lab,
and
our
comparator
system
was
running
ubuntu.
B
In
both
cases,
we
used
the
pi
torch
version
that
is
optimized
for
nvidia
gpus
and
on
down
the
line,
you
see,
there's
a
slightly
different
level
of
cuda
and
cuda
driver
that
were
used
in
the
two
systems,
and
everything
else
was
the
same.
The
exception
of
the
fact
that
we
had
additional
software
running
the
entire
time
that
we
were
benchmarking
so
that
we
could
monitor
our
performance
so
that
we
could
go
back
and
then
tune
those
applications.
B
There
are
computer
vision
and
natural
language
processing
models
in
this
benchmark,
so
we
trained
four
models:
two
were
object,
detection
models
and
two
were
natural
language.
Processing
models
mask
our
cnn
and
ssd
are
object,
detection
and
gnmt
and
transformer
are
natural
language
processing
models.
B
So
the
public
data
sets
are
listed
here
as
well
that
we
use
to
train
the
models.
So
coco
is
the
common
objects
in
context,
data
set
created
by
microsoft
and
cornell,
and
it
contains
300
000
more
than
300
000
images
that
are
that
have
objects
in
them
that
are
labeled
and
for
the
translation
benchmark.
B
So
these
these
public
data
sets
are
the
way
that
you
train
the
neural
network
and
all
of
these
benchmarks
were
donated
by
companies
like
facebook
and
google
and
they
are
used
to
solve
real
world
problems
such
as
google
translate
and
applications
like
that.
So
along
the
bottom,
we
have
the
timings
that
we
received
and
we're
gonna
that
we.
A
B
B
B
We
didn't
do
quite
as
well
on
these
there's
a
random
seed,
that's
used
to
get
these
benchmarks
started,
and
that
affects
the
number
of
epochs
that
the
model
takes
to
train.
So
if
we
had
run
this
benchmark
more
times,
we
feel
that
we
could
have
gotten
it
down
to
about
half.
As
much
or
twice
as
much
training
time
as
the
dgx
one,
so
now
we'll
look
at
these,
we
look
at
these
same
numbers.
Only
normalized
per
gpu
and
again
you
see
like
we
did
better
for
the
mask.
B
So
now,
moving
on
the
next
set
of
benchmarks
are
the
inference
benchmarks,
and
this
is
when
you
put
your
trained
model
to
work,
you
give
it
queries
and
it
gives
you
resolve
and
you
have
a
quality
metric
that
you're
aiming
for
you,
your
benchmark,
fails
if
you
don't
meet
that
quality
metric
and
all
the
benchmarks.
We
ran,
of
course,
did
meet
the
quality
metric
and
were
successful.
B
B
The
single
stream
benchmark
is
intended
for
smartphones
and
the
multi-stream
benchmark
is
intended
for
embedded
systems.
So
what
happens
here?
Is
you
have
something
called
load
gen
which
generates
an
artificial
load
to
run
the
benchmarks?
It
sends
queries
to
the
model
and
in
the
case
of
the
offline
scenario,
it
sends
all
the
queries
in
one
big
batch
and
then
you
get
the
answers
all
in
one
batch
and
in
the
case
of
the
server
scenario,
it
uses
a
pison
distribution
to
simulate
the
burstiness
of
queries
coming
in
that
are
more
realistic
to
real-life
situations.
B
A
B
Per
inference
results
that
we
are
comparing
to
are
from
the
mlp.org
website.
You
see
them
off
on
the
right
here
and
all
the
published
results
for
these
benchmarks
had
ecc
turned
off
error
correction
turned
off
so
turning
ecc
off
does
give
you
a
performance
improvement
of
13.14
to
14.29
percent,
depending
on
the
models
in
this
group.
B
But
the
cons
of
turning
off
ecc
are
that
single
bit
memory
errors
would
not
be
detected
or
corrected
and
double
bit
errors
would
not
be
reported,
so
we
decided
to
leave
ecc
on
so
that
is
going
to
slow,
slow,
our
results
down
a
little
bit,
but
any
of
our
customers
in
the
financial
industry
or
the
scientific
computing
community
and
any
situation
where
safety
is
involved.
You
would
want
this
extra
level
of
security
for
your
data
and
basically
it's
just
cheap
insurance
that
you
you
want
to
keep.
So
we
decided
to
leave
ecc
on
that.
B
And
you
can
see
that
that
these
are
the
server
results
and
in
this
case
higher
is
better,
so
we're
doing
inferencing.
We
want
to
answer
as
many
queries
as
we
can
as
quickly
as
we
can
and
we
are
comparing
one
t4
on
the
left
with
four
t4s
on
the
right.
The
other
difference
between
the
left
and
the
right
is
that
on
the
left,
we're
running
openshift
and
on
the
right,
the
comparator
numbers
that
we
got
from
the
mlperp.org
website.