►
From YouTube: Kubernetes For GPU Powered Machine Learning Workloads In... - Camille Rodriguez & John-Paul Robinson
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2023 in Amsterdam, The Netherlands from April 17-21. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Kubernetes For GPU Powered Machine Learning Workloads In Academia - Camille Rodriguez, Canonical & John-Paul Robinson, University of Alabama at Birmingham
Speakers: John-Paul Robinson, Camille Rodriguez
This talk aims to inform the architects and users of Kubernetes, as well as teams planning to transition for Kubernetes for research purposes, how we designed a high-performing Kubernetes cluster specifically geared towards machine learning and AI workloads. On the architectural side, the use of NVIDIA DGX A100 machines provides unprecedented compute density and performance for those workloads. Those nodes are integrated to the cluster with open-source software. We will also cover our challenges & successes in integrating to other components, such as external CEPH storage, gitlab registry and runners, and SAML authentication. The University of Alabama at Birmingham team will cover how they leverage container-enabled GPUs for their research and development workloads. Research workloads increasingly demand access to ad hoc, GPU-enable compute capacity, with complex software environments to power cloud-native workflows. K8s helps address needs ranging from regular ML training runs to supporting software development via CI pipelines.
A
So
when
we
talk
about
bare
metal
deployments,
we
want
to
have
some
infrastructure
nodes.
You
can
also
call
them
like
management
nodes.
Essentially,
those
are
outside
of
the
kubernetes
deployment
to
be
able
to
manage
the
kubernetes
environment,
so
the
First
Technology
I'll
highlight
here
is
the
metal
as
a
service
platform.
So
Mars
is
a
burmesole
provisioning
tool
super
useful.
When
you
have
a
large
estate
of
bare
metal
servers,
it
provides
you
with
your
asset
inventory.
A
You
can
do
a
layout
of
the
type
of
storage,
layout,
networking
and
all
of
the
layout
that
you
want
to
do
when
you
pixiboot
and
deploy
your
servers
can
be
automated
through
Mass.
So
that
was
a
really
useful
tool
to
use
then
to
choose
which
machine
we're
deploying
and
to
deploy
them
with
application,
to
deploy
the
OS
and
to
deploy
different
applications.
We're
using
Juju
Juju
is
called
also
a
operator
lifecycle
controller.
Essentially
it's
one
of
the
best
aspects
of
it.
A
Is
that
there's
a
lot
of
day
two
operations
built
in
so
once
you
deploy
applications,
you
can
relate
them
to
each
other.
So
let's
say
you
want
to
relate
your
kubernetes
worker
to
the
control
plane.
It's
a
simple
relation
in
a
model
that
you
define
in
yaml,
so
it
makes
it
simple
and
when
you
want
to
scale
your
cluster,
for
example,
you
want
to
add
your
kubernetes
node.
You
can
simply
do
like
Juju,
add
unit
kubernetes
worker
and
it's
going
to
scale
for
you.
You
can
also
reduce
you
can
upgrade
and
update.
A
So
that's
the
main
manager
that
we're
using
In
This
Cloud
in
terms
of
observability.
It's
pretty
standard
we're
using
open
source
project
that
I'm
sure
a
lot
of
you
are
using
as
well
so
elasticsearch
grafana
Prometheus,
since
the
whole
deployment
is
done
on
Ubuntu
servers
we're
also
using
the
landscape
server.
That
tool
lets.
A
So
now
for
the
actual
kubernetes
deployment
control
plane,
we
decided
to
do
the
control
panel,
three
nodes
with
standard
components.
All
of
these
components
are
operators
or
charms
that
are
deployed
with
Juju,
so
Calico
for
the
networking
we
got
hcd,
qbpi,
load,
balancer
and
then
for
the
workers.
We
had
two
different
sets
of
workers.
A
It's
not
the
case.
In
this
case,
we
connected
to
an
external,
well
Standalone
theft,
cluster
separate
from
the
kubernetes
cluster.
The
main
reason
for
that
is
that
that
set
cluster
is
used
for
a
few
other
clusters
in
the
environment.
So,
for
example,
the
openstack
connects
to
it.
Some
users
can
use
the
file
share
system
from
it
separately
from
the
case
cluster,
so
ceph
provides
persistent
volumes
for
for
kids,
it
can
provide
block
storage.
We
have
S3
and
also
yes,
ffs.
A
A
Essentially
that
lets
you
slice
your
GPU
into
smaller
instances.
One
dgx
node
has
eight
GPU
cards
and
they're
really
powerful,
and
those
nodes
are
also
really
expensive.
So,
there's
a
limit
to
like
how
many
you
want
to
purchase
for
your
cluster
in
this
case,
if
you
slice
your
GPU
into
like
smaller
slices
you're
able
to
provide
more
gpus
for
different
types
of
workloads
that
may
have
different
requirements,
and
it
also
it
makes
it
so
that
each
GPU
is
independent
from
each
other.
A
So
in
our
in
our
case,
we
deployed
three
nodes
with
no
make
profiles
and
then
one
node
with
the
seven
slices
per
GPU
profile,
and
that's
also
something
that
you
can
change
later
on
when
you
see
that
the
need
or
the
request
for
one
type
of
GPU
is-
is
greater
than
the
other.
So
it
makes
it
a
pretty
easy
to
use
and
then
to
configure
those
profiles,
it's
simply
a
label
on
the
nodes.
A
A
On
the
networking
side
we
use
metal
lb.
So
for
those
of
you
who
are
familiar
with
kubernetes
and
public
clouds,
well,
you
have
access
to
applicational
balancers
that
you
don't
have
to
to
do
magic
to
have
set
up
in
your
environment,
but
on
bare
metal.
Unless
you
purchase
expensive
appliances
that
do
you
know
hard
roll
bouncing,
you
don't
have
that
many
options,
so
metal
B
is
a
really
useful,
open
source
project
that
can
simplify
your
life
for
load,
balancing
in
kids.
A
So
essentially
you
give
a
set
of
external
or
public
IPS
that
you
want
metal
B
to
be
able
to
assign
to
pods
and
if
a
pod
requests
a
service
type
of
a
type.
Well,
if
service
stuff
type
load
balancer,
it
will
get
additional
IP,
there's
two
different
modes:
Layer
Two
is
preferred
unless
you
have
switches
that
are
able
to
do
bgp.
So
in
our
case
we
did
a
layer,
2
setup.
A
A
The
sample
part
was
probably
the
trickiest
part
a
little
bit
because
of
networking
when
you
have
to
make
sure
that
your
pods
talk
to
the
right
thing
that
you
have
Keystone
talking
to
the
saml
backend
and
all
that
I'm
not
going
to
go
in
detail
through
this.
But
I
wanted
to
give
a
shout
out
to
Gustavo
Sanchez
a
guy
on
my
team
who
helped
a
lot
with
making
this
work.
A
B
Thanks
Camille,
so
what
I
thought
I'd
do
is
give
you
a
little
bit
of
a
overview
of
the
University
of
Alabama
at
Birmingham,
just
to
kind
of
help.
You
understand
the
the
scope
and
the
place
in
which
this
deployment
is
occurring.
So
UAB
is
a
large
public
institution
located
in
Birmingham,
but
the
metro
area
has
about
a
million
a
little
bit
more
than
a
million
people
in
it.
That
represents
about
a
fifth
of
the
state,
the
population
of
the
state.
B
We
also
are
the
largest
employer
in
the
state
and
have
a
very
large
economic
impact
on
the
region,
so
academically
we
have
about
over
20
000
enrolled
students,
a
good
third
of
those
are
in
The
Graduate
research,
space,
post
undergraduate
space,
and
we
generate
about
600
million
annually
in
research
funding
from
the
different
national
funding
agencies
like
the
NIH
heavily
and
the
NSF
research
Computing.
The
group
that
I'm
with
we're
part
of
the
I.T
organization
at
UAB
that
serves
the
campus.
B
B
B
So
a
lot
of
the
Kate's
use
cases
that
we
have
are
what
you
might
consider.
You
know
a
standard
fare
for
microservices.
We
have
a
self-service
application
that
we're
building
for
users
at
our
University,
for
our
researchers
to
be
able
to
kind
of
manage
their
lab
environments,
manage
the
resources
that
we
provide
for
them.
B
We
also
are
working
with
essentially
the
leading
adopters
of
Kate's
on
campus.
We
have
a
few
users
that
have
built
their
own
case
platform
that
they're
running
in
the
cloud
they're
interested
in
bringing
that
onto
campus,
and
we
have
some
use
cases
that
are
kind
of
leading
research
use
cases
for
applications
that
already
exist
that
are
case
oriented.
B
So
one
of
the
primary
spaces,
though,
that
we're
working
on
right
now
is
that
CI
CD
integration.
Basically,
as
we
build
these
applications,
we
want
to
be
able
to
make
sure
that
we
don't
fall
behind
in
our
in
our
workloads.
One
of
the
problems
that
we
had
in
our
own
development
was
that
we
would
just
essentially
get
a
huge
merge
backlog
in
our
workflow
and
so
we're
able
to
use
the
CI
CD
pipelines
that
we're
now
moving
over
to
Kate's
to
help
clear
that
out
by
having
nightly,
builds
and
moving
forward
with
that.
B
B
That's
why
we
kind
of
peppered
it
with
some
gpus,
also
along
the
way,
so
I
think
it's
also
helpful
to
understand
what
we
mean
when
we
speak
in
terms
of
like
research,
Computing
and
high
performance
Computing,
we
tend
to
think
of
our
platform
as
units
of
high
performance,
Computing
and
a
high
performance
computer
in
our
environment
is
something
that
has
a
considerable
amount
of
ram.
You
know
three
to
100
gigabytes
is
very
common.
We
have
a
few
that
have
a
above
a
terabyte.
We
have
a
couple
of
CPUs,
they
have.
B
You
know
24
to
64
cores
on
them
and
then
we
may
have
accelerators
in
those
nodes.
Each
of
those
nodes
definitely
has
a
high-speed
Nick
on
them
to
let
the
data
move
on
and
off
the
node
itself
and,
of
course,
in
in
the
HPC
context,
you
think
in
terms
of
speeds
of
network,
and
so
you
want
to
keep
your
your
data
close
to
your
CPU
with
the
highest
speed
and
then
your
internal
networks
at
the
next
highest
speed,
and
then
your
data
ingestion
onto
the
node
at
the
the
lowest
speed.
B
If
you
will
so,
we
combine
those
compute
resources
into
clusters.
So
we
just
kind
of
buy
a
bunch
of
them
at
a
time
and
we
stick
them
together
and
we
operate
them
on
a
connected
Network
and
then
we
make
make
it
possible
for
them
to
pass
data
in
between
each
other
and
reach
the
essentially
the
global
file
system
that
we
run
as
gpfs.
B
We
we
combine
those
into
a
fabric
that
we
then
expose
in
different
compute
flavors
and
our
kind
of
bread
and
butter
compute
flavor
is
the
HPC
batch
compute
flavor.
We
run
the
slurm
scheduler
if
you're
familiar
with
that.
What
you
do
inside
slurm
is
you
ask
for
a
certain
amount
of
ram,
a
certain
amount
of
cores
and
nodes
for
a
certain
period
of
time
and
then
slurm
allocates
that
when
it's
available
and
ensures
that
nobody
else
is
competing
for
that
resource.
B
It
also
provides
essentially
web
proxy
capabilities,
so
you
can
tie
a
web
proxy
into
a
job
and
you
can
run
things
that
are
kind
of
web
native,
like
Jupiter,
notebooks
and
rstudio,
and
things
of
that
nature,
but
a
lot
of
our
researchers
just
run
their
traditional
SMP
or
MPI
applications,
or
even
their
pleasantly.
Parallel
single
core
independent
data,
workflows
on
the
HPC
batch
and,
like
I,
said,
that's
our
biggest
workload
engine
right
now
today.
B
For
for
research
at
UAB,
we
also
have
an
openstack
cloud
and
we
provided
that
primarily
so
that
we
could
have
the
an
easy
to
use
platform
when
the
impedance
mismatch
to
our
cluster
is
high
right.
So
if
you
have,
for
example,
a
science
Gateway,
that's
a
very
common
tool
that
people
want
to
run
it's
some
sort
of
a
web
application
that
hides
the
details
of
a
complex
platform
behind
the
back
end
and
exposes
to
the
and
users
something
that
they
can
get
started
with
right
away
kind
of
like
our
on-demand
platform.
B
But
often
these
are
geared
towards
specific
science
domains
that
lets
a
new
graduate
student
or
new
researcher,
get
on
board
using
computation
and
Analysis.
In
their
world
much
more
quickly,
so
you
can't
readily
deploy
one
of
those
or
easily
deploy
one
of
those
into
an
HPC
batch
cluster.
There's
a
lot
of
things
that
don't
work
with
that.
So
that's
one
of
the
reasons
why
we
have
openstack.
We
also
have
it
to
help
with
our
development
workloads.
We
actually
do
development
cluster
Builds
on
on
that
openstack
platform.
B
Even
when
you
want
to
use
a
container-like
model
in
HPC
bash
with
Singularity,
you
still
need
an
environment
where
you
can
be
route
to
build
that
container.
So
for
all
of
those
reasons
we
have
an
openstack
cloud
and
then,
as
I
mentioned,
when
we
were
pursuing
Kate's
because
we
know
we
need
a
container-based,
abstraction
and
kind
of
an
orchestration
workload
manager
for
you
know,
next-gen
machine
learning
applications.
B
Those
are
a
lot
of
the
tools
that
we
see,
as
you
know,
Desiring
the
container
back
ends
and
right
now
we
we
didn't
have
a
platform
on
which
to
make
that
easy
to
just
consume
those
containerized
components
of
those
workflows
easily,
so
just
as
a
kind
of
a
reference
related
to
Camille's
slides.
This
is
where
we're
currently
using
Maz
and
JuJu
and
charm
deployments
inside
of
our
environment,
so
we
use
it
for
our
openstack
Cloud.
B
We
use
it
for
our
Kate's
container
environment
and
we're
working
actively
to
migrate
our
ceph
platform
over
to
that.
It's
still
going
to
be
an
independent
theft
cluster,
for
the
same
reasons
that
we
needed
to
have
it
independent
to
begin
with,
but
the
provisioning
is
going
to
happen
through
through
mass
and
JuJu.
B
Our
HPC
batch
is
still
done
in
a
traditional
HPC
batch
way
and
the
gpfs
is
an
independent
file
system,
so
kind
of
in
conclusion,
I
want
to
just
talk
a
little
bit
about
the
the
next
gen
use
case
that
we
really
see
for
for
Kate's
the
one
that
I
think
is
among
the
more
exciting
spaces
that
we
can
spend
our
time
with.
Over
the
coming
years,
as
you
can
see
on
the
you
know,
in
the
in
the
research
environment,
we
have
a
lot
of
different
data
sources.
B
B
That
already
have
existing
data
sets
that
come
in
and
they
come
into
your
environment
and
ideally
those
just
kind
of
stream
into
your
environment,
and
they
have
the
associated
workloads,
kind
of
just
managed
and
the
customizations
that
you
have
to
do
to
your
data,
to
publish
that
data
automatically
run
and
then
on
the
on
the
right
hand,
side.
We
have
our.
Essentially,
our
analysts
sitting
there
composing
analytical
workflows
through
the
through
the
Kate's
platform,
so
they
can
work
with
things
like
Jupiter
up
at
the
top
as
an
example
of
a
Jupiter
application.
B
That's
loading
a
lot
of
data
up
inside
it
and
presenting
a
you
know,
a
satellite
image
of
the
of
the
globe,
obviously,
but
that
needs
to
basically
allow
the
users
not
just
to
consume
the
data
coming
off
the
platform
but
package
up
that
computation
that
they
did
to
produce
this
new
data
set
back
into
the
platform
so
that
it
can
be
used
by
other
researchers
Downstream.
So
they
don't
just
become
consumers
of
data
sets
they
also
become
kind
of
instantaneously
or
immediately.
B
Even
though
they're
kind
of
a
slow-moving
animal
their
os's
are
reasonably
stable.
They
don't
change
very
often
the
the
os's,
the
applications,
the
the
storage
environments,
They
do
change
over
time,
and
so,
if
you
want
to
come
back
five
years
later
and
reproduce
a
run,
it's
going
to
be
very
difficult
to
do
that
so
being
able
to
essentially
capture
all
of
that
up
into
a
container.
B
Reproducibility
tool
for
researchers
over
time,
and
that
requires
that
it
doesn't
you
know
this
is
not
something
that
the
researchers
really
want
to
sit
around
and
design.
So
you
want
to
make
it
possible
for
them
to
essentially
say:
okay,
now,
capture
this
environment,
that
I
used
and
give
me
a
container
that
allows
me
to
come
back
and
use
it
again
in
the
future.
B
So
this
is
what
I'm
kind
of
determining
like
the
The
Next
Gen
Science
Gateway
this
platform,
where
you
essentially
are
able
to
move
across
the
different
areas
of
your
work
and
flow
in
research
and
modify
it
and
reproduce
it
effectively.
So
with
that
I'd
like
to
thank
you
all
for
attending
our
our
talk,
you're
welcome
to
come
reach
out
to
us
individually.
If
you
want
to
know
more
about
either
of
these
efforts
or
stop
by
the
booth
or
just
to
kind
of
keep
a
conversation
going
here.
If
you
have
any
questions.
C
D
Do
you
already
have
an
infrastructure
like
that
in
place
for
kubernetes
to
handle,
say
you
know
a
similar
Resort
resource
contention
between
researchers?
So
if
one
researcher
wants
all
the
gpus
and
definitely
is
there
another
tool
or
middleware
to
kind
of
handle
that
you
know
batch
queuing
so
that
not
everything
is
allocated
to
specific
to
a
single
or
specific
workload,
yeah.
B
So
that's
a
that's
a
really
great
question
and
the
answer.
The
quick
answer
is
no.
We
don't
yet,
but
we
know
that
that's
an
issue,
obviously
in
the
in
the
batch
Computing.
The
reason
why
people
do
batch
Computing
is
for
really
kind
of
I.
Guess
two
two
reasons
is
one
access
to
Bare
Metal
performance
right,
you
don't
get
any
kind
of
interference
from
abstraction
layers
and
two
is
that
you
can
share
the
resources.
B
Back
schedules
are
very
good
at
disciplining
that
over
time
our
Kate's
environment
does
not
have
anything
to
essentially
stop
a
researcher
from
having
all
the
resources.
What
I
would
like
to
see
and
where
I'm
exploring
with
my
team,
the
capabilities
of
Kate's
and
HPC
to
come
together
is
how
we
might
be
able
to
man
Fest
some
of
the
HPC
capacity
as
a
Kate's
worker
platform.
B
C
E
Thank
you.
My
question
is
regarding
Cuda
versions,
so
we
run
in
the
cloud.
I
have
some
ml
Engineers
that
want
to
use
cuda10
some
that
want
to
use
cuda11
right
now,
I
have
to
spend
out
two
different
node
groups,
one
with
two
with
manual
Amis.
Some
of
them
have
q
to
10.
Some
have
Kudo
11..
Is
that
a
good
way
to
manage
that?
How
would
you
suggest
maybe
doing
something
like
that.
B
Well,
I
mean
when
we,
when
we
had
that
requirement,
we
just
spin
up
different
environments
as
well.
We
we
have
typically
been
able
to
move
forward
with
Cuda,
so
we
tend
to
be
on
Cuda
11
on
our
cluster
right
now
and
so
we're
pretty
consistent
across
Code
11,
but
for
something
where
we
would
have
to
go
to
Cuda
10.
That
would
be
a
case
where
we
would
use
our
openstack
environment
to
use
a
GPU
from
that
environment.
B
So
every
one
of
our
environments
has
gpus
available
to
it,
but
also
with
the
one
of
our
kind
of
our
Visions,
for
starting
to
leverage.
Mass
more
heavily
is
to
be
able
to
kind
of
say
well,
we
need
more
capacity
for
this
kind
of
workflow
in
one
environment
over
the
other,
and
so
we
can
potentially
move
that
compute
capacity
over
for
a
period
of
time.
But
right
now
it's
just
we're
kind
of
in
the
same
space,
You
Are,
no
great
solution.
F
C
B
I
wouldn't
say
that
there's
something
specific
in
the
architecture
to
handle
that
we
have.
We
have
used
nodes
in
our
environment
that
are
generously
provisioned,
so
our
a100s
have
their
40
gig
a
100,
so
they
have
a
lot
of
RAM
on
there
and
then
we
have
a
lot
of
memory
available
on
the
nodes
that
are
the
worker
nodes.
So
I
guess
we're
kind
of
maybe
cheating
a
little
bit
in
the
sense
that
we
we
know.
We
have
those
that
capacity
in
our
environment
and
we
don't
have
it.