►
Description
Don’t miss out! Join us at our upcoming events: EnvoyCon Virtual on October 15 and KubeCon + CloudNativeCon North America 2020 Virtual from November 17-20. Learn more at https://kubecon.io. The conferences feature presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Monitoring GPUs at Scale for AI/ML and HPC Clusters - Bharti L Agrawal, NVIDIA
At Nvidia we have several large GPU K8s clusters for running deep learning training (AI/ML) workloads. On these clusters we need monitoring to support a range of user personas . First we have the end users (AI/ML researchers) who want to get an insight into how well their workloads used the GPUs and the system. Then we have the operations team who would like to monitor the general health of the cluster and be alerted in real time to any issues. Finally we have the stakeholders who would like to see the GPU utilization and saturation over time for capacity planning. These requirements cannot be satisfied by a standard “out of the box” setup. In this presentation we will show how we used a combination of open source tools to address our requirements. We will discuss various deployment, maintenance, security and scale challenges we hit and how we resolved them for monitoring GPU data.
https://sched.co/Zeoh
A
A
A
A
A
The
scheduler
sends
metrics
back
to
the
cpu
control
plane.
The
users
are
also
able
to
leverage
the
ngc
registry,
which
contains
containers,
data
sets
and
pre-trained
models
for
them
to
use.
This
is
great
users,
get
the
latest
technology
and
are
able
to
deploy
their
jobs
and
be
guaranteed
that
they
will
have
resources
to
run
on.
However,
we
need
to
ensure
we
can
meet
this
guarantee.
A
A
A
We
have
the
stakeholders.
These
are
the
product
managers,
managers,
architects
and
other
org
leaders
who
would
like
to
get
an
overall
occupancy
and
yield
of
the
system
for
capacity
planning
purposes.
Then
we
have
the
nsp
developers.
These
are
developers
like
me
who
worked
on
the
scheduler
and
the
observability
stack.
A
How
do
we,
how
do
these
use
cases
map
into
requirements
for
the
end
user,
who
wants
to
see
how
efficiently
their
jobs
are
running
in
terms
of
resource
usage
and
their
custom
metrics?
We
need
to
collect
the
resources,
resource
and
application
metrics
and
send
them
to
the
nsv
cloud
control
plan
for
the
nsv
admin
user.
Who
wants
to
monitor
the
overall
health
of
the
system,
get
real-time
notifications
on
incidents
and
be
able
to
root
cause
them
effectively.
We
need
to
centralize
the
metrics
and
logs
in
real
time.
A
A
A
A
A
A
A
A
A
This
is
native
to
kubernetes
and
widely
used
with
a
lot
of
community
support
behind
it.
It
is
highly
performant
in
collecting
metrics
and
writing
to
remote
storage
to
support
in-cluster
data
persistence.
We
use
influx
tv,
thanos
and
swift
stack
object,
store
are
used
for
data
durability
and
storage.
A
A
A
These
metrics
are
sent
to
influx
cv
for
class
in
cluster
persistence
from
here
they
are
forwarded
to
the
metrics
transporter
via
the
n
clutch
cv
subscription
setup.
This
transporter
then
forwards
them
to
the
external
cloud
systems,
having
matrix
transporter
on
a
different
node
from
the
source.
Node
with
a
separate
vpc
allowed
us
to
introduce
a
level
of
security
to
ensure
that
the
data
was
not
would
not
be
hacked.
A
A
An
application
can
also
add
a
fluency
login
sidecar
and
send
their
logs
to
the
same
centralized
service
as
well.
The
users
will.
This
will
see
this
data
in
the
nvc
ui
and
the
stakeholders
will
use
the
elk
stack
on
the
data
lake
to
visualize
the
job.
Metrics
stakeholders
also
get
a
weekly
yield
and
showback
report
that
uses
the
data
from
the
data
lake.
A
A
A
Let's
take
a
quick
look
at
the
details
of
how
we
collect
gpu
metrics
in
this
diagram.
You
will
see
on
the
gpu
node
on
the
bottom.
We
have
several
jobs,
running
node
exporter
pod
has
a
data
center,
gpu
manager
or
dcgm
exporter
as
a
container
that
collects
gpu
metrics
from
the
dcgm
library
running
in
the
pod
as
well.
A
A
A
To
break
the
flow
a
bit,
I
want
to
show
one
example
of
view
that
comes
out
of
the
first
architecture
we
covered.
This
is
a
view
of
the
job
telemetry,
the
end
user
will
see
at
the
top.
You
can
see
the
job
runtime
and
the
time
the
gpus
are
active
and
the
average
gpu
utilization
across
the
8
gpus
being
used
by
this
job.
A
A
A
A
A
A
A
Let's
now
look
at
how
we
optimize
the
components
for
our
scale
for
prometheus
and
other
components,
we
had
to
understand
the
resource,
requests
and
limits
that
would
work
well
for
our
loads,
with
room
to
grow
to
size
each
component.
Well,
we
had
to
understand
how
it
used
the
resources
prometheus,
for
example,
keeps
two
hours
of
data
in
memory.
A
A
A
We
do
regular
evaluations
of
our
system
to
ensure
that
the
sizing
is
working.
Well,
as
we
add
nodes
to
the
cluster,
we
review
these
settings
to
make
the
system
highly
available.
We
had
to
learn
what
load
could
be
supported
by
prometheus,
with
acceptable
restart
times
versus
where
we
needed
to
add
another
replica
for
scaling
from
ethios.
We
rely
on
sizing
of
the
instances
from
our
load
test.
A
A
We
used
influx
tv
with
for
remote,
read
and
write
to
cover
this.
This
worked
well
up
to
a
certain
load,
after
which
we
started
seeing
data
gaps
to
resolve
this.
We
have
to
look
into
the
code
of
prometheus
and
the
downstream
components
we
found
that
prometheus
would
drop
metrics
when
its
buffer
was
full
or
if
the
downstream
components
were
not
expecting
metrics.
A
So
there
were
several
points
of
failure
with
the
2.8
version
of
prometheus,
where
prometheus
introduced
wall
for
remote
drive,
this
occurrence
was
reduced,
but,
prior
to
that
we
had
to
go.
We
had
to
optimize
the
downstream
components
as
well
as
the
buffer
of
the
prometheus,
so
metrics
did
not
get
dropped.
A
A
A
The
influx
db
tuning
for
sizing
and
scalability
was
similar
to
prometheus
influx
tv
keeps
a
lot
of
data
in
memory
and
restarts
times
along,
as
the
container
needs
reads
the
wall
files.
We
also
found
that
influx
db
did
not
scale
well
for
prometheus
data,
which
has
a
lot
of
labels
which
need
to
be
indexed.
A
A
A
A
As
I
mentioned,
we
had
added
input
tv
in
our
architecture
for
data
persistence
in
the
cluster,
since
this
is
a
kubernetes,
cluster
parts
can
restart
for
various
reasons
and
land.
On
new
nodes
to
process
data
for
influx
tv,
we
are
using
host
path,
persistent
volumes
with
pod
restarts.
We
could
easily
have
parts
land
on
different
nodes
and
blues
data.
A
Our
solution
of
in
cluster
application
helped
here
here
we
had
to
ensure
that
the
data
we
are
sending
to
external
system
had
no
gaps
as
well.
This
turned
out
to
be
a
very
hard
issue
to
debug
and
solve
for
and
needed
detailed
investigation
into
the
component
codes
to
understand
how
best
to
solve
it.
The
solution
involved
updates
to
the
code
of
the
services.
A
Sizing
for
tunnels
involved
understanding
each
of
these
and
how
best
to
optimize
them
for
our
loads,
for
example,
thanos
store
for
thunderstorm
we
needed
to
set
max
cache
size
for
thanos
receiver.
We
discovered
that
setting
the
tstb
min
block
and
match
block
to
15
minute
would
reduce
the
wall
size
which
allowed
us
to
reduce
the
restart
times
and
have
thanos
receiver,
not
use
increasing
amounts
of
memory.
A
A
A
A
Luckily,
we
were
able
to
upgrade
prometheus
and
thanos
versions
and
tune
the
configurations
to
solve
for
our
loads.
We
had
to
do
it.
We
had
to
do
rigorous
load
tests
to
set
up
the
max
shards
max
samples
and
capacity
for
prometheus
report
drive
for
our
loads
down.
Sampling
allowed
us
to
support
longer
data
retention
and
querying.
A
A
A
A
Just
adding
replicas
did
not
help.
We
had
to
look
at
sizing
as
well.
We
learned
there
was
a
heap
size
recommended
for
elasticsearch
that
made
it
perform
optimally
for
data
persistence.
We
added
the
persistent
volume
for
greylock
journal.
Data
written
to
elasticsearch
was
already
well
managed,
relog
and
elasticsearch
handle
data
as
derivative.
A
It
shows
the
cluster
capacity
with
total
nodes
in
the
cluster
and
total
number
of
code
nodes
cordoned
on
the
top
and
below
that
it
shows
nodes
coordinate
by
node
type
on
the
right.
It
shows
the
availability
of
the
different
node
types
with
this.
The
admin
users
can
get
a
good
view
into
the
cluster
capacity.
A
A
A
B
B
B
A
A
How
is
the
gpu
usage
calculated
for
different
applications
running
in
the
kts
so
that
their
sras
are
satisfied?
So
basically,
we
are
monitoring
the
actual
utilization
of
the
gpu,
so
we
have
several
profiling
metrics
that
we
grab.
We
grab
the
percentage
usage
percentage
active
for
the
gpu,
the
percentage
active
of
the
tensor
core.
So
while
the
job
is
running,
we
are
monitoring
it
almost
every
second,
but
we're
collecting
the
metric
at
every
30
second
rate,
so
engineers
can
see
that
what
rate
they're
using
the
gpu
add
and
optimize
it.
A
B
A
A
A
I
am
not
really
clear
on
what
that
anomaly
would
look
like
the
the
issues
that
might
happen
at
the
infrastructure
level
would
be
like
they
may
not
be
able
to
mount
their
cf
volumes
or
their
data
sources,
and
then
that
is
when
we
would
step
in
and
try
to
address
that.
But
if,
in
terms
of
because
they're
in
a
in
a
pod
in
a
container-
I
don't
you
know,
usually
they
have
to
tell
us
exactly
how
many
gpus
they
want
so
they're
restricted
to
the
number
of
gpus.
So
they
can't
really.
A
A
So
at
that
time
the
industry
most
of
the
community
was
using
influx
db
for
the
long-term
retention
from
prometheus.
So
that's
why
we
started
with
infrastb
and
we
did
see
scaling
problems
with
it.
So
our
second
iteration,
the
second
architecture
that
I
shared
with
you
has
thanos
in
it.
It
doesn't
have
influx
tv,
so
we
quickly
upgraded
to
thanos
as
soon
as
as
soon
as
it
was
ready.
A
Is
gpu
sharing
talked
into
account
taken
into
account
in
your
jazz
scores?
Gpu
sharing
is
not
yet
taken
into
account
in
the
dashboards
we.
We
are
just
talking
about
gpu
affinity
in
our
scheduler.
We
haven't
really
talked
about
gpu
sharing
as
such
between.
I
assume
you
mean
between
multiple
jobs
by
a
team
of
users.