►
Description
Don’t miss out! Join us at our upcoming events: EnvoyCon Virtual on October 15 and KubeCon + CloudNativeCon North America 2020 Virtual from November 17-20. Learn more at https://kubecon.io. The conferences feature presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
MLPerf Meets Kubernetes - Xinyuan Huang & Elvira Dzhuraeva, Cisco
Kubeflow is maturing as a cloud native Machine Learning (ML) platform that simplifies the journey of development, deployment, and management of ML on Kubernetes. As Kubeflow gets increasingly adopted, practitioners are looking beyond functions and starting to explore its performance and cost efficiency in the real world. MLPerf is a state-of-the-art benchmark suite that aims to set an industry standard for end-to-end performance evaluation of ML systems with real-world workloads covering both training and inference phases of ML lifecycle. This talk will provide a brief overview about MLPerf, followed by detailed discussions about how MLPerf can be adapted to evaluate ML performance on Kubeflow and Kubernetes, as well as how the performance results can be leveraged to guide the future design and optimization of cloud native ML platforms based on Kubeflow and Kubernetes.
https://sched.co/Zepu
A
A
A
As
I
said,
given
the
amount
of
tools
we
have,
how
do
we
know
what
to
choose,
and
especially
when
it
comes
to
performance,
how
do
we
know
which
direction
to
choose
and
what
think
you
need
to
look
in
first
in
order
to
improve
it,
do
you
need
to
improve
your
network?
Do
you
need
to
improve
your
hardware
accelerators
or
maybe
something
wrong
with
all
the
model
code
so
which
direction
to
choose
in
order
to
achieve
the
biggest
impact.
A
A
So,
as
I
said,
we
have
so
many
technologies
and
because
of
the
complicated
nature
of
machine
learning,
workload
and
infrastructure
behind
it,
as
well
as
the
models
are
so
different
from
one
another.
So
how
can
we
compare
the
effectiveness
and
performance
if
these
tools
are
not
even
close
to
each
other
and
we
use
benchmark
in
ei
to
achieve
this?
A
So
last
I'm
going
to
introduce
an
oper.
It
is
a
fair
and
useful
benchmarks
for
measuring
training
and
inference
performance
of
machine
learning,
hardware,
software
and
services.
So
mlport
was
created
by
a
large
community.
We
have
more
than
60
organizations
and
eight
plus
universities
and
more
than
1
000
members
in
this
community.
Contributing
to
this.
C
C
C
C
Individual
components
or
aspects
of
a
machinery
system
people
quickly
realize
that
micro
benchmarks
cannot
reflect
the
complexity
of
and
challenges
in
a
real
world
system.
So,
in
order
to
address
this
problem,
people
gradually
move
to
use
workloads
that
are
at
real
world
scale
and
starts
to
evaluate
a
new
system
from
end
to
end.
C
C
C
C
However,
there
is
a
side
effect,
which
is
often
overlooked,
that
in
some
cases
or
in
most
cases,
actually
increasing
batch
sizes
will
have
a
mathematical
impact
to
your
optimization
process.
That
will
cause
your
model
to
convert
slower
and
you
might
end
up
training
a
model
for
more
epochs
to
reach
the
same
quality,
which
means
that
you
might
have
a
higher
total
cost
for
your
training
job
when
you
train
at
a
higher
throughput.
C
C
And
vice
versa:
in
the
inference
world,
there
is
a
wide
range
of
different
use
cases
and
different
systems
that
range
from
clouds
and
data
centers
to
edge
and
mobile
devices.
The
requirements
for
all
those
systems
and
use
cases
are
largely
different
from
each
other,
and
hence
we
need
to
evaluate
them
on
the
different
scenarios
in
mlps.
There
are
four
types
of
scenarios
today:
the
single
stream
scenario
targets
those
use
cases
where
interest
queries
come
in
at
a
constant
rate
and
a
single
sample
at
a
time.
C
The
multi-stream
scenario
is
similar
as
the
single
stream
in
a
way
that
it
also
involves
inference,
queries
that
comes
in
a
streaming
manner,
at
constant
rate.
However,
in
this
case,
it
involves
multiple
streams
of
data.
At
the
same
time,
and
an
example
case,
is
that
say
in
an
autonomous
driving
use
case,
where
many
sensors
data
are
injected
into
an
inference
engine
at
the
same
time
and
they
need
to
be
processed
altogether
in
parallel.
C
The
measurement
of
multi-stream
scenario
is
to
measure
the
maximum
number
of
streams
possible
under
the
condition
that
the
latencies
can
be
guaranteed
under
a
certain
threshold.
Now
there
is
the
third
scenario
that
is
the
server
scenario,
which
is
most
mostly
suitable
for
machining
applications
as
an
online
service.
C
For
example,
recommendation
services
and
translation
applications
are
some
of
the
popular
use
cases.
In
this
scenario,
we
issue
queries
at
an
arrival
rate
that
follows
a
poisson
distribution
which
mimics
the
influence
queries
in
real
world.
Then
we
measure
the
performance
by
setting
a
latency
threshold.
First
then
measure
the
highest
qps.
The
system
can
reach
on
the
condition
that
the
predetermined,
latency
threshold
can
be
met.
C
C
C
As
you
know,
the
number
of
benchmarks
are
designed
for
evaluating
the
whole
software
and
hardware
stack
of
a
machine
learning
system,
and
our
kubernetes
can
certainly
take
a
part
of
such
a
system.
So
we
want
to
see
how
we
can
run
a
number
of
kubernetes
and,
more
importantly,
how
we
can
leverage
ml
curve
to
optimize
the
machine
learning
performance
on
a
kubernetes
based
machine
learning
system.
C
So,
first
of
all,
let's
look
at
training.
The
training
workload
itself
has
no
difference
between
running
on
a
bare
metal
environment
and
running
on
a
kubernetes
cluster.
The
main
difference
comes
in
orchestration
part
where
you
turn
the
model
training
program
into
a
kubernetes
jaw
and
also
on
the
fit
of
of
shared
shared
hardware.
Resources
such
as
accelerators
network
devices
and
storage
devices
that
are
now
managed
and
allocated
by
kubernetes.
C
C
Kubernetes
so,
first
of
all
to
run
training
at
real
world
scale.
You
might
need
help
from
some
distributed
com
distributed
computing,
libraries
or
frameworks
such
as
openmpi
or
horrorboard,
or
disability,
tensorflow,
etc.
You
might
want
to
take
some
extra
care
when
deploying
programs
using
those
frameworks
on
the
kubernetes
cluster
and
luckily
the
support
for
those
distributed.
C
Training
do
exist
in
kubernetes
community,
for
example,
you
can
check
out
powerful
tools
such
like
control
or
other
similar
works
that
can
help
you
simplify
the
deployment
of
distributed,
work,
disability,
training,
jobs
on
kubernetes
and
talking
about
distribution
distributed
training.
Another
important
problem
we
want
to
address
is
the
resource
scheduling.
C
Now
that
all
the
compute
and
network
and
storage
devices
are
managed
by
kubernetes
to
maximize
performance.
You
want
to
make
sure
that
you
have
the
right
affinity,
settings
that
co-locate
your
kubernetes
resources,
so
here
we
also
want
to
talk
a
bit
about
topology
management
that
can
be
considered
as
another
kind
of
scheduling
in
a
more
micro
scale.
C
The
temperature
management
we
are
talking
here
is
basically
managing
the
alignment
of
cpus
and
peripheral
devices
such
as
gpus
and
mix
within
the
newmar
system,
so
that
you
will
get
the
best
performance
when
you
correctly
align
the
cpus
and
peripherals
to
the
same
local
memory
in
a
given
water,
node,
and
also
we
do
see
that
there
is
a
better
there's,
a
better
feature
for
topology
management
in
latest
kubernetes
release.
So
this
is
going
to
be
very
helpful
for
us
to
leverage
the
pneuma
system.
C
C
Another
inference
provides
a
load
generator
that
works
like
a
harness
for
benchmark
experiments
during
a
benchmark
experiment.
The
load
generator
will
first
query
the
system
of
the
test,
or
we
call
it
sut
to
load
a
portion
of
test
data
that
fits
into
its
memory,
the
sdut
loads,
the
query
data
with
help
of
a
data
module
and
then
returns
to
the
load,
generator
with
ready
signal.
C
Then
the
load
gen
will
start
issuing
influence
queries
at
a
given
distribution,
where
the
sut
will
be
responsible
to
pass
the
queries
to
the
backend
inference
runner,
where
the
increased
results
will
begin,
the
location
will
be
able
to
load
the
output
afterwards
and
then.
Finally,
a
post
processing
script
will
be
used
to
process
the
outputs
and
get
the
summarizer
report.
C
So,
as
you
can
see,
all
this
process
is
happening
in
a
local
device
where
the
influence,
queries
and
outputs
are
all
simply
transferred
from
memory
to
memory.
This
is
an
ideal
way
to
test
your
performance
when
you
want
to
eliminate
all
other
factors
that
are
now
relevant
to
the
influence
task
itself.
C
However,
now
in
a
kubernetes
environment,
it
is
very
common
to
run
a
machine
application
at
the
service.
For
example,
you
might
be
leveraging
some
inference.
Services
such
as
kf
survey
or
zelda,
or
you
might
be
simply
running
a
tf
server
to
serve
your
machine
learning
models
on
kubernetes,
and
you
might
also
want
to
be
interested
to
know
that
what
is
the
end-to-end
cost
of
your
inference
service
from
a
client
point
of
view,
and
in
order
to
do
that,
we
will
need
to
make
some
modifications
to
the
original
mechanism
as
defining
an
output.
C
C
C
So
finally,
we
want
to
show
some
example:
performance
results
from
a
very
simple,
very,
more
experiment.
We
did
before
we
go
into
details.
I
just
want
to
clarify
that
the
experiment,
settings
and
the
target
requirements
are
all
fitted
to
our
specific
testing
environment
and
are
not
directly
aligned
with
mlk's
official
rules.