►
From YouTube: Benchmarking Machine Learning Workloads on Kubeflow - Xinyuan Huang, Cisco Systems, Inc. & Ce Gao
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Benchmarking Machine Learning Workloads on Kubeflow - Xinyuan Huang, Cisco Systems, Inc. & Ce Gao, Caicloud
Benchmarking is an essential part in machine learning research and productization, that provides useful performance information from the perspectives of both models and systems. While Kubernetes and Kubeflow give us a great platform for ML workloads to run on, they do not automatically provide a straightforward way to perform benchmark tasks, especially for complex ML workloads based on distributed jobs. In this talk we present Kubebench, an open sourced benchmarking tool based on Kubeflow, that helps us better understand the performance signature of our ML workloads on Kubernetes through automated and consistent benchmarks. We also show how we can leverage other benchmarking efforts from academia and industry like MLPerf and Dawnbench.
To learn more click here: https://sched.co/FuJw
A
However,
right
now,
there's
no
straightforward
way
to
run
benchmarks,
jobs
on
communities,
and
it
is
not
very
easy
to
run
benchmark
jobs
and
communities
today
and
for,
for
example,
there
are
lots
of
manual
steps
involved
in
this
process
such
like.
You
have
to
prepare
your
data
manually
and
you
have
to
create
your
communities
resources.
You
have
to
copy
all
the
results
and
outputs
from
your
containers,
and
you
also
need
to
take
care
of
rerunning
your
benchmark
experiments.
So
all
those
manual
steps
are
preventing
us
to
from
running
good
benchmarks
and
hereby
good
benchmarks.
A
A
For
example,
we
want
to
be
able
to
run
benchmarks
in
multi-cloud.
We
don't
want
to
be
locked
into
a
single
cloud
provider
also
we
want
to.
We
have
data
scientists
that
using
different
machine
learning
frameworks
such
like
tensorflow
titles
and
even
MX
net,
etc.
So
a
good
benchmark
suit
should
be
able
to
support
a
different
benchmark,
different
machine
learning
frameworks
and
also
when
we
run
large-scale
training
jobs.
We
would
like
to
understand
our
system
performance
on
the
distributed
workloads.
A
So
the
second
dimension
is
that
we
want
to
use
cook
bench
to
help
user,
to
manage
benchmark
jobs
easier,
for
example,
how
we
can
keep
our
workloads
consistent
and
how
we
can
get
reproducible
results
and
also
an
important
part
is
that
how
can
we
make
the
benchmark
as
a
part
of
the
whole
machine
learning
lifecycle?
How
can
we
make
it
are
fully
integrated
into
the
life
cycle
instead
of
separate
tool
that
you
use
a
standalone
tool
that
you
use
separately.
A
So,
given
all
those
requirements
here
is
the
tech
stack
that
we
choose,
so
let
me
start
from
the
bottom.
So
from
the
infrastructure
side
we
use.
We
look
at
the
cloud-based
infrastructure
that
are
both
public
and
on-premise
private
clouds,
and
on
top
of
that
we
have
kubernetes
for
container
orchestration
I
guess
we
don't
need
to
introduce
kubernetes
here
and
on
top
of
communities.
There's
a
coop
flow
for
deploying
and
managing
the
lifecycle
of
machine
learning
jobs
and
Cubans
as
a
thin
layer
on
top
of
qu
flow.
That
provides
some
benchmark.
A
This
is
a
deeper
look
into
the
coop
bench
architecture,
so
the
yellow
part
is
the
functionality
that
is
provided
by
a
coop
bench
and
the
blue
part
is
what
is
user-defined.
So,
let's
look
at
the
blue
part
first,
so
what?
As
a
user?
What
you
need
to
provide
is
basically
the
main
codes
that
runs
in
your
in
your
machine
learning
job.
That
is
to
be
benchmarked
and
also
you
need
to
provide
a
little
bit
more
codes
that
process
your
outputs
to
get
benchmark
numbers.
A
So
that
is
the
main
thing
provided
by
user
and
what's
provided
by
Cook
bench
is
first.
One
of
the
main
thing
is
that
we
provide
a
workflow
on
the
left
side.
We
implement
these
workflows
using
our
go
and
there
are
three
main
steps.
So
one
is
the
configuration
step
that
it
takes
in
users,
configurations,
job
configurations
and
turn
it
into
a
description
of
could
flow
job.
A
So
a
good
flow
job
is
description
is
like
a
high
level
description
of
machine
learning
tasks
in
Yama
style
and
in
the
second
step
here
we
just
deployed
this
approach
of
using
highest
income
flows
API,
and
we
wait
for
the
job
to
complete.
And
finally,
once
the
job
is
finished,
we
have
a
reporter
that
collects
all
the
results
from
all
your
benchmark
jobs
and
it
exports
the
results
to
some
external
data
storage,
so
that
you
can
persist
your
benchmark
results,
and
here
we
also
provide
a
storage
layer
that
provides
some
shared
storage.
A
For
example,
if
you
have
some
benchmark
configs
and
you
have
some
data-
and
we
also
need
to
have
a
storage
to
store
all
those
benchmark
records,
so
the
storage
layer
will
provide
a
shared
storage
that
can
be
easily
used
to
share
information
between
the
user-defined
workload
and
components
workflow
and
also
on
the
right
side.
We
have
this
monitoring
part
we
use
Prometheus
to
collect
all
the
metrics
from
the
benchmark
job
and
also
we
use
graphing
to
visualize
the
metrics.
A
And
here
is
how
our
coupe
bench
looks
like
from
users
perspective.
So
there
are
two
kinds
of
users:
I
think
one
is
the
job
developer.
So,
as
you
have
seen
in
the
previous
slides
for
job
developer,
what
you
really
need
to
provide
is
the
code
that
runs
in
the
benchmark
job
and
also
some
some
extra
codes
to
process
your
results
and
a
crew
bench
will
provide,
is
shared
storage.
A
So
if
you
run
a
distributed
job
order
order,
the
will
automatically
mount
the
volumes
to
your
job
pod
so
that
you
can,
you
don't
need
to
worry
about,
say,
copying,
results
and
outputs
from
your
pods
and
from
experiment
run.
At
this
point
of
view,
what
you
need
to
provide
is
basically
a
job
configuration
and
the
job
configuration
is
consists
of
two
parts.
One
is
the
job
template
and
one
is
the
parameter
files.
A
So
a
template
is
in
case
on
it
formats
right
now
and
it
basically
defines
its
kind
of
Job
Description
with
some
configurable
variables
and
the
parameter
file
basically
defined
actual
value
of
those
parameters.
And
you
can
just
give
those
to
cook
bench
and
we
will
run
this
benchmark
jobs
and
you
will
be
able
to
see
the
results
from
dashboard
or
visualizations.
A
Here's
where
we
are
today,
so
we
have
just
released
our
first
release:
V
0.3,
along
with
coop
flow
and
for
the
first
release
we
provide
something
for
functionalities
such
like.
We
can
support
local
and
distributive
training
workloads,
and
we
can
support
two
frameworks
right
now,
that
is
tensorflow
job
and
pilots
job,
and
we
are
planning
to
support
more
in
the
future
and
for
results.
A
Aggregation
we
are
able
to
agree,
creates
results
in
file
system
and
we
will
soon
support
cloud
database
based
result,
aggregation
and
also
we
provide
end-to-end
a
simple
example
for
your
new
users
to
get
quick
started
and
we
have
a
tensor
flow.
Cnn
example
that
comes
with
it
and
in
upcoming
releases
we
are
going
to
add
a
bunch
of
new
features
here,
and
one
is
the
dashboard
and
the
resultant
matrix
visualizations
and
one
is
that
we
are
planning
to
formalize
our
API
using
community
CRT,
and
also
we
are
planning
to
support
more
benchmarking
scenarios.
A
A
So
so
this
is
the
dashboard
of
cook
bench
and
there
are
two
ways
to
launch
the
benchmark
job
right
now.
You
can
either
give
a
Yama
fowl
like
this,
so
the
Yama
file
will
take
full
advantage
of
all
the
functionalities
of
cook
bench
or
you
can
use
this
kind
of
result.
That
will
just
give
you
a
few
options
and
you
can
just
input
some
like
something
basic
information
like
where
to
find
your
job
image
where
to
find
your
configurations-
and
you
can
just
launch
benchmark
job
here
so
here
we
launched
for
benchmark
jobs
here.
A
And
in
this
panel
you
can
see
all
the
jobs
being
launched
can
do.
Some
basic
filtering
is
to
manage
the
job,
and
here
we
can
click
on
this
job
and
in
the
workflow
button
it
will
bring
you
to
the
Argo
dashboard
that
will
show
the
workflow
of
this
benchmark,
and
here
you
can
keep
track
of
the
progress
of
all
your
benchmark,
jobs.
A
A
A
And
finally,
we
can
check
out
the
shared
storage
where
all
the
benchmark
records
are
kept.
So
here
you
can
see
the
list
of
benchmarks
examples
and
you
click
on
a
particular
example,
and
that
will
show
all
the
conflicts
of
this
benchmark
and
the
output
of
other
workers,
and
it
will
also
show
you
the
results
of
this
benchmark,
so
that
will
be
all
persisted
in
this
shared
storage.
A
C
B
Learning
training
is
damage
based
on
genetics,
then
overhead.
Well,
this
overhead
will
influence
the
training,
so
we
use
cringe
and
open
flow
to
do
the
benchmark.
So
let's
look
at
the
local
training
benchmark
on
single
machine
motor
training
job.
Let's
look
at
the
overhead
on
the
training
in
Spain
how
the
overhead
influences
training
speed.
So
we
use
things
of
those
benchmark.
B
C
B
C
B
B
B
C
B
B
C
B
C
B
Memory,
which
means
that
in
the
distributed
tensorflow
training
we
can
sense
the
impact
from
overhead.
However,
the
cube
flow
benefit
is
that
cube
flow
use,
machine
learning
and
kubernetes
aid.
The
cube
flow,
lower,
the
end-to-end
machine
learning
deployed
difficulties,
so
these
kind
of
overhead
is
acceptable
can
be
tolerated
and
when,
when
we
are
doing
distributed,
training
benchmark,
we
use
manual
dispatch
because
on
the
community
there
is
no
mature
solutions.
B
There
are
other
scheduler
based
on
kubernetes,
open
source,
cube
patch
or
arbitrator,
or
pave
open
source
per
set
on
the
set
of
them.
So
those
platforms
trying
to
solve
distributed
training
can
scheduling
issue
would
so
if
they
can
solve
that
if
they
can
solve
distributed,
trainings
impact
on
internet
and
new
resources,
then
I
believe
that
in
the
future
distributed
training
overhead
could
maintain
within
five
to
ten
percent,
which
is
tolerable.
B
C
B
Is
our
test
baptists
parameters
and
based
on
this
test
patch?
Our
experiment
tells
us
that
could
flow
open
source
project.
We
can
facilitate
tensorflow
and
low.
Although
there
are
overhead,
it
is
tolerable
and
it
can
facilitate
training
in
terms
of
low
trainings.
We
need
those
learning
stack
open
source
programs
to
emerge
in
the
future,
so
that
is
the
some
of
the
contributors
for
our
presentations,
some
advisors
for
presentation.
So
now
the
floor
is
open
for
your
question.