►
From YouTube: 2021-10-26 - David Kanter - Challenges and Directions in ML System Performance: The MLPerf Story
Description
NERSC Data Seminars Series: https://github.com/NERSC/data-seminars
Speaker:
David Kanter, MLCommons
Title:
Challenges and Directions in ML System Performance: The MLPerf Story
Abstract:
As the industry drives towards more capable ML, workloads are rapidly evolving and the need for performance is nearly unlimited. We explore the challenges and design choices behind MLPerf, the industry standard benchmark for ML system performance.
Bio:
David Kanter is a Founder and the Executive Director of MLCommons™ where he helps lead the MLPerf™ benchmarks and other initiatives. He has 16+ years of experience in semiconductors, computing, and machine learning. He founded a microprocessor and compiler startup, was an early employee at Aster Data Systems, and has consulted for industry leaders such as Intel, Nvidia, KLA, Applied Materials, Qualcomm, Microsoft and many others. David holds a Bachelor of Science degree with honors in Mathematics with a specialization in Computer Science, and a Bachelor of Arts with honors in Economics from the University of Chicago.
Host of Seminar:
Hai Ah Nam, Advanced Technologies Group
Steve Farrell, Data Analytics Group
National Energy Research Scientific Computing Center (NERSC)
Lawrence Berkeley National Laboratory
A
So
I
am
very
excited
that
david
cantor
can
join
us
today
to
give
the
data
seminar.
I
I
heard
about
his
talk
at
mod
sim
and
was
told
we
absolutely
must
get
him
here.
So
I
would
like
to
introduce
david.
He
is
the
founder
and
the
executive
director
for
ml
commons,
where
he
helps
to
lead
the
ml,
perf,
benchmark
and
other
initiatives.
He
has
over
16
years
of
experience
in
semiconductors
computing
and
machine
learning
and
has
a
lot
of
opinions.
A
You
know,
you
know
reasons
as
well:
he
founded
a
microprocessor
and
compiler
startup
was
an
early
employee
of
aster
data
systems
and
has
been
consulting
industry
leaders
such
as
intel
nvidia,
kla
applied
materials,
qualcomm,
microsoft,
many
others,
so
david
comes
to
us
with,
with
a
bachelor's
degree,
with
honors
and
mathematics
and
specialization
in
cs
a
bachelor
of
arts
in
econ
from
the
university
of
chicago,
and
we're
really
excited
to
hear
about
the
challenges
and
directions
in
ml
system
performance
and
the
ml
perf
story.
So
with
that
david,
it's
all
yours.
B
B
So,
yes,
I
do
have
a
lot
of
opinions
on
I'll
have
to
be
a
little
bit
circumspect
with
those
in
a
recorded
environment,
but
I,
I
hope,
it'll
be
a
fun
talk
nonetheless.
So
you
know
this
talk
is
really
kind
of
an
overview
of
some
of
the
unique
challenges
in
ml
benchmarking
and
how
the
ml
perf
community
has
approached
it.
B
Since
we
are
kind
of
focused
on
sort
of
more
scientific
computing
and
hpc,
I
I'm
going
to
focus
a
lot
on
training
and
we
might
talk
about
inference
if
I
have
time
at
the
end.
The
last
time
I
gave
this
talk,
I
did
not,
so
I
don't
want
to
get
your
hopes
up
too
high,
but
you
know
thank
you
for
the
fantastic
introduction,
so
I
wanna
first
start
by
talking
about
ml
commons,
which
is
the
organization
that
runs
mlperf.
B
We
are
an
open
engineering
organization.
That
kind
of
focuses
on
the
intersection
of
engineering
and
ai.
So
if
you
think
about
a
lot
of
ai
organizations,
a
lot
of
them
do
policy
work
and
don't
really
build
things
per
se
and
then
a
lot
of
collective
engineering
organizations
like
ieee
or
3gpp,
just
don't
really
focus
on
ai
or
ml,
and
so
it's
sort
of
at
the
overlap
of
the
venn
diagram.
As
you
see
here,
is
sort
of
what
we
think
of
as
the
ml
common
sweet
spot.
So
that's
kind
of
shapes.
B
B
So
our
mission
is
building
better
ml
for
everyone,
and
we
really
look
at
that
accomplishing
that
through
sort
of
three
key
pillars,
one
is
the
benchmarks:
that's
ml
perf.
Today
there
are
other
things
that
we
have.
You
know
maybe
on
the
roadmap
or
in
gestation.
At
the
moment,
oh
man,
I'm
getting
pretty
washed
out.
Oh
well
and
then
building
large
public
data
sets.
So
you
know
if
you
look
at
ml
to
a
large
extent.
You
know
what
has
kicked
off.
B
This
latest
revolution
was
the
the
the
recognition
that
neural
net
deep
neural
networks
could
beat
humans
at
image,
recognition
on
imagenet,
and
so
you
know
you
can
trace
it
back
to
imagenet
and
say
it's
all.
You
know,
thanks
to
that,
and
so
you
know
when
I
look
at
what
do
we
want
to
do
to
help
the
industry
grow?
Part
of
what
I
say
is
we
want
to
do
imagenet
for
other
things,
whether
it's
speech
or
you
know,
mris
or
or
or
whatever,
but
imagenet
for
blank.
B
So
you
know
things
like
that:
help
make
ml
more
usable
to
the
entire
community,
and
then
we
do
have
a
research
group
where
we're
always
working
on
new
ideas-
and
you
know
some
of
them
eventually
bubble
up
to
be
full-on
working
groups.
So
you
know:
ml
perf
hpc,
which
you
know
steve,
has
played
a
really
big
role
in
is
one
of
those
that
sort
of
started
out
conceptually
as
a
research
project
and
you
know,
is
now
really
standing
on
its
own
two
feet.
So
that's
a
little
bit
of
background
about
ml
perf.
B
You
know
our
benchmarks
now
dialing
into
sort
of
the
the
focus
for
this
talk
really
do
span
from
microwatts
to
megawatts.
So
you
know
that
gives
us
many
many
many
orders
of
magnitude
you
know
at
the
high
end,
we
have
training
hpc,
where
you
know,
we've
seen
benchmarks
run
on
something
like
a
tenth
of
fugaku,
the
world's
largest
supercomputer,
all
the
way
down
to
microwatt
level.
You
know
iot
devices
and
along
the
way,
we've
been
adding
benchmarks.
B
We've
been
improving
how
we
handle
things,
especially
on
training,
for
convergence
and
hyperparameter
definitions,
to
make
the
benchmarking
easier,
more
reproducible
and
and
so
forth,
we've
added
power
measurement
to
inference
and
we
actually
even
have
a
mobile
app
that
runs
on
ios
and
android,
which
is
kind
of
cool.
But
you
know
maybe
not
more
super
relevant
for
this
crowd
here.
B
But
before
I
start
out,
I
want
to
like
acknowledge
that
when
it
comes
to
mlp
training,
I'm
merely
one
of
many,
and
in
fact
you
know,
I
think
the
folks
who
played
the
most
pivotal
role
are
probably
you
know
the
sort
of
the
top
half
the
top
three
lines
here.
You
know,
and
so
I'm
getting
to
report
on
a
lot
of
their
good
work.
B
We
have
a
data
set,
you
want
a
model,
that's
going
to
accomplish
some
tasks,
say
image
recognition
and
then
you
want
quality
and
you
want
that
quality
to
be
pretty
leading
edge
right
if
you're
training
to
an
uninteresting
quality
target.
You
know
you
don't
have
an
interesting
benchmark,
and
so
one
of
the
things
to
note
here
is
you
know
we
haven't
said
what
model
and
so
in
the
definition,
one
of
the
things
that
we
originally
looked
at
is
we
said.
Well,
you
know
if
we
really
want
apples
to
apples
comparisons
to
be.
B
You
know
straightforward
right.
We
need
to
specify
the
model.
So
if
something
for
like
image
recognition,
we
want
to
use
resnet,
but
the
reality
is
that
when
you
specify
a
model,
you
are
implicitly
picking
some
winners
and
losers
and
you're
also
precluding
algorithmic
innovations
right.
You
know,
you
might
say:
oh
you
know
just
to
take
resnecks
for
resnet.
For
example,
you
know
there
isn't
there's
the
res
next
networks
that
are
sparse
as
opposed
to
dense
layers
and
that's
going
to
favor
a
different
type
of
architecture
right.
B
So
we
have
an
open
division
where
you
can
modify
the
model,
bring
your
own
model.
Do
it
almost
whatever
you
want
to
really
highlight
more
innovative
approaches,
and
then
we've
got
this
closed
division,
where
you're
going
to
use
something,
that's
mathematically
equivalent
to
our
model
to
ensure
a
high
degree
of
comparability,
and
so
this
is
a
sort
of
key
principle
that
is
applied
to
almost
all
of
our
benchmarks.
If
not
all.
B
So
when
you
look
at
the
training
suite,
this
is
sort
of
an
inventory
of
the
models
the
data
sets
and
the
quality
targets
that
we
have
for
the
different
training
benchmarks.
Now
this
does
not
include
the
three
ml
perf
hpc
benchmarks.
You
can
read
about
those
on
our
website
or
you
know
you
can
go
talk
to
steve.
I'm
sure
he'd
love
to
tell
you
about
it
and
get
you
involved
in
helping
to
contribute
to
them
and
run
them.
But
you
can
see
here
that
we've
got
a
pretty
nice
range.
B
We've
got
recommendation
right,
that's
a
very
different
workload
than
anything
else
up
here.
We
so
we've
got
a
bunch
of
convolutional
based
neural
networks
for
vision.
We've
got
a
bert
based
nlp
model.
We've
got
a
recursive
neural
network
for
speech.
We've
got
some
3d
segmentation
and
reinforcement
learning.
So
there's
a
nice
span
here
you
know,
and
so
that's
important,
right
and
and
the
other
thing
is
again.
Our
our
quality
targets
are
pretty
close
to
state
of
the
art.
B
You
know
there
certainly
are
probably,
in
fact
I
know
for
a
fact
you
can
find
say
image.
Classification
that'll
hit
over
75.9
top
one,
but
not
with
resnet50
right.
That's
you
can.
I
think
there
was
a
very
recent
paper
that
showed
through
very
careful
pre-processing.
You
might
be
able
to
get
better
than
this,
but
you
know
this
is
pretty
close
to
state
of
the
art,
so
it
actually
does
represent
something
that
is
both
reasonably
fair
but
also
meaningful
and
generalizable
right.
B
You
know
if
you're
doing
recommendation,
you
know,
dlrm
is
a
really
nice
representative
of
that,
and
you
know
bert
certainly
is
a
very
good
representation
of
what
a
lot
of
nlp
is
going
to
look.
Like
I
mean
the
ones
highlighted
in
blue
are
the
ones
that
we
added
most
recently
for
anyone
who's
curious
about
that.
B
So
we've
talked
about
models
and
data
sets,
but
one
of
the
questions
is
well.
What's
the
metric
you
want
to
establish-
and
this
is
actually
really
important,
because
I
think
one
of
the
points
of
a
benchmark
is
not
just
getting
vendors
to
beat
themselves
up
and
improve,
but
it's
actually
promoting
a
shared
understanding
of
what
it
means
to
be
good
right,
and
so
this
metric's
actually
really
important.
B
You
know
someone
you
know
right.
We
were
talking
before
the
talk
about
glenn
wrote,
a
blog
post,
saying
that
iops
are
an
evil
metric
for
storage,
because
you
know
to
summarize
they're,
probably
not
terribly
representative,
they're,
overly
simplistic
right,
and
so
that's
a
great
illustration
of
why
it's
important
to
pick
the
right
metrics.
So
the
metric
that
we
picked
was
time
to
train
and
the
only
real
other
option
that
was
suggested
was
sort
of
throughput.
B
Like
you
know,
samples
per
second
processed
now
time
to
train
has
some
some
very
significant
advantages
that
I
want
to
talk
about,
one
of
and
some
disadvantages.
So
one
is
it's
the
most
representative
of
what
your
customers
care
about
right,
like
knowing
the
time
to
train
that
directly
corresponds
to
what
a
user
will
experience.
B
So
that's
good.
It
also
captures
a
lot
of
important
algorithmic
and
numerical
trade-offs
right.
So
if
you
look
at
a
lot
of
training,
there's
actually
a
lot
of
training
done
in
fp32,
because
it's
the
most
accurate
right
and
so
there's
a
lot
of
people
who
are
interested
in
doing
training
in
reduced
precision
and
that's
fine.
But
in
some
cases
it
may
increase
the
number
of
epochs
that
you
need
to
converge
in
an
extreme
case.
B
You
may
not
converge
to
the
right
accuracy
right
and
so
again
in
keeping
with
let's
align
the
benchmark
to
what
creates
value.
That's
a
very
important
step
because
it
forces
people
to
make
trade-offs
against
you
know
actual
accuracy
and
and
time
to
train
right.
If
you
can
train
an
intake,
that'll
certainly
boost
your
throughput,
but
if
you
can't
get
to
the
target
accuracy,
then
you
know
you
do
not
have
a
solution.
That
is
valuable
and
I
think
that
enforces
these,
these
trade-offs,
as
well
as
exposes
things
like
increasing
batch
size.
B
You
know
pretty
typically
increases
the
number
of
epochs
right,
so
we
want
to
expose
those
trade-offs
which
we
think
are
critically
important.
Now
the
downside
is,
it
is
computationally
expensive.
You
know
a
lot
of
our
benchmarks
take
about
a
v
100
week
and
some
of
the
hpc
benchmarks
might
be
longer.
B
It
is
computationally,
expensive
and
it's
high
variance.
Unfortunately,
convergence
is
an
intrinsically
stochastic
thing
and
we'll
talk
about
that
in
a
little
bit
later,
but
ultimately
we
felt
it
was
the
best
choice,
and
this
also
requires
that
you
use
real
data.
I
want
to
point
out
and
that's
actually
critically
important,
because
in
a
lot
of
cases
you
know
using
synthetic
data
will
not
get
you
meaningful
results
on
ml,
and
so
you
have
to
be
very,
very
careful
about
that
right.
You
know
if
the
data
distribution
is
not
right.
B
You
know
as
a
simple
example.
If
you
have
some
sort
of
like
mask
rcnn,
which
has
you
know
a
proposal
based,
you
know
you
identify
regions
of
interest
and
then
you
start
doing
segmentation
and
object
detection
within
there.
If
you
use
bogus
data,
you
will
not
get
the
right
numbers
of
regions
of
interest
as
you
train,
and
so
you
kind
of
don't
get
a
useful
answer
out
of
that.
B
B
That
depends
a
lot
on
how
your
cluster
is
implemented
and
we
don't
really
want
people
to
go
out
and,
like
optimize,
you
know
pretty
significant
clusters
just
for
our
benchmark
right.
You
know,
if
you
look
at
some
of
the
clusters
that
training
is
run
on,
you
know,
in
the
early
days
it
was
64
processors
or
accelerators
these
days.
It's
4
000.,
that's
a
really
big
cluster
and
we
don't
want
people
kind
of
trying
to.
B
You
know
really
game
the
system
in
ways
that
don't
add
direct
value
so
again,
model
initialization
that
becomes
a
disproportionately
large
part
of
the
training
time
for
a
big
system.
Unless
you
use
a
big
data
set
right
and
in
order
to
keep
the
runtime
reasonable,
we
can't
use
giant
data
sets
right.
You
know,
if
you
look
at
like
some
of
the
recommendation
systems
out
there.
You
know,
for
example,
I
think
baidu.
B
We
don't
want
people
to
worry
about
like
what
data
format
things
are
in,
so
we
let
you
sort
of
pack
and
unpack
and
rearrange
in
a
sort
of
principled
way
for
free
right,
because
you
know
otherwise,
then
people
start
arguing
about
well.
What
format
did
you
put
the
the
the
data
in?
Oh,
you
picked
a
format
that
doesn't
advantage
me
like.
We
shouldn't
be
arguing
about
those
things,
so
I
want
to
now
dive
into
some
of
the
challenges
that
we've
run
into
so
on
the
training
side
right.
B
The
one
issue
is
that
there's
no
agreement
on
what
you
know,
software
stacks
and
hardware
systems,
look
like
right.
If
you
look
at
the
number
of
startups
doing
training,
it's
a
lot,
there's
like
over
a
dozen
and
with
wildly
varying
architectures
right.
You
know
you've
got
like
cerebral
systems
with
a
wafer
scale
system.
With
you
know,
hundreds
of
thousands
of
very
simple
processing
elements.
You've
got
tpus
with
very
large
matrix
multiplication.
Arrays
you've
got
all
sorts
of
other
things
in
between.
B
You
could
conceivably
train
on
an
fpga,
et
cetera,
et
cetera,
et
cetera,
and
you
also
have
different
software
stacks
right.
Some
companies
have
proprietary
ones
like
baidu
uses,
paddle
paddle,
you
know
many
cloud.
Vendors
have
their
own,
preferred
stack,
there's,
tensorflow
and
pi
torch,
which
are
you
know
the
two
heavyweights,
and
so
what
this
means
is
you
know
you
from
an
apple
from
a
comparison
standpoint.
We
can't
even
use
the
same
code,
let
alone
the
same
executable,
to
get
this
apples
to
apples
comparison.
So
this
makes
things
pretty
tricky.
B
Another
issue
which
I
sort
of
alluded
to
before
is
that
we
run
at
different
scales
right.
Certainly,
training
with
a
4000
accelerator
system
is
pretty
interesting
for
some
of
these
cutting
edge
models,
but
you
know
there's
also
people
who
you
know
all
they
want
is
maybe
two
four
or
eight
accelerators,
or
even
maybe
just
a
standard
server
running.
You
know
in
sort
of
a
workstation
like
configuration,
and
so
that's
valuable
too,
and
those
are
like
very
different
scales,
which
means
in
many
cases
you're
gonna
want
different
batch
sizes
or
different
optimizers.
B
So
you
know
you're
going
to
have
fairly
different
tuning
and
then
as
well.
Hyperparameters
need
to
be
tuned
as
well,
but
parameter
tuning
can
be
very
expensive,
and
is
it's
not
clear
that
that's
actually
super
productive
you've
also
got
different
numerics,
as
I
alluded
to
before
right.
You
know.
If
you
look
at
mlperf
submissions,
I
think
you'll
see
everything
ranging
from
fp32
to
fp16,
to
bfloat
16
on
the
training
side
and
in
inference
there's
an
even
wider
variety
right.
So
we
want
to
support
that.
B
You
know
we're
going
to
need
a
wide
variety
of
you
know,
potentially
different
hyper
parameters,
batch
sizes,
etc.
So
that
makes
things
hard
as
well
and
then
the
last
thing
that
I
mentioned
before
is
convergence
is
stochastic
right.
So
you
know,
typically,
when
you
train,
you
will
initialize
with
random
weights
that
random
weight
initialization
can
have
a
pretty
big
impact.
B
Anecdotally,
I
can
say
you
know:
I've
seen
cases
where
performance
is
varied
by
twenty
percent
run
to
run
due
to
weight
initialization,
so
you
can
get
lucky,
you
can
get
really
lucky
and,
and
that
makes
benchmarking
hard
you
know.
In
addition,
you
know
we're
talking
about
floating
point
and
and
floating
point.
Numerix.
Are
you
know,
sort
of
inherently
hard
to
reproduce,
especially
once
you
start
talking
about
some
accelerators
have
stochastic
floating
points
where
sort
of
the
rounding
may
be
performed
somewhat
stochastically.
B
So
there's
a
lot
of
things
that
that
make
ml
training
just
really
tough
to
tackle,
and
so
you
know
next,
I'm
going
to
talk
about
and
illustrate
some
of
these
in
more
depth
and
then
talk
about
the
answers.
So
here's
an
example
of
convergence
variation
on
resnet
and
this
actually
looks
pretty
good.
So
what
you
can
see
is
we've
got
five
different
runs
and
they
all
basically
converge
around
60,
epochs,
more
or
less.
So
that's
great!
You
know
the
curves
are
a
little
unstable
early
on,
but
the
convergence
looks
pretty
good.
B
B
I
mean
the
average
is
probably
around
40
epochs,
but
you're,
seeing
things
as
low
as
30
as
high
as
probably
40.
You
know,
somewhere
between
35
and
45,
is
the
range
that
it
looks
like
and,
and
it's
not
clear
that
there's
really
a
rhyme
or
reason
to
that.
So
you
know
it's
a
benchmark,
specific
thing
how
the
convergence
works
and
some
benchmarks
are
just
trickier
than
others.
You
know,
so
that's
an
example
of
how
convergence
can
be
tough.
C
Yeah,
I
think
normally
we
we
allow
some
questions
in
line,
but
if
it's
getting
crazy
and
you're
not
getting
to
stuff
that
you
want
to
talk
about,
then
just
tell
people
to
stop.
But
I
was
just
trying
to
understand
this
and
whether
it
was
like
saying
the
different
experiment
with
the
same
seed
was
actually
still
right.
So.
B
B
So
when
it
comes
to
the
diverse
software
and
hardware
stacks,
we
sort
of
pioneered
this
notion
that
is
carried
over
to
almost
all
of
our
benchmarks,
of
a
reference
implementation
that
sort
of
defines
what
canonical
correctness
is
and
then
rules
for
re-implementation.
So
we'll
say:
hey
this
model
in
tensorflow
is
what
it
means
to
be
correct.
B
I
think
there's
probably
five
different
software
stacks
that
have
been
used:
tensorflow
and
pi
torch,
jacks,
mxnet
usinian,
which
is
from
alibaba
and
then
paddle
paddle
from
baidu.
So
I
think
that
actually
gets
us
up
to
six.
So
that's
been
like
super
successful
and
it's
nice
because
it
does
allow
four
comparisons
in
some
cases
between
different
frameworks.
B
So
just
as
an
example,
you
know
there
was
one
submission
that
google
did
where
they
had
both
jackson
tensorflow
at
fairly
large
scale,
and
you
know
their
their
point
was
really
that
despite
the
fact
that
jax
is
a
relatively
new
framework,
you
know
it's
getting
performance
that
is
very
close
to
tensorflow,
but
in
a
potentially
much
more
programmer
friendly
way.
So
one
of
the
nice
things
about
having
done
this
is
it
turns
ml
perf
into
a
very
flexible
tool
right.
B
It's
not
just
for
hardware,
it's
not
just
for
systems,
but
you
can
make
software
comparisons
as
well,
potentially
so
in
dealing
with
the
different
scale
and
numerics,
we
decided
to
address
this
by
having
a
limited
set
of
tunable
hyper
parameters
right
and
so
we
sort
of
started
with
okay.
Let's
talk
about
changing
batch
size
and
numerics
and
then
downstream.
B
Let's
then
make
other
hyperparameters
tunable
that
you
would
want
to
change
as
you
change
those
two
things.
So
you
know,
for
example,
you
might
change
learning
rate
as
you
change
batch
size
right
and
then
actually,
we've
sort
of
in
recent
years
are
in
the
most
recent
around
maybe
two
rounds.
We
started
something
called
reference
convergence
points
where
you
know
for
a
given
set
of
hyper
parameters,
we
sort
of
have
a
suggested
number
of
epochs
something
should
take,
and
this
helps
to
provide
a
lot
of
guidance
to
submitters
and
sort
of
longer
term.
B
So
this
is
actually
some
work
that
we're
doing
that's
actually
pretty
novel
and
researchy,
but
could
be
a
great
contribution
to
the
community
overall
and
then
with
respect
to
the
stochastic
convergence
issue,
which
I
illustrated
previously,
you
know
we
did
sort
of
the
most
obvious
thing,
which
is
you
know,
we're
going
to
do
multiple
runs
and
average
it.
Actually.
B
What
we
do
is
olympic
scoring,
where
you
know,
you'll,
have
five
or
ten
runs
and
drop
the
highest
and
lowest
and
average
the
rest,
and,
generally
speaking,
the
number
of
runs
required
is
picked
such
that
you
know
our
actual
reported
score
is
expected
to
be
plus
or
minus
five
percent
of
the
true
value
right.
So
we're
aiming
for
a
certain
level
of
uncertainty
and
once
we
get
within
that
we're
good
to
go.
B
Now
I
want
to
talk
a
little
bit
about
what
the
submission
process
looks
like
with
some
illustrations
here
so
and,
and
there's
a
little
bit
more
details
here
as
well,
so
we've
got
two
divisions.
I
mentioned
them
before
the
closed
in
the
open
and
here's
a
little
bit
more
examples
of
how
they
vary
right
in
the
closed
division.
You've
got
very
limited,
hyper
parameters.
You
don't
get
to
change
things
like
padding
on
images.
B
Sorry,
no!
You
do
get
to
change
padding
in
numerics,
but
you
can't
change
the
data,
sort
order
or
number
of
layers,
for
example,
whereas
in
open
you
don't
need
to
be
mathematically
equivalent
to
the
reference.
So
you
can
bring
your
own
network
right
and
you
can
say:
hey.
I
don't
like
resnet,
I'm
going
to
use
res
next
or
I'm
going
to
use
efficientnet
or
you
know
whatever
it
is.
B
As
long
as
you
hit
the
target
accuracy,
it's
fine
and
I
think,
even
for
some
things,
we
may
not
even
require
that
you
hit
target
accuracy,
but
it
must
be
reported
and
then,
in
addition-
and
this
may
be
a
little
bit
more
of
there's-
some
impacts
on
submission.
But
this
is
also
somewhat
of
a
marketing
aspect.
We
have
three
different
divisions
and
so
that's
the
commercially
available
there's
the
available
division,
where
every
component
of
the
system
is
commercially
available.
B
B
You
know
anyone
else
should
be
able
to
go
and
rent
the
stuff
run
it
and
get
this
about
the
same
scores
right
and
if
they
can't
then
there's
a
problem
right
now,
then
we've
got
a
preview
division
which
is
for
for
systems
that
contain
one
or
more
components
that
are
not
yet
general
availability
but
will
be
within
six
months,
and
so
we
let
you
submit
into
preview,
but
then
you
must
subsequently
submit
into
available
and
sort
of.
The
idea
here
is
this
benchmark
happens
every
six
months.
B
You
know
basically
every
two
quarters:
we
don't
want
to
force
engineering
teams
to
alter
their
software
release
cycles
to
accommodate
us.
So
we
say:
look
if
your
software
isn't
ready
in
time.
You
can
do
preview
and
then
you
just
submit
again
and-
and
you
know,
we
can
be
sure
that
you
weren't
like
doing
anything
funny
with
non-ga
software.
That
does
weird
things
or
weird
non-ga
hardware,
and
then
last
we
have
this
rdi
category
for
research
prototypes.
B
You
know
development
systems
and
internal
systems,
so
this
would
be,
for
example,
for
things
that
are
just
not
publicly
available.
So
we've
actually
seen
a
bunch
of
people
submit
fpga
prototypes
of
systems
which
is
really
cool.
You
know,
if
you
think,
about
what
that
does
for
the
community.
You
know
that
actually
gives
folks
the
ability
to
start
working
on
their
software
stack.
B
If
you
can
make
it,
I
mean
we
have
had
some
folks
who
did
this,
who,
like
literally,
showed
up
like
right
at
the
11th
hour
and
had
only
read
the
rules
and
had
not
really
attended,
and
you
know
there
were
some
problems
in
the
rules,
and
so
they
ended
up
with
an
open
submission.
Instead
of
close,
that's
that's
not
an
uncommon
thing,
so
you
know
once
you've
sort
of
played
around
and
gotten
a
lay
of
the
land.
You'll
want
to
re-implement
the
benchmark
for
your
system.
You
know
to
tune
for
maximum
performance.
B
B
Is
this
peer
review
process
where
you
and
your
closest
nearest
and
dearest
competitors
rivals-
or
I
guess
in
the
case
of
hpc,
you
know
peer
labs,
look
at
each
other's
submission
and
find
the
errors,
and
I
you
know
we
have
not
had
a
submission
go
by
where
there's
not
an
error.
Some
of
them
are
most
of
them.
Are
you
know
pretty
innocuous
and
bone-headed
things,
but
you
know
many
eyes
make
light
work,
and
this
is
one
of
the
ways
that,
with
such
a
complex
submission,
we
can
keep
the
quality
high.
B
How
do
we
keep
things
fair
for
startups
and
big
companies
alike
and
while
also
enriching
the
field?
So
you
know
if
you
find
a
lucky
set
of
hyper
parameters
now
everyone
knows
about
it
and
the
world
is
a
better
place
and
then
last
once
the
review
is
complete.
We
actually
post
all
of
those
artifacts
under
an
apache
2
license,
so
customers
and
end
users
can
go
grab
it
and
and
run
it
themselves
and
learn
from
it.
B
So
you
know
again,
how
do
we
drive
the
whole
industry
forward
and
and
that's
the
point
of
the
process
and
then,
of
course,
we're
gonna
celebrate
because
it's
a
non-trivial,
endeavor
and-
and
you
know
we
like
to
celebrate
with
cake
when
we
can
actually
get
together.
But
you
know
not,
it's
been
a
while
it's
probably
good
for
my
belly,
but.
D
D
B
It's
not
it's
not
arbitrary.
It
has
to
be
a
set
of
hyper
parameters
that
were
already
submitted.
So
what
this
means
is,
you
know
you
have
to
submit
the
hyper
parameters,
and
you
know
it
just
limits.
The
advantage
of
you
know
really
going
out
and
trying
to
find
the
best
type
of
parameters
that
that's
really
the
point
there.
B
Yes,
so
during
the
post
submission
process,
everyone
can
see
everyone
else's
submission,
or
you
know
all
the
people
who
submitted
get
to
see
all
the
submissions.
So
you
know
if
you
and
I
both
submitted.
I
would
get
to
look
at
your
hyper
parameters
and,
if
I
think
they're
better,
I
can
go
and
borrow
them
right
and
if
you
think
mine
are
better,
you
can
borrow
them
right.
So
that's
the
way
it
keeps
the
playing
field
level.
Does
that
make
sense,
but
you
can't
use
like
an
arbitrary
new
set
of
hyper
parameters.
B
B
B
B
B
B
So
you
know
but
yeah
I
mean
open.
Submissions
are
reviewed,
just
like
closed
submissions.
Okay-
and
you
know
we
do
require
specifically
for
for
any
available
system.
It
must
be
something
that
can
be
reproduced
right.
That's
not
optional,
for
a
commercial
system
for
preview
and
for
rdi
it
is
you
know.
As
an
example,
I
think
google
submitted
the
tpu
v4
and
rdi
well
before
it
was
publicly
available
to
anyone
else
right
and
so
there.
In
that
case,
you
can't
really
reproduce
it
now.
B
I
think
other
submitters
did
reproduce
sort
of
the
number
of
epochs
to
train
using
that
model
to
make
sure
that
it
passed
the
smell
test,
but
obviously
no
one
else
could
get
the
actual
execution
time,
because
no
one
else
could
get
a
tpu
v4
at
the
time
right.
But
for
you
know
something:
that's
commercial,
all
commercially
available
components
it.
You
know
it
should
be
absolutely
reproducible.
B
B
So
I
think
the
first
thing
is,
it's
pretty
great
to
see
the
impact
of
good
benchmarking
right.
You
know
we
we
set
out
with
a
clearly
defined
set
of
problems,
as
I
articulated
before,
and
you
know,
the
net
result
is
we're
getting
all
sorts
of
folks
from
all
across
the
industry
to
really
compete
on
best
performance
right
and
we've
got
results
that
show
what
works
best
and
the
net
result
is.
First
of
all,
we
are
now
getting
a
much
better
understanding
of
performance.
B
You
know,
one
of
the
things
I
like
to
say
is
you
know,
sort
of
before
ml
perf
came
around.
Everyone
would
talk
about
flops
as
a
good
performance.
Metric
and
flops
is
a
good
performance.
Metric
is
not
good
right.
It's
just
you
know
very
misleading
in
in
a
large
number
of
ways
right,
so
we've
kind
of
established
a
common
understanding
of
what
performance
really
means
that
aligns
with
customer
value
number
one.
We've
got
gotten
a
lot
faster
software
stacks.
I
have
some
data
on
that
in
a
bit.
B
I
believe,
and
we're
now
starting
to
see
hardware
that's
being
designed
with
mlperf
in
mind
and
that's
both
a
blessing
and
a
curse
right.
It's
a
blessing
in
that
to
the
extent
that
mlperf
is
representative
and
generalizable,
then
that's
great
right.
It
helps
point
architects
in
the
right
direction.
You
know
designing
for
the
benchmark
is
always
you
know,
there's
a
little
bit
of
mixed
feelings
on
that,
and
you
want
to
make
sure
that
there's
not
over
optimization
there,
but
it's
also
sometimes
a
little
hard
to
prevent.
B
B
You
know
sort
of
how
many
more
transistors
would
we
naively
expect
to
get
with
this
green
line
on
the
bottom
right,
and
so
you
know
it's
about
two
and
a
half
times
more
transistors
over
this
period
in
time,
and
when
you
look
at
the
actual
performance
that
we're
seeing
it's
more
like
you
know,
16
or
30
x,
more
performance
right,
so
ml
perf
is
helping.
The
industry
drive
performance
very
fast,
like
30x
over
even
16x
over
two
years
or
two
and
a
half
years
is
a
pretty
good
pace
right.
B
That's
that's
fantastic,
and
so
the
benchmarks
here
are
are
the
three
ones
that
have
been
in
all
the
benchmark:
suites
resnet50
is
image.
Classification
ssd,
resnet
34
is
an
object,
detector
and
there's
an
asterisk
by
both
of
those,
and
what
that
means
is
that
we
actually
improved
the
accuracy
target
and
made
it
harder
between
version
.5,
this
first
dot
and
version
0.6,
the
second
dot.
So
if
anything,
this
actually
understates
the
performance
gain
and
then
mask
rcnn
is
an
image
segmentation.
B
Although
the
the
accuracy
went
up,
but
you
know
2x4
over
2x
for
nascar
cnn
about
2x
for
resnet,
you
know
for
transformer
about
1.75x
those
are
pretty
significant
improvements
due
to
software
alone
and
again
same
system
size.
So
you
know
this
is
a
great
example
of
how
ml
perf
can
help
to
drive
value
for
everyone
right.
These
are
free
and
performance
improvements
for
everyone
in
the
community,
at
least
everyone
who
updates
their
software-
and
the
other
thing
I'd
point
out
is
this:
this
is
a
comparison
against
the
fairly
highly
optimized
baseline
right.
B
This
is
not
like
you
know,
comparing
gcc
o2
versus
gcc
or
sorry
gcc,
o0
versus
o2
right.
This
is
someone
tuned
for
the
benchmark
as
best
they
could
the
first
time
around
and
every
subsequent
time.
So
this
is
actually
a
really
nice
comparison,
so
one
of
the
other
ways
that
we're
seeing
a
lot
of
performance
is
system
scale
and
as
hpc
folks,
I'm
sure
this
will
resonate.
Since
you
know
you
look
at
some
of
the
large-scale
exascale
supercomputers
that
we've
got
and
they're
at
a
very
large
scale.
B
So
this
is
a
comparison
of
the
number
of
processors,
whether
cpus
or
accelerators,
in
the
largest
submission
for
ml
per
version,
0.5.6
and
0.7
over
five
different
benchmarks
that
were
in
those
suites,
and
so
what
you
can
see
is,
first
of
all,
there's
a
bunch
of
interesting
things.
First,
each
benchmark
scales
differently
right.
You
look
at
mask
rcnn
and
it
is
capped
out
at
about
512
processors.
In
this
comparison,
while
quite
a
few
of
the
other
ones
are
at
4
000.
right,
each
network
is
different,
has
its
own
challenges,
and
so
that's
great.
B
It's
an
argument
for
diversity
in
the
benchmark
suite.
But
what
you
also
see
is
that
the
amount
of
parallelism
has
gone
up
by
you
know
in
some
cases
64x
over
time
now,
performance
does
not
increase
quite
the
same
with
parallelism,
since
typically,
you
will
need
to
increase
the
number
of
epochs,
so
you're
doing
more
work
to
get
that
parallelism,
but
you're
driving,
much
higher
performance
and
better
overall
time
to
solution-
and
you
know
this
is
going
to
capture
both
improvements
in
software
improvements
in
algorithms
and
potentially
improvements
in
sort
of
model
representation
right.
B
B
B
B
So
I
wanted
to
share
a
selection
of
thoughts
here.
So
first,
you
know
when
you
look
at
time
to
train
smaller
is
going
to
be
better
and,
as
you
look
at
the
results,
there's
a
lot
that
you
can
learn
by
digging
into
them
and
doing
you
know,
clever
comparisons
right
ml
perf
is
a
full
system
benchmark.
It
encompasses,
as
I
said,
your
model,
your
hyper
parameters,
the
training
algorithm.
You
know
your
optimizer
etc.
B
It
includes
software
things
like
your
numerics,
your
compilers,
your
math
libraries.
It
includes
the
hardware,
of
course,
and
not
just
you
know
what
is
your
accelerator
or
processor,
but
what's
the
networking
that's
available?
How
are
the
servers
configured
and
so
there's
a
lot
you
can
learn
by
digging
into
these
submissions
scale
really
matters.
So
you
know,
training
on
eight
processors
is
just
fundamentally
different
from
training
on
4000
right.
You
know,
you're,
going
to
stress
your
interconnect,
a
lot
more
you're
going
to
need
model
partitioning
at
large
scale.
B
Your
efficiency
will
drop
at
large
scale,
and
so
you
know
you
can't
you
know
that
makes
some
of
the
comparisons
really
tricky
over
different
system
scales
right
you're
going
to
need.
You
know,
as
you
go
to
larger
batch
size,
which
is
commonly
done
for
more
nodes,
you're
going
to
need
more
compute
to
converge
right
and
there's
some
sort
of
rules
of
thumb.
I've
seen
where
I
want
to
say
something
like
you
know,
increasing
I'm
going
to
get
it
wrong.
B
B
When
you
look
at
the
results,
I
mean
you
really
have
to
go
in
with
an
open
mind
right,
you
know,
every
submission
is
going
to
say
something,
but
you
have
to
sort
of
dig
in
and
figure
out
what
that
really
is.
So
you
know
try
to
look
at
submissions
that
are
different
on
some
dimension
and
similar
on
others
right,
so
you
could
look
at
you
know
how
does
a
given
accelerator
scale
or
processor
scale
and
system
size?
B
You
could
look
at.
How
does
the
software
evolve
over
time?
How
does
a
new
software
stack?
Do
I
mentioned
the
the
the
comparison
that
google
did
between
jax
and
tensorflow?
That's
a
good
example.
You
can
look
at
how
systems
improve
as
they
go
from
preview
to
available
or
from
rdi
to
preview,
to
available
and
sort
of
see
the
impact
of
tuning
right.
You
know
how
much
better
can
you
get
as
the
vendors
tune,
their
software
stack
and,
of
course
you
can
always
look
at
new
processors
as
well.
B
B
B
One
of
the
big
differences
with
inference
is
we
have
a
lot
of
different
scenarios
where
there
are
different
metrics,
so
we've
got
sort
of
a
latency
optimized
single
stream
case
where
the
metric
is
actually.
What
is
your
query?
Latency
90th
percentile
you've
got
multi-stream,
which
is
sort
of
focused
on.
If
you
have
like
many
cameras
driving
at
once,
how
many
streams
can
you
handle?
B
We've
got
server
which
is
sort
of
mimics.
You
know
any
sort
of
server
type
scenario
and
the
throughput
in
it
and
the
metric
is
sort
of
queries
per
second,
so
throughput,
but
with
a
latency
bound.
So
you
know
you,
it
might
be
queries
per
second
subject
to
a
latency.
Bound
of
99
of
queries
must
be
handled
within
15
milliseconds,
something
like
that,
and
then
we
have
an
offline
scenario
where
it's
just
pure
throughput,
no,
no
latency
constraints
at
all.
B
Submitters.
On
the
infrared
side
again
have
to
use
common
weights,
but
they
can
re-implement
and
they
have
to
use
our
load
generator,
but
they're
allowed
to
re-implement
their
inference,
engines
use
whatever
they
want,
etc
and
the
load
generator
generates,
queries
times
them
and
then
validates
the
the
accuracy
of
the
network
to
make
sure
it's
sufficient.
B
In
order
to
avoid
a
quantization
contest,
we
allow
quantization,
but
it
must
be
described
sort
of
in
a
you
know.
Imagine
like
a
short
page-long
papers,
type
thing
and
you're
allowed
to
use
a
very
small
calibration
data
set.
You
know.
Maybe
a
hundred
or
a
thousand
images,
for
example,
but
you
can't
actually
retrain
and
part
of
the
reason
we
don't
want
to
allow
retraining
is
that
it
would
provide
an
advantage
to
folks
who
are
good
at
training.
B
B
So
that's
a
brief
whirlwind
tour
through
inference,
and
now
I
think
we've
got
five
minutes
for
q.
A
oh
and
I
should
mention
that
the
lh
ml
perf
training
rules
have
been
modified
by
the
hpc
working
group
and
I
think
the
biggest
modification
is
that
the
data
has
to
start
in
like
persistent
shared
storage
for
the
cluster
rather
than
in
any
like
local
node
cache.
Is
that
right,
steve.
C
You
know
new
software
that
pops
up
that
does
amazingly
well
like
jacks
and
and
the
way
that
you
can
look
at
people's
submissions
and
see
the
actual
code
that
was
implemented
and
use
it
yourself
and
all
these
aspects
so
yeah.
I
think
it's
great.
I
just
wanted
to
ask
a
couple
of
questions.
Well
in
this
session
about
the
you
know,
you
talked
about
software
improvements
and
I
was
just
wondering
so
that
was
across
all
software
right,
not
it's
not
for
a
particular
framework.
C
Yeah,
because
I
think
there
it's
also
been
surprising.
How
well
I
mean
for
me
like
there,
isn't
a
tensorflow
isn't
really
on
resnet
competitive
like
mxnet,
and,
I
think
is
the
leading
exam.
I
mean
it's
often
interesting
how
yes,
different
models
or
different
problems,
have
different
frameworks
that
ended
up
being
the
best
one.
For
that.
B
That's
correct
yeah,
that
is,
that
is
absolutely
correct
right
and
I,
I
think
the
results
do
show
that
you
know
some
folks
prefer
mxnet
from
a
performance
standpoint
right
and-
and
you
know
that's
why
we
want
to
have
re-implementation
now.
You
know
the
the
flip
side
is.
You
know
in
my
understanding.
Is
that
mxnet
is
you
know
vastly
less
popular
than
tensorflow
in
right?
So
you
know,
can
you
get
that
performance?
You
know
if
you're
willing
to
use
mxnet
right,
but
you
know
we
can
also
see
what
the
you
know.
B
C
B
B
C
A
B
B
That
will
be
very
clear
about
who
is
number
one
and
they
will
slice
and
dice
it
in
in
peculiar
ways
and
in
the
in
the
as
any
good
marketing
org.
If
you,
if
you
slice
things
fine
enough,
you
can
always
be
the
best.
You
know
I'm
thrilled
to
be
the
best
david
cantor
that
lives
in
san
francisco.
I
think
that
is
true.
B
Sadly,
I
seem
to
have
a
rival
who
lives
in
new
york
and
is
a
movie
producer
or
maybe
he's
in
la.
I
don't
know,
but
you
know
you
know,
I
think
one
of
the
things
that
you
know
so
frankly,
like
first
of
all,
I
love
the
clarity
that
top
500
brings
and
I
really
wish
we
could
get
that
you
know.
I've
talked
with
jack
about
that
and
you
know
because
I
think
one
of
the
things
that's
like
critical.
B
That
mlperf
does
is
you
know
real
data,
real
workloads,
you
know
things
that
are
performance
relative,
sorry,
relevant
and
I'd
love
to
see
it
being
widely
used.
I'd
love
to
see
ml,
perf
hbc
be
widely
used.
You
know
for
all
super
computers-
and
you
know
top
500
is
phenomenally
popular
and
you
know
it's
at
a
different
point
in
the
design
space,
and
you
know
I
think
it
was
designed
to
scale
nicely
right,
whereas
you
know
ml
perf
is
designed
to
make
scaling
hard
and
force
people
to
do
hard
work.
B
You
know
in
in
the
united
states
you're
subject
to
the
wings
of
congress
to
get
money,
and
I
don't
think
that
ml
perf
is
necessarily
fully
comprehensible
to
a
congress
person,
whereas
you
know,
for
starters,
bigger
is
better,
is
always
more
comprehensible
to
anyone,
but
then
yeah,
you
know,
should
you
know
we
we
initially
did
discuss?
Do
we
want
like
a
geometric
weighted
mean,
and
but
the
reality
is
that
not
everyone
cares
about
all
of
the
same
things.
Right.
B
You
know
they
don't
really
care
about
recurrent
neural
networks
for
speech
or
recommenders,
and
so
you
know,
I
think
that
was
sort
of
a
point
that
we
we
kind
of
got
to
to
keep
everyone
happy
and
yeah.
I
don't
know
if
you
have
any
great
solutions
on
how
to
get
you
know.
One
number
that's
easy
to
understand
and
you
know
still,
you
know,
really
meaningful
and
doesn't
get
all
of
the
vendors
wanting
to
strangle.
You
please,
please
see
me
afterwards
and
you'll
become
my
best
friend.
Okay,.
A
B
And,
and
by
the
way,
please
feel
free
to
reach
out
and
talk
to
me
about
these
things,
we're
always
looking
to
engage
more
folks
in
the
community.
You
know
and
and
input
and
collaboration.
You
know
I'd
love
to
to.
Actually
you
know
one
one
thing:
it
might
be
nice
to
actually
get
some
collaboration
between
the
ml,
perf
hbc
and
the
top
500
folks.
You
know
things
like
that.