►
Description
Over the past year I’ve been spending about 20% of my time driving efforts to improve developer-facing parts of TVM, like the Jenkins CI, docker scripts, and pytest infrastructure. In 2022, OctoML plans to invest more heavily to improve the TVM developer experience. In this talk, I’ll discuss the progress made over the past year and some future plans for 2022.
This session is broken into two sections, a 20 minute talk and a 10 minute community breakout session.
Speaker: Andrew Reusch
- - -
Recorded at TVMCon (https://www.tvmcon.org), the Machine Learning Acceleration Conference in Dec 2021.
TVMCon covers the state of the art of deep learning compilation and optimization, with a range of tutorials, research talks, case studies, and industry presentations. We discuss recent advances in ML frameworks, compilers, systems and architecture support, security, training and hardware acceleration.
Connect with us for the latest in ML Acceleration and Deployment:
Website: https://octoml.ai
LinkedIn: https://www.linkedin.com/company/octoml
Twitter: https://twitter.com/OctoML
B
A
Have
this
as
a
way
to
kind
of
encourage
some
discussion,
so
please
feel
free
to
ask
questions
or
or
just
paste,
in
ideas
for
things
that
you'd
like
to
talk
about
in
that
same
chat,
room
chat
box
over
over
there
and
david,
and
I
will
kind
of
try
to
keep
an
eye
on
that
so
yeah
I
just
to
kind
of
give
a
little
bit
of
an
agenda.
I
wanted
to
start
with
just
kind
of
a
taco
review.
Then
we'll
talk
a
little
bit
about
a
couple
different
pieces.
A
The
continuous
integration
we'll
talk
about
some
work.
We've
been
work
doing
on
the
python
dependencies,
as
well
as
some
improvements
to
kind
of
the
way
that
we
write
unit
tests
and
then
we'll
kind
of
open
it
up
for
discussion
there.
I
think
we'll
also
talk
about
future
ideas
in
there
as
well
too
somewhere
in
there.
So
just
a
brief
talk
overview,
just
kind
of
about
me,
those
of
you
who
don't
know
me.
I've
been
achieving
contributor
for
about
a
year
and
a
half.
A
Now
I
work
at
optomel
software
engineer
here
and
I
actually
started
kind
of
my
tvm
journey
primarily
focused
on
micro
tvm.
So
my
background
is
actually
in
embedded
systems,
but
before
that
also
in
web
development
and
kind
of
back-end
systems
as
well
at
google,
and
so
I
kind
of
started
off
working
on
micro
tvm,
which
is
basically
this
effort
to
bring
tbm
to
work
on
bare
metal
systems.
A
A
Hopefully,
we
I'd
just
like
to
kind
of
go
over
some
efforts
that
we've
worked
on
so
kind
of
about
this,
I'm
going
to
just
like
go
over
a
bunch
of
things,
hopefully
kind
of
quickly.
This
is
just
a
mix
of
work.
That's
been
going
on
in
2021
plus.
Some
of
this
is
work
that
hasn't
landed
yet
that
you
know
we're
kind
of
anticipating
landing
on
2022,
and
you
know
this
is
all
a
community
effort.
A
number
of
things
in
here
are
things
that
I
propose.
Some
things
are
I've.
A
I've
tried
to
credit
people
where
I've
taught
I'm
gonna
talk
about.
Excuse
me
stuff
that
other
folks
here
have
proposed
as
well
again.
Everything
in
tbm
has
decided
on
the
forum
through
rfc,
so
I'd
like
to
prompt
some
discussion
here,
but
you
know
again
we'll
we'll
try
to
take
that
back
to
the
the
forum
so
that
those
people
who
aren't
following
along
here
can
can
follow
up
there
again.
If,
if
I
miss
something
here,
I
I
apologize.
B
A
So,
let's
start
with
runtime,
because
I
think
that's
probably
everyone's
most
anxious
part
of
using
of
contributing
to
cvm
ci
certainly
was
mine.
When
I
first
started
there
are
a
lot
of
people
have
had
a
lot
of
days
where
we
see
you
know
eight
hour
plus
builds
here,
as
you
can
see
in
this
screenshot
of
jenkins.
A
I've
placed
at
the
top
right
hand
corner
here
and
there's
a
lot
of
different
causes
behind
this
early
in
the
year
we
were
kind
of
fighting
some
problems
with
having
our
jenkins
executors
backlogged.
So
when
you
submit
a
pr
it's,
it
gets
run
on
several
different
types
of
computers
and,
depending
on
the
the
low,
the
the
workload
basically
on
these
machines,
those
machines
can
sort
of
get
backlogged
and
there
can
be.
A
A
Another
problem
that
we'll
talk
about
here
in
a
second
is
test
running
sequentially,
so
we'll
kind
of
go
through
that
here
on
that
last
bit
test
running
sequentially.
This
is
something
that
I've
worked
on
a
bit
this
year
and
we
haven't
quite
managed
to
land
it.
Yet,
but
it's
something
that's
you
know
getting
close
to
ready
and
pi
test.
A
Xdis
is
essentially
a
model
for
excuse
me:
a
module
for
pi
test
that
allows
you
to
spin
up
a
set
of
of
executor,
nodes
on
exceeding
executive
processes
on
your
machine
and
then
sort
of
dispatch
tests
to
each
executor.
And
so
you
know
kind
of
a
question
you
might
have
then
is
like
well.
If
we
can
do
this,
why
don't
we
just
run
all
the
unit
tests
in
parallel
and
not
every
unit
test
and
tdm
can
actually
be
run
in
parallel,
some
gpu
tests?
A
A
The
the
typical
approach
here
is
to
choose
the
number
of
cpus
essentially-
and
I
guess
kind
of
our
current
approach
is
to
run
multiple
pr's
at
once
per
executor,
and
so
you
know
choosing
number
of
cpus
for
each
different.
Each
pr
is
going
to
be
problematic
because
it
means
we're
going
to
sort
of
overload
the
the
node
by
by
twice
or
or
more,
and
so
you
know
just
kind
of
circling
back
on
what
I
mentioned.
C
A
On
runtime,
now
I'd
like
to
talk
about
reliability-
and
you
know,
we've
all
seen
a
lot
of
various
flaky
unit
tests
here
there
are
a
bunch
of
different
causes
of
of
these
tests,
and
we've
been
kind
of
working
on
the
infrastructure
side.
To
start
addressing
the
problems
behind,
essentially
the
the
infrastructure
side,
problems
behind
these
flaky
tests,
so
things
like
cuda
running
out
of
memory,
things
like
network
problems
and
things
like
disc
problems,
that's
kind
of
our
focus
kind
of
for
at
least
folks.
B
A
Have
a
few
other
things
that
we
can
also
focus
on
or
ways
that
we
can
improve
these
tests?
One
of
the
things
that
is
kind
of
interesting
about
the
way
that
we
run
our
tests
is
we
run
everything
in
the
root
directory
of
of
tvm,
and
there
are
some
good
things
that
come
out
of
this,
because
it
means
that
the
ci
is
kind
of
doing
the
same
thing
you
do
locally.
A
You
know
erroneously
finish
certain
past
certain
tests
or
or
have
tests
behave
differently
than
they
would
either
on
the
next
pr
run
or
even
even
locally,
so
it
makes
it
kind
of
harder
to
reproduce
okay.
So
the
other
thing
to
say
is
that
we'll
probably
add
some
monitoring
as
a
way
to
kind
of
a
way
to
kind
of
understand.
What's
going
on
here
and
I'll
talk
about
that
a
little
bit
more
here
under
reporting
as
well
sorry
and
yeah
we'll
get
there
in
a
second.
A
This
is
kind
of
an
issue
we
run
into
with
tbmci,
where,
if
you
wanted
to
rerun
a
test
that
failed
locally,
how
might
you
do
that
and
I've
we've
heard
well.
I've
personally
heard
multiple
people
that
come
to
tvm
and
kind
of
try
to
get
started
with
it,
and
they
kind
of
ask
me
this
question.
I
say:
well,
you
just
have
to
go.
A
You
know,
read
this
jenkins
file
and
figure
out
what
it
does,
and
this
is
a
pretty
challenging
thing
for
someone
who
isn't
familiar
with
jenkins
jenkins
is
a
pretty
complicated
system,
and
not
all
of
the
shell
commands
that
we
run
in
our
jenkins
file
are
explicitly
encoded
there.
So
you
know
we're
one
of
the
things
that
dave
has
been
working
on
and
I
apologize.
I
should
have
credited
his
name.
A
And
so,
if
you
see
in
this
log
here,
there
really
isn't
an
obvious
way
for
you
to
address
these
problems.
But
with
a
script
like
this
at
left,
we
can
add.
You
know
information
to
this
log
like
to
rerun
these
tests.
You
know,
run
this
command
locally
and
and
provide
kind
of
a
guidance
for
people
to
be
able
to
do
that.
A
Also,
on
the
the
kind
of
reproducibility
side
is
kind
of
about
the
way
that
we
build
our
docker
images
here,
and
you
know
if
you
take
a
look
at
kind
of
how
jenkins
runs
unit
tests
here
in
tvm
you'll
notice,
that
everything
is
essentially
a
series
of
these
docker
bash
commands,
followed
by
a
docker
image
name.
So
all
of
our
tvm
tests
run
inside
these
docker
containers
and
this
docker
image
you
know,
specifies
the
container
that
that's
used
and
we
often
see
this
very
painful
process
happening
in
the
tvm
project.
A
Which
is
you
know,
essentially
someone
decides
they
want
to
add
some
new
dependency
of
tbm.
Maybe
it's
a
python
package.
Maybe
it's
a
new
library.
Maybe
they
even
just
want
to
update,
for
example,
the
version
of
tensorflow
or
pytorch.
That
teo
depends
on
well,
though
they're
tasked,
then
with
producing
a
set
of
new
docker
containers
that
and
then
validating
those
those
containers
on
the
tvm
build.
So
what
do
I
mean
by
that?
I
mean
basically
you'd
run.
A
It
can
take
multiple
days
one
of
the
things
that
leandro
who's
on
the
call
here
did
for
us
this
year
was
he
made
a
an
automatic
rebuild
process
so
every
day
our
jenkins
will
check
out
what's
in
in
maine
and
then
automatically
rebuild
all
of
the
images
and
then
test
them
for
everyone,
and
so
that
way
we
know
kind
of
on
a
daily
basis.
You
know:
do
we
have
essentially
how
we
for
the
the
configuration
of
the
unit
test
infrastructure
checked
into
main?
A
Can
we
actually
rebuild
our
docker
containers
and
then
can
we
run
tests
against
them?
So
just
wanted
to
point
this
out.
I
think
this
has
been
a
great
advancement
and
there's
you
can
kind
of
pull
these
containers
on
tlc
pack
staging
down
and
and
test
them
before
we
actually
kind
of
include
them
in
the
ci
okay.
A
So,
for
example,
if
we
see
flaky
tests,
we
don't
have
currently
a
way
to
actually
produce
a
list
of
tests
that
we
actually
think
are
flaky
at
maine
right
now,
if
we
are,
you
know
thinking
about
kind
of
what
we're
working
on
earlier
in
the
year
as
far
as
load
statistics,
you
know
we
see
basically
sorry
as
far
as
backlogs
to
to
to
ci
workers.
We
have
kind
of
this
very
simple,
jenkins,
dashboard
that
gives
us
an
idea
of
okay
for
all
cpu
jobs.
A
How
many
nodes
are
are
available,
and
you
know
this
is
looking
kind
of
after
we've
added
a
whole
bunch
of
nodes
to
the
picture
here.
But
you
know
we
don't
have
any
sort
of
monitoring
or
alerting
that
you
might
expect
in
a
traditional
production
system
and
then,
lastly,
kind
of
on
the
user
experience
side
when
you're
kind
of
trying
to
look
at
a
at
a
kind
of
test
that
are
tests
that
have
failed.
You
might
notice
here
that
we
we
do
have
this
nice
github.
A
Sorry,
this
nice
jenkins
ui
here
for
listing
out
kind
of
all
of
the
the
python
tests
that
ran
but
you'll
notice
that,
in
the
top
right
hand
corner
we
have
you,
know,
26400
tests
that
we're
passing
and
in
this
this
test
result
below.
You
can
see
that
only
23
000
tests
passed,
but
the
run
as
a
whole
failed
and
what
happened
to
the
remaining
3
000
tests.
A
A
It
also
lets
us
document
this
and
kind
of
provide
the
community
with
a
way
that
they
can
run
their
own
jenkins
kind
of
internally
if,
if
they
want
to
based
on
all
this
documentation
that
we
have
so
we
just
published
a
prfc
on
this,
and
I
just
wanted
to
point
everyone
to
that.
If
they
kind
of
wanted
to
take
a
look
at
work,
that's
been
ongoing
there,
okay,
so
that
was
kind
of
a
long
stint
about
ci.
A
So
in
the
python
dependency
size,
there's
a
lot
of
things
that
you
might
want
out
of
tbm
in
terms
of
dependencies
as
a
starter,
if
you're,
a
tbm
user,
you'd
really
like
to
be
able
to
run
pip,
install
tlc
pack
and
then
arrive
at
an
installation
that
has
a
reasonable
set
of
dependencies
alongside
of
it.
So,
for
example,
we
might
you
might
want
something
that
says
install
you
know:
tensorflow
2.4,
we
probably
test
against
2.4.3,
so
you
know
install
that
one.
A
A
big
challenge
we
have
with
getting
to
this
kind
of
world
is
finding
where
the
dependencies
are
inside
of
tbm
and
a
number
of
them
are
sort
of
documented,
and
I
put
this
in
quotes
here
instead
of
dot
pi
and
also
in
gen
requirements-
and
I
put
this
here
in
quotes
because
you
know
it's
sort
of
maintained.
You
know
by
convention
right
now
and
there's
nothing
to
enforce
that.
A
What's
documented
in
setup.pi
and
gen
requirements
is
actually
what's
tested
in
ci,
the
real
set
of
dependencies
that
actually
live
in
ci
are
existing.
Ci
live
in
these
various
install
scripts
underneath
docker
and
they're
all
in
the
form
of
these
pip3
install
commands.
Excuse
me,
so
nothing
is
particularly
machine
parcel.
A
So
you
know,
how
are
we
planning
to
address
this?
I
just
want
to
go
over
this
brief
approach
and
it's
an
rfc
that
I
submitted
earlier
this
year.
Essentially,
the
idea
is
to
put
all
the
dependencies
in
one
place
in
python,
gen
requirements.pi,
including
functional
constraints.
So,
for
example,
if
we
know
that
we
can't
use
tensorflow
2.5
list
that
in
gen
requirements
as
as
kind
of
a
known
problem,
don't
try
to
upgrade
past
2.4,
for
example,
then
we
would
sort
all
the
containers
into
kind
of
python
architecture.
A
A
That's
these
five
in
the
middle
here
and
what
are
the
containers
that
use
i386?
Well,
that's
the
i386
container.
Finally,
then,
for
each
architecture,
you
can
run
this
poetry
lock
command,
which
arrives
at
a
set
of
python
dependencies
that
are
all
declared
to
be
compatible
with
one
another,
and
this
kind
of
produces
essentially
a
constraints
file,
which
is
a
constraints
file
referenced
on
the
last
side,
and
that
says
for
all.
X86
64
containers
use
this
set
of
python
packages.
A
With
these
exact
versions,
then,
from
there
in
the
docker
build
process,
you
can
consult
that
list
when
you're
installing
the
dependencies
in
each
container.
Once
that's
those
docker
containers
have
all
been
built.
You
then
construct
the
setup.pi,
sorry
construct.
The
install
requires
argument
in
setup.pi
from
kind
of
these
constraints
files
here
and-
and
you
may
find
that
when
you
do
this,
that
may
be
a
slightly
different
revision
of
tensorflow
was
chosen
in
ci.
A
A
The
thought
is
that
if
you
have
tensorflow
2.4,
perhaps
when
you
say
you
know
what
you
install
tvm
against
you
install
it
against
tensorflow,
2.4
and
so
lastly,
then
speaking
of
kind
of
this
developer
flow,
you
populate
a
top
level
pi
project
optoma,
which
includes
these
kinds
of
constraints
in
a
way
similar
to
setup.pi,
okay.
So
that's
kind
of
what
I
want
to
say
on
python
dependencies.
Moving
on
to
unit
test
improvements.
A
This
is
work,
that's
due
to
eric
lunderberg
and
I
think
it's
been
really
helpful
kind
of
as
a
way
to
gain
more
visibility
into
our
unit
test
coverage.
We
previously
had
a
lot
of
unit
tests
that
were
written
with
for
loops
and
they
would
for
basically
have
a
for
loop
over
a
few
different
targets,
say
and
run
the
same
test
over
those
targets.
So,
for
example,
you
can
see
here
that
this
test
relu
operator
on
the
right
hand
side.
A
This
is
what
the
parameterized
stuff
basically
includes,
the
the
target
that's
being
used
as
well
as
some
parameters
that
kind
of
influence.
The
way
the
test
is
laid
out
and
on
the
left
side.
Here
is
what
we
started
with
before
this
rfc
landed
this
year,
and
you
can
see
that
it's
really
hard
just
from
reading
the
test
report
to
tell
exactly
which
targets
are
being
tested
and
which
kind
of
operator
sizes
are
being
tested
as
well.
A
Pi,
test,
parameterize
fixtures
and
so
kind
of
the
idea
here
is
that
each
function
instead
of
writing
a
for
loop.
It
just
accepts
an
argument
that
is,
you
know
the
for
loop,
iteration
variable
and
you
just
write
kind
of
the
body
of
the
for
loop
in
the
function
and
that
kind
of
pi
test
will
automatically.
You
know
loop
over
the
the
possible
options
there.
A
So
in
the
parameterized
tests
you
can
kind
of,
you
can
pass
a
target
and
a
dev
parameter
if
you
want
to
sort
of
automatically
run
your
test
over
all
the
enable
targets
at
a
particular
ci
step,
and
you
can
also
write
kind
of
your
own
by
defining
kind
of
these
array.
Size
parameters,
for
example,
here
with
this
tdm
testing,
parameter
kind
of
in
in
your
unit
test,
and
so
this
so
when
you
create
this
tbm
testing
parameter
declaration
at
the
top
level.
A
I
think
this
is
this
slide
here
is
just
to
show
basically
a
final
example
of
kind
of
the
parameterized
test.
Here.
Oh,
I'm
sorry,
and
now
this
this
last
slide
here
is
about.
Sometimes,
if
you
have
a
unit
test
parameter,
you
may
want
to
actually
only
compute
this
one
so,
for
example,
if
you're
computing
sort
of
like
test
data
dependencies
like
with
a
giant
numpy
array,
you
may
want
to
actually
build
that
just
one
time.
A
Lastly,
I'm
going
to
talk
about
this
is
one
of
david's
ideas.
Is
the
idea
of
sort
of
having
a
merge,
train,
merge,
queue
or
train
workflow,
and
we
have
no
rfc
for
this.
This
is
just
an
idea
at
this
point,
but
you
know
right
now.
The
idea
is
that
right,
landing,
a
pr
involves
kind
of
running
the
tests
by
pushing
the
pr
and
jenkins,
then
someone
kind
of
merges
the
code
and
then,
lastly,
a
post,
merge
build,
runs
on
main.
A
You
know
which
may
actually
group
a
couple
different,
merge,
pr's
together
the
merge
queue
actually
kind
of
flips
this
around,
which
is
where
you,
you
upload
your
code
and
then
test
or
run,
and
then
you
know
tests.
Basically,
when
you
hit
the
merge
button,
it
essentially
enqueues,
essentially
a
set
of
tests
and
and
waits
until
those
tests
are
actually
green
and
auto
merges
at
the
end
of
that
test
run.
Guaranteeing
that
main,
then
is
is
maintained
as
green
after
the
merge,
and
so
anyway.
A
A
If
anyone
has
anything
they
wanted
to
discuss
or
or
talk
about,
we'll
open
it
up
so
yeah
manupa
says
any
thoughts
of
actually
rebuilding
the
docker
image
for
every
single
pr,
using
the
image
from
docker
hub
as
a
cache
and
you're
saying.
Basically,
so
if
someone
submits
a
pr
that
changes,
the
docker
image,
you're
saying
essentially
to
automatically
rebuild
that
and
then
just
run
it
in
the
regression.
A
B
Can
you
hear
me
yeah
yeah,
definitely
yeah
yeah,
so
I
guess
the
suggestion
was
that
yeah
we
rebuild
the
docker
image
but
use
the
existing
image
as
a
cache.
So
ideally
it
should
be.
You
know
if
there
are
no
document
changes,
but
if
there
are
docker
changes
it
will.
You
know,
build
that
and
use
that
to
test
it.
A
B
A
C
Questions
we
have
like
we
do
one
more
minute
before
we
go,
what
just
a
sort
of
a
tactical
item.
Perhaps
we
could
follow
up
on
this
discussion
and
dedicate
the
next
community
meetup
to
to
have
sort
of
a
a
full
slot
to
discuss
about
this.
I
mean
I've
got
plenty
of
questions
so,
but
I
don't
think
we
have
time
to
discuss
about
them.
Sure.