►
From YouTube: Apache TVM Community Meeting, May 20, 2021
Description
* Agenda
* Introductions
* Announcements
* New TVM Plugin for GDB, written by Eric Lunderberg
* Trial run of Discord Server
* Regular µTVM Community Meeting
* AOT Roadmap (slides)
* Open Discussion on AOT
A
Okay,
everyone
welcome
to
the
may
edition
of
the
apache
tvm
community
meeting.
We
have
a
really
big
agenda
ahead
of
us
today,
so
we're
going
to
try
to
move
through
it
as
as
effectively
as
possible.
The
majority
of
the
meeting
is
going
to
be
taken
up
with
discussions
about
about
micro
tvm,
so
we're
going
to
move
through
some
of
the
earlier
agenda
items
because
I'm
sure
that
we're
going
to
take
up.
A
You
know
most
of
the
45
minutes
to
an
hour
on
on
this
topic
so
but
we
like,
but
we
like
to
start
off
every
meeting
with
introductions
and
so,
if
you're,
if
you're,
if
this
is
your
first
time
to
the
meeting
or
you're
new
to
the
apache
tvm
community,
but
please
feel
free
to
introduce
yourself
is
there?
Is
there
anyone
who
wants
to
say
hello
right
now.
A
F
A
A
Okay,
fantastic!
Well,
if
there's,
you
know,
feel
free
to
you
know
to
reach
out,
if
there's
anything
that
I
I
hopefully
you've
been
reaching
out
to
some
of
the
engineers
who
have
been
working
on
this,
but
I'm
the
developer
advocate
for
the
community
and
so
feel
free
to
reach
out
to
me.
If
you
have
any
questions
or
need
pointers
and
any
directions
to
looking
for
things
that
you've
been
working
on.
A
Okay,
anyone
else
all
right
with
that,
we'll
with
that
we're
going
to
move
on
to
some
announcements.
First
off
we
wanted
to.
I
wanted
to
have
have
eric
talk
a
little
bit
about
a
new
plugin
that
he's
written
for
tvm,
that
plugs
into
gbd
and
and
and
helps
with
helps
with
debugging
tvm
specific
actions,
including
debugging
models
and
so
eric.
If
you
want
to
take
a
few
minutes
to
describe
this
work
this
this
this
this
plugin
and
how
it
works,
that
would
be
fantastic.
D
D
D
The
difference
is
that
on
see
the
right,
I'm
going
to
source
the
a
extension
file
on
the
left,
I'm
not
going
to
everything
else
will
be
identical
between
the
two.
So
I'm
going
to
put
a
quick
say,
break
point
just
so
that
way
I
can
simulate.
Oh,
there
was
something
that
went
wrong
and
where
does
it
happen,
so
import
tvm
and
then
let's
go
somewhere
that
triggers
that
break
point.
D
I
A
A
A
But
we've
had
some
requests
also
from
from
users
who
are
in
japan
in
china.
Kind
of
you
know
in
you
know,
in
a
similar
region
to
you
know,
maybe
have
alternate
meeting
times
so
that
so
that
it's
easier
for
them
to
come
to
the
meeting,
because
you
know
typically
it's
in
the
middle
of
the
night
for
for
these
meetings
and
so
look
for
an
announcement
on
the
exact
time
that
the
meeting
is
going
to
happen.
A
If
there
are,
if
there's
anybody
who
has
specific
suggestions-
and
we
can
some
of
this
on
this
on
the
discus
channel-
also
to
to
do
this-
you
know
about
about
a
time
that
works
and
then
we'll
send
that
out
and
we'll
we'll
work
towards
adding
that
to
to
the
agenda.
Another
thing
on
here
that
I
I
didn't
quite
add,
but
it
was
also
important
to
note-
is
right.
A
Is
we,
after
a
poll
on
the
discuss
forum,
we're
going
to
do
a
trial
run
of
a
new
discord
server
for
the
apache
tvm
community?
We
talked
about
it
a
little
bit
within
within
within
discuss
and
essentially
what
this
platform
is,
is
a
synchronous
way
for
community
members
to
be
able
to
talk
to
one
another
about
tbm,
and
so
this
is
largely
about
hey.
How
do
I,
how
do
I
get?
How
do
I
work
on
something
if
you're
collaborating
with
someone?
A
You
know,
you
know,
maybe
sort
out
how
to
work
some
things
or
how
to
get
some
help
and
then
bring
those
decisions
back
to
the
community
in
the
discussion
forum.
So
please
there's
an
invitation
link
there.
I
have
a
link
to
the
initial
forum
post
about
this
and
and
and
try
it
out.
This
link
should
be
good
for
everyone
for
forever
and
if
and
if
you
do
run
into
any
problems,
feel
free
to
reach
out
to
me.
You
know
either
inside
of
the
discord
server
or
through
my
email,
which
I
will
put
down
here.
A
J
A
Yeah,
I
think
that
you
know.
I
think,
that
what
we
would
like
to
do
is
so
so
one
of
the
reasons
that
we're
looking
why
we
started
up
this.
This
discord
server
was
because
we
were
running
so
there
is
a
for
those
of
you
who
are
unaware
there
actually
is
a
tlc
pack,
slack
server
that
that
some
members
of
the
community
have
been
using,
but
we've
run
into
some
pretty
significant
limitations,
particularly
in
the
number
of
people
who
can
actually
be
invited
and
and
how
and
how
we
can
manage
that.
A
And
so
my
recommendation
is
that
discussions
that
would
be
happening
on
that
slack
server.
If
we
can
take
those
over
to
the
discord
server
and
really
try
to
see
that
community
there
and
so
favor
discord
in
you
know
over
the
slack-
and
I
think
that's
a
really
good
point,
like
you
know,
like
there's
a
micro
tvm
channel
in
there,
that
people
are
talking
that
people
are
talking,
and
so
we
need
to
create
a
micro
tvm
channel
within
that
server.
I
I
think
to
answer
your
question
more
yeah.
A
lot
of
the
frustration
with
the
slack
is
that
we
tried
to
make
it
more
open,
but
without
paying
an
incredibly
large
amount
of
money
to
slack,
it's
really
hard
to
get
all
the
features
out
of
it,
and
so
I
think,
in
the
old
days
a
lot
of
us
were
meeting
in
person,
and
you
know,
like
people
weren't
using
slack
that
much,
but
it
seems
like
as
we're
growing
across
companies,
it's
really
important
to
communicate
for
just
some
development
synchronization.
I
A
A
K
Yeah-
and
so
you
know,
one
thing
is
like
you
know:
we
currently
have
the
slack
and
we're
calling
this
a
trial
period
of
the
discord
server.
So
let's
all
move
to
a
micro
channel
on
the
discord,
server
and
try
stuff
out
again.
You
know
for
for
anything,
that's
kind
of
a
material
decision
that
we're
gonna
make
within
the
tbn
community.
We
we
can't
make
it
on
either
one,
so
we
wanna
make
that
those
on
the
discuss
forum.
So
hopefully
it
shouldn't
be
too
big
of
a
deal.
K
L
M
M
A
Are
there
and
aren't
at
the
meeting
understand
that
we're
doing
that
migration?
I'm
also
going
to
be
posting
the
link
to
the
discuss
forum
later
on
too
so
that
it's
more
widely
available,
so
that
anyone
who
sees
it
in
the
discuss
forum
can
can
join
the
can
join
the
group.
Now,
as
with
any
of
these
other
community
platforms,
when
you
post
a
public
link
like
that,
we
also
want
to
make
sure
that
everyone
in
the
community
feels
like
feels
like
they're
safe
and
that
they
and
that
and
that
this
is.
A
A
A
M
Sure,
well,
we'll
just
keep
it
short
and
sweet.
I
think
that's
probably
the
most
important
thing,
so
you
know,
as
velocity
has
picked
up
in
microtvm
development
and
sort
of
facilitating.
What's
all
going
on,
you
know
in
the
community,
it
sort
of
makes
sense
that
we
should
get
together,
maybe
on
a
more
regular
basis,
sort
of
having
a
little
side
huddle.
That's
about
micro,
tdm,
but
of
course,
with
the
provision
that
again
it's
the
discuss
forum
that
counts
when
it
comes
to
business
and
along
those
lines.
M
So
really
this
is
a
you
know,
just
meant
to
be
a
way
to
you
know
help
what's
going
on
already
with
with
with
micro
tvm,
as
speed
has
picked
up
on
it.
So
we've
got
all
the
information
in
here
and
we'll
we'll
try
things
out
as
an
experiment,
and
we
are
looking
at
a
two.
You
know
bi-weekly
cadence,
we'll
dial
that
up
or
dial
that
back
depending
on
how
well
things
work
out.
So
maybe
you
know
monthly
is
more
that
the
the
correct
cadence
will
again
make
adjustments
along
the
way.
M
As
far
as
the
meeting
time,
I
just
picked
a
random
time.
That
would
try
to
appeal
to
a
good
set
of
time
zones.
I
realize
it
is
not
best
for
those
people
on
the
west
coast
of
the
united
states
and
it's
terrible
if
you're
in
china.
So
you
know,
as
you
sort
of
alluded
to
earlier
in
the
meeting
chris,
you
know.
Maybe
we
need
to
do
something
that
flips
back
and
forth
with
something
that
would
be.
M
H
M
Yes,
yeah,
so
micro
tvm
intends
to
make
use
of
the
aot,
so
I
think
it
makes
sense
that
you
can,
you
know,
sort
of
mix
those
two
things.
I
wouldn't
be
surprised
from
time
to
time
that
there
will
be
other
topics
that
drift
in
because
micro
tvm,
you
know
in
some
way
interacts
with.
You
know
some
feature.
That's
you
know
in
the
larger
framework,
you
know
we'll
just
sort
of
let
those
things
organically
go
as
they
need
to
make
sense.
A
Okay
and
and
also
reminding
everyone
too,
that
that
we
are
going
I'm
going
to
be
setting
up
a
new
apache
tvm
organization
that
will
allow
us
to
share
some
of
these
resources,
particularly
the
calendars
for
this
and
so
watch
for
the
announcement
on
the
discuss
forum
about
the
new
calendar
and
you
should
be
able
to
subscribe
to
the
calendar,
and
my
intention
with
the
calendar
is
to
make
sure
that
we
put
all
of
the
public
tv
events
onto
that.
And
so
that's
going
to
include
the
monthly
community
meeting.
A
And
so
you
know,
and
that's
and
a
big
part
of
the
reason
that
we
want
to
make
this
a
community
owned
resources
so
that
folks
in
the
community,
who
are
planning
events
like
say
that
we're
planning
local
events
or
planning
you
know
they
want
to
have
like
local,
meetups
or
or
other
things
that
are
going
on.
We
have
a
place
for
community
members
to
add
that
and
be
able
to
broadcast
it
to
the
wider
community
in
kind
of
a
a
you
know,
a
repeatable
and
discoverable
way.
K
I
figured
what
we
could
do
for
this
is
I
I
think,
there's
been
a
lot
of
discuss
posts
and
forum
posts
and-
and
I
myself
have
even
at
times
become
confused
about
just
the
number
of
things
in
flight.
So
I
thought
to
start
with
the
discussion
here.
We
could.
I
threw
together
just
like
a
couple
of
slides
to
kind
of
just
give
an
overview
of
what
we've
been
working
on
some
of
the
the
some
of
the
work.
K
That's
been
kind
of
done
and
some
of
the
work
that's
outstanding,
and
then
you
know
we
can
kind
of
talk
about
whatever
people
feel
like
is
a
concern
from
here.
I
know
a
number
of
the
contributors
to
aot,
or
maybe
even
all
of
them
are
on
the
call
here.
So
if
you
have
questions
for
for
them,
I've
certainly
been
serving
in
more
of
a
reviewer
role.
For
for
this,
and
so
so
kind
of
we
have.
K
You
can
ask
me
questions,
you
can
ask
them
questions
and
we
can
kind
of
go
from
there.
So
let
me
give
a
quick
overview
of
that
and
then
we'll
kind
of
open
it
up
to
to
discussion.
After
that,
so
I
thought
I
just
would
talk
about
like
what
is
the
aot
effort
in
case
people
here
have
not
kind
of
engaged
with
it.
What
are
the
sub
projects
and
then
kind
of
what
are
the
rfcs?
K
K
But
but
aot
is
an
effort
within
tdm
to
to
generate
basically
glue
code
and
so
that
that
runs
kind
of
all
the
pieces
without
a
need
for
like
a
runtime
library
to
do
that.
So
let
me
explain
a
little
bit
more
so,
like
I
said,
we're
going
to
compile
a
model
and
and
run
it
and
isn't
basically
compiling
the
model
and
calling
the
pieces
what
we're
already
doing.
Not
quite
what
we
do
is
we
break
things
into
sorry?
K
I
think
I
skipped
this
should
just
skip
this
to
this
slide
before
we
break
the
program
into
schedule,
pieces,
there's
sort
of
one
function
for
each
piece,
and
I've
kind
of
illustrated
that
with
this
graph
at
the
right
here-
and
we
export
basically
three
things.
We
export
kind
of
the
generated
operator
implementation
for
each
piece,
so
you've
got
in
this
graph.
You've
got
com2d
bias
ad.
That
was
from
our
relay
model
before,
as
well
as
max
pool
2d.
K
We
then
have
like
a
graph
that
explains
how
to
link
together
the
pieces
as
well
as
how
the
data
dependencies
flow
sort
of
between
the
inputs.
So,
for
example,
if
you
want
to
run
this
comptonity
bias
ad,
the
graph
explains
that
you
need
to
supply
an
input
called
com2d
input
and
a
p1
which
is
a
parameter
and,
and
then
the
output
is
going
to
be
placed
in
this
intermediate
buffer
here,
and
so
these
three
outputs
here
are
what's
currently
generated.
K
K
You
know
what
why
do
we
need
to
to
rely
on
a
library
for
this,
and
the
answer
is
that
we
we
don't.
We
could
do
this,
and
so
I
made
this
slide
to
basically
kind
of
show,
roughly
the
tvm
compiler
architecture-
and
you
can
see
here-
I've
got
the
outputs
written
here
at
right
on
the
right
here.
We
get
simplified
parameters,
we
get
an
operator
graph
and
we
get
the
generated
operator.
K
Implementations
and
the
thing
that
we
wanted
to
to
do
in
aot
then
is
take
this
operator
graph
and
somehow
output
a
function
that
implements
it
and
so
in
the
aot
world.
What
we
do
is
we
actually,
instead
of
outputting
this,
we
feed
it
forward
into
a
new
pass
that
then
creates
sort
of
a
top-level
function
that
that
lives
alongside
all
of
the
the
operator
pieces
and
and
then
this
whole
thing
becomes
kind
of
the
code
generated
output
from
tvm.
K
Let
me
make
sure:
oh
I
see.
Okay,
I
think
I
let
me
this
is
the
example
slide.
I
wanted
to
show
you
guys
sorry.
This
is
not
quite
as
polished
as
I
wanted
it
to
be,
but
just
to
give
you
an
example
of
what
this
top
level
function
looks
like
people
that
have
been
working
with
tvm
for
a
while
are
kind
of
familiar
with
this
function.
Signature
at
the
top.
K
This
is
our
packed
c-function
function,
signature,
and
so
all
of
our
generated
functions
are
invoked
using
this
signature
here
and
so,
but
you
can
see
that
this
function
here
is
not
actually
an
operator
function.
It's
a
the
top-level
glue
function.
That's
meant
to
call
all
the
operators
in
the
graph
kind
of
in
the
order
that
reconstitutes
the
original
model
and
so
to
do
this.
There's
a
couple
challenges
one.
K
K
It's
then
got
to
assemble
kind
of
a
call
stack
that
it
can
use
to
call
sub
functions
with
with
that
same
signature,
and
then
it's
got
to
call
operator
functions
in
order,
and
I've
just
shown
one
example
of
a
function
call
here
you
know
in
in
the
true
implementation.
Of
course,
you
can
expect
kind
of
these
last
two
blocks
to
repeat
themselves
several
times,
as
as
it
works
its
way
through
the
model,
and
you
can
see
in
calling
this
operator
function
effectively.
K
K
So
in
doing
this,
this
basically
allows
us
to
get
rid
of
the
need
to
have
a
large
runtime
library,
because
if
you
remember
that
runtime
library,
the
graph
executor
that
is
basically
was
supposed
to
parse
json
reconstitute
the
call
graph
and
then
do
all
this
code
and
what
we've
done
is
we've
effectively
codified
this
in
the
output
of
tvm
here
so
okay.
I
hope
that
was
somewhat
comprehensible
too
to
everyone
here,
but
that's
kind
of
what
is
aot.
K
I
want
to
say
one
thing
about
this
too,
and
that's
that
a
lot
of
the
aot
effort
has
been
kind
of
started
under
the
micro
tvm,
guys
and-
and
I
think
that
micro
tvm
in
particular
has
quite
a
bit
of
interest
in
the
aot
project,
because
resources
are
pretty
constrained
on
microcontrollers,
but
we
don't
intend
to
nothing.
We've
been
presented
so
far
is
particularly
specific
to
microcontrollers,
so
in
developing
the
aot
project.
K
K
I
guess
in
we
intend
to
make
this
this
core
part
of
the
aot
effort,
more
broadly
reusable
across
the
tdm
code
base,
and
so
particularly
if
there
are
people
on
the
call
here
that
are
interested
in
non-embedded
applications
or
applications
outside
the
tvmc
runtime.
You
know
we'd
especially
love
to
hear
any
thoughts
or
concerns
from
from
your
site
as
well.
K
Okay,
so
I
wanted
to
talk.
This
is
a
fairly
large
effort,
and
so
we
kind
of
have
an
initial
rfc
about
basically
well
a
number
of
things,
but
what
we
wound
up
doing
was
was
merging
a
core
piece
and
then
you
know
we
kind
of
expect
several
pieces
to
follow
this,
and
this
is
by
no
means
a
comprehensive
list.
K
I'm
sure
there
will
be
additional
pieces
as
we
continue
developing
the
effort,
but
just
to
kind
of
give
everyone
kind
of
a
brain
dump
from
me
of
where
I
think
we
are,
and
if
anyone,
if
I've
forgotten
something,
please
let
me
know
so.
The
first
thing
we've
first,
I
guess
project
or
rfc
this.
These
all
roughly
correspond
to
rfcs
the
first
rfc
or
project
that
we've
landed,
is,
is
basically
making
a
tensory
hour
function
to
mimic
this
graph
executor
run.
K
Ultimately,
it
would
be
great
if
we
could
materialize
these
on
the
discuss
forum,
but
just
to
give
you
an
idea
of
kind
of
from
my
mind
where
we
are
you
know,
so
we
have
stuff
needed
by
this
kind
of
broader
non-firmware
or
non-embedded
effort,
and
and
for
that,
that's
basically
implementing
something
that
replaces
the
runtime
api
that
you
see
with
the
graph
run
runtime
today.
We
call
that
the
module-based
model
runtime
api.
If
you
search
for
that
on
the
forum,
you'll
find
a
pretty
lengthy
rfc.
K
That
kind
of
goes
through
current
consensus
and
and
which,
of
course,
we
can
always
change
going
forward
of
you
know
what
should
it
look
like
to
load
and
run
a
model
with
tvm?
So,
that's
you
know
what
function
do
we
call
to
allocate
memory?
What
function?
How
do
we
set
a
parameter
and
how
do
we
drive
inference
and
get
the
output?
K
There
are
a
bunch
of
ideas,
basically
from
the
implementers
of
of
aot
around
basically
reducing
stack
usage,
you'll
notice
that,
in
this
example,
function
here
we're
allocating
a
bunch
of
things
on
the
stack
and
I've
actually
pruned.
This
code
example
down
quite
a
bit,
there's
a
bunch
of
other
things.
I
also
allocated
on
this
act
here
as
well.
K
All
of
this
kind
of
conspires
to
really
blow
up
the
stack
usage
requirements
on
embedded
systems,
which
is
kind
of
a
no-no,
and
so
one
of
the
things
that
we
do
is
allocate
dl
tensor
instances
on
the
stack
and
in
the
embedded
world.
You
know
we'd
like
to
kind
of
avoid
that,
since
for
the
most
part
we
just
need
the
data
member.
So
that's
one
thing
in
flight.
There's
an
rfc
out
about
that.
We'll
talk
about
that
in
a
second
there's.
Another
effort
is
basically
to
if
you
notice
this
function.
K
Signature
here
is
not
the
most
friendly
thing
for,
like
a
c
firmware
programmer.
Of
course
it's
like
a
c
function
interface,
but
specifically,
what
are
the
arguments?
What
order
do
you
present
them
in?
You
know,
there's
no
documentation
that
you
would
that
you
would
kind
of
normally
expect
of
a
tbm
function
and
there's
also
a
bunch
of
sort
of
extra
arguments
in
a
sense
that
out
return
the
return,
value
and
return
type
code
arguments
are
are
typically
are
actually
typically
contained
in
the
args
here.
K
I
believe
I
think
I
said
that
right.
So,
basically,
there's
just
a
lot
of
a
lot
of
pieces
here
that
that
could
be
minimized
and
so
there's
some
interest
in
creating
basically
a
smaller
api
that
doesn't
rely
on
this
packed
function,
type,
signature
and
and
can
also
just
kind
of
be
more
oriented
towards
embedded,
embedded
development.
So
doing
things
like
including
model
metadata
is
part
of
this.
K
K
Okay,
so
that's
what
I
wanted
to
talk
about
just
to
kind
of
give
a
brief
overview.
I
hope
that
was
somewhat
informative
and
not
super
confusing.
We
have
some
of
the
folks
from
that
are
implementing
the
rfc.
I
think
manupa
and
giuseppe
are
on
the
call.
I
don't
know
if
you
guys
had
anything
you
wanted
to
bring
up
or
or
talk
about
or
feedback.
You
were
interested
in
and
I
guess
just
wanted
to
open
it
up
for
for
a
discussion.
N
K
In
a
sense,
the
the
folks
from
sd
micro
on
microelectronics
have
kind
of
up
pushed
a
pr
that
shows
basically
their
kind
of
a
demonstration
of
how
to
integrate
tbm
with
their
embedded
api,
and
I
think,
there's
a
lot
of
really
good
things
there.
That
kind
of
were
were
built
kind
of
from
kind
of
a
firmware
developer's
perspective.
K
So
there's
a
lot
of
there's
metadata.
If
you
need
to
write
reusable
functions
to
pre-process
model
input,
a
lot
of
the
mechanisms
for
getting
input
and
output
are
pretty
well
thought
through.
For
you
know,
in
terms
of
a
world
where
you
know,
the
application
developer
wants
to
be
able
to
kind
of
manage
all
the
the
memory
on
the
system,
and-
and
so
you
know,
kind
of
one
of
the.
K
H
Yeah,
so
I'm
not
sure
if
this
really
fits
hundred
percent.
Your
question
so
my
question
kind
of,
is
I
I
I
think
you
covered
the
right
point.
So
number
one
would
be
for
me
getting
rid
of
the
dl
tensors
and
the
tm
values
in
the
in
a
generated
code.
I
think
this
is
not
a
not
a
large
issue.
So
the
second
point:
what
would
be
interesting
is
do.
K
Yeah,
that's
a
good
question.
I
mean
we've
talked
about
this
a
little
bit
too,
with
I've
been
talking
about
this
with
manufa
and
others
on
the
the
arms
side,
who
have
been
kind
of
implementing
a
lot
of
these
operations
themselves,
and
I
know
that
you
know
memory
memory.
Planning
is
kind
of
one
of
the
big
concerns
here
and-
and
there
is
kind
of
a
question
of
what
we
want
to
do
is
get
away
from
a
world
where
we
have
this
sort
of
dynamic
memory
interface.
K
You
know
sort
of
a
mallet-like
interface.
If
we,
if
we
have
it,
you
know
at
the
best,
it's
like
we'd
like
to
sort
of
pre-plan
every
mallet
call,
so
that
we
can
target
it
to
a
sort
of
a
predefined
memory
location,
and
I
think
what
the
the
the
aot
implementation
today
does
is.
There's
actually
like
an
alternate
stack
based
memory.
Allocator.
K
You
can
pick
that
sort
of
as
long
as
your
model
is
feed
forward
and
straight,
and
then
all
of
the
memory
allocation
should
be
first
and
last
out,
just
like
a
stack,
and
so
you
can
use
this
stack
memory
allocator
with
kind
of
very
little
waste
and
so
kind
of.
How
does
this
all
relate
to
the
original
question
here,
which
is
basically
like?
K
Should
we
allow
these
operator
kernels
to
to
invoke
malik
basically
for
scratch
pad
memory,
and
I
think
that
kind
of
one
of
the
proposals
that
has
been
pushed
so
far
is:
can
we
implement
basically
a
tir
pass?
That
then
goes
into
the
generated
kernels
and
hoists
up
all
of
the
allocations
into
the
main
function?
So
basically
this
would.
K
This
would
look
like
basically,
each
each
of
these
scratch
pad
buffers
would
be
passed
in
as
another
argument
sort
of
a
magically
added
argument
to
each
operator
kernel
and
then
within
the
main
function.
You
know
that
kind
of
gets
to
the
level
of
user
interface
and
kind
of
graph
level
memory
planning
and
then
there's
options.
You
can
either
access
a
set
of
memory,
pools
and-
and
you
know,
use
the
memory
planner
to
target
those
areas
or
you
can.
O
O
So
yeah,
as
andrew
mentioned,
we
are
currently
kind
of
using
a
lightweight
stack
allocator,
because
that's
what
tier
code
could
get
lowered
to
which
has
this
lifo
pattern,
but
yeah
we
have
planned.
We
replace
that
with
user
provided
workspace
buffer,
so
the
idealizing
the
in
the
application
itself,
they
would
be
able
to
pin
them
into
particular
memories.
K
I
guess
implementation
and
the
interface
currently
there's
kind
of
a
implicitly
user-facing
aspect
of
the
tvm
graph
memory.
Planner,
that's
called
it's.
Basically,
how
do
we
identify
a
memory
region
where
a
sensor
is
going
to
live
and
the
tbm
graph
memory
planner
currently
works
by
basically
looking
at
all
the
memory
allocations
that
are
visible
at
the
graph
level
and
currently
that
doesn't
include
scratch
pad
buffers.
K
I
think
that
the
the
first
step
in
that
direction
is
basically
to
make
the
the
storage
identification
process
kind
of
a
first
class
user
level
concept
and
and
so
kind
of
moving
forward.
I
think
that's
that's
one
thing
I
want
to
do.
I
see
giuseppe's
raising
his
hand,
and
I
don't
want
to
ramble
on
about
this,
so
go
ahead.
N
No
yeah,
I
would
just
make
sure
that
there
is
a
another
pr
that
I
just
pushed
like
40
minutes
ago.
So
you
you,
you
didn't
list
here
for
obvious
reasons
that
he's
trying
to
decouple
the
couple
aot
from
the
the
memory
planner
just
described
so
because
the
the
the
relay
planet,
the
graph
memory
planner
in
tdm
as
the
problem
that
it
treats
the
output
buffer
as
a
temporary
as
well.
So
this
is
bad
because
it
the
the
email,
the
output
buffer,
is
provided
by
the
user.
So,
yes,
you
know.
D
N
Cannot
share
it,
you
cannot,
you
can
share
it,
but
you
cannot
change
the
size
of
the
buffer.
Basically,
you
cannot
expand
the
buffer,
that's
the
problem.
So
there
is
an
issue
raised
and
in
order
to
solve
this
issue,
we
decided
as
a
first
move
in
the
direction
of
memory
planning
and
to
to
do
a
in
aot
memory
planning.
We
we
try,
we
are
trying
to
to
decouple
the
the
running,
the
the
airt,
the
the
uft,
the
ut
pass,
the
ulti
function
from
the
tvm
graph
memory
planner.
N
N
K
K
Yeah,
I
was
just
going
to
to
speak
a
little
bit
more
about
this
issue
you
raised,
which
was
just
like
a
thing
that
mood
had
ran
into
the
other
day
and
and
previously
one
thing
that
the
kind
of
module
based
model
runtime
interface
implies.
Is
that
there's
a
tvm
library,
that's
managing
all
the
memory?
K
And
so,
if
you,
you
know,
if
you
want
to
change
the
way
that
the
memory
is
allocated,
it's
kind
of
buried
in
kind
of
shared
tdm
library,
code
and,
and
so
you
kind
of
have
to
change
that
code
and
as
essentially
as
we're
moving
into
this
microcontroller
world.
We're
gonna
have
to
change
the
paradigm,
at
least
so
that
you
know
potentially
there's
memory
blocks
that
that
tvm
wants
to
generate
for
you.
So,
for
example,
it
knows
it
needs,
say
32
kilobytes
of
cpu
facing
ram.
K
It
should
provide
you
some
some
con
construct
that
you
can
use
to
easily
allocate
that
memory,
whether
it's
a
defined
constant.
That
says
how
big
the
cpu
facing
ram
is,
or
a
struct,
or
something
like
that.
That
contains
it,
and
in
particular
why
this
is
important.
Is
the
thing
that
that
giuseppe
just
raised
with
the
aot
api?
K
The
input
buffers
were
kind
of
assumed
to
be
kind
of
don't
touch
these,
but
the
the
graph
memory
planner
then
for
for
all
following
buffers
in
the
graph
we'll
try
to
reuse
them.
So
if
there's
an
intermediate
tensor
that
lives
for
exactly
that,
just
needed
as
a
a
go-between
between
two
calls
that
are
subsequent
and
then
later
you
know,
there's
another
intermediate
buffer.
It
will
try
to
use
the
same
memory
for
those
two
intermediate
buffers
and
that's
great.
K
K
G
Oh,
I
also
hit
this
issue.
I
don't
know
if
you
guys
noticed
in
that
copy
to
from
api
there's
a
new
dl
tensor
overload,
which
you
can
extract
the
shape
of
the
output
tensor,
and
so
then
you
can
look.
You
can
grab
out
the
slice
of
the
output
storage
that
you're
needed
that
you
need
for
that
particular
output
tensor.
So
you
shouldn't
have
to
copy
out,
like
the
full
storage
pool,
you'd
only
just
cut
the
copy
of
that
slice
right.
So
I
think
that
that
should
solve
that
issue.
K
That's
great
yeah,
yeah
well,
we'll
have
to
see
if
we
can
kind
of
mirror
all
those
kind
of
analogies
across
the
interfaces
there
so
kind
of
looking
forward.
I
guess
there's
there's
just
five
minutes
left,
I
guess
is
there?
Is
it
am
I
missing
anyone
that's
raising
their
hand
or
has
anyone
wanted
to
say
anything
that
I'm
missing.
P
Yeah
hi,
this
is
michael
vogelseven
you're
from
one
thing
that
I
would
like
to.
I
mean
I'm
very
interested
by
this
activity,
but
not
for
necessarily
the
full
microcontroller.
What
I'm
interested
in
particularly,
is
limiting
the
current
runtime
that
we
have
with
tvm,
especially
when
models
are
run
within
a
streaming
environment.
Let's
say,
for
example,
when
you
want
to
put
a
model
generated
by
tvm
as
a
filters
inside
gstreamer.
P
Currently,
if
you
do
that,
the
runtime
actually
that
gen,
that
tvm
generates
serialize
everything
and
it
creates
quite
a
problem
actually
with
gstreamer
and
probably
some
other
libraries.
So
I
see
this
work
that
you
are
doing
as
actually
a
bit
a
bit
higher
than
the
mcu.
You
know
a
bit
bigger
devices,
but
to
reduce
the
runtime
to
be
really
self-contained
and
that
can
be
put
inside
a
streaming
environment
like
gstreamer
and
others.
P
That
would
be
absolutely
awesome,
and
here
you
got
the
same
problem
you
mentioned
with
the
memory
that
is
allocated
by
the
external
application.
Like
a
g
streamer.
You
know
that
is
going
to
be
passed
into
this
function,
generated
by
gstream
via
tv
and
the
output
that
is
going
to
other
filters
so
to
have
it
well
self-contained
with
the
memory
management
and
the
multi-threading
management.
You
know
that
would
be
generated
outside
of
the
of
the
tvm
itself
generated
function
is
act
very,
very
paramount.
P
I
mean
I
have
done
this
in
in
the
past
and
unfortunately
I
had
to
change
completely
the
runtime
itself
that
the
tvm
generates,
because
otherwise,
as
I
said,
it's
just
impossible,
everything
is
serialized.
You
got
plenty
of
clash
with
memories.
You
have
to
duplicate
memory
all
the
time,
so
that
would
be
really
helpful.
I
mean
I
see
this.
The
work
that
you
are
doing
guys
with
aot
has
a
really
great
duration.
K
P
What
I
mean
by
cellular
is
serializing
the
runtime.
Sorry,
maybe
I
was
wrong
here.
What
I
mean
is
what
happened
with
the
tvm
runtime
currently
is
instead
of
running
in
parallel.
You
know
with
gstreamer
and
the
filter
that
generated
by
that
is
wrapping
the
code
generated
by
tvm,
the
tvm
runtime
that
you
need
to
start
within
this
filter
to
run
the
tvm
code,
actually
serialize
everything,
that's
the
one.
P
It
so
you
end
up
with.
Actually,
you
know
a
big
slowdown
in
performance
when
you
are
using
g
streamer,
if
you
don't
change
it,
and
that
is
really
purely
because
of
the
runtime.
The
way
it
is
today,
so
removing
it
making
it
as
as
pluggable,
let's
say
to
other
framework,
would
be
actually
really
really
a
good
thing.
Great.
K
K
You
know
quantization
aot
memory
planner
a
bit
of
these
meetings
as
well,
and
so
I
I
just
like
to
say,
I
think
what
we'll
do
is
we'll
post
up.
You
know
agenda
items
and
if
you
know,
I
think
the
focus
of
the
the
meetup
should
definitely
be.
You
know
firmware
oriented,
but
if,
if
people
want
to
attend,
you
know
just
to
kind
of
learn
what's
going
on
in
the
firmware
world
or
you
know
raise
concerns
about.
K
K
A
H
K
Yeah,
so
we're
still,
I
think
that
we
definitely
do
want
to
create
an
interface
that
has
kind
of
here's
what
you
get,
and
I
think
the
idea
is
what
you
would
provide
the
memory.
Planner
is
basically
a
tir
program,
basically
a
let's
see
an
ir
module
containing
tir
code,
and
then
you
know
kind
of
here's.
What
the
interface
is
expected
to
produce
and
and
that
kind
of
is
those
that's
basically
the
the
first
class
storage
identifier.
K
But
whether
or
not
that
means
I
think
we
would
probably
still
try
to
provide
a
more
flexible
implementation
that
can
kind
of
be
retargeted
for
for
small
changes
in
memory
planning
and
then,
if
there's
something
trying
to
do
something
really
crazy,
you
know
you
can
you
can
reimplement
yourself
as
as
you
need
to
so
that's
that's
kind
of
my
view
and
we'll
definitely
talk
about
that
more
in
the
micro,
tv
and
meetups
going
forward,
because
I
think
that's
related
to
things
like
accelerator
offloading
and
things
like
that
as
well.
So.