►
From YouTube: Using ONNX w/ Qualcomm powered devices from smartphones to the cloud edge and everything in between
Description
Whenever our clients target high performant AI cloud inferencing servers, create new and exciting AI based experiences on mobile phones or improve our lives by adding more and more AI features into cars, many of them use ONNX models as an interchange format. Qualcomm helps to deploy and accelerate natural language processing, computer vision, classification, segmentation, and transformer based models in various verticals: Mobile, IoT, XR, Compute and Automotive. We created a link between ONNX and Qualcomm AI Engine direct that allows us to run the same model not only on various backends such as CPU, GPU, Hexagon processor or Low Power AI subsystem of the same SoC, and migrate it to run on range of the devices due to the portability that ONNX provides. In addition to the above, we would briefly cover in this session the work we are doing with Microsoft on collaboration for ONNX RT Execution Provider for a range of our AI accelerators.
Felix Baum is responsible for AI software products at Qualcomm Technologies Inc. (QTI). Felix has spent 20+ years in the embedded industry, both as an embedded developer and as a product manager. He previously led QTI product management for Hexagon software, supporting DSPs with scalar, vector and tensor accelerators for camera, video, machine learning and audio verticals. Prior to that, he led marketing and product management efforts for various real-time operating system technologies. His career began at NASA’s Jet Propulsion Laboratory at the California Institute of Technology, designing flight software for various spacecrafts. Felix holds a Master’s degree in CS from the Cal State Northridge and an MBA from the UCLA.
A
All
right
so
with
that,
let's
get
going
I
think
that
there
were
a
number
of
presentations
that
I
listened
today,
where
folks
talked
about
different
tools
and
how
they're
using
Onyx
and
things,
and
that's
what
I'm
here
to
talk
about
as
well,
but
a
little
bit
different
I'm
not
going
to
go
into
the
models.
I'm,
not
going
to
show
you
any
tools,
I'm
just
going
to
talk
to
you
in
general,
what
we're
using
Onyx
for
and
how
do
we
collaborate
with
with
community?
Does
that
make
sense?
A
A
So
how
do
we
do
that
very
easily?
First
of
all,
if
I
look
at
the
what
our
customers
and
development
Community
has
to
deal
with
is
first
of
all,
there
are
a
ton
of
different
devices
that
are
being
built,
some
of
them,
as
I
said,
really
small,
really
tiny
footprint,
edbd
power
consumption
that
they
can
afford
to
something
that
takes
about
two
Watts
or
less
to
about
five
watt
device,
which
is
what
our
laptops
are
plus
minus.
A
Depending
on
how
Power
hungry
your
computer
is
and
I
know,
some
of
the
folks
here
from
competitive
companies
and
their
socs
are
a
little
bit
power
hungrier
than
what
we
make,
but
nevertheless
to
the
Automotive
Systems,
where
you
have
a
lot
of
compute.
That's
necessary
with
a
lot
of
cameras
and
a
lot
of
other
things
that
consumes
a
lot
of
power
and
and
even
further
than
that.
A
The
next
thing
that
we
have
to
deal
with
the
next
thing
we
have
to
deal
with
is
the
the
challenging
architectures,
because
we
have
to
support
all
of
these
different
devices.
We
have
folks
that
come
to
us
with
a
little
bitty,
itty
bitty,
tiny
noise
cancellation,
algorithms
that
have
to
fit
into
I,
don't
know
100K
of
memory
to
folks
that
use
devices
that
have
to
go
and
navigate
and
drive
a
vehicle.
So
they
have
a
much
more
complicated
models,
much
more
data
that
they
have
to
process.
A
They
have
a
plethora
of
cameras
that
at
the
same
time
taking
images
and
have
to
figure
out
how
to
drive
a
car
without
hitting
pedestrians,
and
things
like
that
and
the
models
are
different.
The
implementations
are
different
requirements
and
timing
requirements
are
different.
Lastly,
the
desired
features
believe
it
or
not,
but
we
have
customers
that
come
to
us
and
say
you
know
what
we
build
this
model
and
it
runs
on
this
device.
A
But
it's
not
enough
for
us,
because
we
want
to
run
the
same
device
or
the
same
model
on
a
different
configuration
and
the
best
example
I
can
give.
You
is
how
many
folks
still
remember
that
I
can
burn.
I
mean
my
burn
into
my
eyes
that
Apple
commercial,
with
a
guy
dancing
in
the
market,
buying
things
with
his
like
face
detection.
Anybody
remembers
that
commercial
Apple
used
to
play
it
like
non-stop
on
TV
so
four
years
ago,
plus
minus
Apple
built
into
the
iPhones
the
depth
sensor
and
what
they
did
with
the
depth
sensor
is.
A
They
said
you
know
what
we
integrate
a
camera
with
a
depth
and
using
these
two
things,
we
have
unbreakable
3D
face
authentication
and
you
can
use
it
to
actually
buy
things
using
your
cell
phone
right.
Your
children
won't
be
able
to
buy
it.
Nobody
else
can
buy
it.
Only
the
the
payment
system
will
recognize
you
as
a
your
face
and
because
they're
using
depth
and
a
camera,
they
can
make
sure
that
it's
not
a
picture
of
you
and
other
things
and
foreign
authentication
path.
That
was
three
four
years
ago
in
Mobile
phones.
A
Next
thing
you
know,
is
you
have
the
same
exact
algorithm
same
exact
capability,
but
now
folks
want
to
use
it
in
in
a
secure
cameras
where
you
walk
up
to
your
home,
you
already
have
probably
like
a
smart
door
knob
or
a
ring
one
of
those
things
that
recognize
you
they
look
at
you.
They
use
exactly
the
same
model.
Exactly
the
same
thing
authenticate
it's
you
open
the
door
for
you,
you
can
get
in
to
your
house.
Okay,
move
the
time
to
today's
states.
A
There
are
plethora
of
cameras
these
days
into
inside
Asama,
bill
right
and
outside.
So
what
they
can
do
is
the
automotive
guys
now
come
to
us
and
say
we
want
to
use
exactly
the
same
algorithm
to
exact
the
same
technology.
All
these
things
that
you're
already
certified
for
payments
and
certified
for
processing
and
unlocking
your
funds.
Now
we
want
to
have
it
in
a
car.
So
when
you
walk
up
to
your
car,
the
car
recognized
you
opens
the
door.
A
You
sit
in
the
inside
your
vehicle,
they'll
see
that
you're
licensed
an
authorized
driver
using
exactly
the
same
algorithm.
Nothing
changed
nothing
much
same
exact
model,
same
exact,
train
material
and
everything
else.
Now
they
look
at
you,
they
recognize
that
you
is
you
you
are
authorized
driver
and
in
addition
to
that,
sometimes
they
even
look
at
you
make
sure
you
know
destructed
or
drunk
or
things
like
that
and
then
they'll.
Let
you
start
the
car
and
drive
off
the
vehicle
so
scalability.
A
Thank
you.
So
the
way
we've
done
it
at
Qualcomm
is
and
I
know.
There
are
folks
that
talked
about
how
you
can
process
machine
algorithms
on
the
CPUs
good,
for
you
and
I've
heard
other
folks
talked
about
GPU
and
how
fast
this
GPU
is
and
how
awesome
it
is
that
they
can
get
better
processing.
The
way
we
look
at
it
is,
if
you
take
a
if
I
take
a
model,
I,
don't
know
Inception
V3,
maybe
something
like
that
and
I
can
run
on
a
CPU
on
a
decent
frame.
A
So
the
different
resolution
I
can
probably
get
about
I.
Don't
know
five
frames
per
second
inferences,
a
second
on
a
good
day
right.
So
you
give
me
a
picture
five
times.
A
second
I
can
look
at
the
picture
and
tell
you
there's
a
dog
in
it
in
that
picture.
Okay,
if
I
take
this
exact
algorithm
and
it's
modal,
it
was
it's
got
converted
to
onyxrt
to
Onyx
and
then
you're
running
running
on
a
CPU
I.
Take
this
model,
I
removed
it
and
said:
I
want
to
run
on
a
GPU,
much
faster.
A
A
We
take
this
exact
architecture
and
we
make
it
multi-core
if
you're
talking
about
like
Automotive
kind
of
equipment,
and
if
you
want
to
go
into
the
cloud
and
do
inferencing
at
the
edge,
we
have
a
devices
where
we
stack
up
a
whole
bunch
of
these
AI
accelerators,
paralyze
them,
and
then
we
can
do
kind
of
cloud
level
workloads
on
that
and
that's
how
we
scale
our
Hardware
but
scaling.
Hardware
is
not
good
enough
to
deal
with
that.
You
have
to
actually
scale
software
as
well,
and
these
are
just
few
things
that
add
pictures.
A
You
don't
have
to
change
the
model
we
will
handle
all
the
translations
and
all
of
the
compilations
behind
the
back.
Well,
I
already
covered
the
second
bullet
point,
the
direct
conversions
and
that
helps
them
to
scale.
It
helps
them
to
scale
across
the
capabilities
of
device.
The
same
Onyx
model
can
run
on
a
small
device,
more
powerful
device,
more
capable
device,
and
it
also
helps
to
scale
across
the
the
verticals.
A
Now
to
us,
and
that's
something
that
we
talked
about
during
the
launch:
it
saves
a
lot
of
time
to
Qualcomm
because
instead
of
me
having
a
a
paddle,
paddle,
converter
and
accelerators
and
be
able
to
accelerate
that
and
all
other
Frameworks
from
other
regions
and
geographies
I
just
have
an
onyx
converter
that
takes
Onyx
interchange
format
and
compiles
it
into
a
binary
that
executes
on
the
device.
That's
it
I
just
need
to
have
one
rather
than
dealing
with
all
these
other
Frameworks
one
at
a
time.
A
A
lot
of
those
are
folks
that
built
four
purpose
devices.
They
know
what
they're
looking
for.
They
know
exactly
how
to
they
want
to
partition
their
models
and
how
to
set
up
system
parameters
and
CPU
power
and
everything
else.
Then
we
have
a
second
type
of
customers
that
come
to
us
and
say
you
know
what
I
just
have
a
model
and
I
want
to
run
it
on
as
many
devices
as
possible.
I
don't
want
to
go
and
worry
about
specific
Qualcomm,
Hardware
I.
A
Don't
want
to
worry
about
other
things
so
to
them
they're
using
if
they
use
onyxrt,
then
that
will
kind
of
protect
them
and
and
allow
them
for
this
portability.
So
one
of
the
things
we're
doing
right
now
is
we're
working
with
Microsoft
on
onyxrt
executioner
provider
for
our
AI
accelerator,
again,
not
CPU,
not
GPU,
but
specific
AI,
accelerators
that
the
onyxrt
will
directly
delegate
to
foreign.
A
Again
to
Qualcomm
the
way
we
solve
it
is
we
put
together
a
unified
software
stack
that
allows
you
to
scale
across
the
verticals
and
across
the
tiers
and
complexity
of
these
devices.
And
how
do
we
do
that
very
simple?
First
of
all,
we
have
to
support
and
I.
Don't
know
how
about
other
vendors
that
talked
about
before
a
plethora
of
operating
systems.
I,
don't
know
if
you
guys
recognize
some
of
them.
Anybody
has
their
preference
preferred
operating
system
for
devices.
So
if
it's
a
mobile
device,
it's
obviously
Android.
A
If
it's
a
automotive
device,
things
are
a
little
bit
trickier
like
qnx
and
others
so,
depending
on
the
type
of
device
you're
using.
We
have
to
support
that
operating
system,
including
even
windows,
so
the
folks
that
haven't
used
it
I
have
right
there.
For
the
last
two
years,
I
used
windows
on
Windows
on
our
device
from
Microsoft
extremely
happy.
A
Last
me
Dale
or
more
dealing
with
system
interfaces
and
drivers
and
support
kind
of
hiding
it
hiding
all
the
underlying
hardware
and
always
the
details
from
you
and
then
what
we've
done
at
the
top
is
we
have
a
unified
software
library
that
Qualcomm
put
together
a
qualcommer
engine
Direct
and
the
reason
we
call
it
AI
engine
and
not
CPU
or
GPU
or
or
a
hexagon
AI
accelerator
is
because,
when
you
come
to
us
with
an
onyx
model,
we
will
take
care
of
routing
it
to
one
of
those
things.
We
can
dynamically
change
it.
A
A
It
detects
that
it's
next
to
the
air
in
terms
of
display,
so
we
can
take
your
air
model
and
push
it
to
the
GPU,
since
it's
sitting
there
doing
nothing
and
then,
on
top
of
it
again,
as
I
said,
we
take
Onyx
runtime
and
we're
providing
a
delegation
to
it.
That
way,
we
can
take
your
model
and
get
it
the
best
performance
that
you
guys
deserve
when
you're
using
Qualcomm
Hardware
Frameworks.
We
have
to
support,
obviously,
and
that's
kind
of
how
our
software
stack
looks
like
and
then
the
rest
of
the
tools.
A
I
don't
have
time,
but
I
would
love
for
you
guys
to
come
to
me
and
tell
me:
what
do
you
use
today
for
tools
we
have
profilers,
we
have
compilers,
we
have
analyzers,
we
have
enablement
for
the
nas
tools,
Network
architecture
search
tools.
If
you
want
to
build
your
model
from
scratch,
go
for
it.
If
you
want
to
go
and
quantize
it,
we
have
ton
of
tools
that
help.
You
helps
you
to
quantize.
A
I,
didn't
know
the
experience
of
the
audience
and
I
don't
want
to
spend
your
time.
Talking
about
all
the
intricacies
of
infor
support
versus
intake
versus
fp16
or
in
16,
and
how
much
data
throughput
and
power
it
saves.
But
if
you
would
like
to
I'll
be
more
than
happy
to
send
me
not
and
I'll
be
more
than
happy
to
take
your
questions
and
answer
those
questions
about
the
tools
or
the
underlying
Technologies.
A
A
The
smart
speaker
probably
has
ton
of
models,
but
they're
all
audio
based
translation
transcription
things
like
that
versus
XR
glasses,
where
a
lot
of
the
a
lot
of
the
it
is
Imaging
and
be
able
to
deal
with
computer
vision,
and
things
like
that,
and
that
is
my
last
slide.
Thank
you
very
much
for
giving
me
a
chance
to
talk
to
you
guys
if
you
have
any
questions,
feel
free
to
find
me
in
the
hallway
and
I'll
be
more
than
happy
to
answer
them.