►
From YouTube: Deploying on Desktop with ONNX
Description
ORT provides the foundations for inference for Adobe's audio and video products (Premiere Topaz Labs develops deep learning based image quality software for professional and hobbyist photographers, which means running on the user's desktop or laptop. ONNX is an essential part of our solution to producing consistent results while making the most of a variety of consumer hardware. This type of deployment poses unique challenges and opportunities. Some experiences in this task have driven us to adopt certain useful strategies, tools, and techniques. Others remain interesting avenues for future improvement.
Alexander Zhang is a software developer at Topaz Labs, primarily responsible for Gigapixel AI and the inference pipeline for image models.
A
I'm
alexander
zhang
I'm
a
software
developer
at
topaz
labs,
which
is
what
could
be
described
as
an
image
quality
company
gigapixel
is
the
product
of
ours.
I've
worked
on
the
most.
It's
a
desktop
app
that
lets
people
with
no
knowledge
of
ai
or
programming
use,
super
sampling
models
to
enlarge,
say
old
photos
to
the
size
needed
for
a
poster.
A
A
That
means
we
need
to
a
way
to
specify
the
same
model
to
many
different
libraries
and
that's
exactly
what
we
use
on
an
x4
after
training
models
in
tensorflow
flow
or
pi
torch.
We
convert
each
through
onan
x
to
create
a
specification
of
the
model
architecture
that
can
be
used
by
each
of
the
different
inference.
Libraries
that
way
we
can
provide
a
relatively
consistent
result
on
different
hardware,
while
saving
a
great
deal
of
manual
work
on
each
model.
A
A
One
example
is
that,
in
many
of
our
older
models,
we
found
that
in
some
libraries,
the
output
of
the
model
was
shifted
by
a
pixel
and
on
some
the
result
was
blurrier
than
on
others.
It
turned
out
that
those
models
included
bilinear
resize
operations
and
when
they
were
converted
using
onan
x,
offset
nine.
I
believe
the
alignment
of
the
resize
was
not
specified
and
this
led
to
different
backends
implementing
it
differently.
A
Some
degree
of
inconsistency
is
inevitable
when
using
different
sets
of
used
operations
or
intermediate
accuracy,
but
it's
not
necessarily
easy
to
qualify,
quantify
what's
acceptable
to
users
and
finally
performing
and
troubleshooting
all
the
library.
Specific
conversions
may
require
getting
your
model
to
a
lot
of
different
machines.
Just
during
development.
A
A
Getting
consistent
quality
and
speed
from
desktops
isn't
just
a
conversion.
Time
issue
there's
also
a
lot
of
tasks
in
the
runtime
inference
pipeline
that
we
must
do.
Besides.
Just
calling
a
library
we
need
to
determine
which
models
can
actually
run
on
the
user
system
and
select
the
appropriate
configuration
things
like
numerical
precision,
input,
format
and
inference
device
for
our
use.
A
To
address
compatibility
and
selection
without
having
to
actually
test
every
library
at
runtime,
which
would
be
slow
and
lead
to
unpredictable
crashes
for
some
libraries
at
the
time
of
conversion,
we
also
write
a
json
file
describing
the
system,
requirements,
memory,
usage,
input
names
and
other
library
specific
details
for
each
variation
of
a
model.
This
can
be
matched
against
system.
Information
found
at
runtime
to
determine
compatibility
selection
is
based
on
preferring
the
discrete
gpu,
with
most
vram
and
using
the
library
with
the
most
specific
requirements
compatible
with
it.
A
This
doesn't
guarantee
the
best
speed
but
works
well
in
99
of
cases
to
handle
arbitrary
input.
We
split
images
into
blocks,
select
an
appropriate
size
based
on
vram
and
then
run
the
blocks
through
inference.
This
allows
us
to
maximize
throughput
through
batching
and
potentially
run
on
multiple
devices
and
libraries
in
parallel.
A
In
the
future,
some
concerns
are
how
to
fit
larger
models
onto
consumer
gpus,
despite
lower
vram
availability
and
the
perhaps
related
problem,
how
to
change
between
different
models
quickly
when
training
or
performing
comparisons.
There's
also
continuing
difficulty
in
ensuring
new
model.
Architectures
can
be
added
and
behave
well
across
all
libraries.
The
fact
that
we
use
more
than
one
means
it
can
take
a
lot
of
work
and
time
for
some
newly
introduced
layer,
type
or
operator
to
be
implemented
in
all
of
them
and
the
bugs
ironed
out
the
result
of
all
the
conversion.
A
Work
can
be
difficult
to
handle
with
hundreds
of
files
for
different
models
and
configurations.
It
can
be
easy
to
miss
one
with
the
wrong
permissions
or
a
quality
issue
and
just
moving
them
around
takes
a
long
time.
We
are
hopeful,
though,
that
the
general
strategy
we've
taken
so
far
will
be
up
for
these
challenges
and
enable
us
to
frequently
deliver
consistent
image
quality
improvements
to
get
the
best
speed
out
of
all
kinds
of
desktop
systems.