►
Description
In many companies, Java is the primary language for the teams to build up services. To have ONNX model onboard and integration, developers faced several technical challenges on the resource allocation and performance tuning. In this talk, we will walk you through the inference solution built by DJL, a ML library in Java. In the meantime, we will share some customer success stories with model hosting using ONNXRuntime and DJL.
Qing is a Software Development Engineer in AWS. He has been working on several challenging products in Amazon, including high performance ML inference solutions and high performance logging system. Qing’s team successfully launched the first Billion-parameter model in Amazon Ads with very low latency required. Qing has in-depth knowledge on the infrastructure optimization and Deep Learning acceleration. Qing is also a PPMC of Apache MXNet
A
Thanks
very
much
for
joining
my
my
name
is
jinglan,
I'm
one
of
the
software
engineer
at
amazon
web
services,
and
today
I
would
like
to
introduce
you
with
the
dvd
library
integration
with
onyx
runtime.
Basically,
we
have
a
few
kind
of
the
use
cases
that
are
alarming
inside
of
amazon
and
also
outside
of
amazon
that
have
been
successful,
proving
that
the
onyx
onyx
runtime
actually
have
a
very
good
performance.
A
A
Okay,
so
here
I
would
like
to
share
with
you
guys
about
like
a
cup
of
the
successful
customer
by
leveraging
awnings
and
also
on
its
runtime
in
in
by
by
leveraging
and
also
like
with
ddl,
you
use
along
the
way.
The
first
use
case
is
basically
from
came
from
amazon
advertising.
We
have
been
helping
with
the
teams
that
are
trying
to
launch
a
model
that
with
a
very
heavy
infrastructure
and
basically
previously
the
latency
in
in
advertising
that
was,
that
does
not
meet
the
the
actual
infrastructure.
A
That's
basically
the
number
on
the
p90
and
also
like,
after
switching
to
onyx
runtime,
we
can
also
providing
the
the
stress
saved
multi-threading
inference
by
leveraging
the
multi-core
cpus
on
the
other
side.
After
doing
the
conversion
between
the
magnetonics,
we
we
do
the
correction
test
to
ensure
that
the
outputs,
after
the
conversion
is
still
being
the
same
and
we
observe,
is
less
than
e
to
the
minus
seven
difference
on
the
floating
numbers.
A
A
So,
after
converting
to
to
onyx
the
the
inverse
time,
are
re
reduced
from
12
milliseconds
to
to
less
than
four
milliseconds,
and
since
deep
jaw,
library
is
supporting
multiple
framework
they're,
also
converting
a
couple
of
but
of
their
scalar
models
to
onyx
is
also
have
a
very
good
performance
and
finally,
is
something
that
is
that
that
wasn't
from
our
expectation,
it's
actually
from
the
chinese
market,
because
there
is
a
huge
amount
of
people,
are
actually
leveraging.
Paddle,
paddle
and
paddle
paddle
is
also
one
of
the
the
deep
engine
that
dpr
library
actually
support.
A
However,
most
of
the
people,
especially
for
people
who
is
using
onyx
runtime,
they
always
facing
a
very
big
challenge,
is
how
because
there's
no
built-in
operators
they
they
cannot
easily
doing
the
pre-processing
and
post-processing
inside
of
java,
like
on
the
like,
especially
on
the
un-dimensional
array
operation.
So
deep
jar
library
offers
solution.
A
So
we
introduced
a
new
concept
called
the
hybrid
engine,
and
essentially
is
I
mean
in
fact
it
is
just
trying
to
load
the
both
two
engines
together
at
the
same
time,
and
also
it
provides
us
a
smooth
trend
transition
for
for
customers,
especially
when,
when
they're
trying
to
switch
the
engine
be
between
like
to
run
for
the
inference.
So
basically
you
all
you
need
to
do
is
just
have
a
master
imprint
engine
in
here
we
just
use
the
onyx
runtime
and
one
process
engine
in
here.
A
We
also
use
the
pi
torsion
here
and
since
we
we
we're
supporting
other
operators
by
using
high
torch,
tensorflow
and
mxl.
The
user
can
choose
one
of
the
engine
to
be
served
as
the
processing
engine
and
to
to
reduce
the
data
copying
time.
Instead
of
like
trying
to
copy
the
data
from
java
heap
into
the
c
process
layer,
we
introduce
the
way
by
using
the
direct
buffer
to
to
send
the
pointers
from
the
from
the
pythons
directly
to
to
onyx
runtime.
A
This
avoids
the
the
data
copying
and
also
providing
a
very
huge
performance
boost
on
the
on
the
infrared
side.
If
you're
trying
to
comparing
you
run
the
end
to
end
inference
with
pytorch
and
this
hybrid
mode
and
finally,
the
way
for
us
to
to
control
the
memory
management
we
creating
something
called
nd
manager.
It's
a
tree-like
architecture
that
has
been
implemented
in
deep
drill
library
to
replace
the
chicago
collection
system
in
in
inside
of
the
java
to
to
provide
the
the
more
cost
effective
memory
collection
stuff.
A
A
So
here
you
can
actually
see
how
the
how
how
the
simple
the
the
experience
is
going
to
be
so
deep
draw
library
offers
us
a
criteria
class
where
the
user
can
define
their,
how
how
their
model
is
going
to
be
load.
They
can
offering
a
model
url.
Basically,
it
can
be
a
file
url.
It
can
be
a
general
hp
url.