►
From YouTube: ONNX and the JVM
Description
Integrating machine learning into enterprises requires building and deploying ML models in the environment's enterprises build their software in. Frequently this is in Java, or another language running on the JVM. In this talk we'll cover some of our recent work bringing the ONNX ecosystem to Java. We'll discuss uses of ONNX Runtime from Java, and also our work writing model converters from our Java ML library into ONNX format.
Adam is an ML researcher in Oracle Labs' Machine Learning Research Group. He's worked on feature selection, scaling up Bayesian inference with GPUs and more recently NLP. He's the lead developer of the Tribuo ML library, maintains the Java API for ONNX Runtime, and co-leads the TensorFlow-Java project.
A
So
one
question
everyone's
talking
about
python
today.
Why
would
we
want
to
support
onyx
on
the
jvm
on
the
java
platform?
Well,
ml
models
are
increasingly
important
in
basically
every
application
and
most
applications,
especially
like
large
business
applications.
At
the
time,
we're
interested
in
oracle
are
written
in
languages
that
are
not
python,
so
we
could
either
get
all
those
developers
to
move
to
python,
which
doesn't
seem
very
likely,
or
we
could
bring
machine
learning
to
them
in
the
languages
they
work
in
like
java
or
c-sharp
or
javascript.
A
So
I've
spent
the
past
few
years,
sort
of
building
java
ml
tooling,
both
for
onyx
and
other
ml
libraries
and
we'll
talk
a
little
bit
about
some
of
that
today.
So
the
first
thing
was
the
onyx
runtime
java
api.
So
this
was
something
that
was
built
in
our
group
in
oracle
labs
in
springer
2019.
We
contributed
to
the
upstream
onyx
runtime
project
in
december.
The
binaries
have
been
available
since
2020..
A
This
is
used
in
production
in
oracle
and
some
other
companies.
I
believe
some
of
them
will
be
talking
after
me
about
their
production
use
cases.
So
I'm
not
going
to
talk
about
that
very
much
and
our
goal
is
basically
to
provide
the
onyx
runtime
c
api
in
java.
So
it's
easy
to
use
and
it's
sort
of
idiomatic
and
you
can
get
all
of
the
functionality
that
you'd
expect
you
can
deploy
all
the
models
that
you
need
to.
We
want
to
try
and
keep
feature
parity
with
all
the
other.
A
Apis
was
a
few
things
that
were
missing
at
the
moment,
around
custom
memory
allocation
and
I
o
binding,
because
it's
difficult
to
sort
of
express
pointers
and
native
memory
operations
in
java.
It's
not
really
for
that,
but
if
there's
something
else
missing
or
you
need
those
features,
please
open
an
issue
on
github
and
we
can
talk
to
you
about
it.
Our
aim
really
is
that
the
java
api
is
a
very
thin
layer
over
the
c
api
with
minimal
performance
impact.
A
We
have
a
zero
copy
path
through
from
java
into
onyx
runtime
for
inputs,
and
you
have
to
do
a
copy
on
the
way
out.
Although
we're
looking
at
ways
of
avoiding
that,
and
but
fortunately,
your
inputs
tend
to
be
a
little
smaller
than
your
outputs,
so
your
outputs
tend
to
be
a
little
smaller
than
your
inputs.
A
So
it's
less
of
a
problem
doing
that
copy,
and
you
have
to
do
a
copy
if
you're
on
a
gpu
anyway,
the
api
targets
java,
8
and
runs
on
all
versions
later
than
that
and
there's
no
other
dependencies
apart
from
the
onyx
runtime
naked,
live
native
library
and
that's
packaged
into
it.
So
it's
very
easy
to
deploy,
so
we're
going
to
look
a
quick
code
example.
It
looks
very
much
like
the
python
and
the
other
apis
right,
we're
trying
to
be
simple
and
similar
to
other
things.
A
So
you
have
to
make
an
rx
environment.
You
can
control
like
a
per
environment
thread
pool
there
if
you
need
to
order
logging,
we
didn't
bother
in
this
example.
We're
going
to
make
some
session
options
set
the
number
of
threads
that
we're
using
and
then
we
load
a
model
by
reading
an
onyx,
modeling
off
disk,
or
you
can
read
a
byte
array
in
from
memory.
A
A
A
So
when
we
want
to
feed
data
into
it,
as
I
said,
there's
a
zero
copy
pathing
you
have
to
use
a
byte
buffer,
which
is
a
java
representation
for
just
a
stream
of
bytes.
You
have
to
be
a
little
careful
about
how
you
order
it
because
for
some
reason,
java
defaults
to
using
networking
style,
byte
buffers
which.
A
Which
doesn't
actually
do
any
copy,
just
wrap
some
extra
metadata
around
it
and
gets
the
pointer
out.
Then
we
run
the
model
by
feeding
in
the
tensor,
along
with
the
name
of
the
input
that
it
is.
We
get
a
result,
object
back
that
result
object.
We
can
query
to
get
all
of
the
outputs
out
and
we
can
see
here
that
we
have
10
values
which
represent
the
likelihood
of
each
of
the
digits.
A
So,
as
you
might
expect,
if
you've
done
a
lot
of
work
with
with
sort
of
managed,
runtimes
and
interacting
with
native
code.
Basically
all
of
the
work
here
is
shuffling
memory
around
between
different
heaps
and
in
python.
That's
a
little
easier
because
you
can
directly
access
pointers,
but
jvm
does
not
let
you
do
that.
So
it's
a
bit
a
bit
more
tricky
and
we
want
to
be
as
efficient
as
possible.
So
we
can
maximize
throughput
and
latency
so
to
sort
of
minimize
overheads
for
people
now.
A
This
means
that
all
of
our
java
objects
have
to
have
like
a
paired
native
object
and
that
native
object
has
to
be
closed.
Otherwise,
you
end
up
leaking
native
memory
and
the
jvm
can't
see
that
because
it
doesn't
know
anything
about
native
lamp.
So
this
is
usually
done
with
a
try
with
resources
statement
which
we
saw
before
in
the
future.
We're
going
to
look
at
adding
more
safety
nets
to
ensure
that
everything's
always
cleaned
up,
even
if
it's
a
little
bit
more
expensive
to
do
that
cleanup.
But
that
requires
newer
java
versions.
A
We
do
actually
have
paths
through
this
for
using
regular
arrays
in
java,
but
we
recommend
all
users
use
the
byte
buffers
because,
as
I
said,
there's
a
zero
copy
path
and
you
can
reuse
that
buffer
again
and
again,
so
you
have
to
do
lots
of
allocations,
but
more
generally
java's.
Existing
multi-dimensional
arrays
are
a
really
bad
abstraction
for
machine
learning,
because
they're
not
flat
and
there's
lots
of
pointer
chasing,
so
they
tend
to
slow
everything
down
as
you
try
and
use
them.
A
So
a
little
bit
about
where
we
want
to
go
with
this,
I
think
we
want
to
move
to
a
more
modern
version
of
java
java.
8
is
8
years
old,
there's
a
lot
of
new
things
in
the
platform
that
will
make
in
working
with
native
code
much
easier
and
safer
and
there's
a
bunch
of
language
features
that
just
will
make
it
easier
to
develop
with
we're,
also
interested
in
adding
new
features
that
are
coming
from
onyx
runtime.
A
So
recently
there
was
single
op
execution
support
added,
and
that
would
be
great
just
to
be
able
to
sort
of
drive
a
single
matrix
multiply
or
some
operation
that
you
can't
easily
get
from
the
jvm
and
also
there
seems
to
be
some
work
around
training,
and
that
would
be
great
to
support
as
well.
Training
training,
deep
learning
systems
is
quite
difficult
on
the
jvm
at
the
moment,
and
we're
also
going
to
continue
to
build
out
to
match
the
ortc
apis
new
things
added
new
execution
providers
methods.
A
You
know
the
extra
memory
support
contributions
are
welcome.
This
is
all
done
upstream
in
microsoft's,
onyx,
runtime
tree
and
we'd
be
happy
to
talk
to
you.
There,
okay,
so
that
was
all
about
inference
in
models,
but
it'd
be
kind
of
nice
if
we
could
actually
train
models
in
java
and
then
export
them
as
onyx
for
use
in
the
wider
community.
You
can
do
this.
You
can
write
the
protobuf
directly
right,
you
can.
A
You
can
take
the
prototype
of
schema
files
and
emit
a
java
prototype
object
and
then
write
that,
but
it's
quite
difficult
to
write,
well-formed
onyx
models,
there's
no
validation
in
your
protobuf.
You
can
have
a
perfectly
valid
protobuf,
which
is
an
entirely
invalid
onyx
model.
So
we
wrote
a
small
open
source
library
for
writing
onyx
models
from
java.
A
We
did
this
for
exporting
onyx
models
from
our
our
java
ml
library,
but
there's
actually
no
dependencies
on
it.
It
only
really
depends
on
protobuf
and
the
onyx
schemas.
So
you
can
use
this
anywhere
on
the
jvm
to
write
out
onyx
models.
It's
apache
2
licensed
it
lives
currently
in
our
tree
for
our
library,
but
we're
happy
to
share
this
with
people
if
they
want
to
use
it,
and
it
gives
you
sort
of
attribute
validation,
checking
that
your
graph
is
correct.
A
A
A
So
we
currently
support
a
subset
of
onyx
offset
13
and
the
onyx
ml
operators.
That
was
basically
all
the
things
we
needed
to
export
the
set
of
trivial
models
that
we
had
we'd
like
to
expand
this
to
cover
the
full
set
of
onyx
ops.
It's
really
easy
to
expand
the
operator
enum
and
we
might
actually
just
look
at
automatically
generating
it
from
the
op-def
files
in
c
plus.
A
That
was
just
a
bit
out
of
scope
for
our
initial
work,
we're
at
at
the
moment,
looking
at
abstracting
over
offsets
to
allow
users
to
supply
different
versions
and
to
use
their
own
custom
ops.
We
didn't
do
an
initial
version.
We
haven't
needed
it,
but
it's
actually
pretty
easy
to
do
and
we've
got
our
prototype
up
and
running
and
we'd
also
like
to
integrate
the
provenance
and
metadata
into
these
converted
models.
A
We
actually
export
tributes
or
very
detailed
model
prominence
object
with
the
training
data
and
the
hype
parameters
in
a
field
in
the
onyx
metadata.
But
it's
not
standardized.
You
know
we're
really
interested
in
the
work.
Intel
are
doing
on
standardizing
this
so
that
we
can
emit
standard
metadata
formats.
Contributions
are
welcome
to
the
onyx
export
stuff
in
tripo
and
we'll
have
to
talk
about
it.
If
you
want
to.