►
Description
Allowing people to overlay data on the real world & create user interfaces in 3D, augmented reality (AR) and virtual reality (VR) — referred to together as "XR" — are rapidly emerging on our smartphones and on specialized headsets. Some believe that XR will change everything from entertainment & art to business software, so how does JavaScript play a role? Andrés Cuervo will look at how XR platforms today are adopting JavaScript through the Immersive Web API, developers are using Node to run JavaScript in embedded XR devices, and what the future of JS XR development might look like.
A
My
name
is
Andres
Cuervo
and
the
long-winded
version
of
our
title
of
my
talk
is
Jas
and
the
virtual
and
augmented
reality
ecosystem
and
the
reason
I
named
it.
That
is
because
I'm
gonna
be
talking
about
a
lot
of
stuff
today,
but
before
I
get
into
that
before
I
get
into
it.
I'm
gonna
talk
a
little
bit
about
Who
I
am
I'm
an
artist
and
engineer
a
software
developer
right
now,
I'm
an
AR
engineer
at
an
marketing
technology,
company
called
moveable
Inc
and
we're
building
a
web
AR
platform.
A
I
am
NOT
gonna
be
talking
about
that
specifically
today,
but
you
can
come
find
me
afterward.
If
you
want
to
know
more
and
before
I
talk
about
sort
of
the
place
that
web
XR
has
and
VR
and
AR
have
I
want
to
talk
a
little
bit
about
the
web
platform
as
a
whole,
so
how
many
of
you
have
heard
this
phrase
before
the
web
platform?
A
The
most
reliable
source
on
the
Internet
wikipedia
says
that
the
web
platform
is
a
collection
of
technologies
developed
as
open
standards
by
the
w3c
and
other
standards
bodies
like
open,
J's,
like
ACMA
and
so
on,
and
that's
one
that's
sort
of
one
way
of
looking
at
it
another
way
of
looking
at
it
is.
If
you
go
to
platform
that
html5
org,
you
see
this
giant
checklist
and
it's
rather
overwhelming
of
all
the
various
specs
that
can
cut
that
can
possibly
fit
into
a
browser,
and
this
is
another
way
of
thinking
about
it.
A
A
If
we're
talking
about
an
error
context,
then
we
mean
something
like
grabbing
the
camera
and
telling
you
where
the
floor
is
or
telling
you
where
or
telling
you
being
the
developer,
telling
you
giving
you
back
a
mesh
of
geometry
and
allowing
you
to
place
here,
a
green
cube,
but
anything
into
the
real
world.
When
we
talk
about
VR.
A
A
A
The
easiest
way
to
get
started
sort
of
hacking
around
with
those
api's
was
actually
to
use
a
few
different
custom
browsers,
one
from
Google
called
web
AR
on
a
kit,
slash
or
webinar
on
AR
core.
It
doesn't
really
roll
off
the
tongue.
Basically,
all
it's
doing
is
taking
those
proprietary,
AR
ap
eyes
so
AR
kit,
and
they
are
core
on
iOS
and
Android
respectively
and
fusing
those
to
a
an
instance
of
a
web
page.
So
this
isn't
Safari.
This
isn't
Chrome
well.
This
is
Safari
on
iOS.
A
But
the
point
is
like
this:
isn't
a
shipped
browser
like
this,
like
you
would
have
you.
This
link
goes
to
a
github
page
and
you
would
have
to
download
that
and
manually
builds
build
an
a
custom
app
in
Xcode
and
then
get
that
onto
your
phone
in
order
to
get
this
running.
But
it's
really
it's
really
interesting,
because
you're
able
to
actually
use
WebGL
and
JavaScript
and
it
has
access
to
things
like
the
plain
or
two
things
like
hyper,
accurate
device
orientation.
A
The
other
option,
that's
a
little
bit
easier
to
get
started
with,
is
the
WebEx,
our
viewer,
which
is
basically
the
same
thing
but
provided
from
Mozilla,
and
they
actually
put
this
on
to
the
App
Store
because
they
realized
that
getting
like
not
everyone
either
has
a
has
an
iPhone
but
doesn't
have
a
Mac
or
it's
just
not.
Everyone
is
comfortable
building
a
custom,
iOS
app
randomly
off
of
github,
and
so
they
provide
a
similar
thing
and
those
are
two
sort
of
like
custom
browsers
for
a
long
time.
A
Vr
was
the
same
way
where
you
would
have
to
fork.
You
would
have
to
download
a
custom
fork
of
the
web
of
Google,
Chrome
or
Firefox
in
order
run
web
VR
before
that
was
standardized,
but
now
we
have
a
bunch
more
options,
and
so
the
rest
of
the
the
rest
of
the
tools
in
here
are
going
to
be
things
that
can
actually
run
in
regular
browsers.
So
the
first
and
sort
of
most
common
one
in
my
mind,
is
three
Jas.
A
The
all
these
examples
are
actually
the
WebGL
context
running
on
a
VR
device,
so
you
would
navigate
you
would
be
in
your
VR
headset.
You
would
navigate
to
the
three
Jas
examples
website
and
you
could
click
a
button
and
say
enter
VR,
and
that
would
that
would
automatically
move
you
into
one
of
these
experiences
and
that's
that's
sort
of
where
we're
at
right
now
and
you
can
do
really
basic
things,
but
you
can
do
a
lot
with
these
really
basic
things
like
position
and
hand,
orientation
and
stuff
like
that.
A
The
other
option
that
is
actually
built
off
of
3GS
is
this
library
called
a
frame
and
whereas
3GS
itself
is
a
proper
JavaScript
library,
a
frame
has
a
completely
different
goal.
A
frame
is
in
was
designed
explicitly
for
people
who
only
have
familiarity
with
HTML
and
CSS
and
don't
and
are
and
are
possibly
intimidated
or
just
don't
want
to
deal
with
WebGL
or
three
or
you
know
disparate
three
3d
contexts.
So,
in
order
to
understand
a
frame
a
little
bit,
this
slide
shows
you
all
of
the
code
for
the
scene.
A
That's
gonna
run
in
the
next
slide.
So
here
you
can
see
in
this
is
a
normal
body,
and
then
everything
inside
of
here
is
a
frame
code.
It
all
looks
like
HTML.
In
fact
it
if
there's
library,
precedes
the
web
component
spec,
but
it
was
modeled
off
of
web
components,
so
everything
has
a
dash
in
it
and
it's
everything
starts
with
a
dash.
A
A
dash
scene
is
the
top
level
and
that
sort
of
gives
you
your
3d
environment
and
then
anything
you
put
inside
of
here
that
a
frame
recognizes
it
will
actually
create
for
you
in
a
full
3d
world
and
see
what
that
actually
looks
like.
So
this
is
an
iframe
and
right
now,
I'm
dragging
around
with
my
cursor
and
I
can
actually
press
either
with
my
arrow
keys
or
my
WASD
Keys
I
can
move
really,
naturally
in
here,
if
I
was
on
a
mobile
device.
A
This
button
at
the
bottom
right
would
allow
me
to
enter
Google
cardboard
view.
If
I
was
hooked
up
to
a
VR
machine,
it
would
give
me
the
VR
instance
that
we
saw
earlier
and
just
to
sort
of
prove
out
like
that.
This
is
actually
what's
running.
You
can,
even
in
real
time
at
the
scene
from
the
inspector
they're
like
these
are
just
normal
HTML
elements
that
are
bound
to
the
WebGL
context,
and
so
that's
a
really
easy
way
to
get
started.
A
A
One
other
tool
is
model
viewer,
which
is
itself
a
proper
web
component.
That
Google
provides
and
it's
relatively
new
I
think
it
came
out
at
the
end
of
last
year,
or
at
least
it
was
open
sourced
at
the
end
of
last
year
and
again,
it
is
just
HTML
with
a
script
import
tag,
and
you
can
see
on
the
on
the
right
here.
A
This
is
sort
of
what
you
see
if
you're
on
the
desktop
view,
but
if
you're
on
a
phone
that
either
is
an
Android
phone
and
provides
this
this
API
called
scene
or,
if
you're,
on
a
iPhone,
and
it
provides
this
thing
called
quick
look
model
viewer
knows
about
it.
It
knows
how
to
call
out
to
it
and
so
similar
to
a
frame,
but
for
a
very
specific
use
case.
A
A
One
other
tool
is
this:
is
the
second-to-last
tool
is
tensorflow
j/s,
so
the
attentively?
There
are
a
few
tensorflow
J's
people
at
this
conference
and
they
gave
a
talk
earlier
today
and
they're
there's
a
lot
of
low
level
stuff
you
can
do
with
with
machine
learning
on
the
web,
but
one
thing
that
they
provide.
Is
this
higher
level
sort
of
ready-made
model
that
they've
trained
on
a
bunch
of
different
samples
of
bodies
and
what
they
actually
provide
back
is
not
just
a
skeleton
but
away
you
can
see
over
here
on
the
right.
A
A
way
to
map
out
like
the
purple
is
the
head,
the
green,
the
light
green
is
the
torso.
You
could
theoretically
sort
of
build
a
whole
mask
over
over
these
people
in
WebGL.
You
could
add
3d
models
onto
them.
You
could
do
you
can
do
a
bunch
of
interesting
things
that
all
begin
to
sound
exactly
like
they
are
sort
of
like
what
what
five
years
ago
would
have
been
like
very
hard
to
do.
Industrial
AR
and
speaking
of
Industrial
a
are
there.
Is
this
one
other
toolset
that
is
sort
of
web
adjacent?
A
So
there's
this
iOS
prototyping
tool
called
torch
and
torch
is
designed
to
very
quickly
get.
You
started
with
AR,
it's
a
no
code
tool,
but
it
actually
just
recently
like
last
month,
introduced
an
export
option
and
they
partnered
with
a
company
called
eighth
wall.
Eighth
wall
is
a
web
AR
company,
so
under
the
hood
they
take
all
of
their
sort
of
iOS
prototyping
code,
transfer
it
over
to
eighth
wall,
and
then
you
can
view
it
in
a
normal
browser.
This
is
run
you
can
see
here.
A
A
A
A
It
gives
you
a
full
nodejs
environment
anyway,
seeing
like
big
companies
like
magic
leap
or
microsoft,
with
hololens
sort
of
release
web-based
tooling
is
really
interesting,
and
it
means
that
there
is
sort
of
is
industry
support
for
all
of
this,
something
that
is
very
nascent
is
this
website
called
immersive
web
dev,
which
is
actually
run
by
this,
is
maintained
by
the
w3c
a
few
folks
on
the
w3c
immersive
web
group,
and
this
sort
of
gives
you
a
nice
starting
place
for
figuring
out
what
are
all
the
requirements?
How
do
you
do
vrn
AR
on
phones?
A
This
means
that
a
lot
of
VR
browsers
when
they,
when
the
next
time
that
they
update,
are
going
to
get
a
lot
of
the
features
or
again
I'm
gonna
get
these
features
for
free,
and
this
sort
of
this
is
just
the
first
step
right
like
next.
We
need
every
browser
to
implement
it,
but
everyone
is
on
board.
The
standard
has
been
like
actually
drafted,
and
so
this
work
is
like
I'm,
not
gonna,
say
nearing
completion.
A
It's
it's
a
very
big
milestone
and
then
the
last
thing
is
I
know:
I
threw
a
lot
of
like
random
resources
at
your
face
and
there's
a
lot
of
overhead
here,
but
there
because
there
is
so
much
work
to
be
done
and
so
much
work
actively
being
done.
The
last
thing
I
want
to
say
about
the
future
of
XR
is
that
it
probably
starts
with
all
of
you
there,
it's
a
small
space
right
now,
but
there
that
means
that
there
is
a
lot
of.
A
There
was
there's
a
lot
of
low-hanging
fruit
for
open-source
contributors
for
people
to
do
work,
getting
JavaScript
ready
for
for
this
immersive
future,
with
all
the
tools
that
I
talked
about
yeah
and
this
QR
code
goes
to
the
link
for
these
slides
yeah.
Sorry,
these
slides
so
yeah.
That's
all
I
have.