►
Description
In this talk, Steve Kinney will talk a little bit about the Web Audio API. He’ll explore the browser as a vehicle for creative expression. He’ll fire up some audio contexts and connect some nodes. He’ll also leverage the getUserMedia Web API, WebSockets, and others to build unique musical instruments that could only be possible in the web browser. He’ll not only talk about the API itself, but also some of the fundamental concepts for working with audio and making music.
About CodeConf
CodeConf improves the software community by providing a forum for thought-provoking talks and forging social connections. The third installment of the CodeConf series took place in Nashville in 2015. Attendees came together to discuss open source, best practices, documentation, and community.
For more information on this year's CodeConf, go to:
https://codeconf.com/
A
A
Hello,
I'm,
Stephen
and
I
am
the
co-director
of
academics
at
the
Turing
school
of
software
and
design.
What
is
that?
Well,
I'm
glad
you
asked
it's
a
7-month
developer
training
program
in
Denver,
Colorado,
focusing
on
Ruby
rails
JavaScript
and
the
like
taking
people
who
want
to
become
great
developers
and
making
really
great
developers.
A
So
if
you're
interested
in
hiring
great
junior
developers
come
talk
to
me,
you're
interested
in
mentoring,
up-and-coming
developers
also
I'd
love,
love
love
to
talk
to
you
after
this
cool,
so
I
wanted
a
little
bit
today
about
doing
fun
and
creative
stuff
in
the
browser.
Can
one
raise
your
hand
if
you
are
into
having
fun
and
being
creative
you're
against
it?
You
can
leave
your
hand
down
right.
I,
don't
want
it
like.
You
know,
force
anyone
to
be
into
fun.
So
it's
a
little
bit
today
about
making
some
music
right
and
I.
A
Don't
really
have
a
great
way
to
tell
you
this
I'm
just
going
to
tell
you
it
right
now
somebody
put
a
synthesizer
into
your
web
browser
I,
honestly,
don't
actually
know
what
happened
and
you'd.
Think
I
would
have
looked
that
up
beforehand,
but
it's
cool,
but
it's
in
most
it's
in
most
your
web
browsers,
soon
to
be
in
to
explore.
In
the
rest,
my
apologies
to
the
Opera
Mini
user,
and
so
how
do
we
go
about
using
this
synthesizer?
That's
built
into
our
browser?
A
Well,
it
starts
with
some
context
if
you've
ever
played
around
with
like
html5
canvas,
you
know.
The
very
first
thing
you
have
to
do
is
like
get
a
canvas
context
and
the
same
thing
is
true
for
using
the
Web
Audio
API.
So
we
start
by
creating
an
audio
context
which
is
pretty
straightforward
to
do.
We
have
two
for
Safari.
We
got
a
little
bit
of
a
rigmarole
with
like
some
vendor
prefixed
scene,
but
we
can
abstract
away
around
that.
A
It's
fine
and
basically
just
a
new
audio
context
and
we
get
back
an
audio
context
and
what
exactly
is
an
audio
context?
Well,
it's
a
place
where
we
start
to
work
with
some
different
audio
processing
nodes.
The
most
important
one
to
start
out
with
is
this
idea
of
a
destination
right.
The
destination
is
like
the
operating
system
and
whatever
it
is.
It
does
to
get
sound
from
the
OS
to
either
your
headphones,
your
earbuds,
your
speaker's
what-have-you.
A
The
other
thing
that
we
need
is
the
ability
to
actually
make
some
noise
right.
There's
a
bunch
of
ways
to
do
that.
We'll
cover
in
a
moment,
what's
really
cool
about
the
Web
Audio
API
is
that
we
can
take
the
the
noise
that
we
create
and
begin
to
process
it.
We
can
add,
delay
reverb
gain
all
sorts
of
fun
stuff.
So,
let's
start
by
talking
about
this
whole
context
and
the
destination
that
we
can
produce
sound
through.
A
The
next
thing
we're
probably
going
to
need
to
do
is
to
create
some
sound,
there's
a
bunch
of
ways
to
do
that
that
we'll
cover,
but
one
of
them
is
to
like
take
sound
from
an
audio
file.
That's
been
done.
The
other
thing
we
could
do
is
we
can
take
the
user's
microphone
and
maybe
pump
in
some
sound
to
an
audio
input
or
something
along
those
lines.
That's
cool.
A
A
An
oscillator
is
what
we
use
to
like
produce
tones
inside
the
browser,
and
it's
basically
creates
a
sound
wave
right
and
sound
ways
of
frequencies
as
a
frequency
goes
up,
as
the
wave
gets
smaller
and
smaller
pitch
gets
higher
and
a
wave
has
amplitude,
which
is
how
short
and
tall
the
wave
is,
and
that
allows
us
to
have
some
amount
of
volume
to
it.
Creating
an
oscillator
super
easy
that
context
we
created
before
it
has
a
creator
method.
That's
really
useful
for
creating
oscillators.
A
A
Like
here
is
like
it's
plugging
a
guitar
into
an
amp
right,
in
this
case
the
keytar,
because
why
not-
and
that's
really
easy
to
do,
because
the
oscillator
has
a
connect
method
and
we
basically
take
the
oscillator
and
we
connect
it
to
the
audio
context,
is
destination
and
finally,
we're
not
actually
producing
a
sound
deck.
So
we
haven't
turned
the
oscillator
on.
So
let's
go
ahead
and
do
that,
so
the
method
is
pretty
straightforward.
The
one
thing
you
might
notice
this
is
zero.
That's
when
you
want
it
to
start.
A
If
you
give
it
a
value
of
a
thousand
will
start
in
100
milliseconds
or
a
second.
If
you
give
it
zero
it'll
start
right
now,
Chrome
and
Firefox.
Don't
actually
care.
If
you
put
anything
in
here,
Safari
does
so.
If
you
would
like
to
wonder
if
you're
wondering
why
your
thing
is
not
working
on
iOS.
That's
why!
A
Lastly,
if
you're
like
me,
you
will
probably
set
the
pitch
way
too
high,
you're,
probably
wearing
earbuds
and
you're,
probably
like
shrieking
and
pain
at
this
point,
so
you're
probably
probably
going
to
want
to
stop
this
thing
and
that's
very
easy
to
do
as
well
and
again,
you're
probably
going
to
stop
it
right
now.
The
problem
is,
if
you
want
to
start
it
again,
once
you
stop
an
oscillator
game
over
you're,
not
turning
it
back
on
you're
done
with
it
go,
get
a
new
oscillator.
A
The
other
thing
you
might
want
to
do
is
then
disconnect
it.
If
you
don't
disconnect
it,
the
garbage
collector
never
grabs
it.
It
stays
around
forever
and
eventually
you
get
yourself
a
nice
friendly
memory
leak.
So
when
you
disconnect
it
garbage
collection
comes
around
and
takes
it
away,
and
you
can
create
a
bunch
of
these
really
easily,
but
we
talked
about
this
before.
What's
really
cool
about
the
audio
API
is
that
you
can
actually
put
in
these
intermediary
nodes.
So
what
what?
A
If,
instead
of
having
to
like
get
rid
of
a
bunch
of
oscillators,
we
simply
put
something
in
the
middle.
That
would
turn
the
volume
up
and
down
right
when
we
turn
it
down
to
zero.
It's
off
when
we
turn
it
up
to
one
it's
on
it
turns
out.
We
can
do
that
with
a
gain
node.
That
gain
is
pretty
much
volume.
For
our
intents
and
purposes,
there's
no
mystery
in
how
you
create
a
gain
note.
A
It's
just
like
we
how
we
create
an
oscillator
we've
got
this
create
gain
method,
and
so
we
create
the
three
things
that
we
need
step.
One
is
we
hook
up
the
sound
source,
in
this
case
the
oscillator
to
the
volume
or
the
gain
node.
That's
really
straightforward
kind
of
how
we
did
it
previously.
The
next
step
is,
then,
we
take
the
gain
node
and
we
hook
it
up
to
the
destination.
So
right
so
now,
we've
plugged
our
guitar
into
a
like
a
volume
pedal
the
volume
pedal
into
the
amplifier
and
we're
ready
to
rock
cool.
A
At
this
point,
we
can
start
up
the
oscillator
when
we
want
it
to
be
loud.
We
turn
the
gain
up
to
one.
What
we
want
it
to
be
quiet,
we
turn
the
game
down
to
zero.
We
don't
actually
have
to
like
get
rid
of
oscillators
and
create
more
oscillators.
We
just
take
it,
give
an
oscillator
plug
it
into
a
volume,
pedal
great
awesome
and
we
can
turn
it
on
and
off
as
we
need
to.
A
A
So
we
can
create
some
sound,
that's
great,
but
making
sound
is
necessary,
but
not
sufficient
for
making
music
right.
The
next
step
is
to
actually
take
in
some
basic
music
theory
here,
so
we
can
begin
to
take
these
tones
or
create
various
frequencies
and
make
something
that's
less
annoying
than
what
I
just
exposed
you
to
so
by
default,
an
oscillator
starts
at
440
440
significant
because
you
get
like
one
of
those
tuning
forks
to
tune
a
piano.
So
they
all
know
that
you
sparetime
your
piano.
A
Tuners
440
is
the
default
frequency
for
that
and
the
way
that
sound
works
is
if
I
were
to
double
that
frequency
to
880
I
would
get
a
but
an
octave
higher,
and
if
I
was
to
have
it,
I
would
get
a
but
an
octave
lower
the
rest
of
the
notes
come
somewhere
in
the
middle
right.
You
might
say
to
yourself
I'm
an
engineer.
I
can
handle
this.
What
I'm
going
to
do
is
I'm
just
going
to
divide
it
by
12
and
I'll,
get
the
rest
of
the
notes
and
that
totally
works.
A
It's
cool.
It
totally
works.
If
you
only
ever
want
to
play
in
the
key
of
a
should,
you,
let's
say
playing
the
key
of
C,
because
I
don't
actually
know
what
those
Black
Keys
do
on
a
keyboard,
then
you
actually
need
to
actually
need
you
see.
This
doesn't
work.
You
start
from
C
and
do
the
same
thing.
Well,
D.
If
you
start
from
a
is
different
than
D.
If
you
started
from
C,
so
this
doesn't
really
work.
A
So
you
can
try
to
be
really
clever
eating,
say
alright,
I'm
going
to
write
this
really
sophisticated
algorithm
and
I'm
gonna,
like
figure
out
what
key
they're
in
and
I'm
gonna
like
find
the
correct
tone
and
shift
only
to
that
one
or
you
can
realize
the
fact
that
we
solved
this
like
five
hundred
years
ago
and
just
leverage
that
so
here
we
have
basically
a
keyboard.
What
what
piano
does
basically
is
it
made
a
whole
bunch
of
compromises?
A
There's
a
bunch
of
intervals,
there's
a
whole
bunch
of
different
choices
and
them
very
thick
music
theory
textbook.
So
you
can
buy
on
Amazon
for
like
a
hundred
dollars
each
and
read
all
about
this.
But
the
idea
is
like
we
solve
this
problem.
So
when
you
have
a
problem
and
a
simple
solution
and
a
cute
name,
you
have
an
open-source
library,
that's
how
it
works
and
basically
I
have
a
little
IRA
called
Octavian,
where
I
never
have
to
think
about
this
ever
again,
I
pass
it
a
sharp
four.
A
It
gives
me
back
the
right
frequency,
and
so
here
it
is
I
get
for
sixty
six
point.
One
eight
one
64,
which
I
would
never
remember
and
I,
can
also
like
get
a
whole
bunch
of
other
properties
about
it
and
even
better
I
have
a
bunch
of
different
intervals.
I
can
use
and
I
can
shift
up
a
major
third
up,
a
major
fifth
and
that's
really
cool
for
like
creating
chords
and
stuff.
So
let's
actually
take
a
look
at
this
in
action
here,
we've
got
a
let's
make
a
small
little
piano
based
off
of
this.
A
Loom
alright,
so
how
do
we
go
about
building
something
like
this?
It's
actually
not
as
complicated
as
it
might
seem,
and
this
graph
makes
the
same
way
more
complicated
than
this.
Basically,
we've
got
a
bunch
of
keys,
they're
hooked
up
to
a
bunch
of
oscillators.
Those
oscillators
have
a
bunch
of
volume
notes
that
would
basically
turn
the
volume
up
and
down
right
based
on
different
keyboard
events.
So
user
comes
in
they
press
the
key
emits
a
key
down
event.
A
We
basically
figure
out
what
key
they
press
and
we
turn
that
oscillator
on
or
off
little
syntactic
sugar
here,
which
basically
like
fires
up
an
oscillator.
If
we
don't
have
one,
it
makes
it.
If
we
have
it,
we
just
turn
it
on.
That's
it
when
they
take
their
finger
off.
Oh
so
we
start
the
oscillator,
that's
cool.
When
they
take
their
finger
off,
we
basically
stop
the
oscillator.
A
That's
it.
We've
created
we've
like
creative
synthesizer
in
our
browser,
so
that's
cool.
The
other
really
cool
part
is
that
somebody
put
a
web
browser
in
your
synthesizer
right.
It's
cool
to
like
emulate
a
piano,
but
like
maybe
it's
just
by
a
piano
at
that
point.
What's
really
interesting
is
using
the
other.
Api
is
available
in
the
browser
to
create
some
different
instruments
that
wouldn't
be
possible,
otherwise,
so
what
if
we
took
our
audio
API
and
mix
it
with
the
getusermedia
API
and
grab
the
webcam?
So
here
we
have
another
demo.
A
A
That's
enough
of
that
I
literally
could
do
that
for
the
next
40
minutes
by
the
way,
but
so
basically
what
happens
is
most
of
that.
The
hard
lifting
was
done
by
some
face
tracking
library,
I
totally
don't
understand,
but
what
the
library
does
is
it
omits
face
tracking
events
which
has
a
whole
bunch
of
properties,
I,
really
only
care
about
in
this
case
height,
which
is
when
I
stood
back
or
came
closer
to
it.
A
That's
the
size
of
the
box
that
my
face
was
in
I
got
a
big
head,
so
I
hadn't
stand
really
far
back
and
the
x-axis,
which
is
the
right
or
left
and
basically
I,
take
those
two
properties.
I
do
a
little
funny
business
to
make
sure
I
don't
create
like
a
really
obnoxiously
high-pitched
sound.
You
can
thank
me
later
and
I
do
a
little
rigmarole
to
get
the
game
to
actually
be
a
normal
number,
and
that's
really
it
all
right.
A
Let's
truck,
let's
check
out
under
a
different
combination
of
technologies,
what
happens
if
we
took
the
Web
Audio
API
and
we
decided
that
we
wanted
to
mix
it
with
some
WebSockets?
Well,
there's
some
problems
here.
One
is
like
latency
right,
like
music,
this
whole
rhythm
thing-
I,
never
had
any
of
it
by
here,
it's
important
and
so
having
people
on
some
kind
of
like
even
millisecond,
like
latency,
is
not
going
to
work.
So
what
we
could
do
is
we
can
just
take
inspiration
from
some
other
instrument.
A
This
is
upstairs
it's
the
atom
terminal
and
you
can
see
there's
a
little
step
sequencer
here
and
basically
it
steps
through
each
column
and
plays
the
notes
that
are
turned
on
kind
of
like
lights
out,
but
for
music,
and
we
can
create
something
like
that
and
we've
got
16
different
quarter
notes,
except
that,
like
we're
programmers,
so
it's
zero
indexed,
it's
cool
and
we
have
some
basic
notes.
I
chose
c
e
g
and
a
g
and
a
will
sound
terrible.
We
turn
them
on
together,
but
roughly
it's
the
key.
A
It's
a
c
major
chord
with
an
extra
note.
So
it's
really
hard
to
make
a
really
terrible.
Sound
music
I
mean
I'm
sure
you
can
do
it,
but
it
should
be
slightly
harder.
What
happens
along
that
time
is
that
when
a
user
hits
one
of
the
different
boxes,
we
send
the
entire
sequence
to
a
server.
That's
keeping
track
of
all
the
connected
users,
then,
in
that
case,
node
will
go
out
and
send
out
all
the
sequences
to
all
the
users
and
they'll
get
combined
in
the
browser
to
make
a
fun
sequence.
A
B
A
A
All
right,
final,
final
demo,
other
cool
thing
about
JavaScript-
is
that
it's
widely
the
most
popular
programming
language
in
the
browser,
mostly
because
it's
the
only
programming
language
in
the
browser
that
helps,
but
it's
actually
like
people
decided
that
hey
I
would
like
to
do
JavaScript
on
hardware
and
all
sorts
of
other
plays,
maybe
even
the
server
that's
cool.
What
happens
if
we
start
mixing
the
Web
Audio
API
with
some
Arduino
Hardware
using
johnny-five,
so
this
is
a
little
kind
of
demo.
I.
A
Have
it
sitting
here
next
to
me,
but
this
is
the
kind
of
prototype
for
it
and
it's
got
some
buttons.
It's
got
a
little
rotary,
potentiometer
and
a
tweener
board,
and
what
happens
is
whenever
something
happens
on
the
board
that
sends
a
signal
to
johnny-five.
Now
johnny-five
can
also
fire
up
a
web
server
and
a
WebSocket
server.
So
button
comes
in
on
the
Arduino
board.
Next
thing
you
know
it's
then
sending
a
WebSocket
signal
to
the
browser.
The
browser
will
actually
grab
the
Web
Audio
API
and
make
a
sound.
A
A
So
the
moralist
story
is
my
hope
today
was
that
we
could
take
you
from
I.
Don't
know
anything
about
making
music
in
the
browser.
Yeah
I
could
probably
do
this
right.
I
could
play
around
with
this.
So
music
is
pretty
awesome.
The
web
is
pretty
awesome
and
it's
like
cookie
dough
and
ice
cream
together.
They
are
really
awesome.
So
there's
a
whole
bunch
of
things
we
didn't
touch.
We
didn't
actually
combine
a
lot
of
technologies
here,
there's
a
whole
other
api's
that
we
didn't
even
use.
A
There's
a
gamepad
controller,
ap
I
bet
you
never
knew
about
that.
I
didn't
there's
web
RTC,
which
is
the
ability
to
get
user
media
from
multiple
sources.
You
could
have
like
multi
face
theremin,
if
you
put
your
mind
to
it,
there's
the
gyroscope
and
the
device
orientation,
API
and
a
whole
bunch
of
other
ones,
and
so
all
the
code,
demos
and
everything
that
you
need
is
here
and
pull
request
accepted
right.
If
you
do
something
really
cool,
send
me
a
pull
request.
I
would
love
to
include
it.