►
Description
This presentation was recorded at GOTO Aarhus 2013
http://gotocon.com
Jeff Hawkins - Brain Inspired Computing
ABSTRACT
Understanding how the brain works and building machines that work on the same principles is one of the greatest quests of our time. In this talk I will describe recent advances in neocortical theory, including why the brain uses sparse distributed representations and how the brain makes predictions from high velocity sensory data streams.
I will demonstrate a product called Grok, that uses a detailed model of neocortical memory to act on machine generated data and how developers can contribute to the development of intelligent machines via the NuPIC open source project (www.numenta.org).
https://twitter.com/gotocon
https://www.facebook.com/GOTOConference
http://gotocon.com
A
You'll
see
why
maybe
in
a
little
bit
and
so
I
actually
went
back
and
worked
in
mobile
computing,
because
that's
that
was
what
I
could
do
until
I
got
around
to
what
I
really
wanted
to
do
sort
of
like
a
person
who
wants
to
be
an
actor
waiting
tables
you
don't
because
before
they
get
a
job
so,
but
for
the
last
12
15
years
or
so,
I've
been
full-time
in
the
neuroscience
I
also
started.
A
nonprofit
organization
called
the
red
for
Neuroscience.
Institute
I
ran
that
for
several
years
that
was
the
scientific
Institute.
A
That's
now
at
University
of
California
Berkeley,
so
I've
been
deep
into
neuroscience
for
a
while,
so
I'm
going
to
talk
about
computing
like
the
brain
or
how
we're
gonna
build
intelligent
machines.
So
I
gave
myself
over
30
years
ago.
I
gave
myself
two
things:
I
really
really
wanted
to
do.
The
first
was
discovering
the
operating
principles
of
the
new,
your
cortex.
A
Now
that's,
not
the
entire
brain
I
don't
want
to
start
I,
don't
say
the
entire
brain
I
know
a
lot
about
all
parts,
but
the
new
your
cortex
is
the
is
the
the
big
wrinkly
thing
on
top
it's
about
80%
of
the
volume
of
your
brain
and
it's
the
locus
of
all
intelligence.
So
my
I'm
speaking,
my
neocortex
of
speaking,
yours,
is
listening
when
I
move
my
hands
and
control
things
here,
it's
my
new,
your
cortex
all
things.
We
consider
high-level
intelligence
within
your
cortex,
so
we
should
be
able
to
understand
how
this
thing
works.
A
What
principles
it
works
on!
It's
just
a
bunch
of
cells,
it's
not
magic
and
then
once
you've
been
able
to
do
that,
you
should
be
able
to
build
systems
that
work
on
those
principles
and
we
can
build
those
in
software
and
hardware.
So
this
to
me
seemed
like
the
logical
way
to
go
about.
Building.
Intelligent
machines
is
to
figure
out
what
the
one
intelligent
machine
we
know
how
it
works
and
then
figure
out
how
to
abstract
from
that.
However,
this
has
not
been
a
common
view
and
especially
early.
A
My
career
I
ran
into
a
lot
of
resistance
about
this
less
so
now,
but
it
still
is,
is
still
perhaps
a
minority
view
and
I
want
to
give
you
a
little
bit
of
history,
about
machine
intelligence
and
people
who
thought
about
it
and
influenced
to
give
you
a
perspective
where
I'm
coming
from
and
they
were
going
to
dive
into
some
real
details.
Okay,
so
we're
gonna
go
back
in
time
now
and
want
to
talk
about
other
ways.
People
thought
about
building
intelligent
machines.
A
The
first
one
we'll
talk
about
a
little
bit
about
artificial
intelligence
and
I
call
this
fielding
the
no
neuroscience
approach.
They've,
just
basically
ignore
brains
completely,
which
it
might
be
okay,
but
that's
how
you
go
about
it.
It
got
she
goes
back
in
time
and
you
could
create
at
the
beginning
of
this
to
Alan
Turing
of
famous
of
computer
science
Fame
back
in
1935.
He
started
writing
about
Universal
machines
which
become
the
Turing
machine.
He
also
was
very
interested
in
machine
intelligence.
A
We
should
build
machines
that
are
that
are
operate
identically
to
humans,
that
we
couldn't
tell
if
it
was
a
human
or
machine
and
that's
the
definition
of
a
intelligent
machine
which
I
think
it's
a
terrible
definition,
because
we
can
build
intelligent
machines
with
a
nothing
like
humans
at
all,
but
he
sort
of
set
this
goal
and
the
field
of
AI
based
on
programming
computers
to
meet
this
goal
started
back
around
those
days
and
there's
over
the
years
been
many
many
major
AI
initiatives.
Some
have
been
successful.
Some
have
been
failures.
A
There
was
a
big
one
in
Japan
called
the
fifth
generation.
Computing
project
was
a
big
failure,
lots
of
AI
projects.
Recently,
we've
had
the
Google
driving
car
and
Watson,
beating,
Jeopardy
and
so
on
over
the
years.
The
good
thing
about
this
is
that
some
of
these
have
really
good
solutions.
I
won't
have
a
Google
driving
car
I,
don't
know
if
you
see
him
around
here,
but
we've
seen
where
I
live,
and
that's
just
cool
I
want
to
have
one
of
those.
A
The
problem
with
them
is
that
they're,
very
caste,
specific
solutions
and
there's
a
limited
or
no
learning
and
learning
is
one
of
the
call
marks
of
intelligence.
We
learn
all
the
time
we
learn
about
the
world,
we're
not
programmed
about
the
world.
We
learn
how
the
world
works,
and
so
AI
has
sort
of
been
in
many
ways,
sort
of
stuck.
It
doesn't
really
get.
It's
not
really
gotten
us
closer
to
understanding
what
intelligence
is
and
how
to
build
machines
that
work
like
like
humans.
A
The
other
major
initiative
would,
you
might
think,
of
the
artificial
neural
networks
and
I
call
this.
The
minimal
neuroscience
approach.
Even
though
has
the
name
neural
Network
in
it,
it's
really
not
neuroscience
like
at
all.
We
can
go
go
back
to
these
two
gentlemen,
Warren
McCulloch
and
Walter
Pitts.
A
These
guys
get
the
credit
for
that
now
today,
when
we
think
about
neural
networks,
most
of
them
are
based
on
a
different
type
of
neural
model.
I've
caught,
what's
called
a
you
know,
summation
with
some
sort
of
a
nonlinear
threshold
function
typically
shown
in
that
picture
there,
but
real
neurons
aren't
like
that
at
all.
I
might
tell
you
a
bit
about
more
than
later,
but
those
things
that
most
are
most
artificial
neural
networks
are
not
that
the
neurons
I
have
are
not
like
real
neurons
at
all.
A
You
can
not
even
close
and
then
the
way
they
hook
them
together,
and
these
different
networks
is
not
like
they're
hooked
together
in
the
brain,
so
there's
a
big
disconnect
between
artificial
neural
networks
and
what
real
neural
tissue
looks
like,
but
there's
been
a
number
of
techniques.
The
current
hot
line,
you've
probably
heard
a
lot
about,
is
deep
learning,
which
is
just
a
multi-layer
or
type
of
perceptron,
and
these
systems
are
pretty
cool.
A
These
are
they're
good
for
certain
things,
they're
good
for
classification,
most
people
who
use
artificial
neural
networks,
use
them
for
classification
and
they're
learning
systems.
That's
a
real
plus
over
the
artificial
intelligence.
The
downsides
are
they're
very
limited.
They
don't.
They
just
haven't
broken
out
of
some
very
basic
pattern.
Classification
problems,
not
there's
nothing
close
to
what
a
human
or
a
mouse
or
a
cat
or
monkey,
can
do
and
they're
not
brain
like
at
all.
A
As
I've
already
mentioned
recently,
there's
been
another
approach
which
which
actually
coming
out
of
here
and
in
Europe
and
the
CERN
Switzerland,
is
the
what
I
call
the
maximal
neuroscience
approach
now
the
best
example.
This
is
the
human
brain
project,
which
is
an
attempt
to
sort
of
model,
an
entire
human
brain
from
all
these
levels,
down
to
them
on
the
molecules
up
the
entire
body
that
little
rosette
picture
in
the
middle
is
all
the
principal
scientists
on
this
project.
Tells
you
how
many
there
are.
A
This
is
a
very
big
multi-billion
euro
that's
been
going
just
getting
started.
The
problem
I
have
with
this
there's
some
really.
This
might
be
very,
very
useful,
but
they're
not
basing
on
any
kind
of
theory.
There's
no
Theory
here,
there's
no
there's
no
attempt
to
understand
what
this
should
do
or
how
it
works,
especially
to
be
building
a
big,
simulator
and
hoping
it
does
something
interesting
and
and
they're
not
really
attempting
to
build
intelligent
machines.
So
it's
not
going
to
go
in
that
direction
unless
it
changes
significantly.
A
So
these
are
sort
of
the
background
that's
been
going
on
and
it
comes
back
to
my
approach.
You
know
the
approach
that
I've
been
working
on.
We
just
essentially
start
with
the
principles
in
the
new.
Your
cortex
understand
how
that
works
and
then
build
machines
based
on
those
principles.
This
is
the
process
we
go
through.
We
start
with
anatomy
and
physiology
in
great
detail
I'm
as
much
a
neuroscientist
as
a
computer
scientist,
perhaps
more
so
we
then
develop
theories
about
how
the
neocortex
works.
We
then
test
those
theories
in
software.
A
We
implement
this
stuff
and
we
run
it.
We
test
it
against
all
kinds
of
problems
and
ultimately
it's
going
to
go
to
silicon
and
that's
actually
starting
actually
this
year
and
I'll
talk
a
little
bit
about
that.
This
is
to
me,
you
know
we're
making
great
progress
on
this.
You
may
not
be
aware
of
this,
and
that's
one
of
the
reasons
I'm
here
to
talk
to
you
about
it,
really
good
progress
being
made
and
I
kind
of
view
it
as
today,
it's
sort
of
like
being
in
this
decade
is
like
the
1940s.
A
We
are
very
embryonic
age
part
of
this
thing,
but
it's
happening
right
at
the
moment
and
I'm
here
to
tell
you,
because
my
goal
I'll
tell
you
right
now.
My
goal
is
to
get
one
two
or
maybe
three
people
in
this
room
who
gets
so
excited
about
this
they're,
going
to
they're
going
to
make
this
part
of
their
career,
because
it's
gonna
be
very,
very
exciting.
It
is
exciting
thing
to
be
doing
so.
A
The
New
York
cortex
I'm
gonna
get
into
some
level
of
detail
at
the
end
of
the
talk
I'll
pop
back
up
again,
so
it's
gonna
get
hard.
If
you
don't
know
some
of
these
terms
but
walk
you
through
it
carefully
the
neocortex,
this
big
wrinkly
thing
is
actually
she
good.
If
I
took
it
out
of
your
head
and
flatten
it
out
it's
about
a
size
of
a
large
dinner
napkin,
it's
about
two
millimeters
thick.
It
is
a
memory
system,
it
is
not
a
computer,
it
doesn't
compute
anything.
It's
memory
and
storage.
A
You
have
patterns
coming
in
from
your
senses.
This
is
streaming
data
high
velocity
streaming
that
it's
changing
very
rapidly
in
order
milliseconds
from
your
retina
from
your
cochlea
from
your
somatic
senses
on
your
skin
and
the
new,
your
cortex
stores
those
patterns
in
a
very
interesting
way,
and
it
builds
a
model
of
the
world
and
that
model
the
world
from
it.
A
It
says:
okay,
given
the
current
inputs,
it
can
make
predictions,
it
can
detect
anomalies
something's
changed
what
it
should
be,
it
can
make
it
can
generate
behaviors
now
because
it
generates
behaviors
if
it's
actually
most
of
your
behaviors
move
your
eyes
and
your
in
your
head
and
your
body
when
I
touch
something
I
move
my
hand
over
it.
So
most
of
the
changes
in
sensation
you
have
coming
into
the
brain
or
due
to
your
own
behaviors,
so
you
create
the
patented
changes
in
patterns
to
a
large
extent,
not
completely.
A
So
what
we
would
say
is
the
neocortex
learns
a
sensory
motor
model
of
the
world.
It
learns
how
the
sensory
patterns
change,
based
on
the
current
sensory
input
in
its
own
motor
output
and
what
should
come
next
and
what?
How
can
it
string
those
together
to
make
certain
things
happen?
I
make
it
sound
very,
very
simple
like
this.
A
In
some
ways
it
very
it
is
very,
very
simple,
but
in
the
details
it
can
get
very
complicated
how
it
does
this
okay
I'm
now
going
to
tell
you
the
six
top
six
principles
by
which
the
new
your
cortex
works
and
then
we're
diving
to
detail
on
three
of
them.
So
this
is
my
my
top
six
here
number
one
is
it's
an
online
learning
system
online
learning
means
that
there's
no
batch
processing,
you
don't
store
up
data
and
look
at
the
statistics
of
it
and
decide.
A
What's
interesting
and
what's
not
that's
how
most
machine
learning
does
today,
but
the
brain
doesn't
get
to
do
that.
It
has
to
learn
continuously
every
moment,
never
stops,
and
you
know
you
don't
have
a
chance
to
go
back
and
say
well,
let's
go
sort
through
all
that
data
and
figure
out
what's
important.
So
it's
an
online
learning
system
from
streaming
data
and
it's
an
important
part
of
being
an
adaptive
learning
system.
You
can't
really
shortchange
them.
A
The
second
thing
is
that
sheet
of
cells
that
2
millimeter
thickness
sheet,
that's
about
the
size
of
a
dinner
napkin
is
actually
arranged
in
a
hierarchy
of
regions.
So
different
parts
of
that
sheet
are
connected
together
through
the
white
matter
into
a
hierarchy.
Now
this
hierarchy
doesn't
look
like
this.
Is
this
a
simple
cartoon
drawing,
but
you
can?
A
It
really
is
a
hierarchy
of
region
production,
the
region,
projecting
the
region
now
the
most
amazing
thing
about
this
unbelievable,
but
there's
so
much
evidence
that
it's
it's
incontrovertible
is
that
all
the
regions
in
the
hierarchy
are
basically
doing
the
same
thing.
Nearly
identical,
there's
some
small
variations,
so
the
parts
of
your
brain
that
are
hearing
and
the
party
of
rains
that
are
seeing
and
the
parts
of
your
brains
of
thinking
the
party
brains
are
doing
language.
If
you
look
them
under
the
microscope,
they
look
nearly
identical
and
there's
evidence
that
they
are
identifying.
A
To
out
all
the
regions
in
the
hierarchy,
and
if
you
switched
your
optic
nerve
and
your
auditory
nerve,
the
brain
would
still
work.
We've
done
this
on
animals
like
weasels
and
so
on.
So
it's
an
amazing
thing.
We
need
to
figure
out
what
any
part
of
the
New
York
cortex
is
doing
and-
and
that's
gonna
tell
us
how
all
the
regions
are
doing,
and
then
we
had
to
figure
out
how
they
work
in
a
hierarchy,
so
hierarchy
of
memory
regions.
Now
it
doesn't
there's
a
lot
going
on
in
each
of
these
regions.
A
There's
a
lot
going
on
in
the
tissue,
but
the
primary
memory
function
is
sequence
memory.
When
I,
say
sequence,
memory
I
mean
like
like
when
you
memorize
them
a
melody
and
what
the
notes
follow
each
other
in
punting
time.
That's
sequence,
memory
the
pattern
and
order
in
time
is
essential.
Now,
why
is
that?
Why
do
I
say
that's
the
most
important
memory
component
of
the
of
the
neocortex
well
think
about
what
it
takes
to
recognize.
My
speech,
which
you're
doing
hopefully,
hopefully
I'm,
not
speaking
too
quickly.
A
Yes,
you
have
to
you've
had
to
learn
what
these
words
in
the
English
language
sound
like
in
time
and
how
the
different
variations
are
in
time,
and
you
can
predict
things,
I
might
say
next
and
so
on,
and
it's
sequence
memory.
When
you
touch
things.
It's
the
same
thing:
it's
sequence.
Memories
patterns
the
time.
If
you
mix
up
the
order
of
those
patterns,
it
would
be
completely
on
obviously
what
you're
doing
and
even
vision
and
most
people
think
like.
Oh,
it's
like
a
picture.
I
take
a
picture
and
analyze.
It's
not
true.
A
Your
eyes
are
moving
constantly
three
or
four
or
five
times
a
second
with
each
Socata.
You
move
your
head
and
so
on.
It's
a
massively
changing
stream
of
data,
and
it's
not
just
it's
not
random.
Your
brain
is
controlling
that
stream.
It's
a
sequence
of
patterns
and
your
brains
constantly
make
predictions.
And
finally,
then
your
cortex
is
generating
a
lot
of
your
behavior,
not
all
of
it,
but
a
lot
of
it,
such
as
my
speech,
and
that
is
a
sequence
memory
as
well
I'm
playing
back
patterns
that
I
have
learned
and
practice
and
stories.
A
I
can
repeat:
sentences,
I
can
repeat
sentences
because
it's
stored
and
it's
a
pattern
that
I
can
recall.
So
it's
all
about
sequence,
memory,
not
a
hundred
percent,
but
mostly,
and
that's
the
key
to
unraveling
the
whole
thing.
Finally,
when
you're
not
finally
the
next
ones,
number
four
is:
if
you
look
anywhere
in
the
brain,
even
no
matter
where
you
look
even
the
exiting
the
sensors
or
any
section
of
the
neocortex,
you
find
that
the
activity
is
sparse.
A
A
This
is
not
something
that
brains
do
that
intelligent
machines
don't
have
to
do
and
I'll
get
into
why
that
is
sparse,
distributed
representations,
another
very
surprising
thing
that
occurred,
maybe
in
the
last
15
years
or
so
is
we
used
to
think
there's
part
of
the
neocortex
had
motor
output
with
the
motor
section
that
was
parts
of
essentially
essentially
output.
We
now
know
that's
not
true.
Everywhere
you
look
in
the
neocortex
everywhere
we
have
looked,
we
find
there
are
both
sensory
and
motor.
There
are
neurons
that
generate
behavior
and
there's
neurons
that
are
recognizing
patterns.
A
So
it's
not
like
there's
a
motor
region
in
a
sensory
region.
You
might
see
that
in
some
textbooks
every
region
is
doing
centrally
motor,
which
makes
our
job
easier
because
again
we're
trying
to
understand
the
sensory
motor
aspect,
and
we
can
understand
that
in
each
region
and
then
finally,
there's
an
attentional
mechanism
which
allows
you
to
attend
to
different
parts
of
the
input
stream.
I'm
not
going
to
talk
more
about
that
today
that
these
six
principles
are
both
necessary
and
sufficient
to
create
both
an
intelligent
animal
and
an
intelligent
machine.
A
There's
lots
of
things
that
you
might
have
argued
that
are
not
on
this
list.
I,
don't
put
language
on
this
list.
I
don't
have
I,
don't
see
how
to
have
a
physical
body
even
there's
a
lot
of
things
I
didn't
put
on
this,
but
you
know
every
every
mammal
has
in
your
cortex
and
every
mammal
from
a
mouse
to
humans.
Have
all
these
six
principles
operating
in
their
brain.
So
again,
we're
our
goal
is
to
understand
these
principles,
how
they
work.
We
can
build
artificial
neural.
A
It's
going
to
challenge
you
before
you
have
your
beer
and
then
you
can
relate
it
all
go
I'm
going
to
start
with
sparse
disturbance,
then
we'll
talk
about
sequence
memory
and
then
let's
talk
about
online
learning.
These
are
the
parts
we
understand
the
best.
We
know
something
about
the
other
parts,
but
we,
this
is
the
parts
we
understand.
The
best
is
the
parts
we've
implemented
today
and
tested
extensively.
So
let
me
talk
about
sparse,
distributed
representations.
A
The
best
way
to
understand
sparse
to
stupid
representations
is
a
compare
it
to
what
we
use
in
computers,
which
is
sometimes
called
dense
representations
in
a
computer.
We
take
some
number
of
bits,
maybe
an
8-bit,
byte
or
128
bits
or
64
bits.
We
use
all
combinations
of
the
ones
and
zeros
from
all
zeros
to
all
ones.
An
example
is
like
the
ASCII
code
and
8-bit
ASCII
code.
There's
a
coach
for
the
letter,
M,
it's
a
bunch
of
zeros
and
ones.
A
The
individual
bits
mean
nothing
if
I
said,
what's
the
first
bit
in
an
in
an
ASCII
code,
it's
it's
just
the
first
bit.
It
doesn't
tell
you
what
anything
about
the
letter
M.
In
fact,
if
I
showed
you
that
color,
you
can't
tell
me
what
that
means.
It's
just
it's
an
arbitrary
assignment
from
a
programmer
long
ago,
I
don't
know
who
did
that
and
came
up
with
a
ASCII
code
in
brains.
It's
not
like
that
way
at
all
in
brains,
the
sparse
distributive
resent
ations.
We
we
use
many
many
bits,
then
why
we
say
bit.
A
You
can
also
think
neuron.
Ok,
so
a
neuron
is,
if
it's
active
would
be
a
one,
and
if
it's
not
active
would
be
a
zero
roughly.
You
can
think
of
it.
That
way,
and
so
many
thousands
or
millions
of
these
things
going
on
at
once.
You
can't
have
a
small
representation
with
just
a
few
neurons.
You
have
to
have
many
thousands
before
you
can
do
anything
interesting
and
you
always
find
most
of
them
are
zeros
most
of
the
in
activity.
There's
a
few
ones
so,
typically,
we
use
representations
that
are
about
two
percent
active.
A
So
if
I
had
two
thousand
bits
of
money,
as
my
talk
example
here
today,
I'd
have
forty
one
s
and
1960
zeroes.
Now
the
important
thing
here
is
that
each
bit
means
something
it
has
semantic
meaning.
Now
this
has
to
be
learned.
This
is
not
something
that
you
assign
to.
It
has
to
be
learned,
but
look
let's
pretend
I
could
assign
it
and
I.
Don't
give
me
an
example.
A
So
if
I
wanted
to
represent
a
letter
in
a
sparse
distributive
representation,
I
might
have
bits
to
say
whether
it's
a
fowl
or
constant
I
might
have
its
bits
to
say
how
does
it
sound
as
I'm
an
e,
sound
and
Oh
sound
and
our
sound?
Is
it
have
a
fricative,
sound
or
the
high
pitch
sort
of
SC
sound
to
it?
How
is
it
drawn
does
have
a
senders
or
descenders?
Is
it
open
or
closed?
Where
is
it
in
the
alphabet?
A
What
are
the
meanings
it
might
have
and
so
on
and
when
I
want
to
represent
the
thing
I
would
pick
the
40
best
attributes
for
that
letter.
So
the
representation
of
the
thing
itself
is
the
semantic
representation.
It
tells
you
what
this
thing
is
you
don't
have
to
go.
Look
it
up.
Someplace,
it's
in
the
representation
itself,
there's
no
lookup
table
for
this
all
right
now,
in
brains,
is
always
learned
constantly
in
fact,
well
the
way
the
brain
works.
We
use
the
same
cells
over
and
over
again,
one
moment
is
representing
something.
A
Then
a
different
set
of
those
cells
are
actually
representing
something
else
in
different
regions,
and
we
do
this
in
two
different
regions
of
the
brain.
So
this
this
is
the
basic
idea,
and
then
there
are
some
really
interesting
properties
that
come
out
of
this
one
of
the
first
properties
is,
you
can
ask
by
just
comparing
two
representations
in
seeing
if
they
have
this
bits
in
the
same
location
cells
that
are
active
at
the
same
time
we
can
say
they're,
semantically
similar
and
how
so
we
can
say.
A
Oh
these,
these
share
the
following
attribute
because
they
have
the
same
bits.
Active
and
that's
not
gonna
happen
by
chance.
If
it's,
if
it's
in
a
sparse
representation
that
will
be
meaningful
and
it's
it's
not
a
random
occurrence,
so
we
can
count
on
that.
So
we
can
very
quickly
compare
two
things
and
if
two
things
are
related,
semantically,
even
by
five
bits
or
or
ten
bits,
we
know
that's
a
significant
semantic
overlap
between
them.
A
A
What
we
do
is
we
just
save
the
locations
of
the
of
the
active
bits
we
save
the
locations
are
the
ones
so
if
I
had
40
active
ones,
I
would
have
index
of
40
different
locations,
and
now
I
just
have
to
look
to
see
if
the
new
pattern
has
ones
in
the
same
location,
and
if
it
does
I
say
it's
the
same
pattern
because
they're
all
sparse
they're
all
right
in
the
sparse
and
by
the
way,
it's
a
little
bit
wrong.
It's
still
very,
very
close:
now
we
can
go
one
better.
A
I
can
say
what,
if
I,
don't
let
you
stave
the
locations
of
all
the
one
bits,
but
only
a
subset
like
ten
of
them.
10
of
the
40-
and
you
say
well,
I-
can
make
them
steak
because
I
might
get
ten
of
them
right,
but
the
other
30
would
be
different,
that
we
very
very
unlikely
statistically,
but
even
if
it
did
occur,
it
would
be
making
a
mistake
for
something
that's
semantically
similar,
and
so
that's
not
a
bad
mistake.
A
Sitz
imagine
I
wanted
to
take
I
had
I
had
10
sparse,
distributed
representations
each
of
2,000
bits
each
of
40
ones
and
I
order
them
together.
So
now
we
have
a
new
representation
which
might
have
about
20%
of
the
bits,
active
ones
and
I
can't
undo
that
I
can't
say
what
were
the
10
that
created
this
Union?
You
can't
undo
it,
but
I
can
do
the
following.
I
can
say
here's
a
new
one,
here's
a
new
representation.
Is
it
one
of
the
existing
10?
A
Is
it
one
of
the
original
10
and
you
might
think,
that's
not
possible,
but
it
actually
is
possible.
What
you
all
you
have
to
do
is
again
oops.
You
have
to
go
back
and
look
at
to
see
you
all
in
that
unknown
the
bottom
there
if
the
ones
exist
in
the
Union,
I'm
gonna
say
it's
one
of
the
original
numbers,
and
now
you
could
say
well
that
could
be
a
mistake.
A
I
could
be
borrowing
someone's
from
this
one
someone's
from
that
one
true,
very
unlikely,
and
if
you
did
you'd
be
making
a
Mis
and
error
semantically
similar
to
what
you
were
stored
in
the
Union.
Now
this
you're
gonna
see
where
this
comes
important,
because
brains
actually
make
predictions
all
the
time
you're
constantly
predicting,
what's
going
to
occur,
which
you're
going
to
see
what
you're
going
to
feel
you're
not
aware
of
most
of
these
predictions,
but
you
know
when
something
unusual
happens
in
the
way
the
brain
is
constantly
making
predictions.
A
It's
predicting
multiple
things
at
the
same
time,
and
and
so
it's
forming
these
unions
of
these
things,
but
you
can't
you
can't
say
well
what
are
all
the
predictions
you're
making,
but
you
something
incorrect
happens.
You
know
about
it,
so
that's
the
property
that
comes
out
of
the
sparseness
through
a
representation,
there's
others.
So
this
is,
if
you
want
to
walk
away
from
one
technical
frame
from
this
talk,
you
can
just
remember
this.
Intelligent
machines
are
gonna,
be
built
on
sparse,
disturbed
representations,
there's
no
two
ways
around
it.
Okay,
now
we're
talk
about
sequence
memory.
A
This
is
a
big
innovation
we
had
about
four
years
ago.
It
took
us
a
long
time
to
figure
this
out.
You'll
see
why?
Because
you're,
probably
going
to
get
really
confused
by
Tory,
explain
how
it
works,
but
we
really
doing
you
can
read
about
it
and
in
fact,
that
you
can
get
the
code
for
two
if
you
want.
So
how
do
you
form
sequence
memory
again?
We
need
that
for
inference
in
motor
behavior.
A
So
let's
just
give
you
a
little
neuroscience
talk
here:
okay,
here's
your
near
cortex,
remember:
I,
said
it's
a
really
thin
sheet
of
cells
about
to
to
film
through
to
three
million
six.
If
you
look
in
that
little
section,
there's
a
section
about
three
millimeters
thick,
you
see
these
layers
of
cells.
I'll
come
back
to
this
in
a
moment,
but
all
those
layers
of
cells
have
some
similar
properties
and
one
of
those
properties
not
at
which
layer
you're.
Looking
at
no
matter
where
the
neocortex
you
look
at
you'll
see
this.
A
The
neurons
are
very
densely
in
there.
Those
little
circles
represent
neurons.
Now
the
green
arrow
says
that
the
neurons
in
a
very
small
column,
this
is
not
true
everywhere,
most
parts
of
the
neocortex
they
respond
in
a
very
similar
way,
so
called
a
mini
column
or
a
micro
column,
and
very
brace.
It's
only
thirty
16
microns
wide
and
very
skinny
column
those
cells,
sort
of
the
same
feed-forward
response
property,
so
they're
very
tightly
aligned.
However,
the
vast
majority
of
connections
are
the
cells
nearby,
that's
represented
by
the
orange
align
anywhere
between
fifteen.
A
Ninety
percent
of
the
connections
are
just
cells
nearby
in
nearby
columns,
and
this
is
pretty
common
structure.
Now,
when
you
go
and
you
look
at
a
real
neuron,
that's
a
picture
of
a
real
cortical
neuron,
the
connections.
The
synapses
are
on
these
branch,
like
structures
called
dendrites
and
a
typical
real
neuron
has
several
thousand
to
tens
of
thousands
of
these
connections.
They
don't
actually
connect
to
the
cell
body.
They
connect
to
the
dendrites
and
the
dendrites
are
actual
processing
elements.
A
If
you
zoomed
in
on
the
section
of
a
dendrite
and
there's
a
picture
of
one
about
40
microns
wide,
and
you
can
actually
see
the
synapses
on
there's
a
little
spine,
sticking
up
better,
be
micron
across
there.
What
happens
is
if
those
neurons,
if
those
synapses
become
active,
meaning
an
action
potential
from
another
cell
arrives
if
they
become
active
at
the
same
time
within
a
few
milliseconds,
very
close
to
each
other,
not
very
far
apart.
So
you've
got
a
bunch
of
math,
you
know
like
10
15
20
active
at
the
same
time
near
each
other.
A
Then
it
generates
what
we
call
the
dendritic
spike
and
has
a
very
large
effect
on
the
cell
body.
It
depolarizes
it
if
they
don't
come
in
the
same
time
or
in
different
branches.
Hardly
anything
happens.
So
these
little
dendritic
segments
are
what
we
call
coincidence,
detectors
they're
looking
for
a
coincident
activity
at
the
same
time,
and
if
that
occurs,
bingo
you
got
you
got
a
hit
and
has
a
big
impact
on
the
cell,
so
a
neuron
as
I
mentioned
earlier.
It's
not
a
simple
thing.
It's
actually
a
whole
set
of
coincidence
detectors.
A
Here's
how
we
model
it
I
won't
come
into
all
of
it
here,
but
this
is
one
of
our
model
neurons,
the
cell
bodies,
the
square,
the
blue
synapses,
are
the
ones
that
are
on
the
most
of
the
dendrites,
the
green
ones,
on
your
bus
near
the
cell
body,
not
on
it,
but
you
can
see
we
couldn't.
We
call
those
blue
ones
are
sort
of
coincidence,
detectives
and
we
roaring
them
together.
Then
we
want
to
put
these
in.
A
We
want
to
put
these
in
and
arrays
that
are
similar
to
the
way
you
see
in
a
layer
of
cells,
and
the
question
we
want
to
ask
is:
how
does
that
neuron
and
that
structure
learn
sequences
and
I'm
gonna?
Tell
you
how
and
right
now.
So
this
is
where
it
gets
a
little
tricky.
Okay
here
is
I'm
now
showing
a
picture
of
a
bunch
of
artificial
neurons
from
one
of
our
experiments,
they're
little
cubes,
each
of
those
of
the
neuron
and
the
red
ones,
and
ones
that
are
active,
the
white
ones
that
are
inactive.
A
It's
it's
about
2%,
sparse
activity
here,
so
a
few
relatively
few
red
ones.
Now
this
is
occurring
in
very
large
sheets
of
cells,
so
I'm
just
doing
in
one
little
section.
This
is
not
an
isolated
thing.
This
is
this
one
little
picture
of
a
very
large
sheet.
That's
going
on
now
what
happens
is
and
I'm
not
telling
you
how
we
form
this
sparse,
distribute
representation,
some
easy
ways
to
do
this
using
inhibition.
So
that's
not
part
of
the
talk
today.
I
want
to
talk
about
how
we
learnt
sequences.
A
So
here's
here's
a
time,
one,
here's
a
picture
of
of
what
our
active
cells
might
be
and
then
another
time
we
have
a
different
set
of
active
cells.
Now
this
is
occurring
as
you're.
Listening
to
my
speech,
this
is
occurring
on
the
order
of
milliseconds.
These
changing
patterns
back
and
forth,
back
and
forth
back
and
forth.
I
only
have
two
pictures
here,
but
trust
me.
This
is
going
on
all
the
time
in
your
head
and
the
question
is:
what
does
it
mean
to
learn
this
sequence?
Well,
how's
this
work.
A
Well,
what
happens
is
when
a
cell
becomes
active.
It
looks
for
cells
nearby
that
were
recently
active
near
and,
and
it
says,
look
those
guys
are
just
active
recently.
They
are
a
good
predictor
of
me
becoming
active.
So
if
I
see
those
neighbors
become
active,
I
can
predict
my
own
activity
and
that's
what
every
cell
is
doing
every
cell
when
it
becomes
active
and
they
all
become
of
some
time.
A
A
All
of
them
are
doing
it
in
some
local
area,
and
so
what
will
happen
is
if
you
have
an
input
that
comes
in
so
it's
of
these
red
ones,
all
bunch
of
cells
will
be
in
a
predictive
state
which
in
biology,
is
a
deep
polarized
state
and
those
are
all
the
yellow
ones.
You
say
well
how
come
there's
so
many
yellow
ones,
/
red
ones.
A
Well,
what
if
I
gave
you
the
sequences
a
followed
by
B
and
a
followed
by
C
and
a
followed
by
D,
and
then
I
show
you
a
well
you'll
predict
B,
see,
indeed,
that's
a
union
of
predictions
in
there.
Now
this
as
I've
shown
it
here
is
what
we
call
a
first
order
memory,
meaning
it
only
learns
a
single
transition
from
one
state
to
the
next.
It
can't
stay
well.
I
should
predict
something
different
based
on
a
long
history
of
patterns.
A
You
can
only
look
at
the
very
recent
pattern
and
we
need
a
variable
order
or
a
high
order
memory
for
you
to
understand
speech
for
you
to
do
complex
motor
behaviors.
For
you
to
understand
melodies,
you
need
a
higher
order
memory
where
the
next
prediction
could
be
based
on
something
that
occurred
a
long
time
ago.
So
we
have
to
solve
that
problem
in
a
general
way
and
here's
how
I
believe
the
brain
does
it.
It's
gonna
use
those
those
columns.
A
I
just
showed
you
before
I
had
no
columns,
we're
gonna,
use,
columns
and
here's
the
basic
idea.
Imagine
I
here's
a
sparse
distributing
a
bunch
of
zeros
and
ones
mostly
zeros,
and
instead
of
having
each
cell
be
one
of
those
bits,
zeros
and
ones,
we're
gonna,
let
be
a
column
of
cells
so
for
every
bit,
we're
gonna
have
a
column
and
in
this
case
I
just
show
10
cells
in
each
column.
It's
a
very
abstract
picture
here
and
when
the
column
is
active,
I'm
going
to
pick
one
of
those
cells
just
one.
A
So
that's
that
now
what
if
I
showed
you?
The
exact
same
input
mean
the
exact
same
sparse,
distributive
representation,
but
at
a
different
point
in
time
in
a
different
context.
Well,
here's
this
is
the
same
set
of
zero
and
one's
the
same
columns
are
active,
but
it's
a
different
set
of
cells.
I
could
just
to
pick
these
randomly
there's
a
way
of
doing
this.
It's
just
pretty
simple,
doesn't
have
to
be
different.
It's
just
enough
of
them
have
to
be
different,
and
so
it's
this
point.
A
What
we
have
is
that
the
column
row
level
at
this
level,
those
zeros
and
ones.
We
have
the
exact
same
input
pattern,
but
at
the
cellular
level,
it's
very,
very
different
and
so
I
can
take
the
same
input
in
many
different
contexts.
So
in
this
case,
if
I
had
forty
act
of
columns,
forty
ones
and
I
ten
cells,
four
columns,
there's
10
to
the
40th
ways
to
represent
the
exact
same
input
in
different
contexts,
and
this
is
what
you're
doing
if
I
were
to
say
a
sentence
like
there
are
too
many
tutus
to
count.
A
I
used
the
sound
to
four
times
in
that
sentence,
but
each
time
your
brain
interpreted
it
differently
the
same
patter
coming
in
your
ears,
but
a
different
representation
internally.
So
you
don't
get
confused
same
happens
when
you've
been
listening,
the
beginning
of
Beethoven's
fifth,
you
know
bump
bump
bump
you
don't
get
lost
where
it
is.
Okay,
you'll
just
have
to
trust
me
on
this.
Oh
this
I
just
want
to
show
you
down
here.
A
So
if
I
were
very
simple
sequence
like
ABCDE
and
another
one
like
X,
B,
C,
D,
Y,
and
so
if
I
just
showed
you,
you
know,
ABCD
I
should
predict
E
and
if
I
showed
you
X
BCD
should
predict
Y.
That
means
internally
I
have
to
have
a
different
representation
for
BC
and
D.
Then
in
this
sequence,
and
that's
how
it's
done,
it's
a
different
internal
representation,
but
at
the
same
time
it's
token
I
can
tell
you
it
to
be
in
a
see
anything
okay.
A
If
you
do
all
this
and
you
can
read
about
it,
I'm
not
going
to
walk
you
through
it.
It's
very
simple
set
of
rules.
Trust
me.
It
works
that
you
put
this
in
a
hierarchy.
I'll
end
up
with
a
very
interesting
sequence:
memory,
a
high
order,
sequence
memory.
It's
distributed,
it's
just
ribbing
the
fact
that
nobody
is
in
charge,
there's
no
single
points
of
failure,
it's
distributive
because
it
can
work
over
very
large
sheets
of
cells
because
it
can
be
localized
learning
functions.
A
It's
high
order
is
as
much
as
the
statistics
of
the
data
will
support.
It
can
learning
extremely
long
patterns.
It's
very
high
capacity,
even
a
simple
interpretation
can
learn
millions
and
millions
of
transitions,
and
it's
noise
and
fault.
Tolerant
I
mentioned
that
already
it
can
make
multiple
simultaneous
predictions
and
it
can
do
semantic
generalization
if
I
show
it
a
sequence
of
patterns
that
are
not
exactly
the
same
but
sort
of
similar
to
ones
I've
seen
before
it
will
use
previous
knowledge
and
apply
to
the
new
sequence.
Okay,
all
very
cool
stuff.
A
Now
let
me
go
on
to
I'm
going
to
talk
about
online
learning.
This
is
the
last
technical
part
of
this
talk
here.
So
what
is
online
learning
again.
Remember
we're
talking
about
learning
continuously,
you
don't
get
to
stop.
You
don't
get
to
sit
back
and
relax
and
you
know
think
about
it
again
later:
no
process,
that's
processing.
So
when
novel
patterns
are
coming
into
the
brain
which
occurring
all
the
time,
if
you
think
about
it,
you
never
I
really
have
the
same
pattern
ever
twice
on
your
optic
nerve.
It
never
repeats
exactly.
A
The
same
was
true
of
your
auditory
nerve
or
not
it's
just
it's
so
much
so
big
numbers.
It
just
never
repeats
so
you've
got
you.
Basically
have
to
decide
what's
noise
and
what's
not
noise?
What's
going
to
be
useful?
What's
not
gonna
be
useful,
and
so
you
don't
know
up
front.
You
basically
have
to
start
learning
everything
and
then
slide
later
hey
this
repeats,
it's
useful.
A
We
now
know
that
new
foreign
synapses
can
form
between
two
neurons,
even
if
they're
not
connected
very
quickly
in
a
matter
of
seconds.
In
some
cases,
new
connections
can
form,
and
sometimes
it
takes
a
few
repeats.
So
learning
is
the
growth
of
these
things,
and
that
makes
it
a
much
more
interesting
exercise
because
we
have
a
much
larger
learning
pallet
if
you
will
to
work
from
so
the
way
we
model.
A
This
in
software
is
the
following:
we
model
something
called
the
connection
permanence,
which
is
which
represents
how
much
the
synapse
has
grown
and
how
thick
it
is.
It's
not
a
strength,
it's
how
much
it's
grown
across
and
made
a
connection,
so
we
think
we
model
that
from
0
to
1.
We
then
put
some
threshold
along
the
way.
We
say,
maybe
point
two.
A
We
say:
oh,
if
it's
below
point
two,
we
say
it's
unconnected
if
it's
above
point
two
it's
connected
and
we
give
the
synapse
a
binary
weight,
we
don't
have
to
give
it
a
scale
of
weight
by
the
way
real
synapses
are
very,
very
stochastic.
They
they're
unreliable.
So
you
can
e
neural
model
that
requires
you
know,
even
at
one
digit
of
precision
and
your
synapses
is
not
a
realistic
model.
Okay.
A
So
when
we
train
the
system,
we
train
by
changing
the
permanence,
and
then
we
decided
that
connection
is
established
or
not,
and
if
you
repeat
something
over
and
over
and
over
again,
it
gets
up
to
the
one
scale
here
and
all
that
means
is
it's
harder
to
forget.
I've
repeated
something
many
times
it's
gonna.
A
Take
it
a
lot
longer
to
forget
it,
then
if
you
just
got
it
once
a
choice,
so
that's
how
that's
how
we
model
learning
in
the
brain,
and
we
do
all
this
in
software,
so
we
call
this
whole
thing:
the
cortical
learning,
algorithm
or
CLA,
and
it's
we've
published
a
paper
about
this.
You
can
meet
all
the
details
and
I'll.
Tell
you
moment
this:
it's
in
an
open
source
project.
You
can
look
at
the
code,
so
I
just
want
effort
late.
This
is
not
your
typical
computer
memory.
This
is
not
like
memory.
A
You
find
anywhere
in
a
computer.
Any
silicon
chip
today
is
very,
very
different,
but
it
is
a
building
block
for
the
new,
your
cortex
and
it's
a
building
block
for
machine
intelligence.
It's
not
the
whole
thing,
it's
just
like
piece,
but
it's
a
big
building
block
and
we
think
it's
a
significant
enough
advance
that
we
can
really
start
doing
some
significant
things
with
this.
Ok,
so
just
to
give
you
a
flavor
like:
where
would
this
go?
How
do
you
put
this
together
build
a
brain
type
of
thing?
A
A
It's
if,
for
those
who
know
neuroscience,
is
not
this
simple
but
I'm
not
lying
to
you
either
if
I'm,
trying
to
Kristin
get
to
the
core
essence
of
what's
going
on,
real
brains
are
much
messier
than
this.
Ok,
so
what's
next,
what
are
we
gonna?
Do
next
here,
there's
three
current
directions
that
are
happening:
I'll
just
tell
you
a
little
bit
about
all
three
of
them.
First
of
all,
there's
an
effort
to
commercialize
this
and
that's
what
my
company
is
doing.
I
hope
other
companies
do
that
as
well.
A
We're
producing
a
product
called
grok
we're
changing
the
name
of
the
company.
The
grok
and
I'll
tell
you
about
about
its
predictive
analytics
and
one.
The
reason
for
doing
this
is
because
commercial
value
or
commercial
success
generates
interest
and
investment
dollars.
So
there's
nothing
like
getting
a
profit
motive
for
some
people
to
get
them
going.
The
second
thing
we
have
an
open-source
project
which
we
just
started
a
few
months
ago,
called
new
pic,
which
is
all
the
source
code
for
this
and
the
goal
is
we
want
to
improve
the
algorithms
we
want
to
develop
applications.
A
We
want
to
educate
people
about
that
and
then
finally,
some
there's
some
really
interesting
work
going
on
custom
hardware,
which
is
needed
now.
We
actually
have
to
do
this
because
we're
running
the
limits
of
what
we
can
do
in
software
and
to
do
more
research
and
to
build
different
types
of
commercial
applications.
We're
gonna
need
hardware
acceleration,
and
these
are
some
of
the
companies
were
working
with
on
that
I'll
come
back
to
all
these
in
more
depth.
A
Let
me
just
talk
about
grok
our
product
a
little
bit:
it's
not
a
product,
yet
it's
gonna
be
a
product
soon,
but
let
me
tell
you
what
it
is:
here's
a
sort
of
a
block
diagram
what's
happening
here.
We
have
a
series
of
records
of
data
coming
off
of
machines,
of
service
types
of
the
numbers,
categories,
text
dates
and
times
we
run
them
through
a
set
of
encoders,
which
are
just
like
your
sensory
organs.
A
We
can
do
this
in
a
very
generic
way.
I
want
the
time
to
talk
about
it
today,
but
in
a
very
generic
way
that
I
can
take
any
number
and
it
works
pretty.
Well,
we
modeled
the
cochlea
to
do
that.
What
happens?
Is
you
end
up
after
you
run
these
things
to
encoder?
You
end
up
with
a
sparse
distributive
representation.
What
we
came
from
a
scalar
or
category
or
date
or
time,
then
we
run
those
sparse
assistive
representations
through
the
CL,
a
sequence
memory
and
and
from
that
we
can
make
predictions
to
knowledge.
A
So
again,
the
convert
and
the
encoders
convert
native
data
into
STRs
and
the
CLA
basically
builds
a
model
of
the
data
and
predicts
predictions
and
anomalies.
We've
applied
this
to
lots
of
different
problems,
and
we
finally
said
it
on
one
area
that
we're
going
to
focus
on
so
one
of
the
things
we
started.
Looking
at
we
looked
at,
we
did
a
lot
in
the
energy
space.
We're
not
going
to
pursue
this
now,
but
this
is
an
easy
example
to
show
you
what's
going
on,
it's
a
very
simple
one:
you
everyone
can
look
at
and
go.
A
Oh
I,
get
it
pretty
simple:
here's
the
energy
usage
of
a
building.
Actually
in
France
this
is
the
electrical
energy
usage.
You
can
see
a
five-days,
no
one's
working
on
the
weekend.
It
looks
like,
and
now
the
customer
what
the
customer
wanted
to
do
was
to
make
at
midnight
to
make
a
prediction
for
every
hour
for
the
next
24
hours,
what
the
actual
energy
consumption
would
be
and
there's
these
ways
of
trading
on
this
code.
The
man
responds.
A
If
you
can,
if
you
can
guess
what
the
power
is
going
to
cost,
then
you
can
that's
how
much
you're
gonna
use
you
can
do
things
to
save
money
and
energy.
So
if
you
run
this
through
the
system,
this
is
a
very
simple
examples.
A
couple
variables
coming
in
to
grok
rock
learns
the
patterns.
You
have
to
tell
it
anything
about
what
these
numbers
mean.
There's
no
domain
knowledge
involved
here.
It
just
says:
okay,
there
are
some
patterns.
A
After
a
few,
a
few
iterations,
it
starts
picking
up
the
patterns,
and
so
the
blue
is
the
actual
and
the
red
is
the
predicted.
You
can't
tell
how
well
this
is
doing
by
looking
at
a
chart
like
this
trust
me.
It
looks
pretty
good.
Actually,
in
this
case,
it
was
pretty
good
and
it
can
predict
things
better
than
other
algorithms
that
they,
the
customers,
were
looking
at
much
better.
Here's
the
same
basic
problem,
but
one
that's
not
as
obvious.
This
is
one
where
you're
predicting
server
demand
on
a
cloud.
A
Application
in
this
case
was
video,
encoding
or
video
decoding,
a
large
company
that
does
this
and
the
and
you
can
see
it's
very
peaky
its
and
that's
the
actuals
of
blue
and
it's
really
not
really
predictable.
You
know
there's
lots
of
factors
going
on,
but
it's
like
it's
like
listening
to
music,
where
there's
a
little
snippets
of
songs,
you
kind
of
recognize
in
other
parts
that
you
don't
recognize
and
what
grok
does
is
it
learns
those
snippets
and
it's
in
question
is:
can
you
do
a
better
job
of
prediction
than
anything
else?
A
They've
got
in
this
case?
We
did.
We
do
much
better
than
then
the
current
techniques
they
were
able
to
come
up
with,
because
just
because
it's
good
at
that
kind
of
stuff.
Here's
another
example,
instead
of
looking
at
predictions
that
we're
looking
at
anomalies.
So
it's
the
flip
side,
we're
basically
saying
well,
we
can
predict,
what's
going
to
happen,
let's
see
what,
if
we
didn't
put
it
correctly,
how
do
we
do
that?
I
won't
walk
you
through
how
all
this
works.
It's
very
cool
internally.
A
How
we
can
do
this
because
it's
a
distributed
prediction
and
distributed
anomaly
detection.
So
it's
not
like
this
one
thing,
but
in
this
case
this
is
a
temperature
of
oil
in
a
gearbox
in
a
big
windmill
in
the
North
Sea
up
here
in
the
Baltic
Sea
someplace
up
there,
I
and
and
basically
the
croc
is
modeling
this
data,
and
then
this
is
it's:
how
how
pretty
the
changes
in
predictability
like
what
graças
trying
they
were
saying?
Is
this
just
sound
like
it's
like
a
new
male?
A
Is
this
like
a
new
malady
or
do
new
snippets
of
song
as
I'm
hearing,
or
is
this
something
I've
seen
before
and
it's
and
what
happens
like
at
this
point
here?
They
turn
that
they
turn
the
window
off
and
all
of
a
sudden
grog
says.
Well,
here's
buddy
begin.
We
started
training
the
system
and
says
if
everything
looked
anomalous,
because
there's
new
didn't
know
it,
then
it
says:
okay,
then
they
stopped
the
machine
for
some
reasons.
A
A
We're
gonna
be
we'll
be
launching
that
sometime
early
in
2014,
but
really
trying
to
bring
a
really
deep
science
to
anomaly
detection
in
monitoring
applications
and
servers
that
no
one's
been
able
to
do
before.
Okay,
so
that's
that's
the
commercial
example.
There
may
just
a
bit
about
the
open
source
project.
We
call
this
new
pic
and
you
can
find
it
at
Numenta
org,
it's
right
up
there.
Now
we
have
the
source
code
for
the
CLA
and
bunch
of
other
stuff
up
there.
A
It's
a
single
tree,
meaning
you
it's
the
same
code
we
put
into
our
product.
You
can
actually
join
our
sprint
planning
sessions
if
you
want
to
dial
in
and
participate
in
those
which
is
not
it's
not
a
dual
source
tree
here.
It's
under
available
under
GPL
version,
3,
there's
also
on
the
website,
those
papers
and
videos
and
documentation,
lots
of
put
videos
and
be
doing
talks
like
this.
If
you've,
not
tired
of
this
already
there's
a
community
up
there.
A
We
just
started
this
a
few
months
ago
with
like
three
months
ago,
or
maybe
a
little
bit
less
than
three
months
ago,
we're
very
happy
with
the
response.
So
far
we
have
a
very
active
email
group
is
over
200
subscribers
about
20
plus
messages
a
day
we
have
a
full-time
manager.
This
is
very
important
for
an
open
source
project.
We
just
didn't
throw
it
up
there.
We
were
taking
this
as
a
business
and
very
serious
about
this.
A
So
I'm
very
active
on
this
as
well.
I
am
NOT
this
weekend
because
I'm
here,
but
other
than
that,
I,
usually
am
what
you
can
do.
If
your
interest
in
this
and
I'm
hoping
to
get
a
few
of
you
interested
in
this,
maybe
just
to
you,
know,
think
about
it.
You
can
change
the
world
here.
You
can
get
educated,
learn
this
stuff.
It's
not
easy!
I'll,
be
honest.
A
If
your
takes
a
while
to
really
understand
this
stuff,
deeply,
it's
beautiful
when
you
do
it's
very
clear
and
consistent,
but
it's
a
little
bit
foreign
we're
looking
for
people.
There
are
people
doing
new
applications
for
the
CLS
and
people
coming
up
in
very
clever
ideas.
Already
that
regard
there's
an
interest
in
extending
this
to
robotics
language
and
vision.
Some
of
these
are
harder
than
others.
These
are
very.
These
are
kind
of
research
projects.
Some
clever
people
will
be
ones
to
figure
out
how
to
do
that.
First,
I'm
very
interested
in
the
robotics
component.
A
We
have
some
people
helping
of
people
helping
of
just
with
tools
and
documentation.
That's
helpful
to
people
bringing
in
our
tools
and
making
things
work
better.
We
had
our
second
hack.
We
had
our
first
hackathon
back
in
May
before
we
even
opened
it
up.
It
was
very
successful.
We
were
happy
with
it.
We're
having
our
second
one
in
San.
Francisco
I
realized
that
very
unlikely
that
people
in
this
audience
are
going
to
go
to
them,
but
I
mentioned
it.
There's
going
to
be
we're,
gonna
talk
about
natural
language
processing
and
using
partial
stupid
representations.
A
There's
a
company
in
Austria
called
septa
CEP
T
that
has
done
some
interesting
work
there
and
so
we're
going
to
integrating
with
them.
We're
gonna
have
a
discussion
I'm
going
to
lis
discussion
about
sensory
motor
integration
and
we're
looking
forward
and
looks
like
we
have
some
interest.
One
of
our
contributors
is
helping
us
put
together,
one
probably
in
Ireland
in
some
time,
the
first
half
of
2014
that
hasn't
been
finalized,
but
we
want
to
get
over
to
this
side
of
the
Atlantic.
Okay.
Next
question
here:
custom
hardware
I'm
surprised
how
quickly
this
is
happening.
A
I
thought
that
would
be
a
little
bit
longer.
But
let's
talk
about
what's
going
on
here.
First
of
all,
lots
of
hardware
companies
I
mean
ship
companies,
we're
talking
the
you
know,
the
people
who
build
ships,
not
systems
but
chips,
they're.
Looking
for
something
extra
I
was
at
a
conference
called
the
iske
back
a
year
ago,
and
it
was
amazing
they're
all
talking
about
beyond
von
Neumann,
like
they're,
seeing
the
limits
of
Moore's
law,
they
saying
we
need
that.
We
need
to
apply
our
silicon.
We
can
do
all
these
chip
all
these
silicon.
A
You
know
all
these
transistors
on
chip.
What
are
we
going
to
do
and
they're
looking
for
alternate
architectures?
They
love
the
kind
of
stuff
we're
doing
because
it's
distributed
memory,
it's
naturally
fault
tolerant.
They
can
just
pile
the
bits
in
there
and
not
worry
if
1%
of
them
are
bad,
because
these
algorithms
don't
really
care.
There's
like
the
idea
that
it's
hierarchical
so
there's
a
bunch
of
things
that
get
some
of
these
guys
very
excited.
A
We
need
new
hardware
architectures.
We
running
up
against
speed
problems
today
for
research,
so
we
started
doing
some
simple
vision
systems
we
stopped
because
it
just
took
too
long.
The
experiments
are
running
for
four
days.
You
know
at
the
very
small
systems
we're
using
in
grok.
We
can
do
an
inference
learning
step
in
about
100
milliseconds,
but
when
you
build
more
complex,
hierarchical
systems,
it
slows
down
quite
a
bit.
We
need
it
also
because
there's
there's
a
need
to
reduce
the
cost,
the
power
and
build
embedded
systems.
A
These
are
all
the
standard
reasons
why
we
built
hardware
for
the
last
50
years
for
computers.
The
same
thing
is
going
to
happen
here.
These
are
some
of
the
companies
who
have
started
working
with
us
already
IBM
is
we
have
a
joint
research
agreement
with
Alma's
in
research
labs
in
San,
Jose
California,
that's
the
first
one
we
got
going,
there's
an
initiative
at
DARPA.
I,
don't
know!
If
you
all
know
what
DARPA
is
it's
the
Defense
Advanced
Research.
A
As
well
and
they're,
working
on
a
new
program
called
the
cortical
processor,
it's
not
official
yet,
but
they've
had
multiple
meetings
about
it.
It's
basically
using
the
CLA
as
the
prototype
implementation,
primitive,
it's
not
restricted
to
that.
But
we
are
the
the
example-
and
it's
also
focused
on
hierarchical
temporal
memory,
which
is
the
term
I
came
up
with
a
number
of
years
ago
and
then
there's
there's
an
issue
of
at
C,
great
and
Sandia
Labs.
These
guys
want
to
use
photonics.
This
is
a
very
embryonic.
This
is
the
beginning.
A
If
you
go
back
look
at
the
beginning
in
the
early
computer
history,
people
didn't
know
how
to
make
memory.
They
didn't
know
how
to
make
this
stuff.
They
all
tried
different
things,
but
same
thing
is
happening
right
now.
Okay,
so
I'm
just
got
a
couple
more
slides
here,
oh
I'm,
we're
over
late,
already
I'm
holding
him
from
your
beer.
I'm.
Sorry
I
just
have
two
more
slides
about
future
machine
intelligence.
What
is
it
going
to
look
like?
What
are
we
talking
about
here?
Is
it?
Is
it
like
this?
A
These
evil
things
you
know
the
matrix
Skynet,
the
Terminator
they're
gonna.
You
know
consume
the
world,
or
is
it
like
this?
You
can
have
friendly
robot,
Butler's
and
playing
games
and
putting
on
fun
hats
and
playing
games.
That
way,
you
know,
or
is
it
something
like
this
in
the
middle?
What
just
sort
of
looks
good
and
then
it
turns
bad.
You
know
all
right,
so
here
are
some
things
that
are
definite
they're
gonna
happen.
I
can
say
this
without
a
question:
we
can
make
brains,
artificial
brains
that
are
faster
and
bigger
than
human
brains.
A
I,
don't
send
that
to
all
we
have
to
do.
I'm
happy
to
make
very
small
brains
to
very
small
neocortex
are
very
useful,
but
we
can
build
them
bigger.
You
know,
neurons
can't
do
anything
faster
than
5
milliseconds.
We
can
run
artificial
brains
throughout
it,
a
million
times
faster
than
than
biological
brands,
and
we
can
make
it
much
much
larger.
We
can
be
applied
in
all
kinds
of
senses.
We're
not
is
this
not
about
building
robots
at
human-like
things.
A
We
are
already
applying
this
to
sensory
data,
that's
coming
off
of
computers
and
windmills
and
and
other
things
it
doesn't
have
to
be
vision
and
hearing
and
touch.
In
fact,
it's
not
going
to
be
mostly
that
and
we
can
come
up
all
kinds
of
crazy
sensors
that
work
in
3d
and
high
dimensional
space,
and
so
on.
A
We're
gonna
be
able
to
do
fluid
robotics,
which
has
been
nothing,
is
there's
nothing
like
that
today
in
the
world
and
here's
another
interesting,
our
brains
have
to
all
fit
into
one
skull,
but
remember
it's
a
hierarchy
in
memory
regions
and
there's
no
reasons
why
those
those
regions
have
to
be
co-located.
If
we're
in
a
world
that's
connected,
so
you
can
actually
build
distributed
memory
systems
that
are
where
pieces
of
the
brain
are
in
different
parts
of
the
world,
the
different
parts
of
a
machine
or
organization
who
knows
how
this
is
gonna
play
out.
A
It's
very,
very
difficult
to
know
how
it's
gonna
turn
out,
just
like
it's
very
difficult,
50
60
70
years
ago,
and
how
computers
are
going
to
turn
out.
Here's
something
that
might
happen
may
not
happen.
We
may
or
may
not
create
humanoid
robots.
It's
certainly
not
a
goal
of
mine
to
do
this.
I,
don't
care
about
passing
the
Turing
machine,
I'm
happy
with
a
mouse,
or
you
know
some
robot
exploit
space
or
something,
but
some
people
care
about
this
and
to
make
it
humanoid
like
like
this
kind
of
thing.
A
Aren't
you
r2
that
c-3po
scuse
me
I'm,
not
a
big
science
fiction,
guy
CC
field,
you
know
you
need
to
have
all
the
other
parts
of
the
brain
to
you
have
to
have
dinner
to
make
it
human
like
it's
got
a
little
human
life
and
have
human
emotions,
and
we
don't
need
to
do
that
for
most
intelligent
machines.
We
may
or
may
not
have
computer
brain
interfaces
for
all
people,
certainly
for
people
who
are
disabled.
This
could
be
very,
very
important,
but
you
know
I.
Are
we
gonna
plug?
A
A
People
who
argue
for
this,
the
cingulate
Ariens
whatever
it
is,
they
just
don't
know
enough
about
neuroscience.
You
can
talk
to
me
later.
It's
not
gonna
happen.
There's
not
gonna
be
evil
robots
taking
over
the
world.
You
know
the
only
dangerous
thing
in
the
world
is
self-replication,
that's
dangerous
brains
are
not
dangerous
and
these
things
don't
care.
You
gotta
unplug
them.
They
don't
care.
It's
just
some.
It's
just
memory
and
it's
not
get
carried
away
about
this
whoops.
Okay.
So
last
slide.
Why
should
we
do
this?
Why
should
you
care?
A
Why
are
you
here
there's
two
reasons
why
we
should
create
intelligent
machines?
The
first
is,
we
can
make
our
life
better
on
this
planet.
Just
like
computers
have
improved
our
lives,
arguably,
and
it
very
very
dramatically
in
medicine
and
transportation
and
communications
and
numerous
ways
we
can't
no
one
could
imagine
seventy
years
ago
made
life
on
this
planet
better.
We
can
make
life
using
brains
better.
Don't
ask
me
how
it's
gonna
happen.
I,
don't
know
all
the
details.
I
can
speculate.
A
No
one
can
really
predict
this,
but
I
can
tell
you
it's
gonna
happen
and
it's
an
important
thing
to
do.
We'll,
look
back
in
time
and
see
what
that
I'm
glad.
We
did
all
that,
but
I
think
even
more
important
to
me
is
you
know
my
guess:
what's
the
purpose
of
life,
what
why
don't?
What
should
I
try
to
do
in
my
lifetime
and
I
only
got
three
number
years
here
and
less
left
than
you
guys
do
most
of
you
least,
so
you
know
to
me:
it's
all
about
knowledge.
A
It's
about
gaining
knowledge,
accumulating
knowledge
and
sharing
it
in
future
generations.
If
we
don't
know
what
it's
all
about,
we
just
better
keep
learning
and
so
I
think
we
can.
We
can
build
machines
that
really
learn
in
many
dramatic
ways.
I
already
mentioned
they
can
be
faster
and
they
never
get
tired.
We
could
create
physicists
or
mathematicians
that
work
24
hours
a
day,
a
million
times
faster
than
us
and
never
get
tired.
You
know
never
get
diverted
by
the
opposite
sex
or
anything
like
that.
A
You
know,
and
and
also,
if
we're
going
to
explore
the
universe.
I'm.
Sorry
to
say,
you
know,
unless
you
want
to
take
that
one-way
trip
to
Mars,
it's
not
gonna
be
humans
doing
it.
You
know
the
only
way
we're
really
gonna
be
able
to
do.
This
is
we're.
Gonna
have
to
send
out
machines
that
are
intelligent,
that
can
handle
the
contingencies
of
things
that
go
out
and
as
I
explore
the
universe.
So
to
me
this
is
ultimately
what
it's
all
about.
It's
about
accumulating
knowledge
and
learning
more
about
ourselves
and
I.