►
Description
Subutai reviews the paper "Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer" and compares it to our dendrites paper "Avoiding Catastrophe: Active Dendrites Enable Multi-Task Learning in Dynamic Environments".
Paper: https://arxiv.org/abs/1701.06538
Dendrites Paper: https://arxiv.org/abs/2201.00042
- - - - -
Numenta is leading the new era of machine intelligence. Our deep experience in theoretical neuroscience research has led to tremendous discoveries on how the brain works. We have developed a framework called the Thousand Brains Theory of Intelligence that will be fundamental to advancing the state of artificial intelligence and machine learning. By applying this theory to existing deep learning systems, we are addressing today’s bottlenecks while enabling tomorrow’s applications.
Subscribe to our News Digest for the latest news about neuroscience and artificial intelligence:
https://numenta.com/news-digest/
Subscribe to our Newsletter for the latest Numenta updates:
https://tinyurl.com/NumentaNewsletter
Our Social Media:
https://twitter.com/Numenta
https://www.facebook.com/OfficialNumenta
https://www.linkedin.com/company/numenta
Our Open Source Resources:
https://github.com/numenta
https://discourse.numenta.org/
Our Website:
https://numenta.com/
A
Okay,
so
this
this
is
the
paper.
I
thought
I'd
go
over,
it's
a
precursor
to
the
paper
that
we
originally
posted
on
designing
effective,
sparse
models,
but
that
paper
I
thought-
was
really
confusing
and
not
terribly
well
written
and
it
sort
of
re
relied
on
knowing
this
paper
anyway,
and
I
think
this
one
is
well
written,
it's
fairly
straightforward
and
it's
it's
pretty
relevant
to
stuff
we're
doing.
I
think
it
even
in
the
on
the
sparsity
side,
it's
relevant,
but
it's
definitely
irrelevant
to
the
dendrites
work.
A
So
I
guess
kevin
is
here,
so
he
can
potentially
there's
a
bunch
of
sort
of
interesting
performance
improvements
they
do
in
here.
Okay,
so
the
title
is
outrageously
large,
neural
networks,
sparsely
gated
mixture
of
experts,
layer
and
you
know
when
whenever
we
do
the
dendrites,
where
people
always
bring
up,
oh,
is
it
just
like
a
mixture
of
experts,
and
it
kind
of
is
like
that.
It
kind
of
isn't-
and
I
think
this
work
is
much
more
directly
like
a
mixture
of
experts
thing,
but
it's
very
relevant.
A
A
So,
of
course,
this
is
very
relevant
to
our
sparsity
work.
Where
conditional,
if
you
don't
do
things,
if
you
do
things,
the
naively
conditional
computation
can
occur
and
that
sort
of
destroys
the
performance
challenges
and
they
sort
of
point
that
out
here
where
there's
in
practice.
However,
there
are
significant
algorithmic
and
performance
challenges.
A
That's
a
pretty
bold
claim,
it's
kind
of
true
when
they
do
it,
but
there's
some
caveats,
but
it's
kind
of
interesting
what
they,
what
they're
doing
so,
what
they
do
is
they
introduce
a
sparsely
gated
mixture
of
experts,
layer,
moe,
consisting
of
up
to
thousands
of
feed
forward
sub
networks
and
to
do
the
way
they
do.
This
is
they're
going
to
have
a
trainable
gating
network
which
determines
which
sparse
combination
of
experts
we're
going
to
use
for
every
example
on
a
per
example
basis.
A
A
Five
years
ago,
so
it
was
pretty
large
back
then.
Nowadays
this
is
not
considered
large,
so
we'll
go
through
the
model
and
how
they
and
a
little
bit
how
they
get
these
performance
improvements.
I
didn't
have
time
to
go
through
the
experiments
at
all
and
I
don't
think
the
experiments
are
actually
as
interesting
as
the
model
itself
and
how
it
relates
to
our
work.
So
this
may
be
very
different
from
most
machine
learning
journal
papers.
A
A
So
here
in
the
introduction
again,
they
say
various
forms
of
you
know.
Conditional
computation
have
been
proposed
as
a
way
to
increase
model
capacity,
and
then
they
list
a
bunch
of
papers
which
I
actually
wasn't
aware
of
at
all
and
might
be
quite
interesting
for
our
related
to
our
den
rights
work
and
basically,
they
say
is
in
these
in
these
schemes.
Large
parts
of
a
network
are
active
or
inactive
on
a
per
example
basis,
so
that
sentence
is
true
for
our
dendrites
work
as
well.
A
Although
the
way
they
these
guys
do,
it,
I
think,
is
quite
different
than
the
way
we're
doing
it,
and
then
they
say
the
gating
decisions
can
be
binary
or
sparse
or
continuous,
stochastic
or
deterministics.
There's
all
sorts
of
variations
on
this,
so
all
of
that
makes
it
seem
very
interesting
and
relevant
to
the
den
rights
work
so.
D
A
Okay,
so
yeah,
it
might
be
useful
to
go
back
to
some
of
those
original
ones
as
well,
because
the
way
attention
ended
up
is
is
somewhat
different,
but
maybe
in
the
beginning,
it's
very
relevant
to
to
generate
yeah.
So
yeah
I
mean
the
words
they're
saying
sound,
a
lot
like
what
we
say
for
for
dendrites.
A
E
A
A
So
this
is
stuff
that
kevin
and
and
lawrence
have
talked
about
quite
a
bit
and
and
so
the
previous
works,
you
know,
try
to
turn
on
and
off
large
chunks
of
the
network,
with
each
gating
decision
to
avoid
sort
of
fine-grained
branching
at
the
same
time,
on
gpu,
especially
large
batch
sizes
are
critical
for
performance,
but
conditional
computation
reduces
the
batch
size
for
the
conditionally
active
chunks
of
the
network.
So
this
is
very
interesting
point
and
they
this
is
the
piece
they
actually
say
they
solve
in
in
this
in
this
paper.
A
A
So
that's
they
have
a
solution
to
that.
Here.
Talk
about
network,
then
another
interesting
thing
is
here:
they
they
say,
loss
terms
may
be
necessary
to
achieve
the
desired
level
of
sparsity
per
chunk
or
per
example,
and
they'll
talk
a
little
bit
about
this
here,
and
this
can
impact
model
quality
so
we'll
get
into
that
a
little
bit
later.
A
But
their
main
motivation
here
is
really
performance
is
really
to
improve
the
overall
efficiency
of
these
networks
and
by
by
doing
that
they
can
greatly
increase
the
size
of
the
networks.
So
that's
their
motivation,
which
is
a
little
different
than
let's
say
what
we're
doing
with
dendrites,
where
we're
trying
to
avoid
interference
and
we're
applying
it
to
continue
learning
and
multitask
learning
and
so
on.
So
here
their
their
motivation
is
purely
performance
and
increasing
the
sizes
of
these
networks.
A
A
They
introduced
the
idea
of
using
multiple
mystery
of
mixture
of
experts
with
their
own
gating
networks
as
parts
of
a
deep
model,
and
here
the
mixture
of
experts
is
the
whole
model.
So
what
they're
doing
here
I'm
guessing
is
that
it's
what
they're
saying
is
a
top-level
mixture
of
experts.
You
have,
you
know,
let's
say
a
hundred
completely
separate
networks
and
at
the
very
top
they
do
a
sum
or
weighted
sum
of
the
mixtures
to
get
the
actual
output.
So
that's
more
like
a
typical
mixture
of
experts.
A
A
A
And
then,
typically
this,
this
g
here
would
not
be
sparse,
but
what
they
mentioned
is
that
if
g
is
sparse,
then
whenever
g
is
zero,
we
can
you
can
skip
computing
that
that
expert
completely
okay.
So
if
you
have
a
thousand
experts-
and
only
you
know-
20
of
them
are-
are
going
to
be
on
or
not
have
non-zero
g's.
A
Okay,
so
that's
kind
of
what
they
saying
at
the
next
thing
here
in
our
experiments,
we
have
up
to
thousands
of
experts,
but
we
only
need
to
evaluate
a
handful
of
them
on
every
for
every
expert
and
then
they
talk
about
having
a
hierarchical
mixture
of
experts,
which
basically
means
you
have
multiple
layers.
So
I'm
not
really
going
to
go
into
that,
but
it's
just
layers
of
these
mixture
of
expert
things,
but
you
can
understand
everything
by
thinking
of
a
single
layer.
A
So
to
introduce
so
what
is
this
g?
So
it
all
boils
down
to
what
is
this
gating
function
getting
network
doing,
and
so
what
they
do
is
so
the
first
intuition
is
saying:
okay,
well,
a
simple
choice
of
a
gating
function
which
is
non-sparse
would
be
to
use
soft
max.
So
you
take
the
input
here.
Multiply
it
by
some
weights.
Do
a
soft
max
of
it
and
that
can
be
your
gating
function.
A
A
A
A
F
C
D
A
D
C
A
A
A
A
C
C
C
D
A
A
Take
off,
I'm
not
sure
you
the
order
matters
that
much
okay
so
and
then
they
train
the
whole
thing
with
that
prop,
so
this
is
kind
of
their
their
model
and
I
kind
of
drew
a
little
diagram
of
it
to
help
us
think
about
this
versus
the
dendrites
layer.
So
this
is
the
sparse
mixture
of
experts
layer,
so
you
get
the
input
coming
in.
A
I
wish
there
was
like
a
pointer
here.
Oh
there
is
oh
okay,
oh
interesting.
So
what
I'm
sharing
this
pointer
thing
shows
up.
That's
pretty
cool,
okay!
So
there's
you
have
the
input
and
it's
this
is
the
feed
forward
portion
of
the
network.
So
this
is
just
a
standard
network,
except
here
what
I'm
doing.
I
should
have
drawn
these
in
squares
these
each
of
these
are
different
exports.
So
they're
not
necessarily
individual
units,
these
could
be
actual
full
networks,
so
they're
going
so
you
have
the
feed
forward
input
going
through
there.
A
You
know
part
of
the
input
goes
through
here
and
and
through
that
noise
function,
that
noise
thing
gets
added
to
a
linear
weighted
sum
of
the
input
here
then
they
take
a
k
winner,
take
all
here,
then
through
the
softmax,
and
then
that
output
is
multiplied
against
the
output
of
the
of
the
mixtures,
and
then
you
get
the
final
output.
Okay,
so
again,
here
most
of
these
values
will
be
zero.
Only
k
of
them
will
be
non-zero
and
so
coming
out
of
here,
you
only
get
k,
non-zero
things,
so
I
drew
it
this
way.
A
A
similar
they
kind
of
this
is
also
an
interesting
way
to
to
draw
it
here.
Is
you
have
an
input
coming
in
negating
network,
which
is
that
whole
thing
on
the
left?
The
blue
thing
that
I
showed
the
difference
here
is
that
they
have
an
arrow
that
goes
from
the
gating
network
to
the
feed
forward,
and
the
reason
for
this
is,
if
anything
is
zero
from
the
gating
network.
A
You
can
completely
skip
those
experts,
so
in
this
example,
only
to
the
the
out
of
the
gating
vector,
that's
coming
out
only
position
2
and
position.
N
minus
1
have
non-zero
values.
Everything
else
is
zero,
so
as
they're
showing
here,
you
only
need
to
send
the
input
to
these.
Two
experts
only
need
to
compute
those
two
and
then
do
the
do
the
multiplication-
and
I
guess
I'd
forgotten
this
in
my
diagram,
but
then
they
add
everything
together
before
you
get
the
final
output.
So
this
is
like
a
standard
expert
system
thing.
A
A
A
Yeah,
you
know
what
they're
doing
here,
so
the
softmax
will
keep
a
have
a
lot
of
values,
a
few
values
near
one
right,
yeah,
and
then
everything
else
will
be
near
zeros.
What
they
could
do
is
you
know
quite
commonly
in
a
mixture
of
exports.
They
could
add
a
weight
value
for
each
expert
here
and
then
this
could
literally
be
zero
one.
At
that
point,.
D
A
Yeah
same
with
our
dendrites
thing,
so
so
this
I'm
contrasting
their
network,
my
drawing
of
their
network
with
what
we're
doing
here
is
very
similar.
It's
just
things
are
sort
of
flipped
a
little
bit.
As
you
guys
know.
We
have
the
feed
forward
component.
Then
the
input
goes
to
okay.
So
here
suppose
we
have
one
dendrite
per
unit.
We
can
put
the
dendrites
out
here.
A
You
know
we
do
a
linear
weighted
sum
of
the
input
pass
it
through
a
sigmoid
and
then
that's
multiplied
by
the
value
of
these
things,
and
we
do
a
k
winner
take
all
afterwards,
so
I'm
not
sure
there's
a
so
we're,
not
adding
noise.
So
that's
a
big
difference
other
than
that.
I'm
not
sure,
there's
a
huge
difference
between
this.
G
A
Okay,
I'll
get
to
that
in
a
second,
that's
a
very
good
question.
They
do
mention
issues
with
their
k
winner
to
call
so
I'll
get
to
that
in
a
second
but
yeah
so
other
than
that
big
plus.
At
the
end,
you
know,
I
think
the
biggest
difference
here
is
that
each
expert
is
thought
of
as
a
whole
network.
So
each
of
these
is
in
here.
I
should
have
really
drawn
these
as
squares,
because
they're
they're
thought
of
as
entire
networks.
A
D
D
A
If
statements
were
free,
conditionals
were
free.
First
of
all,
kevin
wouldn't
have
a
job.
Second
of
all,
second
of
all,
if
they
were
free,
then
then
this
this
would
save
tons
of
computation
by
doing
the
gating
network.
First,
but
again
it
relies
on
the
fact
that
expert
these
experts
are
much
heavier
than
the
gating
networks.
G
And
it
allows
them
to
also
scale
because,
when
you're
looking
at
at
clusters
of
nodes
computational
nodes,
if
you
can
you
do
a
course
hack
at
the
thing
you
rather
than
at
the
at
the
fine
grain
level,
you
can
see
how
it
can
be
just
more
efficient
to
say.
Okay,
we
only
send
a
few
of
these
models.
We
activate
only
a
few
of
these
models
at
a
time
when
we
distribute
them
all
across
all
these
nodes.
A
Did
you
read
the
paper
already?
I
know,
okay.
Well,
that's
exactly
their
system,
I
think
so.
I
think
you're
you're
you
hit
on
exactly
what
how
they're
going
to
scale
that
very
close
to
what
you
just
said
is
how
they're
going
to
scale
the
system-
okay,
yeah.
So
let's,
let's
get
to
that.
Okay.
So
what
they're
doing
so?
First
of
all,
the
the
problem
that
we
alluded
to
earlier.
Is
this
shrinking
shrinking
batch
problem.
A
Well,
let's
say
you
have
a
single
single
gpu.
You
have
the
copy
of
the
network
there.
If
the
gating
network
chooses
k
out
of
n
experts
for
each
example,
so
let's
say
out
of
a
thousand
experts
is
choosing,
let's
say
two
percent
or
20
at
a
time,
then,
for
a
batch
of
b
examples,
each
expert
is
receives
a
much
smaller
batch
of
approximately
this
many
examples
right
so
they're
only
getting
two
percent
of
the
examples
on
average,
and
this
is
going
to
be.
A
You
know
on
this
batch
only
this
guy
and
this
guy
is
being
let's
say:
only
k
equals
two.
Only
these
two
are
actually
being
computed
here.
Only
these
two
are
actually
being
used,
and
here
only
these
two
are
actually
being
used,
but
you're
using
up
the
memory
for
all
of
these
for
everything,
even
though
only
a
tiny
piece
of
this
is
actually
being
used,
is
it
useful
for
use
for
computation,
and
so
this
becomes
very,
very
inefficient
and
the
larger
the
number
of
experts
the
more
inefficient
it
gets.
A
A
A
In
the
in
the
you
have
a
bunch
of
different
gpus,
you
have,
let's
say
one
gpu
per
expert,
and
so
this
one
is
going
to
get
expert
one.
This
one's
gonna
get
expert.
Two,
this
one's
gonna
get
expert.
Three,
this
one's
gonna
get
expert;
okay,
and
now
what
you
do
is
you
you
have
this
outer
loop
conditional
and
you
only
send
in
the
data
here
that's
going
to
be
relevant
for
expert
one.
So
you
have
some
outer
loop.
I
don't
know
what
they
would.
A
Okay,
so
the
way
yeah,
so
let's
say
expert
one-
is
gonna.
Look
at
batch.
You
know,
example
number
one
example
number
five
example
number
seven,
it's
gonna
not
need
anything
else,
so
what
you
would
send
in
here
would
be
example,
one
example:
five
example:
seven
okay,
so
this
so
this
gpu
is
being
fully
utilized
and
this
expert
is
going
to
be
fully
busy
processing
this
guy-
and
you
know
over
here-
you
might
send
in
you
know-
example
two
example.
A
A
A
They
say
it
is
our
goal
to
train
a
trillion
parameter
model
on
the
trillion
word
corpus.
We
have
not
scaled
our
systems
this
far,
but
it
should
be
possible
by
adding
more
hardware
and,
as
we
know,
google
has
done
a
truly
a
parameter
model
at
this
point,
and
so
I
think,
and
I
believe
they're
using
this
technique
to
do
it.
If
I
remember
correctly,.
C
A
E
A
Yeah
so
lucas,
I
don't
know
if
that
was
cleared
up,
but
it's
if
this
is
the
batch
again
on
the
on
the
right
for
a
given
input.
Only
this
expert
and
this
expert
is
going
to
process
it.
Everything
else
is
going
to
be
zero.
If
k
equals
two,
everything
else
is
going
to
be
zero.
So
there's
no
point
sending
you
know
this
expert,
this
data
point
because
it's
not
going
to
compute
it
so
instead
what
they
will
do
is.
A
A
A
A
A
D
A
G
A
Yeah
yeah
it's.
This
is
like,
let's
think
about
imagenet.
This
x1
is
a
full
image
and
what
this
matrix
says
is
just
which
expert
is
going
to
look
at
each
image.
So
if
you
look
at
image
number
one
only
expert
two
and
five
are
gonna
process
it.
So
why
on
why?
Why
do
you
need
to
send
it
to
expert
one
at
all.
A
A
A
A
That
the
gating
network,
which
includes
a
k
winner
trichol,
tends
to
converge
to
a
state
where
it
always
produces
large
weights
for
the
same
few
experts,
this
imbalance
is
self-reinforcing
as
the
favored
experts
are
trained
more
rapidly
and
thus
selected
even
more
by
the
gating
network.
So
this
is
exactly
the
problem
we
have
in
our
k
winner,
take
all
without
boosting
right,
so
this
will
make
it
make
these
batches
very
imbalanced
right
and
so
what
they
do
is
they
have
something
almost
identical
to
our,
not
identical.
A
It's
the
same
idea
same
the
spirit
of
the
idea
is
the
same,
but
they
do
it
as
a
loss
function,
so
they
basically
compute
the
importance
of
each
expert
by
looking
at
the
sum
of
the
gating
values
across
across
the
batch,
and
so
if
the
getting
value
for
expert
one
is
really
high,
that
means
expert
one
is
being
chosen
all
the
time.
I'm
sorry
if
the
importance
is
yeah.
If
the
expert
value
for
the
gating
and
the
gaining
value
for
expert
one
is
constantly
high
importance
will
be
high
for
that
expert.
A
That
means
that
expert
is
being
is
winning
winning
a
lot
and
it's
being
chosen
quite
a
lot,
and
so
what
you
want
to
do
is
you
want
to
spread
out
the
importance
things
so
they're
as
equal
as
possible
right
if
everyone,
if
every
dating
value,
getting
I'm
sorry
if
every
expert
had
the
same,
getting
value
some
upgrading
values.
That
means
that
everyone's
being
chosen
approximately
equally,
and
so
they
try
to
minimize
the
variance
of
this
importance
vector
and
they
have
a
weight.
So
they
try
to
this.
A
E
A
E
A
They
still
got
even
with
this.
They
still
get
experts
which
get
different
numbers
of
examples.
They
have
a
second
loss
function,
which
I
didn't
look
at.
We
can.
We
can
look
at
that
if
you
want
together
but
which
ensures
balance
loads,
but
at
the
end
of
the
day
I
think
they
might
just
oh
well.
I
guess
we
can
look
at
that
right
now.
A
A
Yeah
they
want
to,
they
want
to
learn
noise
values
and
I
think
such
that
the
load
is,
is
really
balanced,
and
you
can't,
since
the
number
of
examples
received
by
an
expert,
is
a
discrete
quantity,
it
cannot
be
propagated.
Instead,
they
have
an
estimator
upload
and
I
didn't
quite
read
through
this.
I
had
to
go
to
the
lab
test.
I
write
that.
D
G
A
G
C
A
This
is
the
same
as
our
k,
winner
takeoff.
We
just
have
zero
here
instead,
and
they
even
mentioned
that
you
know,
while
this
form
of
sparsity
creates
some
theoretically
scary
discontinuities
in
the
output,
we
have
not
yet
observed
this
to
be
a
problem
in
practice:
okay
and
then
all
right,
yeah
and
then
this
term
helps
with
load
balancing,
as
we
just
discussed.
So
I
think
they
could
have
put
the
you
know
they
could
have
put
the
soft
max
inside
and
just
use
rk
winner
take
all
on
the
outside.
G
A
A
D
A
C
Yeah
another
thing
I
was
thinking
about
in
the
light
of
what
you
just
explained,
they're,
trying
to
save
computation
if
you're
using
a
soft
mac,
especially
if
it's
like
a
low
temperature
soft
max
a
lot
of
those
networks
are
going
to
be
attribute
like
a
very,
very
low
value.
Is
there
even
a
point
of
including
them
in
the
competition
or
do
they
use
like
some
sort
of
cutoff?
C
D
A
A
D
A
A
Yeah
so
here
they're,
showing
how
the
not
having
read
this
is
showing
how
the
perplexity
of
the
error
decreases
as
a
function
of
the
number
of
parameters.
So
the
more
more
experts
you
add,
the
more
you
can
get
and
I
did
not
really
look
into
the
hierarchical
moas,
but
this
there,
you
can
get
more
of
these
experts
running
each
expert
had
about
one
million
parameters.
Okay,
so
this
experts
are
much
much
bigger
than
the
single
unit.
A
A
C
They're
good,
I
was
thinking,
hey
it's
interesting.
This
is
a
mixture
of
aspects
of
transformers
and
each
each
of
them
is
already
composed
of
several
heads.
I
wonder
if
you
could
just
do
that
in
the
head.
Instead,
if
you
insert
like
the
key
winners,
if
you
just
have
a
lot
of
heads
and
then
you
put
some
key
winners
there
before
the
soft
max,
if
you
could
kind
of
replicate
the
same
idea
well,.
A
E
G
G
You
know
you
can
kind
of
do
this.
You
know
I
move
stuff
to
the
first
level
of
the
hierarchy
and
then
fan
it
out
from
there.
So
you're
not
shoveling
from
one
node,
a
massive
number
of
batches.
You
know
and
then
trying
to
source
from
just
one
single
node
out
to
say
you
know
a
million
instances
of
of
gpus.
You
know
to
be
kind
of
gross
about
that.
So
I
think
just
from
a
pure
information
routing
point
of
view,
it
makes
sense
to
divide
and
conquer
in
that
fashion.
A
G
G
One
question
I
had
is
whether
they're
able
to
when
they
when
they
finally
trained
the
thing
and
they
had
their
you
know-
were
they
able
to
come
up
with
the
fact
that
yeah
you
know
we've
been
able
to
identify.
Simply
you
know
these.
You
know
500
problem
domains
and
for
each
one
of
those
domains.
There's
always
this
cluster
of
four
that
that
we
pick
and
and
they're
somewhat
similar.
A
B
C
Do
you
see
a
parallel
of
of
this?
With
with
the
brain
I
mean
it's,
it's
not
a
far-fetched
question,
but
in
dendrites
you're
not
really
like
saving
compensation
away,
you're
just
deciding
on
the
unit
or
not
it's
going
to
be
active
case.
You
know
beforehand
whether
you
need
this
whole
region
of
the
network,
like
this
whole
sub-network,
and
then
you
decide
whether
or
not
you're
going
to
pass
the
input
through
that.
Do
you
see
any
vehicles
with
the.
A
Brain
yeah
well,
but
well,
ben,
put
on
the
chat
that
this
is
a
little
bit.
You
can
think
of
each
expert,
maybe
like
a
cortical
column
and
processing
stuff
in
parallel,
so
you
can
maybe
think
of
it.
I
think
it's
a
little
bit
different,
because
each
cortical
column
will
get
a
different
subset
of
the
input
itself,
so
different,
visual
cortical
columns
will
process
different
parts
of
the
visual
field,
but
there
is
routing
going
on
and
there
is
you
know
we
have.
We
think
there's
voting
going
on
and
stuff
like
that.
A
So
there
is
some
potential
parallel
yeah,
but
I
I
don't
think
we
get
the
same.
I
don't
think
we
have
a
situation
in
the
brain
where
you
have
multiple
networks
that
are
processing
the
identical
input
and
then
they're
just
some
like
a
true
mixture
of
experts.
I
think
the
situation
is
more
like
what
we
have
in
the
dendrites,
where
you
select
some.
You
have
this
big
large
network
and
you
select
subsets
of
that
distributed.
Subsets
of
that.
D
E
E
C
A
Yeah
so
there's
stuff
like
so
there's
attention,
you
know
so
there's
covert
attention
which
might
focus
the
computation
on
some
subset
of
the
cortical
columns
and
tell
the
other
cortical
columns
to
sort
of
ignore
it.
So
you
could,
you
could
think
of
this
as
somewhat
similar
to
biological
covert
attention.
C
A
E
A
Yeah,
but
it's
it's,
it's
never
a
case
it
okay.
It
is
a
little
bit
different,
though,
if
it's
attention,
what
we
think
is
going
on
is
that
the
brain
is
saying:
I'm
expecting
some
input
here.
So
it's
there's
a
you
know
some
area
of
the
visual
field.
Let's
say
where
you
are
expecting
input
and
so
you're
you're
predicting
input
there
and
then
you'll
process
it.
But
it's
not
like
input
elsewhere
is
ignored
completely,
so
it
is
diminished.
A
So,
but
if
something
really
something
happens
outside
of
the
focus
of
attention,
it
can
actually
draw
your
attention
to
that.
Right,
that'll
be
a
surprise,
and
so
it's
not
exactly
this
way.
There's
you
know,
expert,
1,
3
and
n.
Here
are
not
like
sitting
idle.
They
are
still.
They
may
be,
processing
less
or
maybe
they're
diminished
in
some
way.
But
if
something
dramatic
happens
there
they
will
notice
it
and
you
know
it'll,
be.
A
I
think
it's
it's
hard
to
draw
too
many
analogies
to
the
brain
here
and
the
brain
kind
of
like
lucas
was
saying
it's
ben.
You
were
saying
it's
not
a
batch
system,
it's
really
we're
streaming
and
we're
embodied
we're
doing
we're.
You
know
acting
in
the
world
and
it's
a
very
it's
hard
to
draw
too
many
analogies
here.
I
think.
G
C
Yeah,
but
just
I
disagree
with
one
point:
that's
not
fully
batching
because
you
can
even
see
online
as
patching.
If
you
look
at
like
over
the
time,
dimension,
you're
getting
I
don't
like
100
inputs
in
a
second,
and
you
could
see
that
as
a
badge,
because
there
is
a
limit
of
how
much
energy
you
can
dedicate
how
much
units
you
can
actually
activate
without
like
going
overheating.
So
that
is
a
batch
in
a
way.
I.
E
C
Yeah,
no
because
if,
if
you
look,
if
you
look
at
silicon
right
and
if
it's
a
temporal
bag,
you
don't
have
that
sort
of
energy
ratio,
you
can
just
process
all
of
them,
but
the
brain
can't
right.
You
have
that
limit
of
how
much
energy
you
can
spend
in
a
given
like
minute
or
whatever
how
many
units
you
can
activate
so
sure
it
is
a
batch
that
yeah,
the
temporal
dimension
could
be
fought
as
a
patch.
E
C
Yeah,
my
point
is:
maybe
okay
and
then
I'm
just
like
going
over
the
board.
It's
just
theorizing,
but
my
point
here
is
that
maybe
you
receive
inputs
in
a
greater
frequency,
then
you
can
actually
process
them
right.
So,
let's
work
with
this,
so
you
receive
like
100
different
inputs
in
a
minute,
but
you
can
only
process
10
of
10
in
a
minute,
so
you
got
to
choose
so
so,
like
quarter
column
can't
process
100
can
only
process
10..
C
E
F
A
Well,
maybe
one
maybe
one
way
to
think
of
it.
Is
you
know
if
you
think
about
machine
learning,
you
have
a
whole
batch
of
learning.
You
do
multiple
forward
passes
through
a
batch,
and
then
you
do
one
learning
step
right.
So
the
the
rate
at
which
you
learn
is
dependent
on
the
batch
size
in
the
brain,
you're
learning
on
every
input
that's
coming
in,
but
there
are
also
much
slower
forms
of
learning
too.
So
you
know
the
extreme
one
is
in
overnight.
You
consolidate
memories
and
the
next
day
you
learn.
A
You
know,
you've
learned
a
bunch
of
stuff,
and
so
you
could
think
of
there's
some
abstract
batch
of
the
entire
day
of
experience
that
somehow
stored
somewhere
in
your
brain
that
you
consolidate
overnight
and
learn
from
so
it's
nothing
is
as
in
the
brain
is
as
hard
as
fast
as
in
in
machine
learning.
But
if
you
think
of
batching,
as
as
related
to
the
time
scale
of
learning
you
you
could
imagine,
there's
you
know
something
like
a
rough
batch
being
saved
on
the
brain.
C
A
A
E
There's
also
just
the
time
constant
of
neurons,
so
neurons
can
only
respond
to
their
sensory
inputs
so
fast,
and
you
know
a
typical
neuron
has
a
membrane
time,
constant
of
say,
20
milliseconds,
so
you're
already
windowing
over
time,
and
that's
why
I
would
say
it's
sub
sampling
instead
of
of
batching,
but
the
point
that
it's
being
economical
with
its
energy
expenditure
in
multiple
ways.
I
still
agree
with.
C
G
I
I
think
the
writing
point
is.
Is
that
if,
if
you
say
you
can't
process,
you
know
you
can
only
process
10
out
of
100,
you
either
winnow
it
down
or
you
have
to
buffer
it,
and
then
you
have
to
propose
a
buffering
mechanism
that
allows
it
to
re-examine
that
you
know
whether
it's
recirculate
on
something.
C
But
is
it
different,
though,
like
let's
say
I
have
10
columns
associated
with
one
patch
right
and
then
I
get
the
first
sensor
assent
to
one
color
column.
Then
I
get
the
second
but
the
first
column
associated
to
that
patch
is
still
processed.
Now
I
send
to
the
second
one,
then
it's
all
really
buffering.
I
I'm
still
routing
it's
just
that,
since
the
first
column
is
busy
and
sending
to
the
second
one,
and
then
when
I
get
the
third
present.
Second.
G
G
You
know
you
know
things
from
time,
one
time
two
time
three
and
they
get
aggregated,
and
you
can
do
something
off
of
that
so
yeah
that
that
would
be
a
form
of
delayed
memory,
which
is,
I
said,
but
you
have
to
you
have
to
show
that
is
that
is
what's
going
on
there
against.
You
know
how
much
you
know,
how
much
do
we
have
in
at
the
cortical
column,
the
notion
of
working
memory,
or
is
that
a
higher.
G
A
E
A
Yeah,
let's
think
about
that
yeah.
So
the
receptive
field
is
like
the
yeah
it's
defined
at
the
level
of
a
neuron
like
what
does
it
respond
to
in
the
input
and
so
like
you
might
have
a
neuron
inside
a
cortical
column
that
looks
at
a
vertical
response
to
a
vertical
edge.
Other
neurons
in
the
same
cortical
column
might
respond
to
horizontal
edges,
or
you
know,
diagonal,
edges
or
whatever
and
so
they're,
but
they
would
all
be
looking
at
the
same
visual
area
to
determine
whether
it's
a
horizontal
or,
let's.
A
D
A
A
Again,
it's
all
it's
distributed.
You
know
the
next
one
will
be
overlapping,
but
you
can
think
of
it
as
as
one
it's
like
a
convolutional
network.
Like
one
you
know.
Let's
say
you
have
a
convolutional
kernel
that
has
a
five
by
five
kernel
coming
in
and
you
have
a
stride
of
three
right,
so
any
given
part
of
the
input
is
primarily
going
to
one
but
another
cortical
another.
A
C
A
C
D
A
G
G
Input
at
global
primary
input,
if
you
consider
that
there's
a
hierarchy
there
going
v1,
v2,
v3
and
the
receptor
the
quote-unquote
receptive
field,
then
spans
multiple
inputs,
it
it
does
span
that,
but
not
within
the
one
cortical
column,
I
think,
is
you're.
Thinking
of
you
know,
the
the
notion
is
that
the
direct
primary
input
to
the
first
level
of
cortical
column
is
is
going
to
be
segmented
somewhat
from
from
the
rest
of
them.
But
after
that
you
know
who
knows
how
many
things
talk
to
each
other.