►
Description
To learn more about Brains@Bay, visit our Meetup page https://www.meetup.com/BraIns-Bay
Brains@Bay Meetups focus on how neuroscience can inspire us to create improved artificial intelligence and machine learning algorithms. In this special edition we focus on the function of lateral connections (connections between neurons within a level). Long-range lateral connections are ubiquitous in the neocortex and cannot be explained by pure feedforward models. In this meetup, researchers from the Allen Institute for Brain Science discuss their recently published paper on modeling these connections.
The speakers are Stefan Mihalas, Ramakrishnan Iyer, and Brian Hu. The title of their talk is “Lateral Connections can Perform Contextual Integration in Cortical and Convolutional Neural Networks." They present a network model of cortical computation in which the lateral connections from surrounding neurons enable each neuron to integrate contextual information from features in the surround. They show that adding these connections to deep convolutional networks in an unsupervised manner makes them more robust to noise in the input image and leads to better classification accuracy under noise.
Link to paper: https://www.frontiersin.org/articles/10.3389/fncom.2020.00031
Link to meetup: https://www.meetup.com/BraIns-Bay/events/270695139/
A
Hi,
everyone
welcome
to
our
brains
that
pay
meet
up.
We're
glad
to
have
you
here
so
brings
that
bay
is
a
machine,
learning
meets
neuroscience
made
up
and
it
started
in
bay
area,
california,
but
it's
now
online
and
the
idea
is
to
discuss
brain
spar,
machine
learning,
algorithms,
and
we
want
the
perspective
from
both
neuroscientists
and
machine
learning,
researchers
and
we're
trying
to
draw
inspiration
from
neuroscience
and
use
that
inspiration
and
things
we
can
learn
from
neuroscience
in
order
to
improve
machine
learning
algorithms.
B
B
B
C
So
hello,
everyone.
I
am
going
to
talk
today
about
one
particular
line
of
study
which
we
recently
published
and
we
are
going
to
focus
on
one
type
of
connections
which
we
observe
throughout
the
brains
which
are
the
lateral
connections,
and
we
have
an
idea
as
to
how
this
can
be
involved
much
more
generally
in
cortical
computations.
C
The
outline
of
the
talk
today,
I
am
going
to
talk
a
little
on
an
introduction
which
is
much
more
general
as
to.
Why
are
we
interested
in
cortex
and
how
is
it
organized
I'm
going
to
present
our
one
of
our
general
hypotheses,
which
is
the
idea
that
there
isn't
just
one
optimization
function
for
neuronal
circuits,
but
there
are
multiple
optimization
functions
at
different
levels.
C
I'm
going
to
talk
very
briefly
one
or
two
slides
about
the
theory
which
is
going
to
say
how
can
we
implement
such
computations
ram
is
going
to
talk
about
how
some
of
these
theoretical
predictions
map
these
observations
and
brian
is
going
to
talk
about.
How
can
we
take
these
type
of
ideas
from
biology
and
apply
them
in
the
context
of
artificial
neuronal
networks,
and
we
dubbed
the
network
cnn
x
or
convolutional
neuronal
network
with
extra
classical
receptive
fields?
C
So
to
begin
with,
why
is
cortex
interesting
to
me
and
why
study?
One
of
the
things
which
I
was
always
very
puzzled
by
the
cortex
is
the
incredible
diversity
in
term
of
science
regarding
evolution.
So,
for
example,
you
can
see
that
you
can
have
it's
not
as
if
big
brains
are
necessarily
or
big.
Cortex
is
necessarily
a
big
evolutionary
pressure
which
is
going
to
monotonically
increase
throughout
evolution,
for
example.
C
C
The
short-range
connections
are
much
more
lawful.
There
is
a
microstructure
which
is
relatively
similar
across
area.
It's
not
perfect.
I
would
say
that
these
are
images
which
are
going
to
be
as
an
as
far
as
it
goes,
and
not
even
that
it
is
similar
across
areas.
It
has
a
non-trivial
level
of
similarities
across
across
multiple
species.
C
C
C
Different
and
I
would
say
more
complicated
from
some
perspective,
simplers
from
others,
one
which
is
going
to
be
the
fact
that
there
are
so
many
inhibitory,
neuron
subtypes
such
that
beyond
the
fact
that
we
need
to
think
about
what
these
neurons
represent.
What
is
their
function?
We
need
to
think
about
their
cell
type
and
there
have
been
don't
have
the
capacity
of
doing
a
full
literature
review.
Regarding
this.
There
are
many
works
which
have
characterized
this
pretty
well.
C
I
would
recommend
the
paper
from
andreas
tullias
from
science
2015,
one
from
massimo
scanziani
from
2013,
in
which
they
characterized
the
logic
among
inhibitory
neurons.
I
want
to
say
that
these
are
generally
relative
simplification.
So,
if
you
remember,
when
we
talked
about
characterizations
of
cell
types,
I
think
that
from
an
pure
cell
type
classification,
we
can
talk,
maybe
about
50
inhibitory
cell
types
in
just
in
brain
area.
V1.
C
However,
most
of
the
time
when
people
talk
about
these
types
of
connectivity,
they
are
limiting
to
usually
three
cell
classes.
They
come
with
different
names,
for
example
from
tullius.
They
characterized
them
as
pyramidal
targeting
internal
neurons,
inhibitory,
specific
interneurons
and
master
skanziani
prefers
to
use
terms
which
are
dependent
on
more
traditional
cell
type
classifications,
based
on
some
of
the
proteins
which
are
expressed
by
the
cells,
for
example,
somatostatin
expressing
cells,
parvalbumin
expressing
cells
or
vip
expressing
cells.
C
It's
not
while
it
is
relatively
complicated.
It
is
also
not
random.
I
think
that
some
of
these
ideas
of
hierarchical
organizations
again
have
been
present
in
the
literature
for
quite
some
time.
Very
influential
line
of
studies
has
been
done
by
phalman
and
vanessa,
and
this
is
the
famous
van
essen
diagram
of
the
hierarchical
organization
of
the
visual
of
the
areas
in
macaque
to
describe
these
streams
and
one
of
the
things
which
I
want
to
make
a
link
between
neuroscience
and
computer
science
is
that
hierarchy
does
not
mean
feed
forward.
C
D
C
But
at
the
same
time,
very
very
clearly
hierarchically
organized
both
at
the
level
of
areas
which
we
have
and
at
the
level
of
modules,
with
sensory
types,
modules
and
association
modules
being
present
at
relatively
different
levels.
But
then,
even
if
we
zoom
inside
of
one
of
these
areas,
we
still
see
a
hierarchy.
C
E
B
E
C
Sure
the
hierarchy
index
would
classify
how
many
of
the
connections
perfectly
align
with
the
idea
of
the
hierarchy
within
an
area.
So
let's
imagine
that
if
I
were
to
have
three
areas
and
all
the
connections
between
them
would
be
perfectly
lawful
that
from
level
one
all
the
connections
would
be
feed
forward.
All
the
connections
which
we
measured
would
be
feed
forward
to
level
two
from
level.
Two.
All
the
connections
would
be
feed
forward
to
level
three
from
level.
C
Then
what
we
would
have
in
terms
of
a
hierarchy
index
we
would
have
level
one
would
be
in
here
we
put
across
the
average
level.
One
would
be
at
minus
one.
The
second
one
would
be
at
zero
and
the
third
one
would
be
a
plus
one.
Okay
got
it
now.
If
we've
got
a
particular
cell
type
or
a
particular
type
of
connection
which
we
measure
which
does
not
fit
these,
we
look
for
every
area.
C
What
fraction
of
connections
correspond
to
the
lawful
description
of
it
forward
versus
feedback
versus
how
many
don't,
if
you're,
going
to
have
a
connection
which
is
going
to
move
laterally
or
a
connection
which
does
not
respect
these?
These
are
going
to
give
us
the
these
are
going
to
give
us
the
fraction
of
the
fractional
index.
C
Which
is
present,
but
that
that
backbone
we
can
extract
mathematically
if
we
do
lots
of
measurements,
but
that
doesn't
mean
that
there
isn't
the
there
are
lots
of
connections
which
do
not
necessarily
follow
that
to
the
law.
That's
much
more
bio,
that's
much
more
biological
than
physics,
where
you
can
describe
a
law
and
then
you're
going
to
have
a
lot
of
a
lot
of
violations.
C
C
Based
on
that
there
was
a
segmentation
and
we
picked
up
things
which
look
like
axons
and
we
computed
on
a
whole
brain
connectivity
model.
But
this
is
not
like
an
em
model
where
you
look,
one
cell
goes
to
the
other,
but
rather
statistical
which
is
going
to
tell
you
where
one
cell
type
in
a
particular
area
is
going
to
project.
C
We
did
an
unsupervised
learning
algorithm
to
classify
feed
forward
versus
feedback
connections
by
simply
assuming
that
you
can
think
of
this
as
a
one-dimensional,
embedding
in
which
the
type
not
the
strengths
of
the
connection,
defines
the
directionality
such
that
we
end
up
classifying
brains,
not
lots
of
people
like
to
classify
the
connect
to
characterize
the
connections
in
the
brain.
As
a
graph,
I
would
say
that
that
is
in
our
hands.
That
was
relatively
difficult.
C
C
So
in
order
to
make
this
work,
since
I
just
provided
an
example
of
object,
recognition
as
an
example,
our
desire
would
be
to
characterize
local
computations,
which
help
arbitrary
global
computations.
We
don't
want
necessarily
to
put
in
to
pre-build
the
to
pre-built
the
global
with
the
global
computation.
We
want
to
make
it
to
make
it
arbitrary
and
behind
this
I'm
actually
going
to
ask
the
audience
in
here
to
guess
what
is
behind
the
red
square.
C
You
have
a
prior
knowledge
because
you
have
seen
the
because
some
people
have
seen
this
stock
before
and
a
bird
would
be
a
very
good,
I'm
not
going
to
present
some
of
the
future
work.
We
have
that
there
are
some
of
those
descriptions
in
terms
of
trying
to
make
somewhat
more
complex
priors
for
the
time
being,
I'm
just
going
to
use
a
very
simple
prior,
which
is
going
to
be
just
a
back
and.
C
More
moose
sounds
good.
The
general
idea
behind
this
is
that
there
is
a
tremendous
amount
of
information
which
is
present
in
the
context
of
natural
images
and
whatever
you
want
to
do.
Are
you
a
hunter
going
to
shoot
it?
Are
you
a
photographer
to
just
try
to
get
it
or
even,
if
you
don't
necessarily
need
to
identify
it
to
to
try
to
see?
Are
you
try
to
either
try
to
think
as
an
from
any
perspective,
the
function
which
you
need
to
apply
to
this?
C
But
we
can't
make
it
absolutely
anything
for
us.
We
started.
We
need
to
have
a
relatively
extensive
set
of
assumptions
and
I'm
going
to
present
here
the
most
important
ones.
I
can
point
to
a
few
others
as
we
go
through,
but
the
important
element
which
we
need
is
that,
whatever
that
computation
is,
it
has
to
use
very
simple,
retro,
a
very
simple
representation,
which
is
the
idea
that
the
neurons
represent
probabilities
of
features
being
present
in
a
localized
part
of
the
world.
C
For
the
mass
we'll
need
to
make
use
of
the
independence
of
features
of
probabilities
of
features
coming
from
patches
which
are
going
to
be
around.
So
how
am
I
going
to
describe
the
mathematically
pointing
this
is
going
to
be?
The
fact
that
the
notation
which
I'm
going
to
use
fc
is
the
classical
is
the
firing
rate
caused
by
the
classic
by
presentation
of
stimulus
in
the
classical
receptive
field.
C
C
C
C
We
can
expand
this
for
many
patches
surrounding
the
current
one
and
use
a
little
and
use
a
little
bit
of
mathematics
to
go
and
say
what
is
going
to
be
the
dependence
of
one
feature
being
present
inside
of
the
image,
given
the
firing
rates
of
all
the
surrounding
neurons,
and
given
this
formula,
this
looks
very
much
like
an
artificial
neuronal
network.
If
we
are
going
to
take
this
nasty
term
and
consider
it
to
be
a
synaptic
weight,
we
map
these
elements
to
firing
rates
of
to
firing
rates
of
the
neurons.
C
There
is
only
one
problem
that
this
is
not
your
standard,
artificial
neuronal
network,
that,
at
least
for
our
perspective,
we
would
need
lateral
connections
to
be
present
which
have
an
multiplicative
term.
We
can
use
an
approximation
by
doing
a
taylor
expansion
to
do
this
with
additive
terms
as
well.
Now,
how
does
this
particular?
How
does
this
particular
term,
which
I
called
something
which
can
be
presented
as
a
weight?
C
C
So
this
is
a
theory
at
the
moment
again,
let
me
let
me
recapitulate
where
we
started
with
general
idea
of
computation
and
we
make
predictions
about
connections
without
any
type
of
measurements.
These
are
the
only
things
which
are
going
to
be
needed,
so
this
is
a
parameter
free
model
coming
from
a
hypothesis
of
computation,
and
this
can
be
implemented.
C
B
C
C
If
you
begin
to
put
the
dependence
on
the
lateral
one,
you
can
begin
to
think
you
can
get
more
illusory
percepts,
because,
because
of
the
question
of
independence,
there
is
so
I'm
I'm
not
saying
that
biology
needs
to
be
like
this.
I'm
saying
that
the
math
works
really
well,
if
that,
if
that
is
the
case,
I
mean
the
math
works.
If
that
is
the
case,
but
at
the
same
time
there
would
be
if
we
were
to
replace
this
not
by
a
modified
heavy
learning,
but
rather
more
of
a
true
heavier
learning.
C
B
C
F
Yeah,
okay,
so
just
checking
yeah
okay,
so
stefan
has
set
it
up
nicely
for
me
to
be
able
to
describe
how
we
took
this
theoretical
slash.
You
know
theoretical
formalism
or
this
network
model,
and
what
I'll
primarily
be
talking
about
is
how
we
took
this
network
model
and
used.
You
know:
unvivo
experimental
data
from
mouse
visual
cortex
to
be
able
to
try
and
compare
how
well
the
predictions
from
our
model
match.
Some
of
the
experimental
observations
that
that
you
know
stefan
presented
a
few
slides
ago.
F
In
terms
of
you
know
what
the
specificity
of
connections
looks
like
you
know
what
the
distance
dependence
of
connections
looks
like
and
so
on.
So
to
begin
with,
I
I'll
just
briefly
describe
the
data
was
collected
by
our
experiment.
Colleagues,
at
the
allen
institute,
we
were
involved
in
part
of
this
study
and
you
know
being
able
to
understand
how
these
recordings
are
done
and
being
able
to
help
them
analyze
the
data.
F
So
we
took
this
data
and
we
computed
the
average
receptive
field
properties
of
all
neurons
that
were
recorded
as
part
of
this
experimental
setup
and
using
that
we
came
up
with
a
simple
parameterization
for
how
we
would
like
to
represent
our
v1
neurons
in
the
mouse
visual
cortex,
essentially
what
it
what
it
boils
down
to
is
that
we
can
simply
parameterize
them
for
from
the
data
that
we
collected.
It
seems
that
we
can
simply
parameterize
them
as
sums
of
two
gaussian
blobs
of
opposite
polarities.
F
We
chose
to
move
them
around
in
steps
of
45
degrees
based
off
of
the
data
that
we
had
and
using
this
sort
of
an
approach.
We
constructed
a
bank
of
filters,
18
filters
that
represent
mouse,
v1,
neurons
putatively,
eight
of
which
are
shown
here.
There
are
eight
more
with
the
opposite
polarities,
where
the
center
is
white
and
the
surround
is
dark,
and
then
you
have
two
subfields
which
are
purely
on
or
off
meaning
they
are
just
white
or
black
in
the
center.
F
The
orientation
preference
of
these
viva
neurons,
if
you
will,
is
determined
by
the
angle
of
the
line
joining
the
centers
of
these
two
subfields.
So,
for
example,
the
second
filter
here
would
represent
a
neuron
that
prefers
45
degrees,
gratings,
okay.
So,
given
that
we
had
this,
we
had
this
database
of
filters.
The
other
ingredient
that
stefan
had
mentioned
was
the
natural
scene
statistics
that
we
need
in
order
to
be
able
to
apply
our
theory.
F
So
a
natural
database
for
us
was
the
database
of
natural
scenes
coming
from
the
berkeley
segmentation
data
set
shown
here
is
one
representative
image
from
this
data
set
and
what
we
do
is
using
our
assumption
of
the
neuronal
code,
which
is
that
the
firing
rate
represents
the
probability
of
a
neuron
as
a
first
step.
In
that
what
we
do
is
we
take
each
of
our
filters
in
our
filter
bank
convolves.
F
The
images
present
in
this
data
set
with
those
filters
get
an
output
like
this
and
then
eventually
we
convert
this
to
a
firing
rate
of
a
neuron
which
we
then
use
to
calculate
our
weight
matrix
w
the
formula
for
which
stefan
had
just
presented
a
couple
of
slides
ago.
So
you
just
run
this
algorithm
through
and
you
end
up
getting
a
synaptic
weight
matrix
coming
from
the
modified
hepian
learning
rule
that
ends
up
being
four
dimensional.
In
our
case,
the
respective
dimensions
being
in
this
representation
that
I
show
here.
F
F
And
then
you
look
at
the
connections
that
this
filter
receives
from
these
eight
filters
that
are
shown
in
the
top
panel
here.
So
once
we
fix
k1
and
k2,
then
what
you
are
left
with
is
a
2d
grid
of
synaptic
weights,
which
is
shown
in
the
bottom
row
here
as
a
heat
map,
and
what
each
of
these
sub
panels
represents
is
what
the
connections
look
like
from
each
of
those
source
filters
to
the
target
filter.
That's
shown
at
the
leftmost
panel,
some
interesting
things
that
pop
out.
F
So
we
have
constructed
this
four
dimensional
matrix
and
now
we
want
to
be
able
to
compare
it
to
experimental
data.
So
one
of
the
first
ingredients
that
we
want
to
look
at
is
does
our
theory
predict
the
like
to
like
connectivity
patterns
that
have
been
observed,
experimentally
from
the
lab
of
thomas
mercedes
logo,
and
turns
out
very
nicely
that
our
model
predicts
the
like
nature
of
the
excitatory
connections
quite
well.
What
you
see
here
is
tree
of
connection
between
two
filters
whose
preferred
orientations
are
offset
by
delta
theta.
F
That's
shown
on
the
x-axis,
so
what
this
says
is
that
filters
will
have
this,
which
have
the
same
orientation
are
more
likely
preferentially
to
connect
to
each
other
and
as
the
preferred
orientation
as
the
difference
in
preferred
orientation
decrease
increases
between
them.
The
probability
of
connection
goes
down.
F
B
F
Just
those
are
just
a
spatial
grid,
so
we
chose
them
to
be
15
by
15
pixels.
Now,
with
roughly
a
correspondence
that
maps
on
to
the
actual
approximate
size
of
receptive
fields
that
are
seen
in
our
experimental
data
so
give
or
take
the
standard
deviation
of
these
turns
out
to
be
around
7
degrees
and
the
net
size
of
these
filters
turns
out
to
be
approximately
15
degrees.
So
that's
why
that's
what
corresponds
to
the
x
and
y
dimensions
of
those
filters
over
there.
A
F
F
Yeah,
so
in
this
case,
in
this
case,
the
way
this
was
done
is
in
the
experimental
paper.
They
fit
a
gaussian,
a
one-sided
gaussian
to
the
distance,
and
then
they
use
the
standard
deviation
of
the
fit
gaussian
to
estimate
a
spatial
constant.
Basically,
so
we
did
the
same.
We
we
took
the
same
approach
that
was
done
by
that
experimental
lab
in
order
that
we
are
able
to
compare.
F
F
F
B
F
F
These
are
connections,
functional
connections
between
excitatory
neurons,
so
we
know
that
biologically
inhibitory
neurons
exist
in
the
cortex,
and
one
now
can
ask
a
question
as
to
how
one
can
start
thinking
about
introducing
inhibitory
cells
into
the
model
over
here
right.
The
idea
here
being
that
most
of
these
negative
connections
that
you're
seeing
are
most
likely
polysynaptic
connections
that
go
from
excitatory
neurons
via
an
inhibitory
neuron
onto
other
excitatory
neurons.
So
the
net
effect
of
the
interaction
between
those
two
excitatory
neurons
will
be
like
an
effective
inhibition.
That's.
F
Idea
right,
the
other
aspect
of
interest
here,
of
course,
is
that
we
know
that
there
is
a
multitude
of
cell
types
in
the
visual
cortex
granted.
There
are,
you
know,
different
ways
of
what
we
mean
but
different
ways
of
defining
what
we
mean
by
cell
type,
as
stefan
had
summarized
briefly
from
our
studies
right,
but
we
were
interested
in
seeing
if
we
can.
Actually,
you
know,
go
in
and
map
some
of
these
computations
onto
specific
components
corresponding
to
excitatory
and
independent
cell
types
in
our
model.
F
They
had
presented
a
very
nice
way
of
being
able
to
think
about
how
one
one
can
start
thinking
about.
Modeling
inhibitory
neurons
in
you
know,
normative
models
such
as
efficient
coding,
sparse
coding.
What
have
you
right,
but
the
crux
of
their
work?
I
can
get
into
the
mathematical
details
for
people
who
are
interested
later.
I
won't
directly
present
it
here,
but
the
crux
of
their
work
was
that
they
wanted
to
introduce
inhibitory
neurons
into
a
network
model
of
sparse
coding
in
such
a
way
that
you
can
represent,
but
you
can
respect
biological
constraints.
F
The
three
constraints
specifically
were
dale's
law,
which
says
that
you
know
excitation
and
inhibition
come
from
distinct
cell
types
and
each
of
these
cell
types,
just
so
inhibitory
cell
types
can
only
provide
inhibition
to
post-synaptic
cells.
Excitatory
can
provide
only
excitation
to
postsynaptic
cells
right.
You
cannot
mix
them.
F
One
way
of
doing
this
would
be
able
to
take
the
connectivity
matrix
that
you
get
from
your
model
and
split
it
into
just.
You
know
two
pieces
corresponding
to
a
positive
part
and
a
negative
part,
and
then
you
can
assign
them
separately
to
e,
and
I
types,
however,
there
is
another
constraint
which
is
most
frequently
we
have
you
know.
A
lot
of
experimental
data
has
shown
that,
roughly
speaking,
the
ratio
of
excitatory
inhibitory
neurons
in
different
brain
areas
is
approximately
forest
one
right.
F
So
one
needs
to
respect
this
constraint
and
just
splitting
it
into
a
positive
and
negative
would
mean
you
have
equal
copies
of
excitation
and
inhibition,
which
we
don't
want.
So
one
way
to
go
around
that
and
reduce
the
number
of
inhibitory
neurons
in
your
model
would
be
to
look
at
something
like
a
lower
angle:
composition
of
your
connectivity
matrix
all
right,
and
so
that's
one
way
where
you
can
say:
okay,
the
lower
rank
part
now
tells
you
that
you
have
a
lot
more
excitatory
neurons.
F
One
sort
of
thing
that
is
seen
prominently
or
a
number
of
studies
have
seen
this
phenomenon.
I
don't
think
it's
a
settled
issue,
but
there
definitely
seems
to
be
a
tuning
diversity
and
orientation
tuning
diversity
amongst
nematode
neurons,
in
that
there
are
subclasses
of
interneurons
that
don't
seem
to
have
any
preferred
orientation
tuning,
whereas
there
are
other
interneurons
which
seem
to
show
very
strong
preference
for
gratings
of
particular
orientations
right.
F
So
one
way
that
they
propose
to
capture
this
tuning
diversity
of
individual
neurons
is
to
do
a
low
rank,
plus
a
sparse
decomposition
of
your
matrix.
Okay,
the
idea
here
being
that
these
this
decomposition
will
allow
you
to
have
a
part
that
you
know
captures
an
untuned
component
of
the
inhibition
and
then
the
sparse
component
ends
up
capturing
a
tuned
component
of
the
inhibition,
won't
go
into
details
here,
but
I'll
just
leave
you
with
you
know
a
summary
sort
of
figure
from
their
paper,
the
basic
idea
being.
F
If
you
see
the
picture
on
the
right,
you
have
two
populations
of
inhibitory
neurons.
The
first
population
receives
non-specific
input
from
all
excitatory
neurons
and
sends
back
projections
to
all
excitatory
neurons,
whereas
the
second
one
receives
input
only
from
a
subset
of
the
excitatory
neurons,
and
this
is
how
it
ends
up
being
orientation
tuned.
F
You
can
decompose
this
and
now
split
up
these
decomposed
pieces
separately
into
positive
and
negative
aspects.
Right
then,
now
you
can
start
saying
that
there
are
different
pieces
to
your
types
of
inhibition,
an
untuned
part
a
tune
part,
and
then
there
is
a
net
excited
trip
part.
This
is
sort
of
a
very
nice
study
that
they
had
published
in
2015,
and
we
used
this
idea.
F
Our
our
computational
goal
was
different.
It
was
not
sparse
coding,
it
is
optimal
context
integration,
but
in
principle
the
same
sort
of
idea
can
be
applied
to
our
framework
in
order
to
be
able
to
start
thinking
about
what
different
aspects
of
innovation
are
there.
How
do
we
start
splitting
up
our
weight
matrix
into
you
know
specific
cell
types.
F
So
on
this
slide,
I
just
summarize
that
when
we
do
this
sort
of
decomposition
on
our
weight
matrix,
you
can
see
again
that
the
sparse
part
looks
very
similar
to
the
actual
full
weight
matrix
that
I
had
shown.
But
the
sparse
component
ends
up
showing
a
lot
more
of
you
know.
Localized
features,
and
you
know
just
provides
us
an
additional
handle
to
be
able
to
look
at
these
different
components.
F
The
turns
out
that
the
excitatory
connections
shown
in
red
here
natural
with
the
experimental
data
shown
on
the
left,
whereas
the
inhibitory
components
show
sort
of
interesting
differences.
There
is
a
tuned
component
of
the
inhibitory
component
and
untuned
component
coming
from
the
sparse
part,
and
these
are
you
know
things
that
can
be
tested,
experimentally
with
further
refined
studies,
hopefully,
which
will
come
up
in
the
future
yeah.
This
slide
just
summarizes
that
qualitatively
the
distance
dependencies
are
also
similar.
F
F
So,
to
summarize
my
part
of
the
talk
we
we
took
this
model.
We
took
a
database
of
natural
scenes
and
parameterized
data
from
mouse
electrophysiological
recordings,
and
we
were
able
to
take
that
applied
to
the
database
of
natural
scenes,
obtain
a
connectivity
matrix
and
then
potentially
propose
mapping
of
some
of
these
computations
in
the
network
model
as
being
mediated
by
different
types
of
inhibitory.
Neurons,
where
you
know
you
have
a
local
normalization
that
is
done,
and
that
is.
F
We
we
are
just
proposing
as
one
proposal
here
that
they
could
be
mediated
by
parvolube
and
interneurons,
whereas
the
surround
inhibition,
which
is
known
where
ssd
interneurons
are
known
to
be
involved,
could
potentially
be
mediated
by
true
stock
populations
of
ssg
neurons.
However,
I
would
like
to
mention
in
general
that
there
might
not
be
a
direct
correspondence
between
computations
and
actual
cell
types.
This
is
just
you
know,
one
of
the
simpler
proposals
that
we
could
come
up
with
in
our
in
our
in
our
published
work.
D
B
B
F
So,
yes,
we
are
not
read
it
to
just
using
gaussian
blobs.
You
have
actually
applied
the
theory
to
different
types
of
filters
in
zoo
and
roselle.
They
were
primarily
you
know.
They
were
primarily
interested
in
using
gabor
filters.
As
a
way
of
you
know,
it's
a
standard
sort
of
assumption
that
neurons
in
let's
say,
cats
and
monkeys,
and
what
have
you
are
well
represented
by
gabor
filters
right
in
the
mouse.
This
sort
of
a
gabor
sort
of
thing
doesn't
seem
to
exist.
F
However,
what
I
would
like
to
mention
is
that
our
theory
does
not
restrict
itself
to
just
using
blobs
right.
We
use
blogs
only
because
the
data
showed
us
that
blogs
are
a
good
representation
of
mice,
but
in
in
our
paper
we
have
also
explored
gabors,
and
you
know
other
sorts
of
filters
that
could
presumably
be
present
in
animals
with
better
activity
than
mouse,
but
the
general
results
seem
to
hold.
H
Okay,
so
yeah,
so
one
of
the
things
I'm
noticing
is
there
is
a
lack
of
expected
symmetry
in
some
of
these
patterns,
like
between
the
90
degree
patterns
and
I'm
wondering
if
you,
instead
of
picking
45
degree
increments.
If
you
pick
something
like
60
or
30
degrees,
that
you'd
get
a
better
symmetry,
I'm
just
thinking,
hexagonal
close
packing
might
be
somehow
representative
of.
What's
actually
going.
C
C
Again,
this
theory
works
on
any
type
of
filters
it
can
anything
can
be
put
in
here.
We
just
wanted
to
compare
with
the
data
from
the
mouse,
and
mouse
is
not
the
most
regular,
and
some
of
the
lack
of
symmetry
is
something
which
has
also
been
observed
in
here.
It
would
also
be
great
if
we
have,
if
people
have
another
five
minutes,
to
hear
how
this
can
be
applied
to
artificial
neuronal
networks
from
brian.
Do
you
want
to
request
control.
D
D
We
want
to
think
about
how
this
type
of
local
computation
could
also
work
in
networks
which
are
being
optimized
for
more
global
computations,
such
as
a
task
like
object,
recognition,
okay,
next
slide.
So
to
better
understand
this,
I
want
to
pretty
much
introduce
the
ideas
of
supervised
and
unsupervised
learning,
so
this
figure
comes
from
a
recent
review
paper,
but
essentially
with
the
rise
of
essentially
compute
power
and
large
amounts
of
labeled
data.
We
have
computer
vision
systems
that
can
do
all
types
of
tasks
now.
D
D
You
have
different
sets
of
features
and
then
you
within
each
feature
map
you
have
essentially
different
spatial
locations.
So
here
the
purpose
of
supervised
learning
is
to
learn
a
set
of
feed.
Four
awaits
you
here
which
allow
you
to
perform
the
computation,
and
so
it's
essentially
a
set
of
weights
from
one
spatial
location
in
a
given
layer
to
the
layer
on
top
and
he
and
usually
in
convolutional
neural
networks.
E
D
But
in
order
to
get
this
to
work
in
these
networks,
we
make
two
very
minor
assumptions
for
essentially
one
is
that,
instead
of
requiring
all
of
the
information
to
be
independent
from
the
independent
patches,
as
this
is
very
hard
to
enforce
in
neural
networks,
we
allow
for
overlapping
receptive
fields
and
we
essentially
scale
the
contribution
from
these
kind
of
overlapping
receptive
fields
by
a
hyperparameter
which
can
be
controlled.
And
then
stefan
mentioned
earlier
that
the
contribution
from
these
extra
classical
receptive
fields
was
multiple
multiplicative
in
the
most
formal
sense.
But
we
relax
this
assumption.
E
D
That's
present
in
the
essentially
the
feed
four
response
of
a
neuron,
and
so
you
can
try
to
average
out
some
of
this
noise
if
you're
going
to
be
talking
to
some
of
the
other
units
in
in
the
same
layer,
and
the
hope
is
that
these
lateral
connections
will
provide
you
some
robustness
to
noise.
So
to
test
this,
we
took
the
mnist
data
set.
We
applied.
D
Yeah,
so
I
skipped
over
that
detail,
but
essentially
we
take
a
baseline
neural
network.
We
train
it
via
supervised
learning
on
essentially
clean
images
of
mnist,
and
then
we
freeze
the
weights
of
the
of
that
network,
and
so
this
provides
essentially
the
same
thing
as
like
the
three
forward
receptive
fields
that
rama
showed
earlier,
and
these
are
what
we
use
to
compute
the
activations.
D
Yes,
correct
yeah,
yeah,
and
so
what
we
did
is
we
tested
our
models
on
these
two
types
of
images,
noisy
images,
so
here
I'll
walk
through
this
is
essentially
the
final
table
of
results.
So
the
cnn
model
is
the
baseline
model.
It's
a
very
simple
model
which
has
two
convolutional
layers,
followed
by
two
fully
connected
layers.
D
And
then,
in
this
case
the
main
observation
is
that
the
baseline
cnn
model
does
pretty
well
at
low
noise
levels,
which
is
expected
if
the
data
matches
the
distribution
that
it
was
trained
on,
but
at
high
noise,
essentially,
the
cnn
x
model
does
much
better,
and
so
here
we
have
the
baseline
cnn
x
model,
which
is
the
second
row.
We
have
a
control
model
which
is
essentially
take
lateral
connections,
but
all
the
lateral
connections
do
are
average
deactivations
of
the
surrounding
units.
D
Yeah,
so
you
can
imagine
the
lateral
connections
essentially
being
an
additional
layer
of
weights
in
your
model.
So
you
have
your
two
essentially
feed
forward
convolutional
layers
and
then,
after
each
of
those
convolutional
layers,
we
insert
an
additional
layer
of
lateral
connections
which
aren't
trained,
using
supervised
learning
but
they're
trained
using
this
unsupervised
learning
rule
and
they
essentially
connect
they're
convolutional
as
well.
So
they
tile
the
whole
feature
map.
But
then
they
connect
essentially
different
units
at
different
spatial
locations
and
with
different
feature
selectivities.
D
I'm
not
as
familiar,
unfortunately
with
those
models,
but
I
think
the
basic
idea
I
would
say
like
as
a
guiding
principle
for
this
model.
Is
we
start
with
some
inspiration
from
biology.
So
we
take
essentially
this
hebbian
like
learning
role,
which
is
we
observe
in
the
biology
and
is
built
essentially
based
on
first
principles
of
keeping
track
of
which
units
essentially
fire
together
over
a
set
of
images.
And
then
we
take
that
same
rule
and
we
just
try
to
apply
it
to
lateral
connections.
D
And
I
think
in
in
these
neural
networks-
and
I
think,
there's
a
lot
of
room
for
future
research
in
terms
of
how
to
combine
like
supervised
learning
along
with
this
unsupervised
learning,
because
right
now
we're
doing
kind
of
a
hacky
way
where
we
train
the
model
in
a
fully
supervised
manner.
Freeze
the
feed
forward
waste
and
then
learn
the
lateral
connections.
But
you
can
imagine
more
hybrid
schemes
where
you
kind
of
learn
both
simultaneously
or
you
have
an
alternating
way,
but
that's
kind
of
future
research.
B
E
D
All
right
so,
given
time
constraints,
won't
ask
for
audience
feedback,
but
essentially
these
are
the
conclusions
of
our
main
talk
so
again
outlined
by
these
basic
kind
of
bullet
points.
Our
hypotheses
is
that
cortex
is
hierarchically
organized,
but
then
it
also
consists
of
these
canonical
local
circuits,
which
we
try
to
understand
here
and
then
that
the
brain
isn't
necessarily
optimizing.
One
cost
function,
but
there
could
be
different
cost
functions
at
the
local
circuit
level
and
the
global
circuit
level.
D
We
applied
this
both
to
biological
data,
so
in
this
case,
looking
at
receptive
fields
from
the
mouse
and
we
found
that
there
was
a
good
experimental
match
and
then,
as
a
proof
of
concept,
we
started
into
looking.
How
can
we
apply
these
principles
also
to
artificial
neural
networks
and
try
to
incorporate
these
lateral
connections
there
and
what
kind
of
benefit
they
can
provide?
D
B
H
C
H
D
G
D
Of
essentially
what
we
were
proposing,
but
I
think
that
is
that
is
interesting
direction
to
think
like.
Why
is
the
brain,
and
why
is
the
human
visual
system
so
robust
to
these
kind
of
adversarial
perturbations
and
it's
possible
that
feedback
connections?
Lateral
connections
could
provide
some
substrate
for
that,
but
we
haven't
actively
pursued
too
much
of
this
in
this
direction.
But
it's
something
that
I'm
interested
in
okay.