►
From YouTube: Subutai Ahmad on Small-World Structures - April 28, 2021
Description
In this research meeting, joined by Rosanne Liu, Jason Yosinski, and Mitchell Wortsman from ML Collective, Subutai Ahmad explains the properties of small-world structures and how they can be helpful in Numenta’s research.
Subutai first discusses different network types and the concept of small-world structures by reviewing the paper “Collective Dynamics of ‘Small-World’ Networks” by Watts & Strogatz. He then evaluates the efficiency of these structures and how they are helpful in non-physical networks by looking at Jon Kleinberg’s paper “Navigation in a Small World.” Subutai also addresses how small-world structures would apply to machine learning by using concepts from the paper “Graph Structure of Neural Networks” by Jiaxuan You et al.. Lastly, the team discusses how small-world structures relate to Numenta’s research such as sparsity and cortical columns.
“Collective Dynamics of ‘Small-World’ Networks” by Watts & Strogatz: https://www.nature.com/articles/30918
“Navigation in a Small World” by Kleinberg: https://www.nature.com/articles/35022643
“Graph Structure of Neural Networks” by Jiaxuan You et al.: https://arxiv.org/abs/2007.06559
"Small-World Brain Networks" by Bassett & Bullmore: https://journals.sagepub.com/doi/10.1177/1073858406293182
More information on ML Collective: https://mlcollective.org/
- - - - -
Numenta is leading the new era of machine intelligence. Our deep experience in theoretical neuroscience research has led to tremendous discoveries on how the brain works. We have developed a framework called the Thousand Brains Theory of Intelligence that will be fundamental to advancing the state of artificial intelligence and machine learning. By applying this theory to existing deep learning systems, we are addressing today’s bottlenecks while enabling tomorrow’s applications.
Subscribe to our News Digest for the latest news about neuroscience and artificial intelligence:
https://tinyurl.com/NumentaNewsDigest
Subscribe to our Newsletter for the latest Numenta updates:
https://tinyurl.com/NumentaNewsletter
Our Social Media:
https://twitter.com/Numenta
https://www.facebook.com/OfficialNumenta
https://www.linkedin.com/company/numenta
Our Open Source Resources:
https://github.com/numenta
https://discourse.numenta.org/
Our Website:
https://numenta.com/
A
You
know,
normally
we
don't
have
other
people
on.
I
invited
them,
because
we
were
kind
of
discussing
some
of
these
discussions
and
stuff
on
the
side
and
they
had.
It
was
really
helpful
and
I
invited
them
they
actually
joined.
I
don't
know
if
you
guys
will
get
anything
out
of
this,
because
we've
already
talked
about
a
lot
of
this,
but
welcome
anyway.
You
guys
just
want
to
describe
just
briefly
introduce
yourself
and
and
what
you
do
and
and
roseanne
might
be.
A
B
Hi
everyone,
I'm
rosianne,
so
jason
and
I
here
run
ml
collective.
It
is
a
non-profit
that
basically
get
people
together
to
work
on
research
without
having
to
employ
everyone.
Whoever
wants
to
get
into
research,
we
don't
need
to
interview
them
and
yeah,
they
just
show
their
interest
and
then
they
can
come
and
freely
collaborate
with
us
mitchell.
Here
also,
I
think,
get
to
us
as
part
of
mlc
or
work
with
us
through
mlc.
C
I
think
I
think
roseanne
covered
it
work
on
ml
research.
We
don't
have,
we
don't
have
much
money,
but
we
have
interested
people.
Roseanne
runs
a
reading
group
every
friday,
which
everyone's
welcome
to
join,
and
we
worked
a
bit
with
subatai
on
some
ideas
over
the
last
six
months.
Also,
we
were
kind
of
inspired
by
how
dementia
does
research
kind
of
like
doing
research
out
in
the
open.
A
A
Okay,
then
just
to
introduce
you
to
our
team.
This
is
pretty
much
our
set
of
researchers.
Jeff
hawkins
is
our
founder
and
well
so
you
might.
I
don't
know
if
you
know
anyone
else
on
this
this,
but
basically
they're.
We
have
a
pretty
big
mixture
of
sort
of
neuroscience
background
machine
learning,
computer
science
on
so
welcome.
It's
great
to
have
you
guys.
E
A
Okay,
so
let
me
get
started
okay,
so
the
we're
going
to
talk
about
small
world
structures,
I'm
really
unsure
what
people
here
will
think
about
whether
it's
relevant
but
jason,
rosen
and
mitchell,
and
I
had
a
long
discussion
over
you
know.
We
meet
every
few
weeks
talking
about
sparsity
and
sparse
structures
and
stuff,
and
this
kind
of
led
to
investigation
of
small
world,
and
for
me
this
became
a
little
bit
of
a
rabbit
hole.
I
started
reading
a
couple
of
these
papers.
A
A
So
the
main
paper,
oh
so,
here's
the
outline
of
the
I'm
going
to
talk
about
just
a
brief
description
of
small
world
networks.
What
they
are,
this
discussion
of
small
world
structure
and
brains
and
other
places
that
shows
up
everywhere.
It
looks
like,
and
in
particular
I
want
to
talk
a
little
bit
about
small
oral
structure
in
learning
and
neural
networks
and
then
just
a
general
discussion
on
sparsity,
cortical,
columns
and
stuff
and
I'll
I'll
stick,
because
I
wasn't
really
sure
what
the
level
of
interest
would
be
in
this
group.
A
I
stuck
to
pretty
high
level
concepts
here,
but
at
the
end,
if
people
want
to
talk
in
more
detail,
particularly
on
point
three
there's
a
paper
that
I'll
go
through,
there's
a
bunch
of
interesting
kind
of
things
in
there
that
you
know
people
might
be
interested
in.
But
you
know
we
could
leave
that
to
the
end
to
go
into
the
details
in
anyone
who
doesn't
want
to
can,
you
know,
can
drop
off
a
point,
so
one
thing's
like
network
theory.
Why
do
we
even
care
about
this
stuff?
A
Well,
clearly,
neurons
in
the
brain
are
connected
in
a
very
complex
network
and
there
are
a
bunch
of
sort
of
abstract
questions
that
may
be
practical.
That
sort
of
come
up
so
things
like.
Are
there
simple
characterizations
of
these
networks?
Can
we
understand
how
good
these
connectivity
patterns
are?
There's
a
variety
of
different
types
of
ways
it
can
be
connected.
A
You
know,
momenta.
We
want
to
create
really
really
sparse
networks
and
run
them
efficiently
on
hardware.
So
are
there
some
types
of
sparse
connections
that
are
better
than
others,
particularly
when
you
get
to
the
limits
and
then
how
does
this
impact
learning
or
inference?
So
these
are
some
of
these
sort
of
questions
that,
in
part,
are
addressed
by
looking
at
these
small
world
structures.
A
So
let
me
drive
into
the
the
main
paper
I'll
go
over
is
one
of
the
two
main
papers,
watson
strogatz.
This
came
out
in
1998,
it's
only
two
and
a
half
pages.
It
is
really
concise,
very
clearly
written.
I
really
enjoyed
reading
this
paper
and
I
think
I've
emailed
it
out
to
this
to
the
group,
so
you
guys
can
take
a
look
at
it,
but
I
it's
very
very
well
written,
I
thought
so
what
they're
and-
and
I
should
say
this
is
the
main.
A
This
is
the
paper
that
kind
of
started
started
all
of
the
analysis
on
small
world
networks.
I
think
they
had
something
like
45,
000
citations
or
something
ridiculous
like
that.
It's
pretty
crazy,
but
it's
it's
cited
all
over
the
place,
so
they
they
started
their
analysis
by
looking
at
two
different
extremes
of
different
two
different
types
of
networks
so
on
the
left
is
a
regular
graph,
in
particular
a
ring
lattice.
So
here
you
have
nodes
in
this
particular
example.
A
A
A
Okay,
and
so
you
can
see
these
as
they
looked
at
these
as
two
extremes
in
a
spectrum
and
they
created
a
an
algorithm
for
sort
of
smoothly.
Interpolating
between
these
two
types
of
networks
and
the
algorithm
is
the
entire
algorithm
is,
can
be
stated
in
just
one
sentence.
Just
one
thing
I
really
like
about
this
paper
is
that
they,
just
they
just
say
things
very
concisely.
A
So
this
is
a
quote
from
the
paper,
so
you
basically
start
from
a
ring
lattice
with
this,
which
is
that
regular
graph
on
the
left,
with
n,
vertices
and
k
edges
per
vertex,
and
then
we
rewire
each
edge
at
random
with
probability
p.
Okay,
so
that's
it.
So
you
start
with
the
ring
lattice
and
just
randomly
rewire
some
fraction
of
it,
and
so
this
is.
A
Yeah,
I
think
in
their
algorithm
to
keep
the
number
of
edges
fixed,
but
from
what
I
understand
later
on,
particularly
in
the
physics
world,
they've
they've
changed
that
so
they've
sometimes
will
add
edges,
so
they
they
might
keep
a
regular
network
underneath
it,
but
then
add
a
bunch
of
random
edges
on
top.
So
that's
another
way
that
people
have
done
it,
but
I
think.
G
H
F
A
G
I
mean
we
work
so
much
here
in
so
many
high
dimensions
and
then
and
when
we
think
about
high
dimensional
spaces,
some
of
the
intuitions
you
get
from
simple,
smaller
dimensional
spaces
don't
apply.
So
I'm
immediately
thinking
like
oh
yeah,
what's
the
nearest
neighbor,
you
know
if
you
have
a
10,
000,
dimensional
vector
and
there's
neighbors.
That
means
something
very
different,
so
yeah,
okay,
well
I'll,
just
put
it
aside
for
now,
but
that's
the
first
thing
that
jumped
in
my
head.
A
Yeah
yeah,
so
here
this
is
a
diagram
from
the
paper.
It
shows
kind
of
the
spectrum.
On
the
left
hand,
side
p
is
zero,
which
means
you
don't
rewire
anything.
So
you
get
the
regular
network
on
the
the
other
extreme.
Where
you,
you
know,
rewire
everything
you
get
the
random
network
and
then
somewhere
in
between
is
this
sort
of
small
world
structure.
A
They
looked
at
did
two
different
metrics
to
help
kind
of
characterize
these
networks,
so
the
first
one
is
the
path
length,
which
is
the
average
shortest
distance
between
vertices.
If
you
look
at
any
two
vertices,
what
is
the
number
of
hops?
You
need
to
get
from.
One
point
to
the
other,
and
so
random
graphs
have
very
generally
have
very
short
path.
Lengths.
It's
you
can
find
between
any
two
vertices.
Usually
a
pretty
small
number
of
hops
will
get
you
from
one
to
the
other.
A
The
regular
graphs
do
not,
as
you
you
know,
if
you
want
to
get
from
one
to
the
other,
you'd-
have
to
sort
of
hop
multiple
times
around
this
lattice
until
you
get
to
the
other
side,
and
of
course
they
didn't
really
talk
about
this,
but
complete
graphs,
where
everything
is
connected
to
everything
has,
you
know,
always
a
path
length
of
one.
So
that's
always
the
shortest
really
shortest
path
length,
the
other.
So
this
is
relatively
easy.
A
The
other
one
is
something
called
the
clustering
coefficient
and
what
this
says
is
basically,
if
a
vertex
a
is
connected
to
two
other
vertices
b
and
c,
how
likely
is
it
that
b
and
c
are
connected
to
each
other?
Okay?
So
how
clicky
is
this
graph
and
it
turns
out
that
regular
graphs,
like
the
lattice,
when
I
showed
you
are
pretty
highly
clustered
random
graphs
are
not
well
clustered.
They
have
a
very
low
clustering
coefficient
and
again
complete
graphs
are
the
most
clustered.
A
A
Boy,
it's
really
hard
to
navigate
zoom
here.
Okay,
if
you
started
the
blue
circle
here,
if
you
want
to
get
to
the
other
edge
of
the
graph,
you
know
you
can
hop
three
at
a
time,
but
so
you
know
so
that's
that's.
You
know
how
many
that's
going
to
be.
The
average
path
length
is
going
to
be
basically
the
number
of
nearest.
A
You
look
at
the
number
of
total
nodes
divide
by
the
number
of
nearest
neighbors
divide
by
two:
that's
roughly
how
many
hops
you'll
need
on
average
and
then,
if
you,
you
can
kind
of
understand
the
clustering
here.
This
guy
is
connected
to
this
one
here,
and
it's
also
connected
to
this
one.
These
two
are
also
likely
to
be
connected,
so
you
can
see
that
this
this
graph
is
somewhat
clickish
and
that's
true
for
most
of
the
guys
nearby.
A
Of
course,
if
you
pick
someone
that's
three
away
here,
it's
not
going
to
be
connected
to
a
node,
that's
three
away
there,
but
most
of
the
guys
in
its
neighborhood
or
most
of
the
nodes
in
this
neighborhood
are
going
to
be
connected
to
each
other.
So
these
regular
graphs
are
relatively
highly
clustered,
but
random
graphs
are
not
so,
let's
see.
A
You
know
for
a
fixed
number
well
at
least
in
this
in
this
in
this
set
of
extremes.
If
you
start
with
these
kind
of
graphs
and
and
randomly
rewire
everything
that's
going
to
have
the
shortest
path
length
in
general,
I
don't
know
what
we
can
say.
You
know
I
think,
there's
going
to
be
a
spectrum
between
random
and
complete
graphs.
There
might
be
some
others
that
have
a
shorter
path,
lengths
than
completely
random
graphs.
I.
G
G
A
C
A
C
Okay
and
on
the
clustering
coefficient
side,
could
you
create
an
even
more
cluster
graph
by
rather
than
having
a
complete
ring
just
sample
like
a
neighborhood
and
then
connect
absolutely
everything
in
that
neighborhood
and
then
maybe
have
one
edge
going
out
of
the
neighborhood?
Could
you
construct
the
small
world
network,
starting
with
that.
A
A
A
D
D
One
thing
I
think
is
that
the
the
ring
lattice
should
converge
to
to
a
clustering
coefficient
one
because,
as
you
start
to
go
from
connecting
just
to
my
nearest
neighbors,
then
to
my
two
nearest
neighbors,
then
to
your
three
nearest
neighbors
as
you
increase
this.
Doesn't
the
clustering
coefficient
just
rise
and
rise
and
rise
until
the
end
of
the
day,
you're
connected
to
everyone,
in
which
case
it's
one
yeah,
but.
D
As
you
yeah
go
ahead,
sorry
that
asymptotics
that
you're
talking
about
might
be
actually
okay,
let's
actually
increase
the
number
of
like
let's
scale,
the
number
of
nodes,
because
then
I
think
you
know,
then
then,
if
you're
scaling
the
number
of
nodes
as
well,
then
you
you'd
never
get
the
one.
That's.
A
A
Okay,
so
what
are
small
world
networks?
We
haven't
really
defined
them
yet,
so
a
regular
graph
has
well-defined
connectivity
and
high
clustering.
Random
graphs
have
small
average
path
length,
and
so
small
world
networks
are
somewhere
in
between
in
kind
of
an
interesting
way
and
this
graph
kind
of
shows
it.
So
what
this
graph
shows
is
they
plot
path
length
as
a
function
of
p,
which
is
the
solid
dots
and
the
clustering
coefficient
as
a
function
of
p
as
these
open
squares?
A
So
on
the
top
right
sorry
top
left
here,
you
have
essentially
zero
probability
of
rewiring,
and
so
that's
going
to
be
a
completely
regular
graph.
So
this
has
a
high
clustering
coefficient
and
high
path
length.
These
are
actually
normalized,
but
we
can
just
ignore
this.
For
now
high
clustering
coefficient
and
high
path
length
and,
as
you
start
increasing
the
probability
of
rewiring,
you
know
the
path
length
is
going
to
drop
and
on
the
bottom
right,
you
have
a
completely
random
graph.
A
You
can
see
that
the
path
length
drops
dramatically
and
gets
very
close
to
the
random
graph,
but
the
clustering
coefficient
actually
stays
high
for
quite
a
while
and
before
it
starts
dropping
so
there's
this
whole
kind
of
region
in
between
where
you
have
high
clustering
coefficients
and
pretty
low
path.
Lengths
and
that's
kind
of
the
magic
of
small
wall
graphs
is
this
region
here
and
so
in
some
sense,
these
small
world
networks
are
very
efficient.
A
They
have
very
short
path
lengths
and
a
high
clustering
coefficient,
whereas
the
complete
graph
would
have
both
but
they're
extremely
inefficient,
because
you
have
of
n
squared
edges
in
a
complete
graph.
So
it
allows
you
to
get
the
efficiency
of
this.
You
know
very
very
well,
so
here's
some
examples
of
real
networks.
Anyone
want
to
guess
what
this
is.
Anyone
at
momenta.
H
A
E
A
Yeah
exactly
yeah.
I
also
thought
this
is
oh,
this
is
a
cortical
network,
but
so
this
is
a
cortical
network,
so
this
is
from.
Actually
this
is
from
jeff's
thousand
braids,
but
it's
originally
from
feldman
and
venice,
and
so
they
mapped
all
the
connections
between
different
regions
of
the
neocortex,
particularly
in
the
macaque
monkey,
and
they
had
this.
A
You
know
network,
and
this
is
a
small
wall
structure
and
you
can
see
that
there's
a
lot
of
sort
of
local
connectivity,
but
then
there
are
these
sort
of
long-range
connections
that
show
up
every
once
in
a
while,
and
so
oh
and
so
social
networks
are
also
small
world.
So
this
is
a
screenshot.
I
took
from
a
youtube
video
that
explains
it.
So
there's
about
a
billion
facebook
users
in
the
world,
70
billion
links,
but
the
average
shortest
path
between
any
two
people
in
the
world
is
four.
A
A
So
what
why
are
these
properties
useful?
So
this
is
what
I
was
trying
to
think
through
a
little
bit,
so
you
you
can
understand
why
short
paths
are
useful
right.
They
can
transmit
information
very
efficiently,
but
why
is
clustering
useful
so
in
physical
networks
such
as
the
brain
and
the
power
grid,
and
so
on,
clustering
will
optimize
kind
of
the
physical,
the
total
physical
path
length.
A
The
probability
of
these
long-range
connections
can
be
really
really
small,
and
so
this
is
a
very
efficient
way
if
you
want
to
minimize
the
total
length
of
your
wires
or
in
brains.
If
you
want
to
minimize
the
total
length
of
axons,
this
could
be
a
really
good
way
to
do
that
so
physically,
you
can
see
why
these
why
this
property
might
be
might
be
helpful,
but
that
wasn't
really
satisfying
to
me.
It's
like.
If
I'm
building
algorithms
or
networks,
I
don't
really
care
about
physical
length.
A
A
A
Claimed
that
it's
easy,
the
reason
it's
helpful
in
non-physical
networks
is
it's
an
algorithmic
reason,
it's
easier
to
find
short
chains
between
points
in
some
in
small
world
networks,
and
so
in
general
we
know
short
shortest
path.
Algorithms
are
really
hard,
but
with
small
world
networks,
and
it
turns
out
only
with
small
world
networks.
You
could
you
can
algorithmically
find
local
algorithms
for
discovering
these
short
paths.
A
So
if
you're
like
a
a
person-
and
you
have
social
connections-
and
you
you
only
know
about
your-
the
people
you're
immediately
connected
to
you-
can
actually
find
efficient
algorithms
for
getting
to
anyone
else
in
the
world.
You
know
you
might
not
find
it
in
four
hops,
but
you
can
find
very
efficient
algorithms.
A
A
You
know
you
get,
you
can
find
these
efficient
algorithms
and
find
these
really
short
paths,
and
you
can
see
that,
as
you
divert
away
from
that,
you
know
the
chance
of
finding
short
paths
increases
he
didn't
use.
I
think
his
definition
of
clustering
was
a
little
bit
different,
but
I
think
the
basic
intuition
still
applies.
A
So
this
kind
of
explains
a
lot
of
things
like,
for
example,
you
know
airline
networks,
a
small
world
where
they
have
a
physical
constraint
as
well,
but
you
can
very
efficiently
find
you
know
if
you
want
to
get
from
any
place
to
any
place
in
the
world.
You
know,
if
you
do
a
web
search,
they
will
very
quickly
tell
you
what
connectivity
or
what
what
flights
it
should
take.
It's
you
know
if
you
look
think
about
all
the
different
places
in
the
world.
A
You
can
also
think
about
this
sort
of
a
thought
experiment
you
could
do
like
if
you
say
I
wanted
to
send
something
you
know
say
I
wanted
to
give
joe
biden
something
like.
How
would
I
do
that
yeah?
It's.
You
know,
I
don't
know
joe
biden
and
you
know
when
I
thought
through
it.
You
know
it
became
very
clear
to
me
exactly
who
I
should
contact,
who
is
most
likely
to
know
someone
who
knows
joe
biden.
A
In
my
case.
That's
donna
trinsky
our
ceo,
because
she
seems
to
know
everyone,
and
so,
if
I
wanted
to
send
something
to
joe
biden,
I
would
sense
I
would
give
it
to
donna
and
then
ask
her
to
pass
it
on.
So
that's
an
example
of
of
this
local
algorithm.
I
can
just
see
okay,
who's
who's,
who
tends
to
have
the
most
connections.
G
You
know
when
I
looked
at
that
was
intuitively
obvious
when
I
looked
at
this
airline
picture,
because
the
airline
picture
the
way
this
works
is
we
have
these
hubs,
which
are
super
highly
connected
to
lots
of
places
and
then
and
and
then
you
and
then,
and
then
you
just
have
to
get
between
the
hubs
and
you're,
almost
certain
exactly
yeah
and
that's
and
don
is
a
hub,
but
but
that's
exactly
that's
starting
out
with
a
different
type
of
structure.
That's
not
like
the
ring
structure
you
started
with,
which
was
you
know.
There's
these.
G
A
I
think
it's
it's
I
I
don't
know
surprising
is
the
word,
but
it's
certainly
a
property
that
applies
to
small
world
networks
of
all
types.
You
know,
you
know
that
you
can.
The
fact
that
you
can
algorithmically
just
by
knowing
local
information
figure
out
a
good
way
to
get
to
anywhere
else
on
on
the
graph
is,
is
pretty
cool.
G
A
F
G
G
G
I
So
in
in
the
donna
case,
maybe
a
way
that
does
make
the
idea
a
little
interesting
is
that
maybe
donna
doesn't
know
any
more
people
than
any
of
us
know.
It's
just.
She
knows
the
right
set
of
them,
the
the
sub,
the
set
of
people.
She
knows
the
number
of
connections
she
has
isn't
different
from
our
number
of
connections.
It's
it's
the
particulars
of
what
she's
connected
to
which
is
kind
of
interesting.
G
G
G
G
A
A
F
F
A
G
Why
I
get
your
question,
so
I
I
think
I
I
think
I
do.
I
I
think
it's
related
to
my
earlier
comment.
I
was
surprised
that
you
said
there's
a
local
algorithm,
but
but
you're
really
relying
on
the
fact
that
you
know
that
some
of
the
people
you're
connected
to
are
more
connected
than
others,
and
that's
not.
I
wouldn't
include
that
in
the
local
algorithm.
I
think
with
kevin's
saying
all
I
know
is
I'm
connected
to
four
people.
I
know
nothing
about
those
four
people,
then
there's
no
directed
way.
I
can
go
about
this.
A
G
C
It
seems
to
me
that
what
you're
really
doing
or
what
we're
kind
of
approximating
is
you
imagine,
you
start
with
joe
biden
and
his
distance
from
himself
is
zero
and
you
expand
out
one
and
the
distance
is
one
and
so
on.
Basically
imagine
you
ran
the
whole
a
star
algorithm
to
map
the
distance
from
joe
biden
to
every
person,
and
then
what
subutai
is
doing
is
saying.
C
Okay,
I
know
my
distance
to
joe
biden
is
somewhat
large
and
I
know
all
the
people
that
I
know
I'm
kind
of
like
mentally
estimating
who
might
have
the
lowest
distance
and
then
sending
the
packet
that
direction
right,
which
is
kind
of
what
kevin
kevin
was
getting
at.
So
if
you
had
no
information
about
this,
it
would
seem
like
it
might
not.
Work
might
not
work
very
well.
G
C
G
C
So
I
think,
when
the
path
length
is
something
like
close
to
four
so
so
subutai
is
connected
to
some
hub
in
his
local
region,
and
he
has
some
knowledge
about
that
hub
and
then
biden
is
connected
to
lots
of
people
and
there's
a
good
chance
that
this
hub
knows
that
hub.
Then
the
whole
thing
can
kind
of
work.
But
if
you
imagine
that
path
length
is
like
10.
so
so
stupid
tainos
is
a
bunch
of
hubs
whose
distance
is
now
nine
divided.
By
knows
a
bunch
of
hubs
whose
distance
is
now
eight
to
subitize
hub.
C
A
No,
I
think
kleinberg
in
kleinberg
says
that
would
it
would
still
be
a
relatively
efficient
algorithm
to
find
short
paths.
I
I
didn't
go
through
the
details
of
all
of
these
algorithms,
but
I
think,
even
in
that
case
you
you
can.
As
long
as
you
get
yours
in
the
in
the
sweet
spot
of
the
clustering
exponent,
you
can
find
efficient
algorithms,
for
maybe
you
might
not
get
the
optimal
shortest
path,
but
you
get
very
good
short
paths.
G
Well,
you
can
in
any
of
these
situations
you
may
not
get
the
optimal
short
path.
I
mean
you
might
say.
Oh
you
know
I
need
to
get
from
ithaca
to
san
francisco
and
I
said
well,
you
know
if
I'm
going
to
or
it's
good
to
you
know,
I
don't
know
some
other
little
eureka.
California
and
you
might
say
well,
I
have
to
go
photographers,
but
maybe
just
by
chance,
there's
a
connection
to
them.
If
they
could
to
eureka.
California,
you
know
he
wouldn't
have
tested.
You
know
you
wouldn't
have
guessed
that
yeah.
G
A
It's
15
pages,
not
one
or
two
and
a
half
pages.
It's
a
very
interesting
paper,
but
a
bit
confusing
in
parts
I'm
just
going
to
show
a
few
highlights
from
it
and
then
we
in
a
more
detailed
discussion
we
can
dive
into
it
in
more
depth.
If
people
want
okay,
so
what
they
basically
did,
is
they
had
a
one
of
the
things
they
did?
Is
they
had
an
algorithm
for
taking
graphs
and
converting
them
to
neural
network
structures.
H
A
And
so
what?
What
do
I
mean
by
that?
So
here's
an
example
so
on
the
left
is
a
fully
connected,
multi-layer,
perceptron,
just
a
bunch
of
linear
layers
going
down
and
what
they
said
is.
This
is
basically
equivalent
to
this
graph
on
the
right.
Okay,
so
here's
the
fully
connected
graph
with
four
nodes
and
what
you
the
way
to
think
about
it
is
that
these
nodes
are
sending
messages,
so
each
going
from
one
layer
to
the
next
is
sort
of
one
round
of
a
message:
exchange.
Okay.
So
this
is
this
graph.
A
You
can
see
how
it
might
be
equivalent
to
these
two
layers
here.
So
node
one
is
connected
to
all
four
nodes
here
in
the
next
layer.
Node
two
is
connected
to
all
four
nodes
and
so
on.
Okay,
so
this
you
can
think
of
this
as
a
fully
connected
graph
with
four
nodes
with
multiple
rounds
of
message:
exchanges:
okay,
that's
how
they
kind
of
formulated
it,
and
you
can
do
a
variety
of
different
network
types
here.
So
on
the
left.
A
Here
you
have
again
a
four
node
graph,
but
you
don't
have
the
cross
connections
here
and
what
that
translates
to
is
you
have
a
unit
one
here?
It
connects
to
units
one
two
and
four
in
this
case
units
two
connects
to
one
two
and
three,
and
so
on.
Okay,
so
this
is
a
little
bit
like
kind
of
the
in
visual
cortex.
We
have
receptive
fields
that
are
connected
locally
to
a
set
of
neurons
below
it
and
that
that
local
connectivity
is
sort
of
shifted
in
a
trans.
You
know
translationally.
A
So
that's
an
example
of
this.
You
can
also
have
disconnected
stuff
here,
so
you
have
these
four
nodes
that
there
are
no
connections
between
this
click
and
this
click,
and
that
corresponds
to
a
graph
like
this.
So
they
went
through
a
bunch
of
different
variations
and
in
some
cases,
contortions
to
convert
a
bunch
of
to
see
how
you
can
take
graph
structures,
different
types
and
and
think
about
the
graph
properties
of
resnets-
and
you
know
multi-layer
perceptrons
with
differing.
A
You
know,
different
layer,
widths
and
so
on.
So
we
can
go
into
that
later.
If
people
want,
but
basically
they
had
a
way
of
thinking
about
a
very
large
range
of
neural
network
structures
as
simpler
kind
of
graph
structures
like
this,
so
then
they
also
created
a
slight
modification
to
the
watts
strogatz
algorithm,
to
create
neural
networks
with
a
variety
of
path,
lengths
and
clustering
coefficients-
and
this
is
a
example
of
the
range
of
networks
that
they're
able
to
to
create
so
on
the
bottom.
A
So
this
shows
clustering,
coefficient
and
path,
length
and
so
on.
The
bottom
right
is
a
completely
fully
connected
graph,
so
this
would
be
like
the
multi-layer
perceptron.
I
showed
you
where
every
layer
is
fully
connected
on
the
the
left
here.
It
would
be
a
completely
random
graph,
so
this
has
very
low
clustering
coefficient
but
reasonably
low
average
path
length
and
then
up
here
would
be
the
equivalent
of
a
ring
lattice.
A
So
here
you
would
just
have
you
know,
nodes
just
connected
to
its
nearest
neighbors
and
that's
it,
and
that
would
be
again
very
similar
to
a
typical
model.
People
have
a
visual
cortex
where
you
have
a
set
of
you,
have
a
neuron,
that's
just
connected
to
a
small
set
of
neurons
below
it
and
that
sort
of
continues
on.
A
And
then
you
have
a
whole
bunch
of
stuff
in
between
so
small
world
networks
would
be
kind
of
in
between
here
and
then
for
what
they
can
do
is
they
can
map
points
in
here
to
different
network
structures
and
then
run
different
benchmarks
on
it,
and
you
can
look
at
the
accuracy
of
networks
at
different
points
in
this
space.
So
you
know
here
it's
showing
five
layer,
multi-layer,
perceptron
or
resnets,
and
then
they
ran
a
whole
bunch
of
diff.
Oh
I'm,
sorry
yeah.
They
ran
a
whole
bunch
of
different
experiments.
A
Here's
sort
of
one
example
experiment
using
scifir10
and
what
you
can
do
is
you
can
then
look
at
the
accuracy
of
these
networks
as
a
function
of
path,
length
and
and
clustering,
okay,
and
so
what
they
found
is
in
this
case
is
that
the
best
graph
was
right
in
the
middle
here.
This
is,
you
know
the
small
world
structure
and
you
can
look
at
accuracy
as
a
function
of
path
length.
G
A
H
A
A
G
A
Thanks
what's
kind
of
interesting:
is
this
looked
a
lot
like
kleinberg's
graphs?
To
me,
you
know
you
have
the
sweet
spot
of
of
clustering
coefficients
that
was
kind
of
a
cool
relationship.
The
other
thing
that's
important
from
our
standpoint
is
what
they
did
to
do.
This
is
you
know
these
graphs
are
sparser
right
and
so
at
the
bottom
right,
you
have
a
fully
connected.
You
know
fully
dense
network.
A
What
they
did
is
in
order
to
create
these
sparse
networks.
They
kept
the
number
of
operations.
The
flops
count
constant
as
close
to
constant
as
possible.
Okay.
So
this
is
a
fully
dense,
fully
connected
network,
so
to
get
a
sparse
network
here
that
has
this
similar
flop
sound,
they
increase
the
layer
widths,
okay,
so
they
effectively
increase
the
dimensionality
of
the
network.
So
these
these
are
sparse
networks
that
are
embedded
in
a
higher
dimensional
structure
than
than
these
guys.
A
G
A
H
A
G
A
Exactly
we
would
try
to
cluster
the
we
would
sort
of
create
more
clustering
in
our
randomness.
If
you
will
instead
of
complete,
when
we
have
two
layers,
instead
of
completely
randomly
connecting
them,
we
would
try
to
include
a
little
more
clustering
and
we
could
do
that
in
a
few
different
ways.
But
that's
one
implication
of
this:
can
clustering
comes
from
learning
in
yeah?
That's
a
great
question.
They
addressed
that
a
little
bit
in
the
paper.
A
I
don't
have
the
graph
here,
but
what
they
say
is
that
if
you
start
with
a
fully
dense
density
connected
network
and
then
it
learns,
they
look
at
look
to
see.
Does
some
sort
of
this
clustering
naturally
emerge
and
it
looks
like
in
their
their
findings
it
kind
of
did,
but
not
perfectly
so.
It
seemed
like
there
was
some
pressure
or
something
to
move
towards
this,
but
it
wasn't
great
and
if
they
impose
this
as
a
prior
as
a
structure
up
front
and
then
they,
then
it
works
much
better.
G
A
C
Okay,
so
can
I
ask
on
this
plot
so
the
top
one
error
it
seems
like
there's
quite
a
lot
of
noise
like
there's
quite
a
lot
of
variance
for
a
given
clustering
coefficient
in
path
length.
Is
that
just
due
to
run
by
run,
you
know
randomness,
or
are
they
actually
missing
a
much
a
curtain?
Yeah
actually
really
well
predict
performance.
A
Very
good
question,
as
I
was
putting
this
slide
together
this
morning.
I
noticed
something
that
was
a
little
bit
bizarre
well,
this
could
be
an
artifact
of
the
way
they're
doing
it.
I
don't
know
how
literal
they
are
here,
but
you
can
see
this
yellow
rectangle
here
and
they're
saying
this:
yellow
rectangle
corresponds
to
this
graph.
A
Well,
this
yellow
rectangle
has
a
you
know
supposed
to
look
at
average
path
length,
but
actually
it
already
has
a
bias
towards
what
am
I
trying
to
say
there
is
a
correlation
between
clustering
coefficient
here
and
pamphlet
already
right
right
here.
So
if
you
were
just
to
look
at
this
bottom
part
here
where
which
would
be
path,
lengths
between
say,
1.5
and
2.5,
you
know
that
would
be
kind
of
in
this
part
here
now
you
don't
see
this
curve
anymore.
It's
not
it's.
C
C
So
what
you're
saying
is
there
could
be
a
single
metric
underlying
both
of
these
curves
that
happens
to
produce
a
curve
in
two
separate
dimensions,
but
there's
really
only
one
metric
that
matters
yeah.
What
I'm?
What
I'm
wondering
is?
Let's
say
we
zoom
in
and
just
take
a
little
tiny
square
region
right
in
the
middle
of
this
plot.
C
C
Some
of
these
biases,
even
in
that
square
you're
going
to
have
a
large
range
of
performance
is
like,
is
that
just
this
specific
instantiation
of
the
graph
was
lucky
or
unlucky
yeah,
if
so
like.
What
is
it
about
that
instantiation?
That
makes
the
graph
luckily
you're
unlucky
like?
Could
you
could
you
like
poke
around
that
graph
and
find?
Yes,
there
are
two
clusters
that
are
barely
connected
or
something
like
that.
A
A
B
A
A
Yeah,
I
think
that's
a
great
question
and
the
question
about
hyper
parameters.
I
think,
is
very
pertinent.
We
found
that
the
hyper
parameters
at
this
end
should
be
quite
different
than
the
hyper
parameters
here,
so
it
might
be
that
at
each
of
these
points
you
really
need
to
do
an
exhaustive
hyperparameter
study
as
well
to
really
get
the
answer,
and
maybe
that
might
explain
some
of
what
jason
was
asking
about
as
well.
E
Super
diet,
could
you
talk
a
little
bit
about
on
the
last
slide
there,
the
sparsity
element,
so
you
said
that,
like
this
best
graph
had
a
certain
sparse
d
yeah
that
you
know
that
corresponds
to
it's.
I
guess
yeah.
My
question
is
basically
how
should
we
think
about
these
graphs
in
relation
to
sparsity?
A
A
F
F
A
F
A
A
E
G
A
A
It
with
a
grain
of
salt,
so
this
is
using
the
feldman
and
veneson
style
connectivity,
which
has
a
lot
of
flaws,
but
it
is
what
it
is
and
you
get
these
coefficients.
I
will
say
that
in
this
paper,
which
I
I
skim
through,
they
look
at
a
many
different
types
of
connectivity
in
the
brain.
They
also
look
at
like
fmri
range
stuff,
where
they're
looking
at
causal
connections
between
different
brain
regions
and
stuff,
but
you
know.
H
A
G
Styles,
yeah,
I
I
it
is
interesting,
but
you
know
even
think
about
something
like
this
in
cortical
networks.
You
know
it's,
it's
not
just
how
many
hops
to
get
some
place.
You
have
to
have,
let's
say
20
synapses,
to
a
pattern
of
other
sparse
patterns.
Someplace-
and
you
know
you
can't
say.
Oh,
I
got
all
my
connections
because
three
of
my
half
three
times
into
them.
G
A
Yeah,
that's
true.
You
know
that
that
particular
thing
that
you
need
to
have
a
sufficient
number
of
connections
to
recognize
patterns.
I
think,
is
true
for
neural
network
architectures
too,
so
you
can't
you
couldn't
just
have
one
connection
and
transmit
all
the
information
you
still
need.
You
know
yeah.
A
A
I
think
I
I
I've
sort
of
convinced
myself
that
this
really
is
a
pretty
fundamental
property,
and
yes,
this
type
of
this
is
not
the
only
type
of
pathway
there
is
of
the
neocortex
and
many
other
pathways,
but
I
would
be
actually
very
surprised
if
those
other
pathways
also
didn't
have
a
small
world.
So.
A
E
A
I
I
A
That
is
yeah,
okay,
just
a
couple
of
slides,
more
on
discussion,
and
then
we
can
kind
of
start
wrapping
up
or
go
into
more
details.
But,
as
I
mentioned
before,
for
static
sparsity
for
our
networks
with
static
sparsity,
we've
mostly
used
completely
random
sparsity.
So
it
could
be
the
case
that
small
world
sparsity
could
be
a
lot
better.
It's
something
I
really
want
to
try
out
now
with
our
stuff.
A
We've
done
dynamic,
sparse
algorithms,
so
I
forget
who
it
was
that
asked
the
question?
Maybe
jeremy,
you
know:
do
they
end
up
learning
small
world
structures
so,
like
the
stuff
that
marcus
has
worked
on,
you
know,
do
they
eventually
end
up
learning
small
world
structures
or
if
they
don't,
you
know,
should
they
be
implemented
to
prefer
small
world
structure
when
they're
doing
their
rewiring?
A
You
know.
Could
that
help?
This
is
the
point
jeff
you
brought
up
this
to
me
really
explains
why
block
sparse
works
weights
work.
So
when
we
did
hardware
implementations,
we've
had
very
constrained
block,
sparse
weights,
where
you
literally
had
to
have
like
fixed
numbers
of
blocks
in
each
row
and
columns
and
and
regions
and
so
on.
A
When
we
first
started
doing
that,
we
were
very
concerned
because
this
is
really
constraining
the
set
of
weights,
whereas
it
felt
like
totally
random
sparsity
should
work
well,
because,
there's
fewer
constraints,
you
can
really
connect
all
sorts
of
stuff
up,
but
the
blocks
bar
spades
might
actually
be.
Small.
World
might
actually
have
a
small
world
structure
because
they're
locally
clustered
plus
you
have
these
other
connections,
but
more
generally
small
world
sparsity
could
be
very
good
for
hardware
implementations,
because
you're
really
limiting
communication
is
a
big
issue
with
hardware
implementations
and
with
small
world
structures.
A
It
seems,
like
you,
could
kind
of
optimize
the
communication,
efficiency
or
fidelity
in
there.
So
if
you
can
map
your
problem
to
small
world
structures
and
if
you
have
hardware
implement
structures
that
can
that
can
efficiently
run
those
that
might
actually
be
a
really
nice
sweet
spot
for
cortical
structures.
F
G
G
G
F
G
F
F
A
Well,
in
our
networks
we
have
say,
a
set
of
weights
would
be
projecting
to
some
cluster
of
neurons
at
the
next
layer
and
then
another
cluster
of
neurons,
also
in
the
same
layer.
So
we
there
would
be
multiple
blocks
that
are
typically
connected
and
that
might
be
sufficient
to
have
the
small
world
properties.
I
don't
know
because
it's
not
enough
to
have
just
have
clustering.
You
also
have
a
few.
G
G
A
A
A
You're
talking
about
yeah,
that's
a
great
point.
You
know
skip
connections
also
in
the
neocortex
can
act
as
some
of
these
random
connections.
You
can
also,
but
they
tend
to
be
a
little
more
structured
as
well.
So
you
can
also
think
of
them.
They're
also
clustered,
but
some
of
the
long-range
lateral
connections
we
have
in
the
brain
or
they're
just
a
bunch
of
connections
that
are
just
not
explained
by
theory
in
neuroscience,
and
they
may
be
just
there
just
to
satisfy
the
small
world
property.
C
C
The
same
graph
structure
every
layer,
so
you
can
imagine,
rather
than
copy
pasting,
that
graph
structure
every
layer
kind
of
like
generating
a
new
graph
structure
in
subsequent
layers,
but
then
you'd
like
to
still
fold
the
properties
that
are.
You
know
desirable,
like
this
short
path,
length
and
cluster
clustering.
C
C
If
you
tuned
the
shuffling
operation,
for
example,
if
you
had
a
shuffle
that
mostly
didn't
move
anything
but
occasionally
randomly
switched
to
nodes,
I
bet
layer
to
layer.
You
would
get
both
fast
computation,
efficient
computation
and
the
small
world
kind
of
effective
connectivity
that
could
be
kind.
A
Yeah
that'd
be
a
that's
a
really
interesting
idea,
yeah
exactly
if
you,
if
you,
and
particularly
your
permutation
or
shuffling,
you
know
mostly
left
things
intact,
but
randomly
shuffled
a
few
things.
You'd
get
these
effectively
these
random
long
range
connections,
but
you
could
still
maintain
the
block
sparse
nature
for
hardware.
A
A
A
If
you
could
think
of
the
neocortex
is
basically
a
2d
sheet,
that's
scrunched
up
and
stuffed
into
our
skulls,
but
you
get
all
these
particle
columns
that
are
laid
out
in
this
way.
We
often
draw
them
as
being
connected
laterally
like
this,
because
in
our
theory
there's
a
voting
operation
that
goes
on
laterally
between
cortical
columns,
so
we
often
draw
them
this
way
and
we've
actually
in
our
implementations,
we
implement
them
this
way.
Well,
what
does
that
mean?
A
It
might
be
that
the
number
of
long
range
connections,
a
small
number-
should
be
sufficient
in
we.
We
do
see
a
lot
of
long-range
connections
throughout
the
neocortex.
That's
not
explained
by
theory.
I
think
the
small
world
stuff
sort
of
says
that
that
does
not
have
to
be
a
very
large
number
in
order
to
get
very
efficient
setup
communications
and
across
the
different
regions.
A
A
The
other
okay,
these
are
less
well-formed,
so
first
of
all,
small
world
networks
seem
to
be
great
at
synchronizing
oscillations
across
brain
areas.
So
we've
talked
a
little
bit
in
our
theories
about
synchronizing
oscillations,
but
small
world
structures
are
good
at
that.
So
that's
nice,
jeff
briefly
mentioned
sdrs
and
population
codes.
I
think
those
things
and
active
dendrites
naturally
lead
to
locally
clustered
graphs.
A
So
again,
this
this
clustering
property,
I
think,
would
naturally
show
up
anyway
in
general,
it
may
make
really
large
scale
learning
easier
to
have
these
structures.
Okay,
so
that's
all
I
had
you
know
actively
prepared
anything
else.
You
guys
want
to
talk
about.
We
can
go
if
you
want,
we
can
go
to
the
paper.
I
know
it's
been
a
it's
been
a
we've
been
over
an
hour
now,
so
we
can
also
stop
any
time.
C
A
Yeah,
so
what
so,
if
small,
world
structures
really
lead
to
better
accuracy,
what
I
was
wondering
is
what
does
that
say
about
the
lost
landscape?
We
know
that
dense
networks,
if
you
look
at
dense
networks
compared
to
completely
random
static,
sparse
networks,
the
random,
sparse
networks
have
their
lost
landscape,
looks
really
bad.
It's
like
extremely
lumpy
and
and
not
smooth
at
all.
I
think
I
saw
a
paper
about
that
somewhere.
A
Could
it
be
that
the
lost
landscape
also
looks
reasonably
sufficiently
nice
to
allow
learning
to
happen?
That
was
just
a
conjecture.
I
think
it'd
be
interesting
to
look
at
that.
I
don't
know
what
measures
there
are
of
you
know.
We
could
measure
the
smoothness
of
lost
landscapes
or,
if
there's
some
other,
better
measure
that
can
characterize
this.
Maybe
speed
up
learning
could
be
one.
A
We
found
in
in
with
random
static,
random
static,
sparse
networks
that
momentum
really
hurts
in
inaccuracy,
which
was
kind
of
makes
sense
if
the
lost
landscape
is
really
bad,
but
momentum
is
a
really
nice
thing
to
have
and
it
speeds
up
learning.
So
maybe
if
we
have
small
wall
structures,
we
could
put
momentum
back
in.
Maybe
it'll
work
better
if
it
looks
reasonably
well
anyway,
these
are
sort
of
some
of
the
thoughts
that
are
going
in.
I
wonder
if
it
would
be
interesting
to
you
know
actually
actively
investigate
that.
F
So
to
this,
if
it's,
if
it's,
if
it's
oscillating
up
and
down
that
to
me,
implies
a
couple
of
things
a
it
doesn't
matter
which
way
the
values
are,
but
if
basically,
you
get
some
very
strong,
stable
region,
I
want
to
kind
of
converge.
You
know
quickly
to
the
point
where,
okay,
this
is
locked
on
down
and
if
these
other
guys
are
oscillating
up
and
down,
I
I
don't
really
want
to
to
magnify.
I
want
to
critically
damp
whatever
that
is
or
ignore
it
all
together.
F
So
it's
I
think,
when
you're
solving
a
set
of,
I
don't
know
stiff
differential
equations
or
something
like
that.
What
the
analogy
is
is:
is
that
you,
if
it's
not
uniformly
stiff,
if
it's
variable,
you
need
to
have
an
adaptive
technique
that
looks
to
see
the
nature
of
things
once
you've
explored
the
landscape,
characterize
the
landscape
and
then
converge
quickly
on
to
where
you
can
put
your
most
effort.
F
C
C
No,
not
many
people
saw
it.
We
found
some.
We
found
a
really
interesting
behavior,
where,
if
you're
training,
all
the
layers,
at
the
same
time,
with
the
same
learning
rate
and
momentum,
it
seemed,
and
we
measured
it
later,
that
some
layers
kind
of
like
learn
faster
than
others,
and
then
it's
like
they
adapt
faster
and
other
layers
adapt
more
slowly.
And
if
you
slowed
down
the
slow
layers
too
much
and
the
whole
system
kind
of
became,
unstable
and
like
did
create
one
of
these
oscillatory
modes.
C
F
Really,
you
know
that
are
used
in
these
large
systems
of
equations
and
not
just
two-dimensionally
could
be
applicable.
You
know
to
this
thing
to
really
speed
things
on
up
locally,
it's
it's
harder
to
visualize
how
you
do
that
efficiently
on
gpu,
because
gpu
likes
to
look
at
stuff
really
uniformly
and
just
ram
things
through.
You
know
as
much
as
possible,
but
it
might
be
worth
it
if
you,
if
you
break
it
up
in
the
blocks
or
or
or
quantize
it
in
some
kind
of
spatial
sense.
F
F
So
there's
that
aspect
of
it
as
well.
If
you
make
it
too
sparse,
then
there's
not
enough
channels
to
you.
You
have
like
a
a
really
bad,
auto
encoder.
You
know
you
just
don't
have
enough
capacity
to
get
things
through.
If
you,
if
you
make
it
too
dense,
then
it
has
all
these
alternatives,
which
means
you
know,
things
can
back
and
flow
and
oscillate,
because
there's
more
than
one
way
of
doing
this
thing,
there's
no
clear
winner.
F
You
know
on
many
levels,
this
small
networks,
kind
of
small
world
networks
make
sense
to
me
because
the
the
extremes
obviously
have
problems
and
there's
obviously
going
to
be
a
sweet
spot
in
any
one
particular
thing,
and
I
think
your
point
of
of
of
saying
that
we
don't
necessarily
use
the
same
structure
on
each
of
the
levels
is,
is
true
from
the
hardware
point
of
view.
If
you're
working
on
a
level
you
would
like
the
in
and
out
graphs
to
be
identical
across
the
thing,
so
you
can
use
the
same
allocated.
F
G
A
D
There
there
are
techniques
for
visualization,
but
I
don't
know
how
much
you
want
to
read:
read
into
them.
There's
a
paper
by
leigh
adolf
from
2017
that
visualizes
the
lost
landscape
for
things
like
dense
nets
and
resnets,
and
they
you
can
kind
of
see
that
okay,
these
are
pretty
smooth
compared
to
when
you
do
it
with
other
networks.
But
in
terms
of
there's
you
know
it's
a
super
high
dimensional
space.
So
so
I
don't
know
how
much
you
can
trust
them
any
visualization
of
it.
C
Just
one
thought,
if
you
guys
do
want
to
pursue
this
direction
in
particular,
if
you
do
want
to
switch
from
your
random
sparsity
to
some
type
of
structured
scarcity
and
then
try
training
networks
and
see
if
they
happen
to
work
better
jeff
mentioned
or
rooted
earlier
that
there's
this
awkward
awkward
choice
of
a
one-dimensional
domain.
To
start
with
this
ring
this
ring
graph
or
you
could
just
as
awkwardly
choose
a
two-dimensional
great
domain
to
start
with,
but
in
any
case
you
might
not
want
to
have
to
make
that
sort
of
choice.
C
So
there
are
small
world
there
are
methods
of
generating
small
world
networks
that
don't
require
you
to
choose
this
type
of
manifold
up
front.
So
I
think
mitchell
could
probably
talk
more
about
this.
There's
like
barbasi
albert
something
or
other
intuitively.
You
can
just
you
can
start
randomly
connecting
things
and
then,
when
it's
time
to
make
a
new
connection,
you
can
evaluate
all
possible
connections
and
you
could
say
directly
sample.
C
D
No,
although
yeah
it
would
be
really
interesting
to
think
about.
I
think
that
this
is
something
that
came
up
in
our
in
our
meetings.
One
time
was
like
a
certain
dimension
out
like,
depending
on
the
dimensionality
of
the
space.
If
you
just
imagine,
you
know
a
grid,
and
then
you
connect
nearest
neighbors
there
and
then
three
dimension,
like
as
you
sort
of
increase
the
dimensionality
there.
D
You
are
actually
increasing
your
your
clustering
coefficient
and
because
it's
almost
like
increasing
k
in
there
on
the
algorithm
that
you're
presenting
so
it's
like
it
would
be
interesting.
I
mean
okay,
I
don't
know
what
I'm
talking
about
here,
but
it'd
be
interesting
to
see
what
sort
of
sparsely
patterns
are
induced,
for
example,
in
2d
versus
1d
and
in
3d
versus
2d,
and
you
know-
maybe
I
don't
know
like
whatever.
If
there's
any
sort
of
thing
that
you
know
that
the
brain
looks
like
to
see.
D
A
Well,
thank
you
all
for
coming.
It
helped
satisfy
my
need
to
talk
about
this,
because
I
was
just
stuck
in
this
right
house.
Hopefully
it
was
helpful
to
some
some
of
you
and
hopefully
interesting,
I'm
going
to
keep
looking
into
this
a
little
bit
and
see
if
we
can
apply
to
our
stuff,
but
thank
you
so
much.
Everyone
for
participating.