►
From YouTube: Bay Area Rust Meetup May 2016
Description
Bay Area Rust Meetup for May 2016.
Help us caption & translate this video!
http://amara.org/v/2Fhx/
A
Oh,
hey:
we're
live
we're
on
the
Internet,
hey
everybody
welcome
once
again
to
another
rust
meet
up.
Thank
you
all
so
much
for
coming
this
month
we
have
the
rust
birthday
and
Frank
McSherry
flew
in
all
the
way
from
Europe
to
talk
about
timely
data
flow.
As
always.
Thank
you
so
much
mozilla
for
feeding
us.
There
is
lots
of
pizza
and
lots
of
cake.
A
So
anyone
on
the
internet,
if
you
want
to
get
here
and
less
than
an
hour,
feel
free,
but
before
we
actually
get
into
the
passage
at
festivities,
I
think
that's
how
you
say
it.
I
wanted
to
at
least
do
a
call
out
that
we
are
doing
a
survey
of
the
rust
community
and
also
the
not
rust
community,
but
is
potentially
interested
in
it.
So
you
know
here
are
some
links
on
on
how
to
actually
get
to
the
the
survey
itself.
You
can
actually
just
go
to
a
blog,
ruslan,
org
and
fill
it
out.
A
It
should
take
about
5-10
minutes.
It
would
be
really
helpful
in
a
house
help
us
figure
out
where
to
take
rust
over
the
next
couple
years.
So
this
is
good
man
annual
tradition,
so,
oh,
oh
so
anyway
we
have
a
birthday.
So
first
off
Aaron
wanted
to
talk
about
Oh,
Nico,
Nico
and
Aaron
someone.
Someone
is
going
to
talk
about
birthday
stuff,
so
please
give
them
warm.
Welcome.
B
B
So,
first
of
all,
we've
had
some
people
gathering
some
numbers,
so
this
is
mostly
thanks
to
Hugh
and
Wilson
I'm,
just
what
has
happened
numerically
in
rust
in
the
past
year,
so
we've
had
almost
12,000
commits
by
seven
hundred
and
two
contributors
to
the
core
rust
repository.
So
this
is
not
counting
the
many
other
libraries
that
live
under
the
rustling
organization,
and
this
is
particularly
encouraging
right.
B
So
we
had
over
a
thousand
contributors
prior
to
in
one
point:
oh,
you
know-
and
there
was
some
worry
that
like
oh
maybe
this
was
sort
of
the
hype
around
one
point
out,
but
actually
people
are
still
contributing
a
year
after,
in
almost
the
same
volume,
we
got
all
the
way
up
to
being
in
one
point
out:
that's
extremely
encouraging.
We
merged
88
rfcs.
We
introduced
18
new
compiler
targets
and
ships,
10
releases
and,
of
course,
delivered
one
year
of
rust.
That
does
not
break
the
way
that
we
were
doing
before.
One
point
out.
B
Now
I
want
to
say
one
of
the
things
so
I
didn't
put
this
on
the
slide,
so
those
88
RFC's
merged.
So
that's
an
average
of
almost
20
RFC's
a
week,
which
is
a
pretty
aggressive
evolution
story
and
there
are
some
other
interesting
figures
in
terms
of
trade
publications.
So
not
a
single
day
has
gone
by
in
the
past
year
that
somebody
hasn't
published
a
new
rust
rate
every
single
day.
Someone
is
publishing
a
brand
new
library
in
rust
and
in
fact,
most
weeks
we
get
about
53
brand-new
crates,
not
new
versions
of
crates.
B
New
crates,
I
think
that's
incredible.
This
community
continues
to
blow
me
away
every
day.
Okay,
so
those
are
some
numbers
there's
one
more
number.
This
was
probably
my
personal
favorite
number
of
the
year,
so
stack
overflow,
you
know,
did
their
usual
annual
survey
of
developers
and
one
of
the
things
they
measure
is
sort
of.
How
do
people
feel
about
the
platform's
of
developing
with?
So
this
is,
if
you're
developing
with
language.
B
How
much
do
you
want
to
continue
developing
with
it
and
not
only
is
rust
on
top,
but
it's
on
top
by
a
significant
margin
and
I
feel
the
love
like
this
is
amazing
and-
and
you
know,
I
think
if
you're
active
in
the
rest
community,
you
can
fill
this
too.
We
all
have
a
lot
of
affection
for
the
project.
It
makes
it
really
fun
to
be
involved.
B
Okay,
we
are
also
starting
to
see
rusts
cropping
up
more
and
more
in
production,
so
Brian
Anderson
has
been
working
on
this
amazing
page
with
friends
of
russ,
where
people
can
submit
logos
of
their
organizations
if
they're
betting
on
rust
in
some
way
so
they're
using
rest
and
production
prototyping
things
with
rust,
whatever
somehow
they're
putting
some
money
behind
rust.
So
there
are
a
lot
of
stories
here
and
a
lot
more
icons
than
I
can
show
on
this
screen,
but
I
wanted
to
talk
about
a
couple
of
the
biggest
stories
of
the
ER.
B
So
one
probably
a
lot
of
you
heard
about
because
it
was
featured
in
a
big
wired.
Article
is
Dropbox,
so
Dropbox
is
using
rust
in
production
today
in
a
pretty
big
way.
So
over
the
past
two
and
a
half
years
or
so
they've
been
moving
away
from
Amazon's
web
services
toward
their
own
storage
back
end,
so
they've
developed
new
server
architectures
and
an
entirely
new
sort
of
back-end
software
stack
to
manage
them.
B
A
lot
of
that
software
is
written
and
go
which
has
been
the
traditional
sort
of
Dropbox
backend
story,
but
there
were
some
core
pieces
for
managing
the
data
on
these
servers
where
go
was
just
using
too
much
memory
and
there
was
no
way
that
they
could
sort
of
get
the
footprint
down
where
they
needed
it
to
be,
and
so
they
turned
to
rust
and
they've
been
extremely
happy.
We've
been,
the
core
team
has
been
in
touch
with
them
regularly
asking
them
you.
B
What
do
you
need,
and
their
answer
is
always
faster
compiles,
but
otherwise
they're
they're
pretty
happy
so
that
I
think
that's
that's
been
one
of
the
most
encouraging
success
stories.
That's
a
pretty
big
bet
for
a
pretty
big
company
that
makes
on
rust
and
I'm
sure
that
we'll
see
a
lot
more
following
in
the
years
to
come.
Of
course,
I
would
be
remiss
not
to
mention
Mozilla
another
fairly
big
company
betting
on
rust.
B
So,
of
course,
most
of
you
have
heard
about
the
turbo
project,
which
is
sort
of
the
sister
project
to
rust
a
new
research
browser
engine,
but
that's
not
actually
the
story
I
want
to
tell
today
so
as
of
some
weeks
ago,
I'm
not
sure
exactly
when
it
shipped
Firefox
is
shipping
rust
code
in
production
on
linux
and
OSX.
It'll
start
shipping
in
windows
in
a
few
more
months
now
we're
taking
things
in
a
somewhat
conservative
fashion
right.
B
So
the
way
that
this
is
starting
out
is
we
have
a
c++
component,
and
then
we
have
a
rust
component.
That's
doing
the
same
thing
alongside
and
sort
of
minority
report
style
we're
asking
both
what
they
think.
The
answer
is
and
reporting
any
time
that
the
rust
component
gets
it
wrong.
Well,
I'm
glad
to
say
in
almost
a
billion
recorded
runs
that
have
been
set
up
by
telemetry.
Rust
has
a
hundred
percent
success
rate
and
it's
cos
0
crashes,
so
yeah.
B
Now
this
this
component
is
an
mp4
demuxer.
It's
just
one
place
to
get
started,
but
I
can
tell
you
the
sense
inside
Mozilla
is
a
lot
of
teams
are
chomping
at
the
bit
to
start
rolling
out
rust
inside
of
gecko,
and
so
we've
been
waiting
to
sort
of
see
it
proved
out,
get
it
integrated
into
the
build
system,
but
this
year
you
can
expect
to
see
a
lot
more
big
news
in
gecko
coming
out
of
rust.
B
B
C
Right,
Nico,
alright,
so
that
was
kind
of
what's
happened
over
the
last
year
and
this
is
now
we're
going
to
talk
a
little
bit
about
what's
coming
up
in
the
near
future.
So
first
off
we
have
a
lot
of
meetups
going
on.
You
can
see
here
all
over
the
world,
basically
and
I.
Think
a
number
of
them
are
also
having
special
events
right
around
now.
C
But
after
that
we
have
a
couple
of
conferences
coming
up,
so
one
is
in
is
russ
conf
and
it's
going
to
be
in
September
in
Portland,
that's
kind
of
for
the
west
coast,
people
which
I
guess
is
the
local
audience.
But
if
you're
on
the
East
Coast
like
myself,
you
might
be
interested
or
even
if
you're,
on
the
west
coast
in
Rust
Belt
rust,
which
is
coming
out
of
the
Rust
Belt
in
Pittsburgh,
and
that's
going
to
be
october,
27
and
28,
and
I
think
there
accepting
or
not
yet
people
did.
C
Is
closed
but
I'm
not
sure
if
you
can
register
yet
but
I
know,
hopefully
soon.
No
and
finally,
we
don't
have
a
lot
of
details,
but
we
can
say
that
there
will
be
something
in
Berlin
and
around
similar
time
frame
and
hopefully
that
will
be
announced
in
the
next
few
days
and
maybe
some
other
European
events
as
well.
So
you
should
definitely
attend
some
of
those.
It's
going
to
be
very
interesting
and
that's
about
all
we
have
to
say,
but
you
know
we
love
rust.
We
hope
you
love
rust,
thanks
for
being
here,
happy.
A
D
Hello
folks,
ya
know
I,
thank
you,
Eric
guys,
I'm,
not
here
for
other
things,
but
eric
has
been
trying
to
get
me
to
give
a
talk
for
a
while
and
I'm
delighted
to
be
here
and
I
didn't
realize.
I
was
a
big
celebration,
there's
cake
for
me
and
everything
it's
so
sweet
and
I'm
going
back
to
Berlin.
Actually
at
the
end
of
the
week
and
I'll
go
to
their
happy
celebrating
event
probably
eat
some
cake
there.
2
and
0
life
is
good.
D
You
know
here
so
I'm
gonna
tell
you
about
sorry,
you're
talking
about
timely
data
flow
and
rust.
This
is
a
project.
I've
been
working
on,
I've,
been
playing
with
rust
for
about
the
past
year
and
a
half
or
so
pre
stable
stuff,
where
I
picked
it
up.
Mostly
by
chance
I
told
some
friends
I
was
gonna,
learn
go
and
they
said
oh
you're,
stupid,
so
I
felt
embarrassed
and
so
I
picked
up.
D
Ralston
said
it
turned
out,
like
you
couldn't
have
told
at
the
time,
but
it
turned
out
that
I
was
a
very
good
choice.
I'm
delighted
I'm
from
one
of
those
seventy
nine
percent
or
whatever,
who
are
going
to
keep
doing
this
for
a
little
while
so
I
time
we
did
flow
and
rust
is
you'll,
see
it's
a
collection
of
some
things
that
that
I've
been
working
on
this
built
on
a
bunch
of
work
that
I've
done
in
the
past,
with
several
other
people
how
to
put
their
names
up
here
because
they're
important.
D
These
are
people
who
are
at
Microsoft
when
I
was
working,
microsoft,
research
and
also
at
ETH
in
Zurich,
where
I
had
been
relatively
recently
up
until
about
january,
or
so
you
can
find
stuff
about
me
on
github
I'm,
delightfully
unaffiliated
at
the
moment,
which
makes
me
very
happy
so
I
don't
actually
have
a
proper
home
page
or
anything
like
that.
But
I
have
github
and
you
can
find
lots
of
things
that
I
do
they're
all
yeah.
A
D
That
I'm
sorry,
this
is
I'm
now
in
screen,
okay
cool,
so
the
talk
is
going
to
be
primarily
about
large-scale
computation,
doing
it
and
rust.
How
rust
in
particular,
has
made
my
life
a
lot
easier
working
with
big
data
and
big
computation,
I'm
gonna,
it's
a
little
bit
of
a
weird
talk,
because
it's
going
to
hit
a
few
things
that
are
interesting
and
new
in
Big
Data
computation.
But
it's
also
going
to
try
to
touch
on
if
you're,
keen
about
rust
and
and
trying
to
figure
out.
D
Is
it
a
good
idea,
bad
idea,
the
moments
that
I've
had
where
I've
had
some
pretty
good
experiences
and
done
things
differently
than
I
have
in
the
past,
because
of
Rustin
have
felt
eventually
smarter
and
better
for
it
after
a
little
bit
of
mortal
combat
with
the
various
components
of
it
of
the
rust
system?
So
let
me
start
with
some
fun
numbers
that
I
like
to
just
motivate
where
I
am
and
why
I
do
things.
These
are
some
numbers
for
some
big
data
computation.
They
were
there
a
little
old
at
this
point.
D
They're
still
I'll
get
some
more
fresh
numbers
towards
the
end
of
the
talk,
but
this
paper
on
graph
processing
and
there's
a
bunch
of
exciting
popular
systems
for
doing
they
drank
in
this
case,
20
iterations
of
page
rank
on
a
few
different
data
sets
on
lots.
Of
course,
this
is
a
cluster
of
in
this
case,
16
machines
grinding
away
and
doing
some
things
in
hundreds
of
seconds
and
it's
pretty
exciting,
and
no
one,
no
one
had
a
great
sense
of
where
there's
a
brilliant
numbers
and
not,
but
they
thought
sure
they
look.
D
Serve
difficult
people
went
and
I
sir
pulled
up
my
laptop,
which
is
somewhere
up
there.
Please
don't
run
away
with
it
and
that
you
know
you
can
write
fairly
simple
things.
This
could
happen
to
be
in
rust
that
I
wrote,
but
it
actually
works.
Similarly
in
c-sharp
we
also
did
it
and
not,
but
you
know
a
little
bit
of
care
and
attention.
You
can
get
relatively
good
performance
numbers
without
all
of
the
exotic
paralympic
data
systems,
which
was
a
little
surprising.
I
mean
you.
D
These
scalable
systems
seem
to
have
a
lot
of
gunk
sort
of
built
into
them.
That
makes
things
go
slow
and
you
know
these
are
the
people
we
went
to
to
try
to
actually
get
the
computations
that
were
too
hard
to
manage
to
go
to
go
excellently
and
in
these
particular
cases
at
least
they
didn't
really
go
any
faster
than
then
our
laptops,
so
that
was
a
little
bit
of
a
surprise
and
for
a
while
we
were
quite
smug.
I'm
like
look
how
clever
we
are
at
which
point
people
started
to
say
things
like.
D
Maybe
you
could
stop
being
so
obnoxious
and
start
to
be
helpful
and
that's
sort
of
what
I
tell
you
about
in
this
talk,
which
is
a
system
that
takes
some
of
the
similar
ideas
in
just
like
attention
to
detail.
Basically,
it's
gonna
be
in
rust,
but
let
me
to
be
honest:
it
doesn't
have
to
have
been,
and
rusted
Russ
just
happened
to
make
my
life
delightful.
D
When
I
did
this
and
I'm
very
happy
when
I
use
rust-
and
that
was
the
only
way
this
is
going
to
come
into
existence
anyhow,
so
so
we're
going
to
talk
through
some
of
these
things.
Actually
let
me
the
next
slide
is
a
stack
picture.
So
we
have
some
parts
of
the
talk.
That's
made
timely
data
flow,
which
I'll
tell
you
a
bit
about
it's
a
few
things.
D
D
So
I'll
show
you
one
of
those
differential
data
flow,
which
is
essentially
a
collection,
oriented,
programming
language
that
magically
updates
in
real
time
and
does
a
few
things
that
previous
systems
just
haven't
been
able
to
do.
That
I
think
is,
is
is
pretty
cool
and
you
can
put
on
top
of
these
neat
applications.
Hopefully,
we'll
get
to
a
few
examples
of
these
towards
the
end
and
I've
drawn
the
picture
this
way,
so
that
you
can
see
that
they,
because
of
the
way
we've
layered
things
they
touch
down
wherever
you'd
like.
D
If
you
want
to
write
stuff
against
core
timely
data
flows.
Great,
that's
that's
delightful!
If
you
want
writes
up
against
bris
higher-level
libraries,
both
on
top
of
it.
That's
all
so
delightful
they
all
sort
of
compile
down
to
the
same
thing.
So
you
can
pick
and
choose
what
you
want
to
use
and
hopefully
that's
very
exciting,
appealing
to
well.
You
know
two
people
who
want
to
use
the
goods
thing
of
the
pre
propagated
stuff
when
it's
appropriate
and
when
they
really
need
to
get
in
there
and
change
bits
and
pieces.
That's
that's
open
to.
D
So,
let's
start
just
with
the
timely
data
flow
stuff,
the
delightful
keno
innovations
if
anyone
hasn't
used
those
there's
nice.
If
people
have
questions
at
any
point,
especially
clarifying
questions,
if
I've
said
something,
that's
confusing
or
you
don't,
you
don't
entirely
understand,
feel
free
to
ask
I'm
gonna,
be
here
after
the
talk.
So
if
there's
like
deeper
dives,
you
want
to
take
into
things
you
know.
Maybe
you
hold
on
to
those,
but
if
at
any
point
I've
said
something
that
seems
weird,
it
probably
seems
weird
to
other
people
too.
D
So
good
more
than
welcome
to
jump
up
and
down
and
make
me
say
it
again,
maybe
with
different
words.
So
this
is
based
off
of
some
work.
We
did
in
SOS
p
in
2013,
but
it's
there
was
some
fundamental
changes
that
went
on
when
this
was
ported
over
to
rust,
mostly
because
we
couldn't
write
the
code
the
same
way.
Oh,
but
it
was
all,
is
all
for
the
best
as
it
turns
out.
You
know,
things
became
better
because
rust
made
me
a
better
person.
D
It
pointed
out,
although
the
places
where
I
did
something
totally
inappropriate
and
I'd
fix
that
and
now
I'm
dang,
so
timely
data
flow
I'm
going
to
walk
through
a
very
simple
example.
It
is
a
crepe
then,
and
such
for
doing,
data
parallel
programming.
So,
what's
going
to
happen
in
the
fullness
of
time,
is
you're
going
to
write
a
program.
That's
going
to
get
spun
up
on
many
potentially
many
different
computers.
D
D
Shortest
sort
of
hello,
world
style
program
you
could
write,
then
it's
going
to
print
some
numbers
to
the
screen.
Basically,
first
off
there's
your
program.
Now
your
program
is
the
whole
thing.
You
write
this
once
even
though
you're
going
to
run
it
a
few
times
on
a
few
different
computers,
there's
one
hunk
of
code
that
you
have
to
think
about
and
put
together
that
actually
gets
run.
You
don't
write
the
program
differently.
If
you
have
a
different
set
of
machines,
you
don't
have
to
write
different
programs
for
each
of
the
different
machines.
D
So
there's
just
one
program
you
and
your
laptop
or
whatever,
wherever
you
enter
code
into
that
gets
written
once.
There's
this
book
within
sort
of
the
main
region
which
so,
as
you
might
expect,
is
the
process
of
each
time.
You
you
spin
up
one
of
these
computations
on
a
machine.
Maybe
this
is
only
once
because
you're
just
doing
on
your
money
laptop,
this
is
going
to
get
run
and
this
isn't
the
most
exciting
part.
This
typically
just
calls
into
timely
and
says:
hey
I'd
like
to
participate
in
a
big
data.
D
G
D
In
this,
this
inner
little
bit
of
copious
is
inside
this
disclosure
that
gets
passed
into
into
timely,
and
this
is
the
the
per
worker
logic.
So
there
are
lots
of
people
lots
of
threads
that
are
going
to
participate
in
some
computation,
and
this
is
the
point
of
our
program.
We
tell
them
what
should
you
do?
What
you
know
you
want
to
help
out
with
this
big
data,
parallel
computation.
D
What
should
each
of
you
would
treat
each
of
you
go
and
do
and
it's
you
know
it's
just
gonna-
be
rust
code
right,
so
nothing
particular
exotic,
we're
not
reading
a
different
language
or
anything
like
that.
Yeah.
Let's,
let's
walk
through
what
was
it
example
dose
for
example,
so
it's
rust
code.
We
start
in
this
case
just
with
a
rust
iterator,
if
you've
seen
this
before
great
I'm
a
little,
how
many
people
are
absolutely
brand-new
to
rust,
never
seen
it
before.
D
Just
running
computation,
random
bits
of
rust
code
on
that
you
can
write
whatever
we
want
and
we're
going
to
start
with.
Just
a
rusted
Raider
00
up
to
10
is
0
through
9.
It's
just
a
little
bit
of
code
to
return
some
some
data,
nothing,
nothing
magical
about
that.
Yet
no
no
fancy
distributed
anything.
D
Yet
though
there
are
methods
proposed
exposed,
sorry
by
timely
through
the
dataflow
operators,
space
that,
let
you
convert
things
like
rusted
Raiders,
into
distributed
streams
or
collections
in
particular,
is
to
stream
operator,
takes
this
scope,
which
is
basically
a
handle
to
the
data
problem.
Runtime
and
says
you
know
what
I'd
really
like
to
do.
Is
I'd
like
to
take
this
data
that
I
have
here
and
convert
it
into
distributed,
pile
of
data
distributed
stream,
I've
written
and
it's
still
conceptually
0
up
through
9
on
each
of
the
the
independent
workers
there.
D
Now
all
of
the
workers
know
that
it
has
a
name
every
all
the
other
workers
know
about
it,
sort
of
a
shared
pile
of
data
that
we
can
start
to
interact
with,
in
particular,
for
example,
we
can
do
something
like
a
data
exchange,
so
we
have
a
bunch
of
workers.
Let's
imagine
hypothetically
five
workers
up
there,
who
are
each
thinking
about
the
number
0
up
through
nine,
and
we
just
told
them
to
take
the
number
0
through
9
and
make
them
into
some
potentially
distributed
collection
of
data.
D
They
did
that
the
reef
sitting
now
on
the
number
0
through
9
and
they're,
not
really
sure
what
to
do
with
him
yet,
but
they
know
that
there
are
four
other
people
who
have
a
similar
pile
of
data.
The
exchange
operator
is
going
to
tell
them
now.
Next
thing
you
should
do
is
shuffle
all
this
data
around
between
all
of
you.
This
is
a
fairly
common
pattern.
E
D
Lot
of
these
big
data
systems
right
a
lot
of
them
work
by
describing
what
individuals
should
do,
individual
workers
should
do
and
how
you
should
move
data
around
between
between
the
individual
workers.
What
the
exchange
operator
here
does
is
allows
you
to
say
for
each
record
that
flows
by,
if
you
show
me
the
record
I'll
tell
you
a
number
as
it
turns
out,
and
the
number
is
going
to
drive
which
workers
should
I
route.
This
particular
piece
of
data
tube,
so
we
send
all
the
zeros,
for
example,
to
the
same
worker
worker
0.
D
D
Now
now
we
have
a
distributed
collection
of
data
and
the
data
in
different
places
than
where
they
started
out.
So
that's
sort
of
a
this
is
again
it's
a
hello
world
example.
Hopefully
you're
not
too
excited
about
this.
You
haven't
gotten
your
checkbooks
out
yet,
but
we've
moved
some
data
around
and
we're
now
going
to
do.
A
thing
is
inspect,
which
we
try
to
borrow
as
often
as
possible,
iterator
idioms
from
rust,
so
that
you
still
you
have
the
same
sorts
of
vibe
when
you're
working
with
iterators,
as
you
have
with
these
distribute
streams.
Well,.
D
Of
thing,
this
will
every
record
that
goes
by
it'll
just
call
some
rust
code
in
this
case
we're
going
to
print
something
to
the
screen,
but
rather
than
have
each
of
the
workers
print
0
through
9
worker
0
is
going
to
print
a
whole
bunch
of
zeros
and
then
a
whole
bunch
of
fives
ones.
And
six
is
that
sort
of
thing.
So
we've
caused
the
data
to
move
around
a
little
bit
and
do
something
different
on
each
of
the
workers.
Then
then
it
would
have
done
if
we
hadn't
actually
had
a
distributed
computation
with
communication.
D
Let's
do
something
more
exciting,
then
that's
a
good
point,
so
we
do
something
a
little
bit
more
exciting.
Now
the
font
is
a
little
smaller
I
hope.
It's
still
readable
in
here
I'm,
going
to
show
a
bit
more
of
an
exciting
example.
Now
the
previous
one
was
basically
a
batch
of
computation.
It
was
there's
a
pile
of
data,
it
exists
and
we
act
on
which
is
cool.
You
know
a
lot
of
people
like
to
do
that
sort
of
thing,
but
it's
not
really
a
data
stream.
D
Yet
we're
not
feeding
data
into
it
and
seeing
cool
stuff
come
out
the
other
end
or
anything
fun
like
that.
We're
just
going
and
we're
done
and
program
ends.
So
let's
write
something:
it's
a
bit
more
interactive
and
it's
pretty
similar,
there's
I
feel
very
bad
about
lying
on
my
slides
about
codes,
so
the
timely
examples
think
doesn't
technically
apply
anywhere
side
to
change
it,
but
timely,
execute
from
args
is
the
right
thing
to
call
to
run
this
example.
D
Oh
with
that
input,
we're
going
to
and
bear
with
me
for
just
a
moment
in
the
context
of
some
method
that
I
haven't
explained
yet
we're
going
to
turn
it
into
a
stream.
We're
gonna
do
this
exchange
you're
gonna
print
some
stuff,
and
then
we
were
going
to
call
the
probe
method
on
the
end
and
that's
okay.
Who
knows
why
we
did
that
I'll.
Try
to
explain
the
timely
part
of
timely
data
flow
is
about
attaching
logical
notions
of
time
to
the
bits
of
data
that
we
have
in
our
in
our
hands.
D
When
we
just
have
this
batch
computation,
we
had
some
data
just
existed.
There
wasn't
really
a
notion
of
time,
which
is
just
there.
What
we'd
like
to
do
is
create
a
new
scope
to
work
with
data
in
which
the
data
have
an
interesting
notion
of
a
non-trivial
notion
of
time,
for
example,
in
this
case,
we're
going
to
have
unsigned
integers
that
go
up
from
0
to
10.
D
This
could
be
10,
it
turns
out,
but
you
know,
could
go
on
indefinitely
as
we
interact
with
with
the
data
flow,
so
the
scooped
operation
basically
builds
a
little
bit
of
our
data
flow
canvas
that
we're
going
to
work
within
we're
inside
this
canvas.
All
of
the
data
that
circulate
around
have
some
notion
of
time
associated
with
them.
This
is
unsigned
integer.
D
That
is
going
to
correspond
to
rounds
of
in
the
probe
operator
is
a
little
bit
of
sort
of
a
dual
here
where
it
lives
on
the
end
of
the
data
flow
and
is
the
thing
that
can
tell
us
how
much
progress
that
we
actually
made
through
this
stream.
You
know
great
you're
talking
about
zeros
and
ones
and
twos
with
respect
to
your
time
stamps.
Have
we
seen
all
the
zeros
yet
and
we've
seen
all
the
ones
that
we've
seen
all
the
twos.
D
So
just
to
finish
out
the
example
there's
a
second
having
built
up
the
little
bit
of
data
flow.
They
remember,
but
there's
a
second
hunk
of
computation
will
read
drive.
It
basically
take
the
input
and
we
take
the
probe
and
in
this
case
we're
going
to
go
through
the
numbers
again:
zero
up
through
ten
we're
going
to
do
a
few
things
we're
going
to
feed
the
current
round
into
the
input.
So
everyone's
going
to
stuff
a
zero
into
the
input
sort
of
introduces
a
zero
as
it
turns
out
at
time,
zero.
D
The
input
gets
advanced
and
what
that
means
is
that
it
moves
the
time
stamp
forward
to
one
and
that's
a
very
important
action
to
take.
That
tells
everyone
else
in
the
system
that
this
worker
is
no
longer
planning
on
sending
anything
that
has
a
zero
time
stamp
on,
which
is
very
helpful,
because
now
all
the
other
workers
that
they're
listening
can
say.
Oh
good,
if
we
finish
all
of
those
messages
that
you
sent
and
they
get
out
of
the
system,
we
can
comfortably
decide
collectively
that
we're
done
with
that
round
of
computation.
D
We
can
get
announced
to
whoever's
interested
that
we've
made
some
progress
and
we're
good
to
go,
which
is
actually
what
we
do
in
this.
The
third
step
here
is
basically
tell
the
worker
why'd
you
keep
running
for
a
little
while
in
fact
keep
running
until
that.
It's
no
longer
the
case
that
that
probe
thing
you're
looking
at
is
strictly
less
than
the
current
input
time.
D
I
D
D
That's
basically
scheduled
everyone
in
the
dataflow
wants
go
through,
go
through
all
the
people
who
could
do
some
work,
the
exchange
operator,
the
expect
inspect
operator,
we'll
see
in
the
very
next
slide
what
it
is
they
might
do,
but
you'll
give
them
a
chance
to
do
a
little
bit
of
work,
to
look
at
some
data
and
then
and
then
come
back.
That
might
not
have
finished
everything
so
step,
while
is
just
a
helpful
little
function.
D
That
says
as
long
as
this
predicate
evaluates
to
true
I,
just
keep
doing
that
it
says
many
times
through
the
body
as
it
takes
two
to
get
all
the
work
done.
In
this
case.
Sorry,
this
probe
less
than
input
time
will
become
false.
Once
all
the
data
hub
for
epoch,
0
have
flowed
through
the
graph.
You
know
maybe
there's
some
other
worker
you've
set
up
on
some
foreign
continent
who's
taken
quite
a
while
to
get
the
data
to
you.
You'll
hang
around.
D
D
So
in
actually
so
the
question
is
I
think
where
does
inspect
run
run
the
actual
code
and
the
way
it's
going
to
work
is
each
worker
is
going
to
have
a
copy
of
inspect
so
every
worker.
If
there
are
ten
workers
out
there,
there
may
be
10
copies
of
inspect
and
there
can
be
driven
by
what
data
they
actually
receive.
So
if
you
send
all
they
did
it
to
one
worker,
only
one
worker
is
actually
going
to
do
that
particular
work.
D
If
you
spray
the
data
across
many
workers,
many
different
people
will
do
the
work,
but
it's
driven
by
how
you
distribute
the
data
as
opposed
to
trying
to
place
a
particular
worker
on
a
particular
machine.
Yeah,
cool,
okay,
there's
a
question:
no,
no!
That's
good!
Okay!
No!
Sorry!
It's
not
good
I'll
carry
on
I'm,
sorry
again
feel
free
to
grab
me
or,
if
you're
actually
seriously
like.
If
there's
an
issue
jump
up
and
down
and
and.
D
I'll
keep
going
until
this
more
jumping,
so
I
did
want
to
point
out
that
this
you
know
I.
Maybe
I
called
this
up
a
little
bit,
but
this
is
a
very
exciting
moment
in
terms
of
timely
data
flow,
this
a
little
bit
of
probe
less
than
input
that
time.
This
is
the
moment
where,
although
we've
written
a
little
bit
of
local
code,
this
is
you
know
local
thread
running
and
saying.
G
G
D
Is
where
we
take
information
about
the
distributed
state
of
the
computation,
so
things
that
might
be
true
on
other
machines,
other
other
workers,
wherever
they
happen
to
be?
If
we
get
information
back
that
helps
us
make
a
local
decision
about,
should
we
keep
doing
a
thing
or
not?
So
this
is
where
a
lot
of
the
magic
is
hidden
with
respect
to
timely
data
flow.
The
thing
that
it
does
that's
pretty
cool
is
let
you
write
interesting
data
flow
graphs
and
give
you
information
about.
D
Are
we
there
yet
or
not
at
any
particular
point
in
the
data
flow
graph,
and
that
information
can
drive
a
lot
of
fairly
useful
computation?
Oh
yeah,
it's
a
question
all
the
workers
getting
back
their
states,
so
the
question
is
what
is
probe
actually
doing,
and
the
answer
is
actually
it's
a
lot
more
passive
than
that
probe
is
just
sort
of
chilling
out.
There's
some
background
coordination
traffic
that's
going
on.
D
It
doesn't
block
or
anything
like
that,
it'll
return
and
tell
you,
according
to
the
current
state
of
information
where
they're
done
we're
not
done,
but
it
doesn't
doesn't,
doesn't
block
or
anything
like
that.
Almost
all
of
timely
data
flow
everywhere
is
non
blocking
it
sorted
by
design
in
a
data
flow
system.
You
don't
really
know
if
you
can
actually
do
some
work
or
not,
so
it's
great
not
to
block
assuming
that
you
can
it's
this
design
sample
throat.
D
So
if
you
use
streaming
systems
before
you
might
know,
I
can
do
all
of
this.
This
is
not.
This
is
not
particularly
new.
There
are
lots
of
systems
out
there.
Let
you
do
this,
so
let
me
show
you
something:
that's
new!
That's
cool,
but
I!
Think
it's
cool
man.
You
will
get
rid
of
that
silly
exchange
and
inspecting
and
put
in
some
more
exotic
computation,
we're
going
to
throw
in
there
that
stream
of
numbers
they
get
fit
in
and
we're
going
to
feed
them
into
an
iterative
sub
computation.
D
Like
a
million
you
put
a
million
in
and
it's
going
to
count
down
from
a
million
down
to
zero
and
each
time
it
counts
down,
it's
going
to
do
it
exchange
probably
hand
it
to
a
different
worker,
essentially
we're
getting
a
little
data
data
exchange
micro
benchmark
going
on
here
and
there's
a
few
things
that
are
cool.
One
of
the
things
that's
cool
is
that
it's
actually
still
works,
that
you
can
still
get
information.
That
probe
still
comes
back
and
tells
you
yeah
we're
done
or
not-
and
you
might
say
it's
a
big
deal.
D
Nyad
and
timely
de
Flore
notes
were
the
first
systems
that
were
able
to
this
correctly.
Basically,
it's
really
hard.
The
reasoning
mechanisms-
people
I
used
before
I
had
a
really
hard
time
breaking
out
of
loops.
If
you're
in
the
loop,
you
could
tell
what
iteration
you
were
on,
but
it's
really
hard
to
have
a
computation
that
would
spin
for
a
little
while
until
yeah
I'm
done.
Let's
go
on
to
the
next
one.
D
C
D
In
this
case,
actually
something
the
Dakota
has
written.
This
is
great,
actually
brilliant
segue
into
the
very
next
minor
transition
the
code
has
written.
What
will
happen
is
zero
will
go
in
from
all
each
of
the
workers.
Zero
will
get
thrown
away,
they
will
drop
out
and
say,
hey.
Are
we
done
in
the
combinations
yeah
we're
done?
Apparently,
everyone
threw
away
their
zeros
every
well,
then
sticking
a
one.
D
D
Each
worker
introduces
some
number
between
the
apt
up
to
10,
all
of
them
get
dumped
to
the
same
worker
who,
who
picks
them
all
up
and
dumps
them
to
another
worker,
a
victim
of
thompson
to
another
worker,
but
they're
done
in
sequence.
Everyone
does
zeros,
then
they
sort
of
block,
while
everyone
agrees
that
we're
done
with
all
the
serious
and
they
move
to
the
ones
and
they
look
to
the
twos.
So,
there's
a
very
strict
in
this
case
serialization
going
on,
though
Madonna
you
can
like
I
mentioned
before.
D
This
is
a
totally
passive
operation,
this
probe,
so
it
doesn't
actually
block
the
computation
until
something
is
done.
It
just
tells
you
what
the
current
state
of
affairs
is
if
you'd
like
to
build
in
a
bit
of
slack,
for
example,
because
it's
it's
productive
to
have
concurrent
concurrent
bits
of
data
cycling
around
you
can
totally
do
this
example
says
hey
if
we're
within
two
of
the
the
current
time.
D
Let's
keep
things
going,
which
means
that
we
can
have
in
this
case
three
outstanding
rounds
of
data
circulating
at
the
same
time,
and
that's
just
it's
a
productive
way
to
keep
more
of
your
cpu's
occupied
I'm.
This
particular
example
is
a
bit
silly
just
to
begin
with,
but
this
principle
general
a,
but
let's
keep
a
few
outstanding
epochs
going.
At
the
same
time,
it's
a
really
helpful
way
to
get
higher
throughput
out
of
a
streaming
system
at
the
expensive
a
little
bit
of
latency.
D
D
What
round
are
we
on
or
what
was
the
current
time
with
respect
to
walk,
walk
so
yeah,
okay,
I
said
this
already
sorry,
so
what
I
thought
I
do
next
is
show
you
just
a
little
bit
of
a
flavor
of
how
you
might
write
one
of
these
operators.
I
mean
I,
put
a
bunch
of
stuff
up
that
say
things
like
filter
and
map
and
and
inspect
and
whatnot.
The
code
is
actually
pretty
pleasant
or
I
think
pretty
pleasant
to
write
these
things
so,
for
example,
yeah.
D
Okay,
we,
like
writing
things
like
stream
map,
that's
really
nice!
Of
course,
you
can
write
that
it's
something
that
we've
we
put
into
the
core
timely
library,
but
you
could
also
write
this
from
scratch.
If
you
wanted
to
I'm
going
to
show
you
writing
this
operator
from
scratch
and
at
the
same
time,
I'm
going
to
pull
out
a
bunch
of
really
cool
things
about
rust
that
made
this
experience,
I
think
very
pleasant
and
cool
and
saved
a
lot
of
headaches.
D
So
here's
the
same
code
written
out
in
sort
of
a
little
bit
more
like
raw,
timely
stuff.
There
are
a
few
fairly
generic
operators
that
you
can
write
that
describes
the
topology
of
the
operator
in
this
case
unary.
So
one
input
one
output
with
a
few
parameters.
It
sufficient
number
of
parameters
to
describe
the
full
behavior
of
the
operator,
and
there
are
three
parameters
that
get
shown
in
this
case.
There's
a
requirement.
D
D
If
you
want
to
do
distinct,
for
example,
you'd
need
to
move
the
data,
all
the
data
that
are
the
same
move
them
to
the
same
workers,
so
that
worker
can
make
sure
there's
only
one
copy,
but
in
the
case
of
map
they
don't
even
need
to
be
moved
anywhere.
We
want
to
give
a
tasteful
name
to
the
operator,
because
otherwise
it's
really
hard
to
tell
these
things
apart
and
then
yeah.
Finally,
we
give
it
a
closure.
D
It's
actually
give
it
some
logic
which
says:
if
I
were
to
run
you
if
timely,
were
to
schedule
this
operator.
What
should
it
do?
You
know
if
I
give
you
a
handle
to
the
input
on
which
you
can
tug
and
get
some
data
and
handle
to
the
output
at
which
you
can
push
some
data?
What
logic
would
you
actually
like
to
run
and
I've
written
some
here,
which
is
actually
the
logic
we
use
for
for
map
and
I'll?
D
Just
I'll
talk
you
through
it,
especially
if
your
benefit
over
here,
because
there's
some
very
cool
rust
idioms
that
show
up
that
I
take
for
granted
at
this
point,
but
they're
cool
to
just
call
attention
to
so.
The
code
here
is
just
taking
the
input
and
saying
as
long
as
you've
got
data
there.
For
me,
the
importance
or
keep
keep
pulling
as
long
as
they're
still
dated
their
particular
sorry,
a
pair
of
a
time
and
data,
because
all
data
have
time
associated
with
it.
D
D
D
So
you
know
the
data:
are
there
we
go
through
each
of
them
and
for
each
of
them
we
apply
the
logic
and
we
shove
that
into
the
into
the
session.
That's
that's
the
whole
implementation
of
a
map.
So
let
me
call
attention
to
a
few
cool
Russ
things
again.
You
know
some
of
you
were
like
okay,
yeah
thanks
great,
but
first
of
all
like
this.
This
whole
wall
that
some
thing
is
this
I
think
absolutely
delightful
and
I've
totally
taking
it
for
granted
now.
D
But
this
is
a
destructuring
which
ensures
that
you
only
ever
get
to
interact
with
valid
data,
and
that's
just
really
pleasant.
You
there's
no
more
testing
of
nothing
like.
Is
this
null
or
not?
What
is
the
data
still
valid?
Has
it
been
already
have
I
seen
it
already
notes,
you
only
get
to
interact
with
data
that
the
system
has
decided
are
valid
and
it
just
it
rules
out
a
large
class
of
errors
that
previously
I
had
men
of
in
c-sharp
and
I.
Just
don't
have
any
more.
You
don't
have
to
worry
about
this
stuff.
D
So
this
is
nice.
You
know
elsewise,
you
would
have
a
whole
bunch
of
tests.
Saying,
like
you
know,
is
this
null
is
valid
all
sorts
of
code
that
you
just
don't
bother
with
anywhere
we're
using
a
closure
in
the
middle
logic.
Is
a
closure
we're
using
it
here
and
delightfully,
unlike
a
bunch
of
other
systems,
the
closures
happily
result
in
specialization.
This
mon
amour
fixation
that
produces
a
new
hunk
of
code.
D
Each
time
we
give
a
different
batch
of
logic,
which
causes
all
of
this
to
be
inlined,
which
means,
if
we
had
that
map
proper,
they
just
subtracted
one
from
everything.
It's
not
a
virtual
function.
Call
it's
just
code
that
gets
in
lined
and
probably
gets,
gets
vectorized
and
goes
really
fast,
because
subtracting
one
from
a
bunch
of
things
is
pretty
easy.
It's
doable
it's
the
code
that
you
actually
would
want
to
show
up
without
the
whole
pain
in
the
butt
of
manually,
enrolling
the
loop
or
putting
horrible.
D
D
In
a
data
parallel
processing
environment,
where
data
movement
and
data-driven
computation
really
cares
about
who's
holding
on
to
what
piece
of
data
who
actually
is
responsible
for
it,
where
you
know
when
I
hand
it
to
you,
what's
the
contract
here
and
and
ownership
and
mu
semantics
just
make
this
abundantly
clear
logic
gets
the
piece
of
data
in
this
case,
and
these
in
the
the
signature.
Sorry
of
logic
makes
us
one
hundred
percent
clear
that
when
you
write
that
code,
you
now
own
this
resource.
D
If
it
was
a
string
that
I
was
sending
to
you,
you
can
do
whatever
you
want
with
that
string.
Now
you
can
push
characters
on
to
it,
make
it
lower
case
all
the
fun
I.
Don't
we
do
with
this
string,
but-
and
you
can
and
you
can
drop
it
at
the
end
and
rust
is
bright
enough
of
course
to
to
know
if
you,
if
you
just
dropped
it,
it
will
collect
all
the
resources
for
you
afterwards.
D
You
don't
have
to
worry
about
about
that,
and
this
has
always
been
a
big
thing
with
a
lot
of
these
data
parallel
systems.
When
you
write
this
code,
you
might
say,
am
I
the
only
person
who
is
going
to
look
at
datum,
I,
don't
know
if
I
should
d
allocate
the
resources
at
the
moment
cuz
there
might
be
someone
else,
who's
gonna
see
it.
It's
a
disaster.
There's
no
disaster!
Here,
it's
nice
at
the
same
time,
data
the
backing
store
for
all
these
is
dayton's.
D
Ownership
is
not
released
by
drain
it.
It's
retained.
The
the
system
in
this
case
gets
to
hold
on
to
this
this
resource
and
reuse
it
and
that's
also
very
clear
and
all
the
signatures
that
you
don't
actually
get
to
own
data,
and
it
makes
my
life
very
easy,
as
they
system
implementer
I,
get
to
reuse
this
buffer
and
not
allocate
stuff
all
over
the
place,
which
is
great.
D
There
are
few
more
exotic
things
in
this
case
time.
The
time
that
you
get
handed
back
is
in
addition
to
telling
you
what
the
current
timestamp
looks.
Like
you
know,
maybe
the
round
of
data
or
the
iteration.
It's
actually
a
capability
sort
of
an
RA
style
capability,
as
long
as
it
exists,
you're
able
to
send
data.
This
is
something
that
timely
data
flow
is
very
interested
in
paying
attention
to
which
is
which
operators
out
there
might
still
send
data.
D
This
is
sort
of
the
fundamental
thing
it
has
to
think
about
is
okay,
you've
written
an
operator,
I
saw
that
you
received
some
messages
or
you
can
produce
any
outputs
and
the
existence
of
this
little
dude.
The
time
object
is
what
drives
that
you
have
to
hold
on
to
one
of
those
in
in
order
to
produce
outputs
you
have
to
in
some
such
exert
or
demonstrate
the
capability
in
order
to
send
data.
D
But
at
that
point
you
can
you
can
do
a
few
things
you
can
throw
it
away,
in
which
case
the
system
will
notice,
you
can
stash
it
in
a
vector,
and
the
system
will
also
implicitly
notice
that
you
haven't
actually
released
the
capability
yet
so
it's
it's
a
very
pleasant.
It's
a
pattern
that
rust
makes
it's
very
simple
and
consequently,
mix
makes
the
code
a
lot
more
pleasant,
a
lot
less
prone
to
error.
D
In
our
experience,
there
are
a
few
other,
very
clever
things
that
go
on
in
here:
I
had
put
them
down,
but
I
started
to
realize
at
some
point
they
transition
into
anti-patterns,
so
I
was
going
to
stop
talking
about
how
clever
I
was
with
drf,
mute
and
stuff
like
that,
but
we
can
talk
about
that
over
substantially
more
beers
than
I
have
yeah.
Let
me
do
a
few
high-level
points,
though,
about
you
know
things
that
about
rust
that
have
made
my
life
very
convenient
as
a
as
a
programmer.
D
The
clarity
and
resource
management
is
absolutely
a
huge
one.
In
building
a
big
system,
the
things
that
have
made
systems
like
spark
Hadoop
Nyad,
even
when
we
do
them
slow,
was
the
resource
management.
The
fact
that,
for
example,
have
a
garbage
collector
this
constant
looking
at
your
tens
or
hundreds
of
gigabytes
of
data,
saying
you
know
still
valid,
who
knows
I
have
no
idea.
What's
going
on,
rust
makes
this
very
pleasant
and
clear,
and
it's
just
delightful.
So
you
know
you
collect
resources
at
very
obvious
moments.
D
You
can
reuse,
reuses
resources,
also
at
very
obvious
moments,
and
you
end
up
with
again
fewer
bugs
than
I
had
before.
There's
a
lot
of
really
nice
compiled
time.
It's
a
safety,
both
memory
safety,
but
also
control
safety.
In
a
few
ways,
you
know,
programming
patterns
that
you
can
enforce
and
cause
people
to
write
more
likely
to
be
correct
programs.
D
Fewer
errors,
predictable
performance
was
a
really
great
thing.
One
of
the
reasons
again,
that's
things
like
Nyad
and
other
systems
went
slow
was
because
you
had
this
crazy
garbage
collector.
That
would
wake
up
and
say:
hey
guys,
I'm
gonna,
take
take
hundred
milliseconds
now
hope,
that's
cool,
and
you
know
you
have
a
hundred
computers,
each
of
them
doing
this.
Randomly
it's
just
a
mess.
D
If
you
arresting
is
great,
I
have
something
I
don't
know
if
this
is
something
you
like
to
be
proud
of,
or
not,
but
I've
written,
this
I
think
sort
of
cool
distributed
system
around
overly
goes
real
fast.
All
these
thing,
I,
don't
even
know
how
gdb
works
yet
or
lldp
works
I've
not
actually
dropped
into
a
debugger,
yet
there
are
bugs
for
sure.
Like
don't
get
me
wrong,
I've
had
bugs,
but
they've
not
been
the
class
of
bug
that
are.
Why
does
memory
look
like
this
I
have
no
idea
what
actually
happened.
D
They're
mostly
constrained
to
these
logical
bugs
at
the
higher
level.
You
can
look
at
your
code
and
say:
oh
yeah,
okay,
I,
see
why
that
happens.
So
Russell's
really
ruled
out
for
me
at
least
this
large
class
of
bugs
that
were
the
real
head
scratchers
that
you're
like
I.
This
wasn't
intended
in
any
way
what
happened?
I?
Don't
worry
about
that
class
of
bugs
nearly
as
much
anymore.
D
We
get
to
run
your
code
and
not
that
much
more
than
your
code
and,
if
you're
confused
about
what's
going
on
you're
running,
oh,
it's
you
slow,
I'm
annoyed,
you
can
you
can
profile
it
and
so
yeah.
What's
your
code,
you
wrote
it.
You
want
change,
it
feel
free.
You
know
if
you're
computing
too
many
hash
codes,
it's
your
fault!
You
can
fix
it,
there's
not
a
lot
of
crazy
infrastructure
in
there.
That's
this
boggling!
D
You
know
that
it's
hidden
what
you've
written
or
that
we've
we've
read
you
written
for
you
Russ,
does
a
great
job
of
taking
hunks
of
code
that
you
write
and
actually
turning
them
into
what
runs,
as
opposed
to
rewriting
what
you've
written
into
some
other
crazy
language
that
you
don't
really
understand.
The
mapping
is
really
distant
and
you
can't
figure
out
in
sort
of
action
at
a
distance
between
the
symbols
you
typed
and
the
code
that
you
actually
have
executing
your
computer.
D
So
so
there's
some
transition.
Now
they
talk
to
the
next
next
section,
so
I'm
giving
you
some
examples.
We
can
write
some
crazy,
crazy
data
flow
graphs.
I've
said
you
can
pretty
much,
but
whatever
code
you
want
in
there,
which
which
is
great
you.
What
should
we
actually
put
in
there
like?
What's
with
some
examples,
I'm
going
to
show
you
know
a
slightly
higher
level
language.
That
is
one
of
the
things
that
you
could
write.
I'm
not
saying
it's
the
one
you
should
write
and
you
shouldn't
write
anything
else.
D
Of
course
you
should
write
whatever
whatever
makes
you
happy,
but
this
is
an
example
of
something
that's
been
written
for
you
and
and
might
also
make
you
happy
or
sleepy
yeah
we'll
see.
So
this
is
the
differential
data
flow
side
of
things
which
I've
had
like
five
different
subtitles
and
I
couldn't
really
figure
out
the
best
one.
But
I
went
with
yeah
real
time,
Big
Data
programming
it
and
it's
just
basically
every
buzz
word
I
could
find
and
we're
gonna
put
deep
Nets
in
there
somewhere
and
it's
anyhow.
D
It's
not
it's
not
exactly
wrong
as
you'll
see,
but,
but
still
it's
yes
buzzword,
bingo
for
sure.
So
our
experience
has
been
that
people
are
pretty
good
at
programming
with
collections
or
aesthetic,
aesthetic
things
they're
good
at
writing
a
function.
That's
like!
Oh!
If
you
gave
me
a
thing,
I'll
tell
you
how
to
act
on
this
thing
and
do
a
few
other
things
to
it
and
give
you
an
answer.
D
They're,
a
little
less
good
at
programming
against
things
like
streams
or
dynamic
objects
that
change,
while
you
aren't
looking
at
them,
or
what
have
you
just
in
terms
of
getting
correct
programs
out
of
people
it'd
be
good
to
restrict
our
attention.
Well,
we
could
hopefully
restrict
our
attention
to
a
slightly
more
aesthetic
setting,
or
at
least
one
were
we
control
that
the
dynamics
that
go
on
so
the
idea
that
we're
gonna
take
care,
and
this.
D
D
These
streams
that
we
just
talked
about
where
we
have
a
notion
of
time,
you
know
the
the
records
in
there
have
have
some
notion
of
time
and
the
records
instead
of
being
data,
are
going
to
be
little
pairs
of
data
and
also
an
integer
describing
the
change
that
just
happened,
like
a
record
has
been
added
record,
has
been
removes
like
a
plus
1
or
minus
1,
no
zeros.
So
we
don't
want
to
see
any
zeros
there,
because
that's
not
a
change.
So
we'll
think
of
we'll.
D
Think
of
these
data
sets
instead
of
yeah,
instead
of
being
big
monolithic
collections,
that
we
just
see
the
same
collection
multiple
times
we're
going
to
show
you
the
changes.
So
I
went
to
show
you
anything
about
this.
We're
gonna
show
my
operators
that
not
not
your
program,
the
changes
that
have
happened
to
the
collection
and
we're
going
to
take
care
of
correcting
your
computation
for
you,
the
the
output
are
going
to
be
similar
they're,
going
to
be
streams,
also
pairs
of
data
and
changes,
but
describing
the
behavior
the
changes
in
the
output
collection.
D
That's
going
on
telling
you
yeah
okay!
Well,
just
a
moment
ago,
your
collection
was
whatever
it
was,
and
it's
just
changed
by
these
five
records
or
these
10
records,
or
what
have
you,
which
is
cool
streams,
all
the
way
down?
That's
really
nice.
It
also
means
that
they're,
nice
and
composable.
So
if
your,
if
your
friend
goes
and
writes
something
from
sets
of,
are
two
sets
of
Z,
you
can
just
stick
them
together
and
it's
all
nice
and
composable.
Yes,
there's
a
question
and
you
have
the.
D
Yeah,
yes,
yeah,
sorry
you'll
see
that
you'll
see
the
languages
we
get
to
it.
It's
definitely
a
very
functional
language
and
that's
going
to
be
an
important
part
of
us
correctly
figuring
out
how
to
update
your
computation.
So
we're
gonna
see
a
lot
of
things
like
maps
and
filters
and
joins
and
group
bias
and
stuff
like
that,
yeah
no
side
effects
I
mean
you're,
welcome
to
put
codon
that
causes
side
effects,
but
you
should
not
be
surprised
when
something
we
I
mean
something
will
happen
all
right.
D
Let's,
so,
let's
look
at
an
example,
so
this
is
a
bit
of
code.
This
is
going
to
be
a
graph
processing
example.
So
we
have
two
collections
initially
I've
called
them
nodes
and
edges.
So
a
set
of
nodes
in
your
graph
in
a
set
of
edges
which
are
going
to
be
pairs
of
pairs
of
nodes
and
the
nodes
are
just
you
know
some
identifiers
and
then
true,
we're
going
to
do
a
reach
ability
example,
and
the
question
is:
can
we
actually
reach
a
given
node
or
not
since
we'll
start
with
some
people
who
are?
D
Yes,
you
can
reach
me
bunch
of
edges
that
are
described
who's
next,
to
whom
and
here's
a
little
bit
of
code,
I'm
using
things
like
join
and
concatenate,
which
I've
just
borrowed
from
relational
algebra.
But
if
I
take
the
set
of
nodes
and
I
joined
them
with
edges,
meaning
I
I
go
and
I
grab
all
of
those
edges
that
match
something
in
in
the
node
step.
D
But
if
you
write
this
you
can
you
can
then
to
your
heart's
content,
go
and
change
the
contents
of
nodes
and
edges
and,
as
you
add
or
remove,
things
from
from
edges
will
receive
updates
to
those
nodes
that
were
reachable
after
after
one
hop
from
the
original
set
of
notes.
You
know,
maybe
that's
cool
again.
You
can
do
this
sort
of
thing
today,
with
incremental
view
maintenance
in
a
sequel
system.
What
you
can't
do
is
that's
cooler
than
this,
which
is
to
say,
maybe
maybe
we
do
that
a
few
times
actually
maybe
I'm.
D
Sorry,
it
skipped
you
stupid.
We
will
do
that
a
few
times.
Maybe
we'll
take
that
little
bit
of
logic
that
steps
through
and
finds
the
immediate
neighborhood
and
will
iterate
it
and
we're
still
thinking
about
collection,
collection,
Ranger
programming,
so
we
wrote
a
little
body
like
an
inner
loop
body
that
says
starting
from
that
set
of
notes.
Oh,
let's
repeatedly
take
whatever
we
can
reach
so
far,
hop
it
out
one
step
and
you
know:
do
it
a
lot.
D
You
know
this
case
we're
due
to
a
fixed
point,
basically
keep
running
until
it
stops
changing,
essentially
running
an
infinite
number
of
tower.
You
know,
use
64
m
x,
value,
nico
yeah.
I
I'm
happy
to
repeat
it
too.
If
it's
hard
to
get
the
mic
over
there
there's
a
weird
box
that
you
talking
to
oh
geez,.
C
D
The
observation
is
that,
if
you,
if
you
would,
if
I'd
put,
can
can't
reach
in
there,
you'd
be
much
more
comfortable
right,
because
rachel
is
what
I
could
reach
just
before
and
I.
That's
obviously
the
thing
I
want
at
it.
I
actually
want
to
add
nodes
in,
and
if
you,
if
you
write
the
math
out
you,
you
realize
that
it
all
actually
works
out
correctly,
because
the
set
reach
is
not
only
the
current
frontier,
but
it's
all
of
the
frontiers,
the
sum
of
all
of
the
frontiers
of
nodes
that
you've
ever
reached.
D
So
every
time
you
take
one
step
forward
from
that
you
get
the
the
gen
one
gen
two
gen
3
gen.
Whatever
things
you
just
need
to
add
the
gens
ero
notes
back
in
it's
a
very
good
point.
I
shouldn't
have
cause
troubles
by
putting
nodes
in
there.
This
is
actually
a
very
helpful
performance,
optimization
to
do
just
to
put
notes
in
there.
Cuz
notes
changes
less
than
reach
changes,
but
but
then
people
ask
questions
when
they
yeah
so
yeah,
so
both
would
work.
Both
would
be
correct.
D
This
one
goes
just
a
little
bit
faster
and
I
forgot
to
change
it
to
so
that's
the
other,
the
other
ways,
but
okay,
so,
but
this
actually
works.
This
is
this
is
a
cool
thing
that
and
again
the
thing
that
differential
dataflow
does
that
as
far
as
I'm
aware,
no
other
system
actually
does
today,
which
is
it
correctly,
updates
the
results
of
this
iterative
computation,
even
in
the
presence
of
removal.
D
So
if
you
go
and
remove
an
edge,
even
a
very
important
edge,
maybe
that
made
half
the
graph
reachable
it
will
correctly
update
the
computation.
Tell
you
about
the
changes
that
need
to
happen
to
your
reachable
set
might
take
a
little
while,
if
you
remove
a
very
important
edge,
but
but
you
get
the
correct
answer
at
least
so,.
D
I
D
Let's,
oh
I'm,
sorry
yeah,
so
this
is.
This
is
good.
This
is
the
part
where
I
tell
you
a
few
of
the
secrets
and
hopefully
just
overwhelm
you
with
mathematical
technology
and
no
1s
anymore,
silly
questions,
so
the
secret
that's
going
on
here,
the
secret
sauce
inside
this
loop,
the
timestamps,
are
a
bit
more
exotic
than
they
were
before.
There's
not
just
a
time
for
the
timestamp
like
a
round
of
input,
there's
a
pair
of
time
and
unsigned
integer
indicating
the
round
of
computation,
which
is
very
helpful.
D
Where
we'll
talk
about
the
difference
between
successive
elements
in
this
sequence,
so
you
know
our
variable
changes
from
one
thing
to
another
thing
to
another
in
a
collection.
This
is
pretty
easy.
We
just
take
the
difference
between
these
two
collections
and
that's
the
diff
at
a
particular
time.
This
gets
more
interesting,
a
bit
more
mathematically
vexing
when
we
have
something:
that's
not
a
total
order,
anymore.
Some
of
the
sequence.
We
have
a
partial
order.
D
In
this
case
we
have
pairs
of
integers
and
it
makes
a
little
bit
more
like
a
grid
in
terms
of
where
things
are,
and
you
might
ask
ok
well
what
does
that
mean
that
you
know
which
of
the
diff
at
five
comma
3
B?
What
do
we
take
the
dip
with
respect
to
and
the
correct
answer
is
not
5,
comma
2,
it's
not
4,
comma
3!
It's
something!
That's
a
bit
more
bit
more
interesting,
though.
If
you
sort
of
take
technology
to
differ
an
integration,
you
can
figure
out.
Okay.
D
Maybe
it's
got
to
be
that
thing
that
when
I
add
everything
up,
it
makes
it
integrate
to
the
right
thing
and
that's
that
as
a
technical
name,
which
is
mobius
inversion,
because
apparently
mathematicians
weren't,
like
weren't
cool
with
the
whole,
you
add
things
up
right,
stuff,
so
mobius
inversion
yeah,
so
other
people
have
done
reach
ability.
There's
a
paper
that's
showing
up
in
Sigma
2016.
This
has
not
yet
come
out
yet,
but
it
will
be
at
sigmod
in
San
Francisco
in
about
a
month,
and
you
can
go
there.
D
It's
only
a
hundred
dollars
to
go
which,
as
I'm
told,
is
like
a
fraction
of
your
rent
anyhow.
So
it's
in
Berlin.
This
is
two
months
rent.
So
just
think
about
that
when
you're
thinking
about
where
you
want
to
travel
next,
they
have
cake
too.
So
any
I'm.
Sorry
so
so
the
the
big
data
log
folks
have
gone
and
done
some
some
evaluation
on
graph
reach
ability
on
a
few
data
sets.
They
have
more
data
sets,
though
the
two
other
ones
who
use
our
bigger
one
of
them.
D
I
I
can't
use
because
the
license
is
weird
and
the
bigger
one
doesn't
run
on
my
laptop
and
I
feel
bad,
but
so
I
evaluated
the
first
two
that
they
have.
You
might
see
that
there's
a
few
systems
up
their
graphics
has
become
the
new
sort
of
punching
bag.
There's
three
systems
are
from
California,
graphics,
socialite
and
big
data.
Log
miriah
is
from
Seattle
University
of
Washington,
where
I
am
also
from,
and
you
might
notice
that
it's
currently
sort
of
cleaning
up
at
the
moment.
D
It's
going
to
change
in
a
few
slides,
but
another
system
that
sort
of
cleans
up
is
differential
data
for
running
on
my
laptop
on
either
one
or
two
cores,
and
you
can
see
that
that
also
does
pretty
well
and
it's
I.
Don't
really
have
a
good
explanation
for
this,
because
it's
shuffling
around
strictly
more
information
than
these
existing
systems
are
they're
all
doing
roughly
the
same
thing,
which
is
to
say
that
we
should
learn
about
the
set
of
newly
reachable
things.
D
Add
the
man
do
whatever
needs
to
be
done
after
that
fact,
differential
on
the
other
hand,
leaves
you
in
a
situation
we're
able
to
actually
do
incremental
updates.
So
I'm
going
to
show
you
next
are
some
numbers
related
to
me
going
and
taking
running
the
computations
and
then
randomly
ripping
out
notes
from
the
computation?
So
I'm
just
number
one.
No
doubt
at
a
time
do
it
a
thousand
times
report
the
report,
the
average
and
the
numbers
you
see
here
are
in
the
tens
of
microseconds
rain.
D
D
D
You
can
run
these
things,
so
the
main
the
main
problem
is
that
the
graph
that
I
would
show
you
that's
on
the
the
right
of
the
screen
is
the
Twitter
graph.
It
has
1.5
billion
edges
and
that's
a
little
bit
more
than
my
laptop
can
handle.
It
takes
about
30
gigs
of
memory.
When
you
run
differential
I
only
have
16
16,
it's
pretty
close
and
I
have
cheated
repeatedly
in
the
past
to
fit
that
comfortably
memory
by
doing
some
inappropriate,
optimizations
but
yeah.
D
So
I
wish
I
wish
I
had
more
exciting
to
show
you
or
just
a
cluster
sitting
around,
but
I
I
mean
I
I.
Have
you
know
amazon
ec2
account,
but
I'm
serve
both
to
just
spin
up
a
hundred
machines
in
order
to
make
a
point,
if
I'm
happy
to
set
people
up
with
the
code?
To
do
that,
though,
and
try
it
out
when
we
were
at
ETH,
we
did
a
few
of
these.
These
sorts
of
runs
I.
D
D
Good
questions,
how
does
update
scale
and
the
answer
I
think
here
is
actually
really
quite
good.
They
do.
The
updates
do
work
proportional
much
more
to
the
amount
of
changes
that
happen
in
the
amount
of
work
that
you
need
to
do
there.
I
think
what
it
sort
of
showing
is
the
defense
fact
that
no
edges
tripled
and
orkut
the
updates
didn't
change
wildly
in
terms
of
the
amount
time
that
they
take.
They
didn't
triple
translate
my
work
that
they
do
there,
the
you're
going
to
eat
a
bit
more,
a
a
bit
more.
D
If
you
have
a
much
larger
hunk
of
memory
that
you
have
to
poke
around
in
to
find
the
correct
answers,
but
in
my
experience
at
least
yeah,
the
amount
of
work
that
you
end
up
doing
is
largely
proportional
to
the
amount
of
change
that
actually
happens
depends
on
a
lot
of
things.
To
be
honest,
it
depends
on
how
dense
the
graph
is
in
this
case,
because
if
the
graph
is
really
dense,
ripping
out
a
node
really
doesn't
change.
Who
can
retune,
because
everyone
is
reachable
already
for
graph
is
just
very
tenuously.
D
Yeah,
it's
to
get
the
absolutely
proportional
to
the
difference
in
the
trace
of
the
previous
computation
on
the
current
computation
with
some
constants
yeah
I
have
some
pictures
to
show
and
then
I'll
try
to
sort
of
wind
down.
Maybe
I'll
tease
one
other
thing,
but
I
just
want
to
show
you
if
you
actually
run
these
things
and
look
at.
I
did
a
thousand
random.
You
know
ripping
outs
of
nodes
and
have
the
empirical
complementary
cumulative
density
function
here,
which
is
out
of
the
thousand
samples.
D
How
long
do
each
of
them
take,
and
you
can
see
that
a
good
fraction
than
forty
percent
basically
do
nothing?
They
don't
change.
The
output
does
not
change
at
all,
and
the
system
just
takes
about
three
microseconds
to
confirm
that
it
rips
the
no
doubt
realizes
this
wasn't
an
important
set
of
edges
for
my
computation
guess
we're
done
quite
quite
quickly.
This
is
choice
should
say
this
is
single
threaded
at
the
moments.
D
These
are
basically
the
best
possible
numbers
with
respect
to
low
latency
stuff,
not
the
best
possible
numbers
with
it
back
to
the
right-hand
side,
though.
Basically,
the
rest
of
them
do
something
non-trivial,
not
wildly
exotic.
You
know.
Typically,
this
is
changing
about
ten
knows
reach
ability
and
stuff
such
that
takes
tends
to
move
up
to
100
hundreds
of
microseconds,
and
then
out
of
these
thousand,
there
was
one
sample
that
took
more
than
a
millisecond.
I
took
20
milliseconds
as
it
turns
out,
and
this.
D
D
D
If
you
just
a
few
more
pictures,
this
is
too
takes
on
the
same
computation,
we're
on
the
left-hand
side.
We
have
single
update
at
a
time
and
on
the
right-hand
side
we
have
thousands
of
updates.
At
a
time
thousand
sorry
exactly
updates
at
a
time
with
both
one
worker
and
two
workers
to
show
you
a
bit
on
how
the
parallelism
works
here,
and
it's
not
too
surprising,
like
in
the
in
the
one
update
case,
where
there's
not
a
lot
of
work
to
do.
D
D
Basically,
a
factor
to
improvement
in
the
heavy
volume
case
of
the
throughput
gets
improved
by
factor
of
two,
as
you
would
sort
of
hope,
and
even
that
that
annoying
20
milliseconds
thing
gets
cranked
down
to
10
milliseconds,
because
in
the
two
workers,
happily
collaborate
and
you
see
the
same
sort
of
trend,
as
you
scale,
broader
to
more
and
more
machines.
As
you
add
more
workers,
the
throughput
absolutely
improves
the
min.
D
So,
let's
see
I'll
lightly
skip
through
a
few
more
things.
You
can
change
the
code
a
little
bit
if
you're
keen,
instead
of
circulating
true
everywhere.
If
each
of
these
nodes
send
around
a
unique
identifier
and
then
instead
of
taking
a
distinct,
you
do
an
argument,
so
you
say:
wow
lots
of
people
sent
identifier
SAT
me.
Let
me
keep
the
smallest
one.
I've
heard
of
you
get
a
connected
components
algorithm,
basically,
within
each
connected
components,
whatever
the
smallest
label
is
in.
D
That
component
eventually
spreads
over
the
entire
Terra
sub
graph
they're
in
a
different
connected
component.
Okay,
there
are
different
identifiers
everyone.
There
gets
it's
a
different
thing.
This
is
it's
a
way
of
competing
connected
component,
so
it's
pretty
similar
in
structure
and
fortunately
this
these
papers
have
evaluated
them.
You
might
notice
that
mariya
here
is
no
nearly
as
awesome
as
big
data
log,
but
that's
cool,
because
I
have
maria's
back
in
this
case,
and
you
know
again
we're
seeing
relatively
good
numbers
on
fairly
simple
deployment.
G
D
D
Yeah,
so
the
question
is:
can
we
handle
things
larger
than
the
memory
of
individual
machines,
and
the
answer
is
absolutely
yes,
because
rust
does
not
shoot
itself
in
the
head
when
you,
when
you
pass
physical
memory,
there
are
some
points
at
which
allocations
do
start
to
fail.
So
it's
certainly
if
you're
on
linux
and
you
have
a
new
killer.
D
Your
process
is
gonna,
get
shut
down
at
some
point
because
you
told
your
system
to
do
that.
But
you
absolutely
you
spell
to
disk
just
fine.
The
internal
data
structures
are
largely
written
to
support
sequential
access
to
main
memory,
because
that's
fast
has
a
nice
property
when
it
suppose
to
disk
it's
all
so
fast
off
disk.
So
this
a
bunch
of
internal
design,
has
has
really
worked
to
make
this
work
well
off
of
external
memory
when
possible,
and
whether
it's
pills
do
disc
or
kills
itself
typically
depends
on
how
you've
rigged
your.
D
Ic,
so
the
question
is
yeah
if
the
data
set
doesn't
fit
in
the
memory
of
one
machine.
This
is
typically
delightful.
These
models,
the
data
pro
models,
will
shard
all
of
the
data
across
all
the
machines,
so
typically,
for
example,
you'll
take
your
graph,
which
has
lots
of
edges
and
you'll
partition,
the
edges,
maybe
by
a
hash
of
the
source,
the
source
vertex.
D
So
the
16
gigs
of
data
get
smeared
all
across
your
cluster,
hopefully
in
a
nice
balanced
way,
and
the
aggregate
memory
of
the
cluster
is
the
more
important
quantity
to
worry
about
dust
does
my
days
to
fit
in
the
aggregate
memory
of
the
of
the
cluster.
So
typically,
the
answer
is
yes
thumbs
up.
Sometimes,
if
you
have
a
crazy
data
set
that
has
aught
of
skewing
it,
something
there
can
be
issues,
but
typically
yes,
great.
What
was
I
saying,
okay,
so
delightful
performance
numbers
I
would
I.
D
I
think
the
program
model
isn't
particularly
offensive.
There's
a
lot
of
a
lot
of
subjective
qualities
to
do
you,
like
writing
iterative
computations
this
way.
Maybe
yes,
maybe
no
other
people
I
gave
a
talk
at
Stanford
and
when
the
professor's
asked
you-
and
you
really
expect
people
to
write
stuff
this
way
and
I
said
yes
and
he
walked
out
of
the
room.
So
you
know
opinions
vary
apparently,
but
yeah,
that's
differential,
dataflow.
The
next
thing
was
abomination,
I'm,
not
really
gonna,
say
anything
else
did
I.
Think
I'm.
D
Is
it
okay?
I
don't.
I
don't
want
to
okay,
no
I'll
keep
going
for
sure.
Sorry,
I,
as
an
academic,
there's
like
a
55
minute
timer
in
your
head,
at
which
point
everyone
just
leaves
the
room,
but
no
let
me
work
this.
It's
actually
pretty
simple
and
it'll
end
on
a
nice
note
where
all
of
you
are
very
disappointed
with
me
and
and
some
liberties
that
I
seem
to
have
taken
and
you'll
be
very
suspicious
of
all
everything
you've
seen
so
far,
so
so
yeah
abomination.
D
This
is
about
serializing
datos
about
your
writing
programs
in
rust.
You
have
nice
little
rust
types
that
thought
were
delightful
on
you.
You
want
to
get
them
as
quickly
as
possible
to
a
big
blob
of
binary
and
then
back
out
again
so
the
name
abomination
you
might
notice
it.
First
of
all,
it's
spelled
incorrectly.
That's
not
my
fault.
This
the
name
is,
is
the
reaction
that
I
got
on
IRC
once
I
first
explained
what
I
was
planning
on
doing
and
someone
once
they
realized.
What
was
about
to
happen
announced
in
all
caps.
This.
D
So
you
know
with
the
moving
sighs
go
through
with
a
little
bit
of
skepticism.
That's
healthy,
I'll
point
out
a
few
places
where
I
know
that
there
are
issues
and-
and
we
can
have
a
conversation
about-
are
these
really
issues
or
are
these
just
like
people
worrying
too
much?
And
the
answer
is
probably
like?
No
other
real
issues,
but.
D
D
So
we're
not
do
this
for
every
type
out
there
we're
going
to
do
it
for
a
few
specific
ones
that
we
we
can
get
our
heads
around,
but
the
two
methods
are
going
to
first
off
take
some
type
data
reference
to
some
type
data
and
a
place
to
put
some
bites.
Yeah
mutable
vector
in
this
case,
and
the
idea
is
going
to
be
that
somehow
that
type
data
ends
up
in
the
bites
on
the
flip
side
and
we're
gonna
do
something
which
is
a
bit
more
suspicious.
E
D
D
D
You
might
just
use
like
mem
copy
right
right,
that's
a
pretty
might
say,
oh
cool,
you
know
you
got
an
array
of
you
60
for
us
when
I
just
copy
the
bites,
that's
awesome,
and
on
the
flip
side
you
might
say
if
you
gave
me
a
whole
bunch
of
bites
now
I'm
supposed
to
interpret
them
as
an
array
of
you,
64's,
that's
cool,
it's
just
a
cast
I'll!
Do
that!
That's
not
a
problem,
doesn't
exactly
work
with
rust,
but
we're
gonna
make
it
work
because
because
we
like
we
like
performance.
D
So
let's
imagine
I
have
some
weird
type
like
a
vacuum:
pairs
of
I,
32s
and
strings
so
on.
In
memory
they
they
look
like
approximately,
so
a
veck
is
a
triple
of
a
pointer,
a
length
and
a
capacity
which
is
telling
us.
You
know.
Where
does
the
thing
start?
How
much
of
it
is
there?
How
much
more
do
we
actually
to
actually
own
when
we
go?
Look
at
that
pointer,
it's
a
whole
bunch
of
I,
32
string
bears,
which
are
you
know
somewhere
in
there
and
I
32.
D
Some
some
padding,
bytes
and
I
got
a
triple
of
a
pointer
length
capacity
as
many
times
as
we
have
elements
in
the
in
the
vector
and
then
each
of
those
pointers
point
out
some
string
stuff.
That
is
pretty
exciting
text
data
that
we've
that
we
put
in
so
mm
copy,
isn't
exactly
going
to
work
here,
but
because
we
have
the
wonders
of
types
and
we
know
that
the
first
thing
is
the
fact-
and
we
know
that
Beck's
their
first
field
is
a
pointer
and
it
points
to
stuff
that
we
could
also
mem
copy.
D
We
can
essentially
write
a
heap
crawler,
which
is
going
to
walk
through
and
every
time
we
get
to
a
type
that
we
know
has
some
own
data.
On
the
other
end,
we
can
go
and
follow
it
and
copy
it
as
well,
and
for
each
of
the
bits
of
type
data
that
we
then
find
on
the
other
end.
We
can
then
copy
anything
we
need
to
there.
C
D
Not
encode
them
as
an
offset
into
the
bytes
array,
as
you
can
probably
do
that
where
you
do
something
very
similar
in
just
a
moment,
I
mean
that's
totally
same
thing
to
do
the
I
guess.
The
point
is
here
that
information
already
exists
in
the
it
might
actually
just
do
that,
and
that
makes
everything
but
simpler.
What
we're
going
to
do
is
basically
do
that.
I
mean
we're
going
to
reconstruct
we're
going
to.
D
We
know
where
the
data
starts,
for
each
of
these
pointers
right
we're
not
going
to
do
relative
offsets
because
I,
don't
think
rust
is
happy
with
relative
offset
pointers.
They
need
to
be
absolute
location.
So
whenever
we
decide
cool,
let's,
let's
deserialize
this
thing,
we're
actually
say
well:
where
does
bytes
start?
Let's
add
the
right
amount,
the
appropriate
offset
to
do
that,
because
that's
that's
where
the
next
bit
of
data
is
well
we'll
do
that
for
each
of
the
pointers,
and
then
this
is
the
better.
D
Then
we'll
just
lie
and
say:
let's
pretend
that's
a
that's
a
veck
of
guy
32
and
string
anyway
see
what
is
ridiculous,
but
it's
basically
right
the
the
thing
that
we
point
out
actually
looks
like.
There
are
a
few
caveats
that
again
come
up,
but
it
basically
looks
like
you
know.
If
we
pretend
that
it
is
in
fact
a
reference
to
a
Bacchae
32
hearing
we
were
like
well,
we
expect
to
find
a
triple
there
as
a
triple
cool.
D
We
expect
the
first
thing
to
be
a
pointer
to
somewhere
in
memory
that,
if
we
follow
it
looks
like
a
whole
bunch
of
I.
32
string
bears
that's
right
and
pretty
much.
It's
all
good.
Actually,
with
a
few
caveats,
it's
very
important
that
we're
returning
only
a
reference
right
like
if
you
try
to
interact
with
any
of
these
objects
in
any
mutable
way,
you're
in
deep
doo-doo
right.
D
You
could
get
a
panic
at
that
point
because
that
that
particular
offset
wasn't
actually
allocated
by
joe
malik,
basically
you're
in
a
world
of
hurt,
if
you
do
anything
other
than
just
look
at
the
data
and
pretend
that
it
is
how
we
saw
you're,
also
in
a
world
of
hurt,
if
you
care
about
some
other
things,
if
you
care
about
alignment,
you
know
if
you're
on
arm
bad
luck.
You
know
x64
is
fine
with
with
alignment
scripts.
D
Sorry
absolutely
so
you
could
do
more
work
than
I
did
I
agree
and
they'll
be
in
just
a
moment.
A
few
awesome
places
to
go
to
contribute
and
stuff
like
that.
Oh
no,
it's
a
good
point,
they're,
a
bunch
of
things.
You
could
do
to
try
to
make
this
a
bit
more
robust
and
it's
a
good
question.
Should
those
be
things
that
that
are
done
and
if.
D
Should
make
this
a
bit
better
I
believe
you
and
yeah
it'd
be
cool
to
have
some
some
help.
There
are
some
other
things
which
I'm
not
really
sure
about
like
according
to
the
rust
documentation.
At
the
moment,
anything
you
do
with
padding.
Bytes
is
undefined
behavior,
it's
not
undefined
behavior
in
llvm
lvm
says
it's
just
an
undefined
value,
so
you
know
is
reading
and
copying
and
undefined
a
padding
bite,
undefined
behavior
I,
don't
know.
D
So
I
I
don't
think
it
should
be
a
problem
but
the
documentation
at
the
month.
Dan.
Are
you?
Do
you
have
a
question
back
there?
Oh
I'm,
sorry.
The
comment
was
just
standard
library.
Does
that
all
the
time
which
which
I
believe,
though,
if
someone
also
said
surprised,
Frank
I
saw
you
did
a
mem
copy
of
a
petting
bite
and
I've
killed
your
program.
The
documents
say:
that's
okay,
I
me,
I,
don't
really
have
a
lot
of
leg
to
stand
on
here.
D
It's
worked
fine
with
no
issues
for
the
past
year,
so
I'm
optimistic
that
it'll
work
for
another
week
or
two
until
someone's
I
gosh,
there's
no
way.
I'm
gonna
kill
it,
but
but
yeah.
So
let
me
put
that
out
there.
Okay
now,
all
of
you
are
like
oh
gee
is
where
the
rest
of
his
projects,
this
bed,
but
oh
shoot,
there's
a
question:
yes,.
D
J
D
D
Know
like
it
works
on
my
laptop,
it
works
on
the
cluster
computers
I've
used.
It
may
work
on
more
and
more
things
as
time
goes
on
and
people
build
more
and
more
robust
architectures.
But
if
you
go
down
to
like
a
raspberry
pi
or
something
like
that,
I,
maybe
it
blows
up.
There
have
no
idea
so
it'll
blow
up
on
something:
okay,
yeah,
good
boy.
Sorry
one
important
thing
here
is
that
I've
only
implemented
abomination
for
a
few
types
that
I
understand.
D
None
of
the
wide
types
have
I
implemented,
abomination
for
and
you're
welcome
to
type
that
in
if
you
want,
but
then
it's
on
you
so
I,
just
I
have
just
a
few
things
to
say
before
ending
this
is
just
some
performance
stuff
about
thought
abomination.
This
is
something
that
I
stole
from
Eric
where's
Eric.
This
is
behind
that.
D
So
Eric
does
a
lot
of
serialization
stuff,
and
this
is
some
code
that
I
stole
this
is
a
cloud
fly
benchmark
that
I
guess
there's
a
recently
ghost
serialization
benchmark
where
they
like
to
take
this
particular
crazy
data
structure
in
serialize
it
a
lot
you'll
read
it
back
and
do
whatever
a
few
different
ways.
Then
you
get
a
sense
for
a
quazy,
realistic
thing.
You
might
write
to
your
log.
How
does
that
work?
And
you
can
sort
of
see
where
this
log
thing
is?
D
It's
got
its
got
a
bunch
of
what
field
some
of
the
fields
are
in
ohms.
Some
of
those
are
integers.
Some
of
the
fields
are
other
crazy,
structs
that
have
things
like
strings
in
them
and
it
has
some
strings
of
its
own
and
and
lots
of
lots
of
gunk
going
on
in
there.
So
first
off
this
does
not
as
written
that
does
not
implement
abomination.
So
we
can't
use
it
it's
too
bad
right,
that's
probably
for
the
best.
But
if
you're
an
adventurous
person,
you
can
type
some
very
simple
things.
D
There
are
some
nice
macros
that
I
have
basically
strokes
are
a
lot
like
tuples
right.
You
know,
struct
has
a
bunch
of
things
in
it
and
as
long
as
each
of
the
things
implements
abomination
sort
of
tells
us,
how
do
we
chase
its
pointers?
We
can
synthesize
an
implementation,
I
put
unsafe
in
front
of
because
I
don't
know
the
right
way
to
force
a
macro
to
be
unsafe,
but
but
you
have
to
type
the
word
unsafe,
and
that
makes
it
true,
but
yeah
you
can
sort
of
see
this
is
this.
D
Is
us
saying
this
log
thing?
It's
got
a
few
of
its
fields
that
need
to
be
chased.
They
have
one
memory
behind
them,
so
when
you're
doing
that
heap
crawler
make
sure
to
go
through
those
fields.
Http
in
origin
have
some
own
things.
These
are
the
strings
that
you
need
to
do
you
need
to
follow
and
it
will
synthesize
an
implementation
that
goes
and
actually
chases
them.
D
So,
let's
go.
That's
all
you
type.
So,
let's
see
what
happens
if
you
turn
out
Eric
and
I
like
cool.
Let's,
let's
see
how
this
works!
Well,
you
can
write
a
few.
This
is
russ
delightful
benchmarking
infrastructure
that
is
very
simple.
To
use
you
just
pull
in
this
test
crate
and
this
venture-
and
you
write
a
little
bit
code
that
says:
what
should
I
do
over
and
over
again
a
lot
and
first
we're
going
to
say
what?
If
what
do
we
just
make
one
of
these
new
log
items?
D
So
we
going,
we
allocate
a
few
strings
and
we
stuff
them
into
these,
feel
there
about
ten
strings
and
I
reality.
We
stuff
them
into
these
fields
and
make
a
new
deal
e,
and
we
run
that
we
see
that
takes
about
two
hundred
sixty-seven
nanoseconds
to
do
to
go
and
do
these
ten
allocations
fill
them
up
with
with
data
etc.
This
number
is
going
to
sort
of
serve
as
a
frame
of
reference
for
our
the
other
numbers
are
seeing
good
or
bad
or
whatever.
Now
imagine
you're
Eric.
You
want
to
serialize
things.
D
D
That's
going
to
tell
us
a
bit
about
the
throughput
in
terms
of
bytes
and
then
we
repeatedly
just
clear
out
the
spike
buffer
fill
it
up
again
with
the
encoded
data
and
do
this
this
black
box
thing,
which
make
sure
that
the
optimizer
doesn't
optimize
out
the
fact
that
bites
is
important
too
to
keep
around
and
you
get
some
cool
numbers.
This
is
sort
of
10
gigabytes
a
second.
This
is
faster
than
any
network
that
I
think
any
of
you
have
operates
up.
It's
not
technically.
D
If
you
have
a
hundred
gigabit,
we
should
talk
yeah.
How
are
you
doing
now?
So
it's
quite
fast.
This
is
just
one
thread
on
the
serialization
side
of
things:
Adi
serialization
side
of
things.
Sorry
code
looks
roughly
the
same,
we're
going
to
encode
it
once
so
that
we
have
some
valid
valid
bytes
and
their
repeatedly
call
this
decode-
and
you
might
remember,
decode-
just-
goes
and
fix
this
and
pointers.
So
it
really
doesn't
do
very
much
work
at
all
which
gives
you
some
really
embarrassing.
Do
serialization
numbers
so
I
mean
it's
not
a
good
number.
D
It's
nonsense!
We
haven't
actually
even
touched
all
of
the
data
in
this
case,
but
if
you
don't
need
to
touch
all
the
data,
if
you
just
wanted
to
go
in
there,
look
at
one
of
the
strings
you
wouldn't
have
to
pay
for
the
cost
of
actually
walking
through
all
of
the
the
other
strings.
You
didn't
bother
with
there's
a
tweaked
version
of
this
that
actually
goes
and
DC
realises
the
end
tests
that
it's
identical
to
the
original,
and
so
it
does
the
derived
a
quality
test,
and
that
is
a
bit
more
like
five
gigabytes.
D
A
second.
Do
you
see
rotation,
so
it
still
so
pretty
good.
Even
if
you
have
to
walk
through
the
entire,
the
entire
thing
d,
serialization
doesn't
actually
allocate
an
object.
Just
gives
you
a
reference
back.
So
if
you
wanted
a
reference,
if
we
want
actually
owned
object
would
be
a
bit
more
expensive,
which
I
guess
is
the
point
of
okay.
This.
D
This
is
really
fast
ray
it's
faster
than
this,
so
it's
actually
faster
to
serialize
something
to
a
big
hunk
of
bytes
and
then
look
at
it
through
the
lens
of
deserialize
than
it
is
to
make
a
second
copy
of
it.
So
if
you
only
need
a
read-only
copy
of
data,
which
often
we
do
in
the
big
data
settings,
it's
actually
faster
just
to
blood
it
down
as
bits
and
hand
it
around
his
bits
and
pretend
that
it
was
actually
validated
because
we
don't
have
all
of
these
different
allocations.
D
We
have
to
screw
around
with
the
thing
that
we
give
up
is,
of
course,
the
mutability
right.
We
can't
take
one
of
these
strings
and
to
lower
it
or
to
upper
it,
or
you
know,
put
a
few
more
two
more
things
on
it,
but
we
do
get
some
pretty
sweet,
pretty
sweet
performance
when
we
need
it.
This
is
great
for
sending
data
around
between
bits
of
our
computational.
It's
also
really
nice
differential
data
flow,
for
example.
D
Its
internal
state
is
essentially
equivalent
to
an
append
only
log
of
the
data
it
has
received
so
far,
so
you
could
just
keep
this
all
memory.
If
you
want,
or
you
could
be
a
bit
more
clever
and
say,
sweet
I'm
going
to
serialize
it
actually
now
I'm
going
to
drop
it
to
disk
and
I'm
gonna
memory
map
it
back
in
and
you
know,
sort
of
nice
lightweight
fault
tolerance
there.
At
the
same
time,
it
all
just
works
out
pretty
nice,
because
things
are
really
fast
because
rust
is
great.
D
The
whole
pile
of
code
is
about
four
hundred
lines
of
code.
It's
not
even
very,
very
complicated.
You
can
look
at
it
and
be
like
disappointed
immediately
without
having
to
look
very
far
so
yeah
and
that's
basically
I'm
gonna.
Just
have
a
few
links
up
here,
and
these
are
the
project.
I've
talked
about
so
far.
Basically
the
pattern
is
you
go
to
github.com,
/,
franck
McSherry
and
just
start
clicking
on
things
and
the
project
pages
there,
with
descriptions
of
what's
going
on
documentation
and
stuff
like
that,
so
few.
D
Other
projects
that
may
be
interested
in
maybe
not
recycler
is
a
sort
of
cool
object,
pool
in
the
same
spirit
of
abomination.
It
sort
of
dives
a
little
deeper
into
the
structure
of
types
and
pools
even
nested
nested
resources,
some
culinary
serialization,
it's
a
cool
blog.
If
you
guys
like
reading
blogs,
that
say
naughty
things
like
all
of
these
comments
about
performance
numbers
and
stuff
like
that
show
up
there
and
raise
some
questions
about.
Are
we
really
doing
the
right
work
in
large-scale
computation
at
the
moment,
which?
D
Maybe
yes,
maybe
no,
usually
don't
post
a
blog
post
about?
Yes,
we're
doing
good
stuff,
because
because
I
have
a
black
heart
and
basically
it'd
be
great
I'd,
be
delighted
to
have
people
either.
You
know
like
try
these
things
out
if
they're
interested
in
trying
these
sorts
of
things
up,
because
they
have
a
burning
passion
to
do
big
data
stuff.
D
You
know,
if
you
have
thoughts
about
things,
are
better
or
just
you
know
comments
like.
Oh,
my
god,
this
is
disaster.
I
mean
you
should
have
done
this
way,
this
this
other
thing
or
even
I.
Just
don't
like
what
you
did
or
if
you
like,
clicking
on
star
things
you
can
go
to
each
of
these
links.
You
click
on
the
star
thing
and
I
will
finally
be
able
to
overtake
the
Naiad
project
from
from
Microsoft
Research
and
retire
live
in
peace,
but
anyhow,
that's
that
I'm
done
I.
E
D
Right
so
all
of
those
strings,
for
example,
we
don't
Traverse
any
of
the
string
content.
We
just
make
sure
the
pointer
points
at
the
beginning
of
the
string,
which
means
we
avoid
touching
a
large
amount
of
data
that
we
have
credited
ourself
with
having
processed,
which
is
partly
why
the
number
is
a
bit
silly.
But.
D
A
good
good
point,
so
what
actually
happens
in
the
code
for
like
a
vector,
for
example,
is
the
pointer
gets
clobbered?
It
gets
something
to
something
that
does
not
actually
reveal
the
memory
layout
of
your
computer.
When
you,
when
you
serialize
it,
the
capacity
gets
set
to
the
same
thing
as
the
length,
which
you
know
in
principle
you
rust
could
happily
say.
I
only
have
power
of
two
capacities.
That's
ridiculous!
Aah
chak
fault
it's
within
its
its
permit
to
do
that.
Yeah!
That's!
E
G
G
J
Curious
more
towards
the
beginning
of
your
talk
and
the
timely
stuff.
You
talked
a
little
bit
about
data
exchange
and
such
and
I'm
wondering
have
you
ever
thought
about
trying
to
map
this
to
a
GPU
in
particular
where
data
movement
is
a
little
bit
different
there,
and
you
would
have
two
more
explicit:
do
control
so.
D
It's
a
good
question:
I
I'm,
going
to
defer
you
to
other
academic
publications,
I
have
other
clubs
in
the
past
and
a
large,
and
nowadays
there
are
a
large
volume
of
people
who
are
trying
to
figure
out.
How
do
we
take
these
non-trivial
models
of
computation?
You
know
there's
somewhere
there
bit
more
exciting
than
just
doing
matrix
multiplication
over
and
over
again,
but
they're,
not
quite
general
purpose.
Computation,
one
of
the
nice
things
about
data
flow,
the
recent
data
flow
sort
of
exciting.
D
Is
you
rip
out
a
lot
of
the
control
flow
potential
and
that's
it
makes
it
a
lot
easier
to
distribute.
The
execution
also
has
the
potential
to
make
it
useful
and
something
like
a
GPU
where
a
control
flow
is
very
scary,
though
you
have
to
be
quite
a
bit
more
careful
when
you
say
well
I'll,
just
add
in
a
loop
or
I'll
add
in
this
probe
operator,
which
is
actually
the
control
flow
part
of
timely
data
flow,
and
it's
some
exotic
information
that
a
GPU
might
have
a
hard
time
producing.
D
The
answer
is
no,
I
have
not
not
tried
very
hard,
but
it's
the
sort
of
thing
that
I
think
will
be
the
subject
of
research
going
forward.
People
have
looked
at
taking
systems
in
particular
is
a
paper.
It
also
at
SOS,
be
2013
called
dandelion,
which
was
about
a
data
parallel
system
that
they
mapped
onto
a
GPU.
That
tried
to
look
like
a
fairly
general
MapReduce,
II
style
system
with
computers
pushed
onto
the
GPU
they
synthesized.
D
G
H
D
I
think
there's
a
few
there's
a
space
of
problems
you
might
ask
about,
and
let
me
try
to
be
as
candid
and
honest
as
I
so
differential.
For
example,
it
is
absolutely
disastrously
on
machine
learning,
style
workloads.
We
basically
take
a
bunch
of
day
and
you
stir
it
together
and
there
isn't
a
lot
of
structure
in
terms
of
what
influenced
what
everything
just
gets
sort
of
put
together
in
a
big
pile.
It
does
work
still
quite
well
on
a
lot
of
relational
OLAP
style.
D
D
We've
done
limited
experimentation
with
this
when
we've
done
experiments,
they've
ended
up
working
it
well,
but
in
part,
because
I
manually
type
in
the
query
plan.
So
we
don't
have
to
worry
about
me
a
bunch
of
levels
of
the
of
that
stack.
So
the
question
is,
you
know:
can
we
get
it
to
work
as
well
as
a
well-tuned
database?
The
answer
seems
to
be
almost
as
as
well,
but
in
the
case
of
differential
is
the
real.
The
special
thing
that
it
brings,
which
is
different,
is
the
introduction
of
iteration.
D
So
if
you're
looking
at
computations
that
involve
iteration,
you
pretty
quickly
get
to
things
like
graphs
relations
that
that
you
know
speak
of
other
relations
in
the
data
set.
So
it's
sort
of
we're
looking
at
those
examples,
because
you
want
to
show
if
what's
new
and
interesting
and
if
you
don't
have
that
problem,
it's
a
little
hard
for
me
to
say
absolutely.
Should
you
should
use
differential,
because
it's
super
important,
that's
better
supported
and
has
lots
of
documentation
and
and
might
not
be
in
rust,
but
but
grown-ups.
You
know
still
use.