►
Description
Ryan has worked as a software engineer for over 15 years, and although he's spent a lot of time with JavaScript and Python, he now defaults to Rust for all but the most ad-hoc of projects.
He first discovered the joys of Rust while working on cross-platform client SDKs at Mozilla, but these days he's putting the language to work on the server-side, building data process pipelines at harrison.ai.
A
All
right
nice
to
see
you,
even
if
only
virtually
everyone,
sorry
I
can't
be
there
in
person,
but
I'll
do
my
best
to
keep
an
eye
on
the
chat
so
feel
free
to
huckle
me
over
there.
My
name's
Ryan
I'm
on
the
software
engineering
team
at
Harrison
AI,
and
if
you
take
a
look
around
the
big
spiral
staircase
in
the
room
there
with
you,
you'll
see
that
our
mission
at
Harrison
is
to
improve
the
standard
of
healthcare
for
millions
of
lives
a
day.
A
A
Obviously
it'll
be
a
lightly
fictionalized
account,
but
I
hope
to
give
you
a
little
bit
of
a
A
vibe
and
a
sense
of
the
journey
that
we
went
through
trying
to
use
rust
when
dealing
with
petabytes
of
data
for
one
of
our
model
building
projects.
So
the
starting
conditions
are,
one
of
our
adventures
had
happened
to
come
by
a
few
petabytes
of
de-identified
Medical
Imaging
data
right.
A
So
in
order
to
put
that
data
to
use,
we
had
to
figure
out
what
was
inside.
Those
Tire
balls
basically
make
them
available
to
some
sort
of
big
cloud
database.
In
our
case,
we're
using
Amazon's
Athena
now
Athena
is
pretty
amazing
right.
It's
sort
of
a
big
cloud
database
designed
for
a
querying
data
at
rest
in
in
a
cloud
storage
provider
like
S3,
and
it
can
read
parquet
files.
It
can
read
Json
files.
It
can
read
things
in
various
compressed
formats.
A
Unfortunately,
can't
read
tarballs
full
of
de-identified
medical
image
data,
so
we
had
to
find
a
way
to
generate
some
sort
of
index
of
those
files
right
process.
The
tarbles
turn
them
into
some
data
that
Amazon
Athena
could
use
to.
Let
us
see
what's
in
there
and
decide
which
files
we
wanted
to
work
with
in
what
order?
Now
we
actually
had
a
few
sort
of
failed
attempts
at
this,
including
one
very
unfortunate
incident,
where
we
sent
too
much
data
to
Amazon
Kinesis
fire
hose
and
it
started.
Dropping
things.
A
Ask
me
about
it
when
I'm
there
in
person
I'm
over
beers
one
time,
but
at
the
end
of
the
day,
what
we
settled
on
was
to
try
and
do
basically
the
simplest
thing
that
could
possibly
work
at
scale
right.
So
we
wanted
to
take
each
one
of
those
millions
of
Tire
archives
process
it
using
a
scalable
compute
function
in
AWS
Lambda
and
turn
it
into
just
a
plain
Json
lines
file
describing
the
contents
of
that
archive
all
right.
A
If
we
got
to
Jason
lines
files
sitting
in
cloud
storage
in
Amazon
S3,
then
we
had
something
that
Amazon
Athena
would
read
read
and
from
there
we
could
use
all
of
the
big
data
processing
capabilities
of
Athena
to
sort
and
summarize
and
compress
that
data
into
some
smaller
indexes.
That
would
help
us
navigate
our
way
around
those
Tire
walls.
Now
this
is
not
a
big
data.
Talk
I'm
not
going
to
go
into
too
many
of
the
things
on
the
right
hand,
side
of
that
diagram.
A
But
let's
take
a
little
look
at
this,
a
kind
of
first
step
right.
The
indexing
piece
where
we
want
to
be
able
to
take
a
look
at
each
of
those
millions
of
Terribles
and
spit
out
some
data
about
what
it
contains.
Now,
that's
the
sort
of
thing
that
sort
of
for
a
lot
of
people
you
could
reach
for
just
about
your
favorite
programming
language,
spin
up
enough
copies
of
it
in
a
cloud
compute
environment
like
AWS
Lambda,
and
it
would
probably
do
the
job.
A
So
when
we
sat
down
to
look
at
this
right,
probably
the
go-to
language
for
it
for
our
team
would
have
been
Python
and,
in
fact,
I've
put
together
a
little
python
demo
here
of
the
core
logic
that
we
wanted
to
use
on
top
of
these
Terribles
right.
At
the
end
of
the
day,
what
we
wanted
to
be
able
to
do
was
take
the
path
to
a
tar
file
open
it
iterate
over
its
contents
right
spit
out
Json
lines
of
like
the
files
that
we
found
there
in
right.
A
A
It
was
at
the
heart
of
this
system,
and
if
we
did
this
in
Python
like
it
would
have
worked,
but
all
of
the
folks
on
the
team
at
the
time
like
we
were
pretty
Keen
to
give
rust
a
try
like
we
hoped
that
we
would
be
able
to
get
some
better
performance
out
of
it.
We
have
to
be
able
to
get
perhaps
some
better
robustness
out
of
it,
and
this
seemed
like
a
really
nice
opportunity
sort
of
self-contained
problem
space
in
which
to
explore
it.
A
You'd
have
a
little
struct
that
we
Define
with
certain
serialize
helper
just
to
help
spit
things
out,
but
at
the
end
of
the
day,
like
that
processing
Loop
in
Rust,
it
operates
at
pretty
much
the
same
level
of
abstraction
as
it
does
in
Python.
Right.
You
open
the
tire
ball.
You
open
your
output
file,
you
iterate
over
each
of
the
entries.
You
write
out
some
some
data
about
it
right
and
I.
A
A
One
thing
that
we
noticed
very
quickly
was
that
rust
was
pointing
out
to
us
a
lot
of
places
where
things
could
go
silently
wrong
in
other
versions
such
as
python
right
here,
rust
is
kindly
letting
us
know
that
the
file
names
in
a
tar
archive
might
not
be
utf-8
right,
and
we
need
to
deal
with
that
error
so
before
we
got
too
carried
away
with
trying
to
do
anything
further
with
that
right.
Let's
take
a
quick
look
at
this
right.
Is
this
actually
going
to
be
any
an
interesting
Expedition?
A
A
A
We
kind
of
get
substantially
better
performance
more
or
less
for
free,
which
is
pretty
cool.
This
is
not
a
talk
about.
You
know
minute.
Optimizations
of
of
rust
code,
so
the
next
step
was
to
go
from
there
to
actually
working
with
data
in
AWS
and
for
rust.
That
gets
a
little
bit
thorny.
So
this
is
where
we
started
to
get
a
little
bit
complicated,
because
rust
has
a
lot
of
high
quality.
He
crates
for
working
with
AWS
right,
there's
a
crate
for
reading
and
writing
files
in
S3.
A
There's
a
crate
for
connecting
your
rust
functions
to
Lambda.
The
thing
about
all
of
these
crates
is
that
they're,
all
async,
all
of
the
networking
stuff
in
Rust,
uses
async
functions,
and
so
we
also
had
to
go
and
find
new
async
versions
of
all
of
the
crates
that
we're
using
for
the
basic
demo
version,
because
everything
inside
of
our
AWS
flight
version
of
this
function
will
need
to
be
async.
Now
I'm
not
going
to
lie
to
you.
This
was
a
little
bit
of
Adventure
right.
Async.
A
Rust
is
pretty
hard
to
to
get
your
head
around
when
you
dig
in
for
the
first
time,
but
kind
of
working
through
some
of
the
issues
and
learning
a
little
bit
about
streams
and
Futures.
We
actually
managed
to
come
up
with
some
code
that
we're
pretty
happy
with.
So
you
know
in
in
order
to
read
a
terrible
from
S3
and
iterate
through
its
entries,
actually
like
the
rust
AWS
S3
SDK
has
a
pretty
nice
wrapper
where
you
can
get
an
object.
A
If
you
give
it
a
bucket
and
a
key,
and
you
can
turn
it
into
what
Russ
calls
an
async
read
trait
right,
which
basically
lets
you
read
the
bytes
out
of
that
file
in
a
streaming
manner.
You
can
pass
that
to
the
async
tire
crate
and
it
will
let
you
sort
of
do
a
very
low
memory.
Usage
efficient
streaming
read
through
the
entries
in
that
tar
file,
which
is
pretty
nice.
A
You
can
do
a
little
while
loop
like
this,
which
is
not
too
bad,
but
we
did
have
a
lot
of
fun,
trying
to
find
all
of
the
various
helper
methods
on
the
stream
traits.
So
here,
for
example,
I'm
iterating
through
the
contents
of
that
tire
ball,
one
at
a
time
using
this
tri-fold
method,
basically
capturing
all
of
the
the
rows
that
we're
trying
to
write
there
as
Json
into
a
VEC,
so
that
we
can
then
write
them
out
again
to
S3.
A
So
I'm
not
going
to
lie
like
we
spent
quite
a
bit
of
time
trying
to
find
out
the
details
like
how
to
work
with
streams
and
how
to
make
this
sort
of
kind
of
fit
in
our
heads.
But
once
we've
got
our
heads
around
it
and
once
we
came
to
understand
how
to
use
these
tri-stream
act
methods
and
the
various
helper
methods
on
a
stream,
you
kind
of
feel
like
a
wizard
right
like
you.
A
A
If
you
go
through
the
motions
of
compiling
it
into
a
Docker
image,
you
can
send
it
off
into
AWS
Lambda
and
run
that
as
a
function,
one
time
10
times
or
in
our
case
millions
of
times,
and
that's
thanks
to
kind
of
cargo
and
rust's
pretty
convenient
to
build
ecosystem.
This
was
actually
really
straightforward.
I'll
be
honest
with
you
like
much
much
simpler
than
I've
ever
had
a
time
of
sort
of
packaging
python
packages
in
the
past.
A
So
the
docs
say
anyway.
I
actually
put
that
to
the
test.
For
this
example,
code-
and
here
are
some
rough
performance
numbers
essentially
of
that
demo
code
running
over
some
some
example:
tar
archives,
like
the
naive
python
version,
you
know
14.5
seconds
to
process,
one
of
those
files
on
x86
and,
as
Amazon
suggested,
like
a
little
bit
faster
to
run
on
their
arm
64
processes.
A
Now,
that's
sort
of
a
a
very
Whirlwind
demo
of
the
kind
of
code
that
we're
trying
to
work
with
here
and
sort
of
what
you
can
expect.
If
you
try
to
dive
into
this
sort
of
thing
in
the
in
real
life
version
of
this,
it
was,
of
course,
a
little
more
complicated
right.
We
weren't
just
listing
the
files
in
the
archives,
we're
sort
of
generating
I
think
six
different
listings
of
different
kinds
of
files
that
you
might
find
in
those
Terribles
and
the
metadata
about
each.
A
A
At
the
end
of
the
day,
we
managed
to
sneak
some
Rust
in
here
and
when
we
ran
it
in
production,
when
we
finally
sort
of
shook
all
the
bugs
out
of
the
system
and
deployed
and
ran
it
at
scale,
it
was
honestly
a
little
bit
of
a
an
anti-climax
like
it
ran.
It
did
it
did
the
job
and
it
finished
I
think
in
in
less
than
a
day,
and
we
were
happy,
we
were
able
to
process
those
files
with
Athena.
We
were
able
to
see
what
was
in
them.
A
A
What
was
really
impressive
for
us
I
think
was
the
stable
memory
usage
that
you
get
out
of
using
something
like
rust,
rather
than
a
scripting
language
like
python,
because
it
means
that
you
can
run
these
functions
on
smaller
Lambda
instances
and
when
you're
doing
compute
in
an
environment
like
Lambda,
you
actually
pay
for
memory
usage
as
well
as
just
CPU
time.
I.
Think
one
of
our
operations
guys
actually
watching
these
rust,
lambdas
Run
For.
The
First
Time
commented
that
he
basically
never
seen
a
Lambda
run
that
steadily
rather
with
that
consistently
low
memory
usage.
A
So
that
was
pretty
amazing.
We're
really
overall
enjoyed
the
the
runtime
robustness
of
doing
this
work
in
Rust
right,
like
it
sort
of
forced
us
to
think
through
a
lot
of
the
error
cases
up
front,
and
it
really
was
the
case
that
when
we
came
to
run
this
stuff
in
production,
it
pretty
much
just
worked,
which
was
incredible.
On
the
other
hand,
some
things
that
we
found
were
pretty
challenging.
I
think
that
the
async
ecosystem
is
still
a
little
bit
fragmented
right.
A
We
sort
of
had
to
find
async
equivalents
of
some
of
the
crates
we
were
using.
We
had
a
little
bit
of
a
challenge
with
some
testing,
some
of
the
AWS
Services,
although
I
will
say
which
we've
got
some
good
mileage
out
of
us
of
a
product
called
local
stack
which
lets
you
do
sort
of
mocking
a
local
hosted,
Services
of
various
AWS
services
and
the
one
comment
I
have
on
things
that
were
challenging.
A
Actually,
when
you're
working
with
rust
or
you're,
working
with
streams
and
you're
kind
of
feeling
like
a
wizard,
and
you
can
use
all
of
the
the
cool
features
that
rust
offers
you.
There
are
a
lot
of
optimization
opportunities
in
there
that
turn
out
to
be
an
attractive
nuisance
right
and
one
of
the
challenges
we
had
was
actually
like
knowing
to
when
to
put
down
the
brush
and
kind
of
just
ship
what
we
had
and
let
it
cost
maybe
200
instead
of
100
and
get
on
with
our
lives.
A
But
overall,
like
I,
said
this
was
a
bit
of
an
experiment
in
seeing
how
rust
would
would
work
out
for
this
kind
of
use
case,
and
will
we
do
it
again?
Absolutely
right
and
it's
in
fact
something
that
we're
trying
to
make
a
core
competency
for
our
team
here
at
Harrison,
try
and
open
source
what
we
can.
A
If
that's
something
that's
interesting
to
you
and
hopefully
we'll
be
able
to
flush
out
even
more
things
on
this
slide
over
time,
so
yeah
at
the
end
of
the
day,
I
think
we're
we're
prepared
to
make
a
pretty
big
bet
on
Rust.
The
combination
of
sort
of
forcing
you
to
think
through
your
problem
up
front
a
little
bit
right.
This
notion
of,
if
it
compiles
it
works,
super
predictable
performance
in
production
makes
it
a
really
good
fit
for
working
with
this
sort
of
data
in
AWS
at
scale.
A
B
B
A
A
So
this,
in
fact,
was
a
team
of
three
people
I
think
at
least
one
of
them
is
in
the
room
there
this
evening,
I
believe
for
Tim,
if
you're
keeping
an
eye
out
for
him,
and
so
if
I
came
to
the
team
with
a
little
bit
of
experience
with
with
rust
I
think
we
were
sort
of
all
rust,
Russ
curious
at
least
but
they're.
Like
not
gonna
lie.
A
It
was
certainly
a
learning
curve
right,
I
I,
don't
think
the
cost,
saving
that
we
would
have
realized
this
time
around
would
have
would
have
paid
for
the
engineering
hours
of
learning
to
get
up
to
speed
with
rust,
but
we're
pretty
excited
about
how
those
skills
are
going
to
compound
over
time.
Right,
as
we
do
more
and
more
of
this
in
more
and
more
parts
of
our
data
processing
pipeline,
you
know
sort
of
those
Baseline
skills
and
patterns
that
were
started
to
develop,
I
think
they're
really
going
to
pay
off
over
time.