►
Description
Agenda ► https://rusty-days.org/agenda
Playlist with all talks ► https://www.youtube.com/playlist?list=PLf3u8NhoEikhTC5radGrmmqdkOK-xMDoZ
Follow ►
Facebook: https://rusty-days.org/facebook
Twitch: https://rusty-days.org/twitch
Twitter: https://rusty-days.org/twitter
This video ►
Zerostash is a deduplicated and encrypted file storage format that is optimized for speed.
The talk will walk through the problem domain, the design decisions that were influenced directly by the need for speed for tools like this, and how to reduce synchronization latency in for remote or p2p sync.
The listeners will learn about how backup software generally works, and where the design tradeoffs lie when designing a storage layer. In addition, they get some insights into generic Rust optimization tricks and gotchas:
* Reducing the number of copies
* Slicing a slice will reduce performance
* LTO
* inlining
* reducing code size on the hot path
* threads and pipelining
* trusting the operating system
* using vectorization for the costliest things
A
A
little
bit
about
I'm
tonight,
I'm
going
to
talk
about
encrypted
backups
and
and
how
you
how
you
actually
create
encrypted
backups
using
rust,
but
before
we
step
into
the
meat
I'm
just
going
to
introduce
myself,
I
work
in
security
software
as
a
day
job
and
I
quote:
roster
the
day
job
at
red
sift.
I
work
on
ingredient,
which
is
a
hardware,
a
cloud
monitoring
agent,
which
runs
rust
in
the
linux
kernel
as
ebpf
modules.
A
It's
been
very,
very
exciting
and
interesting.
How
rust
allowed
us
to
actually
lower
the
barrier
to
entry
to
something
as
technical
and
as
in
the
grain,
so
to
say,
as
the
newest
kernel
and
actually
the
libraries
that
that
we
were
able
to
build
using
rust
are
really
really
pleasant
to
use
as
opposed
to
dc
interfaces
and
and
we
can
provide
the
complete
tool
chain
so
we're
using
ingredient
for
monitoring
the
the
cloud
data
and
cloud
network
traffic.
But
today
I
will
talk
about
a
little
bit
different
aspect
of
cloud
protection.
A
These
are
not
generally
super
helpful
in
terms
of
protecting
your
privacy.
They
are
very
often
designed
to
be
deliberately
vague
and
we
surely
can
do
better,
and
some
of
that
better
is
encryption
at
rest.
If
you
go
to
a
service
provider's
website,
they
very
often
tell
that
they
are.
They
provide
military
grade
encryption
to
your
data,
and
that
is
going.
That
is
what
is
going
to
keep
it
secure.
A
A
The
idea
of
end-to-end
encryption
is
to
encrypt
your
data
on
the
client
and
make
sure
that
only
the
client
can
decrypt
that
data
and
the
server
is
completely
unaware
of.
What's
in
that
now,
if
you
think
about
your
holiday
pictures,
this
is
obviously
much
more
preferable
to
allowing
any
sort
of
systems
administrator
to
even
inadvertently.
A
A
But
I
think
there
was
a
lot
of
development
in
the
past
sort
of
ten
years
in
cryptography
that
that
demolished
this
dichotomy
of
of
performance
versus
security.
So
we
can
actually
approach
the
problem
in
a
way
where
we
are
both
fast
and
we
are
both
correct,
and
this
is
where
us
comes
into
the
picture.
It
really
helps
us
with
the
correct
part.
A
You
can,
when
you're
reading
a
file
from
a
server,
for
instance,
you're
accessing
your
google
drive
or
your
icloud
account
you're,
sending
a
request
to
the
service
provider
and
the
service
provider
will
do
a
lookup
on
their
site,
execute
some
sort
of
business
logic
and
then
return
to
you
with
your
data.
Now
the
latency
is
the
first
bit
of
response.
After
this
entire
cycle
has
been
done
from
the
time
that
you
finished
sending
your
request.
A
Obviously,
in
in
a
network
scenario,
this
is
probably
hundreds
of
milliseconds,
but
when
you're
reading
from
a
disk
or
an
ssd
that
is
going
to
be
orders
of
magnitude
different
and
lower,
even
between
spinning
rust,
as
in
conventional
hdds
or
ssds,
there
can
be
huge
differences
really
really
interesting.
One
is
syscalls.
A
A
A
A
It
is
deduplicated,
so
we
save
some
space,
it's
a
nice
to
have
it's
not.
It
wasn't
really
a
big
effort
to
implement
it
within
the
system,
but,
most
importantly,
we
can
actually
use
this
as
an
online
file
system
that
doesn't
necessarily
expose
all
of
the
data,
and
he
says
this
is
really
exciting,
because
it
also
means
that,
whenever
a
user
logs
into
a
cloud
service,
we
can
also
use
the
xero
format
as
a
key
value
store
where
we
use
their
personally
identifiable
information.
A
So,
in
order
to
achieve
sort
of
fast
programs,
the
first
step
is
to
minimize
latency
a
lot
of
backup
software
that
are
out
there.
They
are
deduplicating
your
data
by
generating
large
numbers
of
files,
which
is
ideal
from
a
lot
of
from
a
lot
of
aspects
of
the
of
the
design
space.
Most
importantly,
it
reduces
the
complexity
within
the
backup
software
itself.
A
It
is
large
enough
to
be
efficient,
four
megabyte
happens
to
also
kind
of
align
with
access
sizes
of
ssds
and
which
gives
us
the
really
nice
alignment
and
and
actually
a
speed
bump.
Theoretically,
but
also
it's
it's,
it's
a
very
versatile
file
size.
I
feel
for
different
use
cases.
You
can
increase
that
or
this
decrease
this,
but
the
key
thing
is
that
we
only
work
on
four
megabyte.
Uniform
objects
that
that
densely
pack,
our
data.
A
And
the
usual
way
of
interacting
with
this
object
is
is
actually
initializing
it
with
the
with
the
block
buffer
as
the
as
the
buffer-
and
I
talk
about
this
later-
why
step
two
choose
the
right
primitives
to
maximize
the
throughput,
we're
still
in
design
space?
We're
not
we're
just
kind
of
playing
around
with
the
theoretical
maximums
so
for
indexing.
A
A
The
the
benefits
of
this
is
being
really
really
fast
and
also
providing
us
with
with
what,
with
the
properties
that
we
would
normally
expect
from
modern
cryptographic
primitives
and
for
deduplication,
I'm
using
c
hash,
which
does
like
10
gigabytes
per
second
for
hashing.
It
is
absolutely
a
non-cryptographic
hash,
but
it
is.
It
is
generally
very,
relatively
hard
and
and
really
really
fast.
A
A
The
crossbeam
channel
offers
quite
a
big
speed
up.
The
crossbeam
utils
package
crate
provides
us
with
a
with
a
threading
implementation
that
is
generally
a
bit
more
versatile
than
whatever
is
found
in
the
standard
library
and
the
rv
rw
lock.
Implementation
in
the
standard
library
is,
I
don't
wanna
offend
anyone,
but
it
doesn't
have
a
great
reputation
in
terms
of
performance,
so
there
are
alternatives
to
like
drop
in
alternatives
to
it
that
exist.
A
A
And-
and
that
means
that
there
is
a
really
efficient
implementation
in
rust,
which
is
the
s.
This
is
the
cmd
implementation
as
a
bonus.
The
rust
argon
2
library
also
uses
this
as
part
of
its
argon
2
implementation,
so
we're
actually
reducing
the
compile
times
while
getting
getting
a
really
really
nice
black
to
speed.
A
Lz4
is
just
a
wrapper
for
the
c
library
I
kind
of
compared
it
to
with,
with
a
couple
of
other
crates
like
minis,
but
ld4
is
just
very
versatile
and
very
tunable
and
c
hash.
It's
it's
again,
a
no-brainer
great
api,
great
implementation
really
fast
and
we're
getting
this
all
of
these
speed
ups
without
actually
having
done
any
work,
we're
just
using
the
right
crates.
Obviously,
the
way
that
I
learned
that
these
are
fast
involved,
a
lot
of
profiler,
driven
development
and
and
kind
of
like
trying
out
everything.
A
A
There
was
a
there
was
a
question
on
twitch
from
frago
daleta.
I
hope
I
pronounced
that
right,
which
asked
whether
it
would
make
sense
to
have
the
hash
be
the
id
of
the
buffer.
I
think
in
terms
of
cryptographic
construction,
I
don't
really
want
to
go
into
the
all
sorts
of
all
the
security
details
that
that
were
chosen.
I've
documented
these
in
the
repository
under
on
in
a
document,
but
generally
there
are
a
couple
of
trade-offs
with
the
indexing
mechanisms.
A
A
Into
our
objects,
it
looks
like
this:
we
open
a
file,
we
unmap
the
file
into
memory
which,
which
basically
creates
a
memory
area
that
is
equivalent
in
size
to
the
file
itself.
So
we
can
just
access
that
access
that
file
as
if
it
was
a
buffer
in
rust,
and
then
we
go
through
this.
We
run
the
duplication
algorithm
and
then
we
check
if
we've
already
stored,
that
piece
of
information
in
our
index
and
if
we
haven't
then
store
it
and
then
just
do
some
other
bookkeeping
and
do
this
do
this
in
a
single
thread.
A
Now,
we've
kind
of
like
have
a
rough
idea
of
the
implementation,
it's
very
very
important
to
start
profiling
and
I
actually
started
profiling
very
early
on
in
order
to
to
figure
out
where
the
bottlenecks
are
perf.
It
works
greatly
on
linux.
You
get
really
really
detailed
output
and
software
like
k,
cash
grind
can
actually
load
per
files
in
a
graphical
environment,
but
perf
on
the
command
line
is
also
very
powerful.
A
Now
that
we
have
an
idea
about
how
much
everything
costs
in
real
life,
we
have
an
implementation.
We
we
know
how
to
profile
it.
We
have
pretty
good
primitives,
we've
optimized
our
data
structures,
so
that
that
we
reduce
the
the
excess
latency.
We
can
start
actually
tweaking
things,
and
this
these
things,
some
of
some
of
the
things
that
I'm
mentioning
actually
like
make
a
lot
of
difference.
A
First
and
foremost,
one
of
the
things
that
I
I
was
trying
to
optimize
for
is
reducing
the
number
of
copies.
This
is.
This
is
super
important
four
reasons:
I'm
gonna
go
into
later.
Slice
access
generates
a
boundary
check.
These
boundary
checks
can
be
very
costly
link
time
optimization
at
the
best
case
you
can,
you
can
get
around
20
30
percent
worst
case
it
probably
equals
out.
A
There's
a
new
kid
on
the
block
called
I
o:
u
ring,
which
is
which
offers
really
exciting
and
interesting
modes
of
async
access
that
that
actually
get
really
really
good
real
life
speed.
Ups,
however,
at
the
moment,
is
linux
only
so,
and
I
kind
of
wanted
this
to
be
a
multi-platform
thing
and
developing
on
mac
os.
I
can't
really
rely
on
platform
dependent
stuff.
A
A
A
And
you
can
design
your
entire
data
flow
around
this
when
you
have
a
file,
you
end
up
that
file
and
then
you
just
start
working
on
slices
and
feed
those
slices
into
the
crypto
algorithm,
and
you
don't
and
compression
and
whatever
whatnot,
and
you
don't
really
need
to
think
about
copying
you're,
just
borrowing.
That's
from
from
that
same
memory,
structure
that
you're
building
that
that
you're
going
to
go
through
anyway.
A
A
So
minimizing
this,
these
open
operations
can
can
can
be
a
huge
win.
I
remember
testing
at
some
point
with
a
kernel
module
that
increased
the
time
to
open
the
file
to
around
700
millisecond,
and
at
that
point
that
is
just
ridiculously
slow.
Obviously,
here
it's
it's
much
better
without
without
kernel
modules,
but.
A
A
And
if
you
just
pass
around
slices
and
are
quite
pragmatic
and
and
principled
around
around
how
you
manage
this
kind
of
data,
then
rusts
one
of
rust's
greatest
power
is
that
you
can
actually
reason
about
what
happens
at
a
low
level
in
the
memory.
So
you
can
you
get
this
very,
very
fine
grain
control
and
you
can
precisely
pinpoint
when
that
copay
is
going
to
happen.
A
A
I
think
it
really
is
a
lot
of
experimentation
to
figure
out
what
it
is.
What
is
the
sort
of
code
that
is
worth
in
lining
and
the
good
thing
about
the
inline
macro
is
that
it
still
allows
the
the
compiler
to
not
do
that,
for
you
there's
an
inline,
always
macro,
which
is
a
lot
more
dangerous,
because
that
will
always
in
line
that
piece
of.
A
A
On
the
picture,
you
can
see
two
different
threads.
One
of
them
is
an
encryption
and
the
other
one
is
a
decryption
thread,
the
the
top
one.
You
can
see
that
cpi
utilization
just
climbs
up,
and
this
is
because
of
all
the
page
faults-
and
this
is
what
the
your
I
o
latency
looks
like
if
you're,
if
you're
looking
at
it.
A
A
A
So
there's
a
lot
of
convenience
that
comes
with
it.
On
the
decryption
side,
the
the
least
recently
used
cash
actually
gave
me
a
performance
hit
around
a
performance
gain
of
around
20
percent
when
I
calibrated
it
correctly
and
it
flat
out,
crashed
the
program
when
I
didn't
so
there's
there's
that
as
well.
A
A
The
link
time
optimization
and
the
and
the
optimization
level
three
settings
best
case
around
30
performance
gain
on
on
top
of
everything
else,
which
is
which
is
definitely
significant.
It
reduces
it,
reduces
compile
time
quite
significantly,
but
if
you're
only
doing
it
in
release
mode,
then
that
that
sort
of
compilation
penalty
is
is
definitely
worth
it.
A
A
A
The
primitives
are
there
the
multiple,
the
multi-threaded
kind
of
architecture?
Is
there
and
I
think,
right
now,
the
async
ecosystem
in
rust
is
in
a
place
where
I
would
be
very,
very
happy
and
comfortable
to
to
build
on
it.
It
wasn't
necessarily
the
case
about
a
year
and
a
half
ago
when
I
originally
started
the
development.
A
A
A
A
A
A
A
A
Another
question
is
object.
Storage
sounds
like
sounds
like
git
and
arc.
They
are
the
inspiration
I
wanted
to
do
something
that's
open
source
and
encrypted
a
lot
of
the
solutions
out
there
and
they
kind
of
work.
Similarly,
but
you
don't
really
know
how
and
the
documentation
on
on
the
exact
crypto
that
the
user
is
very
sparse.
A
Another
problem
that
I
had
is
that
they
were
really
really
slow,
and
so,
if
you
use
a
look
at
something
like
boop,
which
builds
on
git
it
generates,
a
lot
of
small
files
is
generally
very,
very
slow
to
sync.
I
had
the
same
issues
with
tahoe,
loves
and
and
tar
snap
where
they
provide
reasonable
performance
for
putting
stuff
in,
but
there
are
so
many
files
that
actually
retrieving
them
is
is
taking
an
unreasonable
amount
of.
A
A
A
Some
some
things
actually
had
like
a
couple
of
percent
impact
when
I
was
in
lining
on
the
hot
path,
very,
very
simple
stuff,
like
sraf
and
and
other
simple
trade
implementations.
I
think
the
bottom
line
here
is
that
you
kind
of
have
to
play
around
and
see
if
it
works
or
if
the
compiler
does
the
right
thing
anyway,.
A
There's
a
question
about:
how
do
you
protect
encryption
keys
and
manage
them
from
being
swapped
or
read
straight
from
the
memory?
So
there
is
a
is
an
amazing
crate
called
secrecy
which
wipes
the
encryption
keys
from
memory
after
being
used-
and
I
just
had
a
discussion
about
this
recently
with
someone
in
irc.
A
Into
the
memory,
which
means
it
never
hits
the
swap
space,
I
think
this
is
very
often
used
other
than
that.
I
think
operating
systems
started
to
introduce
memory
encryption
schemes-
I
don't
specifically
know
the
details
of
that
and
how
they
work
because
they
tend
to
be
transparent,
but
there
are
no
other
special
protections
apart
from
for
the
key
material,
it's
basically
the
role
of
the
user
to
protect
the.
A
A
A
So
it
wouldn't
be
very
applicable.
The
the
attackers
can
only
observe
four
megabyte
blobs
of
data
that
are
completely
opaque
to
them
and
they
don't
don't
really
contain
any
individually
identifiable
information.
All
of
the
the
root.
The
root
of
trust
in
in
this
is
entirely
derived
from
the
passphrase
and
the
username
that
are
that
are
fed
into
the
encryption.
A
I
started
off
with
failure
and
and
that
that
was
that
was
a
failure
right
now,
I'm
using
this
error
and
anyhow
consistently
everywhere,
and
I
can
highly
recommend
these
they're
great.
They
they're
super
convenient
super
clean,
don't
come
with
any
of
the
baggage
that
failer
does
and,
and
they
are
generally
just
very
easy
to
use
and
understand.
A
A
A
You
can
choose
different
data
formats
that
you
can
you
can
put
in
there.
The
fact
that
I'm
using
this
for
file
encryption
right
now
is
is
something
that
I
had
a
need
for,
but
ultimately
it
it's
extensible
and
versatile
enough
that
you
can
create
your
own
indexing
mechanisms
and
and
build
your
own
cryptographic
designs
into
a
system
that
uses
this
storage
layer.
A
There's
a
question
about
timing
attacks
when
so
what
what
happens
when
somebody
accesses
the
objects
in
quick
succession,
meaning
those
likely
belong
to
the
same
file?
So
this
is
something
that
ingrain
d
can
pick
up.
So
I'm
working
on
the
monitoring
side
and
the
detection
side
of
this
kind
of
stuff
as
well
in
another
project.
A
It
is
completely
completely
possible
to
detect
this
kind
of
stuff
and
and
use
some
sort
of
fancy
statistical
analysis
to
to
actually
start
deriving
this
information.
If
you
are,
if
you
have
compromised
the
server
the
gut
tray,
is
that
you
don't
know
if
somebody
is
trying
to
restore
an
entire
backup
or
actually
just
and
multiple
files-
and
you
don't
know
which
files
they
are
accessing,
and
there
are
other
vectors
where
you
can.
A
You
can
actually
observe
how
much
data
actually
goes
out
on
a
network
connection,
so
the
the
threat
here
is
maybe
identifying
users.
I
think
I
think
that's
possible
and
I
think
that's
that's
very
there's
a
very
legitimate
threat,
but
it
still
still
protects
the
the
user
from
exposing
the
sensitive
data
and-
and
so
I
think,
it's
it
composes
really
well
again,
and
you
can
you
can
deliberately
slow
down
the
server,
for
instance,
so
that
an
external
observer
cannot
cannot
detect
these
timing
attacks.
A
How
does
it
handle
encryption,
encrypting
data
smaller
than
four
megabytes?
It
will
be
just
four
megabytes
if
you
have,
if
you
have
any
any
type
of
data,
it's
gonna,
it's
gonna,
you're
gonna,
see
that
as
multiples
of
four
megabytes,
which
is,
which
is
why,
distinguishing
between
files
and
different
users,
if
they
are
pulled
together
on
the
same
server,
is
sort
of.