►
From YouTube: RustConf 2018 - Project Mentat: a store for evolving data in Rust Delivered by Grisha Kruglov
Description
RustConf 2018 - Project Mentat: a store for evolving data in Rust Delivered by Grisha Kruglov
Prepared by Emily Toop
Mozilla has a data storage problem. There is no unified strategy for storing new user data and moving it between devices, which has led to a proliferation of stores and strategies and a fragile mechanism for syncing that only handles a subset of data collected. Project Mentat was born to help resolve these issues - a structured data store that is designed to be replicated and synchronized and written in Rust to ensure security, safety and maximum portability between platforms.
This talk is about why we decided to "build" rather than "buy", how Mentat is designed to address the problems and requirements associated with data storage and syncing in an environment with trong client-side encryption, - as well as the challenges encountered and lessons learned while using Rust to create portable, performant, persistent, syncing, structured data storage.
A
That
works
okay,
yeah
thanks
everyone
for
coming.
My
name
is
quisha
Kruglov
I
work
for
Mozilla
as
an
engineer
and
I'm
here
as
a
poor
conduit
for
Emily
slides.
She
could
make
it
at
last
moment
so
I'm
here
to
talk
about
projects
meant
at
which
is
a
embedded
rust
or
that
we're
using
at
Mozilla
to
solve
some
of
our
problems
and
I'll
talk
about
what
those
problems
are,
how
we're
trying
to
solve
them
and
a
bit
a
bit
about
that.
That
team
is
working
on
that
time.
A
What's
what
what
changes
we
need
to
make
now
to
solve
our
three
to
five
year
goals
and
for
Mozilla
and
for
Firefox
those
are
expanding,
Firefox
and
so
like
making
the
Firefox
better,
creating
user
experiences
and
also
going
beyond
Firefox
building
the
applications
to
solve
particularly
user
needs,
and
a
lot
of
this
stuff
Revolt
like
there's
a
lot
of
hurdles
involved,
but
one
of
the
big
things
is
user
data
right.
It's
like
okay,
that's
our
big
moat!
A
Essentially
is
we
have
a
lot
of
valuable
data
and
people
would
like
to
access
it
in
different
types
of
applications
and
our
apps
will
like
to
use
them
and
currently
that's
problematic.
We
have
for
reasons
that
I'll
kind
of
go
through
so
part
of
the,
so
one
of
the
problems
is
storing
storing
the
data,
so
we
store
it
like
something
like
a
browser
stores,
a
lot
of
it'll.
You
will
be
surprised
how
many
different
things
it
stores
some.
What
about
is
about
user
behavior?
A
Some
of
is
some
of
it
is
about
just
like
internals
of
the
browser
and
a
lot
of
it,
and
so
there
are
a
lot
of
different
teams
involved
in
installing
this
stuff,
and
it
seems
make
their
own
this.
It's
about
data,
storage,
layers
and
oftentimes.
Those
decisions
are
driven
more
by
immediate
needs
like
how
do
we
make
this
thing
work
for
the
cui,
how
to
make
it?
How
do
we
query
it
quickly?
A
A
Well,
let's
just
do
something
simple:
if
you
like
a
JSON
store
or
something
and-
and
that
doesn't
tend
to
scale
all
that
well
right-
it
might
solve
your
really
immediate
needs,
but
it
doesn't
necessarily
solve
future
needs
and
you
know
like
building
fast
concurrent,
safe
access
to
adjacent
store.
That
will
you
know
what
happened
across
many
different
components?
A
You
will
sorry
I,
don't
know
I,
don't
own
voice,
slides
here,
but
you'll
you'll
end
up
with,
like
with
a
lot
of
will
host
of
other
problems
right
so
like,
if
you
so,
for
example
like
if
you
picked
a
sequel
light
store,
as
your
storage
model.
You'll
still
need
to
handle
a
lot
of
migrations
and
here's
your
project
evolves
as
your
data
needs
of
evolved.
That
tends
to
become
quite
problematic
as
well.
A
When
people
don't
necessarily
think
about
think
in
a
correct
ways
about
those
problems,
because
it's
it's
it's
a
fairly
specialized
skill
doing
this
stuff
like
building
data
stores
that
will
sync
and
it
will
wolf
well
over
time
and
another.
So
that
kind
of
ties
into
the
second
point
is
sharing
good
data,
and
by
sharing
we
mean
sharing
amongst
mullick
and
as
far
as
Firefox,
we
mean
specifically
sharing
it
cause
different
instances
of
Firefox
well
and
if
you
sure,
between
different
products
that
use
Firefox
data.
A
So
we
have
strong
privacy
guarantees
and
it's
part
of
like
a
big
part
of
that
is
enter
an
encryption
and
which
means
that
clients
have
to
do
all
of
the
conflict.
Resolution
work
server
cannot
be
there
like
servers
just
to
dump
a
blob
store,
more
or
less,
and
we
have
multiple
clients
and
the
set
of
clients
is
growing
and
would
like
it
to
grow
right.
A
That's
part
of
our
three
to
five-year
goals
is
that
you
know
building
more
products
that
use
this
data
right
and
then
each
client
is
they
have
their
own
data
storage
implementations
are
written
in
different
languages.
You
have
JavaScript
C,
Java
Swift,
with
sutter
implementations
and
yeah,
and
and
and
the
clients
tend
to
build
data
like
that.
The
teams
that
are
building
this
stuff,
they
tend
to
build
data.
A
That's
oriented
around
their
immediate
needs
so
like
wearing
this
stuff
like
which
isn't
and
the
the
schemas
you'll
you
take
for
that
are
not
necessarily
the
schemas
you'll
take
if
you
want
to
optimize
to
optimize
for
syncing,
and
so
you
run
into
problems
like
you
can't
do
three,
we
merges
of
data
because
you
don't
have
historical
view
into
how
some
the
tables
changed
and
yeah
so
here.
So,
let's,
let's
I,
guess:
let's
try
to
look
at
a
concrete
example
of
what
that
actually
looks
like
in
practice.
A
Like
here's,
a
symbol
like
a
really
simplistic
way
to
model
a
password
storage
layer
so
like
it
will
have
two
clients.
We
have
a
server
in
the
middle
which
it
just
acts
as
a
dump
blob
and
the
schema
is
very
basic.
You
there's
a
URL
for
that
URL
who
have
username
and
password
and
and
there's
a
time
step.
A
So
there's
some
immediate
problems.
As
you
start
making
changes
here
so
say.
Imagine
client
1
makes
a
change
to
a
URL,
maybe
they've
observed
now
that
I
think
there's
a
change.
So
they've
observed
the
URL
changed
and
they
record
one
that
the
change
happen
and
client
2
makes
a
change
to
the
password.
A
So
you'll
see
the
password
is
no
different
and
there's
some
animation
yeah
there
you
go
yeah
so
like
client,
1
syncs,
the
data
is
now
on
the
server
and
client
2
will
sink
eventually,
and
the
data
will
now
slowly
yeah
you'll
notice,
that,
like
we
essentially
get
a
blob,
JSON
blob
back
right
and
now
at
this
point
we
have
to
make
a
decision
like
we
have
to
JSON
blobs
we
and
we
have
to
somehow
smush
them
together.
Right.
A
We
see
that
the
URL
change
the
password
changed
and
this
information
is
on
the
server
already,
so
other
clients
might
have
seen
it.
So
we
don't
have
a
ton
of
choice
here.
We
have
to
take
what
we
see,
which
is
usually
what
happens
since
in
like
simplistic
kind
of
syncing
scheme
flows
like
this
right,
and
so
we
end
up
on
the
client.
We've
just
lost
the
password
so
like
this
is
a
lossy
process.
We
lost
the
password
changed
and
we
end
up
like
with
a
state
that
we've
never
really
observed
and
clients.
A
Week
has
really
no
way
to
recover
from
this,
and
so
there
ways
to
kind
of
fix
that
problem
in
an
existing
world,
but
they
tend
to
involve
a
lot
of
a
lot
of
work.
Essentially
right
like
this,
we
could
track
more
states
on
each
client,
but
as
I
mentioned,
we
have
many
clients
and
they
all
have
different
implementations
of
stores,
and
it
involves
writing
a
lot
of
complicated
code
many
times
in
different
languages
and
trying
to
coordinate
across
teams
to
do
that.
A
But
there's
this
concept
called
so
a
bit
like
one
of
the
lessons
that
we
owe
that
people
tend
to
learn
as
they
write.
This
stuff
is
it's.
It
helps
to
separate
the
way
you
query
stuff
from
the
way
you
modify
your
data
right
and
that's
there's
this
idea
of
CQRS
so
that
you
essentially
segregate
separate
query
from
command
right.
A
So,
like
you
separate
the
way
you
modify
data
from
you
the
way
you
query
it-
and
that
was
you
address
individual,
like
needs
specific
to
each
side
of
that
equation
in
a
more
coherent
way
and
be
very
explicit
about
that
separation.
So,
for
example,
we'd
like
fast
querying,
but
we'll
also
like
data
that
is
syncable
and
those
two
are
kind
of
at
odds
with
each
other,
and
this
kind
of
approach.
Making
this
explicit
lets
us
actually
address
the
Boyd,
make
some
progress
or
the
problem
with
the
problem
and
so
yeah.
A
A
Some
many
different
platforms
with
a
single
implementation
and
so
ideologically
in
the
we've
came
across
the
atomic,
which
is
a
database
that
exists
in
the
closure
world
and
it's
transactional.
It
has
human
data
models
in
the
sense
that
the
like
there
well,
we
have
a
strongly
typed
schema
with
it
and
we
can
mutate
it
for
time
easily.
We
have
transaction.
We
have
a
log
of
everything
that
happens.
A
We
have
rich
querying
via
the
data
log,
which
is
their
querying
language,
but
yeah
it's
only,
but
only
in
the
spirit
is
it
like
a
good
good
fit
visits
it
well.
It
actually
is
a
server-side
kind
of
a
system.
It's
not
open
source
and
yeah.
Then
we
get
to
data
script
which
exists
in
the
same
similar
mindshare.
It's
a
closer
script.
A
A
And
this
like
this
was
a
good
prototyping
decision.
We
ran
into
a
bunch
of
problems
I
guessing
channels
etc,
like
the
transpiling
process
was
slow
and
buggy,
and
it's
just
an
again
like
this
required.
A
JavaScript
runtime
which,
like
it
is,
there's
a
hard
requirements.
If
you're
trying
to
embed
this
across
and
use
this
across,
like
obvious
Android
applications
and
also
in
desktop,
etc,
and
so
we
yeah
so
like
rust-
is
a
natural
next
kind
of
a
choice
like
the
only
real
alternative
to
this
is
C++.
A
It's
and
we
get
predictable
correct
results
once
we
sort
of
work
through
all
of
the
problems
right
and
we
it's
performance
right
so
like
there,
it's
not
all
kind
of
roses.
It's
the
implementation
took
much
longer
to
write
and
something
done,
the
initial
prototype
enclosure
script
and
it's
still
a
kind
of
an
ongoing
thing.
But
we
like
we
get
like
we.
We
get
a
lot
of
things
out
of
it
right,
the
in
the
trifecta.
That
meant
that
cares.
A
The
rostra
effect
that
meant
that
cares
about
this
kind
of
correctness,
combined
with
portability,
so
cross-platform
combine
the
performance,
so
those
sort
of
things
that
really
make
a
difference
here
so
and
if
we
kind
of
peek
under
the
hood
of
Mendte,
it's
essentially
it's
like
the
core
is
the
transaction
log,
and
this
isn't
quite
what
like,
if
you,
if
you
were
to
open
up
an
entire
sequel,
I
databases,
and
this
isn't
an
exact
representation
of
its
close
enough.
It's
a
quad!
Well,
it's
a
tapas
store.
A
And
so
that's
what
this
looks
like
I'm
like
like
under
the
hood
and
so
the
the
thing
that's
missing
here
is
the
you
know.
The
values
can
be
typed
and
we
have
a
bunch
of
different
types
with
support,
and
you
know
that
we
can
enforce
that
at
schema
level
and
then
mentat
will
complain
if
you're
trying
to
write
wrong
types
into
data
yeah
and
so
and
let's
try
to
and
like
this
is
an
example
of
what
the
transaction
table,
how
data
flows
around
right
there.
A
What
it
actually
looks
like
for
real
so,
for
example,
like
it's,
also
simplistic,
but
it's
I
think
it
illustrates
the
problem
well
enough,
so
we
have
the
same
situation
as
before.
We
have
client
1
to
blob
the
service,
still
like
a
Dom
server
blob
everything
is
still
on
to
and
encrypted
and
except
now
we
have
those
transaction
logs,
and
so
let's
try
start
making
changes
all
right.
So,
like
me,
changing
a
password
inclined
to
we're
missing
some
records,
but
this
is
good.
A
Enough
is
essentially
evolves,
appending,
a
new
password
to
the
log
and
you'll
notice
that
it's
now
it's
in
a
different
transaction
and
let's
make
a
similar
change
in
I'm
client
one
will
append
a
new
new
URL
and
there
you
go
now
we're
syncing
stuff
we
append
like
so
log.
A
server
essentially
has
like
an
encrypted
log
which
to
into
which
we
appended
new
records,
and
so
that's
when
we
sync
those
records
back.
A
We
now
have
like
you'll
notice
that
we
have
conflicting
IDs.
So
it's
almost
an
implementational
detail.
Since
we
chose
integer,
you
know
the
cool
like
transactions
are
presented
as
an
integer,
but
like
we
perform
something
akin
to
a
git
rebase,
where
you
playback.
Will
you
rewind
your
current
state
back
to
our
shared
parents?
You
play
in
remote
changes
and
then
your
local
changes
on
top
and
like
that's
one
way
to
do
this,
you
could
also
represent
this
as
a
merge
or
depending
on
your
needs
or
like
the
schema.
A
My
different
things
might
make
sense,
but
like,
if
you're
just
doing
a
simple
rebased.
Well,
you
end
up
with
this
nice
linear
log,
where
you
have
like
an
exact
record
of
everything
that
happened
in
the
system.
You
have
the
old
password,
any
passwords,
the
URLs
everything,
and
then
we
sync
back
the
log
back
to
the
first
clients.
A
There
you
go
and
so
now
both
clients.
They
agree
on
the
log.
They
have
a
coherent
view
into
the
system
and
they
have
a
nicely
like
they
have
a
nice
historically
of
everything
happened,
and
so,
like
your
conflict
resolution,
algorithms
like
specific
to
the
domain,
so
the
data
models
can
make
the
right
decisions,
which
is
nice.
A
So
that's
kind
of
flow
right
and
then
and
then
then
we
get
back
to
kind
of
querying
this
data
so
like
we
talked
about
CRS
and
how
it's
important,
to
kind
of
think
about
that
separation
of
how
you
restore
data
and
and
how
you
modify
the
data
and
on
how
you
read
it
back
right
and
then
those
are
quite
different
like
if
you
were
to
read
the
transaction
log
away
from
the
beginning.
It's
it
will
be
a
terrible
experience,
performance
wise
since
you
have
to
actually
so
like.
A
So
we
do
that
for
you
like,
we
call
that
the
datum
stable
and
it's
a
it's
a
it's
essentially
a
materialized
view
into
the
transaction
like
it's
a
snapshot
of
the
transaction
log
as
it
is
as
of
the
last
transaction,
and
so
so
you
notice
it's
missing
the
added
field.
It
sounds
small,
it's
not
it's
just
the
snapshot!
This
is
what
it
is
now.
A
We
expands
I
was
into
like
from
the
keywords
like
login
URL
into
like
an
actual
entity,
and
the
datum
stable
will
contain
that
I'm
going.
But
when
you're
querying
this
data,
you
don't
have
to
worry
about
stuff
like
this,
you
can
you
describe
the
attributes
you'd
like,
and
this
is
like
a
simple
example
of
data
log
query
that
you
would
use
to
query
this
stuff.
A
You
describe
what
you
would
like
and
you
define
your
bindings
like
URL,
username
password
will
be
bound
to
login,
/url,
etc,
and
you
could
back
and
then
internally
mentat
will
compile
this
down
into
my
chip,
sequel
and
you'll
notice.
That's
it's!
A
self
joins
the
datums
on
adam's
table,
so
we're
querying
we're
not
querying
the
transaction
log.
We're
querying
this
internal
materialized
view
of
it.
It's
a
bit
quicker.
It's
not
like
it's
still
very
slow
or
slow
enough,
but
it's
yeah,
but
like
it's,
that's
how
it
would
sort
of
look
like
internally.
A
It's
like
you
go
from
from
from
this
into
into
this
and
you'll
notice
that,
like
we're
not
doing
anything
really
tricky
here,
like
we've,
resolved
the
attributes
into
their
aunt
IDs
and
we're
joining
on
anti
D.
So
it's
ensure
we
get
the
correct
view
of
the
data
for
each
entity
yeah
and
then
you
get
diagnosed
results,
and
this
is
missing
at
the
anti
Z
column,
but
you
could
buy
data
from
from
the
system
and
and
then
you
do
this
at
infinite,
like
the
the
datum.
Stable
is
just
a
simple:
it's
an
internal
materialized
to
you.
A
It's
one
materialized
to
be
you
in
the
system,
but
like
the
secure
us
lesson
is
the
is
that
applications
are
usually
well
served
by
many
many
different
views
that
define
that
like
represent
the
data
that
the
uise
or
whatever
else,
would
like
to
query
in
in
your
data
store.
So
you
have
a
really
normalized
data
store
and
you
have
the
normalized
views
into
that
data
store,
and
so
we
have
or
will
have
anyway,
is
a
user-defined,
materialized
views.
Where
you
define
like
say
you
might
like.
A
If
you're
in
the
browsing
citation
you
might
only
care
about
the
top
10
visited
websites
and
your
history
might
have
tens
of
thousands
of
records,
but
you
only
care
about
the
10
of
them,
and
you
only
want
your
view
and
when
you're
acquiring
your
view,
you
like
it
will
be
really
quick.
You'll
just
have
ten
records,
and
many
may
maybe
like
5
10
columns
or
something
so
the
Quarian
will
be
really
quick
and
internally.
A
Mentat
will
use
transaction
listeners
to
monitor
changes
to
the
to
its
log
and
updates
the
views
you've
defined,
which
is
nice
and,
like
it's
the
same
mechanism
that
it
uses
internally
for
its
datums
table
for
its
atoms.
You
and
then
you
can
define
as
many
of
those
users
you'd
like
as
your
application
requires,
and
there
will
be
very
domain
specific
for
you
and
that's
your
kind
of
way
to
query
data
and
but
you're
still
when
you're
writing
data
you're,
not
touching
you,
your
views,
obviously
you're
still
writing
into
transaction
log.
A
A
Well,
two
mobile
platforms,
one
too
many
and
as
well
as
Firefox
desktop
itself
and
maybe
other
consumers
in
the
future.
And
so
we
have
a
public
API,
which
is
wrapped
in
the
FFI
layer
which
lets
you
access
it
from
from
non
Brust
code,
and
we
built
an
iOS
sdk
in
swift
and
an
android
sdk
in
java
they're.
A
So
sinking
is
working
progress,
we're
still
working
like
various
merging
algorithms,
we're
working
on
forgetting
data,
which
is
important
from
like
post,
both
privacy
concerns
like
the
right
to
forget
and
just
the
ability
for
people
to
erase
history
from
their
log
that
they
don't
like,
as
well
as
just
from
engineering
perspective
like
you,
have
an
infinite
log
and
I
really
find
that
universe
and
it's
just
not
gonna
work
forever
and
yeah
we're
still
evolving.
Our
IP
API
a
little
bit
as
we're
building
our
internal
consumers.
A
That's
shaping
the
API
and
writing
the
commutation,
and
the
next
step
is
shipping.
This
in
Firefox
log
box,
which
is
a
I
standalone,
password
manager
that
Mozilla's
building
that
gives
you
access
to
all
of
your
Firefox
account
passwords,
and
so
it
will
be
backed
by
mentat
and
eventually
syncing
via
meant
as
well
and
and
in
the
future.
A
We
have
a
vague
transition
plan
and
it's
sort
of
coming
to
fruition
with
some
of
the
efforts
internally
and
the
next
is
I.
Guess
you
guys,
like
perhaps
other
people
have
meant
at
shaped
problems
and
would
love
to
help
you
address
them
and
get
you
involved
in
the
development
of
this
stuff.
So
the
links
are,
it's
everything
is
in
standard
places.
Although
the
development
happens
to
the
github,
the
commutation
is
there.