►
Description
Raphaël Gomès
Developer and Musician. Working on Mercurial SCM at Octobus
Mercurial is a 17 year-old, mainly Python codebase with a bit of C. There are lots of interesting and unexpected challenges that come with rewriting parts of any such software, and some unique to version control systems: from FFI overhead, not reintroducing bugs, testing, mixed encoding issues, to less technical issues like inexperience with Rust and a lack of reviewers, our journey with Rust has not been smooth sailing.
Nevertheless, Rust helps breathe new air into the Mercurial story by being a much better fit to develop a VCS in than Python. The speed improvements alone would be enough justification but the developer ergonomics are the real main selling point on a lot of levels.
A
So,
let's
get
started
for
those
who
are
not
familiar
material
is
a
version
control
system
that
was
started
at
around
the
same
time
as
git
in
2005.,
it's
mostly
written
in
Python.
There
has
been
C
extensions
since
basically
the
beginning,
because
what
would
you
do
in
2005
for
performance
and
now
rusts
with
way
too
much
code
and
I'm
responsible
for
half
of
it?
A
A
So
we
need
to
pick
something
that
goes
that
needs
to
go
fast
or
faster.
It
needs
to
have
a
small
enough
scope
so
that
it
it's
something
that
can
be
reasonably
well
put
into
the
mind
of
a
single
person.
It
needs
to
be
well
delimited
and
by
that
I
mean
that
the
inputs
and
the
outputs
need
to
be
known
in
advance,
and
it
doesn't
need
to
do
too
much
ffi
use.
So
that's
foreign
function
interface
that
the
interface
between
two
languages,
namely
Python
and
rust,
because
ffi
is
very
expensive.
A
It's
a
special
kind
of
director
of
the
cyclic
graph
called
Merkel
trees,
which
is
a
nice
cryptographic
data
structure
that
gives
you
that
the
children
of
the
hash
of
the
children
is
defined
by
the
hash
of
its
parents.
So
you
have
very,
very
nice
properties
of
that
and
you
are
very
familiar
with
Merkel
trees.
If
you've
ever
looked
at
your
commit
history,
your
commit
log-
and
sometimes
they
look
like
this
and
sometimes
like
this.
A
So
some
people
have
called
this
Guitar
Hero
I
have
failed
to
find
a
similar
joke
with
Mercurial
in
it,
so
yeah,
that's
what
I'm
going
to
be
using.
So,
of
course,
if,
if
your
history
looks
like
this,
you
have
to
find
algorithms
that
can
handle
those
kind
of
complexities
pretty
well.
One
of
those
is
called
the
discovery
algorithm
it's
when
you
do
a
pull
or
a
push.
The
first
one
of
the
first
thing
that
it
needs
to
do
is
to
find
the
difference
between
the
two
repositories.
A
That's
called
The,
Discovery
algorithm.
It
has
its
constraints
of
network
and
latency
and
throughputs.
So,
depending
on
the
situation,
it
might
not
do
the
exact
same
thing.
The
exact
algorithm
algorithm
is
very
simple:
it's
like
250
lines
of
code,
it's
quite
intricate,
but
it's
not
too
much,
and
we
rewrote
this
in
Rust.
It
was
up
to
three
times
faster
and
never
slower
in
the
in
any
case.
So
that
was
good,
very
success.
A
It's
very
successful!
It's
like
a
pretty
small
scope
of
stuff.
The
other
example
I
wanted
to
talk
about
was
something
called
the
node
map,
which
is
an
internal
data
structure
to
Mercurial
it's
a
way
of
finding
data
associated
with
a
hash.
So,
if
you
want
to
say
view
the
contents
of
a
commit,
you
will
give
it
a
hash
or
a
partial
hash,
and
it
has
to
find
where
that
points
to
inside
material.
This
is
called
the
node
map
and
we
rewrote
the
node
map
in
Rust
and
it
became
2
000
times
faster.
A
That's
a
bit
of
a
click
bait,
because
we
didn't
just
rewrite
the
node
map
in
Rust.
We
also
made
it
persistent,
so
most
of
the
work
is
already
persisted
to
disk,
which
explains
why
it's
basically
free.
Now,
however,
the
persistent
storage
is
not
really
as
good.
If
you
do
it
in
Python.
That's
why
it's
interesting
to
have
it
done
in
Rust,
because
python
is
actually
really
not
that
good
at
building
a
tree
like
structures
in
memory
efficiently,
which
is
something
that
Russ
can
do
basically
for
free.
A
So
that
was
very
interesting.
This
basically
disappeared
from
Oliver
profiling.
That's
nice
same
slide
as
before,
with
a
bit
of
an
asterisk.
So
when
picking
what
to
rewrite,
we
need
all
of
those
things,
but
you
specifically
need
something
that
is
well
delimited
and
that
you
need
to
refactor
first.
If
you
need
to
do
so
so
I'm
going
to
explain
how
I
did
not
follow
my
own
advice
and
why
you
should
probably
follow
it.
A
The
status
command
you're,
probably
familiar
with
if
you've
used
any
Version
Control
System
git
has
the
same
kind
of
similar
command.
Its
purpose
is
to
show
what
has
changed
since
the
last
commit
what
files
are
modified
added
removed.
Whatever
needs
to
compile
all
of
the
ignore
patterns
from
the
config
from
the
SGA
ignore
file
Etc
into
one
big,
regular
expression,
and
it
has
to
check
every
track
valve
and
every
untracked
file
it
needs
to
show
them
by
default.
It
looks
like
this.
A
It
makes
use
of
a
data
structure
called
the
dur
state
and
the
durstate
is
integral
to
any
command
that
interacts
with
the
working
copy
within
Mercurial,
so
commit
diff
update,
which
moves
in
history
Purge
files,
whatever
all
of
those
commands,
use
the
dur
state
in
some
capacity.
So
it
was
apparent
that
it
was
one
of
the
areas
where
rust
could
really
improve
the
performance
story
of
mercurial.
A
So
a
very
rough
timeline
of
where
I
started,
which
coincidentally,
was
also
the
time
I
started
learning
rust,
which
so
that
was
a
bit
of
a
bit
of
too
much.
At
the
same
time,
I
started
with
the
parsing
and
packing
of
the
deer
State.
Basically
the
file,
the
binary
format
of
the
dirt
State
having
a
routine
that
can
read
this,
put
it
in
some
kind
of
memory
structure
in
Rust,
and
that
can
write
it
back
to
disk.
This
was
pretty
easy.
A
Then
I
moved
on
to
the
computation
of
the
rejects
that
needs
to
be
used
for
ignoring
files,
and
that
was
a
bit
harder
because
for
multiple
reasons,
the
first
one
was
that
the
the
original
python
implementation
grew
over
time
because
Mercurial
is
was
15
year
old
at
the
time
17
now
and
the
code
was
wasn't
as
well
delimited
and
and
nicely
split
as
it
could
have
been.
So
there
should
have
been
my
first
intuition
to
first
start
reworking
a
bit.
The
python
implementation.
A
Also
the
rust
default,
reject
system.
I
know
that
there's
isn't
one
in
the
standard
library,
but
basically
the
rejects
right
doesn't
work.
The
exact
same
way
that
the
python
standard
rejects
engine
works,
so
there's
different
patterns
that
it
can
and
cannot,
support,
which
was
a
bit
of
a
hassle
because
we
need
we
needed
to
find
a
way
of
falling
back
to
the
python
implementation.
If
there
was
one
of
those
around.
A
It
was,
and
I
understood
this
much
later
in
the
in
the
in
in
the
process,
because
I
didn't
really
talk
to
other
call
developers
about
it.
That
that
knew
something
about
this
code
is
that
Everybody
hated
the
API,
so
I
assumed
naively
that
the
the
API
was
built
correctly,
and
that
is
it
was
the
right
API
because
it
was
there
for
15
years.
So
why
wouldn't
be
the
right,
API
right
and
well
looking
at
it?
A
This
is
where
all
of
the
shortcomings
of
the
previous
things
that
I've
stated
were
really
put
into
into
light,
because
the
data
structure
wasn't
good
enough
anymore
for
the
performance
we
were
looking
for.
The
the
file
formats
on
disk
wasn't
good
enough.
Also
any
of
the
new
semantics
that
we
wanted
to
add,
so
that
Mercurial
can
be
a
better
tool.
A
Everything
was
wrong,
so
we
had
to
actually
restart
not
from
scratch,
but
to
redo
a
lot
of
the
work
that
we
had
done,
which
wasn't
very
good,
and
the
takeaway
from
this
is
that
you
need
to
try
to
understand
the
constraints
up
front
when
you're
rewriting
something
before
even
deciding
to
rewrite
something
understanding.
What
exactly
the
the
problem
space
or
the
thing
that
you're
trying
to
rewrite
is.
A
Maybe
the
algorithm
is
actually
really
bad
and
just
rewriting
it
in
the
same
language
could
be
just
good
enough
and
also
what
is
it
trying
to
do
and
is
the
current
solution
good
enough,
or
does
it
actually
need
to
change?
And
it's
just
not
just
a
rewrite
refactor.
First,
it
helps
you
understand
what
the
the
intricacies
of
the
thing
like,
for
example,
just
writing.
A
doc
string
sometimes
makes
you
understand
what
the
implications
of
such
a
method
are.
A
We
have
problems
of
scaling.
The
idea
of
a
big
repository
in
2005
was
really
not
the
same
as
it
is
today
in
in
2022,
so
it's
a
moving
Target
and
sometimes
the
solution.
That
was
the
right
solution
for
many
reasons,
maybe
for
just
time
constraints
or
whatever
was
a
good
idea
at
the
time,
but
it
moves
all
always
so
you
have
to
keep
that
in
mind
and
also
very
good
way,
having
a
way
of
a
b
testing
the
two
implementations,
so
we
have
What's
called
the
module
policy.
A
It's
just
something
that
allows
us
to
choose
which
backend
we're
using
for
materials.
So
you
can
you
test
the
pure
rust
version
of
the
pure
python
version,
the
version
with
some
C
inside
and
the
version
with
rust
inside,
and
we
can
run
the
whole
test
Suite
with
both
you
compare
and
to
to
like
all
of
the
regression
tests
that
we
had
and
to
see.
If
the
the
new
implementation
carries
over
the
bug
fixes
from
the
old
implementation.
That's
pretty
important.
A
So
this
was
all
work
done
from
inside
python.
It
was
python
is
called,
and
then
it
calls
into
a
rust
extension,
but
we
can
attack
the
python
from
the
other
side
and
create
an
executable
in
Rust,
a
pure
rust
executable,
so
that
we
can
allow
the
user
to
have
pure
rust
fast
path
if
they
choose
to
to
use
that,
so
it's
called
RG
very
inspired
and
its
constraints
was
that
it
needs
to
be
transparent.
People
expect
Mercurial
to
always
just
work
and
we
have
broken
backwards.
Compatibility,
maybe
twice
on
some
very
minor
things.
A
Trust
me
I
would
love
to
break
backwards,
compatibility
compatibility
on
many
things
that
I
really
hate
about
the
Mercurial
CLI.
But
you
know
that's
that's
how
it
is
so
it
needs
to
behave
the
same
way
as
AG
does,
and
it
needs
to
fall
back
immediately.
If
anything,
unsupported
is
is
detected,
which
is
unsupported
command,
CLI
arguments
config
any
other
like
extensions
and
that
kind
of
stuff,
if
anything
that
it
doesn't
support,
is,
is
detected.
A
It
needs
to
fall
back
to
python
as
fast
as
possible,
which
means
that
it
needs
to
delay
expensive
computation.
You
cannot
load
a
very
expensive
data
structure
or
do
some
highly
intensive
file
system
work
before
you
figure
out
that
you
can
actually
complete
the
whole
thing
which
actually
the
rewrite
of
the
their
states,
the
their
state
version
2
that
we
did
allowed
us
to
do
a
lot
of
things
that
we
couldn't
do
before,
because
it
it
would
basically
be
instant
to
read
the
dist
State
nowadays
compared
to
pretty
expensive
before
that.
A
A
So
we
started
with
simple
commands:
I
think
the
first
one
was
HD
root,
which
gives
you
the
root
of
a
repository
and
an
error.
If
there's
no
root,
which
is
pretty
simple
in
itself,
but
you
really
early
needed
to
do
configuration
parsing
and
requirements
parsing.
So
requirements
is
like
a
it's
a
foul
in
Mercurial.
That
tells
you
like
the
feature
set
of
the
current
repository.
So
if
your
Mercurial
implementation
does
not
know
about
certain
features,
it
can
just
abort
inside
I,
don't
know
how
to
handle
this
this
Repository.
A
We've
done
the
work
incrementally
in
that
you
don't
need
to
support
100
of
what
any
command
supports,
because
even
AGA
Roots
is
has
some
very
complicated
config
like
Flags,
because
it's
17
year
old
software
and
people
have
added
flags
all
over
the
place.
So
the
idea
was
to
focus
on
the
fast
path
on
what
is
the
most
common
thing
that
people
are
going
to
be
using
and
that
we
can
leverage
to
give
a
better
experience.
A
We've
done
the
same
thing.
We've
run
this
test
Suite
with
RG,
hiding
as
HG
just
to
see
just
to
see
if
anything
broke
at
all
which
it
did,
and
then
we
fixed
that
and
one
of
the
other
issues
that
I
wanted
to
highlight
throughout
all
of
the
problems
that
I've
been
talking
about,
like
the
status
thing
RG
and
many
other
things
that
I
don't
have
time
to
talk
about
with
the
kind
of
unique
problem
of
having
to
deal
with
mixed
encoding.
A
Mixed
encoding
is
a
problem
that
you
shouldn't
have
to
deal
with
ever
unless
you're
dealing
with
historical
data,
so
Mercurial
like
most
Version
Control
Systems,
you
stores
data
of
Unknown
Origin,
so
it's
it's
been
put
there
in
CVS
or
even
pre-cvs
days
in
the
70s
or
something-
and
nobody
knows
the
encoding,
nobody
will
ever
know
the
encoding.
Maybe
you
could
try
to
guess
it,
but
still
that's
a
big
problem
and
it
could
be
added
today
and
now
nobody
nowadays
everybody
uses
ugf8,
but
that's
not
always
been
the
case
install
not
the
case
everywhere.
A
So
the
this
usually
what
we
call
cursed
encoding,
because
nobody
likes
handling
mixed
encoding,
really
garbage
and
garbage
out
people
put
in
the
data,
and
we
give
it
exactly
by
bite
for
byte.
This
is
one
of
the
what
I'd
like
to
call
unknown
unknowns,
it's
something
that
people
don't
budget
for
because
they
don't
even
know
that
the
problem
exists.
I.
Certainly
one
was
one
of
those
people,
so
it's
certainly
like
was
kind
of
a
surprise
to
have
to
figure
understand
that
problem
I,
usually
when
I
need
to
I
need
some
help.
A
I
go
to
the
rust
Discord
right
and
someone
is
trying
to
help
me
with
my
problem
and
I'm
like
oh
I
can't
use
string
type
because
string
type
is
utf-8.
Then
it's
going
to
eat
my
data
and
people
are
like.
Why
should
why?
Aren't
you
using
the
string
type
everything
is
used
if
it
anyway,
then
I
spend
10
minutes
explaining
this,
and
it's
like
just
assume
assume
that
I
need
this
please.
A
So
it
goes
sometimes
against
the
ecosystem,
because
a
lot
of
crates
and
a
lot
of
tooling
expects
you
to
have
nice
utf-8
data,
which
I
completely
understand
why
they
would
do
that,
but
we're
not
the
only
problem
space
where
that
is
an
issue.
Actually,
the
maintainer
of,
for
example,
the
rejects
crate,
has
done
a
lot
of
work
with
byte
strings
because
it's
actually
needed
for
a
parsing
arbitrary
data.
A
A
The
ones
above
are
the
pure
python
versions
of
an
execution
of
Ages
status
on
a
huge
repository,
namely
Mozilla
Central,
which
has
a
600
000
files,
or
something
which
is
like
much
much
larger
than
the
the
Linux
kernel
and,
as
we've
been
rewriting
the
durstate.
The
pure
version
of
python,
which
nobody
uses,
especially
in
those
environments,
pure
version,
pure
python
version
of
mercurial,
started
to
go
way
up
in
terms
of
of
well
time,
which
isn't
very
good.
A
This
is
on
a
machine
without
hyper
threading
without
many
cores,
so
this
is
actually
worse
than
it
looks
because
on
a
better
machine,
you
would
have
much
better
performance
than
this.
So
we've
gone
from
something
that
can
give
you
three
seconds
of
latency
to
get
your
feedback
to
something
that
can
do
maybe
40
milliseconds
for
the
exact
same
results,
which
I
think
is
pretty
cool.
Of
course,
as
you
can
see,
it's
not
all
roses
and
everything
because
we
made
some
other
things
slow.
A
It
turns
out
that
they
don't
matter
to
anyone,
but
still
something
to
keep
in
mind.
If
that's
a
use
case,
that's
important
for
you-
and
this
is
without
some
of
the
newest
optimizations-
that
we're
currently
rolling
in
some
of
the
things
that
you
have
to
consider
when
introducing
rust
inside
of
your
project
is
that
adopting
rust
has
also
a
human
cost
to
it.
It's
not
just
a
technical
cost.
There
have
been
a
lot
of
people
that
have
talked
about
this
at
this
event,
so
I'm
just
going
to
have
one
slide
about
it.
A
Basically,
it's
more
tooling
more
code.
More
code
is
not
good.
Less
code
is
good.
It's
more
tooling
fewer
people
with
experience
with
the
Tooling
in
general
you're,
introducing
a
lot
of
complexity
in
your
code
base,
potentially
code
duplication,
because
we
want
to
still
provide
a
pure
python
pass
path
or
a
a
non-rust
path
to
all
of
our
users.
A
It
leads
to
a
lot
of
code,
duplication,
which
is
sometimes
a
good
thing,
because
you
have
multiple
implementations
of
some
something
and
it
can.
It
can
show
you
that
the
API
wasn't
right
if
you're
rewriting
in
Rust,
because
you
could
do
some
Blackmagic
in
Python,
but
that's
not
possible
in
Rust,
for
example.
So
that's
sometimes
a
good
thing,
sometimes
a
bad
thing.
If
you
get
a
bug
report,
you
need
to
fix
it
in
two
different
implementations:
I've
done
it
and
it's
not
great,
but
you
have
to
fewer
reviewers,
are
comfortable
so
far.
A
Reviewing
rust.
We've
actually
seen
this
move
forward.
Within
the
past
three
years,
I
would
say
people
were
very
hesitant
to
to
review
rest
code
unless
they
were
actively
using
rust
at
work
or
something,
and
nowadays
more
people
aren't
very
interested
and
they're,
like
oh
I,
want
to
start
learning
russet
around
I'm
like
yeah.
Please
do
so
there's
like
you
can
merge
my
merge
request,
please.
A
A
For
example,
if
the
only
thing
you're
interested
in
is
is
making
the
rust
performance
better
than
you
might,
as
we've
seen,
maybe
in
the
performance
slide,
you
might
neglect
the
other
implementations,
and
this
is
something
that
can
impact,
for
example,
the
net
BSD
people
who
still
don't
have
rust
or
not
easily
it's
adding
a
new
language.
It's
adding
a
compiled
language
to
python,
who
is
already
has
a
really
difficult
packaging
story.
A
I
still
think
that
if
I
was
to
do
it
all
over
again,
which
I'm
I
didn't
create
material,
but
if
I
were
to
build
a
new
VCS
right
now,
I
would
use
rust
almost
entirely
pretty
sure
in
2005.
It
was
I.
Think
the
right
choice
to
pick
python
or
C
C
was
obvious
for
performance
reasons,
but
having
the
the
more
high
level
view
of
python
was
very
useful
at
the
time.
A
A
A
If
you
you
have
600
000
files,
you
can
do
an
Aegis
status.
That
gives
you
an
answer
in
about
44
milliseconds,
on
an
8
core
machine
that
with
decent
decent
power,
44
milliseconds
gives
you
a
python
interpreter
that
has
just
started.
So
it's
on
another
class
of
performance,
if
you're
in
the
in
the
command
line,
tooling
VCS
can
do
a
lot
of
things
in
parallel.
There's
a
ton
of
stuff
in
inversion
control
systems
that
benefit
from
being
done
in
parallels.
So
breast
is
quite
good
at
this
in
the
status
algorithm.
A
Everything
is
pretty
much
done
in
parallel
if
you
update,
so
that's
what
you
make
I'll
check
out
in
git.
So
if
you
go
to
another
version
of
the
history,
you
need
to
update
all
of
your
files
and
everything
this
can
be
done
mostly
in
parallel.
Some
of
the
network
stuff
can
be
done
in
parallel,
which
is
all
very
good,
I.
Think
one
of
the
best
things
about
rust.
A
If,
if
you
haven't
read
the
excellent
blog
post
about
this,
it's
very
easy
to
find
nowadays
because
it's
very
popular
but
there's
a
ton
of
stuff
whenever
I
go
back
to
python
or
any
other
language
that
doesn't
have
algebraic
types.
For
example,
I'm
just
super
sad:
it
is
quite
portable
not
as
portable
as
C
or
or
python,
but
it's
quite
portable,
which
is
a
nice
story
like
the
the
windows.
Packaging
story
is
not
not
harder
than
the
Linux
one
for
our
particular
case.
A
Windows
implementation
story,
that's
another
effect
and
all
of
the
other
nice
things
about
rusts.
That
is
not
specific
to
Version
Control
Systems.
It's
rewriting
the
entire
thing
in
Rust
in
Mercurial
is
a
big
step
because
it
breaks
backwards.
Compatibility
in
a
lot
of
ways
it's
a
ton
of
work
and
it's
probably
not
feasible
in
the
short
term,
but
new
tooling
can
be
done
in
Rust
and
we've
shown
and
improved
ourselves.
A
That
rust
is
an
amazing
fit,
even
with
all
of
the
issues
that
I've
mentioned
and
all
of
the
things
that
make
it
a
bit
harder.
The
learning
curve
Etc
it
makes
it
really
worth
it
for
a
Version,
Control,
System,
I
hope.
Some
of
it
was
enlightening
for
you,
I'd
like
to
thank
you
for
staying
and
not
falling
asleep
too
much,
and
thank
you.