►
Description
Whoops! I Rewrote It in Rust by Brian Martin
Three engineers, at various points, each take their own approach adding Rust to a C codebase, each being more and more ambitious. I initially just wanted to replace the server’s networking and event loop with an equally fast Rust implementation. We’d reuse many core components that were in C and just call into them from Rust. Surely it wouldn’t be that much code… Pelikan is Twitter’s open source and modular framework for in-memory caching, allowing us to replace Memcached and Redis forks with a single codebase and achieve better performance. At Twitter, we operate hundreds of cache clusters storing hundreds of terabytes of small objects in memory. In-memory caching is critical, and demands performance, reliability, and efficiency. In this talk, I’ll share my adventures in working on Pelikan and how rewriting it in Rust can be more than just a meme.
A
A
I've
been
at
twitter
for
seven
years,
and
I've
been
using
rust
for
six
years.
I've
worked
on
a
few
open
source
projects
in
rust,
including
a
benchmarking
tool,
a
systems
performance,
telemetry
agent,
and
now
I'm
working
on
a
cache
server
when
I'm
not
at
a
computer.
I'm
volunteering
with
my
local
search
and
rescue
team
and
I'm
training,
my
dog
riker,
to
find
people
who
get
lost
in
the
wilderness.
A
A
A
This
leads
me
to
pelican,
which
is
twitter's
caching
framework
that
we
want
to
use
to
replace
our
forks
of
memcache
and
redis.
Pelican
gives
us
a
single
code
base
to
produce
multiple
caching
solutions
and
to
easily
experiment
with
ways
that
we
can
come
up
with
solutions
that
leverage
new
technologies.
A
A
A
A
They
decided
to
use
tokyo
to
write
the
async
server
runtime,
but
leverage
the
c
components
from
the
normal
cache
server
for
storage
and
some
of
the
other
foundational
libraries.
But
this
was
pretty
neat.
It
showed
we
could
do
a
lot
more
advanced
layering
of
c
and
rust
together
in
the
code
base,
and
I
was
super
excited
to
see
that
rust
could
in
fact
play
a
larger
role
in
pelican.
A
A
A
A
A
The
key
takeaway
here
is
that
multiple
cash
requests
for
a
higher
level
request
causes
us
to
see
the
tail
latency
more
often
than
you
think,
and
that
makes
us
care
about
these
percentiles.
That
might
otherwise
seem
like
they're
way
far
out
in
the
tail,
and
don't
really
happen
that
often
but
with
fanout.
They
do.
A
A
First,
we
need
to
do
performance
testing
and
unfortunately,
we
found
there
were
some
performance
problems
with
this
implementation.
The
throughput
was
about
10
to
15
percent
slower.
This
means
that
we'd
need
10
to
15
percent,
more
instances
to
match
the
throughput
requirements
for
the
cash
cluster.
That
means
it's
10
to
15
percent
more
expensive
to
run.
A
A
A
A
We
start
with
a
ping
server
because
it
allows
us
to
prove
out
the
performance
of
a
design.
We
have
all
the
same
functional
blocks
as
a
real
service,
except
we're
just
not
storing
any
data,
but
we
can
add
all
the
things
we
need
in
terms
of
connection
and
request,
handling
and
debug
ability
and
metrics.
A
A
A
And
then
we
benchmark
again,
just
like
the
previous
chart,
we're
looking
at
request:
latency
verse,
request
rate
and
specifically
looking
at
the
p39
latency.
We
can
see
that
these
are
pretty
close.
This
looked
good
to
submit
a
pull
request
and
merge
into
the
pelican
repo
on
github
feeling
good
about
this.
It
was
time
to
move
on
to
what
I
was
actually
supposed
to
be
doing.
A
The
primary
difference
between
twim
cache
and
the
ping
server
is
that
we
have
a
much
more
complicated
protocol.
The
memcache
protocol
has
a
whole
bunch
of
different
requests
with
variable
numbers
of
fields,
and
my
main
focus
was
on
getting
all
that
protocol
handling,
correct
and
making
sure
that
it
performed
well.
A
A
So
once
I
had
this
in
place
as
a
prototype,
I
was
able
to
do
some
benchmarking
and
make
sure
that
I
was
still
on
the
right
track
in
terms
of
how
the
thing
would
perform
while
responding
with
actual
payloads
right
so
responding
with
a
pong.
That's
only
a
handful
of
bytes
to
respond,
but
with
a
cache
we
might
have
items
that
are
much
larger
that
we're
responding
with.
So
I
wanted
to
make
sure
that
all
the
buffering
and
parsing
and
response
composition
was
all
really
good,
so
I
did
some
benchmarks
and
those
were
looking
promising.
A
But
of
course,
I'd
run
into
some
issues
at
this
point,
choosing
to
use
rust
versions
of
some
of
the
foundational
libraries
left
me
stuck
in
terms
of
trying
to
use
a
storage
library
which
had
metrics
and
logging
deeply
integrated
into
it,
and
we
wanted
that.
We
need
really
good
insight
into
the
performance
and
be
able
to
debug
things.
So
I
had
a
choice
at
this
point.
A
A
And
there
were
a
lot
of
reasons
why
I
didn't
want
to
do
this.
The
storage
library
essentially
winds
up
being
this
place,
where
we
run
into
all
sorts
of
things
that
aren't
super
easy
to
do
with
rust.
We
have
self-referential
data
structures
and
managing
memory
allocations
to
avoid
external
fragmentation.
We've
got
many.
Many
linked
lists,
pointers
all
over
the
place
unsafe
code
everywhere.
A
The
other
hand-
maybe
we
could
simplify
things
a
little
bit.
Maybe
if
we
start
with
something,
that's
only
single
threaded
that
we
don't
need
to
think
about
concurrent
access
that
might
make
it
a
little
easier.
Maybe
we
could
just
implement
the
hash
table
in
rust
and
use
c
code
to
allocate
and
manage
these
big
blocks
of
memory.
A
Luckily,
around
this
time
there
was
a
newer
resource
research
storage
design
by
a
phd
student
who
was
working
with
us,
and
this
new
design
was
a
lot
easier
to
port.
To
rust,
I
even
simplified
that
design
further
and
took
it
back
to
being
designed
for
single
threaded
use.
So
there's
no
locking
or
concurrency
things
that
I
might
get
wrong
rust
does
make
these
things
pretty
easy
normally,
but
for
memory
efficiency
reasons
I
wouldn't
have
been
able
to
use
the
rust
primitives
for
doing,
locking
and
concurrency.
A
But
best
of
all,
once
I
went
down
this
path
I
found
I
was
able
to
manage
all
the
memory
from
rust
and
not
have
to
deal
with
anything
terribly
exotic
like
a
custom,
allocator
or
anything
to
make
this
work.
So
it
was
actually
pretty
straightforward.
Once
I
started
with
a
different
style
of
storage
library,.
A
A
In
terms
of
benefits,
though,
this
kind
of
caused
us
to
move
forward
to
using
this
newer
storage
design,
which
has
benefits
for
a
lot
of
our
workloads
at
twitter,
otherwise
we
still
would
have
been
on
that
older
storage
library
and
not
even
been
able
to
start
tapping
into
some
of
these
benefits
of
the
new
design.
Yet
so
this
kind
of
forced
us
to
move
forward
a
little
bit.
A
And,
lastly,
it
helped
make
it
more
production
ready.
It
was
pretty
much
a
research
reference,
implementation
of
a
storage
library
and
having
another
software
engineer,
come
through
and
take
that
code
and
really
dig
into
it
helped
address
some
of
the
things
that
might
have
caused
us
some
pain
a
little
bit
later.
So
it's
always
good
to
evaluate
this
kind
of
stuff
and
see.
If
there
are
tweaks,
you
can
make
to
improve
things
further
or
areas
where
you
can
clean
up
and
have
a
second
set
of
eyes
really
deep
into
the
code.
A
A
A
A
Additionally,
I
used
cargo
fuzz
a
lot
for
the
protocol
library.
I
wanted
to
make
sure
that,
through
using
fuzz
testing,
I
was
able
to
find
some
edge
cases
that
I
hadn't
quite
considered
in
terms
of
malformed
requests.
So
that's
one
of
the
really
great
things
about
fuzz
testing.
Is
you
don't
need
to
think
of
all
these
edge
cases
yourself?
A
You
can
have
the
computer
start
to
discover
them
for
you
and
when
you
find
something
that
breaks
your
code,
you
can
add
a
new
unit
test
for
it,
and
this
really
helped
me
feel
like
the
protocol
was
going
to
be
rock
solid
and
that
I
wasn't
going
to
have
any
weird
like
crashes,
in
production,
for
having
like
array
indexes,
be
flipped
around
or
all
sorts
of
other
things
that
I
encountered.
While
I
was
fuzz
testing.
So
these
two
things
helped
me
feel
a
lot
better
about
the
code
that
I
was
writing.
A
A
But
really
the
code
between
the
ping,
server
and
said
cache
are
very
similar,
except
they
differ
in
terms
of
what
storage
engine
they
use
and
what
protocol
they
use.
So
here
we're
showing
the
composability
of
these
things
and
how
we're
able
to
neatly
express
that
with
traits
and
generics.
So
that's
I.
I
thought
this
was
pretty
neat
at
the
end
and
when
we
write
a
new
storage
library
or
additional
protocols,
they'll
be
just
as
easy
to
compose
together.
A
Some
more
testing
to
show
off
the
end
results
here,
I'm
comparing
upstream
memcache
in
blue
with
the
rust
implementation
of
pelican,
seg
cache
in
red
and
I'm
going
to
zoom
in
on
the
end
of
this
graph.
Here
at
the
higher
throughputs,
we
can
see
that
upstream
memcache
actually
has
higher
latency
than
this
rust
implementation
of
a
cache.
This
is
super
cool,
because
memcache
is
really
really
fast,
so
it's
neat
to
see
that
we
can
actually
meet
or
exceed
that
performance
with
a
rust
implementation.
A
A
A
There's
future
work
to
do
earlier.
When
comparing
with
upstream
memcache,
I
was
showing
single
threaded
performance,
but
this
implementation
also
has
a
multi-threaded
configuration
and
so
does
upstream
memcache,
and
while
this
rust
version
of
segcash
has
better
performance
than
our
internal
memcache
fork,
especially
when
tls
is
enabled
it
loses
out
to
upstream
memcache
in
multi-threaded
configurations,
I
think
we
can
close
this
gap.
The
performance
is
good
enough
for
us
to
deploy
to
production,
but
I
still
want
to
win
some
benchmarks.
A
A
A
There
were
benefits,
though
the
code
was
easier
to
work
with
once
it
was
all
rust.
I
didn't
have
to
fight
with
cmake
and
cargo
together,
and
some
new
ideas
got
added
to
the
code.
So
that's
probably
a
win
overall
c
and
rust
are
both
very
fast
profiling
and
benchmarking
helped
me
get
the
performance
to
match
that
of
the
c
implementation,
but
it
did
take
work.
A
The
rust
ecosystem
has
all
these
awesome
tools
that
I
mentioned
before.
We
have
cargo.
We
have
fuzzers
and
benchmarks
and
things
that
help
me
feel
that
the
code
is
solid
and
gonna
perform
well,
and
then
we
have
tools
like
rust
format
and
clippy,
which
helped
me
make
sure
that
the
formatting
and
that
the
code
was
idiomatic
and
that
went
a
long
way
to
making
the
code
easier
to
work
with
and
for
my
manager
to
review,
and
I
think
pelican
has
a
really
exciting
future
with
rust.