►
Description
Materials for this talk are available at http://teapotahedron.com/littlelangs.pdf
Languages are an underrated tool for solving engineering problems, in part because creating them has been difficult. Rust's unique combination of features make it an excellent language for writing compilers ranging from the Rust compiler itself to small domain specific languages. This talk will describe the implementation of a compiler for a query language for protocol buffers and how much easier it is to write one in Rust compared to C.
http://www.rust-belt-rust.com
A
Hi
I'm
here
to
talk
about
building
little
languages.
So
first
obvious
question
is
what
is
a
little
language?
Well,
it's
not
a
big
language
like
rust.
It's
usually
something
small
and
simple
for
some
specific
problem
where
it
is
designed
just
to
solve
that
problem.
Really
well,
hopefully
not
turing-complete.
You
know
often
used
in
other
languages,
often
interpreted
or
somehow
otherwise,
not
compiled
to
machine
code.
You
know
your
configuration
files,
hopefully
not
turing-complete,
hopefully
not
like
sent
mail
CSS
as
another
example.
You
know
relatively
simple,
often
in
combination
with
other
languages.
A
Your
query
languages,
often
little
languages,
though
SQL
gets
pretty
big
templating
languages,
and
you
know
there's
kind
of
a
spectrum
between
big
and
little
so
my
particular
little
language
that
I
built
was
motivated
by
the
specific
example
of
transit
data.
So,
in
addition
to
being
a
rust
enthusiast
time
at
transit,
enthusiasts
and
I
like
to
know
all
sorts
of
obscure
facts
about
the
transit
system
in
my
hometown
of
Boston
and
fortunately,
the
transit
agency
in
Boston
publishes
all
sorts
of
data
in
open
formats,
including
real-time
locations
for
all
the
buses
and
trains.
A
So
you
can
see
where
everything
is
and
they
published
this
in
protocol
buffer
format.
You
just
download
it
from
a
URL.
It
gets
updated
every
30
seconds
and,
of
course,
I
have
a
script
set
up
on
my
little
rented
server
out
on
the
Internet's
to
go
and
download
this
data
every
30
seconds
and
keep
a
giant
archive,
so
I
can
go
back
and
see
how
things
were
working
during
the
terrible
snowstorm.
Well
in
theory,
I
still
have
to
actually
have
some
tools
for
dealing
with
this
data
and
part
of
the
problem.
A
Is
that
it's
a
lot
of
data
you
can't
just
you
know
you
can
just
stick
it
in
a
database,
but
it's
like
200
megabytes
a
day.
A
year's
worth
starts
to
add
up
for
my
cheap,
rented
server
on
the
Internet,
so
I
just
store
it.
The
cheap
way,
which
is
concatenated
and
exept
compress,
is
really
well,
but
there's
not
really
a
good
way
to
search
through
it.
So
I
figure
I'll
just
build
my
own
little
search
tool
and
the
way
to
do
that
is
the
build
a
little
query
language
that
lets
me
find
out.
A
For
example,
where
is
boss?
Five
thousand
one
and
protocol
buffers
in
case
you
don't
know,
they're
like
a
binary
format.
They're
the
data
model
was
kind
of
like
your
JSON
or
whatever.
It
has
structures,
it
has
arrays,
it
has
fields
and
messages,
and
so
there's
other
similar
formats
with
other
similar
query.
Language
XPath
was
kind
of
an
inspiration
in
some
way
and
there's
just
you
know
a
basic
example
of
what
it
looks
like
and
so
getting
a
little
bit
more
into
what
is
behind
the
language.
A
There's
sort
of
different
parts
that
you
have
to
go
through.
My
language
has
a
parser.
Every
language
has
to
have
a
parse
or
more
or
less,
which
reads
these
expressions
and
then
produces
a
parse
tree
and
then
because
protocol
buffers
actually
have
a
pretty
well-defined
schema
and
it
tells
you
what
type
all
the
parts
of
the
message
are
supposed
to
be,
and
so
on.
A
There's
actually
a
type
checker,
very,
very
simple
type
checker,
but
it's
there
nonetheless,
and
then
the
actual
evaluator
that
actually
takes
that
expression
and
runs
it
against
my
giant
archive
of
a
year's
worth
of
bus
data
and
spits
out
the
little
subset
that
talks
about.
Just
that.
One
bus
that
I'm
interested
in
because
it's
the
shiny
new
hydrogen
fuel
cell
bus
that
we've
been
waiting
for
for
a
year.
And
we
want
to
know
where
it
is.
A
A
You
know
you
just
take
it
just
reads:
through
the
string
and
either
successfully
matches
it
and
returns
some
successful
result
and
the
remainder
of
the
string
that
it
didn't
parse
or
it
you
know,
fails
the
parse.
It
returns
an
error
and
then
maybe
you'll
get
a
parse
error
and
the
parser
reads
this
expression.
You
know
the
string
that
in
the
earlier
slide,
returns
a
parse
tree
and
that
gets
fed
into
the
next
section,
which
is
the
type
checker.
A
So
the
parse
tree
is
literally
just
like
a
struct
tree
of
strings
and
more
or
less
and
now
we
have
to
make
sure
that
all
the
actual
words
in
there,
like
the
you,
know,
entity,
dot
vehicle
that
you
saw
earlier,
that
there
are
actual
messages
that
are
defined
for
this,
and
so
these
definitions
are
in
this
description
file
that
Google
ships
and
fortunately
Google
ships
a
whole
bunch
of
tools
to
deal
with
protocol
buffers.
Unfortunately,
none
of
them
are
really
designed
with
rust.
A
A
They
produce
a
C
file
from
this
description
of
what
this
transit
data
format
is
supposed
to
be
like
that
I
compile
that
into
an
Esso,
shared
library
and
then
just
load
that
from
rust
and
have
a
little
FFI
wrapper
around
it.
That
lets
me
use
this
C
code
generated
by
this
Google
tool
directly
from
rust,
and
that
lets
me
get
at.
You
know
what
you
know:
what
are
the
valid
field
names?
What
are
the
data
types
for?
A
That
rust
is
pretty
good
at,
which
is
the
iterators,
which
are
a
wonderful,
wonderful
feature
and
make
it
you
know,
make
it
very
easy
to
sort
of
build,
build
things
on
top
of
and
the
way
this
evaluator
works.
Is
it
basically
I
built
a
little
iterator
type
that
will
take
some
generic
input,
actually
something
that
implements
a
specific
type
of
trade
and
then
basically
return
successive
protocol.
A
A
Constructs
this
it
iterator
and
then
it
just
filters
and
C
C's
does
this,
you
know.
Does
this
message
match
the
tag?
Does
it
match
the
filter
expression
and
then,
if
it
is
well
call
the
callback
and
if
and
if
it
doesn't,
and
we
still
have
more
sort
of
messages
more
expressions
to
check,
then
you
know
recur
and
go
deeper
and
check
if
it
continues
to
match,
and
this
is
actually
so.
The
actual
some
of
the
backstory
of
this
is
that
I
actually
originally
started.
A
Writing
this
in
C
and
I
desperately
wanted
to
write
it
and
rust,
but
unfortunately,
Russ
didn't
run
or
the
standard
compiler
didn't
have
a
version
from
my
computer,
which
is
this
lovely
little
arm
thing
and
I
started
writing
it
and
see
they
actually
released
the
arm
build
an
official
arm,
build
of
the
compiler
like
two
days
after
I.
Did
that
and
then
I
kept
going
with
C
regardless
I
got
very,
very
frustrated.
The
code
got
very
big
and
confusing,
and
then
eventually
I
got
so
frustrated.
A
So
I
guess
the
things
that
I
learned
from
all
of
that
rust
is
great
for
writing.
Compilers,
not
really
surprising.
I
mean
the
first
really
big
program.
Written
in
rust
was
the
Rost
compiler,
so
that
was
sort
of
the
guiding
force
behind
a
lot
of
the
way
the
language
was
designed,
and
so
it
was
designed
pretty
well
for
writing.
Compilers.
A
A
Rusts
FFI
makes
it
really
easy
to
reuse
C
code.
It
would
have
been
a
lot
harder
to
actually
get
all
of
these
descriptions
of
like
what
the
messages
are
supposed
to
be
like
and
actually
basically
write
my
own
compiler
for
that
format
as
well
and
get
all
the
data
out
of
the
sort
of
published
standard
or
have
to
basically
code
it
up
by
hand,
and
that
would
only
work
for
transit
data,
but
thanks
to
relatively
easy
FFI
I
could
just
reuse
all
of
that
stuff
and
wrap
it
in
Nice.
A
Safe
interfaces
and
rust
is
great
for
building
composable
abstractions,
so
the
iterators
were
sort
of
very,
very
efficient
and
made
it
very
easy
to
write
this
sort
of
going
back
going
back
to
the
code
filter.
The
filter,
you
know,
is
a
trait
and
you
just
write
some
rated
implementation
for
it,
and
you
have
some.
You
know.
A
That's
eval
function
that
you
can
implement
for
different
types
of
trait
of
different
types
of
filters
makes
it
very
easy
to
write
this
particular
code,
and
then
you
don't
have
a
giant
like
switch
statement
like
you
would
if
you
were
doing
C,
so
that
makes
it
more
compact
and
yeah.
Rust
is
generally
great.
A
B
After
you
had
an
initial
I'm,
assuming
you
wanting
to
be
able
to
see
things
do
things
you
had
an
initial
foundation
that
you
have
expanded
on
added
features
to
after
you
had
the
general
framework
in
place
during
that
expansion.
Were
there
any
points
in
which
you
wanted
to
do
something
that
rust
wasn't
giving
you
a.
A
Very
good
question,
so
one
of
I
mean
there
were
ultimately
I
managed
to
successfully
battle
rust
and
get
what
I
wanted,
but
there
were
definitely
moments
of
like
difficulty
and
like
battling
with
the
borrow
checker
and
sort
of
transitioning
it
from
Oh
will
parse
this
message
out
of
a
buffer.
Oh,
let's
build
a
streaming
interface
and
I
have
all
sorts
of
clever
ideas
for
how
to
do
that
and
then
sort
of
fighting
the
system
and
fighting
the
borrowed
checker
when
it
wouldn't.