►
From YouTube: RustFest Zürich 2017 - Antimony: a real time distributed stream processing system by M. Makhlouf
Description
A real-time stream processing and distributed computation system written completely in Rust designed to out-storm storm and is heavily influenced by the Twitter Heron paper. Antimony addresses all the architectural and efficiency issues of storm just like Heron does however implemented in a much safer language and is not bound to be backward compatible with storm. we will explore the implementation of a cross-platform high performance / lightweight stream processing and distributed computation system in Rust backed by the full power of Tokio.rs
About Mohammed Makhlouf:
Mohammed Makhlouf is a experienced Software / Security Engineer, with a passionate interest in machine learning, security and large scale distributed systems. He is the Threat Intelligence Manager at the National CERT of Qatar and is based in Qatar. He infrequently blogs at: http://blog.mak.my
A
A
So
at
first
I
will
be
rambling
about
some
relevant
and
irrelevant
history
to
why
we
use
a
real-time
streaming.
Why
do
we
need
real-time
streaming
and
our
first
attempt
at
doing
real-time
stream
processing
using
Apache
storm
and
why
we
came
to
write,
antimony
and
I
will
give
some
code
example
and,
in
the
end,
I
will
I.
A
A
Others
are
mega
corporations
that
have
10,000
nodes
and
above
and
we
process
around
10
terabytes
of
data
of
DNS
logs
every
day,
and
we
we
are
mostly
interested
in
DNS
logs,
not
the
entire
network
stream,
because
the
DNS
log
itself
contains
a
lot
of
interesting
information
that
you
can
use
to
infer
a
lot
of
activity
that
is
going
on
into
the
network.
So
if
you
take
a
look
at
a
sample,
DNS
log
entry,
it
contains
a
lot
of
interesting
stuff
that
you
could
use
to
infer
what
is
going
on
inside
a
particular
network.
A
So,
for
instance,
if
you
you
can
weed
out
or
separate
fqd
ends
that
contain
certain
keywords:
you
could
catch
and
detect
very
advanced
malware
stuff
that
could
be
that
was
used
by
the
carbon
act.
Cyber
gang,
who
allegedly
stole
around
500
million
to
1
billion
dollars
and
their
apt
malware,
was
simply
communicating
with
domains
like
this.
That
contains
certain
keywords
like
update,
Java
and
Adobe
AIR.
A
Malware
analysis
of
a
huge
amount
of
malware
and
DNS
is
one
of
the
favorite
channels
that
malware
writers
used
to
do
command
and
control,
and
also
to
exfiltrate
data
through
the
DNS.
So
as
it's
a
great
tool
for
the
attackers,
it's
also
a
great
tool
for
defenders.
If
you
want
to
have
a
kill
zone
to
stop
this
traffic,
if
you
do
the
DNS
log
analysis,
you
can
infer
a
lot
of
things,
that's
going
on
in
a
network
and
you
could
stop
part
of
bad
malicious
activities
within
the
network.
A
A
That
comes
at
various
speeds
and
do
some
analysis
and
we
we
we
were
coming
from
different
programming
backgrounds
and
we
all
agreed
that
Python
is
a
good
middle
ground
for
everyone
for
the
programmers
at
the
analyst
and
we
wanted
to
use
Apache
storm
because
we
thought
that's
a
good
fit
and
we
came
across
something
called
pilots
from
Yelp
that
allows
you
to
write
stream
processing,
real-time
jobs
in
in
Python,
and
you
don't
have
to
fiddle
around
with
anything
related
to
JVM,
and
so
that
was
a
good
choice
for
everyone.
Programmers
and
the
analysts.
A
It's
basically
a
MapReduce
like
of
a
job
but
where
the
input
is
a
continuously
incoming
stream
of
data
tuples
that
never
answers
an
ever
going
process
that
continually
running,
and
you
can
represent
this
computation
in
the
form
of
a
directed
acyclic
graph,
that
the
stream
of
data
is
incoming
and
you
want
to
direct
the
stream
into
a
series
of
computations
without
having
to
worry
about
how
to
do
the
whole
distribution
of
this.
You
don't
have
to
write
complex
networking
layer,
stuff.
A
So
storm
is
great
for
that
and
the
this
directly,
the
cyclic
graph
in
the
terminology
of
apache
storm.
It's
called
a
topology
and
it
contains
nodes
of
two
kinds.
One
kind
is
called
the
spout
and
the
spout
is
a
source
of
data.
It's
a
a
node
that
is
the
reading
data
in
a
continuous
fashion,
so
it
could
be
reading
data
from
a
queue
like
Kafka
or
Anne
Askew,
or
even
grabbing
data
from
network
or
from
a
MySQL
database,
and
we
have
a
bolt.
Both
is
a
the
node
that
is
actually
doing
the
computation.
A
So
if
you
are
computing,
some
value
you're
performing
some
arbitrary
function
or
joins
or
these
type
of
operations.
You
write
this
in
the
form
of
a
bolt,
and
so,
if
we
consider
a
very
simple
topology,
linear
topology
like
that
that
have
a
spout,
that
is
a
low
entry
spout
that
is
grabbing
the
log
entries
from
somewhere
from
a
queue
or
from
a
network
service,
and
it
is
forwarding
this
incoming
data
tupple
to
the
extract
fqdn
bolt
to
extract
the
fqdn
out
of
the
log
entry.
A
And
it's
also
known
as
the
logical
plan
you,
you
simply
represent
your
computation
in
the
form
of
a
topology
and
you
submit
it
to
storm
and
storm
or
handle
how
to
distribute
that
over
a
set
up
of
of
nodes,
cluster
of
nodes
and
four.
So
in
this
example,
if
you
have
this
topology
you
you
want
the
look
to
come
in
and
you
want
to
parse.
So
you
don't
care
which
particular
bolt
is
going
to
parse
they're,
all
equal
you're,
just
extracting
the
fqdn.
So
you
can
shuffle
group
any.
A
You
have
multiple
instances
of
the
spouts
and
it
pushes
this
couple
of
data
into
a
extraction
bolt
that
will
extract
the
fqdn
out
of
the
log
and
then,
when
you
are
passing
it
on
to
the
count
fqdn
you
care,
where
this
exact
that
QD
n
go
to,
which
exact
bolt
to
maintain
the
camp.
So
you
can
filter
by
the
field
of
outgoing
stream.
A
So
what
is
the
motivation
to
rewrite
the
entire
data
stream
processing?
Well,
first,
we
needed
an
excuse
to
write
more
rust
so
and
we
thought
that
the
way
we
were
running
storm
was
in
the
most
efficient
way,
because
we
are
using
PI
layers
which
takes
the
entire
Python
interpreter,
put
it
inside
of
a
jar
and
then
take
this
jar
and
distribute
all
that
onto
the
notes.
A
A
Was
it
first,
the
Nimbus
as
the
master
node
is
very
overloaded,
is
a
it's
responsible
for
scheduling
for
coordination
and
monitoring,
and
it
is
not
resource
aware.
So,
if
you
ask
the
Nimbus
to
make
four
hundred
instances
of
a
certain
process,
it
will
just
do
that,
regardless
of
the
resources
that
are
available
in
your
cluster.
So
it's
not
resource
aware,
and
it
is
also
a
single
point
of
failure.
So
if
you,
if
the
Nimbus
node,
fails
your
entire
cluster
of
storm,
all
the
topologies
that
were
running
there
are
gone.
A
Another
thing
was
the
with
storm
is
that
it
has
a
quite
of
a
complex
execution
model
where
it,
the
storm
worker
itself,
is
just
a
JVM
process
and
it
inside
the
JVM
process.
It
have
multiple
executors,
which
are
essentially
a
pair
of
threads,
and
this
pair
of
threads
are
responsible
for
taking
your
task,
your
code,
logic
that
that
you
want
to
execute
to
execute
them
within
the
JVM
process.
A
There
is
scheduled
with
this
multiple
layers
of
multiplexing,
so
inside
of
a
storm
storm
worker,
the
this
is
a
JVM
process
and
it
has
two
main
two
threads
per
worker.
One
is
called
the
work
to
receive
thread
and
the
other
is
called
the
workers
and
thread,
and
they
pick
up
data
data
tuples
from
the
from
the
network
and
pass
it
on
to
the
to
the
executor.
A
If
you,
when
you
have
a
strong
cluster,
and
it
fails-
but
it's
very
difficult
to
know
where
to
even
start
with
that
and
another
problem
that
we
started
to
to
face
also
is
the
larger
the
the
the
storm
cluster
we
have.
The
more
overloaded
zookeeper
started
to
become
and
it
becomes
very
difficult
to
debug,
because
storm
uses
zookeeper
for
heartbeats
to
maintain
the
the
state
of
the
of
the
of
the
stream
and
and
that
led
to
the
zookeeper
to
becoming
overloaded.
And
if
you
are
using
a
single
zookeeper
cluster
for
other
services
as
well.
A
It
becomes
a
big
mess
to
to
debug
that
so
some
homegrown
wisdom
that
started
to
occur
in
our
team
is
that
if
you
want
to
write
the
topology
just
simply
separate
write,
deploy
a
cluster
of
storm
workers
and
keep
each
topology
at
a
separate
cluster
of
storm,
which
is
terrible,
very
terrible
idea.
And
you
end
up
over
provisioning
for
everything.
And
then
two
things
happened.
A
We
thought
maybe
we
can
rewrite
apache
storm
or
find
a
better
alternative,
and
at
that
time
twitter
published
a
a
research
paper
called
Twitter
heron,
and
we
saw
that
from
the
error
messages
that
we
were
getting
from
stone.
We
learned
that
strong
is
using
Metis
for
I,
think
networking
for
a
synchronous
networking
and
at
the
time
we
watched
the
mile
talk
in
rust
camp,
the
first
row
scam,
who
said
well.
If
rust
have
a
synchronous
networking,
maybe
we
could
rewrite
a
stream
processing
engine
and
rust.
A
So
this
is
the
paper
that
Twitter
published
Heron,
which
addresses
all
the
problems
that
storm
has,
and
we
thought
we
will
use
this
as
a
guiding
principle
into
designing
our
system.
So,
first
of
all,
Heron
does
not
have
a
master
node,
it
doesn't
use.
Nimbus
doesn't
have
a
master
node
that
is
responsible
for
scheduling
it
uses
existing
schedulers
stuff
like
Apache,
misses
or
Marathon
or
aurora,
and
each
you
submit
your
topology
to
the
scheduler
and
each
topology
have
its
own
topology
master.
A
So
the
topology
master
is
a
component
inherent
that
is
responsible
for
translating
your
logical
topology
plane
into
your
physical
plane
and
also
is
responsible
for
maintaining
the
state
and
is
not
it
does
not
handle
heartbeats
or
maintaining
the
the
flow
of
the
data.
There's
a
separate
process
for
that
called
the
stream
manager
and
the
unit
of
abstraction
for
execution
here
is,
is
a
container
not
a
JVM
process.
So
it's
much
easier
to
deploy
on
on
a
larger
scale
network.
A
So
zookeeper
is
the
one
that
sorry,
the
topology
master
is
the
product
that
is
responsible
for
communicating
with
the
existing
scheduler.
That
is
a
resource
aware,
so
it
you
realize,
on
Aurora
or
marathon
to
to
allocate
resources
to
your
topology
and
the
stream
manager
is
the
process
that
is
responsible
for
the
routing
of
the
tuples
and
also
up
backpressure.
A
So
if
we
take
an
example
here
again,
our
same
topology
does
the
extraction
and
when
you
deploy
that
it
becomes
the
physical
plan
when
you
deploy
the
topology,
the
linear
topology,
it
turns
into
the
physical
plan
that
actually
distributes
your
code
on
multiple
containers
and
each
container
will
contain
something
called
the
stream
manager
and
the
stream
managers
of
all
the
containers
in
your
topology
will
form
a
fully
connected
Network.
So
it
will
connect
all
all
of
your
containers
with
if
the
routing
requires
to
go
across
containers.
A
There's
a
TCP
stream
between
the
two
containers
and
if
it's
within
communication
needs
to
happen
between
two
instances
or
two
processes.
Inside
of
the
same
container,
it
was
the
result
to
UNIX
domain
sockets,
and
so
we,
the
whole
purpose
of
writing.
This
was
that
we
wanted
to
move
away
from
our
inefficient
Python
implementation
on
top
of
storm,
and
we
wanted
to
write
that
in
in
rust.
A
So
in
antimony,
you
can
define
a
topology
in
the
form
of
a
JSON
file
that
explains
the
the
components
and
how
it
flows,
how
the
data
flows
from
one
component,
the
other
in
the
form
of
a
JSON
file,
and
you
implement
your
code
logic
of
the
spouts
and
the
bolt
that
you
want
in
the
following.
Folder
structure
like
that,
and
you
simply
implement
the
spout
on
the
components
and
you
can
write
the
code
in
rust.
So
you
can
have
as
spouts
and
you
simply.
A
This
is
a
sample
spout
that
just
emits
continuous
emission
of
log
entries
and
the
bolt
would
do
the
actual
and
computation
on
the
emitted
and
the
emitted
Topo.
And
then
there
is
a
CLI
tool
which
you
run
inside
of
your
topology.
That
would
submit
this
to
the
topology
master
and
would
communicate
with
the
scheduler
to
schedule
to
allocate
resources
for
your
at
the
components
of
your
topology.
A
And
so,
regarding
the
the
name,
we
wanted
to
name
the
project,
some
metallic
kind
of
name,
because
it
seems
that
if
you
go
to
crates
dot,
IO
and
you've
tried
to
find
the
rust
crates.
You'll
find
that
most
of
them
are
metals
and
you've
noticed
that
all
the
metals
and
the
periodic
table
are
already
taking
on
entrees
at
I/o.
So
we
I
mean
there
is
iron,
cobalt,
nickel,
titanium,
aluminium,
even
stuff.
That
cannot
really
rust.
Like
cobalt,
doesn't
realize
a
super
alloy
nickel
also
doesn't
rust
so
anyways.
We
we
couldn't.
A
We
had
to
settle
on
something
that
is
close
enough,
so
we
picked
metalloid
and
antimony
is
kind
of
sort
of
a
semi
metal
type
of
thing,
and
so
we
thought,
okay,
not
close
enough
and
that's
how
it
looks
like
in
reality,
and
the
other
fact
is
that
ancient
Egyptians
used
to
use
this
as
mascara.
So
thank
you.
A
The
the
the
throughput
is
about
two
to
three
times
better
than
our
apache
storm,
but
what
we
really
made
progress
on
as
the
resource
allocation,
because
our
way
of
running
storm
was
to
do
KVM
virtualization
for
building
the
clusters
and
to
move
away
to
something
that
is
using
messes.
We
saved
over
10x
in
resource
usage
so
because,
because
we
started
with
a
not
a
very
wise
way
of
deploying
storm.