►
Description
Tyler Neely can help us to test
if our code is really the best
he is sharing techniques
to test with some tweaks
so our biases can lay to rest
After years of getting whooped by bugs at scale at various North American infrastructure companies, Tyler moved to Berlin where he spends his days reading papers, finding bugs in complex systems, and building Sled - a modern embedded database for the Rust ecosystem. He believes on-call people should sleep as long as they want. The wild success of testing tools like Jepsen is a wake-up call that we’re approaching systems engineering from a fundamentally bug-prone perspective. Why don’t we find these devastating bugs on our laptops before opening pull requests? Rust’s compiler gives us wonderful guarantees about memory safety, but as soon as we open files or sockets, all hell seems to break loose. This talk will show you how to apply techniques from the distributed systems and database worlds in a way that maximizes the number of bugs found per cpu cycle, and reduce the amount of bias that we hardcode into our tests.
(Limerick by @llogiq)
https://paris.rustfest.eu/sessions/building-reliable-infrastructure-in-rust
B
So
another
name
for
this
clock
could
be
the
clear
expression
of
beliefs
or
to
so
Who
am
I
I
kind
of
started
as
like
an
SRE
facing
a
lot
of
bugs
and
production
at
social
media
companies.
And
then
it
worked
for
like
about
infrastructure
companies
and
then,
like
serverless
company,
seemed
like
a
bunch
of
like
weird
infrastructure
things.
Then
I
got
super
burn
it
out
and
I
moved
to
Berlin
and
just
got
lazy
and
to
start
working
on
personal
projects,
kind
of
culminating
and
it
projects
all
the
swed.
There's
like
a
database,
construction
kit
and.
A
B
Testing
things,
so
it's
like
fill
the
database
in
rust.
It's
like
a
collection
of
modular
components
like
a
page
cache.
They
can
just
take
and
basically
build
fairly
high
performance
things
with
very
little
effort
and
I
felt
like
a
pw
tree
and
I
got
the
housing
lines
of
rest
on
top
of
this.
So
even
those
really
complex
things
without
very
much
effort
on
top
of
the
page
cache
anyway,
today
I
worked
on
it
with
the
source
point
which
aims
to
better
fund
and
empower
free
software,
and
you
might
know
me
as
a
space
jams.
B
Anyway,
so
what
I'm
going
to?
Because
we
got
say
ourselves,
the
things
we
create
and
the
context
that
we
exist
in
and
also
mechanically,
confronting
our
assumptions
and
basically
trying
to
test
things
that
are
networks,
these
sockets
and
then
just
like
random
miscellaneous
tips
on
architecture.
Maybe
something
is
driven
fault
injection.
If
we
get
to
it,
it's
basically
ways
it
like
really
bringing
out
the
bugs
from
your
systems,
things
that
I'm
not
going
to
cover
so
reliable
infrastructure.
B
Many
people
need
things
like
metrics,
introspection,
time,
series,
databases,
it
has
to
be
planning,
and
these
things
are
crucial
for
actually
running
the
levels.
That's
where
it,
but
I
think
that
they're
covered
very
well,
particularly
like
to
direct
your
attention
to
duties
in
plugins
use,
one
by
Brendan
Gregg.
B
You
might
know,
because
there's
like
the
flame
graph
guy,
wonderful
book
systems,
performance
monitoring,
particularly
chapter
2
analogy-
is
outside
because
because
our
ebook
is
great
for
particularly
like
building
things
away,
that
is
introspective
oil
and
also
sort
of
like
the
social
aspects
of
running
teams
that
have
to
deal
with
reliable
infrastructure.
This
looks
great
for
it
and,
of
course,
learn
to
cut
my
design
and
data.
Intensive
application
is
like
really
good
for
work
and
like
understanding
material
databases
and
distributed
systems,
correctness
and
things
like
that.
Ok,
so,
let's.
B
B
They
could
be
useful
in
other
ways,
but
to
the
extent
that
we're
trying
to
make
reliable
care.
This
is
important
and
we
don't
do
cache
invalidation
when
we've
realized
that
things
are
wrong,
that
we
believe
and
also
we
kind
of
stacked
beliefs
on
top
of
each
other
in
ways
that
can
like
make
these
like
teetering
towers
of
beliefs
that,
like
you've,
ripped
out
a
Jenga
cube
and
like
you
would
you
would
hope
it
would
like.
B
You
know,
tumble
to
the
ground,
but
instead
of
just
days
floating
there
in
the
air,
and
we
continue
to
base
decisions
on
it.
Not
so
useful
anyway.
I'd
love
to
keep
talking
about
this
like
brain
stuff,
but
let's
kind
of
bridges
into
code.
If
you
want
to
learn
more
about
that,
I
recommend
checking
out
rationality
from
AI
to
zombies,
but
really
I
think
that
reliable
infrastructure
kind
of
boils
down
to
confronting
assumptions.
So
these
things
are
kind
of
nice
talk
about
Andres,
okay.
B
I
should
have
caught
this,
but,
like
my
tests,
just
didn't
cover
it.
It
turns
out
that,
like,
if
you're
just
writing
like
unit
tests,
even
if
you
have
like
100%
line
and
branch
coverage,
that's
only
one
dimension
of
the
entire
space
of
your
program.
There's
also
like
all
the
different
values
of
all
the
variables
throughout
it
that
are
just
totally
unaddressed.
B
Just
relying
on,
unlike
a
bunch
of
unit
tests,
makes
your
code
very
subject
to
was
called
the
pesticide
paradox,
where
your
code
becomes
immune
to
your
tests,
but
that
doesn't
actually
make
mean
that
it's
like
all
that
useful
for
other
things.
So
of
how
can
we
like?
What
should
we
do
if,
like
our
tests,
that
we
write
tend
not
to
be
like
as
effective
as
they
could
be,
don't
write
them?
Just
don't
write
tests
come
on.
Like
testing
is
boring,
I
know,
I,
don't
really
enjoy
writing
tests
manually,
but
we
can.
B
We
can
do
a
lot
better
by
writing
what
we
expect
and
then
mechanically
generating
the
tests,
and
we
can
basically
have
the
Machine
create
more
tests
than
we
would
have
like
many
lifetimes
to
write
by
hand,
and
if
we
can
do
this,
then
we
can
get
a
lot
of
mileage
out
of
things.
We
should
make
our
beliefs
explicit
and
the
beliefs
that
are
present
in
our
code
should
pay.
B
So
if
you
have
not
been
subjected
to
the
cruel
realities
of
modern
computing
machinery,
you
might
assume
that
this
could
be
correct,
like
multiplying
something
by
five
and
then
dividing
it
by
five.
You
might
expect
that,
to
you
know
equal
what
you
put
in
in
the
first
place,
people
who
have
not
been
so
lucky
or
like
subjected
to
the
cruel
whims
of
the
machine
might
guess.
What's
going
to
happen.
We've
run
cargo
tests.
B
Yeah
prop
test
pays
a
lot
of
attention
to
the
edge
cases,
but
also
really
like.
This
is
a
really
cool
thing,
because
it
also
basically
just
wrote
a
regression
test
for
us.
So
prop
tests
just
created
this
file
and
prop
test
regressions,
and
it
basically
shows
the
random
number
generator
seed
that
it
used
to
generate
the
specific
failing
input
for
our
test
and
anytime.
It
runs
a
new
test.
B
It
will
basically
go
through
all
of
the
things
in
this
regressions
file
and
like
basically
make
sure
that
it
tries
those
first
and
if
we
have
in
past
that
yet
it
will
just
keep
deterministically
replaying
it.
So
prop
test
is
nice
because
it
gives
us
so
not
some
non
determinism
in
checking
random
values,
but
also
we
get
deterministic
replay
of
our
bugs
that
it
does
find.
So
it's
a
really
wonderful
way
of
just
you
know,
coming
up
with
tests
for
us,
we
just
say
what
we
expect
and
then
it
does
the
rest.
B
You
can
basically
give
it
like
reg
X's,
and
it
will
generate
text
that
that
satisfies
them.
It's
a
it's
really
cool.
You
can
also
provide
your
own
generators,
for
instance,
and
in
bottles
crate
with
immutable
data
structures.
There's
a
whole
like
like
collection
of
interesting,
prop
test
macros.
So
if
you're
curious
about
looking
at
more
examples
of
practiced
check
out,
mr
s,
there's
tons
of
examples
there.
B
Also,
if
you
have
a
system
with
configuration
that
the
user
can
change,
you
better
expect
your
users
to
configure
your
system
in
ways
that,
like,
maybe
you
don't
expect
so
my
friend
Yash,
is
basically
doing
a
rest
port
of
the
DAT
protocol
and
this
is
a
library
called
spar
spit
field.
That's
a
part
of
it,
and
one
day
we
were
just
hanging
out
at
a
cafe
and
and
said.
Oh,
do
you
hear
about
the
prop
test
library
and
it's
like
yeah?
B
Let's
try
it
out
so
this
was
actually
the
the
first
thing
that
we
wrote
together
to
kind
of
play
with
prop
test
and
it
just
like
immediately
caused
the
code
to
blow
up
and
it's
like
really
not
so
much
testing
but,
like
you
know,
you
might
expect
users
to
configure
things
in
ways
that
we
didn't
expect.
Specifically,
what
this
is
doing
is
its
configuring,
this
library
with
a
particular
page
size,
and
then
it's
like
actually
using
one
of
the
methods
on
the
library.
B
So
it's
like
without
very
much
effort
put
into
testing
you
get
really
interesting
results.
So
how
do
we
apply
this
to
more
complex
things?
Allow
me
to
introduce
you
to
model-based
testing
model
that
this
is
like
a
really
powerful
technique.
Basically,
what
you
do
is
you
build
a
simplified
model
of
the
system
like
if
you're
building
a
database?
Maybe
you
can
just
use
a
hash
map
for
that,
and
basically,
the
point
of
the
model
is
just
to
encode
your
like
expectations
about
how
the
system
should
behave.
B
The
simpler
it
is
the
better
because
it's
also
like
a
cognitive
aid.
Once
you
have
that
simplified
model
of
the
system,
you'll
basically
use
a
library
to
generate
a
random
sequence
of
operations
that
apply
both
to
the
implementation
and
the
model
and
if
their
behavior
ever
diverges,
then
you
know
that
you
either
have
a
problem
with
the
model,
the
implementation
or
both.
B
We
found
a
problem,
and
you
know
if
you
build
something
like
this
on
top
of
prop
test
or
another
popular
library,
you
might
have
heard
of
called
quick
check.
You
can
basically
just
like.he
pre-running
this
long
sequence
of
operations,
which
it
might
initially
be
quite
large,
like
a
hundred
operations
and
who
wants
to
like
actually
like
figure
out,
which
particular
subsequence
of
those
operations
led
to
the
failure.
B
So
these
libraries
are
wonderful
because
they
will
randomly
pick
operations
to
drop
out
of
that
sequence,
rerun
the
test
and
if
it
continues
to
fail,
basically
keep
dropping
elements
out
of
that
operation
sequence.
So
it
basically
is
able
to
shrink
all
these
like
random
operations
into
the
specific
subsequence
that
triggers
your
failure
for
an
example,
let's
start
off
by
building
a
tree,
just
like
a
simple
tree
with
an
optional
root
and
that's
not
actually
implement
the
the
methods,
let's
just
like
have
them
do
nothing
on
our
tree.
B
So
this
is
a
little
library
that
I
wrote
to
make
model
based
test
fit
on
my
slides
for
this
talk,
but
it
turns
out
that
it's
actually
pretty
good
at
significantly
reducing
the
boilerplate
required
to
do
this
kind
of
testing.
So
we
have
this
tree
that
we're
building
as
implementation
and,
let's
just
use
like
a
beech
tree
map
as
our
model
and
basically
what
this
macro
says
is
we.
B
We
have
two
operations
that
we're
defining
set
and
get
and
along
with
their
associated
generators
and
we're
generating
a
key
and
a
value
for
sets
that
we
insert
into
both
the
implementation
and
the
model
and
then
when
we
try
to
get
a
randomly
generated
key
out
of
these
structures.
We
expect
the
values
to
be
the
same.
B
B
B
So
we
can
also
do
more
complex
things
like
imagine.
We
were
building
a
database.
You
could
do
like
a
restart
just
by
like
dropping
the
thing
having
a
closed
its
file
descriptors
and
then
open
it
again,
and
basically,
this
could
check
for,
like
things
like.
Is
the
data
still
in
the
database
that
I
expected
to
be
in
the
database?
B
B
And
if
you
can't
find
that,
then
it's
not
linearizable,
so
I
wrote
this
little
macro,
also
in
the
model
library
that
can
do
this
for
arbitrary
concurrent
data
structures.
In
this
case,
it's
just
like
using
an
atomic
you
size.
So
basically
imagine
like
multiple
threads
trying
to
do
this
buggy
ad,
so
atomic
you
size
is
interesting
because
you
can
basically
have
multiple
threads
change,
a
counter
or
or
rather
au
size
value
without
taking
out
of
mutex
or
anything,
and
it
relies
on
atomic
operations
of
our
CPUs
to
actually
guarantee
atomicity.
B
However,
if
two
threads
basically
get
to
the
line
where
we
load
the
value
like
let
current
equal,
I
load,
then
two
threads
could
basically
read
like
four,
for
instance,
and
if
one
of
those
threads
let's
say
that
both
of
those
threads
are
just
trying
to
add
one
to
it.
If
these
were
happening
at
atomically,
then
we
would
end
up
with
six
like
plus
one
plus
one.
B
But
these
are
some
really
great
papers
by
one
of
the
pioneers
in
model
testing,
John,
Hughes
and
he's
the
creator
of
quick
check
and
yeah
I
highly
recommend
checking
out
these
papers.
If
this
is
something
you're
interested
in
they're,
they're
quite
accessible
and
they
don't
use
a
lot
of
academic
jargon
cool.
So
what
about
things
that
actually
do
I/o
like
things
that
use
files,
things
that
use
sockets
well
like
we
can
do
fault
injection
for
that.
B
We
have
an
issue
where,
when
we
deploy
things
to
production
on
possibly
like
thousands
of
servers
or
even
just
give
things
out
to
a
bunch
of
users,
they
often
encounter
far
more
problems
with
the
issue.
Then
we
were
able
to
detect
when
we
were
doing
testing
on
our
laptops.
So
our
laptops
tend
to
be
these
like
serene
places
where
not
very
many
things
go
wrong,
and
basically,
when
we
build
things
that
work
well
on
our
laptops
that
actually
doesn't
guarantee
very
much
about
how
they're
going
to
behave
in
the
world.
B
B
Things
like
this,
where,
like
there's
like
and
in
almost
all
critical
bugs
that
bring
systems
down
like
we
begin
to
think
about
the
issue,
but
we
don't
really
finish
it.
Furthermore,
in
58
percent
of
those
cases
they
could
have
easily
been
like
this
could
have
easily
been
fixed
if
we
just
like
actually
trigger
those
failures
and
tests.
So
we
actually
don't
really
need
to
go
too
overboard.
With
these
things,
we
just
need
to
actually
exercise
our
testing
code
when
we
think
of
things
a
lot
of
code
ends
up
just
looking
like
this.
B
We're,
like
you
know,
if
there's
an
error
somewhere,
maybe
we
log
it.
Maybe
we
just
have
a
to
do,
or
maybe
we
bring
down
the
whole
system
and
I'm,
not
against
panicking
at
all,
like
I'm,
very
much
a
fan
of
like
the
Erlang,
let
it
crash
philosophy
and
when
you
can't
guarantee
safe
execution
of
a
system
of
crashing
is
usually
better
or
very
frequently
better
than
not
continuing
at
all.
B
Basically,
it
can
be
kind
of
a
pain
to
like
for
every
possible
piece
of
of
error,
handling
logic
to
basically
like
manually
trigger
that
and
the
test
that
we
write.
But
if
we
just
write
one
test
like
this
it'll
get
around
to
triggering
all
of
them
and
like,
depending
on
how
much
we're
able
to
basically
specify
our
expectations
about
general
failures.
This
can
dramatically
reduce
the
amount
of
effort
that
goes
into
it.
Here's
an
example
of
one
test
for
the
sled
project
and
I
would
have
never
written
this
test.
It's
basically
sets
a
key.
B
It
deletes
the
same
key.
It
triggers
one
failed
point,
and
it
just
goes
down
this
list
of
things.
I
would
have
just
never
written
this
test,
but
by
having
this
kind
of
infrastructure
that
just
like,
creates
tests.
For
me,
it
creates
bugs
that
are
like
maniacal
and
it's
also
deterministically
replayable.
So
it's
not
so
hard
to
debug
cool
and
to
learn
more
about
this
or
like
working
with
files.
B
It
will
replay
those
those
operations
in
ways
that
could
have
realistically
happened
on
different
common
file
systems
and
then
just
start
them
up
again
and
see
if
there's
any
any
strange
behavior
and
they
broke
like
every
database.
They
broke
like
sequel
Lite
which,
like
everyone,
thinks
of
as
like
this
incredibly
well
tested
database,
and
it
is,
but
even
sequel.
B
Lite
fell
to
this
incredible
testing
tool
so
check
out
that
paper,
if
you're
interested
in
going
deeper
on,
like
particularly
like,
like
spy
all
testing
cool,
what
about
like
network
sockets
and
just
things
that
end
up
talking
with
each
other
yeah
using
them,
so
one
option
is
Jepsen
which
you
might
have
heard
of,
and
this
is
another
really
cool
tool.
Basically,
it's
like
another
one
of
these
tools
that
I'm,
like
pretty
much
every
system
that
it
gets
turned
loose
against,
tends
to
fall.
B
It's
really
wonderful.
However,
it's
like
it
also
can
be
kind
of
expensive
to
get
set
up.
It
can
take
like
a
month.
It's
actually
like
implement
it,
a
nice
way
for
a
particular
system,
and
it's
like
it
ends
up
being
like
so
expensive
to
use
that,
like
many
companies
that
end
up
doing
it
end
up
writing
like
blog
posts
about
like
yeah.
We
finally
got
Jepsen
working
and
it's
like
it's
a
lot
of
work
and
also
it
takes
like
five
minutes
per
run
to
to
work.
B
Toa
plus
so
Toa
plus,
was
created
by
Leslie
Lamport
he's
the
person
who
created
like
latex
and
Paxos
and
vector
clocks,
so
he's
kind
of
like
the
patron
saint
of
distributed
systems
and
basically
what
this.
What
CoA
Plus
lets
you
do
is
describe
algorithms
and
also
invariants.
That
should
hold
for
different
kind
of
interleavings,
of
different
different
processes
that
are
running
through
these
algorithms
and
it
will
find
violations
of
those
invariants
and
it's
very
effective
at
doing
this.
B
It's
also
intended
as
like
a
learning
tool
where,
like
just
by
using
it
and
learning
how
to
like,
actually
write
Toa
Plus,
you
kind
of
like
dramatically
improve
your
ability
to
reason
about
correct
systems,
so
I
highly
recommend
learning
this
also.
It
has
like
a
wonderful
video
course.
Where
he's
like,
wearing
like
different
fuzzy,
hats
and
things
and
like
it's
worth
watching
just
for
the
hats,
but
the
content
is
also
incredible
and
also
a
Hallel
has
a
site
learn
Toa
comm,
which
is
wonderful
for
getting
into
it
as
well.
B
So
it's
a
really
cool
tool,
but
it
has
this
like.
It
is
another
language
and
then
there's
this
question
of
like
okay,
we
described
our
algorithm
in
this
language
and
then
we
have
this
implementation
and
like
like
there's,
is
there
a
gap
like?
Is
my
implementation
actually
doing
what
I
expect,
sometimes
that
that's
a
that's
a
challenge,
cool
and
add
the
option
that
I'm
going
to
go
farther
into?
B
Is
network
simulation
and
that's
where
we
just
kind
of
like
start
off
building
our
system
in
a
way
that
is
very
amenable
to
fast
testing
in
a
deterministic
way,
and
it
can
be
like
just
like
extremely
efficient,
like
yeah,
basically
can
run
like
thousands
of
tests
per
second,
it's
it's
incredible.
Also
like
one
issue
with
fault
injection
in
general
is
like.
Sometimes
it
can
be
like
a
question
about,
like
which
things
do
you
actually
break
like
you
know,
you
can
break
so
many
things.
So
how
do
you
focus
it?
B
Lineage
driven
fault
injection
I'll,
get
into
in
a
second.
Is
one
way
to
do
that
and
it's
very
it
works
very
well
with
something
that
you
build
from
the
beginning
to
be
to
basically
run
on
a
simulator.
It's
I've
been
building
if
like
distributed
systems
on
top
of
simulators
for
like
about
like
the
last
year
and
it's
I
just
have
to
say
that
like
I
have
never
caught
so
many
bugs
so
early
with,
like
so
much
ease
as
when
I
started.
B
Doing
this,
it's
something
that
kind
of
sounds
like
it
has
a
high
cost
initially,
but
once
you
try
it
you're
just
instantly
hooked
and
and
like
you
never
like,
you
never
feel
safe
writing
distributed
systems
code.
What
simulator
again
in
the
future,
you
just
find
so
many
bugs
that
you're
totally
like
you
would
have
never.
You
would
have
never
thought
to
check
for
it.
It's
incredible!
So
I'll,
just
kind
of
like
give
you
kind
of
like
a
skeleton
view
of
like
how
you
might
implement
a
simulator
one
way
to
do.
B
B
And
then
we
have
like
a
priority
queue
implemented
as
a
binary
heap
of
all
of
like
the
messages
that
are
in
flight
in
the
in
the
network,
as
well
as
like
the
the
responses
that
clients
get,
that
we
can
basically
perform
invariant
checks
on
after
the
fact,
so,
basically
for
every
message
in
the
in-flight
priority
queue
we
just
pop
it
off.
We
deliver
it
to
the
destination
at
the
time
that
we
decided
it
should
be.
B
Then
we
we
basically
deterministically
like
determine
how
long
it
should
run,
or
rather
how
long
it
will
take
to
get
to
its
destination.
Then
we
check
is
that
allowed
based
on
the
partitions
that
we
we
just
came
up
with
if
so
skip
it.
If
not,
we
basically
put
it
back
into
the
priority
queue
that
we
continue
to
iterate
over
and
then,
when
there's
nothing
left
in
the
priority
queue.
We
know
that,
basically,
our
our
cluster
is
quiet,
and
now
we
can
run
our
invariant
checks
on
the
client
responses.
B
You
start
with
the
very
last
thing,
so
imagine
like
a
chain
of
messages
like
you're,
like
passing
a
secret
long
from
one
person
to
another,
and
then
you
just
basically
pick
like
the
last
person
who
sent
it
and
just
like
nope
nope,
you
do
not
send
it
and
just
like
work
your
way
backwards
and
then
see
what
happens
so
Peter
Alvaro,
one
of
the
creators
of
lineage
driven
fault
injection,
has
a
nice
quote,
which
is
fault.
Tolerance
is
redundancy
and
space
and
time.
B
We
want
our
stuff
to
be
deterministic,
though,
if
it's
going
to
work
well
in
a
simulator.
So
for
that
we
need
to
control
our
clocks,
random
number
generators
and
also
our
like
the
way
that
our
threads
are
scheduled.
Basically,
you
can
control
threads
using
Linux,
real
time
priorities,
as
well
as
like
P,
trace
and
other
things.
B
That
was
written
since
the
last
thing
to
drop,
and
you
can
check
out
the
crate
deterministic
for
like
deterministic
versions
of
clocks
or
NGS,
and
also
using
linux,
real-time
as
priorities
and
I'm
working
on
the
file
thing
also
miscellaneous
things
generally
isolate
business
logic
from
I/o
concerns
and
control
flow
logic.
This
is
something
I
noticed
a
lot
of
people
who
are
writing
things
using
futures
and
asynchronous
code
kind
of
violating
the
idea
of
basically
like
we.
We
would
do
well
to
the
extent
that
we
set
like
like
separate
concerns.
B
We
can
take
advantage
of
all
the
performance
benefits
of
asynchronous
code
without
basically
mixing
over
it's
logic
into
it
and
and
there's
I
mean
this
is
not
true
in
every
case,
but
in
most
cases
it
is
and
I
think
this
is
something
that
we
should
strive
for
for.
Understandability
also
use
the
certs
and
debug
sorts
everywhere
when
basically
testing
our
our
code
and
random
ways.
Having
lots
of
these
asserts
will
basically
check
lots
of
assumptions.
B
Use
them
use
debug
assert
if
you're
concerned
about
performance
implications
because
it
gets
compiled
away
and
try
to
use
expect,
instead
of
unwrap,
to
give
the
poor
people
who
have
to
deal
with
the
panics
that
are
inevitable,
with
calling
unwrap
a
little
bit
of
information
to
help
them
when
it
happens,
and
also
when
you
propagate
errors,
try
to
include
context
so,
for
instance,
using
the
failure.
Crate
can
be
nice
for
this
cool
go
break
systems.
Thank
you.