►
From YouTube: A Glimpse of the New Ceph Messenger Built on Seastar - Yingxin Cheng & Vivian Zhu, Intel
Description
A Glimpse of the New Ceph Messenger Built on Seastar - Yingxin Cheng & Vivian Zhu, Intel
Seastar framework brings a new abstraction level over event-driven programming. It is by nature shared-nothing and non-block, hiding asynchronous functions behind its futures and continuations. That’s where the Crimson Project initiated, to liberate Ceph code from complex callbacks and rethink the design of performance from bottom up.
As the base of all other Ceph components which dispatch messages and connected as a cluster, Messenger is very performance-sensitive and also self-contained — a perfect start point for the redesign and validation works in Crimson. Although the new messenger is still under construction, facing challenges with the new programing paradigm and to be really efficient, still plenty can be shared and discussed, including Yingxin's gotchas during his implementations and Jianpeng's performance tests against the existing async-messenger.
About Vivian Zhu
Intel Corporation
Software Engineering Manager
Vivian Zhu is an engineering manager with Intel’s System Software Product Group. She is managing the engineering team focusing on storage-related technologies development and optimization – her recent contributions have been to Ceph, OpenStack Cinder, OpenSDS and Rack-scale design. Vivian holds a Master degree in telecommunication engineering.
About Huamin Chen
Principal Software Engineer, Red Hat
Dr. Huamin Chen is a passionate developer at Red Hat' CTO office. He is one of the founding member of Kubernetes SIG Storage, member of Ceph, Knative, and Rook. He previously spoke at KubeCon, OpenStack Summits, and other technical conferences.
A
Let's
start,
firstly,
thanks
curfew
and
the
great
riddle
of
the
introduction
of
the
chromosome
OS
d.
My
name
is
changing
from
Intel,
and
today's
topic
is
glimpse
of
the
new
self
messenger
built
on
C
star
or
the
so
called
the
Crimson
messenger
followed,
uncover
and
redox
topic.
This
is
more
focused
on
the
practical
experience
to
implement
a
new
messenger
in
self,
and
it
is
the
first
component
that
is
written
with
C
star
and
it
is
a
perfect
one
to
start
with,
because
messenger
is
is
on.
A
Messenger
is
messenger
is
self-contained
the
component
that
makes
it
much
easier
to
implement
and
the
test,
along
with
compared
with
the
Crimson,
OSD
and
I,
have
been
involved
in
his
project
for
Muncie's
and
I
hope
to
share
my
first
handed
experience
to
you
in
this
presentation
so
so
comes
here.
Comes
the
background.
The
whole
refactoring
of
self
core
object
layer
is
under
the
scope
of
the
crimson
project
and
the
goal
is
to
reach
the
optimal
performance
on
the
modern
machines.
A
A
The
the
CPUs
are
grouped
to
his
notes
and
with
dedicated
IO
passes
and
attached
to
a
subset
of
resources,
including
cache
memory,
and
these
are
devices,
and
it
means
that
a
core
is
closer
and
faster
to
access
its
own
resources
with
its
note,
but
it's
slower
and
with
distance
if
it
tries
to
access
the
resources
on
another
Numa
node.
So
it
means
that
practically
the.
A
With
Numa
architecture
it
it
is
compatible
with
the
existing
software,
but
but
it
will
hurt
the
performance
if
the
software
is
not
optimized
with
the
Numa
distances
and
the
second
is
CPU
pipeline.
It
leverages
out
of
order
excusing
and
instruction
level
parallelism
to
for
the
best
performance.
For
example,
if
there
are
four
slots
in
the
CPU
pipeline,
it
can
execute
at
four
micro
operations
per
cycle
per
core
at
the
best
scenario.
A
But
at
the
worst
scenario
it
can
only
execute
less
than
one
operations
and
the
the
pipeline's
divided
two
parts
the
front
end
and
the
back
end.
The
front
end
is
responsible
to
the
front.
End
is
responsible
to
decode,
to
fetch
the
program
code
and
decode
them
into
micro
operations,
and
the
back
end
is
responsible
to
monitor
if
the
data
is
available
on
the
bus
and
executed
the
ready
instructions
from
the
slots-
and
there
are
many
reasons
that
can
impact
the
performance.
For
example,
the.
A
The
dynamic
dynamically
changing
code
and
the
mispredicted
code
will
impact
the
front-end
performance
and
back-end
performance
have
potent
act
at
memory
bound
because
of
the
cache
misses
and
the
slow
access
to
remote
Numa
node,
and
it
can
also
be
a
co
bond
because
of
instructions
compete
for
each
other
for
limited
execution
units
such
as
the
dividing
units,
and
it
means
that
as
our
as
we
programmers,
we
need
to
be
aware
of
this
in
order
to
implement
the
very
optimized
software.
The
the
surly
is
the
emergence
of
the
high
speed
devices
for
storage,
its
nvme
SSD.
A
That
is
much
faster
than
spinning
disks
and
the
revolutionary
persistent
memory
that
will,
at
the
same
level
of
of
the
performance
as
a
similar
level
of
the
DRAM
and
the
network
and
now
up
to
100
gigabits
per
second
bandwidth,
and
it
means
that
we
can
no
longer
expect,
as
is
the
external
external
device.
Io
is
some
levels
slower
than
the
memory,
so
it
is
really
result
in
the
total
refactoring
of
the
software
in
order
to
consume
the
available
computer
resources
to
drive
those
fast
devices
and
the
SAP
architecture
is
not
quite
optimized.
A
With
with
that,
it
has,
the
data
is
allocated
and
accessed
by
all
course,
regardless
of
Numa
topology,
and
the
tasks
are
scheduled
by
the
POSIX
threads
with
the
overhead
of
the
kernel
contact
switches
and
it's
pre-emptive.
That
means
it
can
be
interrupted,
an
expected
lay
from
the
kernel
and
the
block
he
mutex
need
to
to
be
implemented
to
protect
the
shared
data,
and
the
tasks
need
to
be
coordinated
correctly
among
the
among
these
threats,
so
that
will
hurt
the
multiple
core
performance
that
is
required
for
fast
devices.
A
Okay,
so
we
already
have
some
optimizations
with
to
drive
those
limitations,
but
it
is
usually
coupled
with
with
application
code
to
make
it
not
easy
to
be
simplified
and
reduced.
I.
Think
that's
why
we
turned
to
find
out
a
framework
for
this
and
the
C
star
at
the
first
glance
is
meet
our
goal
to
be
a
modern
hardware
friendly,
because
it
is
designed
to
reach
linear
performance
improvements
by
adding
cost
in
the
multi-core
environment,
because
it
forces
the
lock-free
design,
the
traditional
block
him,
you
ticks,
will
causing
blocks
to
the
the
reactor
and
it
will.
A
A
The
logic
can
be
executed
close
to
the
data
for
CPU
pipeline
system
implements
the
user
space
task
scheduling
in
order
to
have
full
control
of
how
workloads
organized
in
in
user
space
and
without
the
effort,
extra
overhead
from
the
kernel
context,
switches
and
the
pre
pre.
If
interruptions-
and
it
also
enforces
the
explicit
message
passing
between
costs,
so
it
makes
develops,
be
fully
aware
of
the
back
end,
but
bottleneck
by
accessing
the
remote.
A
The
memory
on
the
remote
Numa,
Numa
node
and
for
high
speed
devices,
the
sista
adopts
pure
a
synchronous,
event-driven
model
backed
by
a
poor,
and
it
also
integrates
DB,
TK
and
and
the
modern
ways
to
do.
The
direct
memory.
Access
for
fast
Network
and
disk
IO
and
C
Sigma
plus,
is
obviously
another
reason
to
use
C
star
plus.
A
Abstraction
for
a
synchronous
programming,
the
futures
and
the
promises
are
introduced
in
C++
11,
and
these
concepts
are
extended
in
C++,
20
and
C.
Star
has
implementation
of
those
concepts
itself
with
very
low
overhead,
and
it
is
also
a
framework
for
sure
nothing
design
by
providing
shoddy
data
structures,
and
it
also
provides
zero
copy
utilities
to
minimize
the
unnecessary
overhead
from
the
copy,
and
it
is
also
well
tested
and
validated
in
in
many
projects
too,
to
prove
that
it
has.
It
will
have
better
performance
and
parallelism.
A
A
So
so
here
comes
the
goal
of
crimson
messenger,
so
the.
Firstly,
we
want
to
leverage
the
framework
to
uniform
the
architecture
for
a
synchronous
programming
and
secondly,
because
the
cross
core
communication
has
heard
the
performance,
especially
with
Numa,
and
to
minimize
that
we
we
want
to
be
able
to
establish
the
direct
core
to
core
connection
and
the
third
way
we
might.
A
A
The
second
purpose
is
to
enable
a
direct
call
to
call
connection
and
means
to
do
round
to
completion
in
the
main,
our
past
and
with
minimum
cost
quad
core
communication,
because
course
core
is
not
good
for
performance.
For
it.
It
result
in
less
opportunity
to
move
logic
close
to
theta,
because
cross
call.
We
will
have
more
chances
to
let
core
to
access
that.
A
So
it
means
that
the
ownership
management,
if
it
is
quad-core
it
needs
to
be
its
newly,
have
more
overhead
and
the
classical
implementation
of
the
is.
The
connection
is
service
to
service.
It
means
that
the
connection
is
per
or
SD
to
OSD
or
the
client
to
us.
D
and
the
OSD
is
a
multi
core
service.
It
means
that
the
connection
is
not
quad
core,
with
sister
shared,
nothing
design.
A
The
connection
and
the
data
structure
of
connection
and
PGS
are
living
on
the
fixed
cost
and
the
client
can
access
any
P
G's
on
the
primer
OSD
and
if
the
replication
is
set
to
to
the
primal,
SD
will
replicate
odd,
it's
wise
to
a
secondary
OSD
and,
in
short,
if,
if
a
single
write
with
configuration
of
replication
of
replication,
it
means,
on
the
main
I
hope
has.
It
usually
incurs
that,
like
three
cross
core
communications,
that
is
not
good.
A
So
in
a
in
order
to
reach
core
core
connection,
the
simplest
way
is
to
implement
the
single
core.
Osd
and
messenger
should
locate
all
the
connection
to
the
earth
eco
and
it
is
the
current
approach
and
it
is
fully
compatible
with
existing
OSD
OS.
This,
and
the
second
way
is,
is
to
have
multi
core
is
to
still
have
multi
core
as
these,
but
it
may
means
for
messenger
it
should
for
for
its
specially
for
a
multi-core
messenger.
A
A
We
want
an
early
example
to
to
investigate
how
her
step
code
works,
works
with
C
star
and
and
does
the
remember
that
the
a
synchronous
messenger
has
already
been
optimized
for
years
with
similar
design.
So
it
is,
we
need
to
be
implement
very
carefully
with
with
a
new
implementation,
and
the
system
framework
doesn't
provide
the
the
more
control
to
have
multiple
to
have
optimal,
to
implement
optimal
software.
But
it
is
our
responsibility
to
implement
it
correctly.
So
the
the
workload
in
the
messenger
is
is
straightforward.
A
So
I
mean
initial
designs
is
direct
in
from
the
reactor
view.
There
are
many
pending
tasks
that
will
be
executed
in
the
lag
in
the
reactor
in
the
fi
fo
queue
and
the
the
orders
looks
looks
random
because
there
are
many
pending
messages
at
the
same
time,
and
there
are
many,
no
no
messenger
tasks
in
the
same
reactor
and
in
order
to
keep
the
captain
ordered
for
the
streaming
IO.
A
A
So
it
means
that
there
is
only
one
sweep
tax
per
connection
in
the
messenger
and
the
the
messages
are
still
ordered
because
of
the
queue
and
it
is
patched
and
those
there's
also
only
one
running
tax
task
to
to
read
the
messages
the
the
read
will
prefer
to
prefetch
more
buffer
than
your
card
and
the
following
decode.
Well,
we're
running
the
same
task:
if,
if
the
read
will
return
a
readily
immediately
ready
future
with
prefetched
buffer
so
and
the
prefetch
is,
is
also
good
for
to
reduce
the
number
of
expensive
system
course.
A
So
it
and
other
steps
is
at
a
step.
6,
the
the
the
ready
messages
will
be
dispatched
to
the
applications
as
normal
and
we
implemented
it
has
tours
to
compare
the
performance
of
the
optimized
crimson
messenger,
with
existing
a
sync
messenger
and
in
the
in
the
test
environment.
They
are
client
and
the
server
the
clock
in
each
run.
A
We
can
set
the
client
block
size
and
the
server
block
size,
and
we
can
also
set
the
tabs
to
limit
the
the
number
of
pen
the
run
the
sending
messages.
At
the
same
time-
and
we,
we
implemented
the
the
number
of
jobs
to
to
increase
the
pressure
to
the
server
side,
with
multiple
instances
of
messenger,
running
Lockleys,
leh
and
exclusive
course,
and
we
also
have
two
kinds
of
natural
settings:
the
loop
loopback
network
and
the
real
network
with
56
gigabits
bandwidth,
and
we
designed
three
scenarios.
A
We
want
to
evaluate
the
throughput
of
the
saturated
connection
by
setting
the
tabs
excessively
and
the.
Secondly,
we
want
to
evaluate
the
throughput
of
the
saturated
server
call
with
exact
excessive
job
numbers
and
the
third
one
is.
We
want
to
evaluate
the
latency
by
sending
the
messages
one
by
one,
by
setting
the
tabs
two
to
one.
A
A
So
in
order
to
get
better
code
with
with
higher
level
abstraction,
the
the
new
code
is
now
organized
as
continuations
and
the
continuations
are
basically
lambda
functions
that
returning
futures
that
will
be
resolved
later
and
this
these
futures
we
use
with
these
futures.
We
can
use
then
statement
to
link
the
related
logics
together
and
without
a
need
to
define
the
callback
functions
and
implement
and
has
implementations
to
link
them
in
other
places,
and
it
it.
A
It
also
has
continuation
level
semantics
to
allow
us
to
organize
the
continuation
at
higher
level
and
also
specify
the
preconditions
of
the
continuations.
Like
this
case.
The
keep
doing
to
to
let
the
continuation
is
wrong
repeatedly
and
the
the
exception
handling
is
also
extended
to
continuation
level.
If
an
exception
is
throw
out
in
our
chain
of
continuations,
it
will
interrupt
it
and
we
can
handle
it.
A
And
it's
not
the
it
is
not
perfect
and
without
the
costs,
and
the
sister
is
not
that
extensive
to
meet
all
the
corners
of
our
requirements.
For
example,
is
input.
Stream
interfaces
is
seams
not
meet
our
requirements
to
allow
the
memory
to
be
aligned
and
to
specify
lens
and
the
socket
placement,
and
we
we
currently
have
have
walk-around
to
place
sockets
according
to
our
strategies.
But
it
is
not
safe,
so
it
means
to
us.
A
We
might
need
to
fork
the
sister
with
our
custom
implementations
as
a
short
term,
but
in
the
longer
term
we
need
to
rethink
the
overall
framework
and
they
extend
our
implementations
to
be
general
enough
to
upstream,
and
also
the.
Although
sea
star
has
implementation
to
have
a
framework
to
simplify
implementations
of
of
the
code.
But
it
is
still
companies
complex
to
understand
them
because
we
need
to
because
we
need
to
understand
both
programming
models
in
order
to
rewrite
the
existing
code
with
futures
and
continuations,
and
it
also-
and
we
there
are
a
lot
of
things.
A
We
can
use
scheduling
group
to
balance
the
overhead
of
different
tasks
in
the
same
car,
but
if
the
imbalance
is
between
the
course
with
sure
nothing
design,
it
is
difficult
for
us
because
it
we
might
need
to
move
both
data
and
logic
to
different
course,
adaptively
and
as
for
performance,
we
need
to
really
careful
about
the
implementation
and
we
might
need
to
take
into
the
framework
and
with
several
iterations
of
the
code
implementation
and
for
the
debugging.
It
is
annoying
for
at
least
to
me
that
I
sometimes
need
to.
A
The
the
stack
trace
of
the
the
asynchronous
up
operations
might
not
be
useful
because
it
is
a
synchronous
and
I
need
to
dig
into
the
framework
to
find
out
why,
if
there
are
some
bugs
with
it,
so
here's
the
current
status.
We
have
implemented
lost,
Lucie
policy
for
postmessage
of
v1
and
v2
protocol,
and
we
have
necessary
of
optimizations
with
it
and
have
integrated
it
with
crimson
OSD.
A
So
that's
pretty
much
out
of
my
topic
here
and
we
are
a
small
group
with
a
lot
of
work
to
do
so.
You
feel
free
to
join
us.
We
have
a
senior
staff
members
ready
to
help
and
I
think
it
is
a
unique
chance
to
rebuild
the
self
core
object
layer
from
scratch
and
I
bet.
You
will
learn
a
lot
from
it
and
if
you
have
questions
ask
me
thank
you.