►
Description
Messenger V2: The New Ceph Wire Protocol - Ricardo Dias, SUSE Linux
In this talk we will present an overview of the current design and implementation of the new wire protocol (the protocol used for communication between pairs of daemons and daemons and clients), named Messenger V2, that aims to overcome the limitations of the current protocol, such as the possibility to encrypt all data transferred on the wire.
We will start by describing the current wire protocol design and pinpoint its limitations, which will pave the way to present the new protocol features. Then we will present the details of the new protocol design, it's new features, and how to deal with clients from older Ceph versions. We will also talk briefly about some of the possible future features that can be implemented by extending the base protocol.
About
Ricardo Dias
SUSE Linux
Senior Software Engineer
Ricardo Dias is currently working as a senior software engineer, at SUSE Linux, in the Entreprise Storage Team, where his main task is to contribute to the upstream Ceph storage system project. He is also an Integrated Member at the NOVA LINCS laboratory, where he still colaborates in several research projects, and co-supervises a PhD student with Prof. João Lourenço.
He received his doctoral degree from the Universidade Nova de Lisboa, Portugal, in 2013, under the supervision of Prof. João Lourenço, on the topic of Transactional Memory.
During his research career, he has published, and presented, several research papers in high ranked scientific conferences.
A
A
So
today
we
are
going
to
talk
about
I
will
start
with
more
or
less
explaining.
What's
the
SEF
messenger
component,
why
we
require
to
develop
a
new
wire
protocol?
What
was
the
problems
with
the
previous
existing
one?
Well
still,
existing
one
I
will
then
talk
a
bit
more
in
detail
about
the
new
protocol,
which
we
called
messenger.
V2
and
I
will
show
you
some
of
the
details
of
the
protocol.
A
A
Usually
it's
it's
invisible
to
end-users,
and
and
if
it's
not
it's
because
something
is
is
broken,
some
problem
exists
in
in
your
cluster
and
also
one
thing
important
is
that
the
safe
distributed
algorithms,
the
OS,
the
PGP,
ring
all
those
des
monts
quorums.
They
are
built
on
top
of
safe
messengers.
So
the
safe
message
is
just
the
coq
communication
library.
It
doesn't
know
anything
about.
What's
going
on
above
so
basically
the
safe
messenger
component.
A
A
So
yeah
messenger
is
the
it's
the
small
communication
library,
okay,
that
all
self
component
ceases
to
talk
to
with
each
other.
They
it
can.
This
small
every
can
be
used
both
as
a
server
and
the
client
so
because
II,
you
know
that
OS,
these
both
and
other
components
act
both
has
servers
and
in
case
they
are
listening
for
requests.
They
are
sending
requests
to
each
other
and
yeah,
and
our
beady
and
large
lab
are
are
just
client.
A
The
safe
messenger.
Also
abstracts,
the
the
technology
is
for
the
physical
communications,
so
it
it
works
above
the
network
layer,
so
it
and
abstracts
we.
It
can
be
used
in
above
POSIX
sockets
or
our
DMA
or
DP
DK.
It
ensures
a
reliable
delivery
delivery
of
messages.
We
then
exactly
once
semantics
semantics,
semantics,
sorry,
and
it
also
takes
care
of
it
also
handles
the
some
of
the
connection
failures.
A
So
why
did
you
that?
What
did
we
need
to
design
a
new
protocol?
Why
didn't
the
the
existing
one
didn't
was
not
enough,
so
messenger,
v1
or
just
messenger
was
the
was
the
first
wire
protocol
itself.
So
it
was
the
was
there
since
the
beginning,
and
one
of
the
problems
was
that
it
didn't
allow
us
two
weeks
to
extend
the
protocol.
So
if
we
require
to
make
changes
to
the
protocol
in
a
very
early
stage,
it
isn't
it
wasn't
supported.
A
In
the
end,
the
only
way
to
change
the
protocol
would
be
to
introduce
breaking
changes,
so
there
was
no
possible
to
to
extend
the
protocol
and
still
be
compatible
with
previous
with
previous
versions
of
self.
It
also
didn't
support
the
didn't
guarantee
that
authenticity
neither
supported
that
encryption,
which
is
a
security
requirement.
A
A
It
didn't
have
a
very
structured
messages
in
in
their
control
in
the
company
in
the
protocol.
Control
messages
were
not
very
structured
and
it
was
just
it.
It
needed
some
refactoring
and
so
the
these
these
requirements,
or
these
these
limitations
led
to
the
necessity
of
creating
a
new
protocol,
and
this
was
what
led
to
the
v2.
A
So
the
messenger
v2
is
basically
a
complete
redesign
and
implementation
of
of
the
protocol.
It
was
this
design
since
from
the
beginning
to
be
an
extensive
protocol.
So
it's
a
protocol
that
we
can
change
the
behavior
of
almost
from
the
beginning
when
the
the
peer
start
to
connect
so
allows
us
to
extend
it
without
causing
breaking
changes
with
previous
versions.
A
So
in
in
Seffner
Dilys.
The
the
v2
protocol
will
be
is
available
on
port
3,
3
0
0
for
safe
months,
and
that
we
didn't
remove
the
v1.
Otherwise,
we
wouldn't
be
able
to
connect
to
clients
from
previous
version
could
not
connect
to
to
tune
our
dealers,
so
we
are
basically
safe.
Months
are
now
listening
on
two
parts.
One
is
the
v1
part,
the
6
7
8
9,
if
I'm
not
mistaken,
and
the
new
v2
port,
which
is
3
3
0.
Also.
A
Okay,
so
I'm
going
to
show
you
briefly
how
the
protocol
works
for
that
I
will
always
have
a
client
and
the
server
and
the
protocol
can
be
splitted
in
four
four
phases.
So
the
first
phase
is
called
the
Ben
Eric
change,
then
the
authentication,
then
it
proceeds
to
the
session
and
shake
and
and
and
in
the
end,
the
the
message
exchange.
A
This
is
the
message
frame
for
the
no
encryption
mouth
of
the
algorithm,
okay
and
so
the
the
preamble.
It
has
many
many
fields,
but
the
important
so
the
first
one
is
actually
the
type
of
frame.
So
each
each
control
message
has
a
type
and
that's
specified.
On
the
first
eight
eight
bits
of
the
of
the
of
the
preamble,
and
then
we
have
the
number
of
segments
of
data-
okay
and-
and
we
this
will
tell
us
basically
how
much
segments
have
the
payload
data.
A
A
A
Usually
the
the
control
messages
for
the
negotiation
only
use
one
segment
and
then
the
data
messages
use
the
remaining
segments
and
the
after
the
pale
of
comes
the
epilogue,
which
has
some
flags,
and
it
also
has
the
segment
CRC's
to
check
the
integrity
of
each
segment.
Okay,
so
every
time
we
send
a
message
to
the
other
side,
we
basically
pack
the
payload
inside
this,
this
frame
with
this
structure
so.
A
Banner
exchange,
the
first
thing
that
happens
when
the
the
network
connection
gets
established
between
both
between
both
the
client
and
the
server
France.
If
you
are
using
POSIX
sockets
as
soon
as
the
TCP
connection
is
established,
both
the
client
and
the
server
will
send
the
banner
message.
The
banner
message
is
the
the
only
exception
that
is
not
packaged
inside
the
frame.
A
Okay,
it's
a
it's
a
match
of
a
message
that
has
a
simple
string:
readable
string
called
safe
v2,
and
then
afterwards
we
say
we
which
I'll,
which
is
the
banner
length,
so
how
many
bytes
takes
our
banner,
and
currently
we
are
just
sending
two
bit
masks.
What
are
the
supported
features
of
this
of
this
pier,
and
what
are
they
require
features
and
is
by
using
these
bit
masks
is
the
way
we
can
actually
influence
the
behavior
of
the
the
rest
of
the
protocol.
A
Okay,
so
after
exchanging
the
bit,
the
supported
features
than
required
features
with
each
other,
and
they
send
this
message
in
in
parallel
when
they,
when
both
they
receive
the
banner
from
each
other,
they
send
an
allow
message,
a
yellow
frame.
This
our
frame,
serves
to
to
indicate
what
kind
of
entity
I
am.
So
if
I'm
am
on
an
OSD
and
a
client,
okay,
and
also
the
my
address.
A
So
the
first
thing
the
client
sends
another
request
to
the
server
okay
and
in
these
out
requests
it
specifies
what
is
the
authentication
method
that
he
wants
to
use?
Currently,
we
is,
we
only
support,
like
no
authentication
seems
so
I,
don't
I'm
doing
an
Authenticator
at
all
right
or
using
suffix,
which
is
what
implementing
and
maybe,
in
the
future,
using
a
carrier
bearers
authentication
scheme.
A
So,
after
after
the
authentication
method,
we
we
specify
what
are
the
preferred
modes
of
the
connection,
and
currently
we
only
have
two
CRC
are
secure,
so
CRC
is
our
traffic
is
unencrypted
and
not
signed,
and
we
just
check
make
integrity
checks
using
CRC
codes.
Okay,
the
secure
mode,
on
the
other
hand,
will
give
you
the
unwire
encryption
functionality.
So
all
traffic
will
be
encrypted
and
signed.
A
So
then,
there's
also
the
some
payload
bytes
that
are
dependent
on
the
authentication
method
that
that
you
issues
so
now
the
server
when,
when
receives
this,
can
decide
if,
if
it
supports
the
authentication
method
or
not,
and
if,
if
it
doesn't
sue
and
also
if
it
supports
the
mouth
right
and
it
can
really
apply
to
the
client,
say
I-
don't
support
this.
Please
I
only
support
these
methods
and
the
end
this
mode.
So.
A
Then
the
client
needs
to
decide
what
to
do
right
and
it
can
perform
another
out
requests
and
now,
if
the
server
is
is
okay
with
what
with
what
the
client
is,
is
requesting
it
it.
It
replies.
Okay,
with
a
message
called
auto
reply,
more
okay
and
and
the
client
can
can
then
send
another
earth
request
more
okay
and
we
can
go
back
and
forth
as
many
rounds
as
as
as
we
want,
and
this
allows
us
to
plug
in
here
different
authentication
schemes
that
have
more
rounds
or
less.
A
A
So
when
we
are
done
when
server
gets
done,
there
are
two
more
messages
that
are
sent
to
each
other,
both
the
client
and
the
server.
While
they
were
exchanging
these
authentication
messages,
they
were
doing
calculating
digests
of
all
traffic
and,
in
the
end
they
sent
to
each
other
that
that
digest
to
guarantee
that
there
was
no
interfering
interference
on
those.
There
are
no
reply,
attacks
and
other
kinds
of
text
to
make
sure
that
nothing
happens
between
the
authentication
negotiation.
A
A
So
the
first
thing
is
the
client
once
again
initiates
the
session
establishment
and
when
it's
a
new
session,
so
the
periods
connecting
for
the
first
time.
Well,
sometimes
it's
not
the
first
time,
but
when
the
client
doesn't
know
it's
not
aware
of
any
previous
session,
for
instance,
if
the
your
demon
has
crashed,
then
this
it's
restarted.
It
will
send
the
client
identification
frame
to
the
to
the
server,
which
has
a
lot
of
information
like
who.
What
are
my
addresses,
who
is
the
address
of
whom
I
think
I'm
contacting
my
global
ID?
A
Many?
What
are
the
safe
supported
features
and
what
are
they
acquired
and
also
very
important?
The
client
cookie,
which
he'll
be
used,
which
will
identify
this
this
session,
at
least
on
the
client-side,
and
will
be
very
useful
for
when
reconnecting
to
to
session
the
server
on
the
other,
and
when
receives
the
client
identification
it
it.
A
If
everything
is
ok
for
him,
he
will
reply
to
the
client
with
his
server
identification
and
also
the
with
all
these
information,
also
about
the
features
and
also
important
the
server
cookie,
ok,
which
identifies
the
session
from
the
server
from
the
server
side.
So
when
there
are,
there
is
some
connection
failure,
so
the
peers
are
already
sending
message
to
each
other
and
there
is
a
connection.
Failure.
The
the
peers
are
able
to
reconnect
to
to
a
previous
session.
Ok
and
so-
and
this
starts
also
with
the
client.
A
A
It
also
it
it
also
sends
the
some
sequence
numbers.
So
we
know
it's
not
a
duplicate,
reconnect
and
other
stuff,
but
it
also
sends
the
message
sequence
number,
which
is
like
what
was
the
last
message
that
I
saw
from
from
you.
Ok,
the
server
will
will
look
at
this
message
will
check
if
the
client
cookie
matches
with
something
at
that
you
know
in
the
past
will
also
check
if
the
server
cookie
matches
for
that
session,
and
if
everything
is
OK
it,
it
will
send
a
very
simple
message.
Saying
OK
I
accept
your
reconnection,
your
reconnect.
A
So
after
this
step,
everything
is
okay,
so
both
for
the
new
session
and
for
the
for
the
reconnect,
they
are
prepared
to
start
sending
message
together
to
each
other.
Okay,
so
at
this
point
both
Pierce
can
Aneesh
can
send
messengers
receive
and
they
will
also
piggyback
acknowledgement
to
tell
the
other
peer
already
received
that
the
message
with
that
sequence
number
yeah.
This
is
the
traditional
way
of
guaranteeing
some
stateful
the
stateful
session.
So
it's
each
message
frame
as
I
said
before
it,
they
used
the
the
four
the
four
segments.
Okay,.
A
So
forgetting
the
unwire
encryption,
so
if,
during
the
authentication
phase,
if
they
agreed
on
using
the
secured
mode
right,
they,
it
means
that
now
all
the
traffic
starting
from
the
session
negotiation
is
it's
supposed
to
be
encrypted,
okay
and
and
but
what
we
do
is
we
we
encrypt
the
whole
frame
and
the
frame
it's
very
similar
to
the
unencrypted
frame.
The
only
difference
is
the
epilogue.
The
epilogue
has
different
different
fields.
A
So
the
preamble
in
an
encrypted
friend
doesn't
change
that
payload
doesn't
or
it
also
doesn't
change,
and
only
the
the
epilogue.
The
epilogue
basically
has
has
the
same
flex
that
he
had
in
in
the
unencrypted
form,
and
then
he
has
some
padding
to
fill
up
the
first
16
16
bytes
and
afterwards
it
it.
It
has
n
bytes
of
data
that
depends
on
the
encryption
algorithm.
A
So
some
administration
step.
So,
as
I
said
in
the
beginning
of
the
presentation,
we
we
implemented
the
new
protocol,
but
we
still
need
to
have
the
v1
protocol,
because
both
the
kernel
drivers
still,
they
only
talked
v1
right,
and
if
you
have
an
old
user
space
client
connecting
to
the
cluster,
he
will
also
only
be
able
to
talk
the
v1
protocol
so
which
means
that
we
had
to
to
support
to
have
the
two
protocols
be
working
out
at
the
same
time.
So
the
man's
need
to
be
listening
at
both
the
v1
and
v2.
A
A
What
is
the
the
port
that
the
v1
protocol
is
is
listening
to
and
what
is
the
part
where
v2
is
listening
and
I
have
here
an
example
of
the
man
host
configuration
up
now
we
can
specify
for
each
month
for
each
month
inside
brackets,
a
list
of
IPL
of
addresses
and
we
have
to
prefix
the
the
IP
address
with
the
type
of
the
protocol,
so
if
it's
v1
or
v2
and
then
specify
the
deport.
Okay,
of
course,
we
we
also
allow
you.
A
We
still
allow
you
to
specify,
as
we
you
were
used
to
by
just
specify
the
list
of
addresses
for
each
for
each
month,
and
these
are
equivalent.
So
if
you
just
specify
the
D
IP
address,
we
will
assume
that
you
will,
you
will
use
the
differ.
We
will
use
the
default
ports
so
the
six
seven,
eight,
nine,
ten
four,
oh,
this
is
switch.
This
should
there's
a
typo
here,
so
we
will.
Basically,
if
you
just
specify
only
the
address,
we
will
be
listening.
A
So
with
the
v2
it
come,
it
also
come
new
configuration
options.
So
the
now
we
have
MS
bind
messenger,
1
and
M
s
by
in
messenger
to
you.
This
specifies
if
the
demon
should
should
bind
to
the
v1
or
v2
protocol,
and
we
can
set
to
control
if
we
want
the
CRC
mode
or
the
secure
mode.
We
have
this.
This
config
options,
the
heavier
the
descriptions
the
MS
montclair
mouth
specifies
the
modes
for
the
the
connections
between
monitors.
The
mont
service
mode
specifies
the
the
mount
the
modes
when
connecting
duane
services
connect
months.
A
A
A
So
messenger
to
is
enabled
by
default.
So
if
you
deploy
a
nautilus
cluster,
you
will
demands
will
immediately
starting
to
listen
on
v2
and
the
other
demons
as
well.
If
you,
if
you
want
to,
if
you
want
to
disable
the
messenger
which
you
just
add
to
your
yourself,
come
file
or
to
the
config
store
the
option.
Ms
mind
messenger,
2
equal
to
false
and
then
reboot
the
demons.
A
Ok
and
demons
will
stop
listening
on
the
will,
stop
listen
on
the
v2
protocol
and
if
you
want
to
re-enable
just
and
do
the
same
things
so
upgrade
so
actually
operates
from
luminous
or
letters
which
would
mimic
you
will
probably
see
you
have
a
luminous
closure.
You
do
the
upgrade
for
for
Nautilus.
You
will
probably
will
see
this.
This
message
saying
that
the
monitors
don't
have
the
messenger
to
enable
okay,
so
this
basically
means
that,
after
an
upgrade,
if
we
want
to
to
use
the
messenger
to
protocol,
we
need
to
explicitly
tell
the
cluster.
A
Please
start
using
the
messenger
2
and
we
can
do
that
with
a
am
on
command
called
safe,
Mon
in
messenger
to
and
but
there's
a
catch
here.
We
can
only
use
this
command
if
your
cluster
is
using
the
default
ports
for
for
the
view,
one
protocol.
So
if
your
Mons
their
configuration,
they
are
listening
on
the
six
eight
nine
six,
seven,
eight,
nine
port.
Okay,
if
if
your
mons
have
a
custom
port
for
some
reason,
you
cannot
run
this
command
you
to
fix
that.
A
You
will
actually
need
to
change
the
Monmouth
manually
and
and
set
the
and
set
the
port
for
the
video,
because,
probably
if
you're,
if
you
are
not
using
the
default
port
for
v1,
probably
you
want
to
also
custom
the
v2
protocol
for
firewall
reasons
or
something,
and
since
we
cannot
guess
which
part
the
user
wants
to
use,
we
just
call
it.
These
are
needs
to
manually,
adapt
the
Mon
map
and
yeah.
This
is
the
presentation,
don't
have
anything
else,
so
any
questions
or
doubts.
A
B
A
A
A
B
B
A
Using
the
encrypted
so
so,
the
question
is:
if
there
are
performance
impacts
when
using
the
encrypted
mode
and
yeah,
so
the
encryption
mode
will
of
course,
will
have
an
impact
on
performance.
We
don't
have
numbers
yet
so,
yes,
I'm
sure
they're
there
are
I,
just
don't
know
how.
How
much
is
it
so
we,
we
still
haven't,
run
the
performance
benchmarks.
My
my.
A
A
So
the
question
is
I
told
during
the
presentation
that
first
happens
the
authentication.
Then
we
only
start
encrypting
messages
afterwards.
Well,
the
authentication
control
messages
they
are.
They
are
sick,
they
are
secured.
They
there's
no
credentials
passing
in
plain
because
they're
using
currently
to
use
SFX
and
they
have
these
keys
there.
They
know
everyone
knows
keys
of
the
mansion
of
the
client,
keys
and
yeah.
So
the
deification
payload
it's
it's
encrypted
and
follows
yeah.
No
security.
Is
that
time
not
that
I'm,
aware
of.