►
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2023 in Amsterdam, The Netherlands from April 17-21. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Improving Longhorn Performance With SPDK - Keith Lucas & David Ko, SUSE
Longhorn is a cloud-native distributed block storage solution for Kubernetes, providing an opinionated solution to cover different storage topology, data protection, and data services like snapshots, replication, encryption, backup restore, disaster recovery, etc. The Longhorn team is working on a new data plane backend for block storage to improve performance. SPDK is an open source, high performance development kit for storage applications. Longhorn’s engine component is being rewritten to take advantage of SPDK’s architecture to improve the performance. SPDK will simplify the architecture of Longhorn and make it more scalable. This presentation will go over the new architecture, some preliminary performance data, and areas of future growth. Longhorn was accepted as an incubating project by the Cloud Native Computing Foundation in November 2021.
A
So
we
just
guessed
up
yeah.
Probably
someone
in
this
room
probably
did
not
use
long
before.
So
we
have
a
one
part
we'll
talk
about
what
is
a
long
home
and
what
a
longhorn
walks
and
how
to
try
to
challenge
long
has
right
now
and
next
we
will
talk
about
based
on
the
challenging.
We
are
facing
right
now,
how
we
leverage
spdk
in
the
future
and
how
it
really
works
and
what
benefit
you
will
be
tagged
from
spdk.
And,
lastly,
we
will
talk
about
our
Benchmark.
A
Okay,
so
long
long
is
a
CNC
project
incubating
project
right
now
and
he's
a
focused
on
the
persistent
voting
stuff.
So
it's
highly
available
and
software-defined
persistent
bus
storage
based
on
kubernetes
and
also
wrong
for
kubernetes.
So
all
the
things
we
just
leverage
on
the
kubernetes
stuff
so
he's
quite
lightweight,
reliable
and
easy
to
use
what
I
say
that,
because
we
based
on
the
user
experience
and
we
don't
have
any
external
dependence
actually
so
take
example,
we
don't
have
any
external
database
or
stuff.
We
just
leverage.
A
The
kubernetes
resource
like
API
resource,
so
this
is
what
we
have
so
deploy.
Logo
is
quite
simple
and
also
we
support
all
the
persistent
voting
and
with
the
different
type
of
volume
modes.
Ss
modes
rewrite
many
rewrite
only
as
a
multimoice
bar
device,
Software,
System
L.
Of
course,
if
you
use
a
long
haul,
you
will
question
about
rewrite
many.
Is
it
when
you
will
get
to
General
Virgo,
so
you
can
check
it
tomorrow.
We
have
another
session,
a
maintenance
track
talk
about
local
roadmap
and
also
it's
a
storage
agnostic.
A
So
let
me
he
will
be
easy
to
deploy
based
on
the
host
the
whole
file
system.
Any
five
stands
for
the
specifier,
because
Longhorns
are
used
the
sports
file
for
his
simple
Vision
capability.
So
here
we
mentioned
the
esd4
and
xfs
is
actually
verified
by
longontin,
so
you
can
use
that
directly
and
also
not
just
in
cluster.
We
have
inclusive
snapshot
but
also
support
external
backup
and
restore
so
different
type
of
backup.
Target.
A
A
Sorry
yeah
definition
is
what
we
use
right
now
and
also
it's
open
source
and
how
long
works
right
now
is
simply
said
that
we
have
five
parts
I
want
to
I
want
to
briefly
introduce
a
little
bit,
and
it's
about
the
volume
and
body
life
cycle
and
data
placement
deployment
and
control
plan
and
Berlin
UK
is
composed
of
three
parts
voting
foreign
volume
itself.
We
call
it
engine
if
you
use
the
longhorn
terminology
and
also
the
data
placement
is
for
the
boiling
replicas
and
about
Life's
a
violent
life
cycle.
A
We
100
rely
on
CSI
protocol,
so
he
will
be
driven
by
the
kubernetes
building
customer
resource
PPC
here.
So
any
following
operation.
Just
for
the
CSI
protocol
and
data
placement
is
all
the
data
located
in
the
longhorn
disk
and
Longhorn
this
case.
Actually,
the
fire
Love
On
Top
of
hostile
fire
system,
as
I
mentioned,
so
you
can
create
different
long
disk
per
node,
and
even
you
can
create
a
mount
point
for
different
partitions.
But
regardless,
like
a
long
disk
and
deployment
for
violin,
you
can
see
the
right
hand
side
the
diagram.
A
The
body
is
composed
with
the
engine
and
replicas
and
each
one
is
like
a
segregate
component,
so
you
will
be
independent
from
each
other.
One
volume
have
some
problem
will
not
impact
others.
This
is
what
we
decide
right
now,
so
you
will
make
your
volume
is
much
itself
will
not
impact
by
others
and
control
plan.
I,
don't
have
a
control
and
data
brand
and
control
plane
based
on
kubernetes,
very
straightforward.
A
Okay,
if
you
dive
into
a
little
bit
about
Longhorn
engine
and
replica
database,
because
today
we
will
talk
about
database
and
the
current
model,
we
use
the
volume
phone
and
based
on
isci
protocol.
So
he
will
let
the
longhorn
volume
by
one
by
the
open
ISC
is
that
we
use
a
as
a
client
to
make
it
happen
on
the
whole
side
and
also
voting
engine
is
like
the
volume
controller.
A
So
he
will
be
have
an
engine
process
along
with
the
TGT
Target
server,
to
make
sure
the
data
can
pass
through
the
TGT
server,
then
down
to
the
engine
and
engine
have
a
two
major
parts.
It's
a
one
is
a
TCP
based
data
server
and
the
other
one
is
the
old
volume
controller
he's
a
grpc
server
to
accept
the
different
comments
from
the
upswing
and
the
last
part
is
replica
replicas
for
the
data
variant
data,
so
the
engine
will
bypass
the
comments
or
even
data.
A
A
So
this
is
a
diagram
to
talk
about
what
I
say
about
a
database.
So
you
can
simply
say
you
have
a
wallow.
The
user
application-
and
you
start
to
do-
have
some
I
o
stuff
and
go
into
your
body
on
top
of
that,
have
a
file
system
and
then
it
will
go
into
the
iska
support
device
is
posted
by
the
longhorn
and
it
will
go
to
the
iska.
Cd
is
a
host
library
to
make
sure
you
can
write
to
the
longhorn
violin
and
you
will
be
running
on
the
TGT
and
use
the
leave.
A
Longhorn
Library
communal
care
with
each
engine
engine
is
what
I
say
is
a
violin
controller
then
make
sure
the
I
o
it
will
Downstream
to
a
down
pass
to
Longhorn
replicas,
but
this
is
currently
what
Longhorn
Works.
How
are
the
challenge
right
now?
Long
has
the
first
one.
It's
too
many
components:
I
mean
too
many
components
along
the
database.
Like
you
see,
the
user
need
to
prepare
setup.
A
Iscosity
and
long
will
create
a
TGT
engine
process
for
each
balance,
so
he
will
be
a
little
complicated
from
the
database
point
of
view
and
the
second
one,
because
too
many
components
say
we
will
close
the
different
Coast
extra
cost
of
photo
communication,
especially
for
the
database.
You
need
to
I
o
through
the
different
parts
like
here
TGT
and
engine,
and
that's
one
I
want
to
emphasize-
is
I
o
models
limitation?
A
Because,
right
now
we
based
on
the
spouse
file
and
we
set
array
I
o
and
with
the
blocking,
read
and
write
so
far,
so
you
will
have
some
challenge
for
the
performance
perspective
and
also,
if
we
want
to
leverage
like
a
synchronized,
I
o
stuff,
you
will
be
a
little
challenging
right
now
for
the
current
architecture
and
also
for
sure
different
language
or
different
integration,
especially
a
special.
You
want
to
integral
system
stuff,
have
a
different
effort
for
us
for
sure
yeah.
A
B
Spdk
is
a
is
a
software
performance,
development
kit.
It's
used
in
a
lot
of
high
performance,
Cloud
applications
and
one
of
the
things
that
it's
kind
of
based
off
of
is
the
data
plane
development
kit,
which
is
a
a
tool
that
is
used
by
other
cloud
service
providers
to
utilize
Network
device
drivers
in
user
space
to
provide
better
performance
than
using
the
kernel,
and
this
kind
of
includes
a
dpdk
in
its
source
code
and
uses
some
of
the
methodology
behind
that
to
kind
of
provide
better
performance
for
any
kind
of
application.
B
And
one
of
the
things
that
it
also
has
is
a
generic
block
device,
application
layer
with
several
different
implementations
and
a
relatively
easy
way
of
implementing
it.
And
we
kind
of
got
to
a
dead
end
with
the
current
implementation
of
Longhorn,
where
we
have
multiple
go
routines.
Each
using
blocking.
I
o,
and
it
kind
of
didn't,
provide
any
means
to
easily
support
the
asynchronous
methodology
to
improve
performance.
B
But
spdk
already
has
some
some
of
these
generic
block
devices
that
use
Frameworks
like
lib,
AIO
or
IOU
ring
to
provide
better
performance
and
a
lot
of
basically
plug
and
play
and
try
those
out
and
test
those
more
quickly
than
rewriting
longboard
engine.
The
way
it
is
and
as
we
were
discussing
earlier,
we
were
mentioning
that
longhorn
currently
used
his
eyes
scuzzy
by
using
TGT,
which
is
an
open
source
software
product
that
we
modified
to
interface
with
the
longhorn
engine.
B
Spdk
supports
that
as
well
as
a
newer
type
kind
of
as
a
as
in
we
no
longer
just
use
salad
drives,
we
use
nvme
drives.
So
when
so
nvme
over
Fabrics
is
like
an
improved
Network
protocol
for
a
network
of
available
volumes,
and
it
has
a
feature
called
logical
volumes,
which
we
allows
us
to
store
data
in
a
way
that's
equivalent
to
how
Longhorn
currently
works
with
its
sparse
file
methodology
of
storing
data.
B
So
we
can
have
an
equivalent
of
future
equivalence
between
the
two
and
also
basically,
it's
designed
for
asynchronous
programming,
which,
for
the
most
part,
seems
to
perform
better,
especially
for
intense.
I,
o
performances,
I
mean
Longhorn.
Currently
provides
relatively
good
but
performance,
but
we
want
to
make
it
better
and
I
think
one
thing
that
we
do
at
each
level
of
that
data
path.
Is
we
allocate
memory,
then
deallocate
it
in
TGT,
we
kind
of
malloc.
It
then
free
it,
and
in
the
go
portion
of
that
we
allocate
it.
B
B
We
it
kind
of
just
stores
it
in
a
huge,
so
it
can
the
kind
of
the
equivalent
of
a
file
system
and
store
all
of
its
data
within
a
huge
disk
and
manage
it
as
if
it
was
a
file
system.
But
it's
able
to
do
it
more
efficiently
because
it
doesn't
have
to
support
all
the
features
of
the
file
system
and
basically
sbk
sbdk
is
the
only
user
of
it.
So
it
doesn't
have
to
deal
with
any
of
the
contention
that
a
normal
file
system
would
have
to
deal
with.
B
So
I
took
the
picture
of
how
Longhorn
sparse
provisioning
works
from
our
Longhorn
website
and
I
took
a
picture
of
spdk's
logical
volumes
from
sddk's
website
and
I.
Think
both
of
these
show
the
methodology
that
both
systems
use
for
kind
of
determining
like
how
to
read
a
read
or
write
a
block.
So
when
we
have
like
a
hierarchy
of
snapshots
or
data
and
all
that
data
can
be
sparse.
B
So
if
I
looked
up
to,
I'd
have
to
use
an
ioctal
or
ICC,
I
o
CTL
to
see
what's
available
at
that
particular
block
and
then
and
see
if
that
is
sparse
or
not.
If
it's
a
sparse
or
it's
an
empty
location
in
the
file,
I
go
to
the
to
the
next
snapshot
and
so
forth,
and
in
this
diagram,
two
I
have
to
go
through
three
layers
before
I
actually
find
the
data
in
the
oldest
snapshot.
B
And
similarly,
this
SPD
K
has
the
same
functionality
except
it
implements
that
the
ability
to
query
the
sparse
data
like
internally.
So
we
don't
have
to
use
the
kernel
to
get
that
as
far
as
information.
So
we
go,
we
can
just
Traverse
that
hierarchy
then
find
our
data
without
some
of
the
stuff
and
basically,
and
all
these
snapshots
in
sbdk
are
within
the
block
device
that
we
don't
have
a
file
system
on,
whereas
with
Longhorn.
Each
of
these
snapshots
is
a
separate
file
within
the
file
system
that
we're
traversing.
B
So
one
of
the
things
that
sbdk
kind
of
makes
it
easy
to
do
use
is
to
program
it
asynchronously
and
for
the
most
part,
this
kind
of
allows
you
to
achieve
better
performance
and
with
this
methodology,
basically,
I
o
functions,
don't
block
if
you
perform
a
read
or
a
write,
it
doesn't
block,
but
it's
like
you
register
in
the
event,
then
we
have
a
main
Loop
that
checks
to
see
the
status
of
that
and
continuously
check
to
see.
If
this,
if
it's
complete,
then
we
are
notified
via
a
callback.
B
So
so
we're
going
to
redesign
the
the
data
plane,
part
of
all
Longhorn,
to
work
with
sbdk
and
for
the
first
point,
we're
going
to
switch
to
using
nvme
over
fabrics
for
Longhorns
block
device
instead
of
iscos
E
Linux
has
mature
support
for
nvme
over
Fabrics.
The
Linux
kernel
has
kind
of
a
better
interface
for
nvmeola
Fabrics
than
iSCSI,
for
example,
is
because
he
has
a
demon
process.
B
B
We
only
need
to
use
one
for
a
read
operation
and
we
currently
will
be
doing
that
in
a
round
robin
we'll
just
go
from
one
to
the
next
one
for
each
read,
operation
and
kind
of
distribute
them
throughout
all
the
nodes
and
we'll
also
support
snapshots
and
rebuilding
new
new
volumes
when
added
so
snapshots
is
that
kind
of
functionality?
That's
built
into
Longhorn
to
you
know,
have
an
atomic
point
where
we
stored
a
state
of
the
volume
and
rebuilding
is
what
happens
when
you
add
a
new
replica
when
the
volume
is
already
up.
B
So
you
need
to
copy
that
data
that
exists
to
a
the
new
node
that
you're,
adding
and
one
of
the
departures
that
we're
using
doing
with
long
run
with
sbdk.
Is
that
we're
going
to
have
only
one
spdk
process
for
kubernetes
node?
Each
SPD
process
will
handle
multiple
block
devices
from
prior
or
with
the
current
Longhorn.
B
There's
one
long
run.
Engine
per
process
and
like
tgtd,
is
the
thing
that
handles
all
the
volumes,
so
we're
going
to
be
generally
reducing
the
number
of
processes
and
spdk
is
handling
more
of
the
scheduling
than
using
blocking
IO
that
to
run
the
to
allow
the
never
mind.
Okay,
so
I
have
a
diagram
of
kind
of
what
I
just
explained,
and
this
is
a
example
of
a
volume
with
three
replicas.
B
I
o
methodology
in
Linux
IOU
ring
and
we
had
two
tests
that
we
we
did
first
test
is
we
tested
a
raw
disk,
the
existing
Longhorn
and
longer
with
sbk
on
a
single
node
I
think
the
main
motivation
was
to
see
the
overall
impact
of
using
either
implementation
of
the
longhorn
versus
just
using
the
disk.
Then
the
second
example:
we
did
a
three
node
scenario
with
both
the
existing
Longhorn
and
spdk.
B
So
here
are
our
results
and
so
I
think.
The
one
thing
that
was
very
interesting
is
I.
It
seems
as
if
the
longhorn
with
spdk
performs
very
similarly
to
the
just
using
the
raw
disk
in
terms
of
both
the
bandwidth
and
the
I
think
it's
most
equivalent
in
bandwidth
and
the
I
o
operations
per
second,
it's
relatively
close,
but
I.
Guess
it's
a
little
bit
down
on
that
and
the
latency
is
very
similar.
B
B
The
bandwidth
for
rights
is
improved
to
compare
to
the
current
Longhorn
and
also
the
I
mean
the
bandwidth
is
kind
of
equivalent
and
the
I
o
Ops
per
second
are
very
much
improved.
So
I
guess
that's
kind
of
a
summary.
What
I
just
said.
The
bandwidth
improves
knock
categories,
it's
very
similar
to
just
using
the
disk,
and
the
overall
latency
is
about
like
25
to
30
microsystem
seconds
versus
100
microseconds
with
Longhorn.
B
So
this
is
the
three
no,
that
performance
comparison.
There
was
no
disk
equivalent
because
that's
not
really
possible
so,
and
in
this
case
it's
also
very
much
improved
for
compared
to
the
current
Longhorn.
We
have
better
much
better
I
O
operations
per
second,
especially
compared
to
right,
and
the
bandwidth
has
also
improved
I
mean,
if
you
like,
think
about
it.
B
The
existing
Longhorn
is
already
doing
over
a
a
gigabyte
of
data
per
second,
but
we're
doing
even
better
with
spdk,
but
latency
is
also
improved
and
you
in
general
it's
performing
better
and
you
can
kind
of
see
compared
to
the
single
node
scenario.
Our
three
note
scenario
like
performs
better
than
the
like:
it:
it's
a
kind
of
equivalent
to
raid
one
and
in
this
network
scenario
it
performs
better
with
read
operations
and
just
a
single
disk
on
a
single,
node
and
and
I.
B
Think,
like
we
haven't
even
scratched
the
surface
of
all
the
things
that
you
could
do
with
sbdk
I
think
one
of
the
things
that
sddk
is
noted
for
know
that
the
four
is
doing
user
space
and
Vimy
support.
So
if
one
of
the
nodes
in
our
kubernetes
cluster
actually
used
and
vme
drives,
we
might
be
able
to
access
them
directly
instead
of
using
the
Linux
kernels
driver-
and
this
is
like
one
of
the
SPD
case-
claim
the
frame
it's
kind
of
equivalent
to
how
dpdk
uses
network
drivers
and
user
space.
B
We
didn't
initially
do
this,
because
we
wanted
to
have
a
wide
scope
of
scenarios
that
we
could
support.
So
we
could
support
any
sort
of
environment
not
necessary
only
nvme
environment,
but
I
think
we
could,
as
everyone
moves
to
using
actual
nvme
drives.
Instead
of
other
formats,
we
can
potentially
go
and
investigate
doing
that.
We
can
also
try
to
use
a
IOU
ring
and
other
new
technologies
when
I
tried
to
use
IOU
ring
it
in
spdk
at
one
point
in
time
it
didn't
work
but
I'm
since
Longhorn.
B
Nvme
Fabrics
use
our
dma
using
special
drivers
in
the
Linux
kernel
to
more
efficiently
transport
data
over
the
network,
instead
of
using
the
TCP
approach,
which
is
what
we're
currently
using
and
our
dmaker
that
improved
performance
even
more
so
I
guess
what's
next
for
Longhorn,
so
we
kind
of
developed,
Longhorn
spdk
in
isolation
outside
of
the
rest
of
Longhorn,
which
is
the
one
that
enables
it
to
work
with
kubernetes.
So
we
need
to
do
a
lot
of
changes
into
how
it's
deployed.
B
I.
Guess
the
main
thing
that
handles
how
things
work
within
Longhorn,
how
it
deploys
all
the
pods
or
controls
most.
How
it's
deployed
on
a
kubernetes
network
is
something
called
the
long
run.
Engine
I
mean
the
longhorn
manager
and
we
kind
of
need
to
change
how
it
deploys
Longhorn
on
the
kubernetes
of
that
work,
because
we're
moving
from
One
Longhorn
engine
process
per
node
to
I
mean
multiple
Longhorn
engine
processes
per
node
to
one
spta
process
per
node
that
handles
all
the
volumes
so
we're
going
to.
B
We
have
this
component
called
the
instant
manager
which
manages
all
these
processes,
and
we
need
to
kind
of
refactor
that
and
make
it
basically
remove
it
and
just
have
one
spdk
process
and
I.
Think
as
David
was
mentioning,
we
have
a
lot
of
crds
and
the
crg's
kind
of
reflect
the
current
process
model.
We
kind
of
need
to
update
that
to
reflect
the
new
process
model
going
forward
and
we're
going
to
basically
remove
this
iSCSI
component
and
TGT
and
transition
to
nvme
over
Fabrics.
A
So
we
right
now
we
just
show
the
body
parts.
So
while
we
are
walking
right
now
is
a
control
print
feature
priority
integration,
so
I
would
say
the
better
timing
is
next
year:
I
don't
have
a
quite
periodical
currency
for
release.
So
each
year
we
have
a
two
feature:
release
so
I
would
say
in
the
the
coming
release
is
one
that
four,
the
zero
is
end
of
this
year,
but
this
free,
this
reads
will
not
have
spdk
preview
stuff,
but
it
will
happen.
A
B
A
Draw
the
destructive,
but
we
I
try
to
because
Upstream
spdk
have
a
lot
of
application.
Integration
already
take
example
like
sbdkdd,
so
we
try
to
think
about
how
to
integrate.
To
make
sure
the
sparse
file
can
be
can
be
migrated
to
the
spdk
profile.
So
it
would
be
better
to
do
the
migration,
but
I
would
not
say
you
still
need
to
investigate
to
see
how
it
works.