►
Description
Don’t miss out! Join us at our upcoming hybrid event: KubeCon + CloudNativeCon North America 2022 from October 24-28 in Detroit (and online!). Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Longhorn: Intro, Deep Dive and Q&A - Sheng Yang, Joshua Moody, SUSE
Longhorn is a cloud-native distributed block storage solution for Kubernetes, providing an opinionated solution to cover different storage topology, data protection, and data services like snapshots, replication, encryption, backup restore, disaster recovery, etc. In this talk, there will be several parts to have an introduction of Longhorn and have deep-dive discussions to talk about the technical details, the recent release, and future plans. Longhorn was accepted as an incubating project by the Cloud Native Computing Foundation in November 2021.
A
Welcome
everyone
and
I
think
it's
the
first
time
we
have
the
in-person
meeting
after
such
a
long
pandemic
and
well
in
fact,
the
most
difficult
things
I
found
about
this
time
is
just
fighting
the
room
so,
as
you
can
see
like
we're,
probably
in
a
hurry,
so
sorry
about
that,
we
started
five
minutes
late,
all
right.
So
today,
my
our
topic
is
longhorn
and
I
will
be
joined
by
my
colleague
joshua
and
to
talk
with
you
on
the
longhorn
project.
Give
you
update
and
see
what
is
upcoming.
A
So
how
is
longhorn
different?
There
are
a
few
design
principles
on
the
longhorn
when
we
start
working
on
this
project.
We
have
the
aim
to
for
this
storage
project,
to,
of
course,
be
reliable,
which
is
the
most
important
thing
about
any
storage
project
and
the
longer
is
designed
to
be
crash
consistent,
and
we
also
have
designed
multiple
layers
of
protection
against
the
data
loss.
Like
longhorn
itself
has
built
in
snapshots.
A
You
can
always
create
in-class
snapshot,
but
in
addition
to
that
also
provides
the
backhand
mechanism
like
building
backhand
mechanism,
for
you
to
backup
your
volume
to
a
s3
like
external
of
the
snap
of
the
cluster,
and
also
that
backup
is
very
efficient.
It's
incremental
and
it's
like
the
second
layer
of
protection
will
get
for
your
volume
and
the
third
layer
is
in
fact
longhorns
design.
It's
very
simple.
Even
in
the
worst
case
scenario,
you
lost
all
your
nose
as
long
as
you
have
one
of
the
replica
hard
drives.
A
The
local
provides
like
disaster
recovery
snapshot,
backup
the
disk
tags
and
all
the
others
functions,
and
the
third
thing
is
like,
unlike
many
of
the
storage
projects
out
there,
we
designed
longhorn
to
be
very
maintainable
and
it's
the
concept
is
easy
to
understand,
as
you
can
see
later
and
even
easy
to
recover
in
the
worst
case
scenario,
because
you
understand
how
it
works
and
the
longhorn
also
provides
your
ability
to
upgrade
without
interrupting
the
workload
which
means
you
don't
need
to
schedule
like
a
speed
of
downtime.
If
you
have
something
happens,.
A
Instead
of
parts,
so
this
is
very
simple
design
and
but
it's
very
efficient
and
it's
very
reliable,
because
when
you
see,
if
you
have
scale
to
more
parts
with
more
volumes,
they
are
going
to
create
new
engines
and
replicas
and
more
for
you
to
satisfy
your
storage
need,
and
there
are
two
advantages
of
this
design.
The
first
thing
is,
as
you
can
see,
the
data
path
of
each
volume
is
in
fact
not
interfering
together.
A
So,
for
example,
if
you
take
out
take
out
the
engine
or
take
out
the
replica,
the
worst
case
scenario
is
only
going
to
impact
that
the
volume
of
using
that
engine,
and
if
you
take
out
the
replica
of
course
another
replica
will
take
over.
So
it's
no
even
no
impact,
and
the
second
thing
is,
as
you
can
see,
the
engine
always
collocated
with
the
like,
with
the
consumer,
with
the
workload.
A
So
if
the
node
winds
down
and
the
kubernetes
were
going
to
reschedule
the
part,
for
example,
to
the
node
3.,
even
there
is
no
local
replica
exist
within
in
this
scenario,
lower
still
can
find
the
blue
replica,
which
is
for
the
blue
volume,
still
exist.
So
we're
going
to
create
another
engine
which
is
stainless
by
itself,
connect
to
the
replica
on
node
2
and
restore
your
operation
for
your
for
your
volume.
A
So
it's
very
flexible
and
the
reliable
design
here
and
it's
a
like
satisfies
a
lot
of
scenario
like
if
you
have,
for
example,
eks
running
on
different
ac
and
your
ebs.
One
is
going
to
tie
to
one
az
and
locally,
in
fact,
able
to
do
cross,
ac
scheduling
and
ensure
your
data
visibility.
In
case,
one
ac
goes
down
which
ebs
itself
can
now
do
okay,
so
that
was
the
like
overview
of
the
data
plan
of
longhorn,
and
this
is
the
quick
rundown
of
the
manager
part,
which
were
called
like
a
management
plan.
A
So
this
is
the
very
simple
approach
and
now
we're
also
exploring
some
other
ways
currently
in
the
next
release.
We're
going
to
have
like
more
integration
with
kubernetes
to
enable
you
to
like
manually
change,
the
object
and
the
spec
and
also
reflect
that
change
into
the
local
system.
So
currently,
this
change
is
gated
by
the
long
api,
but
they
were
not
going
to
be
updated
in
the
future.
A
All
right
so
quick
update
on
the
now
the
open
source
community
side
of
longhorn
we're
currently
a
cncf
incubation
project
and
now
github
starts
3.8
000
and
the
worldwide
node
count
is
currently
at
53
thousand
plus.
And
in
fact
you
can
take
a
look
at
the
metrics.local.io
to
see
the
real-time
update
and
we're
connecting
this
data
is
in
fact
like
just
a
get
call
which
we
got
to
send
a
notification
about
the
local
user.
A
A
Okay,
so
upcoming
1.3
release,
we're
already
have
the
1.3
rc1
out,
I
think
a
couple
days
ago
and
by
the
end
of
q2,
we're
aiming
to
have
windows
3,
like
official
ga
release
in
this
release,
we're
going
to
include
csi
snapshot,
support
for
the
longest
snapshot.
Previously
we
have
the
longhorn
snapshot.
Sorry,
csi
snapshot
for
the
longhorn
backup,
which
means
like
whenever
you
create
a
csi
snapshot.
Logo
will
create
a
backup
for
you
in
the
backup
store,
which
is,
as
you
might
know,
is
off
the
cluster.
A
A
So
this
is
the
most
immediate
upcoming
release
and
in
fact,
what
really
excites
me
is
about
what's
in
the
future,
so
on
the
release
after
or
like
two
weeks
after
we're
looking
to
add
in
the
trim
support
to
reduce
the
volume
size.
One
question
we
in
fact
get
a
lot
is
like
a
file
system.
If
you
delete
file
in
the
file
system,
like
the
long
warning
didn't
shrink,
the
reasoning
is
lower
is
by
design
is
a
blog
storage.
So
you
need
to
understand
more
file
system
protocol.
A
For
that
to
happen,
so
we
are
aiming
to
add
that,
in
the
in
the
release,
like
a
in
the
next
release
after
windows,
3.
and
also
the
local
system,
backup
restore
itself
is
going
to
enable
you
to
like,
for
example,
if
you
have
upgrade
scenario,
you
are
going
to
have
a
fail
safe
over,
like
sorry
fail
safe
in
case
like
something
went
wrong
and
the
one
guru
feature
will
be
there
for
your
stay
for
workload.
A
For
example,
you
have
a
application
running
as
a
stable
set,
and
all
of
these
volumes
can
be
grouped
together
for
you
to
take
snapshots
or
backups,
and
we
are
also
aiming
to
add
highly
available
nfs
rearrangement
support
and
helium
s3
support
as
well
the
last
of
the
list.
We
are
also
like,
in
fact
working
on
a
very
exciting
new
thing,
is
a
new
high
performance
engine
based
on
sbdk,
so
yeah.
I
will
leave
more
to
my
colleague
joshua.
Do
you
demo?
B
B
B
A
B
B
Okay,
so,
like
I
said,
we
previously
done
lots
of
long
haunted
demos
for
like
intros,
so
I
figured
I
talked
about
some
setups
you
want
to
have
for
your
production
clusters
or
in
the
enterprise
space.
Let's
start
by
number
one.
How
do
I
do
auto
scaling?
What
how
would
my
cluster
set
up?
Look
like
I'm
using
the
enchant
for
demonstration
as
a
visualization
tool
here,
but
all
of
this
can
be
done
with
operators
and
yamls
and
helm
directly.
B
B
And
we
can
see
that
I
have
like
three
notebooks
general.
You
want
three
note
pools
on
your
non-bare
metal
clusters.
Anytime.
You
want
to
do
auto
scaling.
You
want
to
have
your
control
planes
separated
from
your
worker
nodes.
We've
seen
that
one
generally
people
end
up
putting
them
together,
control
playing
worker
notes
and
then
some
things
we
saw
on
this
like
they
scaled
up
the
cluster,
and
then
they
had
50
control
plane
nodes
effectively,
which
is
a
problem
in
itself.
B
B
You
can
scale
up
the
stable
set,
but
you
cannot
really
scale
it
down,
because
you
need
to
be
able
to
guarantee
that
for
the
lifetime
of
the
cluster.
Therefore,
it's
your
minimum
availability
so
that
that's
the
prem
cluster
here
in
this
case
and
lastly,
your
transient
set,
which
is
just
the
regular
worker
pool
the
worker
pool
itself,
can
be
permanently
auto
scaled.
B
You
would
want
to
have
it
set
up
so
that
it
can
scale
up
down
quickly,
depending
on
demand
based
on
your
metrics.
In
my
case,
all
the
notes
are
the
same
use.
The
same
template
no
template,
but
what
you
would
want
to
have
is
for
the
minimum
resource
set.
You
have
bigger
notes
with
more
cpu,
more
storage,
so
that
you
have
less
change
in
that
environment
and
for
the
transient
set.
You
can
have
lots
of
small
nodes
because
it's
pretty
cheap
to
recycle
them
right.
So
that's
that
now.
B
The
other
question
is
what
kind
of
options
do
I
have
on
setting
up?
How
do
you
use
longhorn
to
define
where
my
data
actually
gets
stored,
and
this
is
when
you're
in
an
enterprise
environment
you
don't
necessarily
have
different
nodes,
some
have
gpus.
Let's
say
you
have
machine
learning,
workflows
some.
C
B
Nvme
some
have
like
spinning
disk
and
you
don't
want
the
slowest
web
server
kind
of
thing,
using
storage
from
your
nvme
meters,
for
example,
on
the
node
that
has
the
gpu.
That
would
be
like
worst
case
scenario,
so
to
solve
this,
we
provide
a
whole
bunch
of
parameters
in
our
storage
classes
that
allow
an
administrator
to
find.
B
B
B
B
B
If
you
don't
have
affinity,
zone,
setup
or
zones
and
region
setup,
you
can
also
tag
individual
nodes
to
ensure
for
right
now
for
the
current
version.
Until
we
have
volume
group
support
to
ensure
that
your
volumes
get
scheduled.
Let's
say
if
you
have
an
application
replica
that
in
itself
has
data
replication
and
you
want
to
ensure
that
the
application
replicas
replica
gets
scheduled
so
that
you
at
least
always
have
two
two
application
replicas
available,
so
that
you
can
lose
one
of
the
application
ribbon.
B
That
doesn't
make
too
much
sense,
but
once
we
have
volume
group
support,
we
can
we
know
that
this
one
is
part
of
a
group
of
set
of
volumes
and
we
can
apply
scheduling
rules
to
the
replicas,
as
I
said,
but
till
then
as
a
workaround.
You
can
tag
individual
nodes
and
ensure
that
you
have
like
three
storage
classes
for
each
of
the
application
replicas,
and
they
would
then
ensure
that
the
rapid
cars
get
distributed
evenly.
B
Okay,
here's
the
most!
The
second
thing
is,
you
can
add
many
this
so
on
your
stable
set,
it's
perfectly
fine
to
when
you,
after
evaluating
your
resource
consumption,
you
say:
yeah,
I'm
not
actually
using
too
much
cpu
or
memory,
but
I'm
getting
low
on
storage.
You
can
attach
additional
disks
via
the
cloud
provider,
block
storage
and
expose
them
as
part
of
the
longhorn
set
which
allows
you
then,
to
tag
these
appropriately.
B
Well,
in
this
case,
this
is
tagged
as
an
nvme
this,
so
any
replica
that
was
provisioned
with
the
previous
storage
class
ends
up
on
one
of
the
nvmedias,
while
the
default
disk
is
just
a
regular
disk,
so
no
tags.
So
no
specific
scheduling
rules
right,
that's
that
and
last
but
not
least,
I
want
to
talk
a
little
bit
about
the
upcoming
spdk.
It's
still
in
the
works
and
here's
some
quick.
B
Okay,
so
this
is
like
a
quick
developer
benchmark
so
take
away
grain
of
salt,
it's
not
production
ready
yet,
but
in
general
you
see
that
the
iops
we're
getting
are
pretty
close
to
the
to
the
local
disk.
This
is
including
a
file
system
right,
so
this
is
a
host
mounted
file
system
versus
an
sdk,
provided
longhand
volume
then
mounted
to
a
local
path
on
the
node.
B
Little
bit
more
latency,
so
we
go
from
132
microseconds
to
160
microseconds
for
the
I
o.
But
again
this
is
only
for
the
local
replica.
Once
you
add
like
remote
replicas,
the
latency
will
go
up
a
little
bit,
but
you
also
due
to
the
architecture,
change
that
with
the
spk
implementation,
we're
going
from
one
cpu,
basically
cpu
users
per
process.
So
previously
each
volume
was
this
independent
process.
We're
now
going
more
into
the
you.
B
Have
one
process
for
the
whole
note
which
allows
it
to
scale
with
the
amount
of
cpu
as
well
as
amount
of
disk
linearly,
which
is
a
for
large
large
permanent
storage
set,
like
I
demonstrated
here,
is
really
beneficial,
while
for
a
hyper
converged
state.
It's
also
beneficial,
but
less
so
because
you
have
less
resources
per
node.
A
A
See:
okay
on
right
side,
you
see
the
longhorn
spdk
that
label
that
is
like
our
currently
in
developer,
sbdk
engine,
which
nowhere
near
finish,
just
very
very
like
big
caveats
there,
but
it's
one
to
show
off
like
what
is
the
like
potential.
We
can
get
so
as
you
can
see
that,
basically,
we
have
no
difference
within.
A
So
this
is
the,
and
also
this
is
a
strictly
local
right.
Now
we
do
have
our
sbk
implementation
right
now.
Do
you
support
replicas
and
start
adding
support
for
the
snapshot,
but
still
this
is
in
a
very
early
stage
and
we're
still
talking
like
working
on
this,
but
for
the
people
looking
for
the
performance
yeah.
This
is
something
I
can
show
you.
B
Definitely
we
have
here
the
community's
concern
in
regard
to
performance
and
we're
working
very
hard,
shout
out
to
my
colleague,
keith
lucas,
who
is
driving
the
spdk
initiative
and,
as
you
can
see
from
this
quick
def
benchmark,
we're
yeah
in
comparison
are
doing
pretty
well.
This
is
just
for
reference.
This
is
on
a
single
cpu
provided
to
spk
on
a
single
disk.
B
B
D
A
Oh
you
yeah
yeah
yeah,
so
currently,
if
there
are
like
a
soft
infinity
rule
enabled
which
is
in
fact
disabled
by
default
and
the
replica
well
was
in
the
same
node.
We
are
going
to
my
other
balance
to
another
note,
but
currently
like
based
on
the
capacity
it's
not
being
automatically
rebalanced
and
we'll.
Like
that's
in
fact,
one
request
we
heard
from
the
community
and
we're
taking
that
into
consideration
for
the
roadmap.
B
Yeah
we
do
it,
we
did
add
some
outer
balancing
features,
they're
not
complete,
yet
we
have
some
least
effort
best
offered,
but
yeah,
not
a
100
change.
Yet
jenny.
What
I
like
to
have
is
like
the
different
affinity
rules,
so
you
can
set
up
zones
as
well
for
your
your
kubernetes
nodes,
which
we
will
also
respect
as
part
of
the
affinity
rules.
We
have
disk
affinity
as
well
as
node
affinity.
B
A
D
B
B
B
A
D
A
B
We
also
changed
the
timeout
handling
mechanism
on
the
io
side.
So
let's
say
you
previously
had
slower
discs,
which
was
one
request
we
had
previously.
We
improved
that
a
little
bit.
It's
still
not
officially
supported,
since,
if
you
want
to
really
good
performance
like
this,
you
need
to
make
dedicated
optimizations
for
that.
But
with
the
changes
in
the
newer
versions,
you
should
be
able
to
run
it
on
the
slower
disk
or
we
yeah.
B
C
C
B
What's
going
on
there
so
right
now,
the
way
we
implement
every
export
is
by
having
us.
We
call
them
share
manager,
but
it's
basically
a
provisioner
that
ensures
that
the
every
old
block
device
volume
gets
attached
to
a
certain
node,
and
then
we
export
nfs
shares
via
nfs
ganesha
on
the
application
level.
Out
from
that,
the
problem
is
for
highly
availability
systems.
You
now
need
a
mechanism
that
is
either
active,
passive,
active
efficacy.
B
You
can't
do
active
actors
with
an
already
blocked
device,
but
we
can
do
something
like
active
passive
right
now.
You
can
use
it.
It
is
marked
as
experimental
it's
usable.
The
only
downside
is
that
if
the
node,
where
the
nfs
server
is
on
goes
down,
we
end
up
terminating
your
workloads.
That's
depending
on
settings
right
depending
on
confusion.
B
We
end
up
terminating
your
workloads,
which
generally
is
a
service
interruption,
but
due
to
the,
if
you
have
them
in
a
deployment
or
stateful
set,
the
replacement
workouts
will
be
spun
up
and
the
replacement
workers
will
attach
to
the
new
lvx
chair
manager.
So
the
interruption
of
service
is
very
small.
B
If
you
have
an
end
up
having
an
issue,
but
therefore
we
classify
that's
not
highly
available
yet
once
we
have
an
active
passive
setup,
it
would
the
transition
time
would
be
much
smaller
and
there
would
not
be
any
downtime
for
the
actual
application
service.
There
will
be
an
io
stall
during
the
handover,
but
that's
handled
on
the
kernel
side,
so
we
should
be
following
there.