►
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2023 in Amsterdam, The Netherlands from April 17-21. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Longhorn: Intro, Deep Dive And Q+A - David Ko & Joshua Moody, SUSE
Speakers: David Ko, Joshua Moody
Longhorn is a cloud-native distributed block storage solution for Kubernetes, providing an opinionated solution to cover different storage topology, data protection, and data services like snapshots, replication, encryption, backup restore, disaster recovery, etc. In this talk, there will be several parts to have an introduction of Longhorn and have deep-dive discussions to talk about the technical details, the recent release, and future plans. Longhorn was accepted as an incubating project by the Cloud Native Computing Foundation in November 2021.
A
A
Okay
long,
basically,
is
a
highly
available
sofa
defined
surgery
for
persistent
or
volume,
so
he's
a
provided
by
device,
but
on
top
of
it
you
can
provide
file
system
as
well.
So
have
a
different
scenario
depending
on
your
use
case,
and
it
is
lightweight
reliable.
What
we,
what
we
said
is
because
all
the
things
we
don't
really
rely
on
the
external
stuff
like
a
database
or
other
other
service.
We
just
rely
on
the
kubernetes
resource
so
and
also
the
installation
weighs
very
easy.
A
No
other
support
dependency
and
the
major
scene
is
precision
volume,
so
a
different
type
of
volume,
modes
and
access
mode.
We
do
provide
by
Longhorn
like
a
part
device
and
the
file
system
for
SS
mode.
We
have
a
re-rounding
and
rewrite
many,
maybe
sample
you
use.
A
longhorn
will
curious
about
the
rewrite
many
when
we
will
get
it
to
the
channel
available.
I
would
say
it
will
be
soon
because
at
by
end
of
this
year
we
have
a
one
photo.
A
Release
will,
including
we'll
make
the
rewrite
many
GA
I
will
talk
about
it
later
and
next
storage
agnostic,
so
because
we
want
to
make
the
long
e
to
use.
So
we
don't
let
the
user
to
worry
about
how
to
measure
loan
this
stuff.
So
any
kind
of
wholesale
fire
system,
supporter
or
tent
spot
file
can
be
used
by
long.
But
right
now
verified
by
by
our
team,
is
est4
and
SFS,
and
this
one
that
means
the
user.
A
You
can
just
rely
on
what
the
policy
thing
you
have
and
use
a
long
territory
and,
of
course
not
just
in
cluster.
We
also
support
out-of-cast
stuff.
So
it's
the
viewing
function
and
you
can
pick
up
your
volume
to
external
NFS
and
S3
are
compatible
servers.
I
pick
a
Target
or
pick
a
store,
storage
and
also
I
will
talk
about
more
new
stuff
in
140,
about
system
backup,
and
we
took
it
later
and,
of
course,
these
kubernetes
the
design
implement.
A
Yeah
feature
is
I,
don't
want
to
go
over
all
the
items,
but
I
want
to
highlight
some
item
major
items
here,
so
the
first
one
are
violin
because
Longhorn
can
provide
simple
Vision.
So
let
you
leverage
your
store
to
use
efficiently
and
also
we
have
inclusive
stuff
station
out
of
cost
backup
restore
and,
of
course,
some
clone
is
pension
based
on
the
CSI
protocol.
So
all
the
things
you
work
on
your
volume
based
on
CSI,
it
will
work
well
with
the
longhorn
and
security
staff.
A
We
provide
the
value
encryption
in
the
current
versions,
arrest
in
in
transit
based
on
the
current
loan
architecture.
And
if
you
talk
about
the
replicant
level
running
data,
we
have
a
different
scheduling
strategy,
but
simply
say
we
provide
the
pros,
no
and
and
software
anti-affinity
for
no
level
and
also
Auto
replica
rebalancing.
This
one
is
actually
for.
A
If
your
classes
have
a
no
download
up,
we
have
some
okay
and
to
make
the
replica
can
advance
and
storage
attack
is
also
the
way
to
let
you
manage
your
node,
and
this
can
use
you
well
for
your
wallow
and
volume
usage
and
about
Disaster
Recovery
cross
Discovery.
This
is
one
thing
we
will
talk
about
later
and
live
migration
and,
of
course,
for
the
backup
restore
external.
We
have
a
front
user
feedback.
A
We
have
a
policy
base,
so
you
can
Define
your
recurring
job
for
in
cluster
snapshot
or
external
pickup,
and
one
thing
I
just
mentioned.
We
write
many.
We
will
make
it
ga
by
end
of
this
this
year.
One
for
zero
is
together
with
the
arm
64.
I'm
64.
We
already
supports
an
experimental
feature.
However,
we
we
don't
we
we
will
not.
We
are
not
very
confident
before
because
we
did
not
run
the
full
regression
on
it,
but
right
now
already
integrate
into
our
testing
pipeline.
A
So
the
testing
coverage
is
the
same
as
MCT
md64
and
you
are
yeah.
Ui
is
a
simple
way
you
can
operate,
but
right
now
we
support
a
crd
admission
web
hook,
this
kind
of
stuff,
so
the
interface
will
be
more
than
UI
and
talk
about
the
community
and
momentum
for
Longhorn
and
Longhorn.
Getting
is
a
long
history,
but
along
the
kubernetes
growth
we
also
grow
together
and
right
now
the
stuff
for
sure
is
keep
going
and
Enterprise
use
a
special.
A
We
have
a
kiosk
this
time
and
her
here's
some
some
feedback
from
the
users
and
some
users
have
a
different
scenario
usage
and
also
we
have
some
Telecom
users
as
well,
and
we
have
a
mattress.
We
know
right
now.
There
are
more
than
17
000
notes
wrong
in
Longhorn
already
keep
going,
and
if
you
compare
with
three
years
ago,
we
donate
the
sensor
part
to
sensitive
it's
right
now,
already
gross
23
23
times
yeah.
This
is
a
mattress.
We
just
collect
like
middle
of
October,
so
it's
keep
going.
A
We
don't
have
a
release
right
now,
but
we
just
have
a
release
one
tree
in
June
and
this
one
we
did
not
highlight
a
lot
in
the
previous
qcom,
so
this
one,
this
page,
can
give
you
understand
what
the
what
the
functionality
major
items
in
One
Tree
and
we
have.
We
introduced
admission
and
conversion
Web
book
for
the
longhorn
resource,
V1
beta2.
So
that
means
you
don't
need
to
worry
about.
If
you
use
the
V1
beta1,
we
will
automatically
do
the
conversion,
so
this
is
different
from
some
other
solution.
A
You
need
to
prepare
some
work.
If
you
migrate
to
the
next
version
of
a
resource
and
storage
Network,
some
people
from
the
community
as
well,
they
want
to
make
the
data
plan.
The
traffic
can
be
separate
from
the
control
plan,
so
storage
Network
right
now
with
their
leverage
based
on
the
models
you
can
Define
your
your
network
so
make
our
data
plane
components
all
the
traffic
through
the
second
interface.
So
this
one,
and
also
some
people
from
the
user
they
use
longhold
on
the
cloud.
A
We
don't
respect
it,
but
right
now,
in
one
tree
we
have
this
kind
of
scene
supports
and
I
I
think
you
can
give
a
try,
because,
right
now
we
still
money
the
experimental
feature,
because
we
want
to
know
more
feedback
from
the
users
and
yeah
snapshot.
Snapchat
will
be
will
be
involved
in
the
space
usage.
So
we
want
to
do
more
about
the
essential
approach
to
make
sure
the
space
usage
much
efficiently
and
right
hand
side.
We
have
a
CSS
Nation.
A
A
CSS
session
is
a
it's
a
feature
already,
but
only
focus
on
the
external
pickup
volume
picker,
but
right
now
for
Longhorn,
we
have
in
in
clusters
nature
as
well,
but
we
want
to
have
a
user,
the
same
user
experience,
so
user
can
use
a
CSS,
essential
store
in
incluster,
sensual
or
external
cluster
backup
and
packing
image.
Yeah.
So
warning
is
not
just
raw
data.
A
You
can
based
on
an
easy
impact
image
to
create
your
volume,
so
you
can
use
the
backing
okay
back
image
as
your
backup
strategy
as
well,
because
you
can
expose
your
body
as
a
back
image,
download
and
reuse
it
later.
So
it
is
a
it's
a
quiet
fiction
for
some
users:
spatial
virtualization,
stuff
security
communication.
We
have
introduced
the
TRS
encryption,
it's
the
option
in
so
you
can
enable
it
to
make
sure
the
traffic
between
the
control,
plane,
developer
and
communication
about
a
spatial
folder
control
print
commands.
A
A
So
if
you
still
encounter
some
upgrade,
if
you
are
using
your
encounter,
upgrade
slow
still
using
one
tree,
let
us
know
we
will
see
how
to
improve
if
you
still
have
some
issues
yep
so
at
least
for
one
three
and
one
four
is
upcoming
release
by
end
of
DC
years,
and
we
focus
on
the
data
itself
and
long
is
very
respect.
The
feedback
from
the
community.
So
all
the
things
actually
you
see
here
is
a
major
vote
from
the
community
as
well.
A
The
first
one
left
hand
side
the
three
three
volumes,
because
Longhorn
is
a
broad
device,
so
it
cannot
really
respect
the
file
system,
the
train
to
let
the
the
unused
space
free,
but
right
now
you
will.
We
will
do
that
in
one
form:
one
phone
zero
and
rewrite
many
violin
GA.
Previously
we
did
not
have
like
a
metal
Service
function
for
the
rewrite.
Many,
our
turn
is
a
share
manager,
so
he
will
met
if
a
shared
manager
don't
client
can
compact.
A
The
data
consistent
will
have
some
problem,
but
right
now
we
introduce
a
meta
service
in
our
share
manager,
so
we
will
make
sure
when
our
share
major
compact
compact,
the
client
connection,
will
be
keep
keep
for
the
data
consistency
and
pick
up
and
restore
yeah
right
now.
Longhorn
have
a
body
impecca,
restore
bodies
for
violin
and
in
one
four
zero
we
will
have
a
long
system.
Pickup
restore
what
that
means.
You
can
have
a
different
strategy
for
your
custom
pickup
and
you
don't
really
really
rely
on
the
vehicle.
We
have
a
documentation.
A
You
can
use
a
barrel
to
do
some
stuff
for
the
Long
Haul
season
pickup,
but
this
is
for
current.
So
right
now
in
the
future.
Wonderful,
we
don't
want
the
user
to
like
use
a
different
tool
to
manage
by
themselves
want
just
one
stop
use
a
service
to
let
the
user
can
can
do
the
longhorn
season
pickup
and
restore
and
people
quotation
together
with
Jason
right
now.
We
we
only
rely
on
the
revision
counter
mechanic
in
in
in
our
violin.
A
It's
not
good
enough
because
they
have
a
percent
possibility
to
make
the
data
inconsistency.
So
in
one
for
sale
we
will
have
a
checksum
for
each
step
Asia
to
keep
calculating,
but
not
just
calculating.
We
have
a
function
you
can
set
in
you
can
enable
detecting.
So
you
will
make
sure
the
out-of-date
data
will
be
removed
and
the
Clone
create
a
review
from
the
healthy
one.
A
So
this
is
one
and
together
with
the
bureau,
protection
yeah,
I'm,
64,
I,
just
told
palette
and
kubernetes
1,
112,
25
and
people
ask
about
the
past
security
stuff
PSP,
and
we
are
working
on
that.
If
you
are
following
GitHub
right
now,
we
are
already
the
pr.
The
code
is
also
merged,
so
you'll
be
sooner
or
later,
but
we
have
some
concern
about
the
back
port
to
the
current
feature
is
like
a
one
three
or
one
two.
So
we
will
want,
because
every
reason
we
will
make
sure
Longhorn
will
be
together.
A
What
kind
of
minimal
version
of
kubernetes
you
can
support.
So
some
different
factors
considered
to
get
1
12
25
support
will
be
happen,
will
happen
in
one
four
zero
and
support
bundle,
and
we
found
some
troubles.
When
used
user
feedback.
Some
issues
we
users
can
download
the
propounder,
but
right
now
the
information
is
not
good
enough.
A
special
Longhorn
also
rely
on
the
host,
so
the
news
for
bundle
integration.
We
will
make
sure
this
part
are
much
better.
A
So
when
user
report
issue
will
be
to
identify
identify
by
our
engineers
in
online
volume
expansion
the
same
it's
a
very
many
people
ask
and
that's
why
it's
local
volume,
local
volume.
We
have
a
data
located
for
your
volume,
but
it's
the
best
effort,
but
right
now
many
wall
loads
actually
running
like
this
real
distributed
application
like
a
database,
so
local
violin,
the
requirement
is
getting
getting
hot.
So
you
want
to
make
this
one
to
extend
the
data
locality
to
have
a
strict
enforce
mode
for
that
to
get
a
better
performance.
A
So
this
one
for
zero,
so
PC
is
packed
for
the
upcoming
release
by
end
of
this
year,
yep.
So
this
what
I
say-
and
recently
we
have
a
Patrice
for
one
two:
six,
because
one
of
a
rarely
happen
data
corruption
probably
happened,
so
we
fix
it
in
one
street
two
and
upcoming
one,
two
six.
So
if
you
are
using
one
two,
you
can
upgrade
to
the
the
upcoming
batteries.
A
Okay,
so
next
we
will
talk
about
Longhorn.
I
will
brief,
introduce
this
slide
and
later
Joshua
will
talk
about
detail
the
other
functionalities
data
print,
how
it
works
so
basically
for
Longhorn,
I
I
summarize
the
five
parts,
the
one
is
volume
and
second
is
life
cycle.
Then
data
placement
deployment
and
control
plan
so
violin
actually
happens
three
parts,
the
one
is
a
lot
in
foreign
right
now
we
will
ice
castle
protocol,
but
we
have
a
different.
We
have
a
new
plan.
A
A
So
these
three
parts
and
how
about
Longhorn
Avon
in
life
cycle,
is
triggered
by
the
PPC
and
we
get
it
customer
resource
and
our
customer
resource
is
actually
building
resource
and
we
have
a
CSI
sidecar
spewing
together
with
a
local
distribution.
So
you
will
watch
those
resources
to
have
a
corresponding
operation
as
well.
Of
course
you
we,
because
long
is
six
to
seven
years
project.
A
We
have
a
different
interface
like
a
restful
API
stuff,
but
we
encourage
a
user
can
start
to
use
a
kubernetes
way
to
have
a
consistent
user
experience,
so
data
placement,
because
loan
provider,
storage
agnostic,
so
it's
long
distance,
actually
like
a
firefight
on
top
of
host
file
system,
so
depends
on
your
strategy.
You
can
define
a
different
one
point
for
different
long
disk
and
have
a
different
tag
for
your
replica
schedule.
Strategy
for
that
and
deployment
I
I
call
here,
segregated
microservice,
because
each
body
is
independent
from
others.
A
B
Friend,
oh
no
I
don't
need
this
okay,
yeah,
so
right
now
our
front
end
is
based
on
icecase
by
using
TGT
in
the
application
mode.
Two
components
for
data
playing
is
like
the
volume
engine
which
you
can
think
of
as
like
a
volume
controller
a
while
a
replica
is
just
a
volume
data.
It's
like
a
damp
data
store.
The
communication
happens
over
TCP
IP.
B
B
So
this
is
a
one
very
interesting
aspect
of
the
Spirit
storage
is
like
you
want
fault,
tolerance
right
in
traditional
setup.
You
would
have
the
port
a
which
is
using
a
volume.
It
would
traditionally
be
just
writing
to
the
direct
host
disk
that
creates
a
problem
that
that
part
is
forever
coupled
to
that
nodes,
because
it
relies
on
the
storage
of
that.
B
So
you
need
to
need
a
level
of
indirection
in
between
that
and
we
provide
that
by
the
engine
and
replica
plug
in
between
this
allows
the
replication
of
the
data
across
nodes.
In
our
case,
we
do
this
synchronously,
so
our
weather
cards
are
equal
at
all
points
in
time.
This
also
allows
us
to
move
the
workload
arbitrarily.
So
if
the
community
schedule
decides
hey
this
notice
resource
exhausted,
for
example,
hey
I
need
to
move
this
workload
because
it's
low
priority,
then
Network
can
move
to
an
arbitrary
order
node
in
a
cluster.
B
It
helps
us
in
the
case
where
you
would
lose
the
node.
In
this
case,
node
one
would
go
down.
Of
course,
the
sensor
work.
Does
this
at
that
point
in
time
on
what
note
it
would
be
down?
It
would
be
rescheduled
by
the
schedule.
We
have
some
improvements
there
like
if
you
ever
use
the
stateful
set
before
you
will
notice
that
the
pods
don't
move
by
default.
So
we
help
clean
it
up
and
help
the
process
along.
B
So
the
community
Sketcher
would
now
schedule
this
node
on
on
the
Note
3,
the
Pod,
a
on
Note
3..
In
this
case,
node
3
is
purely
a
compute
node
right.
So
in
Longhorn
you
don't
necessarily
need
every
note
to
be
a
storage
node.
You
can
have
a
dedicated
storage
pool,
which
is
like
a
traditional,
centralized
storage
texture,
while
if
you
have
storage
on
each
node,
that's
more
like
the
hyper-convert
storage
Arch
texture,
if
you
want
a
benefit,
cost
overview
of
that.
B
One
join
us
at
the
booth
for
an
in-depth
discussion
later,
like
at
the
kiosk,
so
you've
seen
that
the
node
failed.
The
data
is
saved
because
we
have
replication.
Kubernetes
was
able
to
reschedule
the
port
to
an
arbitrary
order.
Node
in
this
case
a
compute
node
and
the
engine
continues,
processing
and
living
on
Happily
after
you
can
see
that
there
is
many
different
volumes
on
on
each
node.
We
have
what
we
call
an
instance
manager
for
replicas
in
an
instance
manager
for
node.
B
C
B
And
you
can
also
have
different
amount
of
disks
in
Longhorn
like
each
node
can
have
different
nodes.
Generally,
it's
very
nice
to
have
like
a
node
template
set
up.
But
if
you
know
like
your
workload,
has
a
certain
storage
to
compute
resource
like
relationship,
then
hyper
converter
storage
is
the
way
to
go
because,
like
your
compute
scales
up
your
storage
scales
up
equally,
a
good
example
use
case
would
be
like
if
you're
doing,
video
transcoding,
you
roughly
know
how
many
compute
units
or
memory
user
units
you
have
per
storage
units.
B
So
that's
a
really
good
use
case.
Hyper-Converter
storage,
a
quick
overview
of
how
does
LongHorn
actually
work.
This
is
how
do
we
integrate
with
kubernetes?
What
the
control
Point
looks
like
kubernetes
has
a
CSI
specification.
It's
an
interface
general
purpose,
interface
for
storage
integration.
We
provide
the
long
run,
CSI
plugin,
which
implements
the
CSI
spec
and
that
one
communicates
with
our
long
monitor,
which
is
a
Daemon
set
that
runs
on
each
of
the
nodes,
which
provides
a
backend
API,
as
well
as
all
communities
controllers.
B
So
if
a
user,
for
example,
tries
to
create,
we
would
go
to
the
CSI
call
for
create
volume,
publish
volume.
All
of
these
would
go
against
the
back
API
backend,
which
then
our
controllers
would
create
CID
resources
on.
So,
if
you
like
want
to
use
this
in
an
API
based
fashion
via
the
cube
CTL,
for
example,
just
because
see
this
directly,
that
would
be
possible
too.
B
B
now
I'm,
just
gonna
go
over
some
things
that
where
Longhorn
is
useful
or
where
it's
useful
to
have
a
software-based
storage
backend
right,
if
you
use,
for
example,
an
Amazon
EBS
volume
directly,
then
that
volume
is
tied
to
a
certain
Zone
that
can
be
okay
for
some
use
cases
and
other
use
cases
you
might
want
to
have
a
distributor
across
different
zones.
So
you
need,
like
another
level
of
indirection
by
having
a
software.
Defined
storage
stack
all
right
on
the
snapshot
side.
We
are
gonna.
B
We
are
introducing
a
1.4
a
bunch
of
stuff
when
we
want
to
make
snapshots
more
performant
as
well
as
more
reliable,
so
we
do
like
the
snapshot
check,
something
which
allows
us
to
do
an
even
faster
Delta
for
Weeble.
So
if
you
lose
a
volume,
we
will
have
to
rebuild
that
volumes
replica
on
a
different
note.
By
doing
snapshotting
Pro
snapshot,
we
can
Skip
and
pre-calculate
the
data
right
now.
B
We
already
do
like
verification
of
an
actual
snapshot
data,
but
that
process
is
tiresome
because
we
need
to
go
to
the
whole
snapshot
chain.
So
if
you
have
a
bunch
of
snapshots,
it
will
take
much
longer,
but
if
we
can
already
identify
that
this
snapshot
is
healthy
over
time,
we
can
skip
the
the
rebirth.
We
use
some
of
the
pre-existing
data
without
additional
verification.
B
How
does
this?
How
does
so
much
work,
I?
Think
I'm
cutting
up?
How
does
storage
work?
What
is
the
representation
of
the
volume?
Well,
it
starts
out
with
the
single
spots
file.
So
the
moment
you
start
you
just
we
try
to
provide
a
fast
system.
On
top
of
that
and
anytime,
you
write
anything
onto
that
file
system.
B
There's
a
bunch
of
blocks
going
into
a
sparse
file
on
each
of
the
nodes
via
the
engine
to
replica
communication,
and
when
you
take
a
point
in
time
snapshot
we
will
create
another
spot
as
a
layer
on
top
of
that
one
and
then
just
as
long
as
there's
no
data
in
it,
we
just
read
through
them.
We
have
a
performant
index
to
do
that,
so
that
we
don't
have
to
scan
the
whole
snapshot
structure
every
time
but
anytime.
B
You
then
write
to
it
like
the
prior
data
is
still
existing,
but
it
no
unless
until
or
it's
no
longer
needed
for
the
live
reads,
but
it
might
be
read
it
for,
like
backups,
restores
restore
to
a
certain
point
in
time.
So
we
built
a
whole
snapshot
graph,
actually
per
volume
you
you
can
have
different
starting
points,
different,
ending
points
and
on
the
right
you
see
what
the
volume
life
content
actually
looks
like.
B
How
does
Backups
actually
work,
so
we
have
some
improvements
on
the
backup
that
we
didn't
mention.
I,
think
it
came
with
1.2
102.,
it's
like
backup
groups.
We
have
a
volume,
backup
policy,
recurring
job
policies.
So
previously,
if
you
use
dollar
before
we
had
the
ability
to
set
recurring
jobs,
but
that
was
per
volume
setting
it's
much
nicer.
B
If
you
can
Define
policies
for
your
backups,
so
let's
say
you
have
your
hourly
daily
monthly
policies
and
then
apply
that
to
all
volumes
like
a
default
policy
or
you
have
a
specific,
sensitive
volume
group.
So
you
have
a
policy
for
them.
You
can
apply
them
in
a
more
hourly
every
hour
for
20
hours.
We
have
a
10
count
Etc.
B
So
that's
very
cool.
If
you
want
the
demo
of
that
one
we're
at
the
kiosk
data
backups
rely
on
our
snapshot
mechanism
and
we
do
some
Delta
there
as
well
trying
to
be
very
efficient.
We
do
like
CSC
verification,
because
well,
the
worst
thing
that
can
happen
to
someone
is
that
you
have
a
backup,
but
that
isn't
actually
useful
for
resource,
so
definitely
something
if
you're,
if
you're
doing,
backup,
testing
or
backup
setups,
make
sure
that
you
once
a
month
test
your
like
backups,
so
actually
restore
ability.
B
B
Rebuilding
yeah:
let's,
let's
talk
about
okay,
how
does
actually
volume
rebuilding
work?
This
is
a
bit
in
depth,
so
I'm
gonna
go
slowly.
We
have
like
synchronous,
replicas
protocol,
so
by
default
it's
three.
This
one
is
gonna,
just
show
one.
What
do
we
do?
We
have
to
create
a
very
shortly
pause,
the
engine
so
that
we
can
hook
in
a
new
replica
you
it
doesn't.
The
user
doesn't
notice
it.
B
We
take
a
snapshot
on
all
the
healthy
replicas
at
point
in
time,
since
the
I
o
is
paused,
they
all
have
the
exact
snapshot
content
at
that
point
in
time.
The
new
snapshot
checksum
mechanism
will
at
that
point
also
do
an
asking
verification
in
the
background.
So
this
is
basically
increases
reliability,
because
we
know
for
sure
that
this
backup
snapshot
is
healthy.
There
was
no
bit
Rod,
none
of
that
happened
in
the
past,
so
we're
good
to
go.
Then
we
create
a
new
write-only
replica,
which
is
basically
gonna
get
once
we
unpause
the
engine.
B
B
We
can
already
start
populating
the
live
head,
while
in
the
in
the
background,
we
will
pick
one
of
our
healthy
replicas
at
random
and
do
like
an
asynchronous
transfer
of
their
snapshot
chain
files
right.
So
we
do
verification,
validation
that
other
CSS
happens
and
that
one
processes
quite
expensive,
even
if
we
don't
have
to
transfer
the
data
right
now,
we
have
to
look
at
the
whole
snapshot
and
do
a
per
data
Block
in
the
snapshot,
CSE,
checksum
validation.
B
So
with
the
new
Improvement,
we
can
see,
look
just
look
at
this
natural
file
and
figure.
Okay.
Last
time
we
checked
this
file
was
three
days
ago.
It
had
a
healthy
CSC
CSC,
so
we
can
skip
it.
This
is
already
present
on
a
new
one,
all
right.
So
then,
yes,
once
the
replica
is
synced
up,
we
can
flip
it
over
from
White,
only
mode
into
read,
write
mode
and
it
becomes
like
a
regular
replica
and
it's
part
of
the
reset
as
well.
Okay.
B
Well,
we
have
a
live
migration
feature.
We
have
actually
two
technically
two
life
regressions.
The
first
life
migration
feature
is
for
Harvester
for
virtual
machine.
Live
migration
was
implemented
for
that
one,
particularly
what's
the
challenges,
if
you're
using
a
file
system,
the
problem
is,
you
can't
have
fvx
mode
unless
you
have
a
distributed
file
system.
So
what
we
do
can
do,
though,
is
like
have
two
parallel
running
engines
and
then
do
like
a
hand
over
for
the
engines.
B
We
do
that.
We
use
this
approach
for
Life
engine
upgrades,
for
example.
So
anytime
you
upgrade
to
a
new
long
run
version.
We
make
improvements
in
the
engine,
so
that's
the
volume
controller
part
or
the
replica
part.
We
can
create
mirror
processes
from
the
original
set
of
processors
and
do
a
live
Handover.
B
B
Now,
with
the
new
Improvement,
it
will
make
it
easier
to
set
up
like
a
mirror,
Disaster
Recovery
cluster,
that
in
a
basic
active
passive
configuration,
so
you
can
like
flip
it
over
and
all
your
volumes
that
they
are
ready
to
go.
This
relies
on
like
an
external,
a
backup
store
as
a
shared
between
the
cluster.
This
is
nice
for
on-prem.
If
you
already
have
like
a
old
beefy
NFS
server
or
for
in
Cloud,
if
you
like,
on
Amazon,
S3
or
Azure
S3,
and
both
classes
can
share
the
same
backup
store
anytime.
B
The
volume
on
the
active
cluster
does
a
backup
it
will
be
populated
to
backup
store
and
the
volumes
on
the
passive
cluster
will
pull
that
regularly
and
apply
the
newest
backup.
So
you
have
like
a
continuous
set
of
data
with
the
clearly
defined
period
of
possible
loss
right
because,
like
the
down
between
the
last
backup
that
you
took
versus
the
live
data,
okay
I
think
that's
it.
B
We
want
to
do
some
q,
a
I
want
to
talk,
really
quick,
ignore
the
numbers
on
here,
but
we
already
did
some
performance
improvements
for
1.2
and
we're
doing
minor
performance
Improvement.
On
top
of
the
102
improvements
for
onenote3.
We
did
that
by
improving
our
memory
allocation
and
switching
also
the
memory
allocator
on
the
C
side
on
the
TGT
side.
So
so
that's
it.
B
There's
a
was
a
session
yesterday
for
the
SPD
case.
If
you're
interested
in
a
performance
Improvement,
that's
like
the
future
of
the
loans
that
I
plane
engine
this
year
is
just
for
the
existing
data
engine
so
that
the
people
that
are
using
it
actively
right
now
get
some
benefit
out
of
it
as
well.
Thank
you
very
much
and
there's
any
questions.
B
B
Right
so
with
my
money
right
now
is
set
up,
as
so
on
the
file
system
that
we
support.
We
made
many
for
file
system
usage.
What
and
how
it's
implemented
initially
was
done
via
an
external
provisioner.
We
now
have
an
integrated
that
nicely
so
that,
from
the
user's
perspective,
there's
no
setup
required.
All
you
have
to
do
is
create
a
PVC
with
access
mode,
read,
write
many
and
we
will
internally
create
like
an
NFS
server,
because
you
need
a
disparate
file
system
map.
B
The
volumes
take
the
manage
the
life
cycle,
export
Etc
of
that
server
and
then
provide
that
anytime.
The
the
client
workload
basically
does
a
CSI
call
for
mounting
all
of
that
for
attaching
that
will
be
wired
via
our
controller
to
the
share
manager.
We
call
the
chairman
report,
which
is
basically
on
NFL
server
and
Export,
that
volume
to
that
client
for
use.
C
I
run
into
customers
with
concern
to
Mike.
Looking
at
this
technology
is
what
happens
when
you
deal
with
large
data
sets
where,
because
of
the
the
migration
costs
and
such
the
time
and
effort
just
simply
restitch
an
existing
volume
underneath
kubernetes.
So
I
have
a
storage
array.
It's
already
got
the
data
there.
It's
you
know,
25
disk
or
whatever.
It
is
right,
just
Stitch
it
back
up
under
the
kubernetes
and
let
it
be
managed
on
the
control
plane.
Is
there
a
methodology
or
concept
to
do
that
so.
A
But
Longhorn
is
actually
have
no
any
strategy
for
that,
but
we
have
some
like
a
tag
disk
attack.
You
can
manage
what
kind
of
disk
you
want
to
do,
but
beyond
that,
like
a
red
level,
like
you
say,
data
center
parts,
we
don't
have
this
kind
of
design,
but
you
can
feel
you
can
feed
them
more.
We
can
see
how
we
can
do
I.
B
Think
I
know
where
you're
going,
though
like
okay,
let's
assume
you
have
this
old
Appliance,
let's
say
your
rate
server
stuff
with
25
disc
gear.
How
this
would
work
on
Longhorn
is
like
we
would
pick
up.
The
new
disk
see
that
it's
there
right.
You
would
need
to
might
need
to
do
some,
this
configuration
initially
to
see
where
that
and
stuff.
B
But
if
you
had
used
that
array
before
for
Longhorn-
and
you
just
took
it
off
for
like
a
day
or
something
to
do
like
maintenance
on,
we
would
be
doing
like
the
rebuilding
of
the
existing
ones.
We
would
pick
the
pick
the
see
that
oh
there's,
like
a
replica
of
this
volume,
we
can
reuse
them.
It
just
depends
on,
like
some
setup
of
the
replica
replenishment
rate
interval
Etc
so
to
know
like,
if
you
you
know,
you
don't
want
to
take
this
off
and
increase
the
replica
replenishment
rate
as
well.
B
So
we
will
give
you
time
for
this
to
come
back
online
and
then
we
can
reuse
the
existing
data
based
on
the
snapshots
right-
and
this
is
where
the
Improvement
of
the
snapshot
comes
in,
because
let's
say
you
have
a
terabyte
virtual
Longhorn
volume
there
you
don't
want
to
scan
every
single
file
in
the
snapshot
chain.
Instead,
you
will
be
like
yeah
I've
verified
this
two
weeks
ago.
This
guy
is
good
to
go.
We
don't
need
to.
We
can
immediately
skip
it.
We
already
have
the
the
mirror
copy
here.
It's
fine.
B
C
And
my
second
question
is
speaking
to
customers
where,
as
we're
stitching
virtual
machines
under
these
covert
Frameworks
when
things
happen
and
the
volume
is
transferred
or
the
virtual
machine
is
my
migration
whatever
it
is,
things
go
bump
in
the
night
fisking
and
how
how
we
help
a
customer
is
there
waiting?
Are
there
any
hooks
that
we
can
talk
through
that?
How
does
it
detect
when
the
volume
comes
over
and
it's
it's
four
parts
of
the
volume
it's
got.
One.
That's
got
to
go
through
a
scan
cycle
or
something
like
that.
B
Right,
so
I
can
only
speak
about
our
implementation
for
Harvester
right
because
that's
the
use
case
I
have
you
can
do
a
trick
on
the
Harvester
side.
You
can
do
a
trick
that,
basically,
you
can
rely
on
the
CSI
calls,
because
you
know
when
the
new
process
goes
online
and
you
know
when
the
old
process
goes
Offline
that
will
can
be
used
as
a
validation
to
see
if
the
migration
actually
worked
correctly
or
not
or
if
you
have
to
roll
back.
B
B
B
Then
that's
a
problem.
I,
don't
know
all
the
full
feature
set
if
they
have
like
IO
scheduling
if
they
have
different
storage
classes,
Etc.
All
of
that
stuff,
I,
don't
know
of
the
top
of
it.
There's
the
cost
Factor,
depending
on
like
you,
your
storage
needs
right,
so
that
that
would
be
a
benefit
like
basically
the
the
conceptual
the
differences
like
Longhorn
is
a
story.
B
Software
as
a
storage
stack
right,
so
you
get
the
software
that
manages
your
storage,
creates
the
storage
deals
with
all
of
that,
while
with
EFS
the
the
control
of
your
whole
storage
is
inside.
Of
the
cloud
provider
which
has
benefits
like
yeah
as
well
as
but
also
downsides,
so
if
you
have
a
specific
use
case,
come
sales,
education.