►
From YouTube: February 2023 OpenZFS Leadership Meeting
Description
Agenda: block cloning; shared slog; mirrored L2ARC; i/o rate limit
full notes: https://docs.google.com/document/d/1w2jv2XVYFmBVvG1EGf-9A5HBVsjAYoLIFZAnWHhV-BM/edit#
A
B
What
happens
if
we
have
some
Zeal
login
trees
that
are
trying
to
clone
from
one
file
system
to
another
file
system,
but
then,
when
we
import
the
pool
before
we
Mount
the
destination
file
system,
we
destroy
the
source
file
system,
so
the
block
pointers
are
no
longer
valid
and
but
suggested
to
use
Zeal
claim.
So
this
is
what
I
did
so.
There
are
two
situations.
First,
when
we
import
the
pool
we,
we
will
use
Zeal
claim
to
just
bump
the
references
in
brt
for
all
the
block
pointers.
B
So
then,
when
we
replay
this
hill,
we
only
add
the
block
pointers
to
the
file.
So
we
don't
bump
preferences
anymore
and
the
other
situation
is
when
we
dozil
climb.
We
bump
the
references,
but
we
never
Mount
the
destination
file
system.
We
destroyed
the
destination
of
our
system
before
mounting
it.
So
before
replying
the
Zeal,
so
then
we
have
to
decrement
all
the
references,
so
Alexander
also
suggested.
Another
issue
is
not
100
percent
I
think
addressed
yet
maybe
but
with
zieval.
B
B
So
my
husband's
understanding
was
that
he's
adjusted
to
just
free
the
entries
and
and
return
success,
and
so
so
during
Zillow,
replay
I
would
just
decrease
references
that
were
increased
during
zero
claim.
But
apparently,
if
I
understand
correctly,
there
can
be
another
or
maybe
previous
Zeal
entry
that
fails
and
I
haven't
checked.
That
much
from
my
understanding
is
that
the
zero
replay
will
stop.
A
B
C
If
I
may
I
said
Recon
there
on
on
APR,
we
some
time
ago,
a
few
months
ago,
will,
when
implementing
for
Linux
support
for
what's
that
null
FS
Mount,
not
all
FS
but
equivalent.
We
added
few
log
entries
which
are
implemented
only
for
Linux,
but
not
for
FreeBSD,
and
at
that
point
we
decided
that
we
are
not
going
to
diverge
to
implementation,
keeping
them
compatible,
but
allowing
like
non-existing,
not
implemented.
C
C
So
as
it's
broken
or
it's
not
working
or
it
was
never
designed
designed
to
work
at
least
that's
what
my
appearance,
appearance
after
like
half
an
hour
there
yesterday
I
had
no
chance
to
look
deeper,
but
if
somebody
knows
that
area
better
I
could
take
a
look,
it
would
be
good
to
understand
as
I
understand.
A
proper
solution
would
be
just
to
replay
normally
as
much
as
we
can.
C
Then
we
get
first
error
and
for
every
record
after
that,
we
must
call
free
to
free
all
their
effects,
we'll
find
it
can
includes
TX
right
data
blocks.
It
includes
brts
and
who
knows
what
to
make
include
in
future,
I
think
if
we
would
pass
additional
arguments
to
zero
parse
I
think
function
that
actually
traverses
through
them.
We
could
just
make
it
okay,
after
first
order,
don't
return,
but
instead
to
perform
on
call
it
normal
process
function,
just
call
free
function
and
continue.
Something
like
that,
but.
C
40
hours,
maybe
different
file
or
who
knows
what
happened
it's
better
to
get
last
known,
valid,
State
and
just
I
think
we
are
printing,
something
in
logs
that
hey,
we
can't
replay
so
user
knows
that
something
bad
happened,
yeah
it's
better
possible,
but
yeah.
It
needs
investigation.
What
to
do
with
free
after
that,
because,
as
I
told
log
destroy
is
called
me,
he'll
destroy
it's
called
after
that.
But
what
exactly
destroys
it's
slightly
cut.
B
Out
of
my
understanding,
so
it's
actually
a
surprise
that
this
is
how
we
handle
this
case,
because
I
was
like,
like
I,
assume,
initially
that
if
we
found
some
entries
that
we
are
not
able
to
replay,
we
won't
be.
We
won't
mount
the
file
system
we
will
deny
mounting.
So
we
don't
lose
any
data
because
currently,
when
we
stop
on
error,
we
still
allow
we
still
continue
with
with
the
mount
right.
So
there
is
no
way
for
the
user
to
somehow
stop
this
from
happening.
B
B
B
C
A
B
B
A
C
A
C
B
Yeah,
but
my
understanding
was
that,
of
course,
most
of
the
time
you
want
to
move
clean
pool,
but
maybe
you
have
your
system
is
broken
and
you
try
to
you.
You
cannot
improve
the
puller.
You
have
some
issue
on
the
Linux
system,
so
you
try
to
import
this
on
the
freebies,
the
as
a
recovery
option
to
diagnose
what's
happening,
and
then
you
lose
those
energy
but
I
I
yeah.
A
A
C
D
A
Like
when
we're
doing
the
Zoe
replay,
can
we
use
a
different
error
code
for
this
entry
is
corrupt
or
otherwise
broken
from
it's?
Just
literally,
your
OS
doesn't
support
this
entry
and
it,
and
so
it's
not
an
error,
her
so
much
as
an
incompatibility
and
what
we
default
to
doing
about.
It
is
different
in
that
case
like,
if
it
seems
like
yourself,
is
corrupt,
I
guess
even
then
you
still
kind
of
would
prefer
to
get
back
into
production,
then
to
sit
there
and
ask
for
help
right.
B
But
I'm
not
sure
I'm,
not
sure
when
exactly
we
declare
the
entry
as
replay
it.
If
I
will
call
0
parse
again
on
error,
would
I
walk
through
the
all
the
entries
as
well?
Let's
say:
I
succeeded
with
you,
but
I
failed
on
like
fifth
one
and
then
when
I
call
0
parse
again,
will
it
start
from
the
beginning
or
will
it
start
from
the
the
last
not
replayed
N3,
because
then
we
could
just
call
0
Parts
again
and
free
the
remaining
entries.
C
Code
itself,
it
considers
only
a
claim
position:
I
haven't
seen
there
any
start
position,
it
I
was
thinking.
Maybe
it
could
start
from
arbitrary
block
if
you're
free,
some
earlier
blocks,
but
I
haven't
found
there
any
pointer
to
first
record
and
obviously
first
record,
like
error
can
be
not
on
the
first
record.
It
should
can
be
within
a
Blog
that
I
was
thinking.
Passing
additional
function
directly
to
Zeal.
Parse
could
be
easier,
but
but
yeah
other
ways
are
possible
to.
B
B
A
B
C
It
probably
matters
only
in
case
of
some
transactional
database
when
you
confirm
with
somebody
as
a
transaction
is
done,
but
then
you
roll
back
couple
seconds
seconds
back
and
you
already
confirm
it.
It's
done
so
then
there's
a
problem,
so
it
doesn't
help
here.
But
if,
if
your
system
will
be
down
from
Earth
now
on
period
of
time,
is
it
better
maybe
to
somebody,
but
probably
not
everybody.
C
A
A
A
Makes
it
worse,
yeah
you
invite
some
of
the
changes,
but
not
all
of
them.
Instead
of
you
right
up
to
a
point
where
it
was
good
and
then
everything
after
it
was
gone,
I
think
I
agree
with
Alexander
here
that
that
doing
a
smattering
of
of
the
later
transactions
would
be
worse,
it
would
say,
lose
less
data,
but
consistency
wise.
You
would
have
much
less
an
idea
of
what
did
it
was
okay,
which
was
what
wasn't
then
just
replayed
up
to
a
certain
time,
and
then
we
had
to
stop.
B
B
But
probably
not
for
the
the
complication
anyway,
I
will
I
will
take
a
look.
I
did
this
yeah,
let's
see
if
the
icon
came
up
with
come
up
with
some
ways
to
not
leak
the
entries
there
is
one
more
issue:
I
found,
I,
think
unrelated
with
byte,
swapping
the
entries,
the
headers
of
zeal
entries
zil
claim,
because
from
what
I
see
on
the
rights
we
do
access
the
headers,
but
I
don't
think
we
do
bite
swaps,
as
we
do
during
replay.
D
D
Also,
if
your
pools
are
moving
around,
you
know
like
it's,
the
the
main
storage
for
the
pool
is
on
like
a
Sam
and
then
the
the
slog
devices
are
locally
attached
to
ssds.
You
now
have
a
situation
where
you
want
to
move
these
two
things
independently,
but
adding
new
rules
to
the
system.
You
might
already
be
out
of
ssds.
D
So
you
use
to
sort
of
integrate
the
volume
manager
with
slog
concept
as
well.
So
it
is
that
you
can
create
a
pool
and
use
that
as
effectively
the
slog
device.
When
you
create
the
a
pool
you
can
say.
Please
use
my
shared
slog
pool
as
this
log
device
and
then
all
of
the
the
slog
rights
will
go
directly
to
this
other
pool,
rather
than
be
stored
locally
on
the
normal
data
pool.
D
And
so
there's
a
bunch
of
advantages
to
this.
The
the
complications
are
that,
like
you
now
effectively
have
these
two
pools
that
are
sort
of
interlinked
with
each
other,
and
so
the
BR
has
a
bunch
of
changes
to
like
make
sure
that
you
know
you
only
can
import
one
when
or
the
these
like
client
pools
when
the
shared
model
pool
is
already
imported.
A
D
D
So
what
we
do
is
we,
like
the
Zilla
header
effectively
just
always
has
a
hole
in
it
and
if
that's
the
case
and
you're
using
a
shared
slog,
the
you
go.
Ask
the
shared
slog
like
what
is
what
do
you
have
listed
as
the
header
of
my
zone,
and
then
it
can
go,
do
and
do
go
through
and
do
Replay
in
the
same
way
that
it
normally
would,
after
sort
of
fetching
the
shirts
launch,
block
planner,
and
so
the
zilb
knows
that
it's
supposed
to
go.
D
B
D
If
anybody
else
has
any
other
questions
great
otherwise,
you
know
you
can
reach
out
to
me
on
slack
or
an
IRC
or
comment
on
the
pull
request.
I
would
be
happy
to
have
full
reviews.
I
have
a
couple.
People
at
delphix
lined
up
take
a
look
at
it,
but
I
would
be
very
happy
to
have
people
from
the
community.
Take
a
look,
and
especially
people
have
ideas
about
use
cases
and
stuff
like
that
that
you
know
where
they
might
have
feelings
about
particular
implementation
details.
D
A
A
The
particular
customer
we're
talking
with
just
depends
on
the
performance
boost
they
get
from
their
L2
Arc.
They,
their
working
set
is
as
such
a
size
that
it's
too
big
for
Ram
but
does
actually
fit
with
our
plus
L2
Arc,
and
so
when
the
SSD
dies
and
they
fall
back
to
reading
from
hard
drives
their
performance
tanks
really
badly.
So
they
were
looking
at
doing
a
mirror
and
we
noticed
that
the
zoo,
zoo
pool
command
just
will
not
let
you
construct
a
pool
with
a
mirrored
L2,
Arc
and
I.
A
Wonder
if
that's
actually,
because
we
don't
really
track
stuff
in
the
right
way
to
be
able
to
update
one
of
the
mirrors.
We
got
out
of
sync
like
the
dtl
or
just
you
know,
attaching
a
mirror
if
you
try
to.
If
you
pull
attach.
How
is
that?
If
it's
going
to
figure
out
what
should
have
been
on
the
the
new
SSD,
because.
C
C
A
Yeah
but
yeah
I
think
you're
right
that
the
rebuild
code
you've
just
scrubbed,
would
then
have
to
do
something
about
it
and
and
rebuild
doing
attacher
and
so
on
means.
It
would
just
be
a
lot
of
extra
work
and
I
agree
that
it
doesn't
really
seem
that
useful
I
was
just
looking
at.
It
looks
like
it
could
have
just
been
a
bug
in
the
the
way
we
parsed
the
Jeep
will
create
string.
A
Where,
basically,
you
know
if
we
see
the
keyword
mirror
and
it's
not
directly
following
one
of
these
keywords,
then
we
think
it's
the
next
vdev,
and
so
the
error
you
get
is
that
you
can't
create
an
L2
Arc
with
zero
devices
in
it.
If
you
do,
you
know
the
log
or
sorry
cashmere
thing
it
thinks
mirror
is
the
next
v-dev
and
it
would
even
try
and
I
wondered
if
that
was
just
a
fluke
that
it
didn't
work
but
you're
right.
A
C
A
D
A
B
Yes,
so
we
are
looking
for
like
cleanway
to
to
represent
the
the
following
situation,
so
we
have
multiple
properties
related
to
rate
limiting,
so
we
can
rate
limit
the
bandwidth,
the
reading
bandwidth
we
can
or
throughput.
We
can
limit
the
right
throughput
and
we
can
limit
the
total
throughput
and
also
we
can
limit,
read
the
number
of
read
operations,
right
operations
and
total
operations.
B
B
So
when
we
just
limit
the
let's
say,
read
throughput,
we
actually
create
a
structure
that
have
the
other
properties
Unlimited
and
we're
wondering
how
to
bust
please
what
would
be
the
best
way
to
to
make
this
configurable
and
like
user
friendly,
because
if
those
are
separate
properties,
then
it
might
be
confusing
that
that
setting
one
actually
sets
them
all.
Or
if
this
is
one
property,
then
it
may
be
some
weird
string
where
we
have
like,
like
shared
certain
of
us
property,
where
we
basically
put
all
the
options
into
one
property.
B
So
they
have
ideas
along
the
lines
where
we
have
a
single
property,
but
we
can
use
at
to
set
those
sub
properties.
Let's
say
that
we
have
rate
limit
at
read
throughput
and
this
would
set
the
only
read
throughput
for
the
rate
limit
property
or
something
like
this.
So
I
don't
know
if
anyone
have
has
any
ideas
how
to
how
to
do
this.
So
it's
so.
It's
like
more
pretty
intuitive
to
set
and
also
I,
don't
know
it
looks
elegant,
I
guess.
A
Well,
the
main
problem
is
that
if
you
set
the
read
quota
on
a
certain
data
set,
then
that
would
if
before
it
was
inheriting
now
it
wouldn't
be.
If
you
set
only
one
of
them,
then
we've
set
all
the
other
ones
to
unlimited
and
suddenly
the
the
total
bandwidth
quota
you
set
higher
up
the
tree
doesn't
apply
anymore.
B
So
it's
it
would
be
a
bit
hard
to
to
explain
it
quickly,
but
because
the
properties
overlap,
I
cannot,
let's
say
if
I
have
to
pause
some
threads
because
they
are
over
rate
limit
at
each
level.
I
have
different
different
cues,
so
I
don't
have
to
go
around
and
look
through
entire
pool
where
the
threads
are
should
wait
and
where
they
should
wait
longer.
B
Let's
say
you
have
on
one
data
set,
you
have
read
throughput
configured
and
somewhere
below,
you
have
read
operations
configured
and
then,
when
you
try
to
read
something,
then
we
would
need
to
decide
where
to
check
the
rate
limit
and
which
weight
Q
to
use
for
this
thread
Etc.
So
it's
it's
getting
very
complicated,
so
so,
basically
the
way
I
implemented.
That
is,
is
that
it's
like
a
whole
package.
So
we
check,
if
you
configure
the
rate
limit
on
this
data
set.
B
A
C
B
Yes,
that
was
our
starting
point.
It
would
be
nice,
but
yeah,
it's
it's.
It's
it's
getting
really
messy,
especially
if
you
like
one
data
set
and
have
multiple
children,
so
you
have
one
rate
limit
here.
The
children
have
different
rate
limits,
but
also
you
have
to
like
take
all
of
this
into
account.
So
then
also.
B
A
Running
all
the
way
back,
like
I,
think
Alexander.
You
brought
that
up
when
we
talked
about
this
initially
a
couple
months
ago
of
you
know,
it's
your
biggest
complaint
about
the
quota
system
is
that
you
recurse
all
the
way
up
to
the
parent
every
time
and
yeah,
so
we're
we're
at
least
the
first
implementation
of
this
we're
trying
to
avoid
all
that.
But
yeah
is
one
compound
property
does
feel
pretty
kind
of
nasty
for
the
user,
but
maybe
pavel's
idea
of
having
the
at-based.
A
C
A
Expectation-
and
you
know
means
that
we
don't
ever
have
to
recurse
and
look
at
anything
else
right
as
soon
as
we
we
rehearse
up
until
we
find
the
a
rate
limit
structure
and
then
we
can
stop,
we
don't
have
to
go
and
see.
Oh,
is
there
a
bigger
limit?
We
also
have
to
respect
from
a
parent
higher
up
the
tree.
C
Yeah,
it
makes
sense.
I
I
just
haven't
got
what
power
do
you
mean
with
ad
science?
I
was
thinking
you
mean
like
at
in
a
property
name
like
if
yeah
yeah,
but
then
it
would
be
not.
It
could
be
Collide
from
the
old
idea
of
limited
using
expectations.
Then
it
would
mean
there
could
be
multiple
properties
like
that
and
then
I
inheritance
would
return
back.
That's
the
original
point,
I'm
not
sure.
B
B
A
A
C
B
C
A
C
From
perspective
of
Satan
separate
sub
property
versus
replacing
completely,
maybe
that
syntax
could
have
sensed
only
for
right
part
not
for
read
yeah,
but
this
would
be
first
of
its
kind
that
maybe
conflicting
with
like
features
where
it's
different
properties
at
science
as
part
of
property
name,
was
its
real
or
fake.
Whatever
I
can't
look,
because.
A
A
Dollar
signs
are
for
some
of
the
Hidden
properties.
I,
don't
think
we
want
to
go
there,
though,
but
definitely
open
if
anybody
has
a
better
idea
of
how
you'd
want
to
manage
this
kind
of
compound
property,
because
they
kind
of
agreed
even
just
Alexander's.
First
point
of
like
six
new
properties
is
kind
of
a
lot
already
and
looks
a
bit
crowded,
but.