►
From YouTube: July 2023 OpenZFS Leadership Meeting
Description
Agenda: RAIDZ Expansion; Gang ABD; BRT; ZED slow io's; metaslab_unload
full notes: https://docs.google.com/document/d/1w2jv2XVYFmBVvG1EGf-9A5HBVsjAYoLIFZAnWHhV-BM/edit#
A
C
Yeah
at
the
last
meeting,
I
was
just
starting
and
hadn't
put
together
the
pull
request
yet
and
that
that
has
happened
and
it
wasn't
too
bad
Brian
there
was
a
collision
in
the
d-rate
section.
You
had
added
some
code,
I
think
if
you
have
sectors
that
are
uninitialized,
there
was
a
potential
for
corruption,
and
so
that
pull
request
was
dependent
on
some
structure,
fields
that
were
removed
as
part
of
the
rate,
expansion
and
so
I
had
to
go
with
the
put
those
back.
C
I
mean
it
was
basically
I
think
it
was
determined
that
they
weren't
needed
anymore.
So
they
had
no
consumers,
but
obviously,
in
the
future
they
did
have
a
consumer.
So
I
put
that
back
and
there's
a
similar
one
in
a
different
section
of
the
code
where
a
Zio
was
being
passed
through
a
function
and
then
it
was
removed
because
no
one
was
consuming
it,
but
turns
out.
That
was
some
of
the
changes
you
had
made.
C
There
was
a
consumer,
so
changes
like
that,
and
then
over
the
past
couple
years,
there's
been
a
lot
of
Cleanup
in
the
code.
Trivial
things
like
assert,
you
know
x,
equals
equals
zero
turns
into
a
certain
zeros
and
things
like
that
and
various
cleanups,
where,
if
you
don't
have,
if
you're
not
consuming
a
an
argument,
there's
a
new
convention
where
you
void
it
in
the
in
the
function.
So
a
lot
of
things
like
that,
so
I'm
hoping
I'm
catching
them
all,
but
I've
probably
missed
a
few.
C
C
Yeah
yeah
and
then
and
then
going
fine
and
then
actually
I
found
to
my
sort
of
astonishment.
If
you
will
is
that
in
the
code
we
weren't
actually
enforcing
the
feature
flag
and
it
was
corrupting
your
pool
so
and
because
most
people
just
take
the
current
code
base
create
a
pool
at
it.
You
know
so
if
you
actually
don't
have
it
enabled
it,
it
did
bad
things
and
hung,
and
and
anyway
so
I
added
that
last
night.
C
So,
but
in
doing
so
like
notice,
the
comments
for
Spa
V
Dev
attached
he's
talking
about
mirrors.
You
know
so
to
go
clean
up
the
comments
there
and
I'm
working
on
a
big
Theory
comment
for
the
rate:
Z
beat
every
Z
code,
so
I'll
I'll
be
posting
that
shortly,
but
but
their
current
Focus
right
now
is
on
testing.
There's
the
ZFS
test,
Suite
I,
think,
is
a
good
shape.
It's
passing
then
on
z-test.
C
It
was
sort
of
two
modes
of
running
it,
one
where
we
just
you,
can
specify
as
the
kind
Ebates
and
then
it
would
throw
you
know
an
expansion
in
in
the
mix.
Then
there
was
a
more
dedicated
approach,
which
I
think
Stuart
maybe
did
where
you
actually
just
have
a
dedicated
sort
of
test,
it's
sort
of
like
the
MMP
test.
C
You
know
it's
like
a
one-off
test
inside
of
z-test,
where
you're
not
messing
with
all
the
all
the
craziness
you're,
just
actually
doing
a
focus
test,
and
so
that
I
kind
of
had
to
rework
it
to
get
it
working,
but
it's
working
now
and
that
one
you
pass
in
a
dash.
X
and
it'll
create
a
pool.
A
great
d
pool
fill
it
up
with
with
a
lot
of
data
about
20
of
the
capacity
of
the
pool
and
then
attach
your
disk
and
then
interrupt
the
Reflow
to
confirm
that
we
can
survive.
C
You
know
a
crash,
unexpected
termination
in
the
middle
of
Reflow,
so
that
test
is
in
there.
I
haven't
added
it
to
zloop,
yet
but
I
plan
to
do
that,
so
it'll
get
mixed
in
with
our
normal.
You
know
regression
testing
that
we
do
so
on
the
z-test
side,
the
other
one,
the
one
I
first
mentioned
I'm,
seeing
about
five-ish
consistently
failing.
C
You
know
that
one's
a
little
problematic
to
figure
out
but
I'll
keep
working
on
it
and
then
there's
some
code
in
there
I
think
the
door
added
it.
But
on
on
every
import
we
checked
that
we
sanity
checked
the
scratch
area
that
we're
using
and
it's
failing,
asserts
and
I.
Don't
quite
understand
he
has
a
code
setup,
so
you
can
actually
in
discrete
areas
of
that
where
we're
writing
out
the
stuff.
You
can
have
it
stop
at
that
exact
point
and
then
do
a
validation
and
some
of
those
are
failing.
C
So
I
don't
know
if
they're,
false
positives
or
not,
but
I
kind
of
think
they
are.
But
anyway
so
there's
a
handful
Z
testings,
it's
fun.
You
know
the
fun
part
of
working
on
z-test,
so
you
have
to
go.
She
go
chase
down
all
the
problems.
It
uncovers
yeah
I'm
I'm
in
that
Loop
right
now,
where
you
know
get
a
whole
bunch
of
them.
C
Look
for
the
pattern
see
if
I
can
figure
out
how
to
diagnose
it
further
so
hopefully
wrap
that
up
this
week
and
then
move
on
to
kind
of
the
final
push
would
be
getting
reviewers
I've
asked
a
few
people,
I
I,
think
I
pinged,
you
Brian
on
that.
But
if
you
wouldn't
mind
looking
at
specifically
at
the
d-rate
all
the
d-rate
changes
because
I
think
they're
fine,
but
you
know
we
need
more
eyes
on
it.
C
C
A
C
C
C
B
D
I
haven't
seen
the
loop
or
hang
in
our
job,
usually
crashing
I.
Think
I
saw
on
some
other
tests
like
regular
tests.
Yeah
I
think
it
was
during
the
zero
suspend.
We
have
some
test
of
removal,
removing
Zeal
or
somehow
else
triggering
zero
suspend
the
problem
is
you
also
spend
that's
I'm,
not
sure
how
it
worked
before
like
written
the
description,
I
haven't
created
issue
an
easy
investigated
myself.
I
should
probably
do
one
or
another
until
it
got
lost
from
history.
Yeah.
C
D
C
C
D
B
A
A
A
C
I
just
made
a
commit
this
morning:
I
think
okay,
I
didn't
beat
base.
If
I
re
can
I
rebase
it
and
retain
my
commit
history,
I
think
I
can
but
yeah
okay
yeah,
but
there
hasn't
been
I've
been
spot.
Checking
there
hasn't
been
significant
changes,
I
mean
it
was
it's
kind
of
slowed
down
when
we
were
trying
to
release
that
the
final
version,
but
yeah.
C
D
C
A
C
B
I
think
the
next
one
was
mine.
We
saw
kind
of
interesting
thing
starting
last
week
if
you
have
gang
blocks.
In
this
case,
the
pool
is
like
90
something
percent
full
and
there's
just
I
think
a
lot
of
fragmentation
and
not
a
lot
of
free
space.
So
it's
resorting
to
using
gang
blocks
and
we
create
a
gang
ADB
to
to
send
down
to
the
bio.
B
So
when
we
prepend
the
512
bytes
in
the
beginning,
we
add
the
right
amount
of
padding
to
the
end
of
the
the
thing,
but
then,
when
we're
issuing
it
down
to
the
bio,
it's
too
big,
so
we're
splitting
it
and
because
we
split
it
on
a
4K
page,
we
end
up
with
a
512
byte
thing
and
then
a
bunch
of
4K
pages.
And
now
it's
not
4K
aligned
and
the
device
will
give
you
a
right
error
and
then
the
second
half
has
the
same
problem
because
it
starts
4K
the
line.
B
But
then
it's
got
this
unaligned
bit
at
the
end,
and
so
it's
throwing
errors.
It
didn't
do
this
in
0.8.
It
was
introduced
in
2.0.0
as
part
of
the
gang
ADB
concept
where
we
wouldn't
have
to,
because
in
0.8
we'd
copy
and
re
make
a
new
linear
buffer
that
was
4K
line
properly.
But
when
we're
doing
the
the
gang
ADB
thing
to
avoid
all
the
memory
copies,
the
problem
is
when
it
gets
when
it
has
to
get
split
across
bios,
because
there's
the
aggregation
was
too
large.
It's
causing
right
errors.
B
D
To
make
it
multiple
of
4K
after
it
was
512
just
can't
work,
not
at
this
point
we
would
either
have
to
fix
somewhere
at
the
dev
level
or
just
originally
do
copy.
It's
again
the
block
creation
instead
of
append
one
or
another,
but
with
my
in
my
Zeal
case,
I,
give
up
on
that.
I
went
different
approach.
D
B
D
Was
there
may
be
one
more
problem?
If,
even
if
we
try
to
make
that
function,
you
mentioned
trying
to
cut
it
properly.
Then
we
then
again
at
the
Block,
Level
HBA
or,
as
a
controller
may
be
unable
to
handle
alignments,
not
multiple
to
page
size
and
that's
another
threshold,
the
side
of
a
shift.
We
also
have
page
size
and
right
now
everything
every
block
that
is
bigger
than
page
size.
It
always
aligned
at
the
page
size
and
every
every
discontinues
are
always
multiple
to
page
size.
D
More
than
any
image
should
support
arbitrary
scatter
Gaza
and
some
HBA
supporter
bitters
cutter
gazard.
But
it's
not
hundred
percent
I'm,
not
sure
how
exactly
Linux
handles
that,
but
I
have
I
suspect,
I
I
try
to
look,
but
if
I
remember
correctly,
it
doesn't
properly
still
kind
of
expects
fan
system
to
do
that,
and
so
it's
just
opens
so
big
can
of
worms
that
maybe
just
making
gank
blocks
aligned
originally.
Maybe
the
easiest
way.
B
D
B
B
D
D
A
list
of
the
list
of
blocks
like,
what's
what's
the
problem
to
allocate
list
of
blocks
with
first
prep
engine
512
bytes,
it's
not
that
we
are
copying.
Data
data
will
remain
in
Arc.
It's
usually
just
temporary
buffer
used
to
allocate
that
again.
I
I,
don't
remember
that
code
out
of
my
head,
but
that
would
I
would
think.
B
D
B
A
B
So
the
ZFS
will
split
it
up
and
and
issue
it
to
the
disk
so
that
the
Linux
block
layer
isn't
going
to
split
every
bio.
We
send
it
in
half
anyway,
I,
don't
know
if
that
will
help
much
I
haven't
had
a
chance
to
measure
it,
but
just
another
thing:
I
I
noticed
there
that,
basically,
because
we
found
by
reducing
the
aggregation
and
never
sending
a
big
enough
thing
that
has
to
get
split,
we
avoided
this
problem.
D
A
question:
what
should
be
the
split
like
as
I've
told
some
devices
some
controller
support.
Arbitrary
scattergas
are
other
requires
pitching
on
page
sizes,
aware
Linux
will
split,
will
split,
but
if
I
remember
I
saw
it
was
splitting
on
sector
boundary,
but
we
cannot
go
in
this
case.
We
would
like
to
go
both
below
the
page
and
below
the
sector
boundary.
D
Yeah,
of
course,
it
would
be
great
to
have
arbitrary,
scatter
Gaza
support.
Arbitrary
ABD
I
found
that
we
have
some
requirements
of
alignment
for
this
ABD
in
in
encryption
in
check
Simon.
In
some
other
places,
it's
not
big
but
somewhere
64,
bytes,
sometimes
32,
something
like
that,
but
and
that
we
could
handle
more
or
less
in
some
places
it's
already
handled,
but
it's
definitely
not
handled
on
the
video
layer
where
it's
worse,
I
saw.
A
B
A
A
B
A
B
B
Shy
about
changing
things
yeah!
Well,
even
looking
at
the
the
previous
thing
that
there's
a
nice
helper
function
to
here's
a
b
Dev.
How
many
segments
will
it
take,
but
it's
not
available
on
Ubuntu,
like
2004,
only
newer
versions,
but
if
you
get
the
queue,
then
there's
a
function
that
can
do
it,
but
not
if
you're,
on
really
old
kernels
and
it's
just
yeah.
A
D
Cool
yeah:
it
should
also
help
an
FS
which
also
has
server
side
copy.
According
to
our
service
teams,
that's
told,
should
bosses
and
B
and
NFS
should
work.
We
are
investigating
that.
How
will
it
go
one
of
my
teammates
Amir
Hamza
already?
Oh
sorry,
not
a
different
mersalim
is
now
playing
with
that
trying
to
run
some
tests.
Maybe
you'll
see
his
feedback
on
APR.
A
B
D
A
referral
link
always
like
one
of
them
uses
once
called
one
user's
analysis
is
called
yeah
like
practically.
If
you
do
demand
it
to
be
colonies
and
colony
is
supported
only
within
the
file
system.
But
if
you
said
you're,
okay,
with
optional
copy
and
then
which
most
people
should
be
okay,
I,
don't
know.
Why
would
somebody
require
a
work?
Loan
usually
work
across
and
science,
if
I
was
told
boss,
NFS
server,
both
support,
both
use
copy
file
range
is
cool,
could
be
file
range.
B
D
B
And
then
the
other
thing
we're
working
on
is
expanding
Zed.
If,
based
on
the
work,
we
did
I,
don't
know
some
number
of
months
ago
to
allow
you
to
configure
Zed
with
vdev
properties
to
control.
You
know,
after
how
many
writers
or
checks
and
errors,
do
we
replace
the
disk
with
a
spare
expanding
that
so
that
if
there
are
too
many
slow,
I
O's
on
a
disk,
it
will
auto
spare
it
since
I
had
yet
another
disk
that
decided
to
go
out
to
lunch
but
not
actually
die.
A
A
B
A
B
0.8.3,
even
we
added
the
kind
of
cool
off
time
where
it's
like
once
we've
loaded
it
don't
unload
it
for
at
least
10
minutes
or
whatever,
to
avoid
this
problem,
but
they're
not
in
a
position
to
change
version
of
ZFS
right
now.
So
we
set
the
metislab
debug
unload
parameter,
which
just
doesn't
unload
the
meta
slabs.
Are
there
consequences
to
leaving
that
on?
Is
there
like
stuff
that
only
happens
when
it
unloads
the
metaslab
I
think
condensing
normally
happens
on
load,
not
unload
right.
A
B
A
B
All
yeah
they'd
be
fine
with
tens
of
gigabytes
as
long
as
it's
not
50s
of
gigabytes,
probably
but
yeah,
and
that's
kind
of
gets
back
to
another
thing.
I
think
we
talked
about
before,
but
I
want
to
get
back
to
is
we
probably
do
need
to
change
some
of
the
tuning
so
that
creating
a
14
wide
raid
Z3
doesn't
result
in
15
000
meta
slabs,
when
the
target
was
200.
B
B
B
And
so
works
great
for
small
VMS,
but
I,
see
I.
Think
one
thing
that
we
talked
about
before
is
maybe
changing
the
tuning
from
I
think
what
they,
what
they
said
before
was
the
default
maximum
size
for
metislab
is
16
gigabytes
of
space,
and
then
it
will
keep
increasing
the
count
until
the
count
gets
too
high.
Then
it'll
bump
the
size
to
the
next
power
of
two,
but
the
max
number
is
128
000..
You.
C
B
Can
you
know
we
created
a
10
petabyte
pool
and
it's
ends
up
with
98
000
meta,
slabs
of
64
gigs,
each
or
something
like
that,
but
yeah.
So
we
were
looking
at
is
maybe
every
time
we
hit
the
limit
to
alternate
between
bumping
the
size
and
the
count.
So
we
aim
for
200
and
if
that's
not
enough,
we
go
up
to
400
and
if
that's
not
enough,
then
we
double
the
size.
And
then
we
double
the
count
and
double
size.
B
B
It
does
give
you
more
fragmentation,
although
not
that
much
more,
it
just
seems
to
be
spend
a
lot
more
time.
Cycling,
especially.
The
other
thing
is,
if
you're
using
large
record
size,
then
if
your
medislabs
limited
to
16,
gigabytes
and
you're
allocating
16
megabyte
chunks,
then
that's
only
a
thousand
segments
for
the
whole
meta
slab
and
maybe,
depending
on
your
use
case,
you
want.
We
made
most
of
the
things
tunables
already.
Those.
D
B
I
think
you're
already
merged
all
those
yeah,
so
that
gives
people
a
bit
more
control.
But
we'll
want
to
look
at
the
defaults
a
bit
more
in
do
some
analysis
to
see
if
there's
advantages
one
way
or
another,
because
we
don't
want
to
just
change
the
numbers,
because
we
don't
want
a
big
number
but
to
actually.
A
D
Yeah
from
too
many
metal
slops
would
require
us
to
sing
them
separately,
and
if
we
do
it,
you
know
first,
we
will
write
all
the
updates
into
Global
lock,
but
then
we
need
to
propose
that
lock
sometimes,
and
for
that
we
need
to
distribute
all
the
difference
between
all
the
meta
slabs.
Just
the
number
of
iOS
is
growing
randomly,
and
if
we
don't
do
it
dynamically,
then
we
would
have
to
do
it
on
pool
import,
but
also
a
lot
of
iOS.
B
A
B
B
B
B
A
A
Be
annoying
but
yeah
I'm
all
for
writing
those
down.
I
guess
I'll
just
make
one
other
call
for
testing
a
week
or
two
ago
we
tagged
the
2-2
rc1
release.
So
that's
out
there
for
people
to
start
kicking
the
tires
on
and
do
a
little
bit
testing
we'll
pull
some
fixes
back
for
an
rc2
I
hope
shortly
and
get
that
out
there
for
some
more
testing
but
yeah.
Just
to
plea
for
additional
test
coverage
on
that
manual
testing
and
the
automated
testing.