►
From YouTube: August 2022 OpenZFS Leadership Meeting
Description
Agenda: ZED config; failmode=error; microzap size
full notes: https://docs.google.com/document/d/1w2jv2XVYFmBVvG1EGf-9A5HBVsjAYoLIFZAnWHhV-BM/edit#
C
B
We
discussed
it
a
bit
last
month
where
Seagate
as
John
from
Seagate
was
here
and
I
was
talking
about
having
this
issue
where
he
was
getting
spurious
errors,
and
sometimes
there
would
be
just
enough
to
set
off
the
threshold
instead
of
10.
You
know
errors
within
15
minutes
or
whatever
the
hard-coded
in
the
C
file
limits
are,
and
we
were
looking
at.
You
know,
improving
the
configuration
of
that
and
Rob
did
a
bit
of
a
survey
of
what
other
issues
around
Zed
configurability
people
have
had.
B
Last
month,
John
from
Seagate
brought
up
the
fact
that
you
know
if
they
have
two
different
pools.
They
might
not
want
the
same
values
for
each
pool,
and
then
the
command
line
option
gets
awfully
Messy
as
soon
as
you're,
trying
to
do
more
than
one
pool
or
something,
and
so
using
the
existing
Vita
properties
framework
does
seem
to
make
the
most
sense
there
and
it
fits
with
the
other
part
of
that
work.
B
We
did
Mario
savorski
did
the
tracking
down
on
that
and
found
that
the
reason
why
I
see
it
was
seeing
different
Behavior
with
ZFS
versus,
if
they
did,
you
know,
EXT
or
xfs
or
butterfs
on
Linux
was
that
ZFS
is
specifically
setting
the
fail
fast
option
on
the
the
iOS
before
it
sends
them
down
to
the
Linux
kernel.
So
that's
specifically
asking
them
scroll.
You
know
never
retry.
This
just
report.
The
error
back
to
me.
B
Instead
of
s,
will
deal
with
it
and
in
this
particular
case,
because
the
errors
are
basically
a
transport
error
in
the
you
know,
getting
the
signal
across
the
the
fabric
they're
using
and
so
on.
They
actually
do
want
the
retries
and
so
we're
looking
at
adding
a
Vita
property
for
that
as
well.
That
would
say
you
know
on
on
these
disks,
we'll
set
the
mask
on
the
fast
fail
to
say
you
know,
ignore
transport
errors
and
and
let
the
driver
retry
those.
B
So
there's
the
number
of
I
o,
or
the
number
of
I
o
errors
in
the
amount
of
time
and
then
the
number
of
checksum
errors
and
then
in
the
amount
of
time.
So
the
number
of
errors
and
in
how
many
seconds
for
both
I
o
errors
and
check
some
errors
that
result
in
Zed
decided
that
that
disk
is
bad
and
should
be
kicked
in
and
then
kind
of
only
slightly
relatedly.
B
The
fifth
Vita
property
we
would
do
is
for
controlling
the
the
mask
of
the
fail
fast
bits
that
we
send
down
to
the
the
living
subsystem
so
that
certain
errors,
the
driver,
like
The
Controller
driver,
will
Auto
retry
instead
of
not
because
currently
ZFS
is
explicitly
telling
the
driver
not
to,
and
we
want
to
be
able
to
assume
that
to
say
you
know:
retry
transport
errors,
but
don't
retry.
If
it's
you
know,
the
drive
gave
an
error
right.
B
But
we
don't
currently
have
inheritance
on
VW
properties,
because
for
some
of
them
it
didn't
make
sense
to
have
it.
But
there
are
some
cases
where
maybe
it
does,
and
so
I'm
definitely
open
to
what
people's
thoughts
are
about,
how
to
deal
with
inheritance
there
like
I
guess
we
do
have
the
presidents
of
like
was
it
I
think,
can
mount
and
a
couple
other
normal
properties
that
don't
inherit.
D
B
We
talked
about
a
pool
property
originally,
but
Brian
was
kind
of
against
that
feed
up
properties
are
a
little
more
out
of
the
way.
So
it's
not
going
to
cut
up
the
user
as
much,
but
it
might
actually
make
sense
to
do
a
pool
property
for
these
instead.
But
you
know
whatever
you
guys
accept
when
we
Upstream
it
I
guess
it
comes
down
to.
D
B
A
B
B
Checks,
whether
the
second
one
is
a
pool
or
not
to
then
decide
if
you're
doing
vdent
properties,
so
it
might
not
actually
be
possible
to
access
the
root
one,
but
we
might
be
able
to
come
up
with
a
way
around
that,
like
we
already,
you
know,
made
a
special
keyword
of
like
all
Dash
B
devs,
and
maybe
we
could
have
you
know,
root
Dash,
V,
Dev
or
something
to
to
exactly
to
access
the
root
of
the
pools.
Vita
properties.
B
D
B
Yeah
we
looked
at
that
too,
and
thought
about
you
know.
If
we
pull
in
like
lib
UCL
or
something
to
do
config
parsing,
we
could
make
a
config
file
for
it,
but
then
starting
to
think
about
how
you
would
lay
out
that
config
file
in
a
way
where
you
could
specify
values
for
different
pools
or
whatever
it
quickly
became
you
know.
Do
we
just
make
this
a
tool,
property.
E
Yeah
yeah
good
morning,
Mr
Rob,
hey
yeah,
the
I
think
the
illumest.
You
know
they
have
just
a
configuration
file
that
they
were
pulling
in
and
they
only
end
up.
Setting
three
values:
it'd
be
quite
a
bit
of
quite
a
bit
of
work
to
bring
in
for
only
having
to
set
a
few
things
for
the
the
thresholds.
But
that's
just
what
I
noticed
as
I
was
looking
at
some
of
it.
D
E
A
D
It
can
come
sometimes
have
orphaned
entries
in
it
and
on
Linux
that
prevents
the
server
from
starting
up.
It
tries
to
stat
everything
and
if
it
can't
it
just
Falls
over
so
I
was
looking
at
ways
to
prevent
that.
The
first
approach
would
be
just
to
have,
let's
reset
that
file,
whenever
we
do
sdfs
share
Dash
a
and
in
our
case
that
actually,
you
know,
solves
the
problem,
because
before
the
Linux
servers,
NFL
server
starts,
it'll
actually
call
the
the
share
service
to
do
that.
D
But
then,
in
talking
with
Alan
about
FreeBSD
and
sort
of
How
It's
architected,
it
kind
of
makes
more
sense
to
push
it
down
into
CFS
proper,
rather
than
on
top
of
it
and
I've
started
on
that.
The
other
issue
is
that
destroy
itself,
at
least
on
Linux
I,
don't
know
about
FreeBSD.
It
looks
at
instead
of
looking
at
all
the
data
sets
to
see.
D
If
you
have
this
property
share
interface
property,
it
looks
at
it
Scopes
it
to
just
what
procafest
knows
about
what's
mounted,
and
so,
if
your
Faust
system
is
not
mounted
or
unmounted
behind
zfs's
back
like
using
u-mount
then
like
when
you
do
a
destroyer
when
you're
tearing
down
it,
doesn't
it's
just
looking
at
product
fast?
So
if
you
don't
have
a
mount
you'll
leak,
you'll
orphan,
that
that
entry,
so
Alan
I
was
thinking
as
a
next
phase
is
we
have
the
r
exports
file
and
I
and
there's
one
on
freebsds
in
addition
to
looking
at?
D
C
D
B
Yeah
because
I
think
the
other
idea
I
have
for
that
was
you
know
we
have
this
mounted
read-only
property
that
tells
us
if
the
data
set
is
matched
or
not,
and
we
could
have
one
for
shared.
So
then
you
know
at
destroy
time.
We
could
just
you
know
as
we're
walking
the
list
of
data
sets
and
and
dealing
with
them
to
unmount
them
and
everything
we
could
make
sure
to
unshare
them
as
the
pool
before
the
pool
goes
away.
D
B
F
B
G
B
D
D
D
Than
that
the
file
is
hard-coded
into
inside
it.
Okay,
you
know
yeah,
so
just
in
that
file,
we'd
have
to
parse
that
file.
Okay,
because
the
trick
would
be
like.
What's
the
format
of
that
file,
I
don't
know,
but
the
extent
we
can
parse
that
file,
then
we
can
generate
a
list
of
additional
things
to
check
so
I
think
yeah
with
those
with
those
two
changes.
H
B
In
FreeBSD,
any
ones
you
manually
configure
will
be
in
ETC
exports
and
then
those
ZFS
ones
will
be
an
Etc
ZFS
exports
and
mount
D
just
reads
both
I'm
less
sure.
What
would
happen
if
you
had
the
same
one
in
both,
but
the
ETC
ZFS
one
is,
has
a
big
statement
at
the
top
about
how
it's
owned
by
ZFS
and
you
shouldn't
modify
it,
but
the
the
problem
Don
ran
into
was,
if
you
manually,
unmount
the
data
set
with
umap
rather
than
like
zfsu
Mount
and
then
destroy
the
pool.
B
Zfs
didn't
notice
that
it
needed
to
remove
that
data
set
from
the
exports
file,
and
so,
if
you
reset
the
machine
with
that
tool
now
destroyed,
it
would
mention.
Hey,
go
share
this
dataset
via
NFS,
but
it
doesn't
exist
anymore.
On
FreeBSD
that'll
just
generate
a
warning,
but
on
Linux
it
will
actually
stop
nfsd
from
starting.
H
D
No
Linux
uses
a
dot
d
convention,
so
there's
slash,
Etsy,
exports.d,
slash
and
then
there's
a
zfs.exports,
and
so
what
ZFS
does
is
it
maintains
that
list
like,
if
you
add,
a
share
or
remove
one,
and
then
it
tells
NFS
to
Via
export
FS,
hey?
We
we
changed
that
list.
It's
sort
of
like
you
know,
system
D,
where
you
say
like
hey,
go,
rebuild
everything
because
I
changed
one
of
my
files,
so
it
gets
consumed
that
way,
but.
G
B
Yeah,
whereas
on
FreeBSD
the
the
rc.d
script
for
Mount
D
checks,
if
ZFS
isn't
enabled,
and
if
it
is
it
also,
in
addition
to
pulling
in
ETC
exports,
it
also
pulls
in
ETC,
ZFS
exports
and
Parcels
in
both,
but,
like
I,
said
the
the
big
difference
there
is
when
it
stats
one
of
them
and
it
doesn't
exist.
It
just
emits
a
warning
and
skips
it
rather
than
breaking
nfsd.
D
D
D
B
C
I
G
B
D
B
B
B
D
B
G
B
But
the
use
case
we've
run
into
is
some
applications
that
would
rather
get
the
error
and
be
able
to
return
that
to
the
the
original
request,
or
you
know,
cause
their
load
balancer
to
send
the
request
somewhere
else,
or
something
like
that.
So
you
know
if
you
do
an
asynchronous
right
and
then
fsync
it
if
the
pool
is
suspended,
that
fsync
will
just
not
return
until
you.
B
Some
of
the
code
for
this
exists
in
the
old
pull
request
for
forced
export,
where
we
go
and
unwind
all
the
pending
iOS
and
having
an
option
like
that,
and
then
you
know
if
we
finish
Force
export.
That
also
raises
the
question
of:
would
it
be
valuable,
have
failed
mode
equals
export
where
you
know
if
this
pool
gets
suspended
and
gets
to
the
point
where
it's
stuck?
B
Go:
Force
export
it
for
me,
so
that
it
doesn't
cause
issues
with
my
other
pools.
You
know
if
it's
hung
on
the
spawn
namespace
lock
or
something
where
is
going
to
cause
every
other
ZFS
command
to
hang
waiting
for
that
lock,
it's
going
to
mean
that
this
one
pool
going
down
is
going
to
mean
all
the
other
pools
in
my
system.
No
longer
work.
B
This
particular
customer's
application
definitely
does
something
useful.
You
know
it
can
read
it's
like
an
API
request,
so
it
can
return
to
the
the
caller
that
you
know
the
put
failed
and
then
the
caller
will
just
try
it
again
and
it'll
get
routed
to
a
different
machine
or
something
and
and
it's
okay.
B
Yeah
at
least
I
remember:
there
was
a
whole
kerfuffle
about
this
with
postgres,
where
I
think
it
was
on
Linux
when
it
returned.
When
The
f-sync
Returned
an
error,
the
data
didn't
stay,
dirty
like
it
was
supposed
to,
and
so
a
second
called
fsync
wouldn't
ever
ensure
that
there
would
succeed,
even
though
it
hadn't
flushed
the
data
that
was
supposed
to
be
still
pending,
whereas
on
like
a
lumos
and
FreeBSD
that
wasn't
an
issue
because
the
the
data
stayed
dirty.
B
B
And
then
the
the
second
one
I
was
just
musing
about.
If
we
get
that
fire
was.
If
we
finish
the
the
forced
export
feature,
do
we
have,
you
know,
fail
mode
equals
export.
So
if
a
pool
encounters
an
error
that
would
result
in
being
suspended,
it
gets
Force
exported
so
that
it
doesn't
interfere
with
any
other
pools
that
are
running
on
the
system,
because
you
know
when
a
pool
gets
suspended.
B
C
B
I
I
B
That's
another
question:
I
had
was
about
some
of
the
metadata
blocks
and
or
indirect
blocks,
I
think
in
a
bunch
of
cases,
the
the
maximum
size
of
like
an
indirect
blocked
is
set
at
128k,
but
I've
not
get
blamed
it,
but
I
assume,
that's
just
from
the
four
large
blocks
were
a
thing
and
in
particular
I'm
wondering
about,
if
increasing
or
having
an
option
to
increase
that
size.
Obviously
that's
probably
a
non-disformat
change
just
because
it
changes
it
to
fine,
but
especially
things
like
microsapps.
B
F
But
you
started
speaking
about
indirect
blocks
and
we
on
our
benchmarks.
We
found
that
default.
Direct
blocks
of
evil,
128
kilobyte
create
is
huge
over
here.
If
we
do
random
block
rewrite,
that's
why
on
tronas
WE
defaulted
to
lower
to
32
kilobyte
now,
but
I
think
64
may
may
be
better
intermediate,
but
like
32
works,
pretty
good
in
many
cases,
if
not
most,
because.
F
Yeah
yeah
you're,
rewriting
one,
for
example,
32
kilobyte
blocks,
and
then
they
write
six
layers
of
128,
kilobyte,
indirect
and
after
it,
and
that's
also
create
this
amplification,
but
also
the
compression
decompression
for
those
blocks.
Indirect
blocks
from
Arc
into
GMU
cache.
There's
a
lot
of
CPU
over
here.
So
just
a
reduction
of
the
block
block
size
saved
a
lot
of
Benchmark,
at
least
on
the
whole
recipients,
from
synthetic
benchmarks.
F
So
it's
it's
like.
It
should
hurt
some
disk
scan
or
sequential
operation,
but
hopefully
prefecture
should
handle
most
cases
there.
So
it
shouldn't
be
too
bad
for
to
have
reasonably
smaller
blocks
but
as
I
think
maybe
664
could
be
better
default
than
128..
I
never
proposed
to
Upstream
that,
but
we
may
consider
that
well,
at
least
I.
Definitely
don't
think
it
should
be
increased
I,
don't
think
it
works.
Unless
somebody
has
some
absolutely
huge
block
sizes
and
definitely
only
sequential
operation,
but
still
doesn't
make
sense.
Well,.
B
One
thing
we
were
considering
was:
should
the
indirect
block
size
follow
the
like
the
record
size,
and
so,
if
you
have
32k
blocks,
should
each
of
your
indirects
also
be
32k
for
the
same
reason
to
avoid
the
amplification?
But
it
would
also
mean
that
if
you
use
larger
blocks,
you
could
have
larger
indirect
blocks.
F
If
you
have
huge
holes
in
a
file,
then
each
access
to
them
will
create
the
tree
and
there
are
destroys
you
recreate
and
Destroy
actually
to
make
sure
you
may
be
separate
problems
that
could
be
optimized
somehow,
because
right
now,
I
think
as
soon
as
we
commit
transaction-
and
we
see
that
block
is
all
zeros
because
there
is
no
indirect
block-
is
all
zeros.
It
will
just
evict
that
block
and
on
next
access.
B
B
F
F
B
B
You,
if
you
modify
it,
then
you're
writing
a
lot
more,
but
do
the
fewer
layers
of
indirection
provide
any
benefit
and
we'll
have
to
test
that
out,
but
it's
mostly
the
zaps
that
we're
looking
at
just
your
files
in
a
directory.
But
we,
you
know,
had
wondered
about
some
of
these
limits
and
like
what
you
were
saying.
It
might
make
sense
to
have
some
way
to
to
control
that.
F
A
F
F
B
I
said
just
because
the
last
one
I
had
was
looking
at
the
the
performance
of
copy
file
range
on
FreeBSD
and
I
saw
thanks
man
for
taking
a
look
at
that
does
Linux
do
something
like
pathconf
or
something
to
return
the
granularity
of
sparseness
in
a
file
foreign
BSD
just
always
returns
like
Spa,
Min
block
size
or
whatever
512,
bytes
and
I.
Think
Alan
Summers
suggestion
was
fixing
that
to
actually
return
the
the
record
size
of
the
specific
v-node
that
was
being
queried.