►
From YouTube: OpenZFS Developer Summit Part 3
Description
http://www.beginningwithi.com/2013/11/18/openzfs-developer-summit/
Vendor Panel (all represented companies sharing their work); Karyn Ritter, Matt Ahrens on the OpenZFS community.
A
A
So
what
does
deltix
do
so?
It's
easiest
to
talk
about
the
the
problem
that
delphic
solves
and
how
why
wend,
my
way
down
into
cfs
so
delphi
solves
this
problem
that
lots
of
big
companies
have
they
have
monstrous
databases
and
for
every
big
production
database.
They
have
many
many
copies
in
non-production
use
cases
and
so
developers
and
testers
and
qa,
folks
and
reporting
and
analytics
and
backup
and
production
support.
All
of
these
people
need
copies
of
the
production
database.
A
So
what
we
do
at
delfix
is
we
call
what
we
say
we
virtualize
at
the
database
tier,
so
we
make
it
very
cheap
and
very
easy
and
very
efficient
to
create
copies
of
databases.
Everyone
are
clear
enough:
okay,
cool
and
what
that
does.
It
means
people
can
run
their
projects
faster
and
make
their
developers
happier
and
trim
time
on
their
dev
cycles
and
stuff,
like
that.
A
A
A
First,
we've
we've
made
a
lot
of
extensions
on
top
of
that
and
I
think
matt
will
want
to
talk
about
some
of
those,
but
we
make
heavy
use
of
snapshots
to
close
heavy
heavy
usage,
snapchats
and
films,
much
more
so
than
we
had
at
sun
or
fishworks
or
oracle
and
any
of
these
other
places.
So
each
virtual
database
starts
from
a
snapshot
of
what
we
call
a
source
and
we
create
a
clone
writable
snapshot
and
then
are
able
to
provision
that
to
some
point.
In
time,
databases,
that's
good!
All
right!
B
Was
gonna
I'll?
Also
I'll
talk
about
maybe
a
few
things
that
are
forthcoming,
that
we've
we've
done
a
lot
of
enhancements
in
zfs
like
you
know
the
great
throttle
that
I
think
a
lot
of
you
have
heard
about,
but
I
wanted
to
mention
a
few
things
that
haven't
been
pushed
into
the
most
yet
that
are
on
their
way.
So
one
is
cfs
bookmarks,
so
not
to
be
confused
with
the
internal
data
structure
called
something
or
other
bookmark
in
zfs.
But
this
is
basically
a
mechanism.
B
That's
specifically
for
doing
a
zfs
send
from
a
place
where
a
snapshot
used
to
be
but
no
longer
exists.
So
the
idea
is
that
if
you're
doing
remote,
replication
you're
continually
like
create
a
snapshot
and
then
send
from
the
previous
version
from
the
previous
snapshot
to
the
new
snapshot
over
to
the
remote
system.
So
you
always
have
to
have
that
previous
snapshot
on
the
sending
system.
Zvs
bookmarks
allows
you
to
create
a
bookmark
at
the
point
in
time
where
that
snapshot
represents
and
then
delete
the
snapshot
and
then
send
from
the
bookmark
instead.
B
A
B
So
another
thing
that
I
I've
worked
on
recently
is
the
zero
block,
compression
or
embedded
data
feature
which
is
going
to
be
going
to
limos
some,
even
after
we
get
through
our
backlog.
But
the
idea
is
basically
that
some
data
blocks
compress
really
really
really
well
and
they
can
actually
fit
down
into
like
about
100
bytes.
Then
we
can
put
that
data
into
the
block
pointer
itself,
so
the
block
pointer
doesn't
actually
point
to
the
data
block.
B
It
actually
contains
the
data,
and
this
helps
a
lot
for
like
small
bits
of
metadata.
So
we
think
this
will
probably
improve,
like
you
know,
zfs
list,
and
things
like
that,
because
you
know
all
those
properties
are
stored
in
like
a
different
object
with
a
different
block
and
instead
it'll
just
be.
You
know,
one
less
random.
I
o
because
you'll
already
have
the
data.
It's
also
very
advantageous
for
our
particular
use
case,
where
initialized,
but
otherwise
unused
oracle
blocks
are
very
common.
B
B
Footer
can
be
compressed
with
lz4
to
fit
into
100
bytes,
so
it
super
accelerates
that
you
know
benchmark
special,
but
it
also
makes
it
a
lot
faster
to
like
read
that
data,
because
we
already
have
the
data
in
the
block
pointer
rather
than
go
for
this.
What
do
you
mean?
You
say
you
keep
the
data
in
the
partner.
D
B
So
the
data
like
the
block
pointer,
so
the
block
pointer,
is
always
checksum
by
the
block
that
points
to
it.
So,
for
example,
you
have
an
indirect
block
that
indirect
block
is
verified
to
be
correct,
because
the
block
printer
that
points
to
it
has
its
checksum
in
right.
So,
if
we're
putting
you
know,
pointers
to
data
in
there,
those
pointers
are
verified
to
be
correct.
We're
putting
compressed
data
in
it
that
data.
E
F
B
G
One
hi
I'm
barbara
mastaki,
I'm
at
joint
we're,
basically
a
cloud
provider,
so
we
use
efs
in
a
few
different
ways,
primarily
first
off
just
for
taking
around
images
and
dealing
with
the
fact
that
customers
have
vms
every
vm
gets
its
own
zfs
data
set,
whether
it's
an
os
based
virtualization
or
using
hardware
virtualization.
If
it's
hardware
virtualization
you
get
z,
volts
and
those
are
what
turn
into
your
disks.
Those
images
need
to
be
moved
around.
So
that's
all
through
snapshots
and
clones
very
similar
to
what
delfix
is
doing.
G
In
addition
to
that,
we've
done
a
bunch
of
enhancements
in
zvez
multi-round
multi-tenancy
for
dealing
with
basically
enhancing
the
I
o
throttle
be
more
container
aware.
So
you
could
basically
define
shares
on
that.
So
you
can
basically
have
a
better
sense
of
fair.
I
o
sharing
stuff
that
we
have
in
the
works,
which
is
kind
of
interesting,
is
fs
limits.
G
So,
basically,
you
can
do
limits
on
the
number
of
snapshots
and
data
sets,
because
if
you're
aware,
you
put
10
000
snapshots
on
your
box
and
things
get
very
sad
right
away
for
that
machine,
and
the
last
thing
you
want
is
one
bad
actor
to
screw
over
everyone
else.
The
other
big
thing
we
do
with
cfs
is
basically
the
core
of
our
new,
basically
object,
storage
and
compute
service,
and
what
happens
is
basically
it's
an
s3
like
data
store,
but
you
can
run
compute
directly
on
there.
So,
basically
spin
up
zones
go.
G
Do
whatever
compute
you
have.
You
have
all
your
packages
that
you
ever
want,
then,
once
that's
done,
you
roll
back
the
zone
and
move
on
to
the
next
customer.
So
that's
already
a
pretty
big
use
case
and
we're
using
rollback.
Probably
you'd
have
like
hundreds
of
roll
backs
in
a
round
per
second,
sometimes.
F
G
D
H
Yes,
very
quickly,
as
you
already
know,
from
boris's
talk.
Nexanza
basically
have
a
you
know:
storage
appliance
based
on
zfs,
so
we
basically
used
every
single
feature
of
zfs
right
from
snapshot,
so
we
basically
use
gfs
as
a
backend
for
for
nas
and
sand
protocol
right,
nfs,
civ
ice
busy,
fibre
channel
kind
of
thing,
so
we
we
at
center
itself.
We
try
to
work
on.
You
know,
fail
over
performance.
H
B
H
I
J
J
K
Connecticut,
I'm
the
gitmaster,
the
zfs
understander
development
operations
engineer
so
we're
a
linux
company
generally
ubuntu.
Our
main
product
is
unfortunately
written
in
php,
but
we
are
working
in
a
lot
of
different
operating
different
languages
there.
So
we
are
using
the
gfs
on
linux,
kernel
module
provided
by
ll
right
now.
So
that
brings
me
to
the
point
of
help
number
one.
We
want
to
talk
about
lib
cfs
how
we
can
integrate
with
that,
how
we
can
stop
using
the
command
line.
K
A
K
In
there
and
helping
the
developers
who
don't
understand
the
intelligent
gfs
safely
get
at
the
things
that
they
want
to
get
at
is
one
of
the
things
that
I've
been
working
on
recently
because
we
had
a
whole
bunch
of
yes,
this
pipe
to
grab.
That
was
great,
so
we
operate
our
own
cloud
and
we
also
operate
lots
of
machines
out
in
the
field
and
we
run
open
cfs
on
all
of
that.
K
K
We
start
up
these
virtual
machines
in
the
cloud
to
let
the
customer
know
that
whenever
they
want
to
come,
get
their
data,
it's
good.
So
we're
constantly
starting
these
things
up
and
they
need
about
as
much
ram
as
they
would
have
needed
to
start
up
anyways.
So
they
have
a
whole
bunch
of
ram
going
to
dfs,
and
then
these
guys
are
constantly
taking.
Then
the
next
thing
they're
missing
out
on
some
of
the
pictures.
I
should
have
just
done
it.
I'm
sorry
is
it
going
to
be.
K
I
mean
some
really
killer
stuff
here,
people
just
I
can
go
into
while
it
warms
up
one
of
the
cool
applications
that
I've
done.
That
was
not
exactly
for
work.
Was
we
had
a
linux
kernel
thing
that
was
working
in
three
five?
No,
that
was
not
working
in
three
pod
and
was
working
in
linux
360.,
and
so
they
asked
the
question:
how
can
we
get
to
that.
B
K
J
K
K
Comes
for
their
multiple
boxes,
they
want
to
start
up
all
those
vms
and
actually
run
them.
Not
just
a
pre-provision.
Does
this
thing
work
so
again,
we've
got
lots
of
ram
going
to
zfs
and
then,
when
something
comes
for
your
six
devices
there,
you
gotta
start
them
all
up.
So
the
the
talk
today
about
vm
integration
with
memory
management
is
definitely
interesting
to
us.
Integrate
with
the
linux
page
cache
dropping
it
more.
K
So
that
means
we've
got
these
pools
getting
up
to
85
percent
full
and
those
are
actual
customer
chains
and
we
can't
break
those
out
and
just
put
them
other
places.
So
we
do
have
to
actually
make
the
transfer
and
that's
really
expensive
on
our
end
on
that
guy
over
there.
So
that
multi-tiered
storage
sounds
interesting
to
us
with
maybe
getting
the
ssds
on
these
guys
so
that
they
can
ingest
it
faster
anything.
We
can
do
like
that.
I
think
I
have
a
help.
K
Point
three
improved
center
receive
speed,
we're
probably
doing
something
wrong.
There's
probably
some
configurations
that
we
can
do
better.
There's
also
things
that
maybe
we
can
help
you
guys
give
back
to
community
a
little
bit
and
pick
some
of
that
stuff.
I
know
there's
a
buffer
and
send
receive
talk
on
the
chalkboard
different
today,
so
that'll
be
great.
So
here's
what
a
basic
product
does
out
in
the
field,
because
that
was
our
cloud.
So
these.
K
When
the
computers
want
to
back
up,
we
expose
over
the
network
of
a
block
device
for
them
to
write
to
they
write,
what's
changed
since
then.
We
closed
it
up,
take
his
efs
snapshot
and
then
send
that
one
to
the
cloud,
but
since
that
first
snapshot
that
you
actually
do
as
a
block
writing
from
your
devices
is
way
more
gigabytes
and
up
to
a
terabyte
of
stuff
that
you
don't
want
going
over
your
network.
We
actually
round
trip,
so
we
take
hard
drives,
put
them
together.
K
Mail
them
to
you,
make
a
z-pool
on
that
zfs
send
onto
it
mail
it
back
to
the
data
center
and
upload
it
directly
to
the
cloud
and
during
that
time,
you're
building
more
snapshots
and
you
send
them
all
at
the
end-
works
pretty
well
cool
uses.
I
guess
this
is
another
one
of
the
things
I
did
off
of
our
devices.
I
back
up
our
github
instance.
I
am
the
proprietor
of
our
github
enterprise
within
the
office.
It's
a
real
game!
Changer.
K
I
think
that
you
guys
have
had
experience
with
if
you're,
using
the
zfs
and
you're
used
to
using
it
to
go
into
a
directory
and
just
have
the
files
there
and
then
have
the
snapshots
being
available
to
you
through
a
zfs
list
and
that
kind
of
stuff,
as
opposed
to
cluttering
it
up
with
files
all
over
the
place.
It
definitely
changes
the
way
you
think
about
how
backups
can
work
and
how
you
can
store
data
and
really
cleans
that
thing
up.
K
I
think,
and
then
the
other
cool
thing
I
did
was
that
linux
kernel,
reverse
bisect
didn't
get
into
too
much,
so
we
set
up
a
machine
basically
with
a
clone
of
the
linux
kernel
line
and
a
script
that
tested
whether
or
not
it
worked.
So
then
you
compile
the
new
linux
kernel,
run
the
test
zero
or
one.
So
you
make
a
new
virtual
machine
exactly
off
of
that
model,
zfs
clone
again
from
the
first
one,
give
it
the
new
shot.
K
It
builds
that
test
it
and
goes
back
and
without
zfs
that
whole
thing
would
have
been
a
lot
more
expensive
to
clone
out
again
and
not
only
the
running.
Linux
operating
system,
but
the
clone
of
the
entire
linux
transfer,
because
you
never
know
where,
in
there
you're
going
to
get
so
in
13
steps
and
really
quick.
We
had
ourselves
the
answer
for
which
cash
fixed
that
change
and
we
were
able
to
look
at
it
and
see
why
the
lynx
kernel
was
hanging.
D
N
Which
just
went
to
beta
a
couple
weeks
ago
closed
or
just
another
term
for
archive
the
idea
being
is
we're
targeting
large
businesses,
with
lots
of
data
with
petabytes
to
store,
and
rather
than
just
storing
that
data
sticking
to
the
wall.
You
don't
know
if
it's
going
to
be
good
when
you
need
it
in
five
years,
we're
targeting
zfs
file
systems
like
they
can
write
the
data
to
it.
The
data
can
be
scrubbed
periodically.
If
there's
a
problem
with
it,
the
operators
can
go
in
and
switch
out.
N
Some
of
some
of
the
challenges
we're
running
into
is
being
a
archive,
we're
pretty
much
going
to
write
once
and
hopefully
never
read
unless
the
disaster
strikes
but
we'd
like
to
power
down
some
of
the
discs,
because
at
no
point
having
thousands
of
discs,
spinning
and
the
problem
we're
running
into
is
powering
down
discs
out
from
under
vfs,
isn't
so
wise
when
there's
all
sorts
of
issues
there.
So
we're
trying
to
find
a
better
solution
for
powering
down
discs,
whether
that
be
part
of
zfs
or
maybe
some
external
utility.
N
O
Thanks
and
actually
I
was
going
to
say,
the
stuff
that
boris
was
talking
about
with
the
storage
tiering
could
help
your
power
on
power
off
properly,
because
if
you
had
metadata
in
power
on
storage,
we
could
power
off
data
that
is
sitting
on
disks
that
potentially
wouldn't
read
yeah.
That
might
be
another
use
case.
Oh
that's
so
interesting,
yeah.
J
D
P
So
the
main
crux
of
those
changes
is
to
do
asynchronous
resolution
of
copy
and
write
faults.
So
you
come
in
and
you
do
an
overwrite.
If
you
grab
a
block
of
data,
we
basically
will
allow
you
to
make
that
update
in
memory
and
return
to
the
writer
and
let
them
write
additional
data
while
in
the
background
pulling
in
the
read
in
order
to
fill
in
the
bits
of
that
block.
P
P
D
P
Say
your
record
size
is
128k
and
you're
writing
8k
at
a
time.
Zfs
will
just
go
and
fetch
that
thing,
even
though
you
might
be
writing
a
whole
stream
of
sequential
8k
things
like,
for
instance.
Maybe
your
vm
is
ntfs
with
a
nice
8k
native
record
size
right,
but
you
really
want
to
use
128k
records
on
your
backing
store
just
to
make
the
metadata
more
efficient
and
things
like
that.
D
P
P
P
As
a
researcher,
let's
say
at
a
laboratory,
I
may
know
that
this
particular
data
set
I'm
not
going
to
touch
for
three
years.
So
the
actual
consumer
of
the
data
can
make
the
decision
to
drag
that
thing
to
deep
storage
and
also
can
go
look
at
our
system
and
see
it
to
drag
it
back,
but
by
making
it
nas
any
client.
You
know
any
operating
system.
Client
that
can
talk
to
nas
can
make
that
data
movement
without
having
to
have
any
middleware
or
anything.
D
P
Order
to
make
that
data
movement
possible
and
then
we
make
sure
that
the
data
is
stable,
so
we
create
a
version
of
the
of
the
data
and
then
we
push
it
out
to
tape.
So
our
main
need
in
the
near
term
is
the
ability
to
create
that
stable
version
of
individual
files,
and
so
that's
ref
links
or
something
like
that.
P
The
main
reason
why
snapshots
don't
quite
work
for
us
in
that
environment
is
that
we
want
to
be
able
to
allow
the
end
user
to
say
yeah
that
data
set.
I
no
longer
need
it
anymore.
I
would
like
to
be
able
to
individually
pick
different
objects
in
the
data
store
and
blow
them
away,
but
with
something
like
ref
links,
where
it's
an
immutable,
hard
link,
we
could
provide
s3
versions
of
of
objects
in
a
very
seamless
way,
just
using
the
copy
and
write
semantics
of
cfs.
L
K
I
remember
a
talk
with
specialized
a
while
ago
about
performance
improvements.
They
have
done
and
comments
in
the
code.
I
remember
that
seemed
pretty
exciting.
You
guys
were
saying
you're
gonna,
get
it
back
in
yeah
yeah
we're
working
on
it.
Okay,
cool!
Was
that
what
you
were
talking
about
earlier
in
the
talk
yeah
yeah
that
looks
really
yeah.
The
comments
are
always
great
yeah
and
those
would
go
freebsd
first
and
then
it's.
P
P
B
Q
So
my
name
is
alex
and
I'm
actually
a
hardware
guy,
and
so
what
we've
done
is
we've
done
a
hardware
accelerator
for
sharp256
that
is
very
easily
pluggable
into
cfs.
You
already
have
a
patch
for
xerocs
on
linux
and
it
works
great.
So
it
looks
like
a
lot
of
people
are
using
shop256
for
check
summing
for
data
integrity,
and
what
I
heard
is
that
it
is
often
ball
neck
for
performance,
especially
if
you're
trying
to
do
many
discs
or
flash
storage
things
like
that.
So
basically
we
have
this
very
smooth
low
cost.
Hardware.
Q
Accelerator
works
with
zfs.
On
linux.
We
are
working
on
illumis,
illumina's
patch.
Again
it
should
be
fairly
straightforward,
design,
specifically
to
be
high
throughput
law,
latency
very
lower
at
the
software
side.
If,
if
there
is
question,
if
there
are
questions
or
any
interest,
I
actually
have
something:
it's
a
pci
card.
Yeah
yeah
can
you
can
you
show
it
to
us.
A
Q
Q
Q
D
Q
E
E
Basically,
we
marry
the
speed
performance
and
reliability
of
dram
with
the
non-volatility
of
slc
flash
we're
very
close
to
formally
introducing
so
we've
had
that
product.
For
a
couple
years
our
driver
came
out
in
january
of
2010
and
when
it
required
external
ac
dc
adapter
for
the
for
the
power
backup
and
we're
kind
of
excited
to
have
a
supercapacitor
solution
which
enables
us
to
remove
the
external
power
which,
for
a
lot
of
customers,
obviously
was
a
real
pain.
So
we're
kind
of
excited
about
the
supercap.
E
B
A
B
B
Then
we
need
storage
for
them.
So
since
we're
a
scientific
computing
center,
we
care
a
lot
about
the
fact
that
the
data
is
right.
So
previously
we
invested
in
a
hardware
solution
for
this
that
did
jackson,
my
read
and
stuff
like
that,
but
upon
looking
around
that
was
not
the
way
to
go,
so
we
made
an
investment
in
pulling
zfs
over
to
linux,
so
we
could
use
it
there
and
get
the
data
integrity
protections.
All
the
other
good.
S
B
Q
K
B
Don't
take
distributed
snapshots
today,
we
could,
but
that
would
require
us
to
coordinate
across
the
whole
distributed
file
system
right,
which
we'd
like
to
do
would
be
a
nice
feature
for
luster
to
take
distributed
snapshotting.
So
that's
certainly
a
cool
thing
we'd
like
to
do.
I
think
the
more
useful
use
case
for
us
is
something
like
upgrades
when
we
did
take
the
whole
file
system
down
and
do
a
major
software
update
to
it,
particularly
on
the
luster
side
right.
B
B
K
B
Snapshot
but
done
manually
basically
form
an
administrative
procedure.
It
would
be
nice
to
be
able
to
like
coordinate
that
through
luster,
but
in
front
of
that.
It's
a
nice
administrative
tool
for
us,
but
the
bigger
win
for
us
is
actually
compression
worked
out
great
for
us.
Without
our
data
on
our
data
sets,
we
didn't
know
very
compressible,
so
so
we
got
a
nice
performance
boost
out
of
that.
So
that's
cool.
A
B
Then
basically
start
luster
there
and
it
will
pick
up
the
services
or
a
sailor
partner.
So
there
is
some
concern
there
about
importing
the
pool
in
the
failures
area
and
accidentally
trashing.
It
that's
kind
of
a
worse
case
for
us,
so
we
do
have
higher
level
scripts
that
coordinate
the
failover
process,
an
automated
way
to
make
sure
we
don't
accidentally.
B
Think
there's
room
for
improvement
in
zfs
without
coordinating
that
and
maybe
allowing
it
to
detect
the
fact
that
it's
been
imported
on
both
pools
concurrently
quickly
and
prevented.
There's
some
papers
out
there
for
ways
to
do
that,
all
right
and
some
write-ups,
but
they
haven't
been
implemented.
Yet
that
would
be
a
nice
feature.
P
A
F
B
B
T
T
We
have
lots
for
customer
data,
so
the
customer
not
might
not
benefit
from
zfs,
but
we
do
as
administrators
in
terms
of
backup
and
just
being
able
to
provision
commands,
so
we
do
simple
hosting
from
just
email,
p
or
web
all
the
way
up
to
vms.
If
you
want
to
run
your
own
os
or
whatever
kind
you
want
basically
and
there's
that's
all
on
top
of
dfs,
we
try
to
do
everything
real
time.
T
T
T
G
B
Any
anyone
else
who
always
will
move
on
to
the
next
sessions-
all
right,
so
I
think,
is
karen:
are
you
there
should
we
do
the
community
organization
we'll
do
community
organization
stuff,
then
channel
program,
stuff
and
then
we'll
see?
Maybe
it's
time
for
a
break
and
then
we'll
then
maybe
adam
will
tell
us
about
performance.
U
U
B
U
So
you
know
we
started
off
just
kind
of
with
this
idea
of
having
this
community.
We
got
a
bunch
of
people
together
and
they
very
generously
gave
their
time
to
try
to
think
of
what
we
should
do
around
the
launch
kind
of
content.
We
should
have
the
kinds
of
things
that
we
should
be
working
on
and
that's
kind
of
how
we've
been
managing
it.
It's
a
little
bit
ad
hoc
people
are
just
updating
the
website,
which
is
all
great,
and
I
think
we
definitely
want
to
continue
with
that.
U
You
know:
we've
got
the
the
mailing
list.
We've
just
got
basically
that
one
for
now,
although
I
think
the
linux
mailing
list
will
someday,
maybe
move
to
open
the
effects
as
well
and
then
we've
got
and
we've
got
the
the
dev
summit
that
we
did,
but
that's
pretty
much
all
we've
done
as
a
community
in
open
zfs.
I
know
all
of
you
guys
have
been
working
on
stuff
for
a
long
time.
D
K
A
R
B
U
F
U
Of
the
things
we're
really
wondering
about
is
whether
or
not
we
should
create
an
open,
zfs
foundation.
Obviously
justin
and
others
have
had
experience
in
doing
this.
I
don't
know
how
much
work
it
is,
but
it
seems
like
having
the
ability
to
transfer
ownership
of
the
logo,
as
well
as
the
domain
names
and
just
kind
of
being
a
resource
for
people
to
show
their
support
through
donations
and
and
do
these
kinds
of
things
separate
from
any
company
would
be
a
good
idea.
U
B
C
E
B
F
F
R
P
Give
you
a
little
background,
I
guess,
on
the
previous
d
foundation,
it
started
in
2000.
It's
it
took
a
really
long
time
to
get
to
the
point
where
it
is
right
now.
So
just
have
a
lot
of
patience,
mostly
because
the
people
behind
foundations
like
this
are
very
passionate
people
that
want
to
have
it
happen,
but
they.
P
As
well
and
at
least
to
start
at
least
the
way
the
foundation
was,
the
frequency
foundation
was
for
the
first
six
years,
or
so
we
had
nobody
that
was
paid
right
before
we
had
enough
of
a
financial
footing
where
we
could
say
okay,
we
think
we
can
go
to
that
next
step
of
our
evolution,
and
so
we
hired
a
part-time
person.
You
know
and
then.
D
G
P
P
P
That's
actually
not
so
bad.
So
when
I
started
the
foundation,
you
have
to
have
three
people
at
least,
if
you're
a
us
one
right,
you
have
to
have
a
secretary
of
vice
president
or
president
right,
and
so
I
was
the
secretary
I
said
I'll
take
care
of
all
the
paperwork
and
the
paperwork
as
far
as
getting
the
non-profit
status,
that's
difficult,
higher
alert,
the
other
stuff.
The
other
stuff,
though,
is
not
so
diverse.
P
P
But
we
have
enough
staff
on
hand
that
if
we
were
able
to
double
or
triple
our
donation
level,
our
income
or
revenue
that
we
could
probably
keep
the
same
staff
and
drastically
reduce
our
overhead.
But
you
run
into
these
places
in
your
in
your
history
as
a
foundation
where
your
overhead
is
going
to
be
higher
than
you'd
like,
and
you
really
need
to
then
go
out
and
and
get
more
revenue
and
really
work
it
so
that
you
can
balance
things
out
again.
It's
this
constant
cycle.
B
So
what
I
mean
we
all
work
for
like
companies
that
would
hopefully
support
a
open,
zfs
foundation
like
what
would
be
compelling
about
that
to
solicit
your
companies
like
what
would
you
be
looking
for?
Is
it
you
know,
development
effort?
Is
it
like
you
know,
should
we
should
we
target
like
we
should
you
know,
fund
somebody
part-time,
to
go
work
on
like
making
all
the
make
files
and
doing
all
this
stuff
with
the
open
for
the
code?
P
D
D
P
Big
issue
for
people:
it's
like
I'm
building
a
product
or
a
service
that
depends
on
this.
I
know
the
technology
is
gonna
change
over
time.
I
can't
do
all
that
development
myself.
How
do
I
make
sure
that
four
years
from
now,
I'm
not
in
a
dead-end
product?
That's
something
that
the
foundation
couldn't
speak
to.
P
The
next
thing
that
they
talked
about
talking
with
you
know
commercial
users.
Is
you
know
what
are
you
working
on
right?
So
can
I
fund
a
specific
development
project,
but
I
think
that's
actually
not
nearly
as
important
as
the
the
ability
to
improve
the
downstream
experience
of
people
who
integrate
the
technology.
P
So
that's
probably
the
third
thing
that
these
guys
are
concerned
about.
When
you
talk
to
a
commercial
consumer,
what
is
your
test
environment?
How
do
I
get
fixes
back
into
your
community?
How
do
I
know
that
the
regression
that
I
fixed
doesn't
come
back
again
in
the
future?
What's
your
testing
methodology
all
of
those
things
about
making
it
a
professionally
developed
piece
of
software
that
you
can
rely
on?
P
P
So
sometimes
it's
talking
about
training
materials
so
that
once
I've
hired
somebody,
how
do
I
bring
them
up
on
this
code
base?
How
do
I
make
them
effective?
How
do
I
then
get
them
into
your
community
right
so
that
our
our
changes
don't
stay
here
and
we
have
to
maintain
them
over
time.
So
it's
really
those
four
things
and
I
think
that,
as
a
foundation,
you
can
speak
to
all
of
them
very,
very
well
and.
P
P
You
know
you
have
to
be
having
a
problem
with
reputation.
George
over
here,
you
know,
is
the
person
to
talk
to
and
by
the
way,
have
you
integrated
this
change
that
whatever
and
that
provides
extreme
value
to
companies
when
you
go
and
talk
to
them.
So
almost
all
of
our
discussions
as
and
when
the
freebies
foundation
goes
and
talks
to
a
consumer
freebsd,
it's
not
about
money
because
oftentimes,
that's
the
that's
the
barrier
to
actually
getting
in
and
having
that
conversation
they
think
you're
just
going
to
come.
Ask
for
money.
O
P
D
F
B
B
F
B
C
C
C
I
would
also
have
a
look.
There
were
some
interesting
talks
at
the
community
leadership
summit,
which
takes
place
before
ozcon
every
year.
Now,
for
the
last
five
years,
henrik
ingo
formerly
of
mysql,
gave
an
interesting
series
of
statistics.
He
pulled
together
on
whether
open
source
projects
really
need
a
foundation
or
benefit
from
it,
and
his
findings
were
pretty
much
came
down
to
no,
so
I
would
think
long
and
hard
about
whether
to
go
down
that
path.
C
F
U
C
C
C
C
B
B
P
Community
management
can
I
mean,
if
you're
just
talking
about
like
list
moderation
stuff
like
that-
that's
probably
not
not
as
interesting,
at
least
from
my
perspective.
But
what
is
interesting
is
strategic
planning,
better
tools
for
accounting.
You
know
what
people
are
working
on
and
where
it
is
and
what
its
state
is,
and
maybe
by
using
github
and
people
having
their
work
pushed
up.
There
that'll
take
care
of
some
of
it,
but
often
times.
P
You
know
during
these
periods,
where
we
had
less
management
coming
from
like
the
frequency
core
team
or
what
have
you
their
lacked
focus
right.
What
are
we
trying
to
achieve
in
the
next
six
months
or
what
are
we
trying
to?
What
is
what
is
open,
zfs
supposed
to
be
really
good
at
and
what
are
the
spaces
that
we're
not
interested
in
right?
That
kind
of
level
of.
P
From
just
you
know,
one
person
or
one
group,
but
having
it
recorded
somewhere
and
and
making
it.
So
it's
not
just
a
couple
people
who
have
to
constantly
you
know
remind
people
on
the
mailing
list.
You
know
that
that's
not
what
opencfs
means,
or
you
know.
We
really
need
to
finish
this
feature,
because
that's
our
goal
for
the
next
six
months.
U
U
Are
there
other
things
that
we
want
this
community
to
be
about?
I
mean
you
know,
we've
seen
on
the
mailing
list,
where
people
will
come
up
and
they're
new
users,
and
they
want
to
know
more
about
just
cfs
as
a
whole
to
try
to
do
a
little
bit
in
terms
of
writing
architecture
documents
and
that
kind
of
thing,
but
are
there
other
areas
that
we
want
to
expand
to
to
try
to
come
out
with
openstack.
W
W
U
W
R
G
X
T
G
B
C
D
B
B
Yeah,
so
you
know
how
like,
how
should
we
go
about
finding
people
to
do
this?
Is
it
is
like?
Should
I
be
going-
and
you
know,
sit
like
posting
on
the
mailing
list
and
saying
like
these-
are
the
priorities
that
we
gathered
from
the
development
summit
like
who
will
who
will
do
them
and
hoping
that
somebody
like
volunteers
or
like,
should
I
be
going
and
like
finding
particular
people
me
like
hey
max,
you
want
to
do
this?
Don't
you
come
on.
P
So
I
think
one
thing
that
might
help
you
there
is
by
lowering
the
barrier
of
entry.
So
when
we're
talking
about
all
these
documents
right
right
now,
they're
these
monolithic,
pdfs
and
you
know
trying
to
understand
well,
can
we
use
any
of
that
material,
it's
right
from
scratch,
what
make
sure
you
do
right,
yeah
and
then
bootstrapping
finding
finding
just
a
couple.
People
will
be
just
bootstrapped
as
in
take
whatever
content
is
available,
get
it
in
some
places.
It
has
easy
access
and
then
you
can
go
to
the
community
and
say
hey.
P
X
G
G
Okay,
so,
like
we've,
been
slowly
converting
them
for
lumos,
but
we
have
the
zfs
one
so
that
it's
just
a
matter
of
just
like
going
through
cleaning
it
up
and
like
changing
references
to
names
and
just
like
doing
a
few
markup
things,
and
then
there
you
go
it's
easy
to
go,
build
and
put
it
on
a
website,
so
they're
just
sitting
around.
So
for
those
it's
kind
of
easier
I'd,
say
for
a
lot
of
those
things
like
the
on
discipline:
you're
better
off
this
copy.
X
B
U
Yeah,
there's
definitely
stuff
that
we
can
get
added
to
the
website
as
well.
I
mean
luke
has
talked
about
you
know,
possibly
setting
up
some
additional
things
like
we
asked
for
opengrock
to
be
out
there,
which
hasn't
happened
yet
you
know
they're,
obviously
limited
in
their
resources
as
well,
but
if
there
are
things
that
we
want
to
see
added,
you
know
definitely
send
me
the
list
and
watch.
R
V
F
B
V
B
V
Yeah,
so
I
started
celtics
about
five
months
ago,
and
one
of
the
first
things
I
worked
on
was
holberth,
which,
basically,
when
you
do
a
zfs,
send
it
sends
things
called
free
records
which
represent
polls
in
a
file
where
it's
basically
all
zeros.
So
we
don't
actually
have
to
allocate
space
on
disks
for
these
holes
because
they're
zeros,
we
can
just
you
know,
pretend
we
actually
wrote
them
disk.
V
But
when
you
do
a
cfsend,
if
you
have
very
holy
files,
very
holy
datasets
you're
sending
a
lot
of
these
free
records,
which
takes
up
a
lot
of
takes
a
lot
of
bandwidth
and
make
sure
sunstreams
take
a
long
time.
And
so
I
did
some
work
on
changing
the
behavior
of
the
birth
times
and
basically
recording
when
things
are
freed,
so
that
we
don't
need
to
do
these
signs
of
redundant
free
blocks.
If
the
target
of
the
send
is
already
aware
that
those
blocks
are
free.
S
So
we
already
do
this
for
data
blocks
like
don't
send
things
that
are
older
in
an
incremental,
send,
don't
send
things
that
are
older
than
the
or
like
the
originated
snapshot
of
your
send,
but
we'd
never
recorded
the.
When
we
freed
blocks,
we
didn't
record
what
transaction
the
block
was
freedom.
So
anytime,
you
saw
a
free
record,
you
had
to
send
it
which
caused
problems.
You
have
like
a
z-ball
with
some
other
formatted
file
system.
B
S
S
These
operations
are
done
in
what's
called
sync
tasks,
and
the
idea
is
we
do
these.
When
you
run
one
of
these
commands,
we
send
the
zfsioptyl
to
the
current
says.
I
want
you
to
create
a
new
file
system
with
this
name,
and
the
kernel
will
cue
that
operation
to
happen
at
the
end
of
the
current
transaction
or
at
the
end
of
a
transaction.
So
the
idea
is
at
the
end
of
each
transaction
group.
S
We
kind
of
we
hold
all
the
locks
say
no
metadata
updates.
A
single
thread
goes
through
and
has
kind
of
the
power
to
change,
update
all
the
metadata,
create
new
file
systems,
take
snapshots
and
stuff,
and
then,
if
you,
so,
this
ioctal
waits
for
the
current
transaction
to
complete,
so
that
you
know
that
this
update
happened,
that
you
created
the
file
system,
that
the
snapshot
was
created.
S
S
Because
we
don't
sync
transactions,
don't
happen
that
often
and
it
scales
based
on
the
load
of
the
system.
So
in
general
I
think
the
target
is
or
was
like
to
do
it
at
once,
every
second
right,
but
if
you're,
if
you're
on
a
heavily
loaded
system
and
kind
of
it,
because
the
idea
is
in
order
to
do
finish,
the
transaction,
all
the
dirty
data
has
to
be
on
disk.
S
S
Each
time
you
do
an
administrative
operation,
you
set,
you
do
the
after
down
to
the
kernel
that
puts
it
in
the
queue
to
be
finished.
You
know
10
seconds
from
now,
basically,
and
so
on
our
system
where
we
have
where
we
store
our
vms.
We
have
kind
of
the
some
python
scripts
that
use
cfs
properties
and
metadata
and
there's
a
lot
of
destroys
and
promotes
underneath
the
hood.
S
S
Another
problem
is
that
these
sync
within
these
sync
tasks,
if
you
say,
take
a
snapshot
there's
a
lot
of.
We
do
a
lot
of
checks.
So
I
was
I
was
saying
earlier
during
these
this
sync
test.
We
hold
all
the
locks,
we
don't
let
any
other
threads
update
the
zfs
metadata
so
that
we
can
consistently
check.
Oh
this,
you
know:
if
we're
taking
this
snapshot,
we
can
check
that
the
file
system
you're
trying
to
take
a
snapshot
of
exists
and
between
that
check
and
the
time
you
actually
create
the
snapshot.
S
Nobody
else
can
modify
the
metadata.
So
this
works
within
the
sync
task.
But
now,
if
you're
splitting
across
multiple
operations
say
you
want
to
do
a
destroy,
but
you
have
to
do
a
promote
first,
because
there's
some
clone
of
this
file
system
you'll,
do
the
promote
in
one
sync
task
and
then
10
seconds
later.
You'll
do
the
destroy
and
between
those
two
points,
someone
will
have
created
another
clone
and
that
won't
work.
S
So
you
kind
of
and
there's
a
lot
of
logic,
for
example,
zfs
destroy
dash
r,
which
destroys
all
the
snapshots
of
a
file
system
and
then
destroys
the
file
system.
It's
actually.
First
it
gathers
the
list
of
snapshots
in
user
land
which
hopefully
isn't
changing
because
it
had
no
locks
held
when
gathered
the
list,
then
it
sends
that
list
to
the
kernel
and
in
one
transaction,
destroys
the
snapshot
and
then
does
a
second
transaction
to
destroy
the
file
system.
So
the
system
crashes
between
transactions
or
things
are
changing
too
much
you
get
well.
S
We
try
to
have
good
results,
but
it's
there's
a
lot
of
complicated
logic
in
user
land
to
ensure
that
we
have
the
right
results
for
some
of
these
weird
things.
Just
because
we
can't
do.
We
have
very
specific
things.
We
can
do
transactionally
in
the
kernel
in
defining
these
sync
tasks
and
everything
else.
We
have
to
kind
of
wire
together
and
worry
about
that
in
user
land
and
worry
about
different
transactions.
So
all
the
different
snapshots,
capital
r,
is
even
worse.
I
don't
know
how
many
transactions
that
does,
but
it's
a
lot.
S
Let's
see
the
last
problem
is
kind
of
we've
been
adding
more
and
more
features.
Matt's
been
adding
for
more
features
like
the
taking
snap.
I
think
you
added
this
in
the
last
year
so
like
taking
snapshots,
giving
a
list
of
different
file
systems
to
take
a
snapshot
of,
rather
than
only
being
able
to
take
recursively
under
a
single
point,
but
each
time
we
add-
or
you
know
add
so-
zfs
destroy,
dash
p
and
rollback.
S
P
are
like
on
my
wish
list,
and
then
there's
and
and
the
idea
is
like
destroy
dash
p,
would
atomically
do
the
promote
and
then
destroy
in
a
single
txg
to
make
that
to
avoid
that
problem,
I
was
talking
about
earlier.
Rollback
would
do
a
similar
thing
if
you
want
to
roll
back
through
a
clone.
Do
the
promotes
to
make
the
rollback
work.
B
And
like
every
new,
like
functionality
that
we
had
like
the
bookmarks
stuff
that
I
was
mentioning
earlier,
like
you
want
to
be
able
to
create
a
bunch
of
bookmarks
at
once
or
destroy
a
bunch
of
bookmarks
at
once,
and
all
that
stuff
like
you,
got
to
go
reimplement
a
bunch
of
boilerplate
stuff.
That
snapshot
also
has
to
do,
and
destroy
also
has
to
do
so.
S
Yeah
for
each
new
feature
you
end
up
with
this
trade-off:
either
you
do
it
tie
it
all
together
in
user
land
and
have
really
complicated
code
to
deal
with
transactions
that,
like
the
fact
that
not
everything's
happening
transactionally,
or
you
write
this
terrifyingly
long
giant.
Sync
task
function
in
the
kernel
and
hope
that
nothing's,
wrong
and
or
everything
will
explode.
V
D
V
Execute
them
all
in
syncing
context,
so,
for
instance,
this
could
be
executing
multiple
snapshots,
destroys,
creates
and
kind
of
force.
The
zfs
kernel
module
to
execute
these
all
in
singing
context,
which
means
you
get
that
atomicity,
you
get
better
performance
because
you
don't
have
multiple
transactions.
Things
like
that.
V
Atomicity
you
get
consistent
state,
you
don't
see,
you
know
somewhere
state
where
half
of
your
snapshots
have
been
destroyed,
but
not
the
other
half,
and
you
also
get
more
rapid
development
of
new
sync
tasks.
So
you
can
quickly
prototype
up
some
new
combination
of
these
different
intrinsics,
like
snapshot
and
delete
and
create,
and
rather
than
having
to
you
know,
create
this
entirely
new
sync
task
and
see
you
can
just
write
it
quickly
in
this
scripting
language.
B
V
And
the
other
cool
thing
about
this:
is
it's
a
single
eye
optical?
So,
in
terms
of,
I
think
it's
like
compatibility
across
platforms.
Is
you
only
have
to
support
a
single
optical
and
the
kernel
can
decide.
You
know
what
intrinsics
it
supports
and
maybe
throw
back
an
error
if
you
ask
for
something
that
it
doesn't
provide.
So
this.
B
S
And
a
couple
people
had
mentioned
asking
for
a
better
programmatic
interface,
and
this
would
be
a
better
programmatic
interface,
especially
if
we
there,
you
know,
there's
some
there
we're
glossing
over
a
lot
of
details,
but
it'd
be
nice
to
have
a
way
to,
for
example,
return
data
from
this,
not
necessarily
from
syncing
context.
So
if.
S
V
V
Just
send
this
constant
stream
instructions
that
we
have
pre-generated
or
whatever
it's
easy
to
port,
so
just
to
make
it
more
concrete.
This
is
a
simple
example
channel
program,
so
cfs
destroy.you,
destroy
all
the
snapshots
of
a
file
system
then
destroy
the
file
system
itself.
So
the
way
this
works
today,
like
chris
mentioned,
is
you
you
list
all
the
snapshots
of
the
file
system.
You
issue
a
destroy
on
all
those
snapshots,
and
then
you
destroy
the
file
system
target
itself,
so
you
could
write
a
simple
script.
Instead,
that
has
nice
scripting.
V
This
is
lua
similar
python.
So,
if
you
don't
want,
you
can
probably
learn
the
other
and
you
just
iterate
with
a
nice
for
loop,
you
do
a
destroy
of
each
snapshot
and
then
you
do
destroy
the
target,
and
so
all
this
gets
executed
in
a
single
syncing
context.
It's
faster!
It's
it's
more
correct
and
the
host
application
doesn't
have
to
worry
about
kind
of
working
out
its
way
around
any
weird
edge
cases
where
you
can
get
into
inconsistencies
or
failures.
Halfway
through
a
long
sequence
of
interconnected
operations,.
B
So
maybe
you're
gonna
address
this
later,
but
do
you
want
to
talk
about
like
let's
say
one
of
these
snapshots
can't
be
destroyed
for
some
reason
right
like
it
has
clones,
then
you
know
how?
How
would
that
error
be
handled?
Could
I
still
end
up
with
something
where
it's
like?
Some
of
them
are
destroyed
and
some
of
them
aren't.
S
Run,
I
think
the
important
thing
is
you
can
run
a
bunch
of
in
within
some
constraints,
you
can
run
a
bunch
of
check
functions
and
that
guarantees
that
if
you
run
the
same,
the
same
sync
functions
of
all
the
check
functions
you
just
ran.
All
of
them
should
complete
properly,
so
you
can
kind
of
do
a
dry
run
ahead
of
time
to
make
sure.
B
S
Right
so
in
this,
as
he's
drawn
it
here,
I
think
this
would
actually
like
if
it
fails
to
destroy
a
particular
snapshot.
You
wouldn't
it
would.
It
would
destroy
the
rest,
but
not
destroy
that
one
snapshot,
but
I
think
the
idea
is
you
would
wrap
so
lua
has
first
class
functions.
You
would
have
some
function
like
driver
on
then
do
or
with
a
better
name,
and
you
wrap
this
in
that,
and
so
it
would
for
each
snapshot
in
this.
S
It
would
first
it
would
run
the
code
once
with
just
doing
the
check
functions
and
then,
if
all
the
checks
functions
pass,
it
would
run
the
second
time
running.
All
of
the
action
do
actually
doing
the
work,
and
so,
since
the
check
functions,
all
pass
you'd
be
guaranteed
that
all
the
actual
functions
themselves
would
pass.
I
think.
B
The
cool
thing
about
this
is
that
puts
that
kind
of
semantics
into
the
control
of
the
application.
So
the
application
can,
you
know,
can
think
about
application
writer
to
develop
what
how
you
know
am
I
going
to
be
able
to
check
these?
What
do
I
want?
There
are
semantics
to
be.
Do
I
wanted
to
just
like
do
as
many
as
it
can
and
if
it
finds
one
that
I
can't
do
then
just
drive
on,
but
you
know
maybe.
S
S
Check
the
error
like,
if
you
cared
about
errors,
you'll
be
able
to
actually
define
that
here
and
then,
like
form,
some
output
object.
That
would
contain
the
errors
that
you
cared
about,
rather
than
having
like
whatever
matt
arbitrarily
decided
would
be
the
best.
According
to
him
to
be
like
what
everyone
has
to
use,
yeah,
which
I
mean
for
snapshots,
I
think
that's
not
as
big
a
deal,
but
when
you
get
into
like
the
oh,
I
want
zfs
destroy
dash
p
that
takes
care
of
the
promotes.
Then
there's
like
oh
well
or
is.
S
Q
V
So
this
is
kind
of
a
high
level,
so
we
we
chose.
What
we
want
to
do
is
we
want
to
basically
add
a
lure
interpreter
to
the
kernel.
It'll,
take
these
streams
of
instructions
that
are
passed
down
from
user
space
and
actually
execute
them.
So
the
great
thing
about
lua
is
it's
designed
as
a
domain
specific
language.
So
we
think
it's
going
to
be
easy
to
add
these
kind
of.
D
S
S
S
Language
in
vm,
whether
that's
right
so
before
friday,
our
plan
was
we're
going
to
make
some
envy
list
format
that
the
colonel
will
parse.
That
will
like
let
it
interpret,
and
it
was
basically
like
we
just
had
all
these
artificial
restrictions.
Based
on
how
much
of
an
interpreter
we
wanted
to
write
ourselves.
S
X
S
S
So
right
now,
so
our
goal
is
not:
let's
put
generic
lua
into
it.
It's
let's
put
lua
on
top
of
the
interfaces
that
you
know
this
was
saying
before
we
should
put
lua
should
anywhere.
First
of
all,
lua
barely
has
to
call
out
into
anything.
You
know
it
has
like
one
place
where
it
does
memory
allocation
that
you
can
that's
once
again
like
something
you
can
configure
and
that's
you
know
pretty
much
it
and
then
it
calls
back
into
your
stuff.
So
the
idea
is,
we
put
the
zfs
interfaces.
S
S
G
D
G
B
B
B
S
B
B
Well,
you
mean
not
necessarily
infinite
loop
but
make
but
basically
say
like
you
can
only
run
ten
thousand
instructions
before
you
actually
like
give
us
something
to
do.
You
know,
and
you
know
that
prevents
some
of
these.
Like
worst
case
scenarios,
I
don't
know
if
that's
really
enough
to
let,
like
ordinary
people,
write
these
programs,
but
I
I
wasn't
sure
what
the.
L
B
D
G
S
C
S
L
B
B
The
same
exists,
the
same
problem
exists
today,
where
you
can
create.
You
can
tell
the
kernel
like
hey.
I
would
like
you
to
snapshot
to
create
these
10
000
snapshots.
It's
like
well.
That
might
actually
make
your
transaction
group
take
longer
than
is
ideal
and
hopefully
like
the
right
throttle
will
kick
in
and
then
everything
will
slow
down
gracefully,
but
eventually
you
know
you
could
you
could
get
some
bad
performance
there?
X
B
S
Well,
I
think,
how
would
you
use
cfs
in
this
situation?
Hopefully
this
is
actually
a
bad
example.
We
should
have
swapped
it,
but
zfs
list
presumably
would
be
better
in
this
case
than
the
current
case,
because
in
the
current
case
you
do
need
to
read
everything
into
memory.
In
this
case,
you
could
iterate
on
kernel
memory.
B
B
S
S
B
The
most
constrained
environment
possible,
like
you,
cannot
allocate
memory.
You
cannot
take
locks
right
versus
here
like
we're
in
the
loosest
in
one
of
the
loosest
kernel
contexts
where,
like
we
can
save
space
and
we
can
take
locks
and
like
all
this
kind
of
stuff.
So
I
mean
it
would
be
cool
if
you
could
write
your
detracts
in
lua
too,
but
that's
a
lot
trickier
than
doing
this
project.
D
F
B
F
I
think
it's
also
being
built.
Just
in
general,
I
mean
we
wind
up
with
I've,
done
some
performance
work
for
other
products,
and
the
request
has
been
help
us
avoid
the
context
between
the
user
and
parallel.
So
just
again,
the
machine
the
system
falls
together
is
actually
really
really
nice.
He
also
provides
some
privileges
that
help
the
security
staff
almost
like
send
files,
but
have
like
ultra
descent
bottle
without
something
it's
pretty
cool.