►
From YouTube: CHERI Memory Safety with OpenZFS by Brooks Davis
Description
From the 2022 OpenZFS Developer Summit: https://openzfs.org/wiki/OpenZFS_Developer_Summit_2022
Slides: https://drive.google.com/file/d/10SVS5psBpBF-fm_k5ATc80vf2K6qufUi/view?usp=sharing
A
So
before
I
talk
a
little
more
about
how
the
Cherry
mechanism
works.
I'm
first
going
to
talk
about
the
motivation
for
running
cherry
on
ZFS
and
the
motivation
is
that
we
have
real
Hardware
with
cherry
extensions
so
due
to
some
funding
by
either
ukri
and
with
joints
funded
with
arm
they've
built
an
actual
prototype
board,
so
arm
took
the
Cherry
ideas
they
adapted
to
them,
adapted
to
them
into
the
arm,
v8a
instruction
set
and
they
built
a
system
on
chip.
So
it's
a
four
core
2.5
gigahertz
N1
based
CPU.
A
Now
the
N1
micro
architecture,
if
you're
not
familiar
with
it,
is
the
one
that's
the
basis
for
things
like
graviton
and
the
Ampere
arm
systems
that
are
in
all
the
other
Cloud
providers.
So
it
is
a
serious
core.
It's
a
you
know
it's
a
modern
super
scalar
processor
and
they
took
it.
Took
that
and
they
did
in
one
year,
which
is
insane
they
took
one
year
and
applied
Cherry
to
that
micro
architecture.
A
It
also
has
a
a
GPU
built
into
it
in
order
to
be
able
to
demonstrate
Android,
but
the
real
goal
was
the
the
CPU.
Then
they
built
a
board
and
they
stuck
it
on
there
and
we
have
a
whole
software
stack.
We
took
out
the
software
we've
developed,
which
is
the
most
developed
part,
is
a
cherryvsd
which
is
a
fork
of
FreeBSD
adapted
to
Cherry,
and
we
have
an
llvm-based
compiler
arms
working
out,
a
GCC
based
compiler,
and
we
also
have
GDB.
We
have
most
of
the
things
you
would
expect.
A
A
So
we
started
to
make
an
attempt
at
a
port
before
I
talk
about
that.
Port
I'll
talk
a
little
more
about
what
Cherry
actually
is
so
Cherry
adds
a
new
hardware
type
the
capability,
a
cherry
capability.
You
can
think
of
it
as
a
pointer
with
bounds.
It's
basically
a
fat
pointer,
it's
held
in
memory
and
in
and
in
extended
registers.
A
So
you
have
an
address
just
like
you
would
with
a
regular
pointer,
which
you
can
also
store
an
integer
in,
because
that's
a
thing
C
programmers
do
and
then
we
also
have
permissions
and
we
have
compressed
bounds
and
that
takes
up
128
bits.
So
we
have
a
bit
of
a
floating
Point
like
compression
of
the
bounds,
because
our
initial
prototype
was
256
bits
and
had
bass
and
bound
had
a
top
and
a
bottom
that
were
just
explicit
and
we
knew
from
the
day.
A
We
started
that
no
system
architect
would
ever
accept
that
that's
not
possible
but
128.
Maybe
that's
inbounds,
there's
also
a
hidden
validity
tag.
The
tag
ensures
that
capabilities
are
have
or
ensures
their
integrity.
So
if
you
have
a
string
that
string
overflow
spring
buffer
and
you
write
over
a
capability
with
a
bunch
of
bytes,
it's
not
a
capability
anymore.
A
A
You
could
actually
go
a
bit
in
and
outside
of
those
bounds,
because
that's
also
something
that
c
does
in
addition
to
just
the
one
past,
the
end
that
your
let's
see
requires
people
actually
go
a
bit
below
or
a
bit
above,
but
either
due
to
compilers
being
lazy,
because
it's
never
mattered
or
due
to
attempts
at
optimization.
We've
like
we
found
some
stuff
with
z-lib,
where
they
pre-decrement
the
the
pointer,
and
so
they
could
increment
it
right
at
access
and
supposedly
it
was
faster
on
a
HP,
something
in
the
90s.
A
You
know
so
the
thing
with
cape,
these
carry
capabilities,
is
they're
used
to
reference
memory
and
in
a
cherry
system
all
memory
access
is
via
a
capability.
So
it's
either
explicit
through
new
instructions
that
that
use
the
registers
or
a
new
encoding
mode
where
all
operations
refer
to
a
capability
register
rather
than
an
integer
register,
or
it's
implicit
via
a
default
data
capability
or
a
program
counter
capability.
A
Now
cherry
and
force
Cherry
allows
you
to
do
two
things.
Two
main
things
there's
other
things
you
can
do.
It's
like
very
flexible
at
its
very
low
level.
So
obviously
there
are
all
sorts
of
opportunities
of
things
to
do,
but
the
first
thing
it
does
is:
it
provides
efficient,
fine-grained
memory,
protection
for
CNC,
plus
plus
we
get
the
sort
of
safety
guarantees
that
you
get
in
modern
languages.
In
many
ways
we
get
we
get
strong
spatial
safety
and
with
an
operation
with
a
operating
system,
assist
add
a
bit
of
improvement
to
the
architecture.
A
A
The
other
thing
you
can
do
is
scalable.
Compartmentalization
I
already
talked
about
that.
But
that's
that's
basically
the
two
things.
So
we
I
said
that
you
know
Cherry
capabilities
can
be
treated
as
pointers.
We
have
two
ways:
we
do
that.
First,
we
have
a
hybrid
mode
in
the
hybrid
mode.
Only
select
pointers
have
been
converted
to
Cape
to
capabilities
and
other
pointers
remain
integers
and
are
access
and
are
used
via
the
default
data
capability.
A
There's
also
a
pure
capability
mode,
where
we
compile
every
capability
every
pointer
into
a
capability
and
and
go
over
there,
so
yeah.
Here
we
are
so.
We've
turned
both
all
regular
pointers
and
uint
pointer
T's
into
capabilities.
We
also
the
compiler,
also
places
bounds
on
objects
where
it's
it's
responsibility,
so
it
so.
If
you
take
a
reference
to
a
stack
allocation
and
pass
it
off
to
some
foreign
function,
that's
a
bounded
reference
so
that
that
function
can't
can't
attack
your
stack
addition.
A
But
there's
the
C,
for
instance,
requires
that
a
reference
to
the
first
member
of
a
structure
be
equivalent
to
the
reference
to
the
structure.
So
things
Break.
However,
when
we
compile
our
kernel
and
peer
capability
mode,
we
turn
on
sub-object
bounds,
and
so
we
have
the
have
FreeBSD
pretty
well
adapted
to
this.
A
A
That's
not
recompiled,
and
then
in
both
cases
we
don't
touch
all
the
regular
integer
types,
so
in
Long
size
T,
all
those
time
T
those
sort
of
things
they
stay
the
same.
This
is
good
because
it
saves
us
a
lot
of
pain.
When
you
do
32-bit
compatibility,
that's
a
lot
more
work.
You
know,
32
versus
64-bit
compatibility,
there's
a
whole
bunch
of
things
that
change
here.
The
number
of
things
that
change
are
relatively
small,
so
yeah
also
there's
implied
pointers.
A
A
That
sort
of
thing
those
in
Cherry
in
the
pure
capability
world
are
capabilities
as
well
and
then
so
hybrid
is
the
default
Avi
in
cherrybsd
or
sorry,
it's
implemented.
Sorry
hybrid
is
implemented
as
a
compatibility
layer
in
cherrybsd,
and
so
we
have
a
freebsd64
compatibility
layer
just
like
our
freebsd32
compatibility
layer.
A
This
is
one
of
the
nice
abstractions
we
have
in
FreeBSD
in
large
part
because
when
it
got
ahead
of
us,
so
we
had
to
implement
compatibility
and
therefore
now
we
have
this
really
great
abstraction.
Whereas
when
we
look
at
Linux
or
we
talk
to
people
with
Windows,
their
compatibility
abstractions
are
not
quite
as
flexible
as
ours,
so
that
that
leads
to
some
sadness.
A
If
you
want
the
search
keyword,
so
how
much
work
did
that
take
well
quite
a
lot
of
work,
but
total
changes
are
actually
really
quite
small.
So
in
the
kernel
we're
looking
at
less
than
0.2
percent
of
the
entire
kernel
and
Driver
code
is
even
less.
You
know
we're
looking
at
0.04
percent,
it's
usually
just
adjusting.
A
I
o
controls,
or
occasionally
people
will
be
doing
something,
unspeakably
clever,
with
structures
and
aliasing
and
and
whatnot,
and
we
have
to
do
a
little
work
there,
but
it's
usually
slapping
annotation
here
slapping
annotation,
there
change
a
type
change,
a
type
that
sort
of
thing.
So
really
quite
small,
and
now
let's
talk
about
cherry
and
ZFS,
so
let's
get
to
the
punch
line.
First,
here's
a
screenshot
of
a
serial
console
on
the
Morello
box
on
my
desk.
A
A
So
it
was
all
inline
assembly,
and
that
was
much
easier
to
do
where
we
have
a
pure
capability
kernel
and
we
do
use
it,
but
we
suspect
the
dis
a
little
slower,
we're
trying
to
understand
which
of
the
overheads
are
inherent
and
which
are
micro,
architecture
issues
or
which
are
things
we
just
need
to
refactor
a
bit.
So
most
of
us
work
on
the
hybrid
apis
or
hybrid
kernel,
most
of
the
time.
Yes,
well,
let's
take
a
look
at
the
ZFS
binary.
A
Well,
it
is
a
c64
which
means
binary,
which
means
that
it's
the
the
arm
instruction
set
in
capability
mode,
where
loads
and
stores
use
capabilities
directly
and
also
Cherry
API
is
the
pure
capability
API
and
then
there's
my
root
file
system.
So
installer
works.
We
can
run
on
ZFS
root.
So
that's
all
great,
but
it's
in
our
tree.
It's
not
in
your
tree.
So
that's
that's
something
to
fix,
and
also
there
are
some
issues
so
I'm
going
to
talk
about
that.
A
So
yeah
I
say
we
can
install
and
boot
one
gotcha
there's
some
minor
on
disk
format
changes.
They
are
related
to
metadata
the
big
one
that
I
know
of
because
I
ran
into
it
and
had
to
work
around
it
and
that
probably
the
workaround
probably
fixed
the
others
or
masked
the
others
is
your
your
view
might
be.
Is
v-dev
labels,
our
xdr,
encoded,
NV
lists
and
unfortunately,
Envy
lists
encode
the
pointer
size
in
their
packed
format?
I'll
talk
more
about
that,
so
changes
are
pretty
small.
A
A
A
Definitely
we
have
flopped
the
lines
in
the
installer,
so
that
is
not
the
default,
because
I
don't
want
to
strand
users
on
a
potentially
incompatible
file
system
format
they
get
upset.
When
you
do
that
and
then
I'll
give
you
an
overview
of
the
changes.
First,
the
bottom
line
change,
47
files,
you
know
100
and
a
couple
hundred
changes
pretty
minor.
Overall,
we
haven't
done
pair
I
haven't
done
a
pure
capability
kernel,
I
haven't
got
one
booting,
yet
it
will
require
a
few
more
annotations.
A
I,
don't
expect
it
to
be
bad,
but
I
haven't
done
more
than
boot,
it
and
say
yep,
it
doesn't
work
and
it
doesn't
work
where
I
expected
it
to
so,
mostly,
you
know,
as
I
said,
we're
improving
the
Precision
around
types.
There's
some
funny
name
types
here,
so
you
know.
If
you
want
to
store
a
planer
plane
or
an
integer,
usually
you
would
use
an
end
pointer
T,
but
we
have
some
other
ones
for
ABI
preservation
or
at
least
try
to
salvage
the
API.
A
We
added
this
un64
pointer
T,
which
is
to
say
something
that
is
60
at
least
64
bits
and
can
hold
a
pointer.
This
preserves
the
existing
use
of
64-bit
and
then
I
have
this
even
weirder
one
over
here
un64
cap
T,
which
is
a
type
that,
if
you're,
not
using
capabilities
and
you're
or
you're,
not
in
any
way
or
you're,
not
in
the
kernel
and
in
hybrid
mode,
then
it's
just
a
64-bit
ant
and
otherwise
it's
an
in-cap
T.
A
A
So
if
you're,
manipulating
an
integer
is
an
address,
we've
introduced,
this
new
type
pointer
adity
might
want
to
spell
it
differently
in
in
something
in
ZFS,
because
it
has
to
go
on
multiple
platforms,
but
arm
is
pretty
confident.
No
one
is
using
this
type
anywhere.
They
did
a
pretty
serious
check
both
of
Open
Source
and
in
their
test
databases,
and
this
is
what
we've
been
using
to
talk
to
the
world
about
this.
A
So
storing
you
know,
if
you
want
to
store
an
integer
and
a
pointer,
we
require
that
you
cast
it
through
in
pointer
T,
rather
than
just
doing
it
mostly
to
say
yeah
I
meant
it
it's
a
fairly
minor
change.
It
often
happens
already
in
the
code
and
then,
if
you
want
to
store
a
pointer
opaquely,
you
know
use
something
that
could
hold
a
pointer,
don't
don't
using
it
64.,
there's
a
few
annotations
related
to
the
user
space
capabilities.
That's
almost
all
the
code.
A
A
A
A
A
My
suspicion
is,
it
was
a
proprietary
tool
that
someone
had
a
goal
to
make
it
pass
and
that
no
one
has
access
to
it.
So
maybe
we
should
just
remove
it,
remove
this
obfuscation
but
something
to
discuss
another
one.
This
is
a
I
think,
FreeBSD
specific.
We
have
support
for
pre-open
zfsi
octals
on
on
FreeBSD,
so
that
we
have
so
that
you
can
install
a
new
kernel
and
your
old
and
your
tools
still
work
seems
important.
A
It
seemed
mostly
pointless
on
Cherry
BSD,
because
I'm
never
gonna,
stick
a
drive
in
from
some
agent
laptop
in
my
cherry
box.
I'm,
not
worried
about
that
case.
So
I
just
turned
I
just
added
some.
If
deaths
to
disable
it
and
I.
Think
there's
a
question
overall
for
how
long
that
code
needs
to
remain
for
FreeBSD
I,
think
it
supports
freebsd12,
so
that's
still
important,
but
so
we're
probably
a
ways
off
from
removing
it.
But
that's
that
now
on
to
Envy
lists,
Envy
lists
were
my
biggest
source
of
trouble.
A
I've
made
several
false
attempts
at
doing
the
port
and
I
finally
have
something
that
at
least
I
think
works,
except
for
the
caveats
we'll
get
to
so
first
off.
Let's
look
at
struct
NV
list,
so
unfortunately,
the
nvl
priv
member
here
is
a
pointer
in
you
when,
when
the
nv-list
t
is
is
in
use.
So
it's
not
you.
It's
not
stored
as
a
pointer
when
it's
serialized,
but
it
is
in
the
structure
and
yeah,
as
you
can
tell
it's,
not
neither
the
right
size
or
the
right
alignment.
A
The
obvious
thing
to
do
is
just
make
it
a
unit
pointer
t
or
a
u
un,
64
pointer
T.
But
then
alignment
doubles
the
size
of
this
thing,
which
is
a
little
Annoying,
not
the
end
of
the
world.
Obviously
a
little
rearrangement
can
fix
that
and
in
fact
it
seems
extremely
unlikely.
Anything
would
be
relying
on
hand
coding
the
offset
of
nvl
flag
because
as
far
as
I
could
tell
there
are
no
users
of
nvl
flag,
so
it
should
probably
be
named
pad
zero.
A
So
good
news
is
Envy
list
is
primarily
in
metamory,
except
when
it's
not
the
size
is
encoded
in
the
size
of
embedded,
Envy
lists
and
envy
list
arrays,
and
in
the
case
of
in
the
case
of
the
Native
format,
it's
actually
encoded
there
because
it
reserves
space
for
that.
Nv
list
object
in
the
in
the
allocation.
It's
unpacked
in
place.
So
let's
talk
about
well,
let's
talk
about
Envy
pairs
now,
which
are
the
the
actual
data
storage
for
the
NV
list.
So
first
we
have
a
struct
MV
pair.
It's
nicely
sized.
A
You
know,
I
might
want
a
a
union
here
to
make
sure
that
the
enum
doesn't
do
the
wrong
thing
or
I
might
want
c21.
So
we
can
specify
a
type
but
yeah,
it's
fine,
and
then
it's
followed
by
a
name
and
that's
you
know
well-defined
size
from
the
struct
and
then
there's
a
pad
and
that's
padded
out
using
NV
align
which
pad
Soto
pad
it
to
zero,
zero
or
eight
bytes,
so
that
the
value
can
be
aligned.
A
So
Envy
align
aligns
to
eight.
Let's
talk
about
the
values:
well,
here's
a
partially
complete
list
of
the
set
of
values
that
are
U.
The
value
of
set
of
types
used
in
open,
ZFS,
I'd,
originally
just
picked
some
random
ones,
and
then
this
morning
I
decided
I
should
actually
read
the
code
and
see
which
ones
were
to
use
and
the
answer
turned
out
to
be
all
the
ones.
I
had
there
and
a
few
more
mostly
a
bunch,
more
arrays,
and
that's
that's
not
quite
complete.
A
What
you
get
is
a
pointer
to
an
array
of
pointers,
and
so
that's
encoded
in
the
size
of
the
NV
pair
and
it's
encoded
in
in
the
native,
the
native
format,
as
well
as
the
xdr
format
and
the
reason
so
at
xdr.
It's
encoded
because
the
format
includes
includes
the
decoded
size
of
the
Stream,
which
is
handy
for
being
able
to
pre-allocate,
and
the
model
is
that
you
find
out
what
size
the
stream
is
going
to
be,
because
there's
a
there's
an
opt
to
do
that.
A
Internally,
you
allocate
that
memory
and
then
you
decode.
This
is
all
fine,
except
that
we've
encoded,
a
not
very
future-proof
size
for
pointers,
seems
like
that
20
years
ago
and
128
bit.
Pointers
that
are
addresses
still
seems
crazy
to
me,
even
though
risk
5
is
talking
about
it.
But
you
know
here
we
are,
we've
got
Cherry,
we've
got,
you
know,
RV
128,
someday
Maybe,
and
so
that's
encoded.
A
We
need
to
figure
out
what
to
do
about
that,
and
then
native
also
includes
the
storage
for
pointers,
because
it
actually
sends
all
the
padding
it
doesn't.
You
know
it
does
all
the
right
things.
It
zeros
out
the
pointers
before
passing
them
across
before
passing
the
empty
space
across,
and
then
it
reinstantiates
them,
but
does
that
and
it
also
embeds
the
alignment?
A
Let's
see
so
yeah
current
attack
I've
got
is
that
I've
got
the
Envy
I've
made
Envy
priv
a
pointer
and
then
on
Cherry
systems,
I've
been
I've,
increased
the
size
of
embedded,
pointers
and
I.
Do
it
unconditionally,
anytime,
you're,
compiling
for
capability
mode
and
envy
align
also
increases
the
alignment
I
initially
bumped
Envy
version
and
the
result
was
I
got
something
that
I
could
create
pools
and
I
could
do
all
the
management
and
stuff
and
then
I,
rebooted
and
I
was
sad.
A
It
appears
there
is
something
that
is
checking
my
best
guess
so
far
is
there's
something
that
is
actually
doing
a
hard-coded
check
of
the
version
rather
than
unpacking,
but
I
have
not
found
the
code.
That's
doing
it
because
I
ran
out
of
time,
so
I
backed
that
out.
I
have
a
patch
to
back
that
attitude
works
for
now,
but
no
formal
testing.
It
is
unfortunately
incompatible
on
disk.
So
how
to
fix
for
real.
This
is
something
I
want
help
with.
A
What
do
we
do
about
the
xdr
format
on
disk?
Do
we
learn
how
to
write
things
with
the
old
decoded
format,
values,
or
do
we
have
some
sort
of
a
managed
upgrade
path
where
we
switch
to
always
packing
to
128
bits
instead
of
64
bits
and
we
teach
the
other
code
how
to
deal
with
it?
I
I,
don't
know:
I
have
ideas
I'd
like
to
talk
about
it
tomorrow,
I
see
that
I
am
out
of
time,
I'm
fairly
close,
if
that's
okay
and
then
what
about
Native
format?
A
It's
used
in
ioctals,
not
on
disk.
For
us,
it's
probably
not
an
issue
because
we're
not
going
to
try
to
run
random
containers
with
old
binaries
on
our
systems,
but
eventually
we'll
need
to
do
something.
Maybe
we
should
just
switch
to
xdr,
so
quick
call
to
action.
I
need
a
better
resolution
for
the
NV
list.
Issues
I
want
to
talk
about
that
talk
about
different
trade-offs,
think
about
what
we
can
do
for
the
future.
A
I'd
like
to
merge
changes
that
are
acceptable,
Upstream
and
again
talk
about
that
tomorrow
and
I
think
this
first
step
would
be
cleanups
and
improvements.
The
Second
Step
would
probably
be
pure
capability
support,
So.
Any
changes
that
are
required
for
that
hybrid
I'm,
not
sure
it
will
ever
make
sense.
We
would
like
to
see
hybrid
mode
go
away,
but
it
may
be
that
it
stays
around
for
performance
reasons.
A
So
we'll
see
it's
a
it's
more
disruptive
and
then
I'd
like
to
get
the
test
Suite
running
on
cherrybsd
also
get
cross
compilation
support
because
it
turns
out
that
pure
Cape
compiling
for
peer
capability
mode
is
usually
enough
to
detect
incompatibilities
that
you
don't
actually
need
to
run
once
the
code's
been
ported,
because
code
changes
typically
don't
introduce
new
ways
that
the
compiler
doesn't
warn
you
about.
So,
hopefully
a
CI
setup
could
work.
We
have
the
ability
to
run
in
qemu.
We
could
also
potentially
run
a
cherry.
Build
bot.
A
University
of
Cambridge
has
a
bunch
of
machines
in
a
data
center,
as
does
Microsoft
research.
Microsoft
research
has
theirs
hooked
up
to
GitHub
actions,
so
we
could
possibly
do
things
there.
So
the
future
I
can't
tell
you
what
the
future
is
of
cherry
because
I
don't
know,
but
I
hope
it
is
the
way
we
can.
We
design
all
future
architectures
so
we're
starting
up
a
Sig
at
risk.
Five
to
do
risk.
5
standardization
arm
is
quite
open
to
this
and
happy
with
it.
They
are
quite
well
aware
they
can't
go
this
alone.
A
A
So
a
no
point
of
reference
is
just
like
today,
it's
a
fault
and
also
an
out
of
bounds.
Read
is
also
also
turns
into
a
fault,
so,
if
you
want
to
build,
you
can
build
availability
on
top
of
cherry
using
compartmentalization,
but
in
a
normal
circumstance
you,
you
now
turn
things
that
might
do
horrible
things
to
your
system
into
in
defaults.