►
From YouTube: Ceph Days NYC 2023: State of the Cephalopod
Description
Presented by: Josh Durgin | IBM
In this talk, we'll provide an update on the state of the Ceph upstream project, recent development efforts, current priorities, and community initiatives. We will share details of features released across components in the latest Ceph release, Quincy, and explain how this release is different from previous Ceph releases. The talk will also provide a sneak peek into features being planned for the next Ceph release, Reef.
A
A
So
today,
I
wanted
to
talk
a
bit
about
the
seat
of
the
stuff
project.
What's
been
going
on
recently
in
the
latest
releases,
what
we're
planning
for
the
future
and
some
of
what's
Happening.
Overall,
it's
been
a
a
big
several
big
years
of
growth.
That's
a
community!
In
general,
we've
changed
to
a
new
governance
model
for
our
technical
governance.
Instead
of
having
a
kind
of
a
single
person
stage.
A
A
few
major
Focus
areas
for
us
in
general
have
been
quality
performance
and
scalability
usability,
so
we've
been
doing
lots
more
testing
at
Large
Scale,
trying
to
detect
things
before
users
hit
them
and
improving
our
processes
like
creating
RC
releases
that
users
can
test
out
at
Large
Scale.
Before
again,
the
final
release
hits
the
ground.
A
We've
also
been
focusing
much
more
on
performance
testing
and
performance
improvements
and
stuff.
A
big
part
of
that
is
the
Crimson
project.
We'll
talk
we'll
talk
more
a
bit
about
later,
but
it's
also
means
trying
to
run
at
much
larger
scales
and
much
more
varied,
Hardware,
more
continuous,
on
a
more
continuous
basis
with
more
realistic
workloads.
A
A
What
what
house
bigger
cluster
is
how
many
osds
it
has
these
sorts
of
information
and
right
now
we
have
somewhere
north
of
800
petabytes
of
of
stuff
clusters,
reporting,
there's
a
massive
amount
of
users
out
there
and
there's
either
all
you
can
inspect
some
of
the
data.
That's
out
there
in
the
public,
toiletry
dashboards,
which
we'll
talk
more
about
later
and
for
the
developers
there's
some
private
dashboards,
where
we
can
go
and
see.
For
example,
if
we're
interested
to
release
how
many
crashes
are
happening,
which
classes
is
it
affecting?
A
A
So,
as
I
mentioned
earlier,
performance
and
scale
has
been
a
big
Focus
area
for
us.
We've
been
doing
lots
more
testing
at
the
number
of
different
clusters.
One
of
the
major
ones
was
the
Super
Computing
Center
in
Australia
are
the
super
competing
Center
which
allowed
us
to
use
a
very
large
cluster
of
theirs
for
testing
out
and
fixing
a
number
of
bugs
in
sefadm
and
other
pieces
of
stuff,
including
the
manager
and
a
very
large
scale.
A
A
We're
also
trying
to
do
more
logical
scale,
testing
on
our
Upstream
in
our
stream
Lab,
which
is
not
as
large.
We
can't
run
thousands
of
osds
and
and
use
that
full
space,
but
we
can
run
thousands
of
osds
with
a
much
more
limited
amount
of
resources
in
space,
so
we're
developing
some
methods
to
actually
test
out
things
you
see
at
scale
without
needing
the
full
scale
of
data.
A
Yeah
finally,
there's
also
more
focus
on
performance
as
well
and
the
performance
it
can.
It
can
be
a
very
interesting
thing
to
to
test
or
many
different
aspects
of
this
one
of
them.
A
lot
of
the
things
you
can
test
for
performance
actually
end
up
showing
up
even
at
smaller
scale,
so
we
have
a
smaller
much
smaller
scale
set
of
high
performance
nodes
in
our
rf3
CPL
lab
and
we're
always
improving
and
and
looking
for
more
help
with
running
tests.
There
currently
I
think
we
have
what
is
it
something
like
16
or.
A
So
within
the
radius
layer,
which
is
the
base
layer
for
everything
and
stuff,
there's
been
a
lot
of
focus
on
a
lot
of
focus
on
stability
and
reliability.
So
one
of
the
major
things
there
is
quality
of
service
being
able
to
maintain
the
performance
of
the
different
clients
versus
when
different
things
are
happening
inside
the
cluster,
like
you
have
lots
of
scrubbing,
going
on
to
check
for
data
Integrity
or
lots
of
recovery
from
failure.
A
If
you
want
to
keep
your
clients
happy
and
keep
the
latency
and
throughput
going
as
you
expect
so
in
Quincy-
and
this
was
made
in
the
default,
we
only
have
a
new
scheduler
called
m-clock
which
implements
this
quality
of
service
for
background
operations.
We
made
further
environments
to
it
are
coming
up
in
brief
and
right
now
it
is
it's
only
for
background
operations
compared
to
clients,
but
in
the
future,
we're
going
to
extend
it
to
be
supporting
different
classes
of
clients,
so
different
clients
can
have
different
reservations
for
amount
of
iops.
A
Let's
try
to
make
this
to
be
easy
to
use,
because
stuff
has
many
many
options
for
tuning
these
kinds
of
things
today.
So
with
with
the
m
clock
scheduler,
instead
of
setting
tens
of
options
to
figure
out
how
fast
you
want
recovery
to
go,
you
just
choose
a
single
profile.
If
you
want
recovery
to
go
faster,
do
you
want
clients
to
go
faster,
or
do
you
want
something
in
the
middle?
A
It's
always
possible
to
go
deeper
and
configure
things
at
a
much
lower
level
than
that,
but
we're
trying
to
keep
things
simple,
to
manage
and
and
maintain.
So,
if
you
don't
want
to,
if
you
don't
need
to,
you,
don't
have
to
carry
around
all
those
nitty-gritty
details
of
thousands
of
configuration
options.
A
This
also
applies
to
some
of
the
other
improvements
we
made
in
radius,
but
in
terms
of
how
we're
reporting
Health
errors
and
how
we're
reporting,
through
operations,
we're
trying
to
make
it
easier
for
folks
to
diagnose
performance
issues
and
stability
issues
in
the
cluster
sooner
and
we'll
see
some
more
improvements
there
in
the
future,
as
well
as
distributed
tracing,
gets
more
embedded
in
the
different
Protocols
of
stuff.
Brief
will
be
the
first
release
where
then
that's
going
to
be
present
and
easily
Deployable.
A
So
also
coming
up
in
Reef,
there
are
lots,
more
improvements
for
blue
store
for
performance
purposes
and
and
also
potentially
addressing
different
kinds
of
performance
problems
that
kind
of
throw
up
like
fragmentation.
This
has
been
a
major
cause
of
issues
in
Blue
Store
in
the
past,
and
there
are
some
thoughts
there
and
improving
the
way.
A
It's
becoming
more
automatic
is
the
general
theme
that
we've
been
striving
for
in
rados,
and
another
aspect
of
that
is
improving
the
balancer,
how
we're
Distributing
data
across
the
cluster.
So
today
the
balancer
Works
only
with
them
look
looking
at
the
purely
there's
a
capacity
used
on
each
OSD
and
moving
it
around
to
try
to
even
things
out
in
Reef.
This
also
includes
balancing
across
the
primaries,
because
when
you,
when
stuff
is
doing
reason,
rights
reads
are
always
going
to
the
primary
OSD.
A
A
So
they
go
away
the
maximum
speed
from
the
wire
to
the
disk.
This
is
in
the
experimental
stage
right
now,
where
it
supports
the
basic
operations
of
RBD,
but
it's
still
undergoing
stabilization
and
has
a
long
way
to
go
in
terms
of
optimizations
before
being
fully
production,
ready
and
being
able
to
be
used
for
all
workloads.
A
So
in
Reef
we
have
some
initial
support
for
multiple
reactors
and
snapshots,
but
we're
continuing
to
build
out
the
test,
Suite
there
and
continuing
to
improve
the
functionality
and
to
make
it
fully
able
to
be
used
for
RBD
in
the
next
release.
Probably
yes,
a
big
aspect
of
that
as
well,
is
the
c-store,
which
is
a
new
store
back
in
the
OSD
designed
specifically
for
high
performance
systems
like
ndme
disks
and
other
sorts
of
very
high
performance
media.
A
So,
as
I
mentioned
earlier,
we
have
this
transformatory
system,
which
is
all
opt-in,
reporting,
we're
trying
to
make
this
easier
to
use
and
easier
to
update.
So
we
introduced
a
number
of
Notions
around
how
to
manage
the
data
there
I
think
you
Read's
going
to
talk
a
bit
more
about
this
in
more
detail
later
so
I'm
going
to
skip
through
this
pretty
fast,
but
essentially
we're
Gathering,
both
metadata
event,
clusters
crashes
that
happen
and
also
information
about
Drive
drives
that
fail.
So
we
can
feed
that
back
into
models
predicting
values
before
they
occur.
A
So
in
Quincy,
it's
much
easier
to
manage
things
like
expanding
the
cluster,
adding
new
host,
removing
them,
and
it's
gained
support
for
a
number
of
different
protocols
like
monitoring
protocols
and
adding
proxies
to
be
able
to
make
things
h
a
and
reef.
It's
expanding
that
support
across
different
functionality
and
RBD
and
rgw,
especially,
and
trying
to
make
things
easier
to
Monitor
and
easier
to
diagnose
by
integrating
the
centralized
logging
as
well.
So
you
can
have
all
your
logs
go
to
a
standard,
elasticsearch
based
platform
to
inspect
and
and
sift
through.
A
So
fadm
is
the
current
way
to
deploy
our
clusters
with
stuff
which
isn't
designed
to
be
as
easy
to
use
as
possible.
It's
deploying
them
via
containers
and
system
D,
but
managing
it
all
through
the
step.
Orchestration
commands,
which
are
common,
Zippos,
fadm
and
Brook,
which
is
the
deployment
method
on
top
of
kubernetes
stuff.
Idiom
itself
has
gained
support
for
a
number
of
different
things
in
Quincy
and
in
general,
we're
trying
to
make
it
easier
to
use
and
easier
to
deploy
in
flexible
ways
that
people
would
like
to
so
in
Reef.
A
We're
also
making
it
simpler
to
upgrade
more
piecemeal
so
that
you
don't
have
to
upgrade
the
whole
thing.
Whole
thing
at
once.
That
was
a
little
bit
more
difficult
in
the
past,
making
it
simpler
to
set
up
multi-site
deployments
for
the
radius
Gateway
and
for
mirroring
with
RBD,
and
we're
also
added
support
for
automatically
rotating
the
keys
for
authentication
for
the
inside
demons,
which
keeps
your
clusters
to
the
gear.
A
So
Rook
is
the
main
way
to
play
stuff
on
top
of
kubernetes
and
I
work
as
I,
always
catching
up
and
keeping
up
with
new
features
and
stuff
and
new
features
in
kubernetes.
So
now
it's
supporting
much
much
easier
ways
to
troubleshoot
a
stuff
cluster
I
believe
there's
even
a
new
project,
a
little
bit
separate
from
work
to
kind
of
gather.
Some
of
these
troubleshooting
commands
together
in
one
place,
to
make
it
simpler
to
diagnose
and
and
to
inspect
a
cluster
deployed
in
a
kubernetes
environment.
A
So
RVD
the
block
device
portion
of
zeph
is
one
of
the
most
popular
uses
for
it.
It's
been
very
stable
for
a
long
time
and
now
we're
starting
to
see
much
more
interest
in
more
advanced
protocols
and
more
advanced
functionality
with
it.
So
there's
a
lot
I'll
generate
around
the
ndme
or
Fabrics
protocol.
A
A
Interesting
research
in
the
future
is
ongoing
from
a
group
at
Northeastern
University
into
a
long
structured
format
for
IBD,
which
has
the
potential
to
drastically
improve
the
performance
for
random.
I
o
in
particular,
but
it
has
some
restrictions
on
what
workloads
it
applies
to
I.
Think
it's
a
very
interesting
effort
for
the
future.
A
In
radius,
Gateway,
the
SV
and
object
storage
interface
for
Seth
has
a
number
of
ongoing
efforts.
A
lot
of
these
are
around
supporting
Ai
and
analytics
types
of
types
of
workloads
like
things
like
SBA,
select,
being
able
to
run
SQL
queries
against
your
objects,
and
this
is
a
particularly
interesting
project,
because
it's
able
to
run
Standalone
as
well,
so
you
can
easily
develop
your
queries
against
a
local
file
and
then
run
them
against
your
cluster.
A
This
will
let
you
in
debug,
where
operations
are
getting
selective
or
where
performance
is
slowing
down,
basically
each
step
of
the
way.
So
you
can
see
if
something's
being
synced
from
site
a
to
type
B.
Is
it
getting
stuck
or
is
it?
Is
it
slowing
down
outside
a
or
site
b
or
which
part
of
it?
It's
going
wrong?
A
If
you
have
a
problem,
there's
a
lot
of
work
and
the
readers
gateways
zipper
back
end,
which
is
a
kind
of
applicable
back
end
to
let
you
put
different
things
behind
rgw
for
different
purposes.
If
you
wanted
to
run
it
in
a
very
constraining
environment,
for
example,
you
could
back
it
with
a
simple
file
or
a
simple
database.
A
A
The
last
piece
I
want
to
touch
on
is
testing
and
quality
in
general,
so
there's
a
few
different
aspects
to
this
one.
A
new
effort
that
we're
working
to
focus
more
on
this
year
is
the
developer
experience.
If
it's
very
difficult
to
run
the
test
for
a
system,
it's
it
makes
it
much
harder
to
add
new
tests
and
keep
them
maintained.
So
if
we
can
create
a
way
to
run
these
these
these
tests
locally,
instead
of
having
to
wait
hours
for
our
builds
and
push
it
through
a
queue
for
days.
A
In
case
there
is
a
problem
with
one
another
aspect
of
that
is
engaging
with
other
organizations
that
have
Hardware
they'd
like
to
help
other
community
with
and
enabling
them
to
contribute
in
by
rather
running
tests
themselves,
an
RC
candidates,
for
example,
or
maybe
providing
some
backup
Hardware
in
case
our
CPA
lab,
has
an
issue.
We
need
to
get
things
going
to
be
able
to
merge
code
and
and
get
releases
out
the
door
and
finally,
as
I
mentioned
earlier,
looking
into
improving
performance
by
a
more
continuous
basis
and
monitoring
it
via
a
CI
system.