►
From YouTube: NYLUG Presents: Patrick McGarry on Ceph, A Distributed Objectstore, Block Store & File system
Description
Ceph is a Free Software unified storage platform designed to present object, block, and file storage from a single distributed cluster.
Ceph's main goals are to be completely distributed without a single point of failure, scalable to the exabyte level, and freely-available.
The data is seamlessly replicated, making it fault tolerant. Ceph is a software-based solution and runs on commodity hardware. The system is designed to be both self-healing and self-managing and strives to reduce both administrator and budget overhead.
This talk will provide a quick overview of Ceph architecture and its primary use cases. Examples will include cloud applications using object and block as well as pure storage configurations. We'll also examine the current state of the community, the near future of the Ceph project, and ways the community can get involved. Projects range from 'low-hanging-fruit' suitable for non-technical or novice developers to more robust architectural work for seasoned developers.
Links:
Ceph Homepage: http://ceph.com/
Speaker Bio:
Patrick McGarry is currently incarnated as a community monkey working to spread the Ceph gospel for Inktank. An experienced community manager, gamer, mischief maker, and all around geek, Patrick spent five years writing and managing Slashdot under the nomme du keyboard 'scuttlemonkey.' Patrick enthusiastically helps companies to understand and adopt Open Source ideals and continues to be a strong advocate of FOSS on the desktop and in the enterprise. He still hates writing these bios.
A
A
I'd
like
to
thank
our
other
sponsors
ibm
canonical
brandor
group,
google
and
o'reilly
media
for
their
continued
support
throughout
the
years
and,
of
course,
nyla
wouldn't
be
able
to
do
what
it
is
today
without
the
gracious
time
of
all
of
our
volunteers,
all
right
without
any
further
ado,
petr
mcgarry.
What's
new
with
steph.
B
Thanks
thanks
all
right:
hey
everybody
here
to
talk
about
seth.
How
many
people
raise
a
hand,
know
what
ceph
is
have
heard
about
it
all
right,
so
we'll
breeze
through
the
architecture
overview,
pretty
quick,
then,
and
get
just
to
what's
going
on
cool
who
I
am
patrick
mcgarry.
I've
been
hanging
out
at
ink
tank
now
acquired
by
red
hat
hooray
for
some
people
and
others
are
have
big
question
marks.
So
hopefully
we
can
answer
some
of
those
tonight.
B
B
I
don't
work
for
slashdot
anymore.
Obviously,
but
then
I
kind
of
went
out
into
the
corporate
world
played
with
alcatel
lucent
and
perforce
a
little
bit
tried
to
bring
in
a
little
the
open
source
gospel
and
now
I'm
back
at
inktank
and
and
the
open
source
world.
Then
it
feels
like
coming
home.
So
that's
a
hooray
for
me.
I
I
guess,
we'll
jump
right
into
it.
This
is
what
we're
going
to
be
doing.
So
I
figured
we'd
go
over
a
couple
of
industry
musings.
B
We'll
talk
a
little
bit
about
an
intro
to
seth
kind
of
where
it
came
from
a
little
bit
of
its
history.
For
those
that
don't
know
about
it
and
then,
like
I
said,
there's
a
big
section
in
the
middle:
that's
the
architecture
overview
I'll,
try
not
to
spend
a
ton
of
time
on
that.
So
it
looks
like
we
have
a
lot
of
veterans
here,
but
you
know,
maybe,
if
we're
breezing
through
it
really
quickly,
then
you
guys
can
make
it
a
little
bit
more
interactive.
B
We
can
ask
questions
and
dive
a
little
deeper
into
the
pieces
that
that
you
have
questions
about
and
then
we'll
talk
about
my
piece,
the
community.
What's
going
on
what's
happening,
there's
a
lot
of
a
lot
of
stuff
going
on
turns
out
when
you
take
a
little
piece
of
of
something
like
ceph
and
you
drop
it
into
a
giant
bucket
of
rocket
fuel.
That
is
all
of
the
resources
of
red
hat.
B
Lots
of
things
can
happen
that
hadn't
been
able
to
before,
and
then
I
can
mention
kind
of
the
commercial
stuff
on
the
the
very
end.
But
we
won't
spend
a
lot
of
time
about
that
because
I'm
more
about
the
open
source,
so
here
looking
forward
and
these
numbers
are
actually
pretty
old-
a
study
was
done.
I
won't
give
you
all
the
sightings,
but
2020
they're,
expecting
over
15
zettabytes
of
storage
to
be
used
in
the
industry,
they're,
actually
thinking.
B
B
So
well,
you
know
the
the
you
got
to
come
up
with
new
ways
to
do
things.
You
can't
sit
there
and
and
do
the
same
old
giant,
big
black
box
forklift.
You
know
you
subscribe
to
somebody,
and
then
they
ask
you
to
pay
for
your
with
your
firstborn.
You
got
to
keep
having
children
to
get
it
infrastructure
and
that's
just
never
a
good
plan.
So
you
know
we're
kind
of
past
this,
the
scale
up
and
we're
going
to
go
out
into
the
scale
out.
B
B
Do
it
cheap
plan
for
failure
as
the
norm
as
opposed
to
the
exception
that
kind
of
thing
you
know
all
of
the
the
nice
mantras
that
got
google
and
amazon
to
where
they
are,
and
on
many
of
the
people
in
this
room's
back,
I'm
sure
so
a
little
bit
about
sef
where'd.
It
come
from
sef
started
actually
a
long
time
ago.
2004
is
when
the
first
code.
Well,
I
think
the
first
code
was
written
in
2003,
but
it
didn't
actually
get
checked
in
until
2004..
B
It
started
as
a
project
out
of
a
department
of
energy
grant
that
sage
weil,
our
founder
and
architect.
He
started
this
as
part
of
his
doctoral
research
at
the
university
of
california,
santa
cruz,
and
actually
that's
where
the
name
comes
from
we'll
take
a
quick
aside.
I
know
I
get
that
question
a
lot
ceph
as
in
cephalopod,
because
the
mascot
at
uc
santa
cruz
was
sammy.
B
The
banana
slug
and
sage
wanted
to
keep
it
at
least
kind
of
in
kind
with
where
he
was
doing
all
this
good
work,
and
so
that's
where
seth
came
from
and
then
inktank
came
kind
of
as
a
result
of
that.
The
funny
part
about
this
is
when
he
started
writing
code.
It
was
part,
as
I
said,
it
was
part
of
a
department
of
energy
grant,
so
he
was
working
at
los
alamos
national
laboratory,
so
he
was
actually
going
to
a
secure
facility.
B
He
had
a
little
radiation
badge
he
had
to
like
badge
in
and
security
guys
with
m16s
all
the
crazy
stuff
and
he
would
go
into
the
base
and
he
would
sit
down
in
his
little
cubicle
and
he
would
write
open
source
code
and
check
it
in
over
ssh
back
to
the
subversion
server
at
the
university.
So
it
was
a
little
strange
and
then
in
2006
is
when
he
finally
kind
of
pushed
out
into
the
world.
B
This
is
where
I
usually
take
a
minute
to
talk
about
the
licensing
for
those
of
you
that
aren't
familiar
with
the
license
of
ceph.
It's
lgb
version,
2
lgpl
version
2,
and
it's
actually
got
a
unique
twist
that
sage
threw
in
there.
It's.
We
have
a
fraud
deliberately
fragmented
copyright,
which
means
that
anybody
that
commits
code
to
sef
owns
that
contribution
forever
and
whatever.
B
So
that
means
that
if
anyone,
a
giant
megacorp
that
just
bought
seth,
for
instance,
wanted
to
change
the
licensing
on
it,
they
would
actually
have
to
go
in
and
get
every
single
contributor
that
had
ever
contributed
to
ceph
to
sign
off
and
say,
yeah
sure
you
can
make
that
proprietary
and
charge
me
money
for
it.
That's
cool,
so
unlikely
that
that
would
ever
happen.
B
B
B
B
B
B
B
So
it's
been
kind
of
a
long
road
coming,
but
it's
been
amazing,
you
know,
and
ever
since
kind
of
the
open
stack
point,
it's
just
been
an
inflection
point
and
kind
of
the
hockey
stick
after
that.
So
you
know
we
also
saw
things
like
the
cloud
stack
integration
and
the
zen
stuff.
So
I
mean
there's
lots
of
stuff,
that's
coming
down
the
pipe,
and
then
we
launched
our
first
commercial
product
in
october
of
2013..
B
This
was
kind
of
the
first
time
we
really
thought
about.
Oh
okay,
we
need
to
be
a
real
company,
we're
looking
for
investors
and
they
said
well,
you
need
something
that
you
actually
hand
to
somebody.
You
can't
just
be
a
support
company,
no
one's
going
to
give
you
money
for
that.
And
so
then
we
kind
of
packaged
up
this
whole
thing
and
said
hey.
B
So
so
what
is
ceph?
The
the
real
thing
that
we
started
to
look
at
here
was
kind
of
a
lot
of
what
drove
sage
to
build.
Seth
was
I
mean
he
was
a
dream
host.
He
was
one
of
the
founders
of
dream
house,
so
he
was
tired
of
paying
lots
and
lots
and
lots
of
money
for
storage,
and
you
know
every
time
he'd
play
a
bazillion
dollars
and
they
would
give
him
this
giant
black
box
and
say
it's
storage.
B
The
whole
kind
of
purpose
of
this
was
to
get
away
from
that
moving
from
instead
of
proprietary
hardware
to
you
know
the
the
commodity
hardware,
the
any
day
everyday
type
of
hardware
sitting
with
open
source
software
instead
of
proprietary
software
and
then,
if
you
want
the
enterprise
products
and
services
that
sit
on
top,
not
hey,
we're
going
to
put
the
thumb
screws
to
you
and
lock
you
in
so
that
was
kind
of
the
the
difference
between
the
two
and
and
where
we
were
headed.
So
now,
how
do
we
do
this?
B
I
guess
the
first
thing
is.
I
should
talk
about
some
of
the
principles
that
sage
had
that
were
kind
of
driving
principles
behind
ceph.
You
have
things
like
no
single
point
of
failure.
I
mean
it
may
sound
like
a
no-brainer,
but
sage
really
took
it
to
the
next
level.
I
mean
even
all
the
way
down
to
this
ffs
portion,
which
we'll
talk
about
in
a
little
bit.
B
He
wanted
to
say
even
like
the
metadata
servers
needed
to
be
able
to
scale
horizontally
and
be
no
single
point
of
failure
and
like
even
if
you
look
at
a
lot
of
open
source
options
out
there,
like
hadoop
like
for
a
long
time.
I
know
you've
got
some
hadoop
heads
in
here
that
single
name,
node
architecture.
B
Tell
me
you
didn't,
have
some
pain
right,
yeah,
so
sage
wanted
to
try
and
get
away
from
that
and
kind
of
just
rethink
and
flip
the
whole
thing
on
its
head,
and
so
we
started
with
an
object
store
which
we
can
get
into
here.
So
if
you
look
at
this,
this
is
kind
of
a
general
overview
of
what
ceph
comprised
or
what
comprises
ceph
all
the
pieces
right.
B
B
I
was
kind
of
new
to
the
storage
world
when
I
started
playing
with
ceph
and
hacking
around
with
sage
and
the
guys
and
and
when
I
started
talking
to
storage
admins,
it
really
surprised
me
because
they
would
tell
me
things
like
well:
every
n
terabytes
equals
a
new
head
count,
it's
a
new
storage
admin
because
we
grew
and
had
more
storage.
It
just
kind
of
blew
me
away
like
really.
You
can't
automate
that
you
can't
do
some
magic,
but
sage
did
he
did
some
magic,
and
so
dreamhost
was
our
first.
B
It
was
our
first
large
customer.
First
production
deployment,
they
built
dream
objects.
It
was
it's
now
an
amazon,
s3
competitor,
the
the
guy
that
ran
that,
and
it
was
a
five
petabyte
cluster.
They
now
have
two
of
them
and
they're
looking
to
build
the
third,
potentially
the
gut
five
petabytes.
So
it
was
two
five
petabyte
clusters,
one
for
dream
objects
and
one
that's
coming
for
dream
compute.
It
was
actually
run
by
one
guy
part-time.
B
So
I
tell
that
to
store
these
storage
admins
that
were
telling
me
these
these
stories
about
how
they
had
to
have
new
headcount
for
new
storage,
and
they
just
kind
of
look
at
me
and
they
don't
really
believe
me,
but
it
is
pretty
amazing,
all
the
stuff
that
it
can
do
so
on
top
of
rados.
I
guess
we
can
get
into
that.
Rados.
It's
a
reliable
autonomous
distributed,
object,
store
rados.
B
So
that's
where
everything
goes
right,
it's
what
everything
is
built
on
top
of-
and
it
looks
like
this
and
and
what
this
means
is
in
ceph.
You've
got
a
bunch
of
disks
right,
and
this
is
one
of
the
acronyms
that
I
really
loved
learning
when
I
first
started
doing
storage
jbod
right
just
a
bunch
of
disks,
and
so
that's
what
you've
got.
B
You've
got
a
whole
bunch
of
machines
with
just
a
bunch
of
disks,
and
that
is
what
comprises
your
your
rados
cluster
and
so
what
you
do
to
make
it
a
raido's
cluster
is
for
a
long
time.
When
we
first
started
things,
we
had
our
own
file
system
that
would
sit
on
top
of
the
disks.
It
was
called
eboffs
we
spent
early
on.
They
spent
a
lot
of
time
working
on
this
and
they
realized
that
we're
reinventing
the
wheel.
B
This
is
stupid,
we've
got
the
guys
are
doing
ext4
and
xfs,
and
ultimately
we
want
to
call
the
butter
fs
place
home,
but
you
know
it's
not
production
ready
yet
and
well,
it's
been
not
production
ready.
Yet
for
like
what
last
10
years
but
it'll
get
there
someday.
We
hope.
So
on
top
of
that,
we
would
put
a
file
system
some
linux
file
system.
Most
of
our
production
customers
are
running
on
xfs.
B
On
top
of
that
file
system
is
a
software
daemon,
it's
your
object,
storage,
daemon,
and
it's
just
a
piece
of
software
that
you
run.
Typically,
we
have
one
diamond
per
physical
hardware
disk,
and
then
these
are
the
things
that
are
doing
the
peering
and
putting
everything
together
for
the
clustering
they're,
the
ones
that
handle
all
the
rebalancing
and
they're
the
things
that
are
actually
in
your
data
path.
B
The
only
thing
that
are
actually
in
your
data
path-
and
so
you
have
a
whole
bunch
of
these
osds
in
a
cluster
and
then
you'll
notice,
kind
of
any
application
that
talks
to
a
rados
cluster
doesn't
care
that
it's
a
whole
bunch
of
things
it
just
treats
it
as
a
single
logical
object
or
a
single
logical
unit.
I
should
say
now
what
you
did
notice:
that
there
were
a
few
other
types
in
there
that
weren't
the
osds.
These
are
your
monitors.
B
B
Usually
it's
just
a
small
odd
number
of
these
monitors
that
will
kind
of
be
the
glue
for
your
cluster.
So
what
these
guys
do
is
they
actually
have
a
paxos
algorithm
that
they
use
to
make
decisions
right?
If
you
have
because,
as
I
said,
all
of
the
osds
are
peering
they're
talking
to
each
other,
so
that
means
they
can
also
report
when
they
think
that
hey
that
jerk
osd
42
he
is
down-
and
you
know
somebody
on
the
other
side
of
the
cluster-
can
say
no.
B
B
So
there's
a
number
of
different
ways
that
you
can
store
things
in
a
cluster
and
most
well.
There's,
I
guess,
there's
three
main
ways
you
can
think
about
it
right,
there's
the
it's
that
single
name,
node
problem
I
talked
about
you
want
to
put
something
in
a
cluster.
You're
gonna
go
right
down.
Okay,
I
put
it
at
this
place
all
right
that
one
turns
out.
There's
scaling
issues
for
that
who
do
the
other
way
is
that
you
can
turn
it
into
what
I
call
the
phone
book
approach.
B
Right
kind
of
a
logical
distribution
of
data
across
your
cluster.
The
problem
you
run
into
with
this
is
that
if
you
have
an
outage
or
you
want
to
shuffle
your
data,
there's
a
lot
of
things
that
need
to
move
potentially
as
you
shuffle
your
data,
so
this
becomes
very
network
intensive.
I
guess,
among
other
things
now
this
is
where
crush
comes
in
crush
is
the
magical
algorithm
that
makes
ceph
the
hotness
it's
controlled
replication
under
scalable
hashing.
B
It's
awesome,
I
am,
I'm
constantly
surprised
at
how
much
it
can
do
and
how
simple
it
is
sage
for
a
long
long
time
has
been
just
urging
people
like,
let's
do
more
stuff
with
crush
crush,
is
cool
because
crush.
Is
the
big
thesis
paper
that
he
wrote
and
published
when
he
did
his
doctorate
and
he'd
really
like
to
do
more
with
it,
but
he
just
doesn't
have
the
time,
but
the
way
that
crush
works
is
it
is
a
it's
like.
B
I
said
it's
an
algorithm,
and
so
what
happens
is
when
you
push
data
in
your
cluster?
You
split
it
up
into
logical
chunks.
We
call
them
placement
groups
they're
the
buckets
where
you
drop
your
data
and
what
will
happen
is
it
will
actually
take
a
few
things.
A
few
pieces
of
information
so
you've
got
your
cluster
state
right.
B
B
So
you
get
to
say:
I
have
n
machines
in
y
rows
in
x,
data
x,
power
circuits
or
whatever,
like
you,
can
kind
of
create
your
own
failure
domains,
and
then
you
get
to
make
rules
for
how
your
data
is
distributed
across
these
physical
hosts.
You
can
say
I
want
three
copies
of
my
data,
but
no
two
copies
can
live
in
the
same
row
in
my
data
center.
So
now
you've
all
of
a
sudden
started,
creating
the
appropriate
amounts
of
redundancy
and
what
crush
does
is
it
takes
that
status
of
your
cluster?
B
B
So
that's
what
crush
does
now.
What
this
means
is
that
also
we
no
longer
have
to
go
through
a
bottleneck
to
get
into
the
cluster.
So
now,
each
of
your
clients,
all
it
has
to
do-
is
a
very
quick
calculation,
and
it
knows
where
I
have
to
put
the
data
and
then,
when
it
goes
back,
it
knows
well.
This
is
where
it
is
based
on
the
calculation
and
that
calculation,
because
it
involves
the
cluster
map,
the
status,
who's
up
and
who's
down.
B
If
it
changes
well,
then
the
output
will
obviously
be
the
new
location
of
the
data
and
the
osds
have
already
handled
the
movement
of
the
data
and
rebalanced
everything.
So
it's
where
you
would
expect
so
crush,
really
cool.
I
guess
is
what
you
could
take
away
from
that
so
sitting
on
top
of
rados,
which
is
this
magic
layer
is
liberatos.
B
B
So
I
have
an
application
and
I
want
to
put
stuff
in
ceph,
but
I
don't
want
to
use
your
interfaces
because
I
want
to
do
it
at
the
lowest
level,
because
it's
got
a
really,
really
fast
messaging
protocol
that
you
can
use
to
talk
to
the
cluster
instead
of
going
with
the
overhead
of,
like
you,
know
the
restful
overhead
or
dealing
with
a
block
device
or
things
something
like
that,
so
you
can
actually
use
liberatos
plug
your
application
directly
into
the
object.
Cluster
and
I've
seen
some
really
cool
experiments
with
this.
B
But
most
people
choose
to
do
one
of
the
things
on
top
of
it
because
yeah,
it's
that
socket
connection
for
libretos
yeah,
touched
on
that
already.
So
on
top
of
liberatos,
which
we
used,
because
that
was
the
easiest
way
to
get
into
the
cluster.
We
have
three
main
interfaces
that
we
use
to
expose
your
object.
Cluster
we've
got
rados
gateway.
B
This
is
our
s3
and
swift,
compatible
restful
gateway
that
you
can
use
for
object.
Storage,
we've
also
got
the
rido's
block
device
rbd,
it's
a
thinly,
provisioned
block
device
that
will
allow
you
to
you,
know,
store
machine
images
or
boot,
vms
or
whatever.
You
know,
a
standard
block
device
and
cfs,
which
is
our
distributed
file
system
which
so,
if
I
had
to
refer
to
these
three
things
I
would
say,
rados
gateway
and
the
raido's
block
device
are
really
awesome.
B
We
had
to
choose
to
focus
on
something
and
turns
out
cloud
scenario,
mostly
object
and
block.
So
that's
where
we
focused
all
of
our
resources.
Thankfully,
the
community
has
definitely
kept
pace
with
us
as
far
as
ffs.
So
it's
a
lot
further
along
than
us,
just
ignoring
it.
I
was
actually
telling
one
of
these
guys
earlier
that
there's
a
one
guy
in
china
who
really
loves
ffs
loved
it
so
much
and
he
over
the
last
six
to
eight
months,
has
submitted
like
800
patches
to
cfs
the
guy's.
B
A
machine
he's
a
monster,
but
we
love
him
and
that's
why
ceffs
is
probably
a
lot
shorter
roadmap
than
it
would
have
otherwise
been
now
that
we're
getting
back
to
making
it
also
awesome,
so
the
gateway,
the
gateway.
It's
again
that
same
horizontally,
scalable
mantra
that
sage
started
out
with
you
can
have
as
many
gateways
you
could
spin
up
six
eight
ten
hundred
of
these
raido's
gateway
machines,
put
them
behind
a
normal
load,
balancer
that
you
would
expect
they're
they're,
pretty
rough
and
tumble.
B
B
It's
really
cool,
so
yeah
we
we
have.
It
supports
all
of
the
normal,
like
buckets
and
accounting
things
that
you
might
expect
it's
a
little
bit
closer
to
the
s3
semantics
than
the
swift
ones.
We
had
to
make
some
decisions.
We
started
with
s3
first,
so
something
like
the
the
global
name
spacing,
I
think,
that's
more
in
s3,
then
swift
has
some
like
weird
sub
name
spacing
stuff.
B
B
Oh,
it's
also
plugged
into
keystone
for
the
openstack
heads
in
the
audience,
so
all
of
the
normal
accounting
and
everything
that
goes
on
with
that
authentication
all
that
magic.
It
plugs
right
into
that
too.
So
it
is
a
drop
in
replacement
for
swift
and
openstack
setting
all
right
the
rails
block
device
it
like.
I
said
it's
a
it's
a
thinly
provision
block
device,
so
it
takes.
B
You
know
dev
rbd,
0
whatever,
and
it
takes
a
normal
block
device
and
it
splits
it
up
into
a
bunch
of
chunks,
throws
it
out
over
the
cluster
and
then
takes
all
of
those
things
via
lib
rbd,
which
sits
on
top
of
liberatos
and
aggregates
them
all
together
exposes
them
to
a
hypervisor
which
then
can
do
all
the
magic
that
you
would
expect.
Now.
B
What
makes
this
really
cool
is
because
you've
taken
your
block
device
and
you've
split
it
up
into
multiple
physical
hosts
is
that
a
lot
of
the
workload
that
goes
into
this
block
device
can
be
parallelized.
Now,
what
that
means
is
your
block
device
can
be
arbitrarily
large
or
arbitrarily
busy
and
rbd
doesn't
care.
B
So
it
you
get
a
lot
of
really
cool
benefits
from
rbd
or
by
using
rbd
for
your
for
your
block
devices
and,
as
I
said
this
for
the
openstack
folks-
and
I
think
I've
got
a
slide
later
on
that
we'll
get
into
this.
But
this
is
what
does
all
the
magic
behind
things
like
cinder
and
glance
so
rbd
plugs
in
there
too.
B
Some
other
cool
benefits
that
you
have
when
you
start
decoupling.
The
storage
infrastructure
from
your
vm
infrastructure
is
that
you
can
do
cool
things
like
live
migration
across
hypervisors
right.
You
can
take
one
machine
and
you
can
move
it
over
here
to
this
other
place
and
because
it's
not
a
shared
storage
back
end,
you
don't
have
any
downtime
interruption,
the
user
doesn't
care.
B
The
other
cool
thing
is
that
with
our
openstack
integration,
you
can
also
do
things
like
copy
and
write
cloning.
So
what
this
means
is
especially
for
large
infrastructure
deployments.
If
I
have
a
golden
image
like,
I
have
an
ubuntu
image
that
I
want
to
spin
up
in
the
cloud
and
I'm
in
a
large
infrastructure.
B
Maybe
we're
going
to
do
a
big
deployment
of
something
right.
We
want
to
play
with
our
new
magical
juju
stuff,
so
we
want
to
provision
100
machines
to
play
with
you
can
say
from
this
magical
golden
machine
image,
spin
up,
100
machines
or
a
thousand
machines,
and
rather
than
having
to
copy
it
that
many
times
it
just
starts,
booting
all
100
of
them
or
all
1000
of
them
just
immediately
start
booting
up
off
of
that
golden
image.
B
Now
the
copy
on
right,
clone
part,
is
now
we
have
100
clones
when
I
write
to
one
of
them
all
it's
going
to
do
is
write
the
delta,
the
change
to
the
new
instance.
Everything
else
is
just
going
to
read
through
straight
to
the
golden
image.
So
you
get
a
lot
of
really
cool
benefits
by
having
this
kind
of
short
or
shared
storage
back
end.
C
B
We
also
have
we
have
a
kernel,
client
and
a
user
space
client,
so
krbd
and
is
the
other.
Is
the
current
client
all
right?
So
the
last
piece,
the
nearly
awesome
cfs
cefs,
is
a
distributed
file
system
that
you
can
use
in
the
back
end.
Now
it
has
some
really
cool
features
and
it
was
designed
in
a
really
cool
way.
B
This
is
not
the
data,
it's
the
metadata
and
actually
that
metadata
server
in
turn
stores
its
metadata
in
the
cluster
itself,
and
actually
do
we
have
yes
all
right.
So
metadata
is
just
that.
It's
the
piece
that
you
don't
need
unless
you're
using
cefs,
if
you're
using
cefs,
that's
the
third
type
of
machine
in
your
cluster,
you
just
need
a
few
of
them
or
maybe
not.
If
you've
got
a
really
busy
one.
You
need
a
lot
of
them,
but
what
it'll
allow
you
to
do
is
it
has
a
thing
called
dynamic,
subtree
partitioning.
B
So
what
this
allows
you
to
do
is
I've
got
a
big
directory
tree
and
based
on
the
heat
of
any
portion
of
it.
Each
of
these
guys
can
start
sharing
the
load,
and
I
can
say
well,
half
of
it
goes
to
this
guy
over
here,
because
that's
really
not
busy,
and
these
three
guys
will
split
the
remaining
tree
amongst
them
and
then
accept
all
the
way
down
to
a
single
file.
B
Sorry,
I'm
going
to
work
cold,
so
yeah,
that's
the
metadata
server
again
only
required
if
you're
using
cfs
yeah
hit
everything
cool
all
right.
I
know
we
got
some
openstack
fans.
This
is
how
seph
plugs
into
openstack.
So
this
is
cfs
notwithstanding.
You've
got
do
I
have
the
oh,
no,
all
right,
so
you've
got
rados
gateway
that
plugs
into
keystone
and
swift
and
cinder
glance,
and
I
guess
no
as
well.
If
you
want
to
talk
about
that
all
served
by
rbd
now
I
have
seen
some
really
cool
instances
where
people
have
decided.
B
You
know
what
I
know
you
said.
Cfs
is
nearly
awesome,
but
we
know
that
our
use
case
will
be
fine,
so
don't
worry
about
it
and
they've
gone
ahead
and
they've
used
ffs
and
they've
done
some
really
cool
stuff.
Like
I
know,
one
guy
was
working
on
the
fact
that
his
images
for
his
hypervisors
were
stored
in
cfs,
so
they
would
be
replicated
out
to
all
of
his
hypervisor
machines.
So
every
time
he
spun
up
a
machine,
it
was
locally,
it
was.
It
was
local
to
that
hypervisor
and
he
just
started
spawning
it's
funny.
B
B
I
guess,
take
a
brief
pause
here.
I
think
I'm
still
good
on
time
any
any
major
questions
on
the
are
in
the
architecture
portion
infrastructure
stuff.
I
don't
want
to
like
go
on
about
the
community
stuff
and
like
have
it
lost
yeah,
okay,
so
the
question
is:
is
there
support
for
other
operating
systems
beyond
linux?
I
know
there's
some
bsd
folks
that
were
playing
with
it.
B
We
aren't
really
focused
there
right
now,
although
we
are,
we
have
several
projects
like
there's
a
hyper-v
project.
We
want
to
start
getting
integration
into
the
microsoft
world
because
a
lot
of
people
have
been
leaning
on
us
to
do
it,
but
right
now,
yeah,
it's
mostly
linux
centric
for
the
most
part.
B
So
but
more
is
coming
security
yeah,
so
ceph
authentication,
insecurity
in
ceph
is
actually
handled
by
something
called
sefax,
it's
kerberos
like,
but
it
was
home
rolled.
So
we
did
our
own
thing
because
there
were
a
lot
of
other
solutions
that
had
to
be
accounted
for
in
a
distributed.
Setting
that
you
know
kerberos
wouldn't
necessarily
handle,
but
it's
kerberos
like
so
suffix
is
what
you're
looking
for
for
security
you've
pretty
much
exhausted
my
knowledge
about
suffix
now.
D
So
you
talked
about
multiple
vms,
launching
off
of
the
same
image
and
copy
on
write
being
awesome.
What
is
the
associated
latency
with
that?
I
could
see
a
lot
of
problems
with
everyone
writing
their
own
deltas,
and
then
you
have
to
read
the
delta
and
the
original
file
and
what
happens
when
you
do
copy
on
write
for
a
three
gig
file
and
like
how
does
that
work?.
B
Now,
when
I
say
original
image,
keep
in
mind
that
if
you're
at
3x
replication,
which
is
the
default,
it
may
not
all
go
to
the
same
image
either
it
will
go
to
one
of
those
copies
of
the
placement
group
that
you're
looking
for,
because
keep
in
mind
that
image
also
isn't
stored
as
a
single
thing
or
your
three
gig
file
isn't
stored
as
a
single
thing.
So
it's
going
to
be
looking
for
like
a
four
meg
chunk
in
the
radius
cluster,
so
those
read
throughs
are
going
to
go
through
at
the
placement
group
level.
B
B
So
our
one
of
our
community
guys
vito
den
holler,
he's
actually
the
guy
he
works
for
a
company
called
42
on
the
netherlands,
he's
the
guy
who
wrote
the
integration.
He
also
wrote
the
java
bindings
for
ceph
cloud
stack.
Now,
I
think,
can
do
pretty
much
everything
that
openstack
can.
So
you
can
do
all
of
your
the
the
object,
storage
stuff.
You
can
do
the
block
storage,
and
now
we
have
both
there's
two
rbd
formats
that
you
can
use.
E
How
does
sef
handle
if
you
have
like
a
heterogeneous
network
where
you
have
different
network
speeds
and
some
may
be
unpredictable
on
some
nodes
specifically,
if
you
have
a
blocked
device
and
you
get
maybe
to
the
middle
of
it,
and
you
have
you
know
some
network
latency
come
up
like?
Does
it
start
ballooning
memory
in
your
kernel
as
it
has
to
fulfill
that
request.
B
I
won't
answer
the
very
last
part
of
that
question,
because
I'm
not
the
guy
to
answer
that
question.
I
will,
however,
say
that
the
way
that
we
typically
suggest
people
to
engineer
their
their
architecture
for
steph
it's
very
redundant
simply
because
we
are
very
big
on
the
no
single
point
of
failure
that
includes
network
override.
It
was
slow,
yeah,
okay,
so
the
other
part
of
what
I
was
going
to
say.
B
There
are
so
many
levers
in
cef
that
you
can
pull
to
adjust
things
like
you
can
adjust
your
network
timeout
for
the
latency
on
the
lsds.
You
know
a
lot
of
people
would
see,
so
I
can
think
of
one
good
example.
Where
a
customer
had
decided,
they
wanted
to
go
wand
scale
with
ceph,
which
really
wasn't
recommended,
given
the
latency
between
the
sites,
and
they
saw
a
lot
of
problems
with
flapping
osds.
B
So
there
are
a
lot
of
configurable
options
within
ceph,
a
nauseatingly
large
amount
of
configurables
and
stuff
when
you
really
get
down
to
it,
which
is
I
mean
ultimately
why
ink
tank
came
to
be
right
and
why
ink
tank
was
ultimately
relatively
successful,
which
was
because,
when
you
get
down
to
tuning
a
ceph
cluster
sure
when
you
just
plug
it
in
and
use
it
it's
great.
But
if
you
want
to
make
it
super
great,
that's
a
lot
of
work
on
the
front
end
on
just
knowing
how
all
these
levers
work.
B
B
I
don't
know
chris,
you
can
probably
say
it,
but,
and
I
can
seth
yeah
just
how
many
customers
do
we
have
before
the
acquisition?
Oh
yeah.
Well
as
far
as
people,
just
using
it,
you
know
commercial
customers,
hundreds,
I
guess,
but
I
mean
we
had
a
lot
of
really
nice
flagship
customers
that
we,
how
who
is
who
is
administering
ceph
clusters,
yeah
yeah,
so
hundreds
of
customers
of
ours,
but
there
were
also
huge
numbers
of
people
in
the
community,
utilizing
ceph
for
various
things
that
we
didn't
even
know
about
some.
B
We
still
don't
know
about.
Actually
the
nudt
guys
national
university
of
defense
technology
in
china,
the
guys
with
the
biggest
supercomputer
in
the
world.
We
think
we
didn't
know
for
a
long
time
that
they
were
using
ceph.
We
just
saw
these
guys
from
china
kept
like
submitting
patches
or
writing
on
the
mailing
list.
B
And
finally,
they
came
to
a
seph
developer
summit
online
and
we
realized
that,
oh,
my
gosh,
it's
a
bunch
of
guys
from
the
nudt
in
china
and
they
were
all
like
collectively
working
together
and
they
have
a
big
ceph,
cluster
and
they're.
Actually
taking
you
know
at
some
points
in
time.
Ceph
is
serving
that
giant,
super
computer,
so
yeah.
That
was
amazing
and
we
didn't
know
about
it
for
a
long
time.
Ceph
actually
scales
down
really
well
and
that's
that's
all
because
of
you
know
kind
of
that
again.
B
F
B
You
know
it's
funny.
You
say
that
ceph
was
original,
like
the
mission
statement
behind
creating
ceph
was
to
create
a
distributed
file
system.
That
would
be
a
better
luster.
That's
what
it
was
originally
was,
and
we've
kind
of
gone
on
this
tangent
around
the
outside
of
object
and
block,
and
now
we're
just
getting
back
to
the
file
system,
but
remember
I
said
it
was
written
originally,
as
a
department
of
energy
grant
right.
B
The
three
national
u.s
laboratories
are
all
still
using
ceph
and
they're
using
ceffs,
and
we
still
get
patches
and
stuff
from
them.
So
in
their
use
case,
apparently
it
does
really
well
now.
The
broader
hpc
use
case
in
general
we're
waiting
for
cfs
to
be
awesome
right,
because
then
that
is
what's
going
to
solve
a
lot
of
the
specific
use
cases
for
hpc.
B
B
Yes,
they
were
one
of
the
first
big
customers
that
that
inktank
had
yeah
tim
bell,
dan
vanderster,
those
guys
they
had
a
really
small
pet
project
that
they
started
stuff
with
to
prove
it
out,
and
then
it
turns
out
that
random
other
groups
at
cern
started
going.
Hey,
that's
that's
kind
of
cool.
Can
we
can
we
play
with
that?
Like
yeah?
B
Give
us
some
money,
we'll
add
a
little
bit
here,
and
it
can
be
your
you
know,
and
and
so
it
just
kind
of
grew
and
grew
and
grew,
and
now
again,
with
the
cfs
part
they're
looking
at
potentially
trying
to
get
accelerator
data
on
it
late
this
year
or
early
next.
But
yes,
cern
is
a
big
big
user
of
ceph.
F
E
B
Card-
and
I
will
pass
along
the
guy
you
want
to
talk
to
is-
is
mark
nelson
okay,
he
is,
he
is
the
guy
to
talk
to
and
he's.
Actually,
if
you
come
to
the
ceph
channel,
irc
he's
nhm,
okay,
mark
nelson
he's
the
guy
he's
the
one
who
does
all
of
our
performance,
crazy
stuff
or
leads
it
anyway
and
he's
a
big
hpc
head.
He
came
from
that
side
of
the
world.
Thank.
H
B
Funny
you
say
that
so
all
right,
so
this
will
be
in
my
community
thing
later,
but
I
got
to
plug
it
so
seth
day,
boston
is
next
week.
If
anyone
wants
to
go
like
talk
to
me
I'll
hook
you
up
with
like
a
75
discount
code
or
something
like
crazy,
but
one
of
the
talks
next
week
at
cef
day,
boston
on
tuesday.
I
think
the
10th
is,
if
you
know,
jeff
darcy
from
the
gloucester
world:
okay,
jeff
darcy.
He
when,
when
we
came
on
board
with
red
hat,
he
said.
B
B
It's
a
big
question
mark
and
nobody
really
knows.
Yeah,
what's
that
I
am
not
the
guy
to
answer
that
question,
but
I
mean
so
ultimately
keep
in
mind.
The
architecture
right
ceph
at
the
at
the
lowest
level
is
an
object.
Store
gluster
at
the
lowest
level
is
files
right,
so
I
mean
there's
you
can
kind
of
infer
some
differences
there,
maybe
probably
better
than
I
can
but
yeah.
B
You
can't
control
the
performance,
hits
for
certain
things
and
it's
pretty
wild.
Some
of
the
like
a
lot
of
my
testing.
When
I
did
early
experimentation
like
when
I
did
juju
deployment
of
ceph
when
juju
first
came
out,
I
did
it
to
amazon
because
they'd
already
written
all
the
magic
for
juju
just
to
like
plug
stuff
into
amazon,
and
it
was
like.
Oh,
this
is
easy.
This
is
great.
Then
I
started
using
that
stuff
cluster
and
all
of
a
sudden.
B
B
B
It
was
the
first
monitoring
tool
to
kind
of
go
out
there
and
when,
when
it
was
a
commercial
product,
calamari
was
actually
closed
source
because
our
investors
all
said,
we
won't
give
you
money.
Unless
you
have
something
you
hand
to
the
customer
and
it's
stupid,
but
now
because
red
hat
bought
us,
it
was
probably
my
favorite
part
of
the
announcement.
B
Last
friday
we
pushed
calamari
out
the
door,
it's
now
open
source,
so
there
is
a
monitoring
tool
for
ceph
and
more
than
that,
it's
actually
a
management
platform.
So
longer
term
you'll
be
able
to
do
more
than
just
monitor
your
cluster
you'll.
Be
able
to
do
things
like
deploy
new
nodes
and
manage
your
pools
and
manage
authentication
all
that
stuff
from
calamari
and
it's
open
source
not
yet,
but
soon,
not
yet
there.
B
B
So
it's
not
really
wan
scale.
You
know
it's
like
hey.
I've
got
10
million
millisecond
pings
over
400
miles
or
something
yeah,
not
not
real
world
for
anyone
else.
But
yes,
soon,
there's
there's
actually
a
number
of
initiatives
in
ceph
in
the
ceph
community
that
want
to
see
ceph
at
lanscale
and,
ultimately,
sage
wants
to
get
back
to
that.
He
wants
to
make
a
rados
cluster
that
has
provisions
for
handling
land
scale
and
one
of
mimos.
B
So
it's
a
r
d
research
wing
for
the
malaysian
government
they're
working
on
a
project
potentially
with
the
someone
who
has
a
big
dark,
fiber
network
and
they
want
to
start
working
on
making
ceph
landscape
so
they're,
probably
the
ones
that
are
at
the
forefront
of
that
effort.
But
not
here
yet
no,
no
there
I'd!
B
Well,
I
shouldn't
say:
no,
not
that
I
know
of
there
may
be
that's
one
of
the
pieces
that
will
come
as
we
start
getting
because
as
we
expand
and
you're
gonna
have
land
scale,
you're
gonna
have
to
have
things
like
availability
zones
and
you'll
have
to
be
able
to
start
saying
things
like
well.
I
want
three
replicas
and
I
need
one
in
each
of
my
three
availability
zones
or
whatever,
but
we
don't
have
that
yet,
really
not
not
in
any
meaningful
context.
B
B
I
just
gave
an
interview
with
this
the
other
day,
and
I
realized
I
don't
have
a
good
elevator
pitch
for
erasure
coding,
erasure
coding
is
basically
instead
of
having
three
physical
copies
of
your
data.
You
have
1.4
copies
of
your
data
plus
parity
and
what
this
means
is.
It
takes
a
lot
less
space
to
store
your
data,
but
it
also
takes
a
lot
longer
to
recover.
If
you
have
a
failure,
it's
what
network
right
now.
Okay,
that's
good!
I'm
going
to
use
that,
and
so
what
we
we
started
introducing
is.
B
If
we
have
the
idea
of
kind
of
tiering
your
storage
within
ceph,
you
can
say
by
on
a
pool
per
pool
basis.
You
can
say
all
right.
I
have
my
pool
cold,
so
all
my
cold
storage,
the
stuff
that
I
don't
care
about,
a
whole
lot.
You
know
my
taxes,
I
get
it
out
once
a
year
right
that
stuff
gets
pushed
to
a
cold
layer
which
is
erasure
coated,
so
it
takes
a
lot
less
space.
B
But
you
know
we
it's
not
critical
data.
You
know
if
we
have
an
outage,
it's
okay,
that
it
takes
10
hours
to
recover
or
whatever
then
there's
your
normal
storage,
which
can
be,
unlike
you
know,
spinning
rust
whatever
and
then
maybe
you
have
some
really
michigan
critical
data
and
you
say
we
get
in
and
out
writes,
reads
all
the
time
and
that's
on
our
ssds,
that's
tiering!
B
The
idea
of
you
know
a
bloom
filter
again,
like
the
heat
of
a
data,
will
push
it
up
or
down
the
tier
that
you
go
through
so,
but
to
answer
your
question
in
terms
of
data
resilience
by
default,
it's
3x
replication
and
you
get
to
decide
that
you
get
to
decide
how
many
copies
where
those
copies
live.
That's
what
the
magic
of
your
crush
map,
and
now
you
also
have
the
tool
of
erasure
coding
to
say
not
as
mission
critical,
don't
take
three
times
the
space.
B
B
I
don't
have
that
on
here,
but
it's
done
on
a
placement
group,
so
it's
not
done
on
a
whole
file
because,
like
I
said
your
files
ultimately
get
split
down
into
and
again
it's
configurable,
but
I
think
defaults
like
a
four
meg
chunk,
and
so
the
osds
are
constantly
peering
with
each
other
and
they'll
know
it.
You
know
machine
number,
34
had
the
copy
of
placement
group
number
128..
B
Now
that
machine
is
gone
and
the
other
two
machines
in
your
cluster,
ostensibly
if
you
have
3x
replication,
know
that
that
guy's
gone
and
so
using
crush.
They
say.
Okay,
now
the
new
calculation
says
that
data
piece
that
data
chunk
needs
to
live
on
machine
number
47.,
and
so
they
will
write
a
copy
behind
on
the
network
on
the
cluster
network.
We'll
replicate
that
to
bring
it
back
up
to
the
3x
replication,
so
it
handles
all
of
that
kind
of
self-healing
stuff
on
its
own.
B
I
do
know
the
answer
is
yes,
we
have
that
hit
me
up
afterwards
or
come
see
me
on
irc
or
something
I'll
plug
you
into
the
guys
who
can
intelligently
answer
that
question
turns
out
really
really
well.
I
know
I'm
just
full
of
just
vigor
and
happiness,
but
there's
actually
been
a
couple
of
customers
that
use
ceph
for
their
logs
dash
and
their
log
data
is
millions
of
really
really
small
files
and
they
got
they.
They
did
some
tuning,
they
worked
with
mark
and
they
tuned
their
cluster.
B
B
B
The
last
twenty
percent
is
really
really
hard,
because
there's
lots
of
moving
pieces
in
a
ceph
cluster
when
you're
talking
about
performance
tuning,
so
performance
tuning
is
actually
one
of
the
things
that
we've
had
to
struggle
a
lot
with,
simply
because
there's
no
easy
way
to
say
hey
if
you
want
good,
I
o
on
big
block
devices.
Do
this
because
it's
never
that
easy,
so
performance
tuning
is
something
that's
definitely
a
challenge.
B
There
isn't
one.
No,
we
haven't
seen
it
yet.
We've
we've
seen
thousands
to
tens
of
thousands
of
osds
in
a
cluster
and
that's
what
it
was
designed
to
do.
It
was
designed
to
scale
to
like
the
exabyte
level
and
beyond,
with
tens
of
thousands
of
of
nodes
in
in
yes.
That
is
a
good
point.
It
was
built
at
data
center
scale.
It
was
originally
built
to
be
a
better
luster.
It
was
built
for
data
center
scale.
We
want
to
get
beyond
that,
but
we'll
see
yeah
you,
okay,
so
running
many.
B
Many
metadata
servers
right
now
is
fine,
because,
ultimately,
the
architecture
by
default,
unless
you
go
in
and
mess
with
stuff,
is
only
it's
it's
hot
standby.
You
know
it's
it's
one
metadata
server,
the
the
dynamic
subtree
partitioning
is
the
part.
That's
nearly
awesome
or
one
of
the
parts
is
nearly
awesome.
But
to
answer
your
question,
actually
you
don't
really
want
to
do
that,
and
I
know
it's
hard
to
get
into
not
having
a
machine
image
you
can
say,
deploy
and
have
a
thousand
of
those
to
spin
up
and
go.
B
But
if
you
had
a
thousand
monitors,
your
cluster
would
be
incredibly
unwieldy
because
you
don't
need
that
many
monitors
in
well
look
at
dreamhost.
They
have
two
five
petabyte
clusters.
I
think
they're
both
running
on
five
monitors
each.
So
I
think
we
had
our
quick
start
guide,
which
was
all
on
a
single
machine.
It
was
three
osds
and
a
monitor
just
for
testing
and
playing
around
with
you
know.
I
wouldn't
recommend
that
for
a
workload
of
any
meaningful
type,
the
smallest
well
to
give
you
an
idea
here.
B
This
probably
give
you
a
better
idea,
ink
tank
when
we
build
when
we
build
people,
we
support
them.
We
do
it
on
the
raw
size
of
your
cluster,
that's
how
we
priced
it
out
and
the
lowest
tier
was
what
64
terabytes
and
below
yeah.
So
I
mean
huge
right
to
most
people,
but
that
said,
you
know:
we've
seen
people
that
are
running.
You
know
100
gig,
on
a
sf
cluster
or
smaller.
I
mean
anything,
but
I
don't
think
I've
seen
an
actual
production
workload
on
anything
below
50
60
gig.
B
So
the
question
is
cpu
requirements
and
and
machines,
and
all
that
I'm
not
really
a
hardware
guy.
I
probably
won't
give
you
an
answer.
I
know
a
good
rule,
yeah
yeah!
Oh,
I
don't
have
it,
but
actually
something.
That's
really
good.
If
you
just
google
dream
compute
all
one
word:
dream
compute
architecture
or
dream,
compute
reference
architecture,
just
google
for
that
you'll
find
it'll,
be
the
first
result.
It's
a
pdf
dreamhost
actually
wrote
up
a
pdf
of
all
of
the
stuff.
B
The
specs
of
their
machines
for
the
osd's,
the
machines
for
the
monitors,
their
network
configuration,
although
I
will
say
it's
awesome
to
read,
but
their
network
configuration
was
wicked
overkill.
They
they
took
like
just
giant
buckets
of
money
and
they
just
kind
of
said
network
activate
like
it
was.
It
was
crazy,
like
40,
gig,
spine
routers,
with
double
interconnects
and,
like
you'll,
see
lots
of
money
not
necessary,
but
it
was
wicked
cool.
B
I
mean
one
good
rule
of
thumb
that
we
had
for
a
while
was
one
core
per
osd,
but
it's
not
you
don't
need
that
as
much
anymore,
but
if
you
have
it
you're
good,
you
can
actually
our
quick
start
guide
had
three
osds
and
a
monitor
and
a
metadata
server,
all
in
one
machine
just
using
directories.
So
it's
on
a
single
spinning
disk.
So
it's
possible.
You
know
you
just
have
to
weigh
performance
versus
you
know
what
you
want
to
put
into
it.
B
I
think
we're
probably
running
into
the
end
of
our
time
here
soon.
I
want
to
buzz
through
the
community
stuff,
and
then
we
can
answer
questions
and
you
know
I'll.
I
will
be
far
more
honest
when
we
get
to
the
bar,
I'm
sure
you
know
so
talking
about
oh
gosh.
Some
of
these
slides
are
outdated,
seth
developer
summit.
We
had
our
giant
one
a
long
time
ago.
Apparently
I
put
the
wrong
slide
in
sorry
self
developer
summit.
We've
got
it
coming
up,
24th
and
25th
of
june.
It's
a
virtual
event.
B
Historically,
it's
been
on
google
hangouts,
but
I
think
we're
going
to
use
a
thing
called
blue
jeans
this
time
around,
because
then
we
can
get
like
a
whole
bunch
of
people
all
in
a
single
room.
We
don't
have
to
well
anyway,
it's
a
video
hangout
we
get
together.
We
talk
shop
about
what's
going
into
the
next
release.
This
one
will
be
a
little
different
because
we
usually
we're
on
a
quarterly
release.
Schedule
right,
so
we're
get.
B
We
would
get
ready
to
release
firefly
and
we
would
talk
about
what
we're
gonna
do
for
giant
every
quarter,
but
firefly
got
delayed.
You
know
turns
out
getting
acquired,
is
lots
and
lots
of
work,
so
the
firefly
got
pushed
so
right
now
we're
kind
of
in
this
interim
release.
So
it's
going
to
be
talking
about,
what's
still
or
already
in
flight
for
giant
and
then
also
kind
of
looking
down
the
road
a
little
bit
at.
B
What's
going
to
come
with
hammer
and
just
talk
about
you
know:
where's
everybody
at
status
wise
for
their
projects.
Is
there
new
work
that
people
want
to
discuss,
so
the
format
is
everybody
that
wants
to
do
something
substantive
for
seth
writes
up
a
little
blueprint
on
our
wiki
and
each
of
those
blueprint
gets
like
a
30
minute
window
where
they
can
talk
to
sage
primarily
and
sage
can
say:
yeah,
don't
do
it
that
way,
that's
kind
of
stupid.
You
know,
unfortunately
he
would
never
say
it
like
that.
B
He's
way
too
nice,
but
he'll
tell
you
like
hey.
Your
idea
is
really
good,
but
maybe
we
should
do
about
like
this
way,
or
we
should
think
about
this,
and
this
this
and
rest
of
the
community
can
weigh
in
and
the
blueprint
owner
can
actually
use
that
30
minutes
also
as
a
way
to
say
hey.
I
need
help
with
my
messaging
bus.
Like
I
don't
know
anything
about
that.
Can
somebody
help
me
out
and
they
can
kind
of
recruit
other
members
of
the
community,
the
end
of
cds?
B
We
end
up
with
a
whole
bunch
of
things
that
are
saying
hey.
This
is
going
to
go
into
the
next
release
of
ceph.
These
are
the
each
of
the
guys
that
own
these
things,
here's
a
rough
idea
of
how
it's
going
to
get
done
for
the
people
that
do
work
in
ceph
really
helpful
for
the
people
that
are
just
really
interested
about
how
ceph
has
developed
it's
great
to
be
a
fly
on
the
wall
for
these
things,
so
keep
an
eye
out
for
that.
What
else
yeah?
B
If
you
want
to
contribute
there,
I
think
sef.com,
contribute
it'll.
Give
you
a
straight
up
blueprint
on
how
you
start
hacking
on
seth
yeah.
We
shifted
to
a
new
wiki
a
while
back.
It's
actually
a
mind
touch
wiki.
If
anyone
knows
what
that
is,
and
if
aaron
aaron's
a
ceo,
my
touch,
if
he
heard
me
calling
a
wiki,
I
think
he
just
like
dies
a
little
inside.
I
think
they're,
like
an
enterprise,
knowledge
based
solution
or
something
I
don't
know.
It's.
B
But
this
this
basically
is
the
place
for
all
of
our
developer
stuff.
So
all
of
our
blueprints,
all
of
our
seth
developer
summits.
If
you
go
back
down
here
on
the
planning,
there's
theft,
developer
summit,
if
you
go
down
there,
you
can
actually
go
into
each
of
our
last
four
summits
that
we've
had
and
there's
time
coded
links
to
each
of
the
youtube
videos
of
each
session.
B
So
if
you
want
to
know
something
about
the
rdma
work
from
our
firefly
session,
you
can
go
watch
that
discussion
and
see
all
the
guys
that
are
way
smarter
than
I
am
talk
about
it.
It's
pretty
cool
google
summer
code
yeah.
I
just
figured
I'd
mention
this.
We've
got
two
students
that
are
working
for
ceph
via
the
google
summer
of
code.
B
B
Supposedly
we
will
have
a
wireshark
plug-in
that
will
handle
all
of
the
ceph
behind
the
scenes
traffic
and
will
be
committed
upstream
to
wireshark.
So
you'll
be
able
to
use
it
natively,
so
fingers
crossed
for
that
and
saf
days.
This
is
what
I
mentioned
earlier.
We've
had
a
number
of
these
events.
I
think
six
or
eight
of
them
we
had
one
in
new
york
last
year.
We
did
one
in
sunnyvale,
california.
I
think
we
did
one
in
london
frankfurt.
B
We
had
one
that
was
kind
of
a
safe
day
in
malaysia,
but
boston
next
week,
next
tuesday
come
on
down.
If
you
can
get
away
these
guys,
these
are
full
day
events.
We
will
do
them
basically
anywhere.
We
can
get
enough
people
kind
of
pushed
into
a
room,
and
if
we
can
get
somebody
to
kind
of
say,
hey
here's
a
room,
then
it
happens
even
faster,
but
these
are
awesome.
I've
been
really
happy
with
how
these
have
been
turning
out.
I
think
frankfurt
we
had
like
200
people
show
up.
So
it's
crazy.
B
It's
basically
we
take
members
of
the
community
that
are
located
in
that
region
and
we
say
talk
about
what
you're
doing
with
ceph,
and
so
you
usually
get
about
six
or
eight
guys.
They
each
give
a
half
hour
talk
or
something
on
on
something
cool
that
they're
doing
with
steph
sage
usually
shows
up
at
these
things
and
gives
his
kind
of
where
chefs
at
and
where
it's
going
awesome
events,
meetups
yeah,
there's
god,
there's
probably
20
30
meetups
now
around
the
globe.
B
If
not
it's
coming
soon.
I
just
saw
it
on
the
list
not
long
ago
foundation,
not
a
lot
to
say
we
are
looking
at
potentially
moving
seth
into
a
foundation
setup
for
the
actual
ownership
we
might
not
need
to
now
that
we
have
red
hat
red
hat
historically
has
been
a
pretty
good
steward
of
open
source
projects
and
beyond
that,
with
the
distributed
copyright
you
know,
there's
really
limited
impact
that
any
commercial
entity
could
ever
do
to
ceph
the
project.
B
So
don't
know
where
this
is
headed,
but
we're
still
thinking
about
it.
If
you
have
input
or
questions,
let
me
know
and
yeah
we've
got
a
whole
host
of
other
things
that
are
going
on
in
the
community
in
terms
of
integrations
and
crazy
stuff.
Like
that,
you
know,
if
you
have
specific
questions,
ask
me.
One
thing
I
will
mention:
is
the
gennady
stuff?
B
B
I
think
so
I
don't,
I
don't
know
all
the
details.
I
haven't
really
dug
into
it
much,
but
I
know
they
did
a
good
presentation
at
geneticon
in
greece,
actually
that
I
think
made
it
up
to
the
web.
So
if
you
google,
for
that,
you
should
be
able
to
see
oh
yeah
metrics
if
anybody's
familiar
with
the
betergia
dashboard
that
runs
the
openstack
metrics
we've
got
the
same
things
at
metrics.com.
You
can
check
out
all
the
crazy
like.
B
B
B
If
you
wanted
to
look
at
calamari,
that's
kind
of
what
it
looked
like
six
months
ago.
It's
come
a
long
way,
but
yeah.
It's
got
some
neat
data.
Vis,
stuff
it'll,
actually
show
all
of
your
osds
as
little
green
dots,
and
if
you
have
a
problem
with
one,
it
gets
a
little
bit
bigger
and
it
gets
yellow
and
if
it
goes
down
or
something
gets
much
bigger
and
it
gets
red,
you
can
just
click
on
it
and
kind
of
see
like
oh.
What's
going
on
with
that.
B
So
I
mean
it's
got
some
cool
viz,
some
neat
stuff
with
graphite
and
whatnot,
but
it's
not
going
to
be
nearly
as
interesting
as
maybe
in
another
three
months
when
it
starts
getting
the
actual
like
hey,
we
can
plug
in
new
nodes
and
do
management
and
stuff.
So
that'll
come
soon
the
roadmap.
Just
so
you
know,
historically
with
think
tank.
This
may
change.
B
This
is
actually
under
a
lot
of
discussion
now
that
we're
part
of
red
hat,
because
we'll
have
way
more
qa
resources,
but
historically
we
would
have
a
release
every
quarter,
every
other
quarter.
We
would
have
kind
of
a
long
term
supported
release
this
may
change,
but
for
now
that's
kind
of
what
it
looks
like
firefly
we
just
launched
like
I
said
it
was
late.
B
Firefly
is
what
introduced
you
know
the
erasure
coding
and
the
tiering
and
a
lot
of
really
big
stuff.
That's
been
people
have
been
waiting
on
for
a
long
time,
so
yeah.
If
you
want
to
get
started
with
steph,
you
go
to
ceph.com.get
that'll,
take
you
to
a
downloads
page.
You
can
get
the
packages
or
the
tarballs
or
anything
that
you
want
to
play
with
there
yeah
the
quick
start
guide
that
I
was
talking
about.
B
B
John
wilkins
is
just
a
monster
and
for
those
of
you
that
have
ever
written
docs,
you
know
it's
not
an
easy
task
and
he
was
never
given
release
notes.
He
would
just
crawl
the
code
and
say:
oh,
hey,
here's
something
new.
I
should
write
that
up.
So
he
turned
he
literally
turned
code
into
documentation.
B
B
Where
volunteers
from
different
people
in
the
community
will
actually
schedule
office
hours
well,
they'll
be
in
irc
and
they'll
watch
the
mailing
list
for
a
certain
number
of
hours
each
week
so
that
you
know
you
won't
get
radio
silence.
If
you
come
during
one
of
those
times
and
they'll,
be
there
to
answer
questions
and
whatnot
so
and
that's
not
just
ink
tankers,
we
have
a
bunch
of
shifts,
but
that's
not
just
us.
So
there's
always
a
way
to
get
an
answer.