►
From YouTube: Brad Hubbard -- Troubleshooting Ceph
Description
A
A
Hopefully,
I
can
provide
some
guidance
on
how
to
handle
problems
when
they
do
arise.
I've
broken
the
issues
into
four
main
areas
of
performance,
hangs
and
hangs,
is
in
inverted
commas,
ear,
because
quite
often
a
hang
is
not
actually
a
hang
might
be
something
different.
That
just
appears
to
be
a
hang
a
process
still
running.
That
is
just
not
making
any
progress.
A
A
Performance
when
dealing
with
performance,
it's
important
to
establish
a
baseline
so
that
we
know
exactly
how
performance
is
degraded
and
hopefully
we're
performances
has
degraded.
If
that
happens,
most
of
this
most
of
the
tools
I've
listed
here
can
help
you
establish
baselines
for
the
various
subsystems,
ratoff
benches,
built
in
to
write
offs
and
is
a
quick
way
to
do
benchmarking,
benchmarking
of
a
safe
cluster.
A
Seftel
OST
bench
does
a
simple
right
to
each
to
the
storage
for
each
OSD.
You
can
specify
bytes
per
right
and
total
bet
total
rights.
If
I.
Oh
now
has
an
R,
B
R
vdio
engine,
which
allows
it
to
talk
directly
to
the
storage
using
lib,
RVD
and
lib
RVD
sits
on
top
of
liberato
switch.
Patrick
was
talking
about
before
NAT
/.
If
she
can
test
your
networking
segments,
you
need
to
test
all
of
them.
So
your
public
network,
your
cluster
network,
Oh
s,
DS
to
mom's
clients
to
moms
I've
listed
ppl
I.
A
A
A
You
can
use
PCP
sis
dad
collect
Dell
anything
you'll
use
to
collect
statistics
on
on
what
the
entire
systems
are
doing.
There's
also
the
SEF
benchmarking
tool,
which
is
under
development,
but
it
allows
you
to
run
everything
under
the
hood.
It
even
can
capture
statistics
from
things
like
Val
grind,
block,
trace
and
graph
those
for
you
and
given
that
it's
part
of
the
safe
project,
it's
you
know
it's
directly
dedicated
to
benchmarking,
Seth,
just
be
mindful
of
caching
and
and
the
effects
that
can
have
on
your
performance
data.
A
A
Slow
requests,
a
request
is
considered
slow
within
the
cluster
if
it
takes
greater
than
30
seconds
to
complete
by
default,
although
that
is
Chernobyl.
As
of
a
recent
commitment,
if
you
seen
these
in
the
logs,
you
should
check
the
health
of
the
cluster
and
check
the
indicated
hosts
for
problems
that
may
be
affecting
performance,
historic,
ops,
command
that
you
see
there
yep
down.
The
bottom
will
show
a
collection
of
the
worst-performing
recent
operations
and
perf
dump.
The
bottom
command
will
less
performance
counters,
and
both
of
these
can
offer
hints
at
where
the
issue
may
lie.
A
Moving
on
to
hangs,
is
it
really
a
hang
as
I
mentioned
previously?
Not
everything
that
appears
to
be
a
hang
actually
is
one.
However,
if
you
are
seeing
a
true
hang,
you
are
likely
to
see
processes
in
prolonged
d
state
in
PS
output
and
probably
a
lot
of
iowait
on
the
cpus
estrace
might
list
output
from
threads
that
are
running,
but
none
from
threads
that
are
stuck.
A
A
If
you
do
have
processes
in
d
state,
that's
an
uninterruptible
state
within
the
uninterruptible
fleets,
sleep
state
within
the
colonel,
so
we
need
to
get
information
from
the
colonel
on
what
those
threads
are
actually
doing
and
what
they
might
be
waiting
on
this
arc.
You
invoked
with
the
T
trigger
outputs,
a
stack
trace
for
all
threads
executing
in
kernel
space
to
syslog.
A
We
execute
it
twice
with
a
20-second
delay,
although
that's
an
average
arbitrary
delay
in
between
to
verify
the
threads
in
question
are
in
long-term
d
state
and
not
just
not
just
in
a
transient
state
as
a
transient
d
state
as
normal
processes
fleetingly
Indy
State
as
they're
doing
I.
Oh,
that's,
common!
A
A
If
you
are
real,
like
if
you're
seeing
a
lot
of
these
and
it's
difficult
to
determine,
what's
going
on,
it's
not
obvious
from
other
data
that
you've
collected
you
may
need
to
collector
of
the
emcore
VM
cos
would
probably
beyond
the
scope
of
this
discussion,
quite
a
big
area
to
get
into,
and
if
you
have
to
go
to
that
length,
you
may
require
assistance
from
your
support
organization.
Unless
you
have
those
skills
in
the
house.
A
A
But
in
that
case
you
probably
won't
see
high
CPU
utilization
stack,
traces
need
to
be
interpreted,
and
this
can
require
a
decent
understanding
of
the
issue,
but
they
are
definitely
worth
looking
at
as
they
can
provide
excellent
context
into,
what's,
what's
ifs
or
the
set
diamond
state
is
with
gdb
scripts
or
system
tap
probes.
You
can
gather
considerable
data
on
the
state
of
the
running
program
at
regular
intervals,
so
you
can
set
up
a
GDB
script
similar
to
instead
of
G
stack
in
that
loop.
There.
B
A
A
A
A
A
A
A
The
hash
in
green
represents
the
git
commit
that
this
surgeon
was
built
from,
and
the
version
number
is
shown
there
in
blue
and
they
can
be
important
and
I'll
mention
them
later
on.
You
can
also
see
this
information
by
running
Seth
minus
V,
Seth
minus
V
will
show
you
the
version
of
set
Seth,
but
it
will
also
show
you
that
sha-1
hash,
which
indicates
what
commit
that
version
of
SEF
was
based
on.
We
can
also
see
the
assert
occurred
on
line
79
of
heartbeat
map,
dot,
CC.
A
Okay
and
down
the
bottom
here
we
can
see
the
line
of
code
where
the
assert
was
called
the
code.
Logic
has
checked
the
suicide
timeout
and
has
determined
it
has
been
exceeded,
and
this
is
considered
to
be
a
fatal
problem
and
that
the
process
cannot
continue
so
we
programmatically
terminate
the
process.
So
the
developer
has
done
this
on
purpose.
He
said
if,
if
we
exceed
that
timeout,
I'm
going
to
call
this
a
cert
and
this
assert
is
guaranteed
to
fail.
This
assert
will
always
fail.
So
it's
like,
like
it
says,
it's
committing
suicide.
A
A
A
These
are
usually
memory,
accounting
or
memory
access
errors,
although
they
could
be
indicative
of
other
memory
problems,
perhaps
memory
exhaustion
or
even
problems
with
the
memory
hardware
itself,
I
would
say
these
can
go
straight
to
a
bug
report
if
one
doesn't
exist
already,
so
have
a
look
at
this
f
track.
Have
a
look
at
bugzilla
see
whether
there's
already
a
matching
error
report?
If
there
isn't,
you
can
go
ahead
and
submit
one,
definitely
here's
an
example.
A
A
Osd
process
that
was
running
the
function
in
red
in
front
nine
is
the
most
interesting
as
it
is
the
last
set
function
to
execute
before
the
program
entered
the
dilib
see
abort
code,
which
is
what
exactly
what
is
executing
in
frames.
All
the
frames
above
that
so
a
through
one,
are
all
the
actual
abort
code
after
the
assert
school.
Sorry
after
the
signal
is,
is
handled.
A
A
Eu
addressed
to
line
gives
the
source
code
line
the
memory
address
points
to
so
that
can
be
very
handy.
You
can
run
that
on
the
memory
address
and
get
sent
to
the
source
code
depending
on
inlining,
an
amount
of
assembly
interpretation
required
object,
dump
output
can
be
tricky
to
interpret,
and
I
find
I
find
the
gdb
output
provides
the
best
information
in
most
cases,
if
you
start
gdb
with
the
ceph
OST
and
a
note,
it
has
to
be
the
identical
binary
that
that
crashed.
A
A
A
In
the
case
of
unexpected
behavior,
it
is
important
to
identify
what
the
expected
behavior
was
and
what
the
actual,
what
actual
behavior
occurred,
always
try
to
get
a
timestamp
of
when
the
problematic
behavior
occurred,
as
that
will
make
things
a
hell
of
a
lot
easier
to
try
and
trace
in
the
logs
start
at
the
user
end
where
the
error
or
behavior
is
seen
and
work
backwards
towards
the
SEF
cluster
tracing
the
process
in
each
log.
As
you
go.
A
A
A
Ok,
these
are
the
debug
logging
options
for
OS
DS
and
their
recommended
values
with
the
top
two
or
three
been
first
to
try,
as
they
are
like
they're,
most
likely
to
show
any
issues
enabling
too
many
of
these
options
is
not
recommended,
as
it
can
flood
the
logs
with
data
making
them
difficult
to
interpret.
So
you
really
don't
want
to
turn
all
of
these
on
at
once,
you're
going
to
get
a
lot
of
output
sensor
logs.
A
If
there's
an
indication
of
a
specific
area
that
that
that
area
is
suspect,
then
you
might
want
to
enable
the
option
that
that
deals
with
that
particular
areas,
such
as
a
file
store
or
the
journal
that
may
be
warranted
if
you've
positively
identified
that
that
particular
area
is,
is
the
problematic
one
same
for
the
moms
and
once
again,
the
top
three
are
probably
the
ones
that
that
you'll
need
top.
One,
obviously
is
for
debugging
normal
monitor
transactions,
second
ones
messaging.
A
A
A
A
Openstack
OpenStack
issues
can
be
difficult
to
debug,
sometimes
due
to
the
number
of
separate
processes
involved
and
amount
of
logs.
Sometimes
the
actual
error
does
not
get
passed
up
to
the
higher
level
in
tax,
so
the
area
you
end
up
with
is
not
really
representative
of
the
issue.
A
timestamp
is
pretty
much
essential
in
these
situations,
so
you
can
follow
the
entire
transaction
through
the
legs
through
the
logs
from
end
to
end,
you
may
need
to
turn
up
log
in
velocity
for
all
processes
involved
or
each
in
turn.
A
In
order
to
get
to
the
bottom
of
the
issue
you
may
find,
the
problem
is
in
OpenStack
code,
rather
than
it
being
a
selfish
ooh.
So
in
other
words,
you
might
you
know
you
might
start
at
the
Nova
level,
trace
it
down
to
glance
or
cinder
and
it
stops
there.
You
find
the
actual
error
there
before
it
ever
gets
to
the
to
the
rbd
level
or
the
ceph
cluster.
A
K
rbd
the
kernel
module
logs,
2d
message
and
syslog,
so
it
should
hopefully
be
obvious
if
you
are
seeing
an
issue
with
it.
If
not,
this
module
may
require
instrumentation
with
system
tab
in
order
to
establish
what's
going
on.
So
this
is
one
area
where
system
tap
may
come
into
play.
Definitely
with
the
colonel.
Are
they
dead
client?
A
A
A
This
invocation
would
enable
debug
logging
for
all
OS
DS
in
the
cluster,
which
may
not
be
what
you
want.
You
can
restrict
it
to
individuals
if
you
want
by
only
specifying
their
IDs
rather
than
using
a
wild
card.
So
if
you
have
10
SD
and
you
want
to
turn
up
debugging
just
for
that
OSD,
you
can
specify
only
it's
I
do
the
second
parameter
and
in
the
command
to
reset
debug
logging
to
the
default
is
the
in
mem
MIT.
A
Seff
creates
a
copy
of
the
most
recent
log
entries
at
the
specified
verbosity
in
that
second
field
there
in
memory
and
that's
what
gets
dumped
into
the
log
in
the
event
of
a
crash.
So
if
you're
logging
normally
with
Seth
and
it
crashes,
it
will
dump
out
the
stack
trace
which
I
presented
in
one
of
the
earlier
slides.
But
it
also
dumps
out
recent
history
of
its
logs
and
those
logs
are
shown
at
a
higher
verbosity
level
by
default.
Then,
then,
what
you
would
have
set
by
default,
which
is
zero
just
normal
logging.
A
A
A
This
gives
you
the
source
code,
as
it
was
at
the
time
of
that
release
and
should
match
the
source
of
the
binaries
you
are
using.
So
when
you're
looking
at
the
source
code,
try
to
always
match
up
the
source
code
version
to
the
binary
version
that
you're
using
otherwise
it
can
be
misleading
and
pretty
confusing.
A
For
downstream
sauce,
you
can
download
the
source
package,
install
it
and
use
the
RPM,
build
command
to
prep
the
source
and
apply
the
net
necessary
patches
in
the
RPM
afraid,
I'm
fuzzy
on
how
this
works
for
Deb,
based
distros,
but
I'm
sure
there
are
equivalent
commands
on
those
to
accomplish
the
same
end.
Has
anyone
done
this
actually
like
downloaded
a
Debian
source
package
and
prep
the
source
to
get
to
get
the
source
tree
for
a
Debian
bond
binary
anyone
using
Ubuntu?
A
A
A
It
allows
you
to
navigate
the
code
base
and
jump
between
files
quickly
and
efficiently,
so
that
g
tag
module
is
the
gnu
global
tags,
module
really
really
good
for
jumping
around
source
code
in
Vinh
works
for
C,
C++,
Java,
C++,
isn't
quite
as
good
as
C,
but
still
very
usable
and
very
handy
a
lot
better
than
trying
to
jump
around
in
the
files
without
it
most
of
the
I
should
mention.
I
guess
most
of
the
most
of
the
safe
code
base
is
c
plus
plus,
so
that's
what
you'll
be
reading
most
of
the
time.
A
Resources
I
hope,
I've.
Given
you,
some
ideas
for
looking
into
SEF
is
as
they
arise,
but
it
is
important
to
realize
that
you're,
not
alone-
and
there
are
various
sources
of
health
available,
don't
be
afraid
to
give
a
shout-out
on
on
the
mailing
list
or
the
IRC
channel.
You
can
also
check
the
SEF
bug,
tracker
and
bugzilla
to
see
if
your
problem
is
a
known
issue.
A
There
are
also
some
excellent
troubleshooting
docks
under
the
SEF
storage
cluster
section
on
Doc's,
f
com,
so
I
haven't
gone
into
those
areas
of
troubleshooting
because
they're
very
well
covered
within
that
documentation,
and
I
highly
recommend
that
you
go
and
have
a
look
at
the
dock.
The
excellent
documents
on
Doc
CF
com-
that's
one
of
the
good
things
about
Seph.
Is
it's
very
well
documented?