►
Description
Cephalocon APAC 2018
March 22-23, 2018 - Beijing, China
Michael Hackett, Red Hat Engineer
Vikhyat Umrao, Red Hat Engineer
A
B
Afternoon,
everyone,
my
packet,
pencil
software
maintenance
engineer
with
red
hat,
SEP
storage-
it's
my
colleague
Nick
at
I'm,
Rob,
also
or
pencil
software
maintenance
engineer
with
red
hats:
f,
storage
combined,
we've
been
supporting
set
for
seven
years
so
been
along
for
quite
the
ride,
with
with
storage.
We're
gonna
cover
today
is
some
common
support
issues
that
we've
seen
over
the
last
or
so
years,
supporting
red
hats
F
and
on
what
you
can
do
to
look
into
troubleshooting
these
issues
and
resolving
these
types
of
issues.
B
The
two
columns
I've
spent
cover
today
are
to
the
more
the
more
the
issues
that
customers
log
with
us
this.
This
is
coming
from
a
larger
presentation
which
had
upwards
of
eighty
slides
in
it,
and
we
cut
it
down
to
about
20
slices
of
these
two
issues.
I'm
gonna
cover
on
slow
block
requests
cover.
What
is
this
low
request
possible
causes
of
these
different
types
of
slow
requests
and
what
you
can
do,
the
troubleshoot
them
foods
within
the
hardware
using
Linux
utilities
and
I
find
a
common
trouble
spots
with
these
also
been
cover
flapping
OS.
B
These
causes,
when
using
out
their
storage
the
front
of
state
way
where
there's
millions
of
objects
into
monitors,
dropping
OS.
These
same
thing
we're
gonna
cover.
Where
can
we
start
looking
into
possible
causes
temporary
solutions
for
alleviating
these,
and
also
permanent
solutions
to
alleviating
these
types
of
issues?.
B
C
B
B
B
That's
adjustable
over
here
the
configure
option,
OS
the
OP
complaint,
I'm,
keeping
it
at
30
seconds
is
recommended
due
to
if
you
decrease
this
or
increase,
this
may
be
seen
self
positive,
false
positives.
They
need
to
actually
have
an
assorted
quest
log
when
this
actually
know
which
is
being
seen
by
the
end
client
server,
slowly.
Press
log,
accompanied
to
always
by
block
requests,
gave
some
examples
of
what
you're
seeing
over
here.
If
you've
been
working
set
for
awhile
you'll
be
familiar
with
seeing
these
by
every
now
and
then,
as
sometimes
scoring
this
occurrence.
B
Basically
just
pushing
too
many
reads:
writes
to
the
system
that
are
overloading
you
there's
either
CPU
RAM
or
your
OS.
These
backing
disks
improper
configurable,
7
OS.
These,
for
example,
BIOS
store
up
threats,
OSD
up
threats
being
set
to
high
for
your
hardware
to
handle
it's
common
common.
Seen
when
you're
going
to
tune
your
cluster
and
you're
like
oh.
If
I
can
set
BIOS
thrown
off
threads
that'll
improve
performance
of
my
SEP
cluster,
you
can
get
Jack
those
up
to
a
higher
count
and
socket
Hardware
I
can't
handle
it
put
some
configuration
issues.
B
This
is
common
with
improper
huge
account
settings
on
under
Melanie
set
pools.
It's
going
to
be
to
imbalance
throughout
the
cluster
and
OSD
know,
is
taking
on
more
data.
More
reads
writes
than
in
other
OS,
these
the
cluster
cluster
on
the
back
pillar
recovery
OSD,
goes
down
back
full
recoveries
prioritized.
We
can
see
slow
request
logging
due
to
the
the
churn
that
we're
getting
from
recovery
on
threads,
also
deep
scrubbing.
We
know
that
when
keep
scrubbing
is
the
currently
and
we
lock
those
objects
at
the
at
the
time.
The
deep
scrub
that's
occurring.
B
We
can
actually
block
and
I/o
that's
coming
to
a
specific
objects,
that's
being
deep.
So
up
at
the
time,
there's
a
work
coming
up
in
luminous,
where
we
actually
have
a
pre-emptive
scrub.
That
gets
him
to
talk
about
that
a
little
later,
where
we
actually
prioritize
it
by
an
I/o
back
off
the
beep
scrub.
Let
the
client
access
the
object
and
then
resume
the
team
scruff
wants
to
fly.
It's
often
please,
most
recently,
a
lot
of
a
lot
of
stuff
we've
been
seeing
has
to
do
with
the
compaction
and
splitting
a
car
in
EOS.
B
B
Common
types
of
slow
requests
that
you
generally
see
waiting
for
rewrite
locks.
Basically,
this
occurs
when
the
the
there's
an
in-progress
off
on
the
the
object
in
question
and
the
backing
up
waiting
is
waiting
to
acquire
a
lock.
So
for
that
previous
write
to
complete
or
or
operation
to
complete,
and
for
with
that
object,
it's
a
way
to
require
that
lock.
This
can
be
due
to
several
issues.
B
So
when
we're
way
to
be
the
primary
primary
OSD
is
waiting
for
our
sub
OS
these
to
process
the
the
write
in
the
journal
a
we
don't
report
the
act
back
to
the
client
until
our
final
replica
copy
has
processed
that
right.
So
when
a
waiting
for
some
pop
system,
they
usually
be
due
to
other
OS,
these
are
replicas
sets
either
being
on
a
slow,
node,
I
they're
having
a
slow
disk
and
overwhelm
disk
in
these
things.
B
B
B
B
So
some
common
troubleshooting
techniques.
So
this
is
the
the
bones
of
this
this
presentation:
what
can
we
do
to
determine
what
the
actual
cause?
The
issue
is,
whether
it
be
physical,
Hardware,
networking
things
of
that
picture,
so
obviously
the
first
place
you
always
want
to
start
when
you're
looking
into
any
type
of
issue
log
in
the
set
clusters
just
have
a
look
at
yourself
status
to
set
help
detail.
B
Thus,
if
there
is
any
back
row
recovery
occurring
in
the
cluster
any
types
of
OS,
these
have
gone
down
and
our
slow
request,
even
logging,
so
start
off
with
a
set
path
set
that
says
lower
step
health
detail,
and
that
will
give
you
more
general
idea
of
where
you
need
to
look
now.
The
other
thing
is
we
can
determine
the
OS
these
log
in
the
solution
flow
requests
to
the
shared
in
type
of
common
piece
of
hardware.
They
in
the
same
storage
is
known.
Are
we
in
the
same
rack?
B
Are
they
sharing
the
same
network,
switch
that
that
usually
can
help
us?
So
if
you
have
a
large
number
of
OS,
these
logging
slow
requests,
but
you
determine
that
they're
all
sharing
the
same
piece
of
hardware,
such
as
a
storage
node.
Then
we
can
just
set
that
storage
node
down
in
the
cluster
and
we
should
be
able
to
service
the
the
requests
across
other
OS
things
verifying
this
performance
on
OS,
these
log
and
slow
requests.
B
We
can
use
Linux
utilities
such
as
iOS
that
look
for
high
utilization
of
high
a
wait
times
against
specific
discs
and
that'll
usually
help
you
determine
cost
of
a
slow
request,
basically
an
overloaded
disk.
You
usually
only
see
these
types
of
issues
we
can
look
into
its.
It
could
be
a
single
OST,
login,
the
slow
requests
or
we
could
have
a
number
of
OS.
B
So
he
back
up
in
the
sub
cluster
another
another
common,
useful
thing
we
can
look
for
is
network
performance
or
throughput
Evers,
so
anything
seen
the
Nix
retransmits
any
any
possible
changes
in
the
network
may
be
an
admin
made,
a
change
in
MTU
or
something
along
those
lines.
Kind
of
feeds
into
the
next
point.
B
Jungle
frames,
so
you
just
just
verified
from
the
network
standpoint
that
your
your
networking
equipment
can
handle
that
frame
type
size
and
to
use
sizes
need
to
match
throughout
the
environment
in
order
to
successfully
pass
frames,
so
that
that's
that
that's
also
a
common
issue
that
we've
seen
in
the
past,
utilizing
ifconfig
or
ping
the
specific
frame
size.
So
what
can
be
used
to
determine
if
network
is
the
cost
of
that.
B
Determining
if
deep
scrubbing
compaction
of
splitting
is
occurring
at
the
time
of
slow
requests
can
be
looked
at
as
well.
If
none
of
the
above
above
fits
your
issues
for
a
dis
performance
network
or
anything
along
those
lines
set
set,
thought
log
file
will
tell
you
if
the
deep
scrub
is
occurring
against
the
specific
OSD.
Give
you
that
time
frame,
and
then
you
can
see
the
slow
request
was
logging
at
that
time
frame
compaction
of
splitting
and
be
a
little
more
difficult
to
determine,
although
with
later
releases
of
of
luminous,.
A
B
B
A
handy
tool
we
use
is
called
the
set
log
parser.
This
is
available
on
github.
It
was
written
by
another
one
of
my
colleagues
at
Red
Hat.
The
set
block
bar
can
be
very
handy
when
determining
tough
to
find
out
slow
as
the
issues
lots
of
times
when
we
get
a
slow
request
and
we're
logging
against
a
single
OSD.
That's
easy
to
determine
that.
It's
that
this
that
that's
backing
up
the
cluster.
B
What
this
log
parts
of
tool
does
is
basically
you
you
can
feed
it
an
OSD
tree
which
basically
takes
into
account
the
crush
deep
Square
where
our
OSDs
and
our
and
our
set
nodes
reside,
and
we
can
feed
it.
A
set
thought
log
file
and
we
run
the
parser
and
it'll
give
you
a
nice
output,
such
as
the
example
that
you
see
down
there.
B
I'm
sorry,
it's
a
little
small,
but
if
you
have
copies
of
the
slides
or
you
want
to
want
to
use
it,
you
can
have
a
better
view
and
we
actually
show
you
and
we
log
the
number
of
types
of
slow
requests
that
logged
the
crushed
root.
That's
actually
logging
the
slow
request,
the
OS.
These
have
a
log
on
the
slow
request.
So
it
gives
you
a
a
much
easier
high-level
view
of
where
the
issue
actually
bus.
B
C
Let's
talk
about
flapping
or
his
teeth,
so
what
is
that
USD?
So
mostly
itself?
We
have
two
type
of
network
which
is
called
public
network
and
the
Questor
network,
so
public
it
was
nothing
but
where
you
are
all
the
clients
that
monitor
the
sites
and
OSD
talk
to
each
other
with
their
peers
in
mythbuster
Network.
What
happens
is
that
when
two
is
these
are
three
as
per
the
replicas?
C
They
talk
to
each
other
and
if
one
host
is
not
responding
to
another
knows
the
earth
and
if
that
happens,
for
multiple
OSD,
if
any
two
is
leaves
which
are
cleared
to
that
OSD,
which
is
not
responding.
Inform
back
to
the
monitor
that
this
is
mrs..
Lee
is
not
responding
to
me.
Monitors
mark
that
was
near
the
top,
and
if
that
OSD
is
not
down
and
the
PIDs
is
still
up,
it
will
bounce
back,
and
it
will
say
that
you
have
marked
me
wrong
later
on.
C
But
if
the
process
really
went
down,
the
quality
will
go
out
which
is
default
300
seconds,
so
the
default
value
which
is
controlled
by
the
Toolman,
which
is
called
Manos
diamond
down.
Recruiters.
You
can
increase
this
to
the
number
of
OSDs
in
one
node
plus
one
so
that
you
can
come
out
of
false
positive
so
that
you
need
minimum
one
number
of
100's
needs
to
inform
on
it
to
that.
Okay,
these
are
these
many
number
of
quality
they're
not
responding
to
each
other.
So
this
is
how
a
flapping
was.
C
He
like
OSD
goes
down
and
comes
back.
This
is
nothing
but
the
traffic
velocity.
Now,
let's
focus
on
why,
with
the
large
number
of
object
cluster
for
the
article
to
work
load,
why
we
see
flapping
hosting
sometime,
so
this
is
where,
to
start,
let's
identify
the
workload.
If
you
have
large
number
of
objects,
it.
A
C
Of
puts,
because
you
are
writing
not
operator
to
the
cluster,
that's
fine,
but
if
you
have
deletes
in
the
same
ratio
like
if
you
let's
say
you
are
writing,
80%
of
your
workload
is
right
and
then,
finally,
you
are
deleting
60%
of
data
in
the
same
amount
of
time
it
can
cause
this
kind
of
issues.
I
will
be
covering
myths
like
that.
Why
why
a
delete
workload
can
cause
this
issue
shake
the
bucket
will
be
sharding
which
is
used
or
not.
C
C
So
what
are
the
possible
causes?
The
first
issue
here
is
that
when
you
have
millions
of
objects,
but
they
would
be
B.
Oh
my
back-end
for
that
OST
is.
It
grows
like
anything
right,
because
each
index
gets
stored
into
the
OMA
keys
and,
oh,
my
keys
are
nothing
but
your
label
any
key
value
pairs.
Now
what
happens
that?
Is
you
run
operations
like
listing
of
the
bucket?
It
needs
to
read
up
on
the
data
from
the
oh
man
and
as
much
as
Rita.
C
C
The
second
issue
is
that
what
I
talked
in
the
previous
slide
night
beneath
workload
so
like
when
you
have
delete
workload,
it
causes
loss
of
the
steal
data
in
the
home,
and
so
what
happens
is
that
you
delete
a
lot
of
objects
within
a
bucket,
always
in
multiple
pockets.
It
will
remove
all
the
entries
for
that
from
the
home,
a
keys,
but.
A
C
A
C
It
will
trigger
the
compaction
to
remove
all
that
steal
data,
and
if
you
have
lots
of
steal
data,
it
will
be
always
compacting
and
as
Michael
covered
compaction
is
a
very
you
know
it's
lot
of
CPU
and
intensive
and
about
this
intensive
operation.
So
what
happens
is
that
it
start
causing
issues
to
OST,
because
it
will
be
always
compacting
and
it
won't
be
annoying
or
cease
to
perform
the
eye
of
the
cluster,
because
most.
C
And
reasons
why
you
can
see
you
have
a
thief
laughing,
because
your
world
is
either
compacting
the
devil,
DB
or
the
deepest
corruption
or
prime
die
Oh
listing
a
is
always
keeping
oh
man.
It
is
always
reading
the
omec
Lisa,
so
till
jewel,
release
or
before
June
release.
The
home
at
back-end,
though,
was
the
default
of
my
back
and
his
label
to
me
and
Annie
Loomis
cluster,
which
were
upgraded
or
which.
C
C
Is
pretty
good
because
it
uses
a
multi-threaded
compaction
and
lot
of
other
enhancements
like
bloom
filters
and
all
that
thing
block
sleep
with
label
DB.
The
drawback
is
that
it
doesn't
even
multi-threaded
compaction.
It
signals
real
compaction,
it
doesn't
have
those
features.
What's
wrong,
with
rocks
really
has
for
improving
the
performance.
So
what?
What
would
be
the
temporary
solutions
as
soon
as
you
see
that
you
have
lots
of
and
millions
of
object
in
your
question?
You
see
this
issue.
C
Waiting
and
failing
because
your
overseas
is
going
up
and
down
in
the
flapping,
it
will
trigger
the
recovery
and
bearing-
and
these
are
the
two
operations
various
bronchi,
because
we
have
to
be
consistent
into
the
data
we
have
to
be
in
consistently
state.
So
what
happens
is
that
your
customer
will
immediately
start
shouting,
like
my
clients,
are
timing
out
and
not
my
plan.
My
application
is
not
able
to
get
the
data
from
cluster.
So
what
would
be
your
immediate
solution
to
solve
this
issue?
C
The
first
thing
is
that
if
the
describe
is
running
in
the
cluster,
so
they
are
two
things
either
you
can
set
the
nori
describe
flag.
Probably
if
it
is
a
hammer
cluster
in
Hamilton,
because
the
feature
for
sadena
no
deeper
flag
to
particular
pool
came
to
jewel,
release
came
in
journalist
so
and
command
is
that
safe
over
steeple
said
poor
name
and
the
no
discover
equal
to
1.
So
this
will
take
care
of
setting
the
no
remiss
/
flag
online
index
book
now
how
it
will
help
as
soon
as
you
will
set
it.
C
C
C
And
find
store
objects
and
recovery
of
threads.
So
what
happens
with
them?
If
you
see
that
in
the
wall
street
dogs,
after
setting
an
OD
Vista
flag,
still
the
vs.
are
flapping
and
they
are
timing
out.
So
what
you
do
go
ahead
and
increase
the
timeout
values
like
this,
so
these
are
configurable.
You
can
use
a
central
command
to
inject
them
dynamically,
but
it
is
highly
recommended
to
put
these
values
in
Sep
calm.
If
you
don't
put
stepped
out
on
these
values.
C
What
will
happen
if
RC
goes
down
and
comes
back
and
the
PID
is
killed
when
the
verse
they
will
come
back?
Both
new
values
won't
be
in
memory
so
to
keep
this.
Is
this
value
a
link
existed
when
you
are
troubleshooting
this
issue
or
when
you
are
helping
your
customer
or
users
definitely
make
sure
that
these
values
are
receptor
con,
and
these
can
be
knitted
by
and
injects,
are
also
so.
C
So
for
objects,
you
have
a
geofence,
that's
what
is
time
out
of
the
trick,
but
after
that
there
is
a
society
timeout,
so
OSD
tries
to
keep
up
that
much
time,
but
if
it's
not
able
to
cook
worked
in
that
time,
what
is
it
hits
that,
aside
time
out
of
time,
it
kills
himself
so
that
it
doesn't
harm
to
the
requester?
That's
when
it
goes
out
and
that's
when
the
PID
goes
of
it.
C
C
It
has
label
DB
to
compact
and
then
was
due
to
put
so
it
doesn't
have
to
combat
always
again
and
again,
and
we
have
seen
that
oh,
my
G
is,
which
are
more
than
50
gigabyte.
They
come
down
to
ten
or
five.
Once
you
do
offline
compaction
because
for
offline
compacts
it
doesn't
mean
a
threshold
to
me.
You
are
manually.
B
C
Was
talking
about
the
size
of
the
index
card,
how
you
can
calculate
it,
so
you
can
use
the
reduce
low
level
come
on
choose
one
of
the
index
card
object
from
Belarus,
and
then
at
that
point
we
see
there's
element.
It
will
give
you
the
number
of
keys
inside
Devadas
index
card
and
each
shard
is
nothing
but
200
by
it.
You
can
multiply
it.
You
will
get
the
size
of
the
object
that
doesn't
get
solid
object
if
it
is
greater
than
20
and
B.
It
means
that
definitely
you
need
to
recharge
your
eye.
C
A
bucket
I've,
seen
the
we
have
seen
the
objects
like
one
gigabyte,
not
500
MB
is
for
time.
So
the
recommendation
is
something
like
you
recharge
your
pockets.
There
is
a
feature
in
luminous
which
can
do
take
care
of
dynamically
shouting.
But
if
you
have
very
good
production
posture
to
take
a
production
downtime-
and
you
can
do
a
family
shouting
and.
A
C
C
C
And
if
you
don't
want
to
use
these
instance,
you
have
another
option,
is
called
file
solo.
My
background
sleeping,
you
can
rebuild
our
sleep
in
summary,
if
I
cover
make
sure
that
you
have
indexed
pool
for
a
production,
large
cluster
make
sure
you
in
it
school
is
backed
by
either
nvme
or
SSDs.
However,
propagated
next
are
certain.
C
Care
of
your
future
growth,
like
this
money,
100,000
personal
object
and
then
how,
in
his
foolishness,
need
to
use
Ross
DB,
which
is
before
it
compaction,
trade
and
completion,
disabled
and
the
rocks
TV
cash
should
be
killed
properly
and
one
thing
which
Michael
talked
about
the
pre-empting
had
be
disturbed.
This
is
coming
in
luminous.
So
now,
whenever
you
have
a
client
area
coming
to
a
DV
scope
object,
the
deepest
club
I
would
be
taken
back
and
the
client
I
would
be
loud.
So
this
is
coming
in
luminous.