►
From YouTube: Ceph Crash Telemetry - Observability in Action
Description
Join us November 3-16th for different Ceph presentation topics! https://ceph.io/en/community/events/2022/ceph-virtual/
Ceph Crash Telemetry - Observability in Action
To increase product observability and robustness, Ceph’s telemetry module allows users to automatically report anonymized crash dumps. Ceph’s telemetry backend runs tools that detect similarities among these reported crash events, then feed them to Ceph’s bug tracking system. In this session we will explore Ceph crash telemetry end-to-end, and how it helps the developer community to detect emerging and frequent issues encountered by production systems in the wild. We will share our insights so far, and learn how users benefit from this module, and how they can contribute.
A
A
A
A
A
We
also
have
the
crash
Channel,
which
is
which
has
information
about
all
the
questions
that
happened
in
that
poster,
and
the
information
is
like
the
back
trace
of
the
crashes
and
exactly
where,
in
this
F
code
on
the
crash
occurred,
this
one
is
also
on
by
default,
and
then
we
have
the
device
channel
that
has
information
about
the
health
metrics
of
all
the
drives
in
that
cluster.
It's
mostly
smart
data.
A
We
collect
nvme,
metrics,
we're
applicable
and
we're
working
on
adding
some
vendor-specific
metrics
as
well,
and
this
one
is
also
on
by
default,
and
then
we
have
the
ident
channel.
We
let
users
have
the
option
to
identify
themselves
if
they
want
this.
One
is
of
course
off
by
default,
and
this
channel
has
the
content,
name,
email
or
organization,
and
so
on,
and
in
Quincy
we
added
the
cluster,
the
first
channel,
which
has
information
about
the
cluster's
performance
counters,
and
this
one
is
also
off
by
default.
A
It's
really
important
to
emphasize
here
that
by
default,
Telemetry
reporting
is
of
users
need
to
explicitly
opt
in
by
agreeing
to
a
cdla
sharing
license
and
they
can
do
that
either
by
a
CLI
command,
Steph
Telemetry,
on
or
with
the
staff
Telemetry.
Sorry,
with
this
FM
dashboard,
wizard
and
reports
can
be
previewed
with
self
Telemetry
show
Dash
all
and
in
Quincy
the
command
would
be
def,
Telemetry
preview.
Dash
all
I
want
to
talk
about
privacy
before
we
continue,
we
require
to
reopt
in
on
staff
upgrades.
A
A
The
reports
don't
contain
any
sensitive
or
identifying
data
like
pool
or
host
names
and
object
names
or
contents.
We
anonymize
the
reports,
so
the
cluster
is
assigned
a
random
uid,
which
is
therefore
Telemetry.
This
is
not
the
the
fsid
the
fsid
is
not
reported.
We
also
remove
the
disk
serial
ID
and
the
device
Channel
and
we
never
store
IPS.
On
our
back
end.
We
also
have
a
separate
Telemetry
reports,
one
for
the
anonymized
cluster
data
and
one
for
the
anonymized
device,
health
metrics
and
they
are
sent
to
different
endpoints.
A
So
why?
Why
do
we
need
all
that?
So
the
motivation
to
have
Telemetry
for
developers
is
to
get
feedback
on
what
features
are
in
use
in
the
wild.
We
can
learn
about
the
upgrade
Cadence
and
version
adoption
rate
from
the
basic
Channel
from
the
device
Channel
we
can.
We
can
see
what
hard
disks
and
SSD
models
users
are
deploying
in
the
wild
and
in
the
long
run
it
helps
us
to
create
an
open
data
set
of
device.
A
The
motivation
is
to
validate
our
installations
by
looking
at
what
is
common,
what
what
do
other
stuff
users
are
doing?
In
the
long
run,
it
helps
to
preemptively
mitigate
failing
devices.
Once
we
have
more
accurate,
failing
prediction
models,
users
can
preemptively
mitigate
these
devices,
which
reduces
downtime
and
shifts
downtime
to
maintenance
window
other
than
peak
hours.
A
With
a
crash
information
users
do
not
need
to
actively
report
issues
or
Open
tickets
for
each
crash.
So
all
the
automation
around
it
saves
a
great
deal
of
time,
and
users
can
also
use
the
open
data
set
of
crashes
that
we
have
in
our
Bud
tracking
system,
tracker.staff.com
to
better
understand
an
issue
and
they
can
see
if
there
is
a
fix
for
it
and
what
version
it
is
fixed
in.
Maybe
they
want
to
upgrade
to
that
version
to
to
have
it
to
have
that
fixed.
A
Let's
take
a
quick
look
on
the
architecture
on
the
staff
cluster
side,
specifically
for
the
the
crash
Telemetry
process.
So
whenever
a
demon
crashes
on
the
on
the
ceft
cluster,
this
can
happen
due
to
various
reasons.
I
can
be
an
unexpected
state,
it
can
be
a
failed
surgeon
or
it
gets
a
bit
even
a
bug.
A
signal
Handler
will
generate
a
crush
dump.
A
It
will
write
it
to
the
local
disk
and
then
the
demon
exits
and
restarts
we
have
in
each
node
a
crash
sap
crash
demon
which
will
check
periodically
that
this
directory
and
it
will
post
new
crashes
to
the
ceph
cluster,
using
the
self-manager
module,
the
crash
manager
module,
which
will
write
it
to
the
database
and
then
once
once
a
day
in
case
the
user
is
opted
into
Telemetry
and
into
the
crash.
Channel
Telemetry
will
query
this
database.
A
So
what
is
the
the
crash
report
contain
we're
talking
about
the
back
Trace
that
that
is
containing
that,
but
we
also
have
a
crush
ID
and
the
crash
demon,
type
and
name
the
step
version
of
of
that
demon
and
some
information
about
the
OS,
the
distribution,
current
version
and
it's
applicable.
We'll
have
some
information
about
the
assert
and
on
the
back
end,
we
can
combine
this
with
other
cluster
information
that
comes
from
basic
channel.
If
it.
A
So,
let's
take
a
look
at
a
couple
of
Rob
Back
traces.
These
are
real
examples
that
they
were
received
in
Telemetry,
I,
simplified
it
so
it
could
fit
on
one
one
slide.
So
we
can
see
that
these
two
questions
they
look
quite
similar
and
from
a
developer
point
of
view,
they
represent
the
same
issue
but
they're
not
similar.
A
So
the
differences
can
happen
due
to
different
SEF
versions,
code
changes,
which
is
each
version
of
course,
or
a
different
compiler
versions
or
even
compiler
optimization,
and
this
affects
the
symbols
and
the
offsets
and
addresses.
So
we
need
to
find
a
way
to
tell
that
there
are
indeed
the
same,
so
we
sanitize
the
back
Trace.
A
We
remove
all
the
offsets
and
addresses
from
the
frames,
and
we
do
that
by
using
the
and
and
sorry
and
after
after
that,
we
use
red
excess
to
search
and
replace
patterns
in
the
frames
and
then
to
filter
out
entire
frames
completely.
So,
for
example,
those
empty
frames
are
irrelevant
to
understand
the
nature
of
that
crash.
A
A
Another
thing
that
is
important
to
say
here,
that
is,
that
we
support
multiple,
multiple
generation
or
recipes
of
signatures
which
allows
for
backward
compatibility.
So
in
case
we
find
optimizations
to
a
recipe.
We
can
detect
that
the
old
signatures
are
actually
the
same
case
as
well,
and
then
the
cross
processor
will
populate
the
database
and
we
will
have
a
lot
of
good
information
in
the
database,
but
having
just
the
data
is
not
enough.
We
need
to
to
do
something
with
that,
and
this
is
what
the
redmine
bot
does.
A
A
For
example,
we
we
saw,
we
fixed
an
issue
that
we
saw
in
Telemetry,
but
then
we
have
new
reports
in
Telemetry
of
newer
versions.
That
has
has
the
fix,
but
now
reporting
the
same
issue
all
over
again.
The
bot
will
know
to
open
a
new
issue
and
to
look
it
to
the
original
fixed
issue.
So
developers
can
have
a
heads
up
to
understand
that.
A
A
So
you
can
see
that,
according
to
each
project,
it's
important
to
to
say
that
not
oh
oops,
not
all
pressures
I
mean
step
bugs.
So
there
could
be
issues
with
the
hardware
or
iOS
that
we'll
see
crashes
reported
due
to
that.
That
could
be
also
environment
or
resource
limitation,
or
configuration
issues
that
end
up
in
crashes
and
there
can
be
issues
of
other
dependencies
as
well.
A
That
career
is
the
prosperous
database
and
Maps
it
to
to
the
redmine
instance,
and
we
also
have
the
grafana
dashboard
that
queries
the
database
and
shows
a
lot
of
analysis
and
panels
to
Developers
and
then
there's
another
component
which
I
don't
know.
If
I'll
have
time
to
talk
about
today
about
the
signature
merger
that
knows
to
to
take
a
couple
of
crash
signatures
and
to
merge
them
together
in
case
the
recipe
has
been
improved,
all
right,
please,
let
me
know
if
you
have
any
questions.
B
A
Okay,
that's
a
good
question.
Basically
right
now,
the
the
crashes
should
be
synced
with
the
red
mines,
so
they
can
search
redmine,
but
we're
working
on
having
tools
for
Developers
for
sorry
for
users
as
well
to
to
see
like
a
bird's
eye
view
of
what's
happening
with
all
the
crashes
in
their
cluster
that
they
reported
and
how
they
are
mapped
to
issues
in
redmine.
So
this
this
this
is
something
which
is
ongoing
work.
A
A
So
right
now,
users
can
access
the
data
with
the
redmine,
so
the
the
back
traces
and
crash
signatures
are
in
redmine
for
developers.
They
do
need
to
have
access
to
this
sepia
lab
currently
to
to
see
all
of
the
breakdown
of
the
crash
dashboard,
and
it's
really
important
just
to
emphasize
here
that
many
times
we
see
that
users
publish
their
crash,
the
the
crush
reports
in
redmine,
but
they
removed
the
signature.
A
A
So
please
record
that
I'll
I'll
go
ahead
and
I'll
show
the
demo
of
the
dashboard
by
using
an
example
of
how
we
can
learn
about
new
crash
reports
of
new
releases.
So
we
just
released
the
queen
c17
2.5,
so
we
will
have
sorry
we
will
have
in
the
dashboard
all
of
the
crashes
that
were
reported
with
this
version.
So
you
can
see
that
there's
the
the
search
result
page
here
that
tells
us
about
how
many
clusters
have
seen
this
specific
issue.
A
A
A
We
can
see
the
breakdown
by
versions,
so
that
was
reported
mostly
for
17-3.
Here
we
have
the
sanitized
back
Trace
that
we
can
search
the
entire
database
for
other
other
signatures
that
have
similar
frames.
So,
for
example,
if
we're
curious,
what
other
signatures
contain
this
Frame?
We
can
just
click
on
that
and
I'll
move
to
that
tab
momentarily
to
see
how
many
other
signatures
there
are
in
case.
It
was
a
insert
a
question
to
a
surgeon.
I
would
see
the
details
here
here.
A
Some
users
choose
to
identify
themselves,
so
here
we
will
have
the
information
which
was
removed
for
the
sake
of
the
demo,
but
we
will
have
contact
details
of
users
that
identify
themselves
and
we
actually
contact
users
every
now
and
then,
when
we
need
more
information,
so
we're
so
grateful
for
you
for
helping
us
with
this
and
providing
logs.
This
is
a.
This
is
very,
very
helpful
for
us.
A
Okay,
I'll
I'll
go
back,
there's
a
lot
to
talk
on
the
on
the
dashboard,
but
I'll
go
back
to
the
slides.
I
want
to
quickly
talk
about
some
some
fixes.
That
happened
thanks
to
to
reports
that
happened
in
Telemetry.
So,
for
example,
we
can
see
that
the
Telemetry
bot
opened
this
issue
and
it's
populated
all
the
relevant
Fields,
such
as
a
the
affected
versions,
and
we
have
here
the
sanitized
pack
trays
some
raw
example
of
a
crash
dump,
and
this
was
picked
up
by
the
first
team.
A
A
So
we
have
this
example
as
well.
Another
another
issue
that
was
opened
by
the
Telemetry
bot
and
the
rgw
team
picked
that
issue
and
again
came
up
with
a
fix
for
that
which
was
also
back
ported
to
octopus,
Pacific
and
Quincy,
and
this
is
everything
here
is
automated,
except
for
the
fix,
of
course,
but
the
boat
will
update
the
the
tracker
with
all
sorts
of
relevant
updates,
such
as
the
affected
versions
and
so
on.
A
Yeah
and
here
here's
another
example
of
an
issue
that
was
first
reported
by
a
user
and
they
reported
the
the
issue
that
happened
in
version
1700
and
the
only
thing
what
they
did
is
to
include
an
example
of
their
of
the
press
dump,
and
this
is
where
they
included.
Oh
sorry,
so
our
Engineers
looked
at
this
issue
here
and
they
searched
it
for
our
Telemetry
dashboard
and
they
they
realized
that
it
started
already
in
Pacific,
so
not
necessarily
in
Quincy.
A
A
So
here
we
have
an
example
of
another
issue:
that's
happened
that
was
reported
in
Telemetry
and
a
user
chimed
in
and
and
said
that
they
also
have
that
and
they
can
provide
us
with
logs
or
any
additional
information,
which
also
is
another
use
case
for
the
sync,
with
our
bug
tracking
system
yeah
and
like
I
mentioned.
Sometimes,
users
identify
themselves,
so
we
can
contact
them
for
more
information
and
we're
very
grateful
for
your
cooperation
on
that.
A
Yeah
and
there's
another
example
here
that
Telemetry
helped
us
to
prioritize
an
issue
that
first
started.
It
was
reported
Downstream
first
and
it
had.
The
original
report
had
crash
information
and
we
could
search
the
dashboard
to
better
understand
where
we
saw
that
in
the
wild
as
well.
I
had
just
one
minute
left,
so
I
think
I
think
we
don't
have
much
time
I.
A
You
can
have,
you
can
see
breakdown
by
minor
versions
as
well
in
in
our
public
dashboard.
So
you
can.
You
can
just
change
it
to
minor
here
and,
for
example,
we
can
we
can
choose
a
Quincy.
Obviously
the
time
window
is
two
years.
So
that's
going
to
show
results
for
two
years.
The
yeah,
it's
actually
a
pretty
big
query.
A
I
can
just
change
it
here.
Maybe
it'll
be
faster.
Sorry,
okay,
it
did
it
returned.
So
so
you
could
take
a
look
and
learn
about
deaf
deployments
in
the
wild
in
our
Telemetry
dashboards.
There
are
some
all
sorts
of
histograms
and
heat
Maps
here
as
well.
You
can
see
some
more
capacity
density
charts
and
have
I
look
at
the
device
dashboard
as
well
I'm
out
of
time,
sorry
about
running
over.