►
From YouTube: Three Pillars, Zero Answers: We Need to Rethink Observability - Ben Sigelman, LightStep
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Three Pillars, Zero Answers: We Need to Rethink Observability - Ben Sigelman, LightStep
Observability has never been more important: the complexity of microservices makes it harder and harder to answer basic questions about system behavior. The conventional wisdom claims that Metrics, Logging and Tracing are “the three pillars” of observability… yet software organizations check these three boxes and are still grasping at straws during emergencies. In this session we’ll illustrate the problem with the three pillars: metrics, logs, and traces are just data – they are the fuel, not the car. We then reframe observability around two fundamental activities: (1) measurement and (2) explanation. “Measurement” must be both robust and precise; “explanation” must present the larger context of distributed transactions and the role that contention plays under peak load. We present both via demos and examples to ground the session in real-world storytelling.
To Learn More: https://sched.co/GrXI
A
My
name
is
Ben
cinnamon
I
am
the
CEO
and
one
of
the
co-founders
at
light
step.
I'm
here
to
talk
about
observability,
I.
Think
a
lot
of
people
here
already
familiar
with
the
term
I'm
a
little
bit
bitter
about
how
many
letters
there
are
it's
just
a
lot
of
typing,
but
we
have
to
live
with
it.
So
I'm
here
talking,
observability
I,
think
there's
a
lot
of
discussion
around
both
what
that
term
means
and
how
we
establish
an
effective
practice.
A
Around
observability
I
spent
my
early
days
at
Google
working
on
what
now
sounds
a
lot
like
observability
I,
built
that
the
dapper
tracing
system
there
and
then
built
Monarch,
which
is
their
high
availability
time
series
monitoring
system
and
at
might
step,
we
work
on
similar
sorts
of
problems.
This
is
very
much
not
intended
to
be
a
product
pitch
for
light
step,
but
is
more
intended
as
a
thought
piece
and
some
a
provocative
way
to
reconsider
how
we
consider
our
own
observability
practice
as
an
industry.
A
A
Approximately
that
we
all
know
observing
microservices
is
difficult,
I,
don't
think
that's
going
to
be
controversial
for
anybody,
who's
deployed
them,
there's
a
sense
that
Google
and
Facebook
solve
this
I
mean
you
can
replace
Google
and
Facebook
with
Netflix
and
Twitter
or
whatever
it
doesn't
matter,
but
there's
this
idea
that
this
problem
has
been
solved
and
that
they
use
metrics
logging
and
distributed
tracing.
So
we
should
I
think
that's
the
essence
of
the
argument,
which
doesn't
mean
it
doesn't
hold
water.
In
my
mind,
it's
not
based
on
a
first
principles,
understanding.
A
It
also
runs
afoul
of
one
of
my
biggest
complaints
about
this
type
of
argument,
which
is
as
someone
who
worked
at
Google
for
a
long
time.
We
should
not
be
emulating
what
they
did.
At
least
we
shouldn't
do
it
blindly.
They
often
were
dealing
with
scaling
issues
that
we
don't
have
to
deal
with
outside
of
Google
they're
processing,
five
billion
RPCs
per
second,
it's
a
lot
of
our
pcs
per
second.
A
So
if
you're
going
to
build
a
system
that
can
observe
five
billion,
our
pcs
per
second
there's,
an
natural
tension
between
scale
and
feature
set,
and
you
will
necessarily
give
up
features
in
order
to
scale
with
that
sort
of
volume,
and
so
the
solutions
that
we
can
use
outside
of
Google
and
Facebook
are
actually
oftentimes
a
lot
more
powerful
from
a
future
standpoint
than
what
we
could
use
at
Google,
because
we
don't
have
to
scale
to
that
order
of
magnitude.
So
this
conventional
list
and,
in
my
mind,
is
totally
busted.
A
It
feels
a
little
bit
dogmatic
to
me
at
times
as
well.
It
is
often
represented
as
the
three
pillars
of
observability
and
their
numerous
blog
posts
and
marketing
posts,
and
so
and
so
forth
about
establishing
a
practice
of
observability
where
we
follow
the
three
pillars,
get
something
in
place
for
each
of
metrics
logging
and
tracing
and
then
voila
we're
supposed
to
be
done.
A
So
this
is
a
what
metrics
look
like.
These
are
a
bunch
of
squiggly
liens
I'm,
not
here
to
lecture
anyone
about
what
these
are.
I
think
people
familiar,
but
you
have
statistics
you
sample
them
frequently,
and
then
you
draw
lines
between
the
samples.
Log
gain
is
also
something
I
think
we
have
a
firm
understanding
of.
A
Of
course,
you
can
do
more
fanciful
things
with
it,
but
at
the
very
lowest
level
you
have
statements
in
code
that
you
write
or
link
they
emit
structured
or
unstructured
data
to
a
log
file
which
is
then
centralized,
and
then
you
can
search
over
those
log
files
to
generate
a
list
of
logs
and
get
some
primitive
statistics
back
as
well
or
in
some
cases
more
sophisticated.
The
statistics,
but
this
is
logging
I,
think
people
are
probably
familiar
with
that
too,
and
tracing
usually
looks
something
like
this
in
people's
minds.
A
This
is
a
screenshot
of
sipc
in
there
are
many
other
tracing
systems
that
have
a
visualization
of
a
distributed
trace.
The
common
elements
are
shared
timeline.
The
fact
that
the
segments
of
the
trace
come
from
different
parts
of
a
distributed
system
and
that
you
can
understand
what
caused
what
which
allows
you
to
essentially
understand
the
critical
path
to
the
trace
so
and
a
quick
show
of
hands
the
metrics
logging
in
tracing
slides
people
familiar
with
these
concepts,
yeah
great,
okay,
cool,
so
we're
aligned
about
that
all
right.
A
So
I
think
when
we
think
of
things
it's
nice,
think
of
them
in
threes.
I've
thought
a
lot
about
why
this
three
pillars
idea
has
caught
on
so
much
I.
Think
part
of
it
is
that
we
like
to
B's
a
lot
as
human
beings.
These
are
three
three
sets
of
three
things:
that
I
really
like
I
tried,
Rice
Krispies
again
for
the
first
time
in
honor
of
this
slide,
I
hadn't
had
them
in
decades,
they're
very
good.
A
Really
good-
and
it's
not
just
the
threes
they're
just
delicious,
but
I-
think
part
of
it
is
well
more
seriously.
Is
that
we,
as
users
of
tools,
have
been
frustrated
by
people
that
are
promising
things
in
terms
of
use
cases
and
value
propositions,
and
we
want
them
to
be
more
factual.
Do
you
or
don't
you
support
metrics,
do
or
don't
you
support
tracing?
A
Do
you
or
don't
you
support
logging,
and
these
are
things
that
we
can
assess
objectively
and
and
use
to
grade
observability
solutions
and-
and
that
does
have
some
appeal,
but
they
all
have
fatal
flaws.
This
is
one
of
my
favorite
cartoons.
In
the
last
year,
we
need
to
be
vulnerable.
I
think
that
it's
difficult
one
day,
I
will
say
it
in
favor
of
what
it
was
like
to
work
at
Google
was
that
there
wasn't
no
one
just
trying
to
sell
anything
to
anyone
else
at
Google,
so
you
would
willingly
represent
your
flaws
as
a
service.
A
You'd
build
something
and
say
well,
this
is
what's
good
about
it.
This
is
what's
not
good
about
it.
It's
very
hard,
especially
at
conferences
like
this,
to
get
people
who
build
systems
to
get
up
on
stage
and
talk
about
what
doesn't
work
about
their
system,
but
we
need
to
do
that.
It's
not
it's.
Nothing
to
feel
ashamed
of.
A
Every
system
has
a
trade-off
and
that's
one
of
the
biggest,
but
if
I
could
get
one
thing
like
propagated
into
this
whole
conference,
it
would
be
that
we
should
all
come
up
here
and
present
what
doesn't
work
as
well
as
what
does
work.
But
we
need
to
talk
about
those
two
and
these
are
still
understood,
but
maybe
not
as
clearly
as
understood
as
what
these
things
look
like.
A
So
so
personal
service,
metrics
and
a
word
that
no
one
knew
in
2015
or
not
very
many
people
knew
anyways,
so
we
have
tags
they're
useful
to
explain
variants
so
here's
time
series
this
is
a
real
example
from
one
of
life's
types:
internal
systems.
Anyone
that's
looking
at
this
can
see
that
there
is
some
kind
of
anomaly
that
happened
around
1:00
p.m.
the
obvious
question
is
what
happened.
So
you
have
a
tag
on
that.
A
You
group,
by
that
tag,
and
you
get
a
new
set
of
lines
where
you
can
now
see
that
this
anomaly
came
from
one
particular
value
for
that
tag,
which
is
not
a
root
cause,
but
is
a
useful
piece
of
information.
We
now
know
that
that
tag
value
is
probably
highly
correlated
with
the
problem,
and
so
we
have
a
lead
that
we
can
go
on,
so
this
is
really
useful.
A
The
issue
is
that
the
number
of
values
for
the
tag
is
expensive
and
and
that's
called
cardinality,
so
the
cardinality
of
the
metric
is
the
number
of
values
for
a
tag.
It
was
previously
something
that
we
learned
about
in
our
College
set,
set
theory
classes
and
then
forgot
the
word
and
now
we're
all
relearning
it
again,
but
this
cardinality
issues
is,
ironically
by
part
of
product
marketing
for
a
lot
of
companies
now
because
of
such
a
significant
pain
point
for
metrics
and
assuming
people
are
familiar
with
this
idea.
A
If
they're
at
this
talk,
logging
also
has
a
fatal
flaw,
which
is
basically
just
the
cost
of
the
data
itself.
I
mean
we
were
already
spending
a
lot
of
logging,
whether
it
is
Splunk
or
elastic
clusters
or
whatever,
when
we
were
just
dealing
with
a
monolith
where
you
had.
You
know
some
constant
number
of
logging
statements
per
transaction
you're
now
seeing
a
transaction
that
is
at
these
different
boxes
or
services.
You
have
regular
transactions
or
traversing
many
service
boundaries.
A
So
if
you
want
to
use
logging
to
understand
these
transactions,
you
have
to
log
about
each
entry
and
probably
exit
point
from
those
services,
so
you're
multiplying
their
transaction
rate,
which
is
a
function
of
your
business,
which
you
know
you
don't
really
that's.
Ok,
as
your
transaction
rate
grows,
you
can
spend
more
money
on
things,
but
you're
now
multiplying
it
by
every
micro
service.
A
So
if
you
build
out
a
micro
service
stock
and
you
go
to
10th
to
potentially
thousands
of
microservices
at
a
large
organization,
your
login
cost
is
scaling
roughly
linearly
with
the
number
of
services
and
that's
a
big
problem,
because
your
value
is
not
scaling
by
orders
of
magnitude.
So
the
law
do
data
volume.
A
When
you
include
the
cost
of
the
networking
and
storage
for
this
data
and
weeks
of
retention
and
that
basically
just
being
way
too
much
money,
I
mean
I
was
at
a
talk
and
I
was
at
a
conference
last
year
and
someone
was
present.
I
won't
say
who
didn't
ask
them
for
permission,
but
he
was
presenting
about
his
company's
logging
infrastructure
at
a
monitoring
conference,
and
this
company
is
like,
in
my
mind,
a
real
honest-to-goodness
thought,
leader
in
the
space
and
they're
doing
lots
of
talks
here
today
and
stuff
like
that
and
I
asked
him.
B
A
A
What
we
would
do
is
say
not
not
for
every
logging
statement,
but
for
the
entire
transaction
will
either
keep
or
discard
the
transaction
and
by
only
keeping
one
in
a
thousand
of
them,
we
were
able
to
significantly
reduce
the
cost
of
getting
this
data
out
of
the
process
and
in
a
regional
storage.
I
thought
that
would
be
enough.
It
turned
out
it
wasn't
we
had
to
do
another
10x
decimation
of
the
data
to
get
it
in
a
truly
global,
centralized
storage.
So
what
has
all
said
and
done?
A
We
had
1
out
of
10
thousand
transactions
restored,
the
other
9999
were
discarded
without
consideration,
so
the
sampling
was
completely
blind
and
there's
no
other
way
for
us
to
accomplish
that,
and
so
this
is
a
major
problem,
because,
if
you're
dealing
with
an
outlier
by
definition,
that
doesn't
happen
very
often
and
you
probably
lost
it.
Similarly,
if
you
want
to
do
some
kind
of
aggregate
analysis
focused
on
a
particular
aspect
of
your
system,
you
probably
don't
have
enough
data
to
use
to
accomplish
that.
Dapper
gets
some
credit
for
doing
some
great
things
from
Google.
A
It
was
really
useful
for
web
search
for
add
surveying
things
like
that
where
we
had
super
super
high
throughput,
but
for
low
for
low
throughput
products
like
Google,
Checkout
or
even
Google
Spreadsheets,
it
was
of
limited
value.
It
did
have
some
value,
it
wasn't
useless,
but
it
had
a
lot
of
problems
because
we
threw
out
so
much
the
data
we
couldn't
make
meaningful
statistical
statements
about
these
systems,
so
those
are
the
three
fatal
flaws.
I
mean
this
is
a
table
kind
of
reviewing
them
for
people,
it's
impossible
to
find
a
solution.
A
That's
just
vanilla,
logs
Tudela,
metrics,
vanilla
traces
that
skills
gracefully
accounts
for
all
the
data
and
is
immune
to
cardinality
issues.
Does
this
resonate
as
people
or
is
anyone
upset
with
me,
okay
anyway,
so
this
is.
This
is
my
critique
of
the
three
pillars
so
I've
talked
to
a
lot
of
companies
as
part
of
my
job,
who've
deployed
the
three
pillars
and
then
they're
still
in
a
world
of
pain,
and
this
is
basically,
why
is
that
you
can
deploy
them
and
that's
not
enough.
A
You
have
to
think
about
something
a
little
bit
deeper
and
we
need
to
be
smarter
as
an
industry
about
how
we
evaluate
observability.
It's
not
just
these
three
checkboxes,
all
right,
so
yeah,
no
silver
bullet,
I.
Think
bullets
are
scary.
I
put
this
slide
in
just
before
the
talk.
I
regret
it,
especially
when
they're
that
big
but
yeah.
This
is
a
silver
bullet.
There's
this
other
problem,
which
is
that
data
and
UI.
So
if
you
search
for
computers,
you
know
here's
computer
and
what
we
get
back
are
pictures
of
monitors.
A
A
So
again
we
can.
We
can
collect
all
of
them,
but
if
we
don't
do
something
with
it
and
we're
not
intelligent
about
the
way
we
consume
the
data,
it's
not
that
useful.
So
this
is
my
segue
to
talk
about
the
second
half
of
my
talk,
which
is
the
scorecard
for
observability,
so
I.
Basically,
you
know
I
found
a
lot
about
this
and
I'm
trying
to
get
up.get
got
something
up
here.
A
That's
not
a
bunch
of
check
boxes,
but
also
isn't
something
that
empty
product
marketing
can
satisfy
something
that
we
can
actually
look
at
to
evaluate
observability
I'm,
going
to
write
a
much
longer
form
version
of
this
as
a
blog
post.
The
next
couple
of
weeks,
which
I'll
publish
and
I
don't
know
if
anyone's
gonna
read
it,
but
it'll
go
into
more
detail
than
I
have
time
for,
and
the
remaining
15
minutes
of
this
talk.
A
But
I
would
love
critiques
from
anyone
in
this
audience
that
you
want
to
reach
out
to
me
separately
or
after
the
talk,
because
I'm
still
refining
these
ideas,
but
but
these
are
basically
the
two
goal
that
we
should
think
about
goals
and
activities.
So
two
main
goals
are
really.
If
we're
talking
about
an
individual
service,
our
goals
are
related
to
how
our
services
perform
in
the
eyes
of
their
consumers.
So
everything
in
my
mind,
takes
a
service
centric
view.
A
If
you're
talking
about
a
devops
kind
of
world,
and
then
we
should
also
think
about
the
activities
that
we
take
as
operators
to
accomplish
those
goals
or
to
further
those
goals,
so
I
divide
observability
into
these
two
pieces.
In
order
to
make
sense
of
this,
we
have
to
talk
about
slis
and
people
are
familiar
with
the
Google
SRA
book.
They
talked
about
sli
SLO
SLA
is
basically
SLI.
Is
a
service
level
indicator?
It's
an
indicator
of
the
health
of
a
service
that
a
services
consumers
would
care
about.
A
So
that's
to
differentiate
from
something
like
like,
for
instance,
if
you
had
a
Kafka
queue,
an
SLI,
maybe
how
long
it
takes
a
message
to
get
into
and
out
of
that
queue
for
a
particular
topic
that
could
be
a
good
SLI
for
Kafka.
You
would
not
want
to
look
at
the
CPU
load
or
the
number
of
nodes
you're
running
as
an
SLI.
Those
are
details
now
Kafka's
is
run.
The
key
thing
here
is
that
most
services
only
have
a
small
number
of
SLI.
A
Is
that
really
matter,
and
that
really
need
to
be
measured,
which
is
a
good
thing.
So
it's
kind
of
like,
if
you
can't
explain
which
SLI
are
going
to
move
for
particular
optimization
you're,
making
you
probably
shouldn't
be
making
it
or
you're
just
guessing
so
slis
are,
in
my
mind
the
first
step
to
properly
running
high
performance
services,
and
there
are
two
basic
goals:
getting
back
to
the
goals
and
activities
you
either
want
to
gradually
and
improve
this
life.
So
this
looks
like
you
know.
A
European
is
saying
that
your
product
is
too
slow,
so
you're
going
to
spend
a
couple
of
quarters,
making
a
series
of
optimizations
and
refactoring
services
in
order
to
make
the
SLI
better
as
a
baseline
and
then
the
other
version
is
that
your
SOI
was
like
this
for
a
while
and
then
suddenly
it's
way
out
of
alignment,
and
you
need
to
fix
it
as
soon
as
you
possibly
can.
This
almost
always
takes
the
form
of
identifying
something
that
you
need
to
undo
like
a
data
push
or
a
new
service
release.
A
It
might
be
someone
else's
service,
but
these
are
very
different
things
and
we
talked
about
observability
tooling,
without
identifying
which
goal
we're
trying
to
move
so
yeah.
This
one
takes
days
weeks
months.
This
one
takes
like
do
it
now.
You
know
this
is
the
one
that
wakes
you
up,
they're
both
super
important
bowls,
but
we
should
think
about
which
tools
we're
using
for
which-
and
if
you
aren't
thinking
about
that,
you're-
probably
going
to
well
you're
more
likely
to
choose
the
wrong
tool.
A
The
two
activities
that
were
pursuing
while
we're
inserting
you
know
in
service
those
goals,
in
my
mind,
are
detection
and
refinement.
These
again
are
things
that
we
can
reason
about
and
discuss
in
isolation,
which
makes
them
effective
ways
to
evaluate
observability
solutions.
Detection
is
about
reifying,
SOS
and
specific
numbers,
so
you
know
p99
or
the
throughput
or
they
error
ratio.
Those
sorts
of
things
refinement
is
about
reducing
the
search
space
and
I'll
talk
more
about
that
in
a
minute.
A
But
this
is
another
thing
where
I
think
there's
some
false
hopes
around
what
we
can
expect
from
our
tools
and
I'll
get
to
that
in
a
minute,
so
for
detection
I
have
to
have
a
quick
interlude
about
staff's
frequency.
This
is
some.
These
are
photographs
of
exactly
the
same
data
from
an
internal
system.
The
only
difference
is
the
smoothing
interval
that
we
use
to
compute
these
percentiles,
so
these
are
I,
think
p99
of
some
signal.
A
You
see
if
you're
able
to
get
down
to
one
second
resolution
or
five
second
resolution
you'll
be
able
to
detect
patterns
of
failure
that
you
won't
see
if
you
smooth
things
out
and
this
bottom
right
graph,
it's
unclear
to
me
whether
or
not
there's
a
steady-state
regression
or
whether
it's
intermittent.
If
you
look
at
this
one,
it's
much
more
obvious
that
something
intermittent
is
happening,
which
is
the
case
in
this
particular
incident,
and
that's
actually
an
important
clue
so
I'm
going
to
talk
about
the
next
slide,
but
I
wanted
a
visual
for
it.
A
So
this
is
my
scorecard
for
detection.
If
you
buy
my
argument
that
observability
is
about
detection
and
refinement
so
for
detection,
we
need
to
understand
the
cost
of
cardinality
I
think
there
are
a
lot
of
people
who
make
card
me
out
the
cardinality
out
to
be
something
where
we
can't
pay
a
dollar
for
it.
That's
not
true.
You
do
need
to
pay,
for
it.
Just
know
how
much
you're
paying
for
it
understand
that
get
it
written
down,
make
sure
you're
thinking
about
it.
A
As
you
structure
your
metrics
strategy
and
inform
your
developers,
and
you
also
need
to
understand
what
stacked
elements
you
can
actually
measure,
can
you
go
way
up
into
mobile
and
web
with
the
same
set
of
the
same
solution
that
you're
measuring
your
services?
Because,
if
you're
trying
to
improve
ideas,
your
latency,
you
really
ought
to.
Similarly,
can
you
go
below
your
application
code
into
your
open
source
dependencies
and
manage
services
to
get
a
sense
of
how
failures
that
that
layer
are
propagating
up
into
application
layer
issues?
So
specificity
is
about
understanding,
cardinality
and
stack
support.
A
Fidelity
is
correct.
Statistics
I'm
putting
escalation
points
here
because
it
blows
my
mind
how
incorrect
p99
is
a
lot
of
solutions
that
people
pay
a
lot
of
money
for
will
take
p99
across
different
hosts
or
different.
You
know
shards
of
a
monitoring
service
and
then
average
the
p99
together,
which,
statistically
as
meaningless,
completely
meaningless
but
I
swear
it's
it's
commonplace.
It's
absolutely
commonplace
and
it's
a
difficult
thing.
So
you
compute
that
you
have
compute
the
p99
at
the
very
end.
A
So
if
you
haven't
been
storing
histograms
or
or
meaningful
summaries,
you
can't
really
compute
the
p99
globally,
which
is
what
you
actually
wanted.
So
be
careful
about
statistics.
If
you're
measuring
p99,
that's
smart,
but
make
sure
it's
actually
p99
because
a
lot
of
times
it's
not.
You
also
want
high
staffs
frequency,
which
is
that
last
slide.
A
I
think
that
for
detection
to
work
really
well,
you
want
to
be
able
to
detect
the
difference
between
intermittent
and
steady-state
failures,
and
that's
only
easy
to
do
if
you
have
high
frequency
data
and
then,
of
course,
freshness
which
is
sort
of
obvious.
But
I
have
seen
people
design,
really
cool
observability
solutions
around.
You
know
spark
and
things
like
that.
A
So
why
refinement
instead
of
something
else?
So
my
observation.
This
is,
of
course,
completely
scientific,
this
graph
that
doesn't
have
any
units
on
it
but,
as
you
add
microservices
the
number
of
things,
your
end
users,
care
about,
doesn't
change,
and
so
as
engineers
or
as
developers.
We
have
to
accept
the
fact
that
we
have
created
a
monster
as
microservices,
because
our
bosses
are
more
more
importantly,
our
end
users
and
customers.
A
To
introduce
all
sorts
of
ways
for
things
to
break,
and
many
of
them
are
pretty
elaborate
and
hard
to
predict,
and
so
I
think
that,
just
even
even
if
you
could
build
a
dashboard
that
had
every
possible
failure
mode
on
it,
we
don't
have
time
as
human
beings
to
sort
through
that
list.
I've
seen
people
building
dashboards
and
even
advertising
dashboards
that
have
this
dizzying
dizzying,
just
pages
and
pages
of
spark
lines,
and
things
like
that.
It
is
sort
of
impressive
from
just
a
consolidation
standpoint
that
we
can
do
that.
A
But
it's
totally
in
actionable,
because
if
there's
a
failure
in
a
microservices
environment,
it
will
crop
up
in
many
services
concurrently,
so
we'll
be
dealing
with
many
lines
that
are
spiking
at
the
same
time
and
you're
left
with
this
kind
of
guessing
Chuck
linear
search
thing,
which
is
really
painful.
So
when
I
think
that
the
crux
of
observability
is
detection
and
then
refinement
of
the
search
space,
we're
not
going
to
find
observability
solutions
that
reliably
tell
us
the
answer
and
fix
problems
for
us
if
they
do.
A
I'm
really
scared
because
we're
off
to
get
different
jobs
we'll
be
out
of
work,
but
thankfully
our
job
security
is
intact.
I
think
we're
still
going
to
need
human
beings
to
actually
make
decisions,
but
we
can
use
tools
to
reduce
the
search
space
and
to
refine
it
down
to
a
set
of
hypotheses
that
are
actually
credible,
and
so
in
observability
solution
is
mainly
about
eliminating
hypotheses
before
you
investigate
them.
That
should
be
the
goal
of
refinement.
A
So
right
must
reduce
the
search
space
so
yeah
we
discover
variants
in
our
systems,
usually
initially
through
an
SLI,
that's
out
of
whack.
We
explain
it
using
observability
tools.
That
explanation
is
usually
in
the
form
of
new
variants,
which
we
then
have
to
explain
again,
and
we
keep
on
doing
this
until
we
find
something
that
we
can
fix,
either
by
adding
cachet
and
worse
or
getting
rid
of
serialization
we're
rolling
back
for
Elise
we're
looking
for
variants
that
could
be
explained
by
a
problem.
A
We
actually
know
how
to
solve
and
what
I've
seen
a
lot
in
organizations
is
that
I
can't
seen
organizations
brand
name,
companies
that
have
really
smart
people.
If
you're
working
really
hard
and
care
a
lot
about
outcome
of
the
company,
where
they
don't
have
good
tools,
then
they
do
have
good
tools,
because
they,
you
know,
use
something
new
and
suddenly
they
fix
problems
that
improve
SLI
is
by
50%
or
70%,
and
they
do
this
in
order
of
days.
A
Just
because
I
think
people
have
lost
confidence
that
they'll
be
able
to
find
the
problem,
there's
so
many
possible
areas
to
look
that
they
don't
even
feel
like
they're,
going
to
be
able
to
enumerate
the
possible
issues,
and
so
they
find
huge
wins
once
they
have
the
right
to
lean,
because
they
aren't
contending
with
this
list
of
a
hundred
thousand
possible
problems.
So
I
also
want
to
talk
a
bit
about
histograms.
A
This
is
I
saw
this
wasn't
a
product
picture,
I
mean
it
I,
just
don't
know
where
else
to
get
this,
so
this
is
a
graph
histogram
showing
out
latency
for
particular
requests.
This
is
slowness,
basically,
the
further
right.
You
go.
The
slower
you
go,
the
further
up,
you
go
the
more
data
there
is
so
it's
just
a
histogram
p99
in
this
diagram
is
right
here.
A
The
refinement
scorecard
is
thus
cardinality
is
again
really
important,
arguably
even
more
important
than
in
detection,
and
that
we
need
to
understand
how
many
dollars
are
spending
for
more
tag
values.
We
do
need
I
think
we
still
need
metric
solutions
to
be
clear,
but
we
should
just
be
really
careful
about
what
we're
gonna
pay
for
them
with
cardinality,
because
if
you're
using
metrics
to
do
the
refinement
process,
cardinality
is
literally
your
only
tool
to
accomplish
that
goal.
A
So
you
have
to
be
careful
about
that
robust
statistics,
so
that's
histograms
in
my
mind,
there
might
be
other
ways
to
do
it
as
well,
but
I
think
going
beyond
percentile
measurements
is
pretty
important
for
refinement.
It
certainly
can
help
you,
you
know,
reduce
this
entire
set
of
hypotheses,
which
this
is
a
log
scale
thing,
but
you
know
we're
only
looking
at
the
top
point
of
5%
of
requests
here
and
and
that's
the
nice
thing
to
be
able
to
refine
and
zero
everything
else.
A
We
also
need
to
think
about
how
long
data
is
retained,
for
this
is
mainly
in
my
mind
in
order
to
explain
what's
normal
and
abnormal
another
common
practice
in
a
micro
service
architecture
for
observability
is
that
you
find
an
issue.
It
looks
bad
and
you
don't
know
if
it's
always
been
like
this,
or
if
this
is
just
normal.
A
It
happens
to
me
all
the
time
Sisto
is
and
if
you
don't
have
a
system
that
can
show
you
exactly
what
you're
looking
at
you
know
two
hours
ago
or
a
week
ago,
it's
really
hard
to
answer
that.
What's
normal
question
and
a
lot
of
time
series
data
is
really
just
trying
to
answer
that
question.
I
would
argue,
there
are
other
ways
to
do
it
than
showing
the
squiggly
line
and
I
think
we
should
actually
explore
this
as
an
industry,
but
you
need
to
understand.
A
What's
normal
and
again,
I'll
Harper
got
this
correct
statistics.
We
have
a
lot
of
incorrect
statistics
out
there
and
that's
dangerous
and
then
the
last
thing
I'll
say
is
about
suppressing
messengers
and
what
I
mean
by
that?
This
is
a
diagram,
that's
simplified,
but
if
your
service
up
here
depends
on
an
intermediary
service
which
depends
on
some
tertiary
service
and
the
tertiary
service
fails,
you
don't
want
to
blame
this
intermediary
service
for
that
failure,
but
it
will.
A
It
will
represent
the
failure
and
its
own
performance,
and
so
you
often
have
incidents
where
you'll
be
woken
up
or
some
will
be
woken
up,
because
there
are
errors
or
latency
down
the
stack.
You
shouldn't
waste
any
time
on
the
intermediary
messengers
of
that
failure.
In
an
observability
system.
That's
able
to
understand
the
global
picture
should
be
able
to
suppress
those
hypotheses
for
you
and
refine
that
search
to
focus
only
on
the
service.
That's
actually
having
a
problem
and
in
turn
you
can
hopefully
figure
out
what's
causing
load
in
that
service
and
refine.
A
The
hypotheses
of
weird
load
is
coming
from
above
it
in
the
stack
as
well.
So
that's
my
scorecard
for
refinement.
Okay.
So
how
am
I
doing
pretty
good,
so
wrapping
up
first
into
my
prospectus?
So
these
are
the
fun
game
to
play
the
designer
observability
problems,
so
durability
systems.
You
can
have
high
to
put
high
powered
quality,
a
lengthy
attention
window
and
a
lack
of
sampling
so
choose
what
you
want
actually
know.
A
I
see
a
lot
of
people
advertising
some
really
cool
Shi
T,
but
it's
because
it's
not
scaling
or
they
advertise
something-
that's
really
compelling,
but
it
would
break
if
you
have
lots
of
tag,
values,
etc,
but
that
this
is
the
essence
of
the
problem
and
why
there's
still
so
much
activity
in
this
space
is
that
no
one
has.
This
is
a
law
of
nature
that
we're
dealing
with
yeah
I
call
observability
whack-a-mole.
So
you
can.
You
can
kind
of
move
things
around,
but
you're
still
gonna
have
to
deal
with
these
problems.
A
Somehow
I
talked
about
dapper
earlier
I
do
think
the
tracing
data
is
probably
the
best
data
source
we
have
in
that
it
represents
it's
the
superset
of
law
and
data
for
transactions
anyway,
and
it
also
understands
context
which
is
super
valuable
for
suppression
of
hypotheses.
So
tracing
data
has
the
raw
material
to
suppress
hypotheses
about
parts
of
the
system
that
are
reflecting
failures,
but
aren't
actually
the
cause
of
failures,
so
I
think
tracing
is
really
useful.
A
A
There
certainly
other
approaches
as
well
that
don't
do
this,
but
I
really
like
this
approach
and
here's
my
scorecard
in
essence
just
summarized
down
to
one
slide,
but
this
is
just
you
know
summarizing
what
we
talked
about
already,
so
I
tried
to
keep
this
half
an
hour
and
I
did
it's
5
o'clock
and
allegedly
I
have
35
minutes
for
this
slot.
So
I
wanted
to
leave
time
for
questions
so
with
that
I.
Thank
everyone
for
attending
my
talk
and
I
open
it
up
to
questions
which
I
can
repeat.
A
Yes,
so
the
question
was:
what
do
we
do?
How
do
we
actually
expect
our
observability
system
to
detect
these
messenger
systems
and
suppress
issues
with
that?
I?
Don't
know
how
to
do
that
without
tracing
data
being
involved
in
some
capacity,
but
I
think
what
we
I'd
like
to
get
to
a
world
where
we
don't
look
at
traces,
anymore,
I.
Think
the
tracing
data
is
the
fuel,
not
the
car.
The
fact
that
we
need
traces
doesn't
mean
we
should
be
looking
at
them
very
carefully.
A
So
I
think
the
idea
is
that
you
have
enough
trace
in
data
that
you
can
automatically
understand
that
the
critical
path
for
the
lion's
share
of
your
p99
requests
are
bottoming
out
in
service
X
way
down
the
stack-
and
you
just
present
that
as
the
hypothesis
to
investigate
and
you
might
even
exonerate
the
services
on
the
way
and
say
these
are
not
your
problem.
You
know
these
are
not
the
droids
you're
looking
for
focus
on
service
X.
So
that's
how
I
think
if
it
happening
so
tracing
tracing
data
would
be
the
raw
data.
A
A
So
the
question
is
how
to
aggregate
99%
out
correctly.
Yes,
so
for
some
with
some
technology,
I
don't
actually
know
how
to
do
it,
because
I
think
they've
made
some
bad
design
decisions,
but
I
think
the
best
way
to
do
it
is
to
keep
track
of
pretty
fine-grained,
Bend,
histograms
or
HDR
histograms,
or
something
like
that.
A
Those
things
have
an
algebra
where
they're
distributive
and
human
commutative,
so
you
can
take
them
globally,
merge
them
up
to
the
very
top,
and
once
you
have
one
aggregate
histogram
with
a
lot
of
fidelity,
you
can
take
a
p99
with
a
bounded
error
bar
on
it.
So
you
can
say
this
is
guaranteed
to
be
within
five
percent
of
the
actual
p99.
If
you
kind
of
do
the
math
out,
but
I
think
it's
about
maintaining
histograms
throughout
your
data
processing
pipeline.