►
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
No-Nonsense Observability Improvement - Cory Watson, SignalFx
Observability has gone from a thing you read about on Twitter or Medium thinkpieces to something your organization “has”. Maybe you’ve got a few new observability tools deployed. How is that working out for you? Regardless of your adoption level – from logs on local boxes up to the highest cardinality traces and feature analysis – at the end of the day these are tools, not magic spells. How do you teach, train, use, evangelize, and measure the impact they have in your organization? Cory has been a part of solo and large observability teams, in-house and vendor, and worked with dozens of companies. In this session he’ll explain some no-nonsense, tool agnostic methods for wringing more value out of what you have, identifying when to use new tools, how to handle migrations, how to measure value, and how to deal with “why does this cost so much?”
https://sched.co/Uabb
A
A
It
says
so
now
you
have
observability
at
your
company.
So,
what's
day
2
day
3
day
4
looks
like,
and
so
this
is
mostly
gonna
be
a
talk
about
how
to
get
more
value
out
of
observability,
how
to
talk
to
leadership.
How
to
talk
to
your
peers,
how
to
talk
to
teammates,
because
at
the
end
of
the
day,
I
believe
this
to
be
more
of
a
social
problem
than
it
is
a
technical
problem,
but
unfortunately
we
tend
to
look
at
it
like
it's
a
technical
problem.
So
why?
Should
you
listen
to
me?
A
That's
me:
I
am
Corey.
I
worked
for
Splunk
I
am
a
Technical
Evangelist,
but
previously
I
worked
for
signal
effects.
We
were
acquired
by
Splunk
a
few
months
back
and
my
role
at
signal
effects
was
in
the
I,
was
a
technical
director
in
the
office
of
the
CTO,
and
so
I
basically
got
to
have
a
lot
of
pre-sales
conversations.
A
A
lot
of
post
sales
conversations,
a
lot
of
quarries
done
a
lot
of
observability
before
let's
talk
to
him
and
figure
out
how
to
deploy
this
a
lot
of
product
conversations
about
what
products
should
do,
because
leading
up
to
this
I
previously
had
worked
at
strike.
Where
I
was
the
principal
engineer
and
where
I
created
and
led
the
observability
team
I
forgot
to
start
a
timer,
so
we're
gonna
do
that
now
reset
starting
and
there
I
bought
a
lot
of
these
tools.
A
I
was
a
customer,
not
a
vendor,
which
is
a
whole
different
kettle
of
fish
right
and
then
before
that,
as
you
can
see
up
here,
I
also
worked
at
Twitter,
where
I
was
an
SRE
and
then
later
was
an
engineering
manager
for
the
observability
team,
and
there
we
built
our
own
stuff.
So
if
you're
thinking
about
building
buying
running
open
source,
doing
any
of
that
stuff,
I
have
done
all
of
those
things
at
a
whole.
A
Bunch
of
different
companies
and
I
have
a
lot
of
school
show
for
it,
and
I've
talked
to
a
lot
of,
maybe
even
some
of
you
about
your
choices
about
doing
one
of
these
things.
If
you
want
to
find
out
more
about
that
kind
of
thing,
occluded
by
someone's
head
at
the
bottom
left,
you
can
find
it's
okay,
yeah
leaned
down
there,
you
go
there'll,
be
later
slides
for
that,
just
frame
it
for
me
like
put
your
hands
around
it
great
audience,
participation
all
right!
You
can
find
more
about
me.
A
There
got
a
roll
with
it,
no
lot
of
demos,
but
this
is
the
version
I
get.
So
today's
goal
is
to
find
useful
vendor
agnostic
ways
to
improve
your
observability
stack.
If
you're
expecting
me
to
tell
you
how
to
name
your
metrics
or
to
tell
you
why
tracing
is
the
coolest
thing
that
you
should
be
poking
at
or
any
of
that
kind
of
stuff.
It's
not
we're
gonna
be
talking
about
today.
A
We're
mostly
gonna,
be
talking
about
the
social
aspects
of
the
work
that
we're
doing
and
how
we
can
try
to
make
ourselves
more
prepared
for
the
conversations
that
we
need
to
have
inside
the
organization.
So
the
first
part
of
this
is
sort
of
about
sort
of
why
we
observe,
if
you've
ever
gone
to
any
talks
on
observability
if
you're
interested
in
the
space.
You
know
that
this
is
the
part
of
the
talk
where
someone
defines
observability
I'm,
not
gonna.
A
Do
that
today
I'm
instead
going
to
talk
to
you
about
the
conversation
that
you're
going
to
have
with
the
organization
so
like.
If
someone
asks
you,
why
do
we
need
to
buy
one
of
these
tools
or
something
like
that?
So
this
is
my
favorite
graphic
for
articulating
why
we
have
observability
in
the
stack.
Some
people
often
ask
me
like
why
what
I
thought
we
had
monitoring?
Wasn't
that
a
thing
didn't
we
already
have
monitoring?
How
is
it
different?
Well,
monitoring
is
because
you
traditionally
know
what
you're
paying
attention
to
I.
A
A
The
problem
is,
there's
a
lot
of
stuff
in
our
stacks
that
nobody
really
ever
anticipated,
and
the
point
of
observability
is
to
prepare
yourself
for
the
inevitable
possibility
that
stuff
is
going
to
break
things
are
going
to
fail
and
you're
not
going
to
have
anticipated
what
it
was
I'm
sure
many
of
you
have
been
in
a
situation
where
you
got
called
that
you
got
paged
and
the
thing
that
broke
was
something
you
would
never
thought
of
happening
before.
Observability
does
not
make
those
things
transparent
to
you.
A
It
is
not
magic,
it's
not
a
silver
bullet,
but
it
does
prepare
you
for
the
fact
that
when
things
are
broken,
you
are
more
likely
to
find
some
of
those
bread
crumbs
or
something
like
that.
That
will
get
you
into
the
right
place.
So
the
unanticipated
things
that
we're
observability
tries
to
help
I
like
to
call
this
for
wind
gets
weird,
because
that's
typically
what
observability
is
more
valuable
for
not
for
the
things
that
you
already
expected
to
happen.
A
That's
just
monitoring,
so
the
next
reason
that
we
need
something
like
this
is
that
the
systems
that
we're
building
just
keep
getting
more
complicated
systems
are
complex.
Not
only
are
they
complex
but
they're,
complex
and
actually
sort
of
academically
interesting
ways.
Systems
are
nonlinear,
meaning
that
the
inputs
that
you
give
them
can
result
in
varied
and
wildly
out
of
proportion.
A
Outputs,
they're
chaotic,
because
the
inputs
they're
very
sensitive
to
a
slightly
different
input
can
result
in
a
really
weird
output
and
then,
lastly,
they're
emergent,
because
these
weird
human
things
keep
getting
involved
in
these
systems
and
setting
up
weird
little
micro
services
that
interact
with
other
little
micro
services
and
before
you
know
it,
you've
got
this
big
complicated
system
that
seems
almost
alive
by
itself
and
trying
to
deal
with
this
stuff
is
pretty
difficult
for
humans.
We
have
to
learn
how
to
deal
with
it
and
the
complexity
that
it
entails.
A
So
another
graphic,
many
of
you
responsible,
probably
for
some
sort
of
service
that
you've
written.
Maybe
it's
request
based,
and
you
know
that
a
request
based
service
can
usually
be
in
a
couple
of
simply
simplistic
states.
So
we've
got
like
waiting
for
a
response
like
listening,
we've
got
processing
we've
got
responding
and
maybe
we've
gotten
got
shutting
down.
So
that's
four
states
that
your
system
can
be
in
and
humans
are
pretty
magical.
A
But
the
complexity
to
this
is
that
this
little
robot
that
starts
off
so
simple
actually
becomes
fairly
complicated
and
before
you
know
it,
the
robot
needs
to
understand
that
every
single
state
that
your
system
can
be
in
and
so
there
happens
to
be
which
I
am
probably
going
to
butcher.
But
there
is
from
cybernetics
the
thing
called
the
law
of
requisite
variety.
That
says
that
for
a
controller
to
properly
control
something
it
needs
to
understand
more
states
than
the
system,
it's
monitoring.
A
So
this
is
a
talk
about
observability
I
think
so
we
should
probably
bring
it
back
to
that.
So
why
do
we
need
observability
for
this?
And
what
can
we
do
with
them?
Sort
of
ability
to
make
it
better?
I
don't
want
to
go
to
that
slide
yet
so,
instead
I'm
going
to
stop
and
say
we
need
this
observability
stuff,
because
it
is
effectively
a
counter
to
complexity.
A
This
is
the
one
slide
that
doesn't
have
any
other
slides
that
go
with
it,
because
I
don't
know
how
to
tell
you
to
do
this.
Everybody's
got
a
different
way
of
doing
it,
but
the
number
one
thing
that
I
would
encourage
you
to
do
is
to
promote
the
work
that
you
and
your
observability
stack
is
doing.
This
is
primarily
aimed
at
people
by
the
way
who
are
on
an
observability
team
or
want
to
encourage
the
use
of
observability
in
their
organization.
A
What
the
winds
are
what
the
games
are,
then,
how
is
anybody
supposed
to
know
why
you're
investing
in
it
right
and
we're
gonna
come
back
to
this
and
talk
to
leadership
in
a
minute
we're
a
few
more
slides,
long,
so
oops
that
went
too
fast.
So
now,
we've
got
five
things
that
we're
gonna
talk
about
each
one
of
these
do
have
real
slides
that
come
after
them,
so
number
two
teach
basic
skills.
How
many
of
you
write
job
descriptions
for
people
that
you
hire
at
your
companies
or
have
contributed
to
them?
A
How
many
of
you
put
a
bullet
point
in
it,
for
you
need
to
be
able
to
create
digest
or
do
anything
with
your
observability
stack,
a
lot
fewer
hands
went
up.
We
don't
really
teach
this
right.
Like
I
learned,
it
all
just
kind
of
on-the-job,
I
learned
it
because
I
couldn't
remember
how
anything
worked
and
I
felt
better,
knowing
that
I
had
things
keeping
track
of
it.
A
For
me,
the
problem
to
this
is
that
your
first
interaction
with
many
of
these
tools-
let's
say
I,
join
a
company
and
there
for
a
few
weeks,
got
my
laptop
provision
that
rolled
code
my
first
day
and
then
I
get
into
an
on-call
rotation
and
suddenly
I'm
thrown
into
a
problem,
and
if
your
first
interaction
with
an
observability
tool
is
at
3
o'clock
in
the
morning
under
duress.
Do
you
think
you're
gonna
be
particularly
happy
with
that
observability
tool
like
it's
a
really
crappy
way
to
learn
so
to
help
people
with
this?
A
We
need
to
actually
teach
them
something
up
front
now.
I
want
to
preface
this
by
saying:
don't
expect
people
to
learn
your
entire
observability
stack
or
really
even
retain
almost
anything
that
you
tell
them
in
this
in
the
spin-up
class?
Or
this
you
know,
however,
you
do
it
it's
your
company,
but
that's
because
humans
learn
by
doing
things
not
by
watching
them.
You
have
to
actually
sit
down
with
them.
You
know
the
mouse
and
hand
or
your
keyboard.
Whatever
you
doing,
you
actually
actually
have
to
make
this
stuff,
but
we
can
help
them
get
started.
A
In
fact,
most
of
the
education
things
that
your
organization's
have
when
you
bring
people
in,
it's
not
meant
to
get
them
to
exhaustively,
understand
your
entire
company
or
your
entire
stack.
It's
just
to
get
them
to
understand
some
of
those
basic
words
and
some
of
those
things
that
they're
gonna
see
later
so
we
can
teach
them
how
to
navigate
the
tools
and
the
products
and,
what's
out
there,
we
can
teach
them
about
how
to
find
metrics
traces,
logline
spans.
A
Whatever
those
things
are,
we
can
actually
show
them
how
to
do
it
any
of
the
common
frameworks
or
libraries
or
things
that
your
organization
uses
to
make
this
more
easy.
These
are
things
you
can
also
teach
and
I
can
sort
of
run
through
this
whole
list,
but
I'm
just
reading
the
slide
to
you,
the
one
that
I
think
that's
important
at
the
end
is
to
plug
your
way
of
doing
it.
How
are
you
different?
Most
of
us
are
probably
lightly
familiar
with.
Oh
yeah,
we've
got
a
stats
library
or
we
met
spans.
A
This
way,
your
log
lines
pop
out
like
that.
But
what
do
you
do?
That's
special
in
this
mixture
we're
teaching
that
to
the
people
that
are
in
the
class
and
then
the
next
part
is
since
we
don't
learn
by
just
looking
and
being
talked
to.
How
do
we
make
sure
that
we're
practicing
and
there's
something
for
people
to
do
after
they
leave
your
class?
So
the
next
thing
we
have
to
focus
on
is
genuine
practice.
A
The
only
way
you're
gonna
get
better
at
this
stuff
is
by
using
it,
and
so
we
need
to
make
sure
that
people
have
more
things
that
they
can
do
on
sort
of
day,
two
or
day
three,
so
that
they
get
out
into
the
world
and
they
start
using
these
tools
on
a
regular
basis.
They
need
to
using
solve
problems
too
see
what's
going
on
in
the
systems
or
what
have
you?
What
can
they
do
week
over
week
over
week,
because
we
want
this
to
be
something
they're
constantly.
A
Investing
in
this
is
where
the
promotion
of
the
work
becomes
important
too.
If
you're
shipping
cool
new
stuff
in
this,
are
we
telling
people?
Are
we
letting
them
know
hey?
You
need
to
come
check
this
out.
You
need
to
see
these
new
features
that
we've
added
I
like
to
use
martial
arts.
As
an
example,
I
will
also
say
that
I
do
not
know
any
martial
arts,
so
I'm
mostly
making
this
up.
So
if
I
get
it
wrong,
you
can
just
not
tell
me
after
it's
fine,
because
it's
just
an
illustrative
example.
A
But
the
point
is
many:
people
go
into
martial
arts
because
they
have
this.
Like
idea,
I'm
gonna
go
and
I'm
gonna,
be
some.
You
know
in
theory
black
belt
or
something.
But
the
truth
is
that
expertise
is
not
evenly
distributed
in
organizations,
and
you
have
to
expect
that
the
people
who
learn
from
you
and
who
work
in
the
company
you're
gonna,
have
some
people
who
are
black
belts
and
some
people
who
just
know
the
basics
and
there's
nothing
wrong
with
that.
A
But
we
need
to
facilitate
this
sort
of
progression
over
time,
that
there
are
people
that
you're
helping
who
are
getting
that
black
belt,
but
also
people
who
have
the
basics,
and
we
need
to
remember
that
not
everyone
is
going
to
come
in
and
become
a
black
belt
in
your
observability
stack.
They
probably
have
more
interesting
stuff
to
do
so.
Number
three
we've
talked
about
promoting
our
work
and
we've
talked
about
whatever
the
last
one
was
I
forget.
So
now
we
need
to
stretch
our
systems.
A
So
what
do
I
mean
by
stretching
our
systems
improvements
in
capacity?
This
is
a
law
that
I'm
ripping
off
from
somewhere.
It's
paraphrased
heavily,
but
improvements
in
capacity
are
consumed
about
productivity
and
efficiency.
If
a
new
programming
language
comes
out
tomorrow-
and
it
allows
you
to
do
the
same
things
that
you're
doing
today
in
half
the
RAM
and
half
the
CPU
you're-
probably
not
going
to
adopt
it
because
I
hate
to
break
it
to
you,
but
pretty
much.
A
All
of
you
are
using
language
that
there
is
a
language
that
is
twice
as
fast
and
twice
as
efficient,
but
you've
chosen
not
to
because
it's
more
efficient
for
you,
you're
more
productive.
If
you
use
these
other
tools
and
the
same
goes
for
the
systems
that
we
build.
If
tomorrow
someone
on
you're,
like
you
know,
resiliency
staff
make
some
changes
and
adjusting
things.
It
adds
some
new
capacity.
A
You
are
almost
certainly
going
to
consume
it
very
quickly,
with
cool
new
features
that
your
users
asked
for,
and
that's
okay,
but
we
need
to
learn
how
to
do
the
stretching
ourselves
so
optimally.
We
are
not
learning
how
to
break
our
systems
from
our
customers,
one
of
my
pet
peeves
of
observability
and
something
that
I
encourage
people
as
a
vendor
to
not
sell
is
we
are
not
going
to
make
in
go
away,
we're
only
going
to
sort
of
help.
You
understand
the
impact
of
them
and
maybe
help
you
communicate
about
them
better.
A
If
I'm
on
a
bridge
and
the
bridge
collapses,
the
Department
of
Transportation
will
not
know
before
I
do
like
that's
just
physics
and
it's
the
same
situation
here
so
optimally.
We
need
to
find
ways
to
take
the
data
that
we've
got
and
feed
that
back
in
and
make
our
systems
better
by
sort
of
doing
the
stretching
ourselves.
I
always
have
this
visualization
of
you
like
using
shoe
horns
to
stretch
things
out
and
I
was
doing
the
practice
talk
this
morning
and
realized
that
this
made
no
sense
so
I'm,
just
not
gonna
use
that.
A
A
If
they
started
to
see
too
much
impact
on
the
system
by
using
the
observability
tools,
they
would
be
able
to
dial
that
back
or
turn
the
test
completely
off
and
again,
that's
something
that
our
observability
stack
is
doing.
They
can't
get
that
anywhere
else.
Also,
if
there
are
things
that
you
find
in
incidents
like
if
you're
ever
watching
an
incident
review-
and
somebody
says
who
glad
that
didn't
happen.
A
That
is
an
excellent
way
for
you
to
find
something
to
use
for
a
game
day
or
something
else
to
sort
of
stretch
out
your
systems
so
know
before
we
got
to
measure
usage.
This
is
the
big
one
that
finance
wants
to
talk
about,
because
at
the
end
of
the
day,
these
tools,
and
by
the
way,
you
may
think,
I'm
talking
about
this
from
the
perspective
of
a
vendor
they're
like
what
the
vendor
costs,
you
may
not
be
thinking
about
it
this
way,
but
your
optics
and
your
capex
is
still
money
getting
spent.
A
So,
even
if
you
build
this
in-house,
if
you
build
something
from
open
source,
all
these
things
are
still
gonna
cost
you
money,
you're,
still
paying
full-time
engineering,
salaries
or
you're
still
reserving
instances
or
whatever
you're
doing
so.
I
can
tell
you
right
now
that
observability
will
probably
be
one
of
the
most
expensive
parts
of
your
company,
whether
it's
something
you
built
or
something
that
you
bought
from
other
people
at
every
company,
I
have
ever
worked.
It
has
been
the
most
expensive
thing
in
the
stack
well
other
than
like
you
know
the
company.
A
So
what
you
need
to
do
is
start
measuring
usage
upfront.
Even
if
you
are
using
something
open-source,
you
need
to
know
where
the
money
is
being
spent.
So
the
first
things
we
need
to
start
doing
is
start
grouping
things.
What
services
did
information
come
from
logs?
Metrics
fans
doesn't
matter
what
team
was
it
associated
with,
because
then
we
can
start
thinking
about
charge
back
and
showing
people
what
they're,
using
especially
relative
to
their
peers.
A
We
had
a
big
initiative
at
stripe
where
we
actually
started
measuring
keeping
up
with
what
people
are
using
and
we
were
able
to
cut
metrics
50%
year-over-year
just
by
recognizing
what
people
were
doing
and
talking
to
them.
We
didn't
even
take
anything
away
from
them.
They
were
using.
We
had
just
never
actually
looked
at
what
was
being
emitted.
I.
A
Remember
it
at
Twitter,
ninety-nine
point
eight
percent
of
metrics
that
were
written,
never
got
read
back
out
again
and
that
sort
of
information
is
really
hard
for
finance
to
understand
because
they
don't
know
why
it's
sitting
there.
So
you
can
also
group
by
region
by
geography
by
cloud.
You
know
where
you're
assigning
costs.
You
can
also
probably
find
a
friend
in
finance
who
knows
had
a
lot
of
FS
in
it.
Who
knows
a
thing
or
two
about
money
and
asked
them
to
come
over
and
help
you
with
this
stuff.
A
They
love
to
teach
that
kind
of
thing.
Lastly,
comparing
what
you
are
spending
to
how
your
organization
is
growing
is
another
great
metric.
So
how
much
are
you
spending
on
infrastructure?
Is
that
going
up?
Is
it
getting
bigger
and
having
more
of
them?
Well,
that's
going
to
increase
your
complexity.
You
will
spend
a
proportionate
amount
more
because
of
that
I'm,
not
sure
what
that
percentage
is
it's
different
for
every
org,
how
about
dependencies,
the
more
interwoven
your
systems
get
the
more
things
that
are
hooked
together.
A
The
more
flows
and
connections
you
make
between
things,
the
more
envoy
metrics
come
out
of
them
for
as
an
example
that
sort
of
stuff
like
it's
going
to
cost
you
more
because
you
need
that
visibility.
You
want
to
be
able
to
see
those
things
when
things
get
broken
there,
you
want
that
visibility
so
be
sure
to
feed
that
back
to
your
team,
your
leadership
or
what
have
you?
Because,
as
you
are
increasing
complexity,
that's
going
to
cost
you
more
as
an
observability
team.
A
Also,
you
can
compare
this
I'm,
not
sure
how
that
one
got
so
big.
You
can
compare
this
to
changes
over
time
if
you're,
an
organization
who
only
shifts
code
once
a
week
and
suddenly
you
start
shipping
code
five
times
a
day.
You
are
probably
going
to
end
up
generating
more
cost
in
terms
of
observability,
because
more
changes
to
the
system
means
more
output
from
it
and
means
more
things
that
people
are
going
to
want
to
investigate
because
change
inevitably
leads
to
problems.
A
So
we
also
need
to
measure
not
just
how
much
people
are
sending
in,
but
also
how
much
people
are
pulling
back
out.
So
a
good
example
to
go
back
to
monitoring
that
we
talked
about
earlier,
if
someone's
monitoring
a
metric,
that's
a
pretty
good
signal
that
that
metrics
worth
something
and
that
it's
valuable,
you
may
also
be
able
to
pay
attention
to
incident
investigations.
So
observability
is
always
very
closely
associated
with
keeping
up
with
incidents,
but
you
also
need
to
be
looking
at.
What
did
people
do
during
these
incidents?
What
were
they
using?
A
Were
there
a
series
of
tools
that
we
use
more
than
others?
This
is
also
some
information
that
you
don't
just
use
to
defend
your
position
of
like
we
have
a
big
observability
stack.
It's
also
how
you
can
make
internal
decisions
when
you
know
you
need
to
shrink
or
grow
certain
aspects,
maybe
nobody's
using
that
logging
platform
anymore.
Maybe
tracing
didn't
pay
off
the
way
that
you
expected
it
to
that's.
A
Ok,
but
you're
not
going
to
know
unless
you're
paying
attention
to
how
and
when
people
use
the
tools
game
days,
chaos
testing
all
that
kind
of
stuff.
These
are
also
things
it
doesn't
just
have
to
be
incidents.
These
are
things
that
hopefully
you're
doing
inside
the
company
anyway,
I'm
not
gonna.
Tell
you
how
great
chaos
testing
is.
There
are
plenty
of
other
talks
for
that.
The
dashboards
are
another
big
one.
We
keep
making
all
these
dashboards
if
a
metrics
in
a
dashboard.
That's
also
something
that's
valuable.
A
How
do
you
encourage
the
use
of
that
more
I
mean
we're
not
selling
ads
in
these
dashboards,
yet
at
least
I.
Don't
think
anybody
has
any
plans
to
do
that,
but
the
more
eyes
you
can
get
on
them
and
the
more
you
can
push
people
into
them.
Hopefully,
the
more
people
are
seeing
those
and
the
more
value
that
you're
deriving
from
those
metrics.
Lastly,
you
just
keep
going
with
this.
Where
are
places
where
you
can
use
this
data
inside
your
organization?
A
We
did
a
pilot
program
once
at
a
place
that
I
worked
where
we
use
tracing
data
to
generate
cost
data.
To
say
how
good
were
our
margins
on
some
of
these
like
or
we've
charged
in
too
little
or
too
much
for
some
of
these
queries,
because
if
they're
long
runners,
we
might
be
able
to
take
the
fact
that
we
know
the
spans
and
what
machine
they
were
on
and
how
much
the
machine
cost
to
generate
budget
information,
that's
something
no
one
had
thought
of
before
so
number
5.
A
We
also
need
to
track
need
in
detail,
which
is
just
a
fancy
set
of
slides
for
incidents.
Is
one
of
them
about
us
talk
about
so
observability
I've
already
said
closely
associated
with
incidents
many
times.
This
is
where
we
start
to
push
like
MTTR
and
MT,
tedious
metrics,
for
how
to
have
observability,
improve
your
organization
and
how
things
are
working.
A
So
the
reason
we
need
all
this
is
because
complex
systems
are
hazardous
like
the
fact
that
you've
just
got
these
big
amalgamations
of
services.
They
are
just
intrinsically
dangerous,
they
change
a
lot
as
you
know,
and
they
require
experience
with
failure.
The
only
way
to
learn
about
failure
is
to
fail
that
hasn't
changed
yet
and
it
doesn't
in
our
business
either.
A
I'm,
guessing
that
your
incident
management
metrics
are
probably
bad
because
they're
probably
the
ones
on
this
slide,
no
offense.
We
can
talk
about
it
after
but
MTD
meantime
to
detect,
for
those
of
you
that
aren't
familiar
I
mean
time
to
remediation
is
another
metric
time
stamp,
the
duration
of
an
incident
who
got
it,
how
severe
it
was,
the
proverbial
root
cause
that
doesn't
really
exist.
These
are
all
things
that
you're,
probably
tracking,
in
your
incidents,
and
this
is
inadequate.
A
This
is
at
best
just
like
a
sort
of
high-level
measure
that
stuff
happens
and
confirming
that
stuff
continues
to
happen
there.
It's
not
to
say
that
improving
MTT
d
or
MTTR
on
goals
that
you
should
have,
but
this
is
definitely
not
enough
information,
so
instead
I'll
make
some
suggestions
for
things
you
should
track
most
credit
for
this
slot
goes
to
Norah
Jones.
Many
of
you
may
know
Norah
Jones
in
her
work
in
chaos,
testing
and
safety
and
systems.
A
These
are
the
types
of
things
as
a
suggestion
that
you
should
start
measuring
things
like
where
are
people
saying
stuff
like
how
does
that
work
like
just
going
back
and
reading
the
chat
logs
when
people
are
dealing
with
incidents
can
be
a
goldmine,
because
things
like
people
expressing
surprise
is
really
really
interesting.
Right,
like
we
could
go
back
and
investigate
that
more.
How
about
tracking
all
that
stuff
that
you're
tracking
for
the
high
level
part
of
your
service
at
every
individual
system?
A
What's
the
MTT
D
or
the
MTTR
of
each
of
these
systems,
do
we
have
systems
that
aren't
failing,
because
that
sounds
like
something
you
should
pay
attention
to?
One
of
the
downsides
of
these
incident
management
programs
is
all
that
you're
doing
by
definition,
is
monitoring
the
incident
or
the
failure.
You
are
spending
a
whole
lot
of
your
day,
generating
success,
that
you
also
need
to
compare
something
called
the
halo
effect
and
the
horn
effect,
like
the
halo
effect,
is
that
when
you
only
monitor
success,
you
only
learn
about
success.
A
The
horn
effect
is
that,
if
you
only
monitor
failure,
you
only
learn
about
failure.
We
have
to
mix
two
of
these
and
see
what
we
can
learn
what's
happening
in
successful
things
or
what
near-misses
did
we
have
in
those
successful
operations?
Nobody
likes
surprises
when
it
comes
to
incidents,
so
that's
definitely
something
you
should
pay
attention
to.
Also
the
fact
that
you
keep
creating
these
dependencies
when
dependent
depend
he's
a
vivify
when
do
new
edges
show
up
in
the
graph,
and
when
do
we
get
connections
between
teams
that
were
no
longer
connected?
A
I'm,
not
gonna,
read
everything
on
this
I
pretty
much
already
did,
but
the
the
point
of
this
is
that
there's
a
lot
of
really
interesting
questions.
You
can
ask
about
incidents
that
are
well
beyond
these
simple
qualitative
metrics
that
we
feel
comfortable
with
over
into
qualitative,
which
we're
perhaps
less
comfortable
with
as
engineers.
A
This
is
also
a
great
opportunity
to
get
a
lot
of
feedback
for
doing
the
internal
stretching
yourself,
because
there's
so
much
to
learn
from
near
misses
and
incidents
or
what
people
say
even
about
successful
operations
number
six,
we
are
almost
through
them
all.
This
is
the
last
one,
so
the
next
part
is
instrumenting
change.
At
the
end
of
the
day,
the
real
goal
that
we
mostly
have
is
not
just
to
create
some
system
that
static
that
just
sits
out
there
and
processes
things.
A
Even
if
you're
not
changing
your
system,
your
users
are
because
users
have
this
really
pesky
habit
of
finding
interesting
ways
that
your
api,
maybe
wasn't
intended
to
work
and
then
coming
to
rely
on
those
features,
and
so
even
the
shape
of
the
way
that
your
users
are
handling
the
system
changes
over
time.
So
pretty
much.
Every
tool
that
we
build
has
a
y-axis
which
is
magnitude
of
change,
and
then
it
has
an
x-axis
which
is
time.
The
problem
is
that
times
not
really
what
break
stuff
like
the
passage
of
time
in
and
of
itself.
A
This
is
a
very
heavily
paraphrase
quote
from
Edward
Tufte.
Who
does
a
lot
of
visual
display,
and
that
was
unnerving?
Hopefully
that's,
okay,
the
the
passage
of
time
is
not
the
mutator
in
systems.
That's
not
what
breaks
stuff
humans
and
robots
are
what
tend
to
break
stuff
and
humans
could
be
users
or
they
could
be
you
making
some
changes,
or
maybe
your
friends
who
are
running
puppet
on
the
fleet
or
something
like
that.
So
the
deploys
are
the
obvious
ones
right.
A
Everybody
likes
to
talk
about
the
fact
that
deploys
often
caused
problems,
and
that's
that's
true,
but
pretty
much
any
change
can
cause
problems.
The
vast
majority
of
them
are
safe.
People,
run
RM
minus
RF
all
day
and
managed
not
to
delete
their
entire
home
directories,
but
every
once
in
a
while.
They
do
right.
So
we
generally
modern
view
of
safety
and
systems
is
that
people
are
actually
pretty
safe
and
that
the
systems
are
just
not
safe
enough
to
help
them.
A
So
we
also
need
to
be
paying
attention
to
things
like
change,
management
or
feature
flags
or
scaling
up
and
scaling
down.
Dns
shifts
things
even
social
constructs
like
imagine
that
if
any
of
you
work
at
Apple,
like
imagine,
if
nobody
told
you
about
the
big
Tuesday
event
where
you
announced
the
thing
and
suddenly
all
your
systems
are
looking
extremely
weird,
but
no
one
told
you
that
a
lot
of
people
were
gonna
start
coming
to
look
at
the
new
I
thing
right,
like
that's,
gonna,
be
pretty
frustrating.
A
You
can
do
about
it,
so
we
need
to
avoid
the
human
being
out
of
the
loop
in
the
situation
when
automation
fails,
which
means
automation
even
needs
to
be
heavily
instrumented
and
provide
information
back
to
the
user.
So
this
brings
us
to
feedback.
We
do
not
have
enough
feedback
in
the
tools
that
we
build.
I'm
complaining
about
this,
even
as
a
vendor,
even
the
one
that
I
represent,
doesn't
have
enough
feedback.
A
We
need
to
find
better
ways
to
get
more
information
about
the
change
in
our
systems
back
to
our
end
users
and
our
end
users,
being
our
peers
and
teammates
that
operate
these
systems.
These
things
need
to
be
toggled
on
and
off
easily.
We
don't
want
to
bombard
them.
We
don't
want
to
like
have
the
screen
flashing,
all
the
time
showing
them
alerts
going
off.
A
We
need
subtle
ways
to
show
them
the
trimmers
and
the
shakes
and
the
system,
so
they
can
see
problems
as
they
start
to
occur
or
diagnose
them
as
they
go
backward
and
they
should
be
easy
to
toggle
on
and
off
and
move
around.
We
need
to
avoid
purely
analytic
interfaces
like
just
telling
someone.
Oh
yeah,
you
can
go
over
here
and
run
this
SQL
query.
That
shows
you
all
the
deploys
that
happen
and
then
filter
by
this.
That's
not
what
people
want
to
do.
They
don't
want
to
snap
out
and
go
into
an
analytic
mode.
A
They
want
to
stay
in
a
perceptive
perceptual
mode
of
looking
at
the
user
interface
and
seeing
what's
happening,
something
that
I
feels
pretty
strongly
about
is
like.
We've
got
all
these
alerts
that
are
firing
off
and
like
going
to
slack
or
going
to
email,
where
you're
totally
paying
a
lot
of
attention
to
them.
Instead,
why
not
direct
those
signals
back
into
the
UI?
Can
they
be
showing
up
in
some
of
our
charts
as
subtle
reminders
that
oh
look?
We
garbage
collection
ran
kind
of
hot
there
for
a
minute,
but
then
it
faded
away
afterward!
A
That's
really
important
information
when
you're
trying
to
backtrack
and
figure
out
what
happened
after
the
fact,
all
right.
So
we've
talked
about
a
bunch
of
stuff.
Here
we
went
through
6
points
in
24s
minutes.
How
do
these
things
actually
improve?
Your
observability
stack,
I
told
you
a
bunch
of
stuff,
but
I
want
to
sure
that
you
see
concretely
how
these
can
actually
help.
I
reorganize
all
these
slides
right
before
this
talk.
So
if
I
get
them
mixed
up,
I'm,
sorry,
no
we're
do
it
we're
doing
good.
So
why
does
promoting
the
work
help?
A
If
you've
ever
studied
biology,
you
may
have
heard
of
a
word
called
stigma:
G,
which
is
how
uncoordinated
systems
like
ants
figure
out
how
things
are
going
on
like
if
an
ant
marches
off
and
finds
a
cookie
and
then
marches
back
with
a
cookie
that
ant
leaves
things
in
the
environment.
Pheromones
that
say,
like
hey,
I
found
food
over
here.
A
Another
answer
like
I
like
cookies,
so
before
you
know
it
they're
all
off
to
getting
cookies
too,
and
so
this
is
a
way
that,
if
you're,
showing
success
and
you're
showing
good
output
from
your
work,
then
other
people
are
gonna
pay
attention
to
this,
no
press
is
bad.
Press
even
being
associated
with
incidents
is
helpful
because
I
can
tell
you
observe,
abilities,
usually
I
have
caused
incidents
as
an
observability
team,
but
it's
very
rare,
so
you're,
probably
not
causing
them
so
even
being
associated
with
something
as
nail-biting
ly.
A
Frustrating
as
incidents
can
be
okay,
if
you're
in
leadership
and
you're
in
this
room-
and
you
want
observability
to
be
an
important
part
of
the
stack
you
need
to
be
promoting
that
work.
What
are
you
doing
to
lift
that
work
up
and
say?
Oh
look
at
this
cool
work
that
this
team's
doing
or
that
this
engineer
is
doing.
This
is
really
helpful
to
the
org
and
it's
really
making
our
user
safer
and
make
me
a
better
product.
You
have
the
power
to
do
that.
A
A
You
probably
got
a
lot
of
cool
tricks
and
stuff
internally
that
make
this
stuff
automatic,
and
then
this
also
is
a
big
way
to
avoid
being
frustrated,
because
one
of
the
most
frustrating
things
is
when
you
are
dealing
with
a
problem-
and
someone
goes
oh
there's
a
dashboard
for
that.
That
just
makes
you
want
to
punch
people
in
the
face
right.
A
That's
not
like
a
really
good
answer
if
you've
been
dealing
with
it
for
a
couple
of
hours,
so
systems
stretching
finding
ways
to
use
this
data
to
sort
of
feed
back
into
the
system
to
make
the
system
more
resilient
and
safer,
you're
collecting
a
lot
of
data
in
these
services,
and
in
this
stack.
So
if
you
can
find
ways
to
sort
of
get
more
revenue
out
of
that
data,
so
to
speak
by
using
it.
A
For
other
purposes,
that's
really
but
you're
also
able
to
ultimately
hopefully
make
the
system
and
make
your
product
better
more
resilient
and
safer
for
your
users
and
then
optimally
people
are
using
this
on
a
regular
basis
which
encourages
them
to
get
better
and
better
and
be
more
practiced
so
that,
when
they're
dealing
with
incidents,
they
know
what
they're
doing
measure
the
usage
know
what
it
costs
ahead
of
time,
don't
be
surprised
by
that
conversation.
We
owe
it
to
the
company
to
spend
the
money
wisely.
A
So
we
should
be
making
sure
that
we're
doing
a
good
job
there.
If
you
have
systems
that
are
especially
risky
because
maybe
they're
new
or
they
use
novel
techniques,
monitor
them
closer.
If
you
have
some
systems
that
are
directly
user
interfacing,
that's
probably
something
else,
that's
risky.
If
the
system
has
been
operating
fine
for
a
long
time,
maybe
that's
a
time
you
can
dial
it
back.
That's
definitely
something
you
should
know
like.
Can
we
take
away
some
monitoring
of
certain
systems
or
some
data,
some
telemetry
volume,
or
something
to
that
effect?
A
And
you
definitely
want
to
be
prepared
when
finance
comes
calling?
You
don't
want
to
say,
let
me
get
you
those
numbers
next
week,
so
in
our
incidents
we
need
to
make
sure
that
we
are
improving
how
effectively
we
measure
them
so
that
the
observability
stack
can
help
and
be
sure
to
examine
success
as
often
as
failure-
and
this
also
will
generate
lots
of
interesting
questions.
You
use
the
incident
management
process
to
get
information
that
you
can
feed
back
into
the
stack
it's
a
great
source
of
those
problems
and,
lastly,
we
need
to
instrument
change.
A
Better
change
is
usually
the
start
of
the
problem.
I
know
most
of
us
when
an
incident
happens
immediately,
go
who
deployed
something
who
changed,
something
who
ran
puppet,
who
changed
a
feature
flag
right,
like
all
these
things
are
the
social
things
you
need
to
know,
and
this
they're
also
terribly
small
volumes
of
data
like
these
are
not
things
that
happen
a
billion
times
a
day
like
many
of
our
metrics.
A
And,
ultimately,
you
want
more
change
if
you
are
more
agile
and
able
to
ship
product
faster,
that's,
ultimately,
something
that
you
can
feed
back
to
leadership
and
say:
hey!
Look
since
we
did
this,
we've
been
able
to
improve
the
rate
of
change,
but
the
number
of
failures-
the
MTBF,
as
it's
often
called
mean
time
between
failures-
is
lower
all
right.
So
this
is
the
slide.
A
If
you
just
want
to
snapshot
it
and
see
the
six
things
all
in
one
place
to
promote
the
work,
teach
the
basics
stretch,
the
systems
measure
your
usage
track,
need
an
instrument
change.
These
are
the
six
things
that
I
encourage
you
to
focus
on
again.
These
are
largely
social,
not
much
technical
about
this
I
think
I
have
one
interstitial
slide
that
I'm
gonna
know
this
first,
so
some
are
all
of
these.
By
the
way
you
don't
have
to
do
them
all
and
not
gate
key
be
about
it.
A
Maybe
you
have
some
related
things,
but
pretty
much
any
one
of
these
or
a
little
bit
of
all
of
them
should
result
in
a
much
more
rigorous,
but
definitely
a
much
more
valuable
observability
program
and
I'm
gonna.
Let
this
slide
linger
for
just
a
little
while
these
are
citations.
If
you're
interested
in
some
of
the
like,
where
some
of
this
material
came
from
I,
encourage
you
to
take
pictures
of
this
slide
and
check
out
some
of
these
papers,
I'm
a
big
fan
of
the
top
one,
the
Norah
Jones.
A
This
talk
also
plugged
that
that
was
just
a
few
weeks
ago-
very
good,
talk
at
mama
drama
and
then
also
Wikipedia
for
thoroughness.
But
thank
you
so
much
for
giving
me
the
opportunity
to
talk
today.
If
you
want
to
talk
about
any
of
this
stuff
I'll
be
here
all
week,
you
can
talk
to
me
on
the
Twitter's
or
you
can
go
to
my
website
at
one
Mogan
comm,
where
I
write
about
some
of
this
stuff
periodically
and
someone
randomly
thank
you
enjoy
the
combo.