►
Description
The RED Method: How To Instrument Your Services [B] - Tom Wilkie, Kausal
The RED Method defines three key metrics you should measure for every microservice in your architecture; inspired by the USE Method from Brendan Gregg, it gives developers a template for instrumenting their services and building dashboards in a consistent, repeatable fashion.
In this talk we will discuss patterns of application instrumentation, where and when they are applicable, and how they can be implemented with Prometheus. We’ll cover Google’s Four Golden Signals, the RED Method, the USE Method, and Dye Testing. We’ll also discuss why consistency is an important approach for reducing cognitive load. Finally we’ll talk about the limitations of these approaches and what can be done to overcome them.
About Tom Wilkie
Tom is the founder of Kausal, a new company working on Prometheus & Cortex. Previously he worked at companies such as Weaveworks, Google, Acunu and XenSource. In his spare time, Tom likes to make craft beer and build home automation systems.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
A
Right,
then,
how
about
mean
I
guess
is
where,
as
we're
here,
we're
probably
also
using
kubernetes
hands
up
so
more
people
using
kubernetes
and
Prometheus
and
using
kubernetes
and
Prometheus
together.
Okay,
that's
it's
not
as
many
as
I
was
expecting.
Actually
with
the
over
set
the
overlap.
All
right,
then,
as
I
said,
my
name's
tom
started,
this
company
called
causal
recently
we
do
a
hosted
version
of
Prometheus.
Basically,
I
also
make
my
own
beer,
which
kind
of
explains,
and
that's
also
the
barbecue
for
lunch
used
to
work
for
a
company
called
weave
works.
A
Who
also
here
and
also
do
a
lot
of
work
on
prometheus
before
that
I
was
at
Google
for
a
few
years
and
before
that,
another
startup,
so
we're
going
to
cover
four
things
today,
we've
kind
of
done
the
first
already
the
use
method.
I
know
that
the
talk
was
pitched
as
the
read
method,
but
we're
actually
gonna
talk
about
the
use
method.
First,
because
that's
kind
of
my
inspiration,
at
least
for
the
naming
scheme,
the
read
method,
which
is
what
we
have
mainly
here
to
talk
about
and
then
the
footballs
for
golden
signals.
A
If
anyone
here
has
read
the
the
Google
SAE
handbook
so
quickly
the
introduction,
why
does
this
matter?
Why
do
we
even
need
patterns,
and
why
should
we
even
care
about
instrumenting
our
applications?
So
I'll
start
first
with
why
I'm
giving
this
talk?
There
was
a
recent
recent
prometheus
conference
in
Munich
and
the
read
method
was
alluded
to
multiple
different
talks
and
no
one
actually
explained
what
it
was,
and
more
and
more
like
I'm
seeing
mentions
of
it
on
Twitter
and
things
like
this
and
no
one's
actually
kind
of
come
along
and
said.
A
This
is
what
the
read
method
is.
So,
as
I
kind
of
tongue-in-cheek
came
up
with
this
name
a
couple
years
ago,
I
thought
there
should
be
an
authoritative
talk
on
what
the
read
method
is,
at
least
from
my
point
of
view.
So
why
are
these
patterns
important?
Well,
if
you're
a
software
engineer,
you
probably
don't
want
to
spend
your
life
thinking
about
the
best
way
to
instrument
your
code.
You
know
probably
more
important
what
your
code
actually
does.
So,
by
having
these
patterns
and
by
having
kind
of
tried
and
tested
methods.
A
A
There's
a
there's,
a
whole
debate
about
white
box
versus
black
box
monitoring,
if
you're
already
using
from
easiest
it
sounds
like
most
of
you
are.
Then
we've
already
won
that
debate
and
white
box
monitoring.
It
is,
and
if
you
want
the
debate
about
push
versus
pull
you're
in
the
wrong
place,
where
that's
a
bit
of
an
in-joke,
sorry,
all
the
examples
I'll
give
will
be
Prometheus
running
on
kubernetes
as
well,
so
to
start
with
use
method.
A
This
was
invented
by
a
chap
called
Brendan
Greg
or
at
least
coined
by
him,
and
defines
three
things
for
every
resource
in
your
system.
You're
going
to
monitor
the
utilization,
the
saturation
and
the
errors.
Okay
and
Brendan.
Has
this
absolutely
excellent,
like
de
facto
webpage
that
describes
this
incredible
detail
and
I'm
just
going
to
kind
of
lightly
skim
through
it
a
little
bit
here
so
utilization?
A
So
resources
are
CPUs
disks
memory
network,
some
things,
people
normally
forget
things
like
interconnects,
yeah
and
so
on,
and
this
is
very,
very
targeted
at
infrastructure
at
real
things,
it'd
be
quite
challenging
to
apply
this
to
your
micro
services,
which
is
why
we
did
the
read
method
anyway.
Moving
on
you
end
up
with
this
kind
of
checklist
approach
and
when
you've
got
a
problem.
A
I
don't
know
about
you,
but
I
find
my
problems,
kind
of
bisects
quite
nicely
and
there's
half
of
my
problems
I
like
a
very
obvious
after
10
minutes
of
fiddling
and
then
there's
the
other
half
which
I
have
no
clue.
What
the
problem
is
and
I
really
don't
know
what
I
don't
know
so
one
of
the
nice
things
about
having
these
kind
of
checklist
based
approaches,
they
really
help
in
the
second
instance
in
the
instance,
when
you
don't
know,
what's
wrong,
it's
quite
reassuring
to
go.
A
Look
if
I
just
go
and
check
all
of
these
things
in
order,
then
I'm
likely
to
find
a
problem.
That
gives
me
the
clue
for
the
next
thing.
So
you
end
up
with
this
kind
of
checklist
and
if
you're
like
me,
you
turn
this
into
a
dashboard
yeah.
You
turn
you're
you're
hunting
for
problems
into
something,
that's
a
bit
more
methodical
it,
but
a
little
bit
less
kind
of
haphazard,
so
diving
into
the
real
stuff
how'd
you
do
this
with
Prometheus.
Well,
CPU
utilization.
A
One
of
them
is
that
I
love
from
atheists,
because
the
queries
fit
on
a
single
page.
So
you
get
nice
neat
queries
like
this.
This
is
using
the
node
exporter,
which
is
a
primitive
exporter.
You
run
on
every
machine
if
you're
using
cuban
Hattie's
use
a
daemon
set,
and
then
you
go
in,
and
you
say
for
each
CPU
in
the
machine.
How
much
time
was
it
idle
for
and
you
take
that
as
a
percentage
of
the
total
time
and
you
take
one
out
and
take
it
away
from
one.
A
So
this
is
super
easy
right,
CPU
saturation,
this
one's,
where
it
gets
a
little
bit
more
interesting,
CPU,
saturations,
digitally
measured
as
load
average,
so
I
picked
the
one
minute
load
average
here,
load
average.
For
those
who
aren't
aware
is
the
number
of
processes
on
the
run
queue
like
a
sampled
average
of
that.
A
This
one
here
is
a
open
do
that
this
one
here
is
a
handy
little
recording
wall
that
I've
written
that
gives
me
per
node.
The
number
of
CPUs
Missha,
punishing
you'll
notice.
I
don't
have
errors
on
this
slide,
so
this
is
really
the
us
method.
So
far,
it's
a
terrible
joke.
A
cpu
errors
are
kind
of
really
hard
to
measure.
It
turns
out.
A
I
looked
into
a
bunch
of
places
and
the
best
advice,
I
got
was
like
wrapping
through
D
message,
for,
like
exceptions
and
machine
check,
exceptions
and
they're
kind
of
machines,
specific
and
they're
kind
of,
like
that.
The
Intel
processors
have
some
nice
counters
on,
but
they're
different
on
other
processors
and
so
on.
So
I
don't
really
have
a
way
to
do
that.
But
luckily
CPU
errors
don't
seem
to
be
a
thing.
I
need
to
worry
about.
A
A
This
memory,
one
of
where
saturation
is
in
some
weird
units
and
the
best
advice
I've
seen
for
how
to
measure
your
memory
saturation
is
actually
to
look
at
how
much
paging
your
machine
is
doing,
because
if
it's
doing
a
lot,
it
kind
of
tells
you
that
your
memory
is
being
is
oversubscribed
and
if
it's
not
doing
any,
then
saturation
is
not
a
problem.
The
list
really
does
go
on.
You
know
I've
already
touched
on
CPU
errors
and
memory.
A
Errors
are
another
thing
that
are
really
hard
to
actually
get
a
counter
for
I
found
it
really
hard
to
get
a
counter
for
hard
disk
errors.
You'd
think
this
would
be
quite
obvious.
It
turns
out.
There
is
a
counter
in
Sisyphus,
but
Prometheus
doesn't
expose
it
yet
so
I
was
hoping
to
get
it
done
before.
I
gave
this
talk,
but
I'm
going
to
plum
that
through
node
exporter,
so
you
can
actually
have
a
counter
disk
capacity
versus
disk.
A
I/O
I,
don't
know
about
you,
but
disk
errors
have
never
caused
an
outage
for
me,
whereas
running
out
of
this
capacity
is
probably
the
root
cause
of
most
of
my
outages.
So
this
capacity
doesn't
really
seem
to
fit
into
this
use
model
very
well.
Maybe
this
is
a
criticism
of
a
model
or
my
understanding
of
it.
Network
utilization
is
a
another
interesting
one,
because
I
really
want
to
have
this
as
a
percentage
of
some
like
total,
but
I've
not
found
a
way
to
kind
of
know.
A
The
actual
capacity
of
my
link
like
programmatically
so
currently
I,
just
in
my
dashboard
just
say,
look
I,
know
that
for
this
machine
type
on
AWS
I
have
this
much
network
traffic
and
network
bandwidth
and
then
I
use
that
as
the
denominator
and
get
my
percentage,
but
I'm
not
very
happy
with
that
ya
know.
So.
For
that
way,
I
also
don't
have
any
clue
how
to
measure
interconnects.
So,
let's
see
if
this
MiFi
works,
we've
done
a
set
of
dashboards.
A
A
A
A
You
end
up
with
these
use
method,
dashboards,
and
so
this
is
a
cluster
wide
use
method.
Dashboard
with
every
single
one
of
those
queries.
I
talked
about
encoded
into
it.
You
know
you
can
see.
My
cluster
is
not
particularly
heavily
loaded
and
it's
just
really
straightforward.
Each
row
is
one
resource.
Each
column
is
a
different
bit
of
the
use
method.
This
is
all
open
source
if
you're
interested
it's
on
github
in
who
else
here
does
ma
know
repos
yeah
me
too.
A
We
have
a
public
and
a
private
mono,
repo
and
I'm,
not
sure
it's
a
good
idea
yet,
but
so
all
of
our
open
source
projects
deliver
it
cause
of
all
public
and
Klumps
is
here.
So
if
you
want
them,
the
lot
of
the
dashboards
Agrafena
dashboards
they're
generated
with
something
called
JSON
it,
which
is
like
this
really
cool
conflict
management.
Language
for
like
merging
JSON
dictionaries
and
doing
all
sorts
of
really
cool
stuff,
so
you
have
to
like
generate
them,
but
this
means
you
can
parameterize
them.
A
If
you
use
slightly
different
job
name
for
various,
like
node
exporters
named
differently
on
your
cluster,
then
you
can
just
bang
that
in
and
generate
out
the
dashboards,
and
that
is
pretty
much
the
user
for
dashboards.
I
want
to
iterate
on
these
and
get
them
to
include
kind
of
the
fixes
that
I've
discussed
here
and
get
them
to
include
disk
errors
and
so
on
and
and
maybe
other
people
will
find
them
useful
and
we're
also
going
to
add
things
like
drill
down.
A
A
Swear
keynote
used
to
make
this
switching
between
mirrored
and
not
mirror
and
much
easier,
so
yeah,
if
you
want
to
do
some
further
reading
on
this,
the
use
method
website
by
Brendan
Gregg
is
absolutely
fantastic,
goes
into
a
lot
more
detail
and
has
a
lot
more
kind
of
pros
and
cons
and
side
effects
and
so
on.
You
should
definitely
read
that
and
then
we've
got
the
Klumps
dashboards.
That
I
just
mentioned
the
read
method.
A
What
we're
really
here
to
talk
about,
so
the
read
method
was
kind
of
inspired
by
a
question
we
had
at
work,
which
was
like:
why
aren't
we
using
the
use,
method
and
I
found
it
really
hard
to
explain
to
people
that,
like
the
use
method
is
for
resources
and
the
read
method
is
for
my
services
and
they're
kind
of
two
different
things
that
we
do
use
the
use
method.
But
the
use
method
doesn't
tell
me
if
my
services
are
healthy.
They
tell
me
if
my
machines
are
healthy.
A
So
the
read
method
is
a
bunch
of
stuff
I
learned
at
Google
that
I
kind
of
miss
remembered,
and
so,
if
you're
familiar
with
Google's
for
golden
signals,
it
includes
saturation
and
when
I
came
up
with
this
I
forgot
about
that
one.
It's
also
like
use,
has
three
characters,
so
this
needs
three
characters
and
I
couldn't
really
fit
away
of.
You
know
it
could
be
the
Reds
method,
but
that's
a
bit
bit
lame.
A
So
the
red
method
is
for
every
service
in
your
architecture,
monitor
the
request
rate,
the
rate
of
errors
of
those
requests
and
the
duration
of
those
requests
is
some
kind
of
distribution.
Why
well?
This
is
a
nice
way
of
knowing
kind
of
without
knowing
what
the
service
does
or
how
it
works,
knowing
if
the
service
is
healthy,
ish
right.
This
is
really
useful
for
when
you've
got
an
on-call
team
that
are
on-call
for
things.
A
They
didn't
write
and-
and
if
you
want
kind
of
your
DevOps
to
to
scale
and
to
be
something
you
do
as
a
company,
eventually
you're
going
to
have
to
be
on
call
for
stuff.
You
didn't
write,
you
know.
Eventually,
someone
else
is
going
to
have
to
be
on
call
for
your
code
and
you're.
Gonna
have
to
be
on
that
call
for
their
code
and
that's
how
you
build
this
kind
of
scalable.
You
know
operations
team
so
to
do
that.
A
You'd,
not
it's
nice
to
have
a
nice
uniform
method
of
like
knowing
what
a
service
does,
knowing
that,
if
a
service
is
healthy,
so
the
read
method.
It
also
is
nice,
because
this
is
kind
of
the
building
blocks
for
being
SLA
driven.
You
should
really
only
care
if
your
customers
are
happy.
You
shouldn't
care
if
your
computers
are
happy,
so
this
gets
like
starts
getting
towards
high-level
metrics.
That
tell
you.
If
your
customers
are
happy,
you
know
if
things
are
slow,
if
your
durations
and
Layton
seas
are
really
high,
then
your
customers
probably
aren't
happy.
A
If
your
errors
are
high,
they're,
definitely
not
happy.
If
your
rates
not
high,
that's
an
interesting
one,
maybe
you're
just
not
popular
enough,
so
this
started.
This
I
think
is
the
first
mention
I
could
do
I
had
a
different
haircut
at
this
time.
This
is
the
first
mention
I
could
find
of
the
read
method
and
this
kind
of
caused
a
big
storm,
because
the
person
said
an
alternative
and
really
it
should
be
a
kind
of
an
addition
like
a
different
method
for
monitoring
a
different
thing.
Anyway.
How
do
we
do
this
with
Prometheus?
A
A
This
declares
the
Prometheus
metrics,
so
I
have
a
laser
pointer,
but
this
declares
the
metric
we're
going
to
use
a
single
histogram
to
represent
all
three
of
the
attributes
of
the
read
method
and
we're
going
to
call
it
by
best
practices,
we're
going
to
include
the
SI
unit,
we're
going
to
record
things
in
seconds,
not
in
milliseconds
we're
going
to
include
various
different
labels,
and
this
is
how
we're
going
to
differ
a
JIT
between
successful
and
failed
requests.
So
we've
got
a
status
code
in
there
and
then
we're
going
to
register
that
metric.
A
So
you
need
this
once
in
your
application.
Normally,
at
the
top
of
like
main
logo,
then
I
prefer
the
approach
of
having
a
piece
of
middleware
that
I
can
wrap
around
my
HTTP
handlers
that
will
go
and
instrument
them
instead
of
kind
of
manually,
putting
the
instrumentation
into
every
single
HTTP
handler.
There's
a
nice
set
of
generic
middleware
out
there
there's
a
Prometheus
one.
The
Weaver
works
guys
have
one
in
their
common
repo,
which
is
one
I
use,
and
this
is
the
the
one
I've
come
up
with
here.
A
This
uses
something
called
HTTP
snoop,
which
is
a
really
cool.
Little
go
library
if
you've
ever
if
you've
never
seen
it.
That
gives
you
this
fake
response.
Writer
that
records
a
whole
bunch
of
metrics,
as
it's
being
used
like
how
long
it
was
used
for
and
what
the
error
code
was
and
how
much
bytes
were
sent
in
so
much
and
then
actually
recording
the
metric
is
just
that:
request,
duration
with
labels
method
path
code,
so
on.
If
anyone
can
spot
the
obvious
mistake
on
this,
they
can
win
a
causal
sticker.
A
A
Or
they've
got
different
paths
to
maps
and
I.
Think
that's
so.
The
obvious
mistake
here
is
I've
used
the
URL
path
as
the
label
and
like
wool.
Zero
of
Prometheus
rules
is:
don't
use
high,
cardinality
values
in
your
labels,
because
this
will
really
overload
the
time
series
data
base
underneath
it
so
actually
what
we
tend
to
use
instead
of
using
URL
paths
which
will
include
things
like
you
know,
whatever
you
know,
random
user
IDs
you've
got
in
your
path.
We
tend
to
erase
out
the
high
cardinality
things.
A
So
if
you,
if
you
defined
like
if
you're
using
like
gorilla
bucks-
and
you
define
your
your
pattern,
matcher
for
your
path,
we
would
use
that
instead
of
the
actual
path-
and
this
helps
reduce
the
cardinality
help
to
keep
your
queries
fast.
Basically,
so
I
get
to
keep
my
stick
anyway,
and
then
we
use
it
by
we
serve
the
Prometheus
metrics,
so
Prometheus
can
come
and
on
and
scrape
them
and
we
serve
a
function
which
won't
compile
because
it's
three
dots
so
then
to
get
the
rate
of
requests
is
super
easy.
A
You
just
look
at
the
request.
Eurasian
seconds
count,
and
so
this
is
kind
of
an
internal
implementation
detail
of
how
histograms
are
implemented
in
Prometheus.
Is
they
actually
export
a
whole
bunch
of
time
series?
They
don't
just
export
a
single
one.
They
explore
account
sum
and
these
buckets,
and
so
we
use
this
as
a
count
to
see
how
many
requests
are
going
and
then
we
do
a
count
and
we
filter
them
by
the
status
code.
A
A
So
actually
I
would
normally
type
it
in
here,
but
I
find
it
much
easier
to
use
my
product,
of
course.
But
this
is
not
a
pitch.
You
can
do
this
just
in
normal
Prometheus.
So
first
we'll
do
some
rate
request,
duration
and
the
reason
I
find
it
easier
because
it's
got
this
auto
company.
So
we
can
do
we're
going
to
need
a
rate
of
that
and
I'm
going
to
filter
this.
So
you're
only
seeing
the
dev
traffic.
A
Okay,
there
we
go
yeah,
but
we
can
start
to
do
more
interesting
stuff
if
we
go
and
break
this
down
by
by
request,
sorry
by
status
code.
So
here
we
go
we're
going
to
drill
down
into
the
status
code,
and
now
we
can
see
that
there
is
no
errors
per
second
and
there
are
36
QPS
to
our
dev
front-end.
So
this
is
like
the
first
two
bits
of
the
read
method
in
a
single
query,
and
then
this
is
where
it
gets
way
more
interesting,
because
I
have
to
write
and
get
right.
A
A
A
If
I
just
look
at
the
raw,
if
I
just
look
at
the
raw
set
of
time
series
that
query
returns,
you
can
see
this
le
dimension
has
like
of
various
numbers,
like
all
the
requests
would
be
in
the
plus
infinity
1
and
then
only
ones
that
took
one
second
or
lower
being
12.5
and
so
on,
and
you
can
see
it
broken
down
my
method
by
namespace.
You
can
even
see
the
route
who
got
it
right
in
America.
Doesn't
include
user
names,
so
there
we
go.
That's
that's
how
you
do
the
use
method
with
Prometheus.
A
Just
see
what's:
gonna
yeah,
okay
and
I
like
this
approach,
because
it's
just
super
simple
there's,
not
a
lot,
there's
one
one
thing
you
need
to
instrument
all
your
apps
with
most
companies
probably
have
a
common
library
that
they
can
put
this
into,
and
so
the
developers
don't
even
need
to
think
about
this
when
it
gets
a
bit
more
interesting
is
your.
You
know.
Your
generic
micro
service
architecture
usually
looks
like
lots
of
services
that
all
talk
to
each
other
and
if
you're
really
lucky,
there's
no
loops.
A
You
know
if
you've
got
some
errors
at
the
top,
you
basically
have
to
scroll
down
until
the
errors
stop
and
it's
the
service
above
that
that's
introducing
those
errors.
If
you,
if
you
kind
of
go
what
I
mean,
this
is
kind
of
me
taking
that
graph
and
traversing
it
row
by
row,
so
I
find
this
a
really
nice
way
of
writing
dashboards
and
then,
to
the
extent
where
this
completely
automated
now
like
the
system,
will
just
generate
these
for
me
using
JSON
the
other
one.
A
This
is
something
I
learned
the
hard
way
when
you're
plotting
your
latency
graphs.
You
should
never
plot
average
right.
You
should
never
use
average.
This
has
been
drummed
into
over
the
multiple
years
of
doing
monitoring,
but
except
for
you'll
notice,
I've
got
a
verage
plotted
here
and
there's
two
reasons:
I
plot
average
one
is
because
Layton
sees
done
some.
You
know
if
you've
got
one
service,
that's
got
a
really
high
latency
and
you
look
at
the
two
services.
It
calls
the
you
know.
A
It
quite
quite
often
happens
that
they've
got
really
low
latency
and
it
just
doesn't
help
you
because
latency
use
do
not
some
like,
like
that
averages
do
some.
So
if
you've
got
one
service
with
a
very
high
average,
then
one
of
them
below
is
going
to
have
a
very
high
average,
and
that
helps
you
point.
The
second
reason
is
that
averages
can
be
used
to
sanity
check
whether
you've
got
your
buckets
right,
and
this
is
the
reason
I
was
talking
about
Elian
buckets
for
histograms,
like
it's
an
implementation
detail,
but
it's
when
people
get
wrong.
A
Sometimes
so
sometimes
you'll
see
a
nice
low,
latency,
no
problems
at
all,
because
you've
only
declared
your
buckets
go
up
to
100
milliseconds
and
obviously
it
can't
report
more
than
100
milliseconds
and
then
you
go
and
plot
your
average,
which
is
taken
just
by
dividing
the
count
by
the
sum
note
the
sum
by
the
count
and-
and
then
you
see-
that's
it
like
you
know
tens
of
seconds
and
you've
realized.
You've
got
your
buckets
wrong,
so
I
like
to
include
average
in
there.
A
If
you
want
to
be
more
about
the
read
method,
there's
a
bunch
of
work.
I
did
we've
works,
but
it's
really
taken
off
with
other
companies.
Now,
there's
Cindy's
got
a
this
fantastic
blog
post
about
logs
and
methods
and
metrics
which
I'd
really
go
into.
The
slides
will
be
online
and
these
are
clickable
links
have
them
so
there's
not
much
porting
taken
photo
but
yeah.
That's
peter
has
good
writings
on
these
as
well.
A
How
can
that
even
work?
It
doesn't
so
you'll
notice.
This
is
very
similar
to
the
read
method
and
it
just
has
saturation.
You
know.
Latency
is
the
same
as
duration.
Traffic
is
the
same
as
well.
Requests
errors
is
the
same,
but
this
saturation
thing
how
full
your
services
is
like.
Okay,
is
that
your
definition,
how
full
your
services
and
one
of
the
reasons
I
excluded?
It
is
because
what
does
that
even
mean,
but
it
turns
out,
if
you're,
using
premier,
if
you're,
using
cube,
Nettie's,
there's
a
nice
definition
of
this
right.
A
You
should
be
saying
this
pod
leads
one
CPU
or
a
tenth
of
a
CPU
or
so
on,
and
this
gives
you
a
nice
denominator
to
tell
how
full
your
services,
so
this
one
I
am
going
to
cheat
with
I'm
afraid
because
it's
quite
a
handful
to
go
into
cube,
State
metrics
and
see
advisor
and
build
this
up
from
from,
like
the
actual
primitives,
but
I've
defined
a
bunch
of
recording
laws
and
it
is
under
I,
don't
know,
I
actually
have
forgotten
namespace
yeah.
That's
not
good
idea.
Let's
do
pod.
A
Okay,
well
even
more
because
I
built
this
into
a
dashboard
because
I
saw
I,
don't
have
to
remember
these
queries.
I
have
these
used
method,
but
these
cluster
resorts
dashboards,
which
tell
me
how
full
my
cluster
is.
How
committed
my
cluster
is
so
how
much
of
the
CPU
I
physically
got
is
in
use
by
it
has
been
committed
in
requests.
So
this
is
what
these
mean,
and
but
these
will
tell
me
David's
fixed
him
awesome.
A
This
is
this
is
a
fork
of
graph
and
we
use
because,
with
a
graph
on
you
can't
have
these
these
multi
column
tables
with
numbers
in
the
columns.
So
we've
got
a
fork,
we're
merging
it
all
back
in,
of
course,
we're
just
currently
running
this
fork,
but
let's
go
and
have
a
look
at
this,
so
yeah
there's
the
ax.
So
we're
going
to
take
this.
So
this
is
the
container
resource
requests.
A
But
slower
I
blame
the
MiFi,
so
here
we
go,
you
can
see
it's
got
it
broken
down
by
instance.
Actually,
the
incidents
the
same
he's
got
it
broken
down
by
node
and
namespace
and
the
container
and
so
on,
and
it
tells
you
basically
how
many
CPUs
are
requested
for
that
pod.
So
this
pot
needs
two
CPUs.
Okay,
it's
a
pretty
straightforward
and
then
this
one
says
container
CPU
seconds
by
namespace.
A
You
should
be
able
to
see
the
same
thing,
and
this
gives
us
the
ongoing
CPU
usage
of
the
containers
in
my
cluster,
and
that
graph
was
a
bad
idea
and
now
we
need
to
divide
out
the
2,
and
that
will
give
me
some
some
idea
of
ego
CPU
usage
of
all
the
pods
in
mica
in
my
cluster,
and
if
we
divide
that
2
it
should
give
us
the
idea
of
how
full
are
my
pods
and
I'm
really
gutted.
That
I
can't
find
the
query
for
this,
because
it's
not
as
simple
as
just
dividing
the
two.
A
A
Yeah.
Sorry
about
that,
you
can't
you
can
divide
the
two,
but
the
problem
is
the
pod
name
is
different.
If
you
look
here
this
one's
using
pod
and
the
other
ones
using
pod
underscore
name,
and
so
you
have
to
do
a
real
able,
don't
think
that
in
my
recording
rules
anyway,
this
is
all
in
the
in
the
open
source.
So
you
can
go
knock
yourself
out
and
and
show
me
how
to
do
this
better.
A
Here
we
go,
these
are
the
various
services
we've
got
running
in
the
cluster,
and
here
you
can
see
how
full
they
are.
You
know
40%
is
a
bit
high,
maybe
but
it's
fine,
and
so
this.
Finally,
you
can
get
an
answer
for
you
know
what
is
the
saturation
of
their
services
in
my
cluster.
So
if
you
want
to
read
more
about
this,
the
Google
s3
book
is
online
and
has
an
absolutely
fantastic
chapter
on
this
and
there's
another
blog
that
mentions
it
that
I
found
on
Google.
A
So
in
summary,
we've
covered
the
use
method
using
it
for
monitoring
the
resource
utilization
saturation
requests
for
every
resource,
really
useful
for
infrastructure.
The
read
method
that
is
request,
errors
and
duration
for
every
service
in
your
cluster
we've
taken
a
quick
look
at
for
golden
signals,
which
is
red
plus
saturation
and
how
you
might
want
to
do
saturation
on
your
cuba,
Nettie's
cluster
and
then
I've
kind
of.
A
Given
you
a
couple
of
demos,
there's
a
couple
of
other
methods
out
there
and
the
TSA
method
is
actually
another
one
of
Brendan's
Greg
Brendan
Greg's
that
looks
at
threading
and
Method
R
is
I.
Have
no
idea
I
need
to
look
at
them
for
the
rest
of
the
day,
so
you're
already
in
the
read
method,
talk
which
means
you're,
probably
interested
in
prometheus,
there's
a
meet
the
maintainer
x',
where
you
can
come
in
and
argue
with
me
about
the
stuff
I've
discussed
here
and
we're
also
going
to
have
frederick
and
girl
from
there.
A
If
you
want
to
ask
them
some
more
interesting
questions,
Ilya
from
weave
works
is
giving
a
talk.
Practical
rights
of
Prometheus
for
app
developers
and
the
guys
from
the
French
FedEx
company
are
giving
a
talk
about
how
they
use
Prometheus
and
then,
with
a
p.m.
after
the
booth
call.
We've
got
some
drinks,
a
local
hotel,
sponsored
by
the
fresh
tracks,
guys
there's
the
link
here.
If
you
want
to
sign
up
I,
take
it
with
probably
getting
close
to
capacity
so
sign
up
quick.
A
A
A
A
Most
services
have
more
than
one
RPC
that
they
accept,
especially
if
it's
your
front
end,
you
might
have
a
hundred
different
parts
in
your
front
end
and
your
one
root
in
your
front
end
might
be
were
returning
a
hundred
percent
errors
and
you'll
miss
it
if
that
route
isn't
called
as
much
as
say
another
one,
you
know
so
in
in
causal,
we've
got
one
route
the
right
path.
That's
called
like
three
orders
of
magnitude
more
fun
than
the
query
path,
so
any
errors
on
the
query
get
completely
drowned
out.
A
If
if
we
just
were
to
alert
on
the
total
error
rate,
so
we
do
error
rate
per
service
per
boot
and
then
latency
as
well,
we
alert
on
latency.
We
have
some
pretty
strict
targets
that
we
don't
always
meet
so
right.
Now
they
don't
actually
pages
unless
they're
like
really
really
bad,
but
hopefully,
as
we
start
to
meet,
our
stricter
targets
will
start
alerting
on
breaches
of
those,
but
that's
pretty
much
it,
and
then
you
I
tend
to
be
of
the
idea
that
you
shouldn't
alert.
A
You
know
it's
google
standard,
google
practice,
don't
alert
on
symptoms.
Don't
alert
on
causes
a
lot
on
symptoms,
so
you
should
alert
on
things
that
feature
SLA
and
we
don't
alert
on.
You
know
random
service
over
here
it
hasn't
been
able
to
elect
a
master
unless
some
something
at
the
top
starts,
then
complaining
the
that
that's
causing
errors
to
a
class
to
a
user.
The
only
exception
I
make
for
that
is
disk
capacity,
because
when
you
run
out
of
disk
capacity,
everything
stops.
A
A
A
So
what
I
tend
to
do,
for
instance,
when
I'm
talking
to
DynamoDB
or
when
I'm
talking
to
you,
know
s3
or
any
other
service
that
I
don't
own
I
instrument
on
the
client
side,
but
normally
when
I'm
talking
services
that
I
do
own
I,
think
it's
good
enough
to
just
instrument
on
the
server
side.
I
mean
you
can
do
both,
but
then
that
kind
of
depth-first
traversal
of
all
your
services
starts,
become
really
complicated
right
and
when
the
network
is
broken,
it
normally
becomes
pretty
obvious
pretty
quickly.
A
A
Do
you
use,
read
scope
great
I,
I
spent
two
years
of
my
life?
Writing
them
and
then
I
moved
on
to
prometheus,
but
we've
scoped
kind
of
gives
you
that
nuggets
screenshots,
but
that's
probably
better
screenshots
yeah
gives
you
that
kind
of
dag
of
of
what
talks
to
what
and
you
could
use
that,
for
instance,
if
you
didn't
know,
but
I
tend
to
just
know
so,
I
use
those
anymore
questions.