►
From YouTube: Monitor the World: Meaningful Metrics for Containerized Apps & Clusters - Nicholas Turner & Nic Cope
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Monitor the World: Meaningful Metrics for Containerized Apps and Clusters - Nicholas Turner, Amazon & Nic Cope, Planet Labs
In this talk, Nick and Nic dive into key metrics and tooling that you should be using to monitor your Kubernetes control plane components and your containerized applications. Nick will disect the metrics that have shown to be critical for Kubernetes cluster operations, experience gained while helping to build and operate Amazon's EKS platform. Topics covered will include monitoring etcd and the Kubernetes control plane components. Nic will discuss Planet Lab's Kubernetes deployment and the application metrics that have proven to be most critical for operating large scale clusters, including how they utilize Linkerd and Prometheus for visibility. Two themes that will persist throughout the talk are how use the minimum number of necessary metrics to still have a complete picture while avoiding noise, and how to do it with open source technology.
To learn more: https://sched.co/GrXU
A
I'm
gonna
briefly
discuss
monitoring
applications
in
a
curb
kubernetes
environment,
but
a
lot
of
it
applies
to
just
monitoring
applications
in
general
and
then
I'm
going
to
touch
on
monitoring
the
kubernetes
control
plane,
especially
in
the
context
of
as
you
scale
your
cluster
and
then
my
co-presenter
Nick
will
walk
us
through
how
planet
labs
set
up
their
monitoring
infrastructure,
which
is
the
actual
interesting
part
of
the
talk.
And
so
our
theme
is
going
to
be
on
sort
of
how
to
focus
on
fewer
but
more
important.
Metrics.
A
So
Who
am
I
I'm,
currently
working
for
Amazon,
eks
and
I
formerly
worked
at
a
couple,
Seattle
startups
porch
and
offer
up
if
you've
heard
of
them
so
quick
story.
The
reason
that
I
came
to
Seattle
was
actually
in
2015
I,
moved
up
to
work
at
porch
and
my
first
day
on
the
job.
My
colleague
mentioned
that
over
the
last
couple
days
he
had
moved
the
entire
production
website
onto
this
really
cool
new
technology,
which
I
had
never
heard
of
called
kubernetes.
So
I
was
like
okay,
that's
that
sounds
very
interesting.
A
I
did
a
quick,
Google
search
I
noticed
that
it
was
at
version,
zero
point
four
and
so
I
asked
him
about
it,
and
he
explained
about
at
the
time
replication
controllers
and
all
this
stuff,
and
then
he
mentioned
he's
like
oh
by
the
way.
Actually,
I
got
a
job
at
Google
and
my
last
days
on
Friday
so
have
fun
with
it.
A
So
that
was
my
very
exciting
introduction
to
kubernetes
and
it
went
well.
So
why
do
we
monitor?
This
is
fairly
self-explanatory,
but
obviously
the
most
important
thing
is.
We
need
to
be
able
to
detect
problems
as
they
happen
and
then
secondarily,
but
almost
as
important.
We
want
to
be
able
to
prevent
future
outages.
For
example,
if
you
have
a
disk
somewhere,
that's
on
its
way
to
filling
up
we'd
like
to
get
an
alert,
maybe
twelve
hours
before
it
fills
up
so
that
we
can
actually
do
something
about
it
or
you
know.
For
us.
A
We
would
like
to
know
if
a
data
center
is
running
out
of
capacity
and
somebody
needs
to
go
and
build
a
rack
before
we
can
before
our
customers
can
launch
more
clusters
and
then
another
reason
to
monitor
would
be
optimization.
So,
for
example,
if
you
want
to
understand
how
to
make
your
applications
more
efficient
and
potentially
save
on
cost
and
additionally
for
me,
because
I'm
incredibly
nosy
I
want
to
understand
why
Nick
told
me
that
his
service
was
going
to
have
a
certain
QPS,
but
it's
actually
a
hundred
times
more
than
he
promised
right.
A
But
monitoring
in
a
modern
micro
service
environment
is
actually
pretty
difficult.
A
lot
of
us
are
in
the
process
of
breaking
our
monoliths
up
into
many
micro
services
and,
as
we
do,
that
each
additional
micro
service
introduces
a
wealth
of
metrics
and
each
additional
micro
service
also
means.
We
have
all
these
complex
interactions
between
them
and
the
way
that
we
often
use
containers
and
kubernetes
environments
is
a
little
bit
more
transient.
So
looking
at
the
picture
at
the
VM
level
doesn't
tell
us
the
whole
story
anymore.
A
So
we
need
new
tools
and
then
something
that
Nick
is
going
to
touch
on
later
is
sort
of
the
organizational
aspects
of
monitoring
right
in
an
organization
where
you
have
many
different
service
teams
running
many
different
services.
They
have
different
requirements,
and
so
it's
hard
to
find
a
one-size-fits-all
strategy.
A
It's
helpful
to
approach
this
problem
with
a
method
and
I'm
sure
many
of
you
are
familiar
with
both
of
these,
but
I'll
reiterate
them.
So
a
common
way
of
thinking
about
monitoring
resources
is
the
used
method.
This
was
first
coined
by
Brandon
Gregg
and
it
states
for
every
resource.
We
want
to
check
utilization
saturation
and
errors
where
utilization
is
a
measurement
of
the
time
that
we
use
the
research
resource
over
the
total
time.
A
Where
and
saturation
is
more
of
an
instantaneous
measurement
of
use
where
we
could,
we
could
think
of
it
as
like
a
counter
and
Prometheus
or
an
integer
that
represents
maybe
the
number
of
processes
waiting
for
a
CPU
or
waiting
on
disk
and
then
the
other
methodology
is
the
read
method
which
was
introduced
by
Tom
Wilkie,
and
this
is
primarily
geared
towards
monitoring
web
services
and
it
states
for
every
service.
We
want
to
monitor
the
request
rate,
the
the
errors
and
the
duration
or
latency.
A
So
here's
a
graphic
of
what
a
kubernetes
cluster
or
metrics
environment
might
look
like
on
your
cluster.
So
we
have
two
worker
nodes
and
we
have
our
applications
themselves.
You
can
see
the
pods
running
there
in
this
case
we've
chosen
Prometheus
as
our
metrics
pipeline,
so
each
of
our
applications
is
instrumented
with
a
Prometheus,
endpoint
and
Prometheus
is
going
to
pull
the
metrics
from
there
and
since
we're
running
in
kubernetes,
we
also
have
a
cubelet
running
on
each
node,
which
has
C
advisor
embedded
in
it,
and
that
gives
us
very
rich
container
level.
Metrics.
A
We
also
have
the
metric
server,
which
is
part
of
the
kubernetes
core
metrics
pipeline.
So
that's
something
that
is,
gives
us
metrics
in
the
moment
on
pods
and
nodes,
so
CPU
and
memory,
something
that
can
be
used
by
consume
by
the
horizontal
pod
autoscaler
and
then.
Lastly,
we
have
cube
state
metrics
running
there
and
that's
something
that
actually
looks
at
the.
It
derives
metrics
off
the
entire
kubernetes
api.
A
So
we'll
quickly
talk
a
little
bit
about
monitoring
applications,
and
what
I
want
to
focus
on
is
how
we
pick
our
most
important
metrics
right.
What
do
we?
What
are
we
putting
at
the
top
of
our
dashboards?
And,
and
so
where
do
we
start
and
our
advice
is
start
with
your
users
and
if
you
can,
that
means
starting
with
your
business
metrics.
So
if
you're
an
online
bookseller,
it
might
be,
orders
fulfilled
successfully,
maybe
you're
a
rideshare
company,
so
it'd
be
rights,
completed
or
maybe
you're
a
company
that
launches
satellites
into
space
and
takes
pictures.
A
So
then
you
might
be
measuring
how
many
pictures
you
successfully
take
and
send
back
down
to
earth,
but
beyond
those
business,
specific
metrics.
What
comes
next
is
essentially
the
metrics
that
work
across
the
board.
So
this
is
where
we
go
back
to
the
read
method
for
monitoring
web
services,
and
these
can
be.
These
metrics
can
be
really
helpful
for,
for
example,
for
an
Operations
team.
A
It
could
lead
to
cascading
failures,
but
there's
some
supplementary
signals
like,
for
example,
one
of
the
most
frequent
causes
of
outages
is
ourselves
when
we
deploy
code,
so
something
that
you
can
do
really
easily
in
a
kubernetes
environment
is
just
attaching,
for
example,
a
version
label
which
I'm
sure
many
of
you
do
to
your
pods
your
deployments
and
then
understanding
as
your
deployment
progresses
throughout
your
clusters.
In
your
environments,
where,
for
example,
the
new
version
is
and
how
many
environments
is
progressed-
and
this
is
this
is
so-
we
can
actually
get
this
metric
from
cube.
A
Another
supplementary
signal
that
we
can
talk
about-
and
this
is
included
in
the
read
method-
is
request
rate
so
that
we
can
understand
when
our
applications
are
overloaded
by
our
users
and
then
saturation
as
well
and
I
want
to
touch
on
what
saturation
actually
means.
So
an
application
is
saturated
when
its
resources
are
saturated.
A
So
how
do
we
know
when
its
resources
are
saturated?
Well,
we
can
actually
get
if
we're
talking
about,
for
example,
CPU
memory
disk
network.
We
can
get
a
lot
of
these
metrics
from
cubelet
or
cube
state
metrics,
but
in
general
I
would
recommend,
starting
with
the
metrics
from
cubelet,
and
that's
because,
as
we
scale
up,
cubelet
is
still
responsible
for
the
pods
that
are
running
on
its
node.
A
So
it's
going
to
be
a
little
bit
more
efficient
in
that
sense,
whereas
cube
state
metrics
is
still
responsible
for
the
entire
cluster
and
driving
metrics
off
the
entire
clusters
state.
So
you
have
to
be
conscious
of
the
resources
that
keep
state
metrics
requires,
but
here
we
have
from
cubelet
from
the
first
from
the
embedded
sea
advisor
itself.
Actually
we
have
the
CPU
and
memory
additionally
there's
some
complementary
metrics,
for
example,
if
you
have
persistent
volumes,
you
can
get
some
interesting
metrics
off
that
and
to
touch
on
cube
state
metrics.
A
So
one
question
would
be:
how
is
monitoring
the
control
plane
different
from
monitoring
applications
and
actually
it's
very
similar,
a
lot
of
the
stuff
that
we've
learned
from
monitoring
applications
applies
to
monitoring.
There's
the
control
plane,
especially
the
API
server
right
we
can.
We
can
basically
treat
the
API
server
is
another
one
of
our
applications.
We
can
apply
the
read
method
to
it,
monitor,
request
rate
errors
in
duration
and
then
two
other
things
that
I
would
say
cover
the
bare
minimum
of
monitoring.
A
The
control
plane
would
be
just
a
general
sense
of
your
everything
running
in
your
cube
system.
Is
it
available?
Do
you
see
a
lot
of
container
restarts
and
then
I
would
also
look
at
SCD
specifically
and
just
make
sure
that
it's
you
know
it's.
The
cluster
is
healthy,
it
has
quorum
its
accepting
rights
and
you
don't
have
like
really
slow
disk.
That's
causing
you
know,
slow,
slow,
writes
and
client
timeouts,
so
that
I
would
say
that
covers
the
bare
minimum
honoring
the
control
claim
and
then,
as
you
start
to
scale
up
your
cluster.
A
The
first
thing
that
you
want
to
be
cognizant
of
is
your
API
server.
Starting
to
consume
more
and
more
CPU
and
memory,
so
you
might
watch
for
dropped
requests
which
manifests
as
429
s,
and
you
can
actually
adjust
the
number
of
requests
that
the
API
server
can
handle
simultaneously
with
the
max
requests
in-flight
flag.
A
But
you
have
to
be
cognizant
of
the
fact
that,
as
you
do
that
it's
going
to
take
more
resources,
so
you
have
to
in
kind
adjust
your
requests
and
limits
based
on
you
know,
your
note
count,
as
your
note
count
goes
up
and
your
pod
density
increases
and
your
objects
churn
increases
and
to
reduce
load
on
the
API
server.
You
can
bring
down
you.
Can
you
can
reduce
the
amount
that
your
API
server
clients
can
call
it,
but,
as
you
scale
up,
you
probably
won't
be
able
to
do
that.
A
You'll
probably
actually
have
to
go
the
opposite
direction
and
allow
them
to
do
more
work
so
so
to
those
clients.
Actually,
as
we
as
we
scale,
we
want
to
pay
attention
to
what's
going
on
with
the
scheduler
and
the
controller
manager.
One
issue
that
that
we
see
is
that
the
scheduler
is
going
to
slow
down,
as
you
have
more
pods
to
schedule
and
more
nodes
to
schedule
them
on.
So
there's
two
metrics.
A
Actually
one
of
them
exposed
from
cube
state
metrics
and
one
of
them
is
exposed
from
the
scheduler
which
tells
us
how
long
the
scheduler
is
taking
to
schedule
pods
and
then
the
controller
manager.
We
want
to
know
for
each
controller,
how
long
it's
taking
to
do
a
unit
of
work
and
how
long
that
unit
of
work
is
sitting
around
in
the
queue
before
it
gets
acted
on,
so
they
keep
weren't
there
is
we
really
just
want
to
for
those
control,
plane
components
at
a
minimum?
We
want
to
understand
how
well
they're
doing
their
key
jobs.
A
And
finally,
the
last
thing
that
I
want
to
talk
about
is
monitoring
sed,
so
it's
fairly
straightforward
that
CD
is
generally
healthy
when
it
has
a
quorum
and
it's
accepting
rights,
and
so
the
first
thing
to
look
at
is
leader
elections
and
just
to
understand
whether
or
not
your
sed
cluster
has
a
leader
and
whether
or
not
you're,
seeing
constant
leader
changes
that
can
signify
an
unhealthy
cluster
and
then
another
thing
that
can
lead
to
degraded
performance.
Is
you
know
your
disk
write
performance
so
how
long
it's
taking
at
CD
to
commit
rights?
A
A
Finally,
so
another
thing
to
keep
in
mind:
is
the
database
size
so
when
Etsy
DB
fills
up
its
quota
for
its
DB
file,
it's
going
to
trigger
a
no
space
alarm,
so
we're
gonna
watch
for
that
and,
additionally,
SCD
has
a
flag
to
where
it
will
periodically
check
for
corruption.
So,
in
terms
of
corruption,
we
can
let
@cd
sort
of
monitor
itself
and
with
that,
I'm
going
to
hand
over
to
my
co-presenter
nick
and
he's
going
to
talk
us
through
a
little
bit
about
how
planetlab
set
up
their
monitoring
infrastructure.
Hey.
B
B
So
my
team,
unfortunately
for
y'all,
we
work
on
the
boring
terrestrial
side
of
things.
There
is
a
team
that
manages
the
spacecraft.
We
do
computer
vision
on
top
of
the
imagery
300
million
square
kilometers
every
day.
That
is
a
lot
of
imagery.
It's
a
lot
to
look
at
with
your
human
eyes.
So,
my
organization,
our
greater
goal,
is
to
come
up
with
a
time
series
database.
We
call
it
the
queryable
earth.
B
So
if
you,
rather
than
looking
at
a
ton
of
imagery,
if
you
just
want
to
subscribe
and
see,
is
there
20
ships
container
ships
in
Seattle
today
and
30
tomorrow?
You
can
sort
of
subscribe
to
that,
so
our
organization
is
about
70
people
across
a
lot
of
time.
Servants,
tens
of
microservices
operational
teams
who
are
on-call,
Pharaon
services,
run
their
own
services
within
that
there
is
the
kubernetes
team
Hobbs
we're
a
tiger
team
originally,
and
we
got
that
name.
I
run
that
team
there's
five
of
us
and
we
do.
B
B
Can
you
have
it
ready
at
the
end
of
the
quarter
and
I
was
like
sure
how
hard
could
that
be
in
practice
a
little
over
a
year
later
we're
actually
between
about
a
hundred
and
four
hundred
nodes
at
any
particular
point
in
time
we
have
highly
elastic
workloads,
we
see
about
fifteen
thousand
requests
per
second
at
ingress
across
to
production
pastures.
We
run
on
Google,
compute
engine.
We
don't
use
DKE
because
reasons
we
probably
would,
if
we
sell
there
again
today,
but
we're
not
using
GK
at
the
moment.
B
So
the
every
different
organization
in
team
has
a
slightly
different
monitoring
philosophy.
I
think
it's
useful
to
talk
about
mine
before
I
talk
about
all
my
teams
before
I
talk
about
how
we
actually
monitoring
practice
the
planet.
So
first
we're
gonna
page
only
when
customers
are
affected,
you
may
notice.
There's
an
L
in
that
word.
At
Planet.
Costumers
is
synonymous
with
service
owners.
Cluster
mate
is
a
customer
of
our
clusters.
If
they
are
affected,
our
business
customers
might
be
affected,
and
that
would
be
bad.
B
So
one
example
of
this
is
when
we
first
set
up
a
kubernetes
clusters.
Core
OS
provided
prometheus,
although
manager
config
that
just
had
a
bunch
of
stock
alerts,
Fred
CD
and
one
of
those
alerts
would
page
you.
If
it
city
disk
latency
was
high
or
if
there
was
a
spike
in
it
and
that's
bad
that
could
lead
to
leader
election,
which
could
happening
all
the
time
which
can
lead
to
failed
Rises
Nick
was
talking
about,
but
then
we
found
that
we
get
page.
B
Four
in
the
middle
of
nine,
oh,
there
was
a
spike
in
Lindsey
and
went
away.
There
was
a
leader
election
and
the
cluster
kept
working,
so
we
don't
get
woken
up
for
that
anymore,
because
there
was
no
actual
impact
on
our
customers
so
sort
of
following.
Oh
that's
thing.
We
want
a
page
only
when
software
won't
fix
the
problem.
Kubernetes
and
most
cloud
providers
are
really
good.
B
They
work
really
hard
to
be
self-healing
so,
for
example,
in
kubernetes
you're,
all
probably
familiar
with
liveness
and
readiness
pervs
that
will
replace
unhealthy
pods
at
the
cloud
provider
level.
For
us,
we
health
check
our
instances
couplets.
So
if
a
couplet
is
unhealthy,
we'll
have
GCE,
delete
it
and
recreate
it,
and
we
don't
want
a
page
until
those
processes
have
had
a
chance
to
actually
fix
the
problem.
Another
more
concrete
example
of
this
is
that
there's
a
bug
that
was
recently
fixed
in
kubernetes
1.1
3.
B
That
would
cause
processes
to
get
stuck
in
uninterruptible
sleep
and
basically
that
would
cause
any
pods
depending
on
those
processes
or
containing
those
processes
to
be
stuck
for
a
long
time
or
forever.
So
we
used
to
get
paged
when
this
would
happen,
we'd
see
wedge
pods.
We
would
drain
the
node
that
they
were
on,
according
the
notes
of
indicate
new
pods
train
the
nodes,
so
that
healthy
pods
got
moved
away
successfully
or
cleanly,
and
then
to
leave
a
note.
B
I
got
really
repetitive,
so
we
wrote
a
piece
of
software
which
is
open
source
called
Drano,
which
uses
the
kubernetes
node
problem,
detector,
other
open
source
software
from
Google.
The
node
problem
detector
looks
at
signals
on
a
node,
for
instance
tailing
logs
or
running
arbitrary
scripts,
and
if
it
sees
something
that
considers
bad,
it
sets
a
node
condition
to
indicate
a
permanent
problem
with
the
node.
B
Finally,
what
a
page
only
the
team
who
can
fix
it,
so
my
team
is
on
call
for
kubernetes
infrastructure.
My
costume
is
were
on
call
for
their
services.
This
is
the
hardest
thing,
in
my
opinion,
if
we
see,
for
instance,
we
use
the
link
a
tea
service
mesh,
so
we
get
standardized
request
metrics
across
all
of
our
services.
B
On
the
other
hand,
if
I
see
that
10
different
services
have
had
an
increase
in
latency
or
increase
in
500,
so
there's
a
good
chance
that
something
to
do
with
my
infrastructure
and
I
want
to
get
paid
for
that.
I
wanna
fear
not
have
my
customers.
So
what
are
we
actually
monitor?
I
will
say
that
we
just
gonna
talk
about
sort
of
time
series
based
monitoring
here
we
do
use
distributed,
request
tracing.
We
do
use
structured
logging,
but
not
gonna.
B
Talk
about
that
in
this
talk,
so
this
is
at
a
glance
dashboard
or
the
top
part
of
it,
at
least,
if
you're
in
my
team
at
planet-
and
you
get
page
or
something.
Your
page
probably
has
a
link
to
this
dashboard
and
went
to
the
rotten
book
for
your
load.
There
are
other
dashboards
that
we
have
that
drill
down
deeper
into
the
cluster
autoscaler
health
coordinators,
health
control,
plane,
health
at
City,
Health,
etc,
etc.
B
This
is
typically
where
you
start,
and
this
is
structured
similar
to
what
Nick
described
before,
based
on
the
business
metrics,
that
we
have
given
that
many
different
things
running
my
cluster
and
they
have
different
actual
business
metrics
like
images
downloaded
and
whatnot.
We
go
for
that
lowest
common
denominator
of
the
read
method,
so
that
top
dashboard
is
powered
by
link
2d.
That
is
just
the
request
rate
error
rate
and
to
the
left
of
that
the
duration,
p99,
I,
think
or
p95
histogram
for
requests.
B
As
you
go
further
down,
we
have
a
lot
of
highly
elastic
sort
of
machine
learning,
workloads
that
cause
many
nodes
to
get
spun
up
and
spun
down.
We
actually,
rather
than
alerting
on
schedule
or
latency.
We
actually
just
all
work
on
the
total
time
that
pods
are
impending,
so
if
pods
are
impending
for
any
reason,
for
too
long,
for
instance,
nodes
can't
spit
up
or
whatever
will
be
alerted
to
fix
that,
but
we
also
graph
that
there
a
fun
fact
all
those
little
purple
triangles
beneath
that
dashboard.
That
is
Draenor
remediating,
broken
notes.
B
So
we
see
that
a
lot
and
Drano
comes
in
and
kicks
in
and
shoots
those
nodes
at
the
very
bottom
you'll
see
what
Nick
talked
about
before
we
show
you
the
running
software
versions.
This
cluster
was
running
pretty
much
all
the
same
set
of
software
versions
and
to
the
right
of
that
the
bottom
right
corner.
You
will
see
the
running
container
version,
so
we
can
correlate
if
there's
a
new
version
of
core
OS,
which
we
run
and
latency
goes
up.
A
B
Completely
get
rid
of
that
alert.
We
just
made
it
not.
Wake
people
up
hired
to
see
alerts
for
us
means
you
get
woken
up
in
the
morning.
You
fix
it.
Low
urgency
means
you
fix
us
some
time
during
your
on-call
shift
and
fixing.
It
might
just
mean
silencing
it
for
a
couple
of
months
and
putting
a
bug
in
to
look
at
it
at
some
point.
The
future
so
high
urgency
alerts.
B
Pretty
much
is
something
that
is
critical
to
the
cluster
broken
and
do
we
feel
that
Auto
remediation
hasn't
worked,
so
our
ingress
have
we
lost
ingress
nodes
and
they
didn't
come
back
and
we
expect
them
to
come
back
within
n
minutes
because
of
Auto
remediation
or
have
we
lost
control
pods,
you
know
core
DNS.
Is
that
any
all
about
control
pods
going
to
biston?
So
basically,
if
any
part
is
unhealthy
and
group
system,
then
that's
bad
and
we
we
load
on
that
after
we've,
given
kubernetes
a
chance
to
fix
it.
B
We
also
we
use
link
81,
so
we
use
that
service
mesh,
and
that
means
that
we
are
providing
as
a
service
to
all
of
our
customers.
So
we
will
also
page
if
we
see
average
errors
increasing
across
like
ten
different
services,
multiple
different
service
dimensions,
as
I
mentioned
before
low
urgency
alerts,
there's
a
lot
of
them.
Some
of
the
interesting
ones
are
if
we
lose
one
EDD
follower,
that's
low
urgency,
because
we
can
tolerate
two
failures,
so
we
lose
one
follower
low
urgency.
We
fix
it
that
way.
B
If
we
lose
two
followers,
one
more
means
the
clusters
bought.
So
we
page,
probably
the
most
noisy
and
also
the
most
useful.
Is
this
alert
that
we've
tuned
a
lot
about
pod
restarts
or
container
restarts
within
pod?
We
actually
just
we
used
to
try
new
fancy
things
now,
just
if
any
part
has
a
container
that
restarts
ten
times
we
page
low
agency,
it's
almost
never
something.
That's
actually
seriously
broken,
but
it's
very
often
a
sort
of
canary
for
underlying
problems
in
the
cluster
that
you
might
want
to
go
follow
up
on
later.
B
B
We
provide
the
monitoring
infrastructure
for
them
and
we
also
try
and
provide
some
standardized
metrics
and
those
come
from
the
lincolni
service
mesh
and
from
Cupido
days
they
mostly
come
from
cube
state
metrics
that
Nick
mentioned.
We
provide
them
with
the
count
upon
restarts
for
their
services.
We
provide
them
with
the
health
of
their
pods,
the
you
know
for
a
deployment
how
many
replicas
they
want
versus
how
many
they
have.
We
also
provide
them
concrete
resource
utilization,
which
is
semi
useful.
We
more
interesting
ly.
B
We
provide
them
CPU
and
memory,
etc
utilization
as
a
percentage
of
their
requests
and
limits,
so
they
can
tune.
Those
I
were
in
the
process
of
open
sourcing.
Some
software
called
Costanzo
that
will
actually
tell
you
roughly
how
much
your
services
cost,
rather
than
just
it's
using
ten
Zieve
use
its
using
fifty
dollars
a
month
or
something
like
that.
I
would
say.
The
main
value
here
is
just
that
these
requests
are
standardized.
So,
if
I
understand
how
my
service
works,
I
understand
how
your
service
works
from
a
large
monitoring
perspective.
B
A
B
Have
noticed
that
we
said
we
were
sort
of
talking
about
open
source,
and
that
was
actually
a
signal
effects.
Dashboard
that
I
showed
you
before
signal
fix.
Is
our
monitoring
vendor
we've
used
them
a
planet
for
a
long
time.
We
like
them
a
lot,
particularly
their
billing
model
works
well
for
us,
and
but
it
ties
to
cardinality.
Basically,
they
bill
you,
four
data
points
per
minute,
or
at
least
that's
what
they
bill
us
for.
B
So
if
you
have
three
time
series
and
you
sample
once
a
minute
you're
paying
for
three
data
points
per
minute,
when
we
first
set
up
our
naive
monitoring
implementation,
it
worked
great.
It
did
everything
that
we
wanted
for
and
we
were
sending
about
110,000
data
points
per
minute
to
signal
effects,
which
was
more
than
a
fifth
of
planet's
total
budget
for
monitoring
just
with
the
kubernetes
cluster
infrastructure,
not
even
the
things
running
on
the
clusters.
So
we
took
somewhat
of
a
novel
approach
to
fixing
this
and
we
basically
put
a
Prometheus
in
it
before
I.
B
Go
too
much
into
this.
I
want
to
say
that
this
is
a
controversial
pattern
within
the
curve
prometheus
community.
You
look
at
Prometheus
issues,
3902,
there's
a
lot
of
discussion
about
this,
but,
as
many
of
you
may
know,
Prometheus
supports
remote
right,
so
it
can
store
its
time
series
in
s3
or
GCS,
or
various
different
Atkins
and
signal
effects
implements
that
back-end.
So
we
can
use
Prometheus
to
basically
write
to
signal
effects.
B
Furthermore,
Prometheus
supports
recording
rules,
so
you
can
take
something
that
might
be,
let's
say,
HD
to
be
a
request
rate
across
100
pods,
broken
down
by
status
code.
That
might
be
hundreds
of
time
series.
We
average
that,
or
rather
we
sum
it
up
on
the
on
the
service
name,
and
we
just
sent
your
signal
effects.
B
B
You
have
to
kind
of
look
deep
when
your
service
teams
are
asking
for
availability,
Sariah
observability
patterns.
One
example
was
that
we
had
one
team
that
made
it
seem
like
the
most
critical
thing
there
are
all
could
do
was
get
distributed,
request
tracing
in
place,
so
we
got
it
in
place.
I
love,
it
I,
use
it
a
lot.
Almost
no
one
else
uses
it.
A
planet
they're
all
sort
of
stuck
in
this
sort
of
less
you
structured
logging.
Instead.
B
So
now
we
sort
of
pay
a
lot
more
attention
when
someone
asked
for
something
we
talked
to
everyone
and
like
hey.
Well,
you
all
really
find
this
useful.
Can
we
like
have
a
big
discussion?
Put
some
design
Doc's
out
there
find
what
what
everyone
actually
wants.
As
I
say
it's
hard
to
be
Prometheus
in
the
monitoring
space
for
kubernetes,
but
a
lot
of
people
shy
away
from
it,
because
you
have
to
run
your
own
alert
manager
and
you're
in
cavada,
you're
on
the
hook
for
like
really
important
stuff
within
your
org.
A
So
with
that,
thank
you
guys
for
coming,
appreciate
it
and
some
links
here.
You
can
find
us
on
Twitter
and
github,
and
some
of
the
open
source
projects
that
Nick
mentioned
are
linked
here
as
well,
so
we're
gonna
take
questions
after
the
talk
and
just
grab
us
come
talk
to
us
for
questions
feedback.
If
you
want
to
grab
beers,
that's
fine
too.
Thank
you.