►
From YouTube: Weighing a Cloud: Measuring Your Kubernetes Clusters - Han Kang, Google & Elana Hashman, Red Hat
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Weighing a Cloud: Measuring Your Kubernetes Clusters - Han Kang, Google & Elana Hashman, Red Hat
Kubernetes is complicated. Instrumenting it can be worse. Measuring the components of a distributed system shouldn't be as daunting as being asked to weigh a literal cloud. In this talk, we'll go over the components of a Kubernetes control-plane and show you where to look to figure out what is actually happening. We will show you common cluster issues and how they would look in your instrumentation, so that you can more effectively diagnose clusters. Starting in version 1.14, Kubernetes metrics were overhauled to provide consistent, high quality metrics. Han Kang and Elana Hashman will go over the changes and the potential ingestion implications of this overhaul and how it may affect you.
https://sched.co/UaeP
A
Hi
welcome
to
our
talk
yeah,
so
today
we're
going
to
be
talking
about
how
to
measure
your
kubernetes
components.
Briefly.
My
name
is
hon
king
I'm,
a
software
engineer,
I
lead
the
cluster
ops
team
at
Google,
which
is
a
part
of
the
broader
gke
API
machinery
team
in
open
source
I
contribute
to
both
sync
API
machinery,
instrumentation
and
I,
actually
wasn't
aware
that
I
still
had
a
Twitter,
but
Alayna
correctly
deduced
that
it
was
exactly
the
same
as
my
github
handle,
and
this
is
Elena
yeah.
B
B
So
what
are
we
going
to
cover
today?
I'll
start
out
by
introducing
how
instrumentation
works
in
kubernetes
and
then
Han
will
talk
about
the
various
components
of
kubernetes
control,
plane
and
dive
into
more
detail
on
their
instrumentation,
then
we'll
both
cover
some
real-world
debugging
examples
using
the
metrics
and
tools
we've
introduced
and
last
I'll
talk
a
little
bit
about
metrics
usability
and
the
sig
instrumentation
roadmap.
So,
let's
start
from
beginning
how
does
kubernetes
instrumentation
work,
kubernetes
components
integrate
with
Prometheus
a
time
series
based
open-source
monitoring
and
alerting
toolkit
your
instrument?
B
Is
software
will
use
the
Prometheus
client
to
export
metrics
in
Prometheus
format
on
some
endpoint,
which
then
can
then
be
scraped
and
stored
in
a
time
series
database
typically
you're,
scraping
mechanism
is
the
open
source
Prometheus
server,
but
there
are
other
cloud
and
vendor
tools
that
can
scrape
Prometheus
metrics
such
as
stack
driver
or
Azure,
monitor.
All
kubernetes
binaries
are
instrumented
using
Prometheus
clients
and
export
me
Prometheus
metrics,
by
default,
and
so
as
soon
as
you
set
up
a
metric
sync,
you
can
collect
metrics
on
your
cluster
health
without
any
additional
configuration.
B
So
what
do
Prometheus
metrics
look
like
each
time
series
has
a
name,
a
series
of
labels
with
values
and
an
overall
value.
In
this
example,
our
time
series
is
called
up.
It
has
a
label
of
job
with
label
value,
cube,
API
server
and
label
instance
with
label
value,
API,
1
and
the
overall
time
series
value
is
1,
presumably
because
our
API
server
is
up
time,
series
names
and
labels.
Let
us
query
and
drill
down
into
our
data.
B
Prometheus
metrics
will
typically
also
have
some
associated
help
text,
which
describes
what
the
metric
means,
as
well
as
a
type
annotation,
which
indicates
the
data
type
of
the
time
series
value.
There
are
4
metric
data
types
in
Prometheus,
first
counters,
whose
values
are
monotonically
increasing
integers,
as
their
name
suggests,
counters
are
typically
used
to
count
a
number
of
occurrences.
For
example,
number
of
requests
or
number
of
failures,
gages
represent
values
that
can
increase
or
decrease.
They
are
a
very
flexible
metric
type.
You
might
use
them
to
track.
B
Payload
sizes
or
record
Layton
sees
or
represent
uptime
like
in
our
example.
Time
series
here,
summaries
and
histograms
are
more
complex
data
types
that
allow
us
to
track
distributions
of
data
and
determine
performance
at
a
given.
Percentile
summaries
are
pre
calculated
for
specific
percentiles,
whereas
histograms
are
served
in
bucketed
form
for
later
aggregation
summaries
are
more
accurate
and
efficient,
but
can't
be
aggregated
across
many
data
sources.
B
For
example,
if
you
wanted
to
calculate
the
99th
percentile
latency
of
pod
launches
across
all
cubelets
in
a
cluster,
you
would
need
to
use
a
histogram,
not
a
summary,
from
a
combination
of
these
simple
data
types
of
labels.
Kubernetes
developers
have
built
metrics
for
each
component
that
we
can
collect
to
help
us
measure
the
health
of
our
cluster
when
I
quantify
the
quality
of
service
of
a
cluster
I
like
to
consider
the
dimensions
of
availability,
latency
and
capacity
of
my
service
as
well
as
any
errors.
What
does
this
look
like
in
practice?
B
As
an
example,
let's
consider
some
of
the
metrics
available
from
the
kubernetes
api
server.
If
we
want
to
measure
our
API
servers
availability,
we
can
look
at
the
uptime
series,
as
we
saw
in
the
last
slide,
which
is
a
gauge
with
a
value
of
1
when
the
server
is
online
and
zero.
When
it's
offline
to
get
an
idea
of
request
latency,
we
can
use
the
API
server
request,
latency
seconds
histogram,
which
allows
us
to
examine
the
distribution
of
server
request,
latency
to
collect
data
on
our
API
servers
capacity
and
throughput.
B
We
can
take
a
look
at
the
API
server
request
total
counter
from
that
metric.
We
can
derive
request
rates
and
get
a
picture
of
load
distribution
last.
We
might
want
to
collect
some
data
on
errors
in
our
API
servers.
Many
things
can
go
wrong
of
course,
and
what's
going
wrong
is
not
always
going
to
be
obvious
from
a
single
metric,
but
here's
a
counter
API
server
dropped
requests
total,
which
we
probably
want
to
stay
close
to
zero.
B
We
can
derive
the
rate
of
dropped,
requests
from
this
and
perhaps
build
an
alert
if
that
rate
exceeds
zero
for
some
period
of
time.
So
I've
talked
a
little
bit
about
what
Prometheus
metrics
kubernetes
exports,
but
not
how
to
use
them
without
diving
into
this
too
deeply
I
want
to
cover
some
use
cases
and
open
source
tools
that
you
can
use
to
aid
and
measuring
cluster
health
and
debugging.
The
Prometheus
query,
language,
prom
QL
can
be
used
for
metric
analysis
and
aggregation
and
we'll
see
some
examples
of
an
action
later
in
this
talk.
B
But
where
do
we
actually
write
our
queries?
If
you're,
using
a
Prometheus
server
back-end
to
prototype
and
for
one-off
queries,
you
can
use
a
Prometheus
web
UI,
which
is
deployed
alongside
the
Prometheus
server.
Once
you've
figured
out
some
stable,
useful
queries,
you
can
display
the
aggregated
data
in
permanent
dashboards
graph
on
ax
is
an
open
source,
dashboard
and
graph
editor,
that's
capable
of
accepting
Prometheus
as
a
data
source.
B
Now
an
individual
metric
is
never
going
to
tell
you
the
whole
story
on
your
cluster.
Lots
of
different
issues
may
manifest
the
same
way.
For
example,
there
are
many
reasons
why
a
node
might
go
offline,
but
in
most
cases
you
will
see
that
nodes
uptime
metric
drop
to
zero.
A
single
symptom
is
not
enough
information
to
form
a
diagnosis.
Maybe
the
node
went
offline
because
its
workload
got
too
high
and
it
can't
report
back
or
maybe
there
was
a
networking
issue.
B
Metrics
can
give
us
a
good
idea
of
how
something
is
failing,
but
not
necessarily
why,
by
exploring
multiple
metrics
and
data
sources,
we
can
pinpoint
the
specifics
of
a
failure
and
figure
out
the
why?
Even
if
metrics
don't
show
the
whole
picture,
they
can
guide
us
to
what
we
should
look
at
next.
If
a
node
is
down,
we
can
take
a
look
at
system
performance
metrics
and
see
if
there
are
any
anomalies
that
might
have
led
to
the
node
dropping
off
wine.
B
We
also
don't
need
to
limit
ourselves
to
metrics
other
tools
we
might
use
for.
Cluster
observability
include
log
files,
audit
logs
cluster
events
at
CD
dumps
and
more,
but
in
order
to
use
our
metrics
most
effectively,
we
need
to
understand
the
context
in
which
they
were
produced
without
a
mental
model
of
how
a
kubernetes
cluster
works.
It's
much
harder
to
pick
up
on
the
breadcrumb
trail
there
leading
us
down
to
illustrate
this
Han
made
me
this
chart
of
a
cube,
API
server
that
keeps
going
online
and
offline.
Hey,
wait!
A
minute!
B
Now,
seeing
this
chart,
you
may
be
worried
not
just
about
the
pun.
You
might
ask
me,
those
are
everything
in
a
kubernetes
cluster
depend
on
a
functioning
API
server.
This
looks
pretty
bad
and
sure
they
do,
but
it
turns
out
our
applications
doing
just
fine,
because
kubernetes
architecture
does
not
necessarily
require
a
functioning
control
plane.
In
order
for
the
data
plane
to
work,
there
are
more
questions
you
should
be
asking
me.
Is
this
the
only
API
server
in
the
cluster
or
are
there
multiple
running
and
a
high
availability
configuration?
B
A
Let's
start
with
a
little
bit
of
context
in
kubernetes,
we
basically
have
a
hub-and-spoke
model
of
network
communication,
which
means
that
point-to-point
interaction
is
not
allowed
between
components.
In
principle,
the
components
of
a
control
plane
are
neither
aware
nor
allowed
to
communicate
with
each
other
directly.
All
of
their
interaction
must
be
mediated
by
a
hub
which,
in
our
case,
is
going
to
be
the
cube
API
server.
On
a
slightly
orthogonal
dimension.
We
have
the
cubelet.
A
Let's
briefly,
go
over
the
cube
lit
model.
There
are
three
icons
so
on
the
left.
You'll
see
the
node
in
the
middle,
the
cubelet
and
on
the
right,
the
pods,
the
nodes
I
like
to
think
of
as
representing
some
sort
of
computational
resource.
These
can
be
VMs
or
even
physical
machines
on
the
right,
our
pods,
which
encapsulate
a
single
deployable
unit
in
kubernetes.
This
can
consist
of
a
container
or
multiple
containers
in
the
middle
there's
the
couplet
and
that's
the
daemon,
which
runs
on
the
node.
A
It
is
primarily
responsible
for
making
sure
that
the
pods,
which
are
supposed
to
be
running
are,
in
fact
up
and
healthy
in
most
master
configurations
and
by
master
I
mean
a
node
which
is
running
the
components
of
your
control
plane.
These
master
configurations
are
going
to
have
a
cubelet
running
your
master
components
as
pods,
and
because
of
this,
each
of
the
master
components
is
going
to
expose
a
health
endpoint
which
the
cubelet
can
monitor,
so
that
it
can
try
to
restart
the
pod
if
it
becomes
unhealthy.
A
We
can
also
rely
on
the
fact
that
we're
going
to
have
some
metrics
in
the
shape
and
the
format
that
Elena
just
described
and
we're
also
going
to
have
logs,
and
while
we
are
always
going
to
have
this
baseline
for
it
in
acting
the
control
plane,
not
all
kubernetes
control
components
are
going
to
be
equally
in
respectable.
Some
of
them
are
going
to
be
way
better
than
others
on
what
they
can
tell
you
about
your
cluster
or
its
operational
state.
A
Specifically,
what
I
have
in
mind
are
two
master
components:
the
cube,
API
server
and
sed,
and
by
no
means
am
I,
saying
that
the
controller
manager
and
scheduler
are
not
important.
They
clearly
are.
You
can't
have
a
functioning
kubernetes
control
plane
without
these
things,
but
the
fact
remains.
If
you
look
at
controller
manager
logs,
you
will
only
ever
see
controller
manager
things.
If
you
look
at
scheduler
logs,
you
will
only
ever
see
scheduler
things,
but
what
about
the
API
server?
A
Well,
since
the
API
server
is
the
hub
of
the
control
plane,
every
single
component
interaction
must
go
through
the
cube
API
server
to
talk
to
communicate
the
cube,
API
server
centralizes
those
interactions.
So
if
you
want
to
go
digging
for
things
which
have
happened
in
your
cluster,
this
is
going
to
be
a
terrific
place
to
start
so
how
do
components
communicate
with
API
server?
Well,
the
cube
api
server
provides
a
restful,
HTTP
api
that
the
human
the
components
will
interact
with,
and
we
can
actually
examine
this
by
adjusting
the
verbosity
of
any
cube
cuddle
command.
A
So
if
you
set
the
verbosity
of
a
cube,
cuttle
command
to
nine
or
ten,
you
can
see
curl,
you
can
see
the
output,
which
will
let
you
copy
and
paste
and
directly
curl
from
the
terminal,
but
that's
just
the
surface
of
cube
api
server,
int
respectability.
The
fact
is,
the
cube
api
server
is
so
central
increases
the
value
of
interest
backing
in.
Since
all
component
interactions
will
go
through
this
restful
api.
We
can
see
those
interactions
in
our
api
server
logs.
A
This
makes
the
cube
API
server
logs
a
terrific
source
of
data
to
mine,
since,
unlike
the
controller
manager
and
scheduler
logs,
where
we
only
see
controller
manager
and
scheduler
things
in
API
server
logs,
we
will
see
a
controller
manager,
things
scheduler
things
and
basically
anything
else
that
interacts
with
the
cube
API
server.
All
of
these
components
are
basically
clients
and
the
API
server
is
going
to
give
us
a
record
of
these
client
interactions.
A
In
fact,
we
could
often
find
various
component
interactions
in
the
metrics
as
well,
which
are
quite
rich
and
we're
going
to
see
some
of
these
in
our
examples
later
on.
We
also
have
health
endpoints,
as
you
can
see,
there's
three
of
them
and,
if
you're
wondering
why
we
have
so
many
that's
because
until
quite
recently,
the
cube
API
server
didn't
ambigú,
eight
between
cube
Ness
lightness
and
readiness,
probes
and
bundled
everything
up
into
a
single
endpoint
which
was
healthy.
As
of
116.
The
cube
API
server
supports
lightness
and
readiness,
as
distinct
health
checking
mechanisms.
A
The
cube,
API
server
can
also
be
introspective
using
audit
logs,
depending
on
your
cluster
configuration.
While
this
is
technically
a
security
feature,
our
logs
tend
to
be
a
terrific
source
of
request
information
because
they
are
structured,
unlike
our
cube,
api
server
logs.
So
if
you
want
to
query
about
request
information,
it's
going
to
make
your
life
a
lot
easier.
A
Our
next
incredible
source
of
cluster
information
comes
from
a
TD
that
CD
is
different
from
other
kubernetes
components
in
a
number
of
ways.
First,
it's
a
third-party
component
which
isn't
created
from
kubernetes
proper
and
it's
basically
vendored
it
for
those
of
you
who
don't
know
about
sedl.
Just
briefly
describe
it.
A
TD
is
the
persistence,
George
litter
for
kubernetes
control,
plane,
it's
a
distributed,
key-value
store
and
it
uses
the
raft
algorithm
to
provide
strongly
consistent
views
of
data
stored
across
its
members.
A
So
that's
what
I
like
to
call
the
elevator
description
for
a
TD
but,
more
importantly,
is
a
database.
It
stores
data
its
doors,
specifically
data
about
your
cluster.
So
if
you
want
to
look
at
what
happened
in
your
cluster,
this
is
going
to
be
a
tremendous
tremendously
powerful
tool
for
you
to
look
at
when
you're
trying
to
figure
out
what's
going
on.
A
So
here
are
some
ways
that
you
can
actually
look
at
sed.
The
first
two
items
in
the
list
are
tools
which
allow
you
to
directly
access
data
from
a
CD
at
CD.
Cuttle
is
the
one
people
are
probably
most
familiar
with,
which
is
a
client
command-line
client
for
interacting
with
SED.
It's
super
useful
for
ad-hoc
exploration
of
sed
data.
The
second
one
is
auger,
aha,
which
is
probably
a
lesser
known
tool,
but
it
shouldn't
be
it's
a
tool
written
by
one
of
my
friends
and
colleagues
at
Google
seems
to
draw
bets
and
auger.
A
The
thing
about
auger
is
that
it
can
interact
with
a
DB
file
directly.
This
is
super
useful,
because
this
means
you
can
query
for
data
without
actually
having
a
TD
running
and
that
ended
of
itself
is
valuable
enough
to
justify
using
it.
But
that's
really
not.
All
there
is
to
a
ger
auger
was
specifically
built
to
interact
with
kubernetes
at
CDs,
which
means
it
knows
how
to
decode
kubernetes
objects.
A
So
if
you've
ever
used
a
CD
cuddle
to
try
to
look
at
a
kubernetes
object,
you'll
will
see
that
you'll
end
up
with
a
screen
of
garbled
protobuf.
It's
basically
terrible
Algar
can
help.
You
actually
read
that
data
anyway
with
a
TD.
You
also
have
metrics,
which
has
a
ton
of
useful
stuff
like
compaction
and
snapshot
and
Linc's,
and
it
also
has
a
health
end
point
since
sed:
isn't
from
kubernetes
proper.
A
The
health
endpoint
doesn't
really
conform
to
the
same
format
and
you'll
get
JSON
output
back
from
the
south
end
point
and
everything
else
in
the
curie
relays.
Control
plan
kicks
back
plain
text.
You
can
also
look
at
the
logs,
which
has
its
own
unique
way
of
marking
sub
levels.
It's
going
to
look
a
bit
different
than
your
other
kubernetes
components.
B
So
Han
and
I
will
show
you
a
number
of
different
examples
to
showcase
how
we
can
use
the
different
metrics
and
tools.
We've
shown
you
in
order
to
debug
some
common
cluster
issues.
I
mentioned
the
scenario
number
of
times,
because
it's
just
so
common.
So
let's
work
through
this
problem,
a
node
has
gone
offline.
I
might
find
out
about
this
in
a
number
of
different
ways:
I
see
that
a
node
is
reporting
not
ready,
or
maybe
the
pod
scheduled
on
a
particular
node
are
showing
unknown
status.
How
might
we
detect
this?
B
Given
the
metrics
we
have
available?
Well,
here's
a
sort
of
obvious
solution.
Look
for
all
the
nodes
that
Prometheus
failed
to
scream.
Here.
We
use
our
favorite
Prometheus
time
series
up
since
one
indicates
a
scrape
target
is
on
line
this
query.
It
detects
all
nodes
that
are
offline.
We
could
use
this
query
to
build
automation,
to
check
for
nodes
in
the
state
for
some
period
of
time
and
then
perhaps
automatically
reboot
them.
So
a
human
doesn't
have
to
intervene
on
node
failures
or
really
think
about
them
at
all.
B
So
we
can
isolate
that
as
a
likely
source
of
the
problem,
but
the
node
hasn't
gone
offline,
so
it
wouldn't
show
up
in
our
previous
query
and
we're
not
quite
sure
what's
wrong
poking
around
some
of
the
Prometheus
metrics
I
noticed
that
commonly
when
a
cubelet
is
in
a
gray
failure
state
and
is
acting
slow,
it's
slow
for
everything,
not
just
pods
and
users,
so
this
shows
up
in
our
Prometheus
scrape
duration
data.
In
this
case,
we
can
detect
such
problems
by
looking
at
unusually
high
scrape
time.
B
Typically,
it
takes
under
a
second
to
scrape
data
for
a
node.
So
if
the
node
metrics
endpoint
is
regularly
taking
more
than
two
seconds
to
respond,
there's
probably
something
a
little
off.
This
query
can
help
flag
nodes
that
are
not
behaving
optimally.
Both
of
these
examples
used
knowledge
of
how
Prometheus
works
and
it's
odd
an
automatically
generated
time
series
to
help
us
debug
cluster
issues.
Prometheus
knows
whether
or
not
it
can
successfully
scrape
a
target,
and
it
knows
how
long
it
took
to
scrape
that
target.
So
we
can
use
that
information
to
our
advantage
here.
B
A
Okay,
so
now
we're
gonna
go
over
an
example
which
is
quite
near
and
dear
to
my
heart,
because
it's
emerged
out
of
the
sweat
blood
in
tears,
of
not
just
myself
but
a
number
of
Engineers
over
Google,
not
just
once,
but
a
number
of
times
in
varying
incarnations,
so
we're
going
to
start
with
a
crash
lipping
api
server.
So
this
is
a
pretty
common
failure,
mode
which
we
should
realize
at
this
point,
because
we
know
that
the
couplet
is
going
to
try
to
restart
broken
containers.
A
So
how
are
we
going
to
figure
out
that
this
is
happening?
Hopefully,
we
can
become
aware
of
this
before
someone
tells
us
that
it's
happening
so
one
option
since
the
cube,
API
server
exposes
health,
a
public
API,
it's
possible
to
set
up
uptime
checks
against
the
help
endpoints
to
generate
some
picture
of
availability.
This
is
pretty
common.
A
Alternatively,
we
can
ingest
metrics
from
the
master
cubelet,
which
exposes
counters
for
liveness
and
readiness
probes
of
the
containers
of
the
pods.
It's
watching
in
this
case
the
master
cubelet,
is
watching
our
QB
API
server.
It's
going
to
look
something
like
this
and
as
a
disclaimer
I've
stripped
out
some
of
the
labels
to
make
it
easier
to
parse
on
screen.
A
So
at
this
point,
we've
detected
that
our
API
servers
crash
looping,
but
we
need
to
ambigú
eight
between
one
of
two
situations.
So
is
the
cubelet
to
start
a
container
when
it
sees
that
the
container
isn't
running,
or
is
it
killing
the
process
and
trying
to
restart
it?
In
the
first
case,
the
cubelet
is
just
in
repair
mode.
It's
it
sees
that
container,
which
is
supposed
to
be
running,
isn't
so
it
just
starts
the
container
initialization
process.
In
the
second
case,
the
cubelet
is
killing
the
process
in
the
in
the
case
of
a
master
node.
A
This
means
that
the
liveness
pope
has
failed
enough
times
consecutively
to
trigger
a
cubelet
restart.
So
how
can
we
figure
out
whether
it's
the
former
or
the
latter?
There
are
a
number
of
different
ways.
We
can
look
at
the
failure
account
in
our
metrics
probes,
endpoint
that
we
looked
at
earlier.
Possibly
we
can
verify
this
by
examining
the
cuba
logs
and
because
it
will
tell
you
if,
in
fact
the
cubelet
did
drive
a
restart.
A
In
our
case,
it's
going
to
turn
out
that
the
liveness
it
was
a
failure
in
the
liveness
endpoint
we
did
have
consecutive
failures
and
the
cubelet
was
actively
killing
our
API
server
process
and
restarting
it.
We
can
further
investigate
by
curling
our
API
server
health
check,
endpoint
and
passing
an
a
verbose
flag.
A
This
allows
us
to
see
all
of
the
individual
health
checks
are
configured
for
this
health
endpoint
and,
as
you
can
see
here,
the
API
server
health
endpoint
is
comprised
of
a
number
of
more
granular
health
checks
and
I
bolded,
the
one
which
is
failing,
which
is
the
sed
health
check.
Since
the
cubelet
is
using
this
endpoint
for
aliveness
probing.
That
basically
means
if
a
TD
is
not
healthy.
Our
API
server
is
actually
going
to
be
crash
living,
so
it
turns
out
that
our
example
isn't
actually
an
API
server
issue.
A
After
all,
it's
an
sed
issues
and
the
slides
from
this
point
will
reflect
that.
But
now
that
we
know
it's
an
sed
issue,
let's
dig
one
of
the
first
things:
I
do
when
I
am
debugging.
An
sed
issue
is,
interestingly
enough.
I,
look
a
cube,
API
server
metric,
specifically
I.
Look
at
the
STD
object
metric.
This
metric
will
tell
you
the
number
of
objects
that
exist
for
is
given
kubernetes
resource
and
believe
me,
when
I
tell
you,
it
is
not
uncommon
for
random
third-party
libraries
that
you
install
in
your
cluster
to
create
a
bazillion
objects.
A
So
in
this
particular
example,
there
are
a
million
such
objects,
and
this
can
be
more
or
less
problematic,
depending
on
the
size
of
the
object.
Let's
say
that
the
average
object
size
for
this
CRD
is
about
one
kilobyte
doing
some
math.
If
we
have
a
million
such
objects,
that's
going
to
take
about
a
gig
of
space
if
the
the
average
size
is
larger
than
that,
we
really
have
a
problem.
Why?
A
Recently
we
ran
into
a
far
more
insidious
variant
of
this
issue.
So
let's
take
a
look
at
these
two
metrics.
The
first
one
is
the
sed
count
metric
and
you
can
see
that
there's
only
one
instance
of
this
object
that
exists.
The
second
one
gives
us
a
count
of
the
number
of
updates
for
that
specific
resource.
So
there
is
one
object,
but
it
has
been
updated
1200
times.
So
why
is
this
a
problem?
So
if
we
want
to
correctly
model
this
in
our
heads,
we
have
to
take
a
step
back.
A
A
But
sed
is
a
database.
It
doesn't
store
just
one
object:
it's
a
key
value
store.
It
stores
an
object
for
each
key.
There
are
many
objects,
so
we
can
see
that
we
have
this
dimension,
but
if
we
were
to
take
the
average
size
of
objects
and
multiply
it
by
the
object
count,
would
that
give
us
an
approximate
size
of
the
data
used
by
a
CD?
A
So
if
we
revisit
the
metrics
from
the
last
slide,
we
can
see
that
while
there's
actually
only
one
instance
of
that
object,
that
1,200
writes
to
that
object
means
that
there's
actually
1200
versions
of
that
object.
So
it's
quite
a
bit
different
than
what
we
initially
thought.
So,
even
though
we
have
1,200
objects,
we
also
now
have
to
consider
the
size.
So
let's
just
consider,
for
instance,
that
the
size
of
this
object
is
large,
say
200
kilobytes.
A
Let's
do
some
math,
so
200
kilobytes,
let's
say
API
servers
crash
lipping
once
every
minute,
so
we
have
1200
writes
in
a
minute.
That's
roughly
20
writes
a
second.
So
if
the
size
of
the
object
is
200
kilobytes,
then
that's
4
megabytes
a
second
or
two
hundred
forty
megabytes,
a
minute
which
is
about
a
quarter
Giga
minute
now:
SCD
compacts,
every
5
minutes.
So
by
the
time
compaction
occurs.
A
A
So
then
what
we
really
need
to
be
able
to
do
is
ballpark
the
size
of
the
object.
If
you
can
catch
the
API
server,
while
it's
healthy,
you
can
use
cube
cuddle,
but
in
our
case
Etsy
D
is
actually
not
healthy
and
the
cube
API
server
is
not
healthy.
So
this
is
when
it
becomes
super
useful
to
use
augur
because
you
can
operate
on
the
bolt
DB
file
itself
and
that's
how
you
would
approximate
the
size
of
an
object
using
augur.
A
So
in
the
nasty
variant
of
this
case
that
we
encountered
recently,
it
turned
out
that
this
custom
resource
had
a
number
of
base64
encoded
images
embedded
into
the
annotations
field
which
drastically
bloated
the
size
of
this
object.
There
was
also
a
custom
controller
which
was
going
rogue
updating
the
object
due
to
a
bug
in
the
Equality
comparison
operation
for
the
CRT.
A
Basically,
there
was
an
an
unordered
list,
which
was
always
returning
as
unequal,
so
the
controller
would
make
an
update
request
because
it
thought
the
object
had
changed,
so
this
high
right
rate
led
us
to
bump
into
IO
throttling
limits,
but
we
were
able
to
mitigate
this
by
contacting
the
CR
the
owners
and
getting
them
to
reduce
the
size
of
the
images
that
they
were
storing.
So
the
moral
of
the
story
probably
don't
store
images
in
your
CR
DS,
if
you,
if
you
have
to
try
to
make
them
as
small
as
possible.
A
B
Last
example,
so
last
fun
problem
for
a
period
of
time,
I
had
an
issue
with
API
server
access
being
slow,
both
via
cube,
cuddle
and
API
calls.
Now
the
API
server
collected
metrics
for
this,
so
I
figured
I'd
dig
into
that
to
see
if
there
was
any
wisdom,
I
might
gain.
Perhaps
the
particular
verb
was
slow
like
poster
patch,
and
that
might
indicate
some
issue
with
write
performance.
So
this
query
allows
us
to
plot
out
HTTP
call
Layton
sees
grouped
by
HTTP
verb
and
see
if
there's
anything
particularly
slow.
B
So
here's
that
query
applauded
in
Griffin
against
a
production
cluster
representing
p99
Layton
sees
of
the
API
servers
by
HTTP
verb
over
a
24-hour
period.
The
Green
Line
is
a
marker
for
5
seconds,
and
this
graph
has
a
linear
scale.
It
looks
like
there
was
a
big
spike
in
post
latency
towards
the
left.
Well,
over
5
seconds
now
you
may
notice
something
a
little
fishy
about
this
data.
B
If
I
look
at
the
Max
and
current
values
of
most
of
the
verbs
and
the
legend
on
the
right,
the
current
HTTP
latency
clusters
just
below
125
milliseconds,
that's
pretty
slow
for
an
HTTP
call
and
also
kind
of
weird
number.
The
second
thing
you
may
notice
is
the
maximum
recorded
HTTP
call.
Latency
is
exactly
eight
seconds
again.
B
I
helped
write.
So,
while
that
query
didn't
help
me
pinpoint
the
source
of
the
API
server
slowness,
it
did
help
me
confirm
that
things
were
slow
and
I
was
also
able
to
confirm
the
issue
and
suggest
improvements
upstream,
which
is
a
good
opportunity
for
me
to
jump
into
talking
a
little
more
about
upstream.
In
our
last
example,
we
saw
there
was
an
underlying
issue
with
a
metric
that
was
providing
bad
data
to
cluster
operators
and
made
it
difficult
for
us
to
figure
out
what
was
going
on.
How
does
the
instrumentation
address
these
sorts
of
problems?
B
Well,
it's
a
bit
complicated
updating,
metrics
between
releases
could
break
monitoring
stacks,
for
example,
fixing
the
API
server
latency
buckets
meant
that
users
would
potentially
see
wildly
different
performance
metrics
between
a
113
and
114
upgrade
despite
the
actual
performance,
not
necessarily
changing.
This
could
result
in
anomaly
detection,
throwing
incorrect
alerts
between
releases,
and
there
are
other
issues
beyond
the
metric
data
itself.
B
If
a
new
metric
is
added
that
causes
a
memory
leak
or
generates
too
many
time,
series
there's
no
real
way
to
disable
it,
meaning
a
full
bug,
fix
and
cluster
upgrade
rollout
needs
to
be
performed
when
bad
metrics
are
discovered.
So
how
can
we
coordinate
developers
to
address
this
and
sure
that
end-users
are
fully
informed
about
these
sorts
of
problems?
B
Cygnus
fermentation
has
spent
a
lot
of
time
identifying
and
cataloging
broken
metrics
like
in
our
API
bucketing
example,
but
that's
not
the
only
one.
For
example,
the
labels
on
C
advisor
metrics
did
not
match
the
kubernetes
instrumentation
guidelines,
so
they
couldn't
be
joined
with
other
kubernetes
metrics
like
those
exported
by
Cuba
State
metrics.
There
were
also
a
number
of
time
series
that
use
the
wrong
data
types.
B
For
example,
cubelets
were
only
exporting
spod
start
time
in
summary,
form,
meaning
you
could
not
aggravate
pod
start
times
across
a
cluster
in
order
to
rank
pod
latencies
by
percentile.
You
need
a
histogram
for
that.
There
were
also
issues
with
unit
standardization,
some
time
series
run
in
microseconds.
Others
were
in
seconds
and
in
some
pretty
bad
cases
they
claimed
they
were
in
seconds
when
they
actually
were
in
microseconds.
B
This
did
not
conform
with
from
easiest
and
kubernetes
best
practices
for
unit
normalization,
so
we
put
together
a
cap
detailing
the
problems
and
we
fixed
it
most
of
those
fixes
rolled
out
in
the
114
release.
Now
for
some
metrics,
we
could
for
follow
an
informal
deprecation
policy,
for
example,
marking
the
time
series
with
non-standard
units
as
deprecated
in
their
help
text
and
in
the
code
and
adding
brand
new
time
series
with
the
correct
units,
but
others
like
API
server
bucketing
needs
to
be
updated
directly.
This
might
cause
confusion
to
end
users
of
those
metrics.
B
So
how
do
we
communicate
that
those
metrics
are
deprecated
and
ensure
users
are
able
to
smoothly
transition
off
them
for
these
purposes?
Sig
instrumentation
has
developed
a
metric
stability
framework
and
we're
working
to
transition,
all
kubernetes
control,
plane
metrics.
To
use
that
framework.
The
metric
stability
proposal
will
allow
us
to
treat
metrics
as
a
proper
API
and
with
an
enforced,
automated
multi
release
notice
period
for
changes
to
stable
metrics.
B
It
will
also
allow
us
to
slowly
phase
out
obsolete
metrics
across
releases
before
deletion
and,
most
importantly,
it
will
provide
tooling
to
enforce
stability,
for
example
via
static
analysis.
In
practice,
what
this
means
is:
we've
replaced
all
directs
use
of
Prometheus
metrics
with
a
component
base
map
wrapper,
which
allows
us
to
annotate
metrics
with
a
stability
level
and
deprecation
status
like
you
can
see
in
the
on
the
screen.
B
So
it's
a
great
first
step
that
we
now
have
the
capability
to
mark
metrics
as
stable
or
deprecated,
but
there's
still
a
bunch
more
implementation
to
come.
Now
that
we
have
the
metric
stability
framework
in
place,
we
need
to
decide
what
metrics
should
actually
be
considered
stable
and
to
accomplish
that
we
need
to
develop
criteria
around
what
constitutes
a
stable
metric
and
select
some
metrics
to
be
promoted.
To
statement.
B
We'd
also
like
to
add
flags
to
the
various
kubernetes
components
to
allow
for
disabling
individual
metrics
at
runtime,
so
that
if
a
new
alpha
metric
causes
stability
issues
by
leaking
memory
or
having
cardinality
explosions,
we
can
just
turn
it
off.
Rather
than
having
to
do
a
full
bug
fix
rollout.
You
may
have
noticed
we
didn't
discuss
tracing
in
this
talk
and
there's
a
sig
instrumentation
cap
under
consideration
for
adding
distributed
tracing,
support,
kubernetes
components
and
of
lot
as
of
last
Friday.
B
A
new
cap
was
also
proposed
to
add
structured
logs
to
all
kubernetes
components,
which
is
a
thing
I
have
been
asking
for
for
a
year
and
finally,
we
have
more
metric
improvements
in
the
pipeline
as
we
continue
to
identify
suboptimal,
metrics
and
iterate
on
them.
So
if
you
want
to
learn
more
about
any
of
this,
we
invite
you
to
intend
the
cig,
instrumentation,
intro
and
deep
dive
sessions
which
are
scheduled
in
the
next
slot
today
and
the
same
timeslot
tomorrow
and
I.