►
From YouTube: Episode 38: Custom Resources in kube-state-metrics
Description
Hey, come and join Christian Schlotter and Xinqi to explore a new feature in kube-state-metrics 2.6 that will allow for any component that uses CRDs to be observable in a cloud native way!!
A
A
Okay,
that's
awesome
and
for
those
who
watch
our
show
for
the
first
time
we
have
like
us
a
YouTube
channel,
and
you
can
find
our
previous
episode
there
and
they're
all
about
like
kubernetes
networking
or
like
tons
of
product
and
the
awesome
things
we
have
here
at
VMware.
So
if
you
are
interested
in
our
Channel,
please
consider
subscribing
it.
B
Yeah,
the
time
of
today
is
customer
resources
in
Cube
State
metrics.
Maybe
we
should
first
talk
about
what
Catholic
resources
and
Cube
stick.
Metrics
is
right,
so
still
a
short
introduction
for
custom
resources.
Basically,
they
are
some
kind
of
extension
to
the
kubernetes
API,
mostly
known
as
some
resources
like
in
cluster
API.
B
B
It's
the
current
state,
you
normally
see
in
you
also
see
using
Cube
cuddle,
but
by
using
the
cube,
stick
metrics
and
you
can
store
the
information
in
some
kind
of
database
and
you
will
be
able
to
query
the
data
on
later
times
to
yeah
and
also
we
will
be
able
to
create
some
alerts.
If
something
is
wrong
with
the
state
right,
yeah.
B
A
B
Yeah
so
yeah
yeah,
so
how
do
you
consume?
Cube,
State
metrics
may
be
also
interesting.
Basically,
Cube
save
metrics
only
exposes
its
information
by
an
HTTP
endpoint
and
you
actually
consume
it
using
Prometheus
later
on.
So
Prometheus
is,
in
this
case
the
database,
which
takes
a
look
at
cubic
metrics
in
an
interval
like
30
seconds
or
every
minute
and
grabs
the
current
state
and
stores
it
into
its
database.
B
So
if
you
take
a
look
at
Prometheus
here,
that's
like
a
Prometheus
I've
got
running
here
in
my
local
system
and
I'm
able
to
query
all
the
data
which
is
stored
in
there
and
if
we
take
a
look
at
at
what's
there,
there
are
a
lot
of
metrics
which
have
the
graphics
Cube,
which
are
mostly
exposed
by
cubic
metrics.
In
this
case,
like
redeem
sets,
there's
a
metric
when
a
demon
set
was
created
so
yeah.
Basically,
that's
the
idea
of
cubic
metrics.
B
Yeah
here
we
can
query
all
the
metrics
Q,
State,
metrics
or
other
yeah.
Other
information
Prometheus
gets
from
other
long
applications
and
query
the
data
and
take
a
look
at
it
and
also
yeah
have
graphs
and
take
a
look
at
it
over
time,
and
so,
for
example,
if
we
have
a
deployment
and
at
the
next
day
you
see
yeah
something
was
broken
tonight.
You
maybe
can
use
the
metrics
to
get
a
Canon
of
an
idea.
What
happened
like
you
could
maybe
see
that
the
Pod
was
restarted
or
something
like
that.
B
Yeah,
okay
yeah,
so
it's
pretty
well
known,
I,
think
in
the
kubernetes
space
to
use
Cube
statements
metrics
to
have
additional
alerts
on
top
of
kubernetes
for
cluster
operations,
yeah
and
so
maybe
to
to
get
to
the
exciting
new
stuff.
Cube
State
Matrix
provides,
as
I
said,
Cube
State
metrics
provides
lots
of
information
or
lots
of
metrics
for
the
core
kubernetes
resources
and
since
the
latest
release
of
cube,
State
metrics
or
the
latest
two
minor
releases,
it's
also
possible
to
have
some
custom
configuration
file
for
cube.
B
Yeah
so,
but
what
could
it
be
used
for
like
if
you
think
of
cluster
API,
you
have
resources
which
describe
the
cluster,
and
you
know
the
state
of
your
cluster
and
by
using
a
custom
configuration
you
can
also
have
metrics
for
these
resources.
Maybe
I
can
show
some
example
here.
Right,
hey
go
to
the
Rochelle
I
have
here
some
development,
cluster
API
running
and
I
can
use
Cube
puzzle
to
actually
it
gets
the
cluster
resource,
for
example,
and
there
I
can
see.
B
B
B
Yeah,
so
maybe
to
take
a
look
at
at
how
this
works
I
know
we
know
we
now
know
we
have
here
some
kind
of
class
resources
and
if
I
take
a
look
at
the
observability
mainspace
here
and
at
the
parts
which
are
running
I
can
see
if
I
have
here,
Prometheus
running
and
also
Cube
stick
metrics
and
on
top
there's
also
grafana,
which
could
be
used
to
create
dashboards.
But
we'll
take
a
look
in
that
into
that
shortly.
B
B
So,
basically
like
for
lots
of
applications,
there's
configuration
for
Prometheus
and
you
can
do
some
big
static
configuration
here
and
two
lots
of
kind
of
things
like
in
scrape
conflicts.
You
can
Define
to
a
reference
to
some
HTTP,
endpoint
and
Prometheus
will
continuously
grab
the
data
from
this
HTTP
endpoint,
specifying
all
this
kind
of
endpoints
could
get
horrible
if
you
have
lots
of
applications
running
in
kubernetes
and
that's
why
Prometheus
also
provides
some
Auto
Discovery
to
detect.
B
B
A
B
If
it
does
not
exist
or
if
it's
too
false
or
some
other
value,
Prometheus
the
the
search
discovery
of
Prometheus
will
say:
okay,
that's
not
something
for
me!
I
don't
have
to
take
a
look
at
it
but
best,
but
by
setting
it
to
True
here
it
knows:
okay,
I
just
take
whatever
is
defined
here
in
ports
and
also
adds
the
slash
metrics
path
to
the
URL
and
to
gather
all
the
metrics
and
yeah
yeah.
B
B
Yeah
maybe
some
background
on
that.
There's
something
called
the
open,
metrics
specification,
which
is
a
big
specification
which
came
from
or
some
wave
together
with
the
Prometheus
project,
which
defines
the
way
how
this
is
done.
So
that's
a
kind
of
a
standard
way
to
expose
metrics
for
in
this
case,
for
example,
Prometheus.
B
B
Let's
add
asked
so
we
don't
get
some
garbage
output
and
there
we
can
see,
there's
a
plain
text
format
following
a
specific
yeah
way,
how
it
should
look
like
and,
for
example,
there
for
for
every
metric.
We
have
some
kind
of
type
and
some
help
information,
so
we
can
yeah.
We
know
that
this
is
a
Gauss
metric
here
and
follow.
The
type
is
followed
by
the
name
of
the
actual
metric
and
same
for
the
help.
Information.
A
B
And
the,
for
example,
the
cube
certificate.
Signing
requests
label
has
this
kind
of
Hub
information
but
yeah,
and
so
that's
the
type
and
help
information,
for
example,
and
there
are
also
entries
with
having
data
like,
for
example.
Here
we
have
a
custom
metric
called
copy
cluster
info,
which
is
of
of
the
type
gosh
and
the
entry
for
the
metric
looks
like
this.
It's
the
name
of
the
metric
and
in
curly
braces,
there's
some
labeling
information.
B
You
can
think
of
it
like
being
some
text
information,
a
key
value
pair
and
at
the
end
zone,
there's
also
a
value
for
for
it.
So
in
this
case
the
metric
has
the
value
of
one,
but
it
could
also
be
zero
or
any
other
kind
of
floating
Point
yeah,
but
you
like
you,
could
have
exposed
by
that.
The
creation
timestamp
the
Unix
timestamp
of
the
creation
of
something
or
have
a
counter
like
how
many
requests
I
had
or
something
like
that.
B
B
A
B
Yeah
exactly
for
all
the
cube
entries
here,
the
information
which
is
exposed
is
hard
coded
into
Cube,
State
metrics.
So
that's
go
code
which
defines
what
what
is
exposed
to
follow
metrics,
which
have
the
prefix
copy.
In
this
case,
all
this
information
comes
from
what
I
Define
in
my
configuration
file
for
my
cluster
API
cluster
metrics
in
this
case
yeah.
So
as
we
can
see,
there's
some
kind
of
topology
version
here
and
maybe
I
can
do
again.
B
The
your
cattle
get
clusters
requests
and
we
can
see
here
this
entry
Max
to
the
second
entry
here,
which
also
yeah
shows
you
the
same.
It
shows
exactly
the
version
which
is
shown
here
or
you
you
can
see
in
Cube.
Cuddle
gets
oeml,
there's
somewhere
in
the
yellow,
there's
this
information,
and
you
can
also
see
the
name
of
the
object
here
and
in
this
case
the
namespace
is
not
shown
because
yeah
of
the
cube
cuddle
output
but
yeah,
basically
that
Maps
one
to
one
onto
the
metric
here.
B
Yes,
you
can
think
of
cube,
save
metrics,
that's
the
same
as
we
do
when
we
do
Cube
cuddler
gets
to
the
objects
or
sorry
for
that.
If
you
could
look
at
cluster,
which
oh
ML,
and
to
expose
information,
we
defined
as
labels
here
and
if
we
take
a
look
here,
there's
somewhere
in
the
J
in
the
yaml
output.
There's
this
version
information
which
gets
exposed
back
Cube,
State,
metrics.
B
B
B
Yeah,
so
let's
just
take
a
look
at
at
maybe
this
metric
here
I
have
a
big
configuration
file,
because
there
are
lots
of
metrics
that
I
want
to
expose
for
my
custom
resource
definitions.
So
this
is
a
pretty
big
file.
I
collapsed
some
parts
here,
so
there
you
can
see.
I
have
some
additional
entries
here,
but
maybe
to
take
just
a
look
at
the
cluster
object
for
now.
So
this
is
the
configuration
file
for
the
cube,
State
metrics.
B
B
So
this
this
kind
of
resources
area
where
I
can
Define
different
kind
of
API
resources
to
or
in
or
normally
custom
resources
to
expose
metrics.
For
so
you
can
have
one
entry
for
each
group
version
kind,
pair
or
or
key,
and
in
this
case
we
have
the
cluster.
We
refer
to
the
cluster
CRT
of
the
group
cluster
X
Kate's
IO,
the
version
V1
beta1.
B
We
can
also
see
it's
the
same
when
doing
Cube
pedal
gets
here.
It's
also
the
same
here.
Let
me
scroll
a
bit
up,
so
this
Maps
exactly
to
the
API
version
and
client.
We
get
using
Cube
cuddle
in
this
case,
so
this
tells
lipstick
metrics
to
get
the
information
from
the
API
endpoint
provided
by
the
cube
API
server
for
this
resource.
A
B
A
B
So
I
we
have
seen
the
shell
output,
but
I
can
also
just
filter
here
for
the
copy
class
or
inform
metric
and
also
see
three
entries
for
each
of
our
clusters
and
the
query.
Language
I
don't
want
to
explain
the
whole
language
for
now,
but
I
can
filter
for
labels
here,
for
example,
I
want
to
filter
for
all
clusters
which
have
this
version
set,
for
example,
and
when
I
query
for
that,
it
will
only
show
me
the
ones
which
have
this
version
label
set.
B
So
that's
how
you
can
later
on
filter
for
for
things
like
I'm,
only
interested
in
the
objects
in
the
default
namespace
and
I
can
filter
for
for
links
basically
equals
to
default.
For
example,
let
me
just
do
that
in
space
equals
default,
and
it
now
showed
me
all
three,
because
all
of
them
are
in
the
default.
Namespace.
A
B
B
Yeah
and
maybe
if
we
take
a
look
at
into
the
information
which
is
provided,
there's,
for
example,
here
the
name
field,
the
namespace
fields
and
the
uid
of
the
object,
and
these
three
labels
are
defined
in
my
configuration.
So
if
I
switch
back
to
that
there's
here,
some
configuration
for
labels
from
path
like
the
name
label
comes
from
the
path
of
the
object
at
metadata
name
or
the
namespace
label.
Is
the
value
of
metadata
namespace
or
same
for
the
uid.
It's
just
a
Json
path
inside
the
object,
in
this
case.
B
A
B
So
these
are
values,
as
you
can
see,
I'm
using
the
the
metadata
information
here
and
I,
don't
want
to
repeat
that
for
every
metric
guy
I
create
I
always
want
to
have
the
name,
the
namespace
and
the
uid
at
the
metrics,
so
I
I
would
be
able
to
distinct
between
the
value
of
T1
cluster
and
the
value
of
the
other
cluster,
and
then
I
can
Define
special
metrics
or
different
metrics,
which
yeah
different
configurations
below
that,
like
the
first
metric
I
did
create.
Was
this
info
this
informatic?
B
A
B
That
this
is
all
configurable
and
you
can
name
your
labels.
However,
you
want
in
this
case
we
try
to
follow
the
examples
or
the
the
samples
we
have
from
the
implementation
of
cube.
State
metrics,
like
the
objects
of
cluster
API,
have
some
yeah
same
or
similar
information,
and
you
can
compare
objects
to
core
resources
like
machine
deployments,
are
pretty
similar
to
deployments
or
huge
button
control.
B
B
A
B
B
Yeah,
so
this
information
is
pretty
straightforward:
it
it
will
always
have
the
value
of
one.
Maybe
why
is
that?
For
that?
We
can
take
a
look
at
the
open,
metrics,
open,
metrics
specification
and
open
metric
specification,
defines
different
kind
of
metrics
and
what
their
value
should
be
and
how
some
naming
scheme
should
be
done,
and
if
you
take
a
look
at
the
types
of
metrics,
there's
also
an
infometric
here.
So
so
that's
the
whole
description.
B
What
an
informatric
is
and
an
infometric
basically
always
has
a
value
of
one
and
just
exposes
information
in
key
value
pairs
at
the
label
labels,
so
so
yeah.
In
this
case
we
have
this
info
metric
and
the
value
for
every
cluster,
as
we
can
see
here,
is
one
and
yeah.
We
have
some
information
here.
We
can
use
for
building
dashboards
or
something
like
that.
Yeah
before
I,
come
to
other
kind
of
metrics
like,
for
example,
replicas
I
may
show
how
this
could
be
used
to
create
dashboards.
B
So
I've
prepared
some
some
small
dashboards
using
grafana
here
and
we
did
take
a
look
a
lot
at
the
at
the
version
field.
For
now,
and,
for
example,
I
can
build
this
kind
of
dashboard
where
I
can
see,
which
versions
are
all
my
classes
at
in
this
case,
yeah.
That's
one
use
case,
for
example,
for
the
info
metric
to
display
this
information
yeah.
A
B
This
is
a
value
photometric
and
for
info
metrics.
This
value
is
specified
to
always
be
one
because
for
the
information
we
only
want
to
expose
information
here
in
the
middle,
at
the
labels
like
some
key
value
pairs,
so
maybe
a
better
example
where
you
can
see
that
this
information,
the
the
value
at
the
end
is
not
always
one
is.
If
you
take
a
look
at
the
copy
cluster
created
metric
and
we
can
see
it's
another
value
and
in
this
case
it's
the
creation
timestamp
of
the
of
the
cluster
object.
A
B
Like
we
Define
to
always
have
the
name
here
again
that
that
comes
from
the
top,
the
namespace
and
the
uid
of
the
object.
Yeah,
that's
that's
the
part.
We
defined
the
additional
key
value
pairs
here,
for
example,
they
come
all
from
Prometheus
in
this
case
or
from
the
service
discovery
of
Prometheus.
So
you
can
use
this
information,
for
example,
to
see
okay,
this
Matrix
comes
from
Cube
State
metrics.
A
A
is
it
possible
that
the
number
at
the
end
of
each
line
is
a
list
instead
of
a
single
number.
So,
for
example,
if
I
want
to
know
that
who
create
this
cluster,
there
may
be
a
user
ID.
Something
like
that.
So
is
it
possible
that
there
are
two
values,
shows
both
user
ID
and
times
timestamp
at
the
end
of
each
line.
B
B
B
You
know
if
we
query,
for
that,
we
get
multiple
outputs
here,
not
only
three
but
multiple
I
know
a
cluster
is
always
in
one
stage
and
for
this
metric,
though,
for
each
cluster
there
are
only
one
entry
will
be
of
value
one.
So
if
I
filter
for
that
I
get
back
to
only
three
entries
and
I
can
see.
There's
some
kind
of
face
here,
which
shows
me,
okay,
that
one
is
in
the
provision
phase.
B
Maybe
I
first
go
back
to
what
the
definition
says
here.
This
is
called
a
state
set
like
and
the
state
said
yeah
it's
a
collection
of
strings
and
yeah
related
to
Boolean
values.
In
this
case,
and
only
if
the
Boolean
value
is
one
yeah,
the
value
will
be
one
like
and
in
our
case
yeah,
maybe
I
should
I
go
back
where
this
value
comes
from.
So
in
this
case,
we
take
another
look
at
the
cube,
cuddle,
outputs,
get
clusters
and
we
see
here
that's
the
phrase
fields.
B
You
can
see
here
in
status,
that's
that's
the
phase
field
and
that's
where
this
information
comes
from
and
now
what's
what's
making
this
estate
set
is
the
status
phase
of
the
cluster
is
always
one
of
a
list
of
well-defined
strings
in
this
case
it's
for
example,
provisioned.
You
could
also
be
unknown
and
we
want
to
know
in
which
phase
my
cluster
is
at
and
by
that
we
expose
multiple
entries
for
my
cluster
object,
but
only
the
the
entry
for
provision
will
have
to
Value
one.
B
B
For
my
off
for
my
cluster
so,
and
only
one
of
them
will
have
the
value
of
one
like
we
know
our
class
is
in
provision,
so
the
value
for
the
provisions
would
be
one
and
for
the
others
the
value
would
be
zero.
You
can
also
show
that
now
on
the
Prometheus
UI.
Maybe,
but
when
we
query
or
we
would,
we
should
see
this
six
metric
entries
from
my
classroom
and
the
value
should
be
zero
for
all,
except
for
the
provisioned.
A
B
B
My
machine
deployment
has
said
so,
if
I
take
a
look
at
adds
another
metric,
which
is
called
copy
machine
deployment
back
replica,
see
if,
in
this
case,
we
have
machine
deployments
which
has
have
their
value
set
to
a
three
of
so
yeah,
maybe
to
to
highlight
here
we're
now
not
looking
to
the
cluster
object
anymore.
We're
taking
a
look
at
the
machine
deployment
object.
B
I
can
show
that
on
the
computer
side
too.
So
if
they
do
Cube
cuddle
cats
machine
deployment,
you
can
see
here
we
have
three
machine
deployments
and
all
of
them
have
a
value
of
three
in
that
case.
So
that's
the
cost.
Metric
I
can
just
expose
the
number
which
is
given
there,
which
is
also
yeah,
which
is
a
very
interesting
feature
like
I,
can
use
the
the
numbers
here,
for
example,
to
show
okay.
B
B
We
see
so
the
spec
replicas
metric
uses
this
information
in
Spec
replicas,
and
there
are
other
also
other
integers
which
get
exposed
which
are
at
the
end
here
in
the
status
field,
which
is
very
similar
to
what
a
deployment
does
and
there
we
have,
for
example,
again
a
status
replicas
field
which
shows
the
current
state,
so
in
Spec
you
say:
I
want
three
replicas
and
the
status
you
see.
Okay,
currently,
there
are
really
three
replicas
existing
and
there's.
Also
the
information.
B
B
Maybe
I
show
the
the
unavailable
replica
metric
here
too.
It's
called
copy
machine
deployment,
status,
replicas,
unavailable,
I,
more
or
less
showed
the
Json
path
here
in
the
metric
game
too.
So
it's
more
human,
readable
and,
as
we
can
see
here,
we
have
three
machine
deployments
and
one
of
them
has
three
unavailable.
Replicas
and
the
others
seem
to
be
fine
in
their
current
state
yeah
and
for
such
metrics.
It's
interesting
to
have
some
kind
of
a
graph,
and
we
can
see
here.
B
So
I
I
will
also
show
the
configuration
for
that
one.
So
here
again
we
have
the
group
definition.
We
use
machine
deployments,
have
some
kind
of
name,
namespace
uid.
You
always
want
to
add,
and
then
here
we
can
see,
there's
the
spec
replicas
field.
Let
me
just
say:
okay,
this
is
a
gosh
metric
and
please
just
set
the
value
to
whatever
they're
set
at
spec
replicas,
which
will
be
three
in
our
case.
Yeah.
B
B
B
B
Yeah,
maybe
we
can
show
some
picture
here
like
this
is
an
overview
from
the
from
the
Prometheus
page,
and
we
have
Prometheus
here
in
the
middle
and
Prometheus
does
get
its
metrics
from
something
called
exporters
like
the
targets,
and
one
of
them
could
be
Cube
State
metrics,
so
it
pulls
its
metrics
from
example,
from
Cube
State
metrics.
Here
it
stores
them
in
there
in
its
database
and
makes
it
available
by
the
HTTP
interface
or
by
its
API
and
I.
B
Think
I
also
got
a
picture
about
the
relation
of
cube,
State
metrics
to
you
know
to
the
API
server
like
in
our
case,
we
have,
as
I
did
show
the
namespace
observability,
where
we
have
the
port
cubesat
metrics
I
I
did
also
show
for
service
Discovery.
We
have
the
service
for
cube,
State
metrics
and
what
Cube
State
metrics
does
is
to
gather
its
information.
B
It
talks
to
the
API
server
of
kubernetes
and
tries
to
pull
all
the
time
the
information
from
the
API
server
just
like
we
would
do
a
cube,
cancel,
get
all
the
time
and
store
this
information.
So
it's
using
the
API
is
going
to
get
the
information
and
then
Prometheus
and
then
go
ahead
and
try
to
pull
the
information
from
the
HTTP
endpoint
and
store
it
in
its
database.
A
A
B
If
I
take
a
look
at
the
CLI
arguments,
there
are
lots
of
options
to
optimize
here
like
in
this
case.
I
was
talking
about
the
sharding,
which
is
the
scale
out
thing
yeah
or
you
can
also
limit
acoustic
metrics
to
take
a
look
only
at
a
specific
name
space
or
have
a
denial
list
of
namespace.
It
should
not
take
a
look
at
yeah.
B
B
Yeah,
there's
also
some
further
documentation
of
the
feature,
which
also
explains
again
how
it
could
get
consumed
and
also
has
some
examples
how
to
use
it.
Yeah
I
think
that's
also
helpful
for
anyone
who
wants
to
start
and
and
use
that
and
maybe
for
ideas
what
it
could
be
used
for.
It
could
be
used
for
any
kind
of
custom
resource
metric,
as
I
said
like
running
excite.
One
use
case
would
be,
for
example,
if
I
have
a
backup
tool
like
Valero
and
the
Valero
custom
resource
shows
me.
B
B
B
B
For
example.
Config
I
have
running
here.
It's
integrated
in
the
Tilt
environment
for
cluster
API,
so
yeah.
If
anyone
wants
to
take
a
look,
how
it's
done
there,
there's
some
documentation
how
to
enable
Cube
stick
metrics
in
there
and
also
the
the
configuration
for
cluster
API
is
stored
in
the
repository.
So
we
could
also
link
that
to
go
here.
B
B
A
B
B
They
should
be
visible,
then,
in
here
like
here,
for
example,
I've
got
one
one
alert
configured
for
the
metrics
I
did
create,
which
is
an
alert
which
yeah
triggers
when
there
are
post
objects.
So
in
cluster
API
there's
there's
a
configuration
in
the
objects
like
you
can
say:
Okay
I
want
to
pause
all
operations
on
my
cluster
and
if
you
set
that
the
salary
will
trigger
would
trigger.
B
B
A
B
B
A
B
A
A
Okay,
so
thanks
Christine
for
Christine
for
joining
us
today,
and
it's
like
a
I
I,
have
a
like
a
very
clear
whole
picture
about
like
how
we
like
use
this
Cube
State
metrics,
to
get
informations
from
cluster
and
do
the
parameters
on
the
top
two
like
career,
informations
and
finally
shows
up
in
the
graph
funnel
graphana
dashboard.
So
thank
you
very
much
for
providing
all
this
information
for
us
today.