►
From YouTube: Collecting Operational Metrics for a Cluster with 5,000 Namespaces - Rob Szumski & Chance Zibolski
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Collecting Operational Metrics for a Cluster with 5,000 Namespaces - Rob Szumski & Chance Zibolski, Red Hat
Kubernetes is popular to provide a multi-tenant, shared infrastructure layer for many eng teams within an org. This is great for the teams, as they have a stable, scalable cluster to build upon. However, the Ops team needs insight into resource usage, availability & more, especially once an app is running in multiple regions/clouds.
This talk will introduce the open source Operator Metering project, which collects metrics from Prometheus about Pods/Namespaces & combines it against cloud resource usage and other metrics to compute cost per Pod and other reports on a periodic interval. This capability unlocks billing for internal biz units, capacity planning, and operational knowledge of Operators running within a cluster.
At the end, we’ll show the project running against a real cluster with over 5,000 namespaces, proving the scale and insight the reports can provide to an Ops team.
To learn more: https://sched.co/GrXg
A
A
So
today
we're
gonna
talk
about
collecting
operational
metrics
across
some
really
large
clusters.
You
know
5,000
10,000
namespaces
and
go
through
some
of
the
use
cases
for
doing
that,
and
then
we're
gonna
take
a
live
demo
at
some
of
the
technology
and
we'll
show
you
some
reports
running
from
some
of
these
clusters.
What
we're
going
to
talk
about
today
is
a
component
of
our
operator
framework.
A
So
the
operator
framework
is
a
set
of
tools
to
build
operators.
Operators
are
pieces
of
software
that
have
specialized
knowledge
about
how
to
run
something.
In
this
case,
we
have
an
SDK
that
helps
you
build
these,
so
you
take
the
operational
complexity
that
makes
up
your
application.
You
embed
it
in
a
piece
of
software
and
we
have
a
bunch
of
tools
to
help
you
do
that
run.
Testing
cycles
all
that
type
of
stuff.
We
also
have
our
lifecycle
manager.
This
helps
you
run
a
number
of
operators
on
a
cluster.
A
You
know
1
5,
10
30
and
manage
the
lifecycle
and
upgrades
of
those.
Then
what
we
have
is
our
operator
metering.
This
is
what
we're
gonna
spend
the
bulk
of
this
talk
on,
and
this
is
all
about
collecting
operational
metrics
from
the
Prometheus
stack
that
we
have
running
in
the
cluster,
and
then
you
know
turning
that
into
useful
reports
for
humans
to
do
things
with
all
this
is
hosted
in
a
vendor,
neutral,
github
board,
we'd
love
for
you
to
check
out
the
code,
get
involved
file
bugs
all
that
good
stuff
and
so
for
metering.
A
The
way
that
this
works
is
we've
got
a
few
different
things
that
we're
interested
in
the
first
is
app
specific
insights.
These
are
you
know,
custom
metrics,
that
your
operators
and
the
applications
that
they're
running
know
about
anything
from
the
number
of
database
tables
you're
running
to.
If
you
need
to
do
data
rebalance
operations,
you
might
want
to
track
that
as
kind
of
look
at
node
failures
and
that
type
of
stuff
see
how
those
affect
each
other,
but
also
we
have
this
really
great
data
source
on
the
cluster.
A
That's
our
pre
installed
Prometheus
stack,
and
this
already
knows
about
all
the
pods
running
on
the
cluster.
It
knows
what
namespaces
they're
in.
So
it's
got
a
bunch
of
rich
data
that
we
can
include
kind
of
out
of
the
box
to
help
you
get
a
bunch
of
reports,
so
I
want
to
talk
about
some
of
those
use
cases
and
what
this
is
useful
for.
A
So
at
the
goal
for
this
is
really
out
of
the
box.
You
can
have
chargeback
show
back.
This
is
reporting
on
resource
usage
by
a
number
of
different
metrics,
and
this
really
encourages
the
correct
behavior
of
folks
that
are
using
your
cluster.
If
somebody
is
requesting
a
gig
of
ram,
you
want
to
make
sure
that
they're
using
that
gig,
because
you
can't
use
that
gig
and
give
it
to
somebody
else.
So
we
use
cluster
monitoring
like
I
said
so.
We've
already
got
this
really
smart
data
source.
So
we're
not.
A
You
know
tackling
this
huge
problem,
it's
a
little
bit
smaller
of
a
problem,
and
then
we
want
to
make
this
framework
extensible
such
that
you
get
a
report
about
RAM
or
CPU
capacity
or
storage
capacity,
and
if
you
want
to
further
post-process
that
we
want
to
make
that
very
easy
to
do
so
before
we
get
into
it.
This
is
kind
of
our
end
result.
This
is
where
we're
going
to
get
to.
A
This
is
a
CSV
file
that
comes
out
of
our
system,
and
what
this
is
looking
at
is
a
number
of
pods
and
namespaces
across
the
cluster.
What
nodes
they're
scheduled
on
exactly
how
long
that
the
pod
was
alive
and
then
getting
some
resource
utilization
out
of
it
in
this
case,
we're
looking
at
CPU
seconds
as
well
as
the
actual
dollar
cost
of
that
when
you
have
a
really
rich
cloud
billing
API,
you
can
actually
then
start
computing
these
very
easily
and
say
this
pod
cost
a
dollar
and
20
cents
to
run
that
type
of
thing.
A
A
So
what
this
looks
like
under
the
hood
is.
This
is
looking
at
two
clusters
that
have
this
installed
I
break
out
Amazon,
because
we
can
do
some
dollar
correlation
there,
so
that
flow
is
a
little
bit
different,
but
you've
got
a
bunch
of
clusters
running.
They
have
namespaces,
there's
pods
in
those
namespaces
and
we
have
metering
software
running.
That's
looking
at
all
that
and
tracking
it
continuously
on
Amazon.
We
can
actually
hit
the
Amazon
billing
API
and
say
this
pod
was
on
this
node.
What
type
of
instance
is
that?
How
much
does
it
cost?
A
What
is
the
percentage
of
the
pot
it
is
being
used
or
the
node
is
being
used
by
that
pod
and
can
get
to
an
actual
dollar
amount.
So
we're
doing
this
on
Amazon
today.
We
hope
to
extend
this
to
any
rich
billing
API
and
there's
some
really
interesting
use
cases
for
doing
this
on
bare
metal
as
well,
and
then
we
want
to
output
these
reports
to
stable
storage,
we're
using
Prometheus
as
our
main
data
source
and
that
doesn't
stick
around
forever.
You
have
to
store
it
Ram.
A
So
when
we
talk
through
some
concrete
use
cases
that
are
pretty
interesting,
this
is
our
first
one,
which
is
your
dis
on
Amazon,
and
you
want
to
do
show
back
to
teams
that
are
using
the
cluster
and,
let's
just
say
every
team
you
know,
has
the
standard
three
projects
or
namespaces
on
the
cluster
dead
stage
prod
and
then
each
team
just
has
a
blanket
budget.
Let's
say
$10,000.
How
do
you
start
computing?
This
well
are
metering
software.
A
You
can
use
one
of
our
default
reports
for
looking
at
plot
dollars,
like
we
just
showed
on
the
previous
screen,
and
then
you
can
either
just
dump
that
into
Excel
and
start
sorting
columns
start
adding
things
up.
You
could
further
post-process
that
process
that
into
a
bi
tool
of
choice.
There
are
a
number
of
these
out
there.
You
could
even
start
piping
this
into
a
graph,
honest,
our
building,
dashboards
or
even
you
know,
starts
sitting
off
like
slack
web
hooks
or
anything
you
wanted
to
do
so.
A
A
So
a
really
popular
one
is
just
shame.
Underutilization,
everybody
kind
of
wants
to
do
the
right
thing
when
it
comes
down
to
it,
but
they
just
need
to
know
that
they're
being
kind
of
a
bad
actor
on
the
system.
So
let's
say
we're.
We
have
a
cluster
or
set
of
clusters
that
are
used
by
15
teams,
and
so
we
just
want
to
say,
hey
look.
Our
threshold
is,
if
you
request
twice
as
much
as
you're.
Actually
using
you
shouldn't.
Do
that
we're
gonna,
just
gonna,
let
you
know
so.
A
You
can
compare
two
of
our
reports
and
look
at
the
ratio
of
that
and
you
know
have
a
piece
of
software.
That's
doing
some
math!
You
could
do
this
by
hand,
you
could
pipe
it
into
something.
It
says
anything
above
2x.
Let's
go
send
off
a
slack
web
hook
into
that
team's
channel.
Let's
go:
send
people
emails,
let's
put
them
on
like
a
leader
board
in
this
case
leading
is
not
good,
so
you
can
do
anything
and
what
you
want
to
do
here
is
incentivize
the
correct
behavior
you're,
just
wasting
company
resources.
A
And
then
here's
a
use
case
for
bare
metal
and
let's
say
we've
got
a
few
different
clusters
on
bare
metal
and
you
want
to
start
looking
at
the
usage
across
both
of
these
clusters
for
a
single
team
and
to
also
do
some
things
like
you
know,
answer
the
question:
how
much
RAM
are
we
using,
but
you
can
also
go
a
little
bit
further
and
start
saying.
Well,
actually
we
know
how
much
that
RAM
cost,
because
we,
you
know,
spend
X
thousands
of
dollars
per
machine.
A
If
this
thing
costs
you
a
hundred
grand
you
know
it
matters,
so
these
are
the
types
of
things
that
you
can
plug
in
with
this
system,
and
so
where
this
is
really
really
useful
is
for
capacity
planning
so
set
aside,
it's
shaming
people
for
a
second,
and
we
can
use
this
knowledge
about
the
cluster
to
know
where
the
cluster
is
going.
So
you
know,
you've
got
real-time
utilization.
You
can
start
turning
that
over
time.
You
can
see
where
this
is
going.
There's
a
number
of
things
you
can
do
here.
A
The
interesting
use
case
for
this
is
what
we're
going
to
talk
about
for
the
rest
of
the
talk.
Is
this
environment
called
open
shift
online?
These
are
these
really
really
big,
highly
multi-tenant
clusters
that
OpenShift
runs
and
we
do.
These
are
unreal
workloads,
so
you
can
sign
up
with
an
email
address
and
get
two
gigs
of
ram
or
a
gig
of
ram
and
go
to
town
running
real
docker
containers,
and
we
use
these
for
scale
testing
for
OpenShift
and
so
they're
super
highly
multi-tenant.
A
You
know
we
can
fit
5,000
10,000
namespaces
on
these
clusters
and
they're
really
unique
environments.
To
start
testing
out
some
of
this
software,
and
so
I've
talked
a
lot
about
these
built-in
reports,
but,
as
I
said
in
the
beginning,
we
want
to
do
this
with
custom
metrics,
all
the
things
that
make
your
applications
tick,
the
things
that
are
really
really
important
to
them.
A
You
also
might
want
to
meet
her
on
those
you
even
just
doing
capacity
planning,
but
you
might
want
to
do
things
like
telemetry
license
enforcement,
if
you
say
built
an
operator
that
is
commercial
software,
and
you
want
to
distribute
that
to
all
of
your
customers.
You
need
to
know
if
you
license
per
core
per
database
table
per
gigs
of
RAM
per
gigs
of
storage
whatever
it
is.
B
All
right,
thanks
Rob,
so
now
that
we've
actually
gone
over
some
of
the
use
cases
we
had
mine
when
developing
metering
and
some
of
the
things
that
we're
trying
to
solve
is
it
let's
go
a
little
bit
more
in
tips
about
how
we
actually
achieve
this?
What
technologies
we've
chosen?
Why
and
also
some
of
the
challenges
we
have
so
one
of
the
most
important
pieces
of
our
stack
is
our
query:
engine
we're
using
a
database
called
presto
which
was
written
by
Facebook.
B
They
market
it
as
a
distributed
sequel
engine
that
is
designed
for
interactive
processing
and
particularly
for
analytics.
This
is
super
useful
for
us,
because
we're
really
just
trying
to
get
up
to
a
place
where
people
can
really
interact
with
our
operators
and
our
reports
without
having
to
you
know,
wait.
You
know
until
the
end
of
the
day-
or
you
know
like
typical
batch
jobs
run,
maybe
once
per
day
or
something
or
even
less
we'd,
really
like
to
be
able
to
enable
people
is
to
actually
iterate
on
these
rapidly
make
changes.
B
Customize
things
potentially
just
use
the
things
out
of
the
box
as
well,
but
we
want
a
lot
of
people
in
this
cluster
being
able
to
interact
with
the
system.
The
other
interesting
thing
is
that
presto
actually
doesn't
really
store
any
data.
It's
not
traditional
like
a
traditional
database
that
actually
puts
everything
on
disk
and
teep
just
treats
it
like
a
bunch
of
files.
It's
actually
designed
to
be
more
of
a
maybe
a
proxy
in
some
ways.
What
it
does
is.
It's
got
this
concept
of
connectors
which
allow
presto
to
actually
connect
to
other
underlying
databases.
B
So
what
this
means
is
you
can
effectively
join
database
tables
from
many
different
database
technologies,
so
you
can
have
it
connect
to
hive
which
I'll
go
into
in
a
second.
You
can
have
it
connect,
Presta
or
Postgres
equal,
my
sequel,
Cassandra
MongoDB,
and
you
can
have
it
join
tables
or
documents
across
these
different
sources
into
a
single
and
through
a
single
interface,
just
using
regular
sequel,
which
is
pretty
powerful.
B
The
other
component
is
Apache.
Hive
Apache
hive
is
a
also
a
sequel
engine
designed
for
sequel
on
Hadoop.
We
leverage
this
because
it
allows
us
to
actually
do
storage
and
compute
separate.
We
can
actually
have
storage
done
entirely
in
s3
or
HDFS,
or
a
local
TV,
with
read/write
mini
semantics
and,
at
the
end
of
the
day,
Presto's
interacting
through
hive,
to
actually
find
where
the
data
is,
and
then
it's
actually
interacting
with
the
underlying
data
wherever
it
lives,
we
also
have
a
few
operators,
so
the
first
main
operator
is
the
reporting
operator.
B
The
reporting
operator
is
pretty
much
what
ties
everything
together.
It
is
what
talks
to
presto
it's
what
talks
to
hive.
It's
what
talks
to
a
Prometheus
and
the
various
cloud
API
is
to
actually
tie
everything
together
in
reality,
what
it's
actually
doing,
what
that
means
is
it's
taking
a
Prometheus
query
executing
against
Prometheus
and
then
storing
those
metrics
into
presto.
B
At
the
same
time,
it's
running
these
other
sequel
queries
to
actually
run
reports
and
produce
the
kind
of
actual
information
that
you
want
to
see
which
might
be.
What
is
the
total
capacity
of
my
cluster
for
this
duration
of
time,
or
maybe
what
is
the
percentage
of
the
utilization
in
that
style?
The
other
piece
is
the
metering
operator.
B
A
custom,
CRT
and
metering
resource
has
spec
just
like
everything
else,
it's
kind
of
like
a
deployment
and
when
you
fill
it
out,
it
will
actually
output
all
these
other
components
as
deployments
staple
sets,
secrets
and
services
for
presto
hive
reporting
operator
and
even
optionally
HDFS.
If
you
don't
have
object,
storage
for
available,
so
I
want
to
talk
a
little
bit
about
why
we
chose
some
of
these
components.
B
As
you
might
be
wondering,
we've
made
some
interesting
choices
about
using
things
like
hive
or
presto,
as
you
might
imagine,
why
not
use
something
like
Postgres
or
why
not?
One
of
the
main
things
is
that
we
want
to
integrate
well
with
existing
systems,
not
necessarily
have
to
fully
replace
other
systems
that
are
good
at
what
they
already
do.
B
So
one
of
the
biggest
pieces
that
we
want
to
work
well
with
prometheus
Prometheus
is
really
good
for
monitoring
collecting
metrics,
doing
queries
against
that
information,
so
we're
taking
full
advantage
of
Prometheus
as
our
actual
part,
that
is
responsible
for
figuring
out,
what's
happening
on
the
cluster
and
we're
using
it
as
a
data
source
to
actually
run
queries
against.
The
other
piece
is
that
we
want
to
make
sure
that
this
works
well
on,
like
bare
metal
environments
in
any
really
any
kubernetes
environment,
which
means
it
can't
be
a
SAS.
B
It
can't
rely
on
entirely
rely
on
cloud.
Api's
I
mean
you
can
use
s3
as
an
optional
piece,
but
it's
not
required.
So
that
was
a
key
piece
related
to.
That,
though,
is
that
we
chose
hive
in
particular
along
with
presto,
because
it
means
that
we
don't
necessarily
have
to
worry
about
the
storage
piece.
There's
a
lot
of
storage
technologies
out
there.
A
lot
of
the
current
data
relational
databases
around
are
designed
to
scale
horizontally
necessarily
or
if
they
are,
if
there's
trade-offs
that
you
have
to
make.
B
So
this
allows
us
to
actually
have
storage
either
outside
of
the
cluster
inside
of
the
cluster
leverage.
Whatever
you
really
want
to
use,
and
then
it
can
do
all
the
compute
inside
the
cluster
and
as
I
mentioned
before,
Prest
is
really
unique
in
that
you
can
just
really
work
with
a
pre-existing
data.
There's
not
a
import
phase,
necessarily
I'll
get
more
into
that
in
a
second.
But
what
that
really
means
is
it's.
Basically,
when
you
execute
a
sequel
query:
it's
not
like
going
to
just
download
everything
all
at
once.
B
So
now
that
we've
actually
got
a
little
bit
of
a
better
idea
on
how
these
components
interact.
Let's
talk
a
little
bit
about
what
the
data
flow
looks
like
when
it
comes
to
metering,
so
the
primary
component
that's
doing
most
of
this
is
a
reporting
operator.
It's
going
to
be
talking
to
Prometheus
performing
Prometheus
queries,
inserting
those
that
by
data
into
presto
and
then
presto,
is
going
to
be
using
hive
to
actually
store
that
in
something
like
s3
or
HDFS,
or
a
local
TV,
where
it
can
be
mounted
at
on
multiple
pods.
B
At
the
same
time,
it's
calculating
reports.
So
when
you
create
a
report
object,
it's
typically
with
a
schedule,
something
like
an
hourly
schedule,
a
daily
schedule
or
longer
and
in
the
background
it's
running
against
the
schedule
and
it's
calculating
running
the
sequel,
queries
against
presto
to
actually
calculate
the
results
and
then
also
storing
them
through
hive
and
something
like
HDFS
or
s3.
B
B
What's
called
our
report
generation
query,
you
can
see
this
referenced
and
the
generation
query
field
here
that
query
I'll
show
you
in
a
second,
but
then
today
you
don't
really
have
to
do
a
whole
lot
to
use
everything
that's
built
into
these
existing
reports
are
really
easy
to
create
you
just
say
what,
where
you
want,
you
set
scheduled
and
create
it.
So
here's
an
example
report:
query:
it's
got
some
things
emitted,
but
the
main
things
you
have
to
know
is
it's
going
to
contain
a
list
of
columns.
B
So
one
of
the
very
key
things
that
we
talked
about
earlier
in
the
talk
was
capacity
planning
OpenShift
online.
As
Rob
said,
anybody
with
an
email
address
can
sign
up
no
credit
card,
no
payment
details,
nothing!
You
want
access,
you
can
get
it!
It's
really
designed
for
kind
of
smaller
apps
we're
not
going
to
give
you
a
huge
amount
of
resources
by
default.
We
give
you
about
one
gigabyte
of
memory
and
two
CPUs,
but
the
way
you
actually
get
that
is
you
sign
up
and
then
there's
somewhat
of
a
queue.
B
These
clusters
are
kind
of
a
fixed
capacity
because
we're
not
caught
charging
you
for
it.
We
can't
just
scale
this
out
with
infinite
nodes
that
cost
money
we're
not
going
to
just
spend
endless
amounts
of
money
for
people
to
run
free
apps,
like
Bitcoin
miners.
So
the
way
this
works
is
we
gate
by
capacity.
If
there's
no
capacity,
you
have
to
wait
until
there
is,
and
so
all
actually
understanding
what
is
being
used.
What's
not
being
used
is
extremely
important
because
we
want
to
make
sure
people
can
have
access
to
these
clusters.
B
So
one
thing
that
we
did
to
actually
solve
some
of
these
problems
are:
we
have
teams
that
wrote
a
pod
D
scheduler,
which
is
basically
designed
to
delete
pods
so
that
the
cluster
can
be
rebalanced
in
a
more
efficient
way.
Kubernetes
doesn't
have
that
by
default,
so
we
need
a
way
to
actually
make
sure
the
pods
are
running
on
well
size
nodes
or
in
a
bin
that
actually
fits
better
for
it.
B
So
what
did
we
find?
Well,
one
of
the
more
interesting
things
that
we
found
is
that,
given
the
default
quota
of
1k
byte
of
memory
and
2
cores
pretty
much,
nobody
actually
sets
their
own
resource
limits
or
quotas.
So
that
means
we're
going
to
be
giving
them
the
default
that
we
set,
which
is
50%
of
their
quit
and
in
reality
we
found
that
the
top
1,000
namespaces
don't
do
anything
with
their
quotas.
B
They
just
leave
it
blank,
which
means
they
get
half
of
their
quota
by
default
and
at
the
end
of
the
day,
we
can
basically
come
to
the
conclusion
that
most
people
aren't
making
customizations
and
more
often
than
not.
We
can
actually
see
that
our
cluster
being
underutilized
by
these
these
users
and-
and
we
could
also
then
maybe
go
to
the
conclusion
that
we
could
probably
oversubscribe
this
cluster
since
the
average
person
is
not
actually
using
everything
we're
giving
them.
B
On
the
same
note,
we
looked
at
the
people
are
using
the
cluster.
What
we
found
was
people
who
are
using
the
cluster,
which
are
apps
that
have
real
usage,
not
like
idle,
say:
I
have
500
megabytes
of
RAM
and
I
only
use
one,
but
actually
are
using
some
percentage
of
their
memory
or
CPU
and
in
practice
most
people
are
using
around
85%
of
their
limits,
for
something
like
memory
and
a
little
bit
less
for
CPU
is
in
full
spike
here.
B
So
in
this
case,
this
is
a
graph
that
we
took
from
the
data
we
produce
and,
as
you
can
see,
most
usage
for
this
period
of
about
I
think
it's
two
hours
or
more,
it's
showing
that
on
average
people
are
using
about
85%
of
their
quotas.
So
what
this
tells
us
is
that
it's
pretty
well
dialed
in
if
they're,
using
one
or
like
always
at
one,
that
would
mean
they're
always
hitting
their
quotas,
and
if
it
was
something
closer
to
50%
like
our
defaults,
it
means
they're
not
really
using
all
of
it.
B
So
most
of
our
most
active
users
are
getting
a
good
usage,
but
they're
not
also
hitting
the
quota
limits,
what
you
might
expect.
So
what
are
the
some
of
the
problems
we
encountered?
It's
a
really
interesting
question.
One
of
the
biggest
pieces
that
was
difficult
for
us
was
actually
ingesting
this
data
from
each.
This
is
really
good
at
this.
It's
really
it's
got
a
good
design
for
this
and
it's
basically
polling
periodically
all
the
data.
That's
are
all
the
metrics
and
points
in
the
cluster.
B
We
followed
a
very
similar
model
with
the
reporting
operator
right
now.
There
isn't
a
good
way
for
presto
to
actually
natively
make
these
queries
for
us.
So
what
we
had
to
do
is
we
actually
did
have
to
import
data,
which
is
turning
out
to
be
about
as
big
of
a
problem
as
we
thought
it
might
be.
So,
at
the
end
of
the
day,
one
of
the
problems
we
encounter
is
that
importing
this
metrics
gets
back
after
a
while,
just
because
of
the
sheer
amount
of
volume,
part
of
that's
related
to
the
high
metric
resolution.
B
Right
now
we
collect
at
about
a
60
second
round
solution
for
these
clusters,
which
might
be
a
bit
a
bit
of
a
low
resolution
for
us
or
high
resolution
rather,
and
it
actually
causes
a
really
huge
growth
in
storage
requirements
again
we're
not
necessarily
building
on
top
of
a
time
series
data
base
here.
So
some
of
those
efficiencies
we
gain
from
Prometheus
are
lost
when
we
actually
import
the
data.
So
longer-term
we're
hoping
to
actually
leverage
Prometheus
more
by
hopefully,
actually
having
presto,
make
two
queries
against
Prometheus
directly
rather
than
importing
the
data.
B
So
it
affects
our
throughput
as
well,
and
then
one
of
the
last
things
that
we've
kind
of
hit
was
when
we
get
these
reports,
they
have
everything
we
want,
but
sometimes
that's
too
much
information
with
5,000
namespaces.
You
can
imagine
with
an
hour
report
that
runs
on
the
hour.
5,000
named
stasis
times
24
hours,
you
get
a
hundred
and
twenty
thousand
rows
for
just
a
day's
worth
of
data
in
the
CSV
file
and
that
can
be
come
a
little
bit
less
manageable.
B
B
We
found
that
our
Prometheus
stack
actually
is
going
way
beyond
some
of
the
typical
recommended
scaling
values
that
even
we
document
for
Red
Hat
on
openshift.
In
practice,
we
have
much
much
more
than
7200
pods
on
these
clusters,
with
a
cluster
of
5,000
to
8,000
namespaces,
one
to
two
pods
per
namespace
and
you're.
Definitely
going
over
that
amount.
B
We
were
actually
able
to
run
these
reports
on
these
clusters
some
tuning
and
it
worked
really
well
with
Prometheus
at
the
end,
despite
some
of
the
issues
I
mentioned
before
all
right,
so
next
I
want
to
actually
show
you
guys
what
this
looks
like
to
interact
with
a
real
cluster
in
this
demo.
I
have
a
test
cluster
setup:
that's
got
roughly
2,000
mm
spaces.
It
turns
out
there
costs
a
lot
of
money
to
run
on
g-cloud,
a
5,000
namesakes
cluster,
so
I
hope.
B
B
Now
it's
replying
rapping,
okay,
all
right,
so
each
one
of
these
reports
is
something
that
you
can
create
out
of
the
box
and
what
each
one
of
these
reports
is
doing
is
running
on
a
schedule
right
now.
It's
gonna
run
on
the
hour
every
hour
and
each
one
of
these
is
running
a
very
particular
query
which
basically
specifies
what
it's
going
to
do,
each
time
so
I'm
going
to
show
you
one
of
those
really
quick.
Let's
see,
let's
show
you
guys
the
usage
for
CPU
anybody.
B
So
what
I'm
doing
is
I'm
playing
the
report
generation
query
custom
resource.
Actually,
let
me
show
you
the
whole
list
really
quick.
So
we
have
a
number
of
built-in
queries.
Some
of
these
are
more
flexible
than
others,
and
some
are
just
kind
of
building
blocks
for
other
queries,
especially
the
ones
that
end
in
slash
raw
but
query,
I
pull
it
out
earlier
here.
It's
basically
just
the
spec
dot
there
we
go.
It's
basically
a
bigger
two
spec
doc
query
for
the
custom
resource
that
I
showed
before.
B
The
actual
contents
isn't
super
important,
but
I
want
you
to
understand
that
this
is
just
raw
sequel,
there's
a
little
bit
of
other
stuff
in
there
there's
some
go
templates.
It's
there
to
make
it
flexible,
and
particularly
this
one
is
designed
for
a
concept
called
roll-up.
That
means
we
can
actually
have
an
hourly
report
which
calculates
data
on
an
hourly
basis
from
raw
data,
and
then
we
can
have
a
daily
report
that
reports
on
the
hourly
reports,
data
which
actually
it
saves
us
a
lot
of
computation
because
we're
basically
spreading
out
the
computation
over
time.
B
B
What
we
have
here
is
a
namespace
memory
utilization
and
a
namespace
utilization
data,
query
or
report
each
one
of
these
is
going
to
run
every
day,
and
then
they
have
some
custom
info
that
actually
tell
us.
I
want
to
aggregate
the
results
of
another
report,
so
let's
go
ahead
and
create
that
all
right
and
then
let's
actually
see
its
status.
B
B
We
can
also
see
the
last
time
it
reported
I
set
it
up
to
start
from
the
8th,
because
that's
when
this
cluster
was
set
up
and
I
want
it
to
calculate
not
only
what
is
the
current
times
data,
but
all
the
data
that
we
art
you
did
have
collected
as
well.
So
that's
cool,
get
the
status
one
more
time
and
make
sure
it's
been
updated
and
we
can
see
it's
already
caught
up
to
a
few
days
worth
of
data.
It's
now
the
11th
from
the
8th
in
a
few
seconds
we'll
catch
up
to
the
14th.
B
As
you
can
see
all
the
other
hour,
they
once
have
and
then
well.
That's
also
running
I
want
to
show
you
the
other
query,
which
is
interesting,
which
is
a
cluster
wide
version
of
the
same
thing,
so
the
previous
one
was
for
namespaces.
This
is
for
the
cluster
wide
utilization
for
CPU
memory
very
similar.
You
can
give
some
custom
inputs.
I
want
to
aggregate
these
other
cluster
capacity.
Hourly
reports,
as
indicated
here
so
I'm
also
gonna,
create
that
and
then,
let's
check
on
the
status.
B
So
we
have
the
name:
six
memory
utilization,
one
and
it's
basically
almost
caught
up
one
day
behind
and
let's
actually
get
some
of
these
results.
So
I
know
this
URL
is
somewhat
hard
to
see,
but
it's
basically
we're
using
an
open
ship
route
to
expose
the
reporting
operators
API.
You
can
also
use
a
load,
bouncer
service
or
ingress.
B
If
you
like
right
now,
I'm
taking
advantage
of
some
open
ship
features
that
make
it
a
little
bit
easier
and
then
the
reporting
API
is
basically
a
simple
REST
API,
where
you
can
create
the
results
in
whatever
format
that
you
prefer
so
right
now
we
support
JSON,
CSV
and
tabular
format.
So
let's
actually
take
a
look
at
this
in
the
tabular
format.
Really,
quick
and
again,
this
is
the
cluster
memory
utilization
report.
B
So
this
report
right
now
it's
caught
up,
look
just
like
to
the
12th,
also
go
to
the
right,
don't
worry,
but
the
first
two
columns
basically
indicate
the
period
that
it
was
reporting
on
for
that
row.
The
third
column
is
the
total
capacity
available
in
bytes
seconds.
So
that's
basically
the
actual
of
memory
times
the
time
that
it
was
at
that
value
and
then
we
also
have
to
use
the
usage.
You
know
far
right.
We
have
the
percentage
of
overall
cluster
capacity
that
this
is
taking.
That
is
taken
up
so
beginning
of
the
week.
B
Last
week,
one
before
I
set
up
2,000
names
faces,
it
was
about
4%
utilized
and
now
it's
about
12%
utilized.
We
also
have
some
averages
in
case.
That's
a
very
interesting
for
you
and
same
for
the
nib
counts,
and
then
the
same
thing
can
be
done
for
the
namespace
level
report,
so
this
one
is
going
to
be
quite
a
bit
bigger
and
that's
because
it's
going
to
have
a
row
for
each
namespace
per
day.
So
this
might
take
a
second
and
don't
worry.
I
also
have
another
view
for
this
in
a
second.
B
So
what
we
have
here
is
a
bunch
of
clusters
or
namespaces
rather,
and
we
have
one
for
each
row.
This
is
our
columns,
the
namespace,
as
shown
we
have
usage
going
to
the
right.
We
have
the
request.
We
have
the
percent
utilization
or
usage,
and
then
we
also
have
the
percentage
of
the
clusters
capacity
and
requests.
So
a
lot
of
these
ones
are
basically
idling
they're,
not
doing
a
lot
they're.
Actually,
the
bad
actors
that
we
want
to
shame
like
Rob
is
talking
about.
B
Well,
we
have
some
even
back
worse
actors
that
I
want
to
show
you
so
I've
already
actually
downloaded
some
of
these
reports
as
CSVs
and
opened
them
up
in
whatever
the
I
told
you
might
want
to
use.
In
this
case,
I'm
gonna
use,
Libre
Office,
and
so
one
of
the
things
Rob
talked
about
was
actually
just
finding
out
what
are
people
doing
on
this
clusters?
So
what
I've
done
is
I've
imported
this
in
to
LibreOffice
and
I've
actually
set
it
up
so
that
it's
going
to
show
us
the
more
recent
data?
B
First,
it's
going
to
show
the
data
that's
most
recent
with
a
high
request,
but
a
low
usage,
which
means
things
that
are
asking
for
too
much
and
not
using
any
of
it.
In
this
case,
the
top
usage
is
metering.
It's
the
only
thing
doing
anything
in
this
cluster.
No
surprise
its
usage
is
about
one
percent
of
the
cluster
and
it's
requesting
about
fifteen
percent
of
capacity,
probably
because
I
over
scaled
it
to
make
this
demo
run
really
fast.
But,
as
you
can
see,
the
following
ones.
B
Are
these
plus
test
or
over-allocated
namespaces
I,
wonder
what
those
are.
So
if
we
look
at
the
usage,
it's
literally
zero.
These
pods
again
don't
do
anything
they're,
literally
the
pause
container
but
they're
requesting
something
like
seven
thousand
or
seven
hundred
seventeen
thousand
cp4
seconds,
which
is
something
like
if
I
recall,
four
cores
and
like
something
like
that
and
over
a
period
of
time-
and
we
can
see
its
usage-
is
really
low.
B
Zero
percent-
but
it's
request
is
quite
high,
eight
percent
and
that's
out
of
the
overall
cluster
capacity,
meaning
each
one
of
these
namespaces
is
basically
adding
up
to
almost
a
total
of
a
quarter
of
this
clusters
capacity
and
we
can
do
the
same
for
namespace.
In
this
case
the
name
space
is
a
little
different
values,
but
it's
very
similar.
B
We
can
see
that
they're
asking
for
a
lot
of
memory,
but
they're
not
actually
using
it
and
then
at
the
end
of
the
day,
if
you
want
to
know
at
a
bird's
eye
view,
what
is
my
cluster
looking
like?
Maybe
not
at
a
namespace
level.
You
can
also
look
at
these
CPU
or
name
space
or
memory
cluster
wide
reports
which
basically
give
you
Oh
on
the
12th
to
the
13th.
You
had
4%
of
your
clusters,
CPU
used,
which
is
shown
here
that
row
and.
B
A
Wanted
to
get
a
show
of
hands.
Does
anybody
have
a
tool
like
this
running
against
a
cluster
they're
running
today?
Nobody
kind
of
ish,
maybe
a
few
people.
Okay.
Does
anybody
ever
run
into
problems
with
pods
that
can't
be
scheduled
because
they
ran
out
of
capacity
yeah
everybody
running
small
clusters
for
dev
tests
yeah?
A
So
this
whole
is
open-source.
It's
on
github
I.
Think
we've
got
a
few
links
here.
We'd
love
you
to
try
it
out.
This
is
a
screenshot
of
our
readme.
We're
currently
testing
this
with
customers,
but
it
works
against.
You
know
any
kubernetes
cluster.
This
is
all
very
agnostic.
We've
got
installation
guides
guide
through
how
to
run
reports.
The
reason
we
showed
you
all
these
sequel
queries
is
because
you
can
write
these
custom.
A
We've
handed
it
off
to
a
customer
and
actually,
like
you
know,
a
few
hours
later,
he'd
already
reported
back
that
he'd
been
writing
custom
queries
because
he
didn't
like
some
of
our
logic,
so
it's
really
really
extensible
and
so
we'd
love
for
y'all
to
try
it
out
get
feedback
file
bugs
all
that
good
stuff.
Thank
you.