►
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Fine-Grained Cost Allocation in Multi-Tenant Kubernetes Clusters - Yang Guan, Google
An organization operating a multi-tenant Kubernetes cluster may want to attribute the cost of operating the cluster to individual tenants for a variety of reasons. For example, a SaaS provider may want to understand how much it costs them to serve each customer, or an enterprise may want to perform chargeback or showback to internal teams or departments that are sharing the cluster. This session introduces you to methodologies to track the usage of cloud resources from various Kubernetes objects, and to associate the usage information with the cost of cloud resources on a per-namespace basis. The session also compares the methodologies with existing products in the market. A demo is included at the end to illustrate how we apply the methodologies to perform namespace-granularity cost allocation in GKE clusters.
To learn more: https://sched.co/GrYY
A
Hello:
everyone
thanks
for
joining
this
session.
My
name
is
a
young
one
and
I'm
a
software
engineer
working
at
Google.
My
work
mainly
focuses
on
making
kubernetes
a
better
place
to
run
multi-talented
workloads
and
today,
I'm
here
to
talk
about
her
uses,
your
metering
and
the
fine-grained
cost
allocation,
Amati
challenge
of
kubernetes
clusters.
A
So
to
know
why
we
need
to
run
a
multi-tenant
cluster.
Imagine
you
walking
the
IT
department
of
your
company
and
the
multiple
teams
within
your
company
know.
Karate
is
a
greater
platform
and
each
love
the
team
wants
to
have
a
creating
environment.
All
your
SAS
provider
wants
to
use
kubernetes
to
hoster
the
workloads
from
your
external
customers,
or
even
you
just
have
multiple
micro
services
that
you
want
to
close
to
end
the
wrong
in
kubernetes.
What
do
you
do?
What
options
that
do
you
have?
A
One
of
the
main
one
of
the
options
is
to
use
what
we
call
a
mati
single,
talented
model
where
you
create
a
separate
cluster
for
each
of
your
issue
of
the
party
and
use
a
cluster
as
a
boundary.
The
other
is
to
allow
the
multiple
parties
to
share
a
single
multi-talented
cluster
used,
for
example,
from
a
19mm
space
as
the
talented
boundary.
Why
are
you
certain
cases?
A
marquee,
single
tenant
class
or
two
does
offer
some
of
the
bandages,
such
as
better
isolation.
A
It
comes
at
your
cost,
so,
for
example,
if
you
are
manager
more
than
ten
clusters,
you
might
need
to
invest
in
tours.
For
automaticall
automaticall
classroom
management
such
as
a
clash
or
creation,
upgrade
and
a
given
deletion
of
clusters
simply
because
it
is
impractical
to
manager
a
huge,
enable
class
or
manually
and
depending
on
your
upgrade
or
patch
policy,
those
classes
may
end
up
being
different
patch,
washing
or
even
minor
versions,
causing
different
workflows
to
running
slightly
different
environment.
A
Secondly,
with
the
multiple
clusters,
you
will
need
to
have
a
magical
control,
and/or
masters.
We
should
the
extra
overhead-
and
sometimes
you
do
need
to
pay
for
these
control
and
resources
more
when
you're
running
applications
from
different
talent
in
different
clusters
that
are
by
nature,
isolated,
like
from
each
other.
A
You
won't
be
able
to
use
the
free
or
idle
resources
from
one
cluster
to
a
different
talent,
causing
high
resource
of
fragmentation
and
a
low
utilization
in
comparison
where
you
have
a
single
multi
tenant
cluster,
there
is
a
better
chance
for
you
to
do
optimization
in
packing
that
improves
the
overall
cluster
utilization.
Lastly,
creating,
for
example,
to
prohibit
namespace
is
usually
much
faster
than
bootstrapping
a
new
cluster,
which
means
you
can
provision
environmental
to
a
new
talent
much
more
quickly.
A
So
how
does
it
look
like
to
share
a
magnetic
faster
among
multiple
talents
in
a
multi-talented
cluster
pause
from
different
talents,
cashier,
compute
storage
and
network
resources?
Of
a
note?
In
addition,
cluster
level
resources
such
as
the
remoter
storage,
storage
or
load
balancers,
can
be
used
by
multiple
talents
in
kubernetes
in
general.
A
talent
issues
just
see
the
classroom
as
a
shared
pool
will
computer
resources.
A
We
are
telling
the
submitter
hey
the
workloads
that
workers
can
be
fitting
into
your
cluster
and
up
and
they're
running
a
talent
doesn't
need
to
care
and
shouldn't
care
about
that
detail.
The
resource
allocation,
the
consumption
within
clusters
such
as
where
the
party
is
running
whether
introducing
extra
rep
secada
war,
certain
parts
that
causes
the
new
notes
to
be
added
into
the
plaster
or
what
kind
of
Natural
Resources
the
is
provisioned
for
their
workloads,
but.
A
There
are
some
people
who
really
care
about
the
resource
usage
reading
the
October
net
cluster
and
wants
the
visibility
into
resource
usage
by
resources
by
usage
readability.
We
need
to
know
what
resources
are
used
by
whom,
most
specifically,
we
wanted
to
understand
how
much
resource
is
consumed
by
a
given
tenant
so
who
killed
about
such
a
visibility?
A
A
We
are
behaving
such
that.
If
you
see
something
like
a
huge
volume
data
being
sent
to
neighboring
IP,
you
know
something
goes
wrong.
Lastly,
if
you
are
an
IT
director
or
protocol
manager,
you
can
use
use
digit
data
to
understand
the
resource
usage
change,
whether
my
application
is
that
gradually
consuming
more
resources
and
use
that
to
decide
whether
it's
time
to
lower
the
footprint
of
your
application
and
perform
in
virtually
heavily
unlike
a
resource,
optimization
of
your
application.
A
A
We
can
keep
a
close
watch
to
report
and
the
monitor
whole
cluster
resources
are
used,
namely
CPU
memory
or
human
GPUs
storage,
etc
and
Link.
The
reported
usage
to
kubinashi
in
native
concept,
such
as
what
had
the
CPU
and
the
memory
being
used
by
a
part.
What
is
the
size
of
a
storage
being
requested
by
a
persistent
volume
same
whether
the
storage
is
a
SSD
or
a
regular
disk?
Why
do
we
want
to
link
the
usage
to
kinetic
concept?
A
This
is
because
it
provides
us
a
powerful
generic
and
flexible
way
to
for
the
map
usage
to
challenge.
So
if
you
are
using
namespace
at
the
tenant
boundary,
you
can
just
sum
the
report
either
CPU
usage
of
pause.
You
know
give
an
m-space
which
tells
you
the
aggregated,
CPU
usage
for
the
challenge,
or
if
user
name
space
has
multiple
applications
or
single
challenger
spans
across
multiple
namespaces,
you
can
use
to
burn
anchi
label
to
do
more.
A
Slack
use
a
grouping
of
filtering
within
a
single
namespace
out
across
multiple
nemesis
across
multiple
namespaces,
depending
on
your
policy
of
managing
talents
within
the
cluster.
So
how
can
we
measure
resource
usage
in
karate?
Luckily,
with
the
decorative
resource
model
used
by
kubernetes,
you
can
you
can
obtain
a
lot
of
information
from
the
cube
API
server.
A
Similarly,
you
can
report
use
data
for
other
resources,
such
as
storage
and
even
to
even
GPU
one
of
the
missing
pieces
or
one
of
the
missing
information
from
the
cuba,
cuba.
Api
server
is
the
natural
traffic
statistics.
We
briefly
mentioned
this
before
so,
and
there
are
several
motivations
to
measure
network
traffic
usage
so
for
all,
you
can
use
natural
to
statistical
to
make
sure
your
application
is
behaving
correctly.
If
you
see
a
sudden
spike
or
data
or
traffic
sena
tune
on
the
region,
there
might
be
a
bug
or
was
your
application
might
be
under
attack?
A
Secondly,
or
some
platform,
natural
egress
is
now
really
free.
So
you
wanted
to
capture
how
much
data
is
sent
from
applications
for
each
talented
from
issue
for
cost
allocation
purposes
and
one
of
the
option.
One
of
the
possible
option
to
capture
and
monitor
natural
traffic
is
to
deploy
a
network
metering
agent
monitoring
agent
onto
each
node
allege.
The
network
monitoring
agent
talked
with,
for
example,
the
contract
able
to
check
so
many
bytes
are
sent
from
part
and
reported
the
usage
to
the
central
usage
monitoring
component.
A
I
want
to
go
a
little
I
want
to
expand
a
little
more
on
why
we
choose
to
measure
resource
requests
to
force
instead
of
the
actual
usage
or
actual
resource
utilization,
particularly
for
CPU
and
memory.
So,
first
of
all,
cube
scheduler
uses
a
resource
request
to
decide
where
a
part
can
be
fit
on
a
node,
and
once
a
party
is
assigned
to
a
note,
cube
scheduler
considers
the
amount
of
CPU
and
memory
being
reserved
for
that
part,
regardless
how
much
your
CPU
and
memory
is
actually
used
by
the
part?
A
Also
cube
scheduler
will
stop
assigning
ball
pass
to
the
node
if
well,
there's
not
enough
on
reserved
resources
available
on
the
node
and
on
the
cluster
level.
If
none
of
the
nodes
in
the
cluster
has
enough
on
reserve
the
resources
those
are
assigned
apart,
we
have
become
pending
and
it's
the
pending
part
that
drives,
for
example,
class
the
autoscaler
to
add
more
notes
into
your
cluster,
which
usually
means
it
increases
the
cost
rate
of
your
cluster.
And
similarly
another
factor
is
the
hall
resource.
Coder
works
in
kubernetes
it
constraints.
A
It
provides
a
per
name
space
limit
on
the
aggregator,
the
resource
request
in
Seattle's,
resource
utilization
and
based
on
these
factors,
we
decided
to
foster
measure
resource
requests
instead
of
actual
resource
utilization
for
CPU
and
memory.
But
we
are
the
met.
Util,
additionally,
is
still
very
useful
and
irrelevant
in
these
contacts,
for
example,
utilization
tells
you
how
many
resources
are
being
consumed
by
Vesta,
Africa
parts
or
butcher
parts
portable
parts.
A
Those
are
parts
that
you
either
don't
have
resource
requests
being
specified
or
occasionally
use
more
resources
and
being
requested
and
having
utilisation
data
allows
you
to
understand
the
detailed
application.
Behavior
nominee
watch
is
a
memory
footprint
over
time
or
is
a
CPU
usage
increase
where
new
version
of
your,
for
example,
a
new
portion
will
code
is
being
deployed
and,
lastly,
the
cut
for
certain
resources.
The
concept
of
requesters
doesn't
apply.
An
example
is
we
can
match
the
natural
egress
natural
traffic
or
natural
to
boundaries
either
into
a
scheduler
resources?
A
A
When
the
resource
usage
data
is
collected,
one
of
the
predominant
use
use
case
is
to
use
the
teacher
for
for
fine-grained
a
costly
allocation,
so
cost
allocation
for
dedicated
resources.
Relative
that
is
used
by
a
single
challenge
is
a
relatively
straightforward.
You
know
which
attendant
to
use
the
resources
from
the
user
data
and
you
can
directly
allocate
the
relevant
cost,
choose
a
specific
challenge.
A
On
the
other
hand,
if
you
are
allocating
Carter
for
sheer
resources
such
as
VM
instances,
it
is
the
slightly
more
involved
mathematically,
but
you
can
apply
some
simple
solutions,
such
as
a
linear
cost
allocation.
For
example,
if
the
user
agent
metering
data
indicates
a
challenge,
has
a
pod
requesting
200
million
use
and
has
been
running
for
60
minutes,
and
you
know
a
single
call,
VM
cos
view.
A
The
feature
currently
reports,
usage
of
for
several
computer
resources,
such
as
the
CPU
memory,
dynamic
provision,
disks
or
even
GPUs,
and
the
network
aggressive
traffic,
and
we
are
continuously
working
on
supporting
more
resources
and
the
report.
Header
usage
is
attached
with
relevant
a
kubernetes
name,
space
and
name
and
the
labels
allowing
you
to
do
flexible,
fine-grained
usage
or
an
emesis,
and
this
is
how
things
working
in
GAE.
A
So
once
you
enable
this
feature
for
your
ticket
faster,
a
usage
amatory
agent
is
a
deployed
onto
the
master
node,
the
master
node
a
is
hidden
from
from
the
user,
so
you
won't
be
able
to
see
it,
but
it
generally
use
agent
record
by
talking
to
the
cuban
api
server
and
sense
the
usage
record
to
a
bigquery
table
specified
by
your
cluster
admin.
Optionally.
If
you
care
about
the
natural
usage,
a
network
monitoring
agent
is
deployed
into
the
cluster
as
a
daemon
set.
So
there
is
a
one
part
of
the
network.
A
Metering
agents
are
drawing
on
each
of
your
cluster
node.
After
the
usage
records
are
streamed,
choose
a
bigquery
table.
The
class
data
mean
can
perform
analysis
in
boo
queries
such
as
a
general
total
CPU
usage
from
a
given
namespace.
Oh,
the
admin
can
join
the
usage
of
either
billing
data
so
that
you
can
understand
it.
For
example,
plenum
space
acosta
break
it
down.
The
enemy
can
also
export
the
data
to
other
tools,
such
as
the
data
studio
for
visualization.
A
A
You
can
definitely
do
certain
queries.
Reading
big
big
query
so
that
you
can
aggregate
a
filter
or
grouping
resources
use
each
other
together
to
understand
to
understand
the
detail,
the
resource
usage
of
ease
in
your
classroom,
but
you
can
also
populate
the
use
it
easier
to
prepare
a
so
that
you
can
chew
all
like
a
meaningful
graphs
to
visualize
the
usage
of
data.
So,
in
this
example,
we
have
pie
chart
tells
you
like
how
much
cost
is
incurred
from
each
of
the
namespace
and
the
like
a
table
in
the
middle.
A
Tells
you
like
the
details,
like
the
actual
members
of
the
total
members
that
corresponding
to
the
cause
to
generate
it
from
machine
m
space.
The
pie
chart
on
the
left
button
basically
break
down
the
cost
by
resource
type,
and
you
can
see
within
my
class
sure
most
of
the
charges
are
coming
from
the
usage
of
CPU
resources.
A
The
red
portion
is
a
culture
for
memory
resources
and
there's
a
little
like
a
yellow
portion.
That
is
a
culture
for
natural,
the
egress
traffic
and
on
the
bottom
right.
This
gives
you
the
cost
of
change
from
different
namespaces
over
time,
and
you
can't
you
see.
There
are
the
abnormal
spikes
from
the
station
namespace,
and
there
was
a
due
to
accidental
error
in
which
we
we
scale
a
deployment
event
by
ten
times.
A
And
that
trades.
That
concludes
the
demo
and
also
brings
me
to
the
end
of
my
presentation
so
just
to
wrap
up.
We
talked
about
why
we
need
to
operate
a
market
can
in
the
cluster,
because
they
easy
is
a
classroom
management
and
lowers
the
cost.
We
understand
that
you
know
Marty
challenger,
cluster
potential
resource
potential
use,
eg
metering
is
important,
has
an
has
many
use.
A
Cases,
for
example,
allows
you
to
monitor
and
folks
has
to
resource
usage,
detect
abnormal
is
telling
a
behavior
and
last
by
not
least,
hello,
teacher
aggregators
cluster,
costing
two
patan
into
pieces
or
pertinent
charges.
We
wanted
to
say
using
a
metering
is
supported
in
GTE
and
beta
feature,
and
if
you
want
to
know
more,
the
documentation
is
a
link
checking
the
jingle
in
the
slide,
and
that
concludes
my
presentation.
Thank
you
them
very
much
for
your
time.
A
Yeah,
this
is
definitely
something
we
are
looking
to
you
before.
We
were
sorting
we.
We
want
to
understand
how
other
things
that
works
in
different
environment,
like
besides
hu
ke,
making
sure
that
the
interface
is
generic,
such
that
you
can
use
a
join
with
the
billing
information
from
a
different
club
provider,
or
you
can
specify
your
own
price
sheet
so
that
we
can,
based
on
the
price
listed
in
the
sheet,
calculate
cost
for
you.
So
there
has
been
an
ongoing
discussion
and
that's
one
of
the
reason
I'm.
A
A
Yeah
so
I
think
certain
platform
provider,
the
piell's
like
CPU
and
memory
in
two-dimension,
but
there
are
some
like
other
intentions,
is
the
actual
VM
is
appeals
as
like
sojanet,
so
like
it
from
the
pinna.
You
see
like
a
culture
for
a
single
VM
instead
of
coaster
for
CPU
and
memory
individually.
In
that
case,
if
you
might
want
you
to
mix
match
like
a
either
by
average
of
the
cost
of
CPI
memory
together
or
to
your,
like
your
own
combination
of
two
costly
factors
together.
A
You
can
also
do
something
like
a
very
simple
or
very
complex.
The
simple
solution
is:
if
we
just
count
company
as
a
negative
credit,
you
can't
do
linear,
like
a
breakdown
for
those
negative
credit
and
apply
the
linearized
accreditor
towards
an
m-space,
for
example,
but
depending
on
environment
is
sometimes
credited
like
from
the
cloud
provider.
A
The
the
tragic
are
not
attributed
uniformly
across
the
vm
insider
cluster,
so
we
see
cases
the
people
who
just
want
to
aggregate
and
aggregate
credit
and
apply
to
a
cluster
level,
but
we
also
see
cases
where
people
see
some
node
is
more
important,
so
we
only
want
to
apply
credit
to
choose
those
nodes,
so
that
is,
the
more
involved
are
like
detecting.
Where
else
things
are
running
mapping
from
usage
to
discount
and
do
you
gather
like
a
getter
like
a
final
result.
A
Hopefully,
like
as
I
said
that
we
are
looking
to
options
to
eventually
maybe
over
open
source
aging
in
the
future,
and
after,
like
one
of
the
tuition
to
open
source,
ager
will
be
to
allow
edge
to
walk
across
the
Tiffany
environment
and
non-primate.
Definitely
one
of
the
environment
that
we
we
all
consider
about.
A
No
right
now,
but
this
is,
that
is
something
we
have
been
considering
to
clash
with
utilization
data,
one
of
the
opportunities
to
talk
with
the
metric,
a
server
which
I
have,
which
I
think
has
been
a
standard
like
component
running
in
kubernetes
and
fetching
data
from
there,
but
depending
on
your
needs
right.
Metric
server
performs
its
own
aggregation
averaging
over
certain
time
period.
So,
depending
on
how
fast
you
want
to
detect
the
utilization,
there
might
be
other
alternatives
to
be
considered.
A
A
A
So
the
question
is
about:
how
do
you
know
BC
GPU
is
being
used,
I
think
this
we
do
have
work.
So
when
you,
when
partner
requester
a
GP
resources
in
the
resource
list,
you
specify
like,
what's
the
type
of
resource
being,
what
type
of
GPO
being
requested-
and
we
came
from
there
understand,
like
water,
is
being
used.
What.
A
Yes,
but
we
do
consider
the
limit,
but,
like
internally
within
kubernetes,
request
reason:
ball
I
get
the
fund
a
mantle
like
a
concept
of
for
resource
allocation
right,
so
you
can
put
it
another
way
like
the
summation
of
requester
were
all
positive
running
on
a
single
node
can
be
much
higher
than
the
know.
Deepika
path.
We
all
know
they
allocatable.
That
would
be
relatively
strange,
like
a
situation
if
you
do,
for
example,
cost
allocation
based
on
that
right.
So
you
seems,
like
you
are
overusing
a
large
than
what
is
actually
available.