►
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2023 in Amsterdam, The Netherlands from April 17-21. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Energy Efficient Placement of Edge Workloads - Parul Singh & Kaiyi Liu, Red Hat
Currently, the energy consumption metrics are only available at node levels. There is no way to obtain container-level energy consumption. Autoscalers and schedulers really need pod-level metrics data in order to obtain energy savings from resizing or migrating containers. The presentation introduces Kubernetes-based Efficient Power Level Exporter (Kepler) and its integration with Kubernetes. By leveraging eBPF programs, Kepler probes per container energy consumption related system counters and exports them as metrics. These metrics help end users observe their containers’ energy consumption and allow cluster admins to make intelligent decisions on achieving energy conservation goals. The next part of the presentation shows that Kepler can be easily integrated into Prometheus and render time series metrics into Grafana. At last, we will demonstrate sustainable management of clusters by leveraging Kepler, Cloud Native patterns, Observability and Kubernetes features like node selector, node labels, node name, affinity and anti-affinity to achieve energy efficient placement of edge workloads based on the ideal load of an edge destination.
A
Now
we
are
going
to
see
how
you
can
incorporate
sustainability
in
Computing
and,
specifically,
we'll
talk
how
you
can
make
energy
efficient
placement
of
workloads
on
edge
so
who
we
are?
I
am
parul
Singh
I
work
as
a
senior
software
engineer
at
red
hat
and
work
in
this
sustainability
team,
where
we
are
incubating
various
tools
and
Technologies
to
make
green
efficient
computing.
B
A
So
we
thought
that
the
best
way
is
to
take
a
community-based
initiative
on
environmental
sustainability,
so
we
already
have
a
cncf
environment,
sustainable
tag.
If
you
are
interested
feel
free
to
check
out
a
proposal,
we've
also
collaborated
in
the
community,
one
of
our
POC
that
we
did
was
carbon
away
scaling
with
Kera.
We
did
this
with
Microsoft,
where
we're
investigating
how
to
use
a
carbon
intensity
to
make
workload
scaling
decisions.
We've
also
collaborated
with
IBM
on
making
a
vpa
recommender
for
kubernetes
that
take
into
account
container
level
energy.
A
So,
let's
get
into
the
agenda.
It's
very
simple.
We'll
start
with
some
background
of
how
things
are
right
now
in
sustainability
and
Computing,
then
we
will
very
briefly
talk
about
a
sustainability
stack
specifically
to
projects
Kepler
and
the
model
server,
and
then
we
will
get
to
their
demo,
which
is
the
main
objective
of
today's
talk.
How
you
do
energy
efficient
placement
of
the
workloads.
B
So
we
have
a
quote
here
that
sort
of
sums
up
how
bad
carbon
emissions
are
in
the
tech
field,
so
in
2021
an
ACM
technology
brief
estimated
that
the
information
and
communication
technology
sector
contributed
between
1.8
and
3.9
of
global
carbon
emissions,
which
is
actually
more
than
the
total
carbon
emissions
emitted
by
Germany
and
Italy.
Combined.
A
So
that
haunts
us
to
ask
question:
how
can
you
do
energy,
how
you
can
measure
energy
in
while
you're
doing
computation,
when
you
don't
have
access
to
the
infrastructure?
Maybe
it's
on
private
on
public
cloud
or
it's
on
premise:
it's
a
hybrid
Cloud
structure.
It's
a
bare
metal!
How
are
you
measuring
energy
and
specifically
indirectly,
where
you
cannot
install
specific
Hardware
to
measure
it
and
how
are
you
attributing
energy
consumption
of
workloads
in
a
shared
environment
where
you
have
shared
resources
for
processes,
pods
and
containers,
so
that
got
us
into
thinking?
A
Let's
build
a
sustainability
stack
and
the
first
project
that
we
got
started
was
Kepler
and
the
Kepler
model
server
before
we
get
into
the
projects.
Let's
talk
about
mythology,
how
do
we
calculate
or
how
we
measure
energy,
so
our
principle
is
based
on.
A
mythology
is
based
on
the
principle
that
CPU
power
consumption
can
be
directly
proportional
to
CPU
power
energy
or
like
how
much
CPU
Energy
power
you're
consuming.
So,
for
example,
let's
say
you
have
a
container
that
is
consuming
10
of
CPU.
A
You
can
say
that
it
attributed
to
10
percent
of
CPU
power,
or
you
have
a
workload
that
has
five
containers
and
it
is
running.
Each
of
them
are
consuming
10
percent
and
totally
you
can
say
that
totally
they
attribute
it
to
50
of
power
consumption
and
we
we
didn't
make
this
up
our
principle.
You
can
check
out
the
white
paper
that
we've
referenced
and
this
a
lot
of
work
is
being
done
in
this
area,
so
our
Metrology
is
based
on
the
many
scientific
papers.
A
A
So
Kepler
does
three
things:
the
first,
its
reports
per
part
energy
consumption,
including
from
resources
like
CPU,
GPU,
RAM,
and
it
supports
bare
metal
as
well
as
VM,
and
it
uses
Cloud
native
stack
like
Prometheus
and
grafana,
to
render
dashboard
where
you
can
see
the
energy
consumption
next
Kepler
is
supposed
to
measure
energy
consumption.
So
it's
very
important
for
us
that
the
energy
footprint
of
Kepler
itself
is
very
low.
A
So
we
use
ebpf
to
probe
software
and
kernels
to
get
these
readings
and
at
last
Kepler
uses
machine
learning
models
where
you
don't
have
access
to
the
existing
power
monitors
on
the
on
the
cluster.
Kai
will
get
more
into
that,
but
wherever
you
don't
have
an
existing
power
meter,
Kepler
uses
its
own
machine
learning
models
to
get
the
readings.
B
So,
let's
take
a
look
at
the
second
part
of
this
Kepler
sustainability
stack,
which
is
the
Kepler
model
server
project,
so
a
little
bit
about
the
Kepler
model.
Server
by
default.
Kepler
will
use
a
supported
power
meter
or
some
power
measurement
tool
to
measure
node
level,
energy
metrics
like
CPU,
core
and
dram,
and
then
it'll
use
that
to
try
and
predict
pod
level
energy
metrics.
So
what
happens
when
Kepler
does
not
have
access
to
a
supported
power
meter?
B
That's
where
the
Kepler
model
server
steps
in
to
save
the
day,
so
the
Kepler
model
server
will
provide
trained
models
for
Kepler
to
predict
these
missing
energy
metrics
and
they
use
software
counters
and
performance
metrics
to
as
the
features
so.
The
current
Tech
stack
of
the
Kepler
model
server
includes
tensorflow
Keras
for
the
models
and
flask
and
Prometheus
for
exporting
these
models
for
Kepler
to
use.
So
let's
take
a
look
at
some
examples
of
the
node
level
energy
metric
models
that
the
capillary
model
server
currently
has
implemented.
B
So
the
first
one
will
predict
CPU
core
energy
consumption,
we're
currently
using
a
linear
regression
model
for
that
and
it
uses
the
following
software.
Software
counters
and
performance
metrics
as
the
features,
and
it's
also
capable
of
online
learning.
That
way
we
can
refit
and
retrain
the
model
in
the
future,
with
with
more
up-to-date
data
to
make
sure
the
performance
does
not
get
worse
and
then
another
example
is
also
a
linear
regression
model,
which
also
is
capable
of
online
learning.
B
However,
this
time
it's
used
to
predict
dram
energy
consumption
and
it
uses
a
different
set
of
performance
metrics
as
the
features.
So,
let's
take
a
look
at
how
the
model
server
fits
in
Kepler
as
a
whole,
we
can
effectively
split
the
model
server
into
two
parts.
The
first
part
is
a
training
phase,
so
here
we'll
have
training.
B
Workloads
on
a
cluster
Kepler
will
retrieve
the
node
energy
metrics
and
performance
metrics
from
the
cluster
and
Kepler
can
do
this,
because
a
power
meter
will
be,
of
course,
provided
it'll,
then
aggregate
these
node
energy,
metrics
and
performance
metrics
in
a
collector
and
then
export
them
as
Prometheus.
The
Kepler
model
server
will
then
scrape
these
node
level,
energy,
metrics
and
performance
metrics
from
Kepler's,
Prometheus,
metrics
and
then
I'll
set
up
training,
testing
and
validation.
Data
sets
to
train
and
evaluate
and
then
save
the
regression
models.
B
B
So
we
can
export
like
the
archive
file
of
the
model,
or
we
can
like
export
the
model
weights
for
Kepler
to
use
to
make
a
a
prediction
of
the
energy
Matrix,
and
we
can
do
that
with
flask
routes
and
Prometheus
metrics
and
in
the
future.
We
also
want
to
try
and
include
try
to
export
the
models
using
open,
Telemetry
metrics.
A
Let's
get
to
the
interesting
part,
how
you
do
carbon
intensity,
aware
scheduling,
so
the
use
cases
are
considered
a
scenario
where
you
want
your
workload
to
be
fueled
by
as
much
as
renewable
energy,
say:
solar,
energy
or
wind
energy.
During
the
day
and
at
night
you
want
to
use
fossil
energy
because
the
renewable
energies
are
not
available
or
you
just
want
to
control
the
carbon
intensity
of
your
workload.
So
the
second
is
what
we
are
going
to
demonstrate
today,
how
you
can
control
the
carbon
intensity.
A
The
use
case
premise
is
you:
have
a
multi-node
cluster
and
the
nodes
are
running
in
different
Zone,
and
this
is
useful
for
long
running,
batch
or
machine
learning
workloads
where
you
are
online
training,
a
machine
learning
model,
and
the
reason
is
because
we
are
migrating
the
workloads
according
to
the
carbon
intensity.
So
let's
say
that
you
have
a
workload
where
just
the
containers
fires
up
it
does.
Some
computation
gives
you
a
result
and
just
dies.
You
don't
need
to.
A
It
won't
be
very
useful
if
you
introduce
this
because
it's
just
running
for
a
very
short
time,
so
this
is
best
suiting
for
our
workloads.
That
are
running
for
many
hours
or
like
over
days.
Best
is
like
the
machine,
learning,
training
models
and
the
demo
setup
is.
We
have
three
nodes:
openshift
cluster.
We
are
using
Prometheus
as
a
monitoring
stack
and
we
would
be
using
features
of
kubernetes
like
tanes,
tolerations
and
node
selector
to
control
the
placement
of
the
workloads
and
we
have
a
carbon
intensity.
Forecaster
Kai
Ken
will
tell
you
about
that.
Right.
B
So
that
leads
to
the
question:
how
do
we
forecast
the
carbon
intensity
for
a
node?
So
we
have
an
exporter
which
will
scrape
from
numerous
data
sources
from
numerous
public
energy
apis
so
such
as
electricity
map,
what
time
National
Grid
and
it
will
retrieve
this-
this
carbon
intensity
data,
As
Time
series
data.
So
then
this
will
be
exported
as
Prometheus
metrics.
B
The
carbon
intensity
forecaster
will
then
come
in
and
scrape
these
Prometheus
Prometheus
metrics
retrieve
it
as
a
carbon
intensity
time
series
data
and
they
can
use
that
to
update
the
models
for
each
of
the
node.
So
the
models
can
now
be
no
used
to
predict
the
forecasted
carbon
intensity
of
a
note
some
hours
in
the
future,
and
it's
important
to
note
that
the
carbon
intensity,
forecaster
and
the
exporter
exporter
are
extendable
interfaces.
So
we
want
it
so
that
the
carbon
intensity
forecaster
can
introduce.
A
So,
let's
get
into
the
flow
of
how
this
work
we
periodically
assign
node
labels
according
to
the
forecasted
carbon
intensity
of
the
Zone,
the
node
is
present
in
and
there's
a
crown
job
that
makes
request
to
the
carbon
intensity
forecaster
and
the
carbon
intensity.
Forecaster
will
return
the
carbon
intensity
in
future
sometime
in
future,
for
example,
like
two
hours
and
for
example,
we
have
three
nodes,
so
the
carbon
intensity
for
Caster
Returns,
the
forecast
for
casted
carbon
intensity
for
the
nodes
we
sort
them
from
high
to
low
the
highest,
get
a
label
red
the
lowest.
A
Has
a
label
green
and
anything
in
between.
Has
a
label
yellow,
not
part,
also
have
to
specify
its
intention.
So
for
our
demo,
this,
the
part
that
we'll
be
running
says
that
I
I,
like
the
note
that
has
the
label
carbon
intensity
as
green,
and
we
also
have
to
provide
the
tolerations,
because
the
part
does
not
want
to
be
assigned
to
a
node
that
has
carbon
intensity
as
red.
So
you
can
see
in
the
diagram.
A
There
are
three
nodes:
node
one
node,
two
node,
three,
the
node
one
has
labeled
red
and
whichever
node
has
label
red.
You
also
add
a
taint
that
this
node
is
very
high,
has
very
high
carbon
intensity
and
node
2
and
node
3
has
no
tense,
but
label
green
and
yellow
and
in
the
pot
spec.
You
see
that
we
have
a
tolerations
for
the
key
carbon
intensity
with
the
value
as
red
and
effect
as
no
execute
and
a
toleration
seconds
of
five.
A
So,
let's
see
how
the
transitioning
happening.
Let's
say
a
node
is
transitioning
from
green
to
Red,
which
is
node
one
and
the
Pod
was
already
running.
So
you
will
label
the
node
as
red
it
get
tainted
and
the
taint
is
carbon
intensity,
red
and
no
execute.
So
the
pot
gets
evicted
from
node
one
and
it
gets
assigned
to
node
two,
which
is
transitioning
from
label
red
to
green
and
the
label.
A
So
you
have
to
label
the
node
as
green,
and
you
have
to
untaint
the
node
and
be
aware
that
tainting
just
ensures
that
pods
are
evicted
by
the
nodes
if
Parts
have
no
Toleration
for
the
tail,
so
the
labels
ensures
where
the
Pod
would
be
assigned.
But
if
a
pod
is
already
running
on
a
node,
the
tolerations
ensures
that
they
get
evicted
from
the
node.
If
the
carbon
intensity
is
higher
than
the
Pod
wants.
B
So,
let's
take
a
look
at
how
it
all
fits
together.
So
on
the
left,
you
can
see
the
public
energy
apis
and
the
data
sources
that
we
have
available
that
provide
us
with
the
carbon
intensity
data.
So
we
have
like,
for
instance,
National
Grid
electricity
map.
What
time
and
in
the
future,
we
want
to
include
more
data
sources.
So
then
the
carbon
intensity
forecaster
will
scrape
from
these
data
sources
and
and
retrieve
the
carbon
intensity
time
series
data
which
will
then
be
exported
with
as
Prometheus
metrics.
B
A
A
Yeah
it's
here
and
it
has
no
taint
available.
So
now,
let's
see
how
my
pod
spec
looks
like
so
my
part
is
called
CI
test.
It
has
a
node
selector
with
carbon
intensity
as
green,
and
we
have
the
chains
over
here
that
says:
carbon
intensity,
red
no
execute
N5
so
and
over
here
in
the
top
left.
You
see
that
we
don't
have
the
Pod
running
so
I'm
just
going
to
run
that
pod.
A
Okay,
so
the
pot
is
being
created
and
it
will
take
some
time
yeah,
so
it
got
terminated.
Now.
Let
me
explain
you
why
it
was
terminated.
We
are
not
modifying
the
scheduler,
so
the
scheduler
just
assigned
the
part
to
wherever
it
wants
to,
but
the
label
and
the
chains
ensures
that
the
Pod
get
assigned
to
the
right
node
so
the
first
time
that
we
rang
it
terminated
because
maybe
that
node
had
attained,
but
now
it
is
running
fine
and
let's
just
see
which
node
it
got
assigned
to.
A
Okay,
so
you
can
see
that
the
part
this
node
has
the
label
as
green
and
the
Pod
has
been
scheduled
on
the
same
node.
So
that's
that's.
We
don't
know
we
are
not
dealing
with
the
scheduler.
The
scheduler
is
not
touch
it's
just
the
node
selector
and
the
chains
and
Toleration
that
is
taken
care
of.
So
we
have
some
lessons
that
we
learned.
B
So
the
first
lesson
that
we
learned
was
that
finding
zone
carbon
intensity
data
is
actually
quite
very
difficult.
There's
not
too
much
public
energy
apis
out
there
that
let
us
use
their
carbon
intensity
data
and
also
of
the
ones
that
do
allow
us
to
use
their
carbon
intensity
data.
Sometimes
there
are
time
points
that
are
missing
so
in
the
future.
We
want
to
try
and
remedy
it
remedy
this
by
trying
to
get
more
and
more
access
to
carbon
intensity
data
sources.
A
We
also
realized
that
our
queries
are
very
simple,
like
we
just
make
queries,
what
is
the
current
or
what
is
the
average
carbon
intensity
in
zone?
But
we
need
to
have
more
a
solution
for
more
complicated
queries
like
which
zone
has
the
lowest
carbon
intensity
and
is
the
carbon
intensity
low
within
the
past
24
hours,
and
we
also
want
to
support
multiple
electricity,
carbon
emission
provider.
A
So
the
roles
ahead
for
us
is
to
take
this
approach
and
apply
it
to
a
multi-cluster
scenario,
and
we
are
pro
we
are
exploring
how
you
can
do
that
with
kcp
and
integrate
carbon
intensity
awareness
in
kubernetes
six
scheduler
profile.
So
we
we
are
not
touching
the
scheduler,
but
we
want
this
to
be
assigned
into
a
scheduler
itself.
So
we
are
exploring
options.
A
That's
all
pretty
much
so
here
are
some
references.
If
you're
interested
in
our
projects
are
a
whole
sustainably
stacked
this
QR,
we
will
lead
you
to
a
GitHub
organization,
where
you
can
see
all
these
projects
and
the
link
to
the
white
paper
that
that's
a
base
for
us.
You
can
explore
that
and
we
are
ready
to
take
questions.