►
From YouTube: Smart Green Computing Cloud Native Operations - William Caban & Federico Rossi, Red Hat
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe in Amsterdam, The Netherlands from April 17-21, 2023. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Smart Green Computing Cloud Native Operations - William Caban & Federico Rossi, Red Hat
Speakers: Federico Rossi, William Caban
The European Union and the United States have set up a target of at least 50% - 55% net reduction in greenhouse gas emissions by 2030. But, with the sprawling of the cloud-native workloads and the increased demand for resources: are we doing enough?
Many community efforts and open source projects enable the observability of the power consumption from software resources to hardware resources. How can we combine the visibility provided by these tools to achieve the organization's sustainability goals?
In this talk, we combine CNCF projects and other open source communities tools to create and continuously improve Machine Learning models for cluster operations. These ML models consider a holistic view of a system: from application runtimes, node metrics, cluster metrics, and network metrics to the tracing of the interactions among the distributed components. These ML models are used for the "smart operations" of the distributed systems aligning to the organization's carbon and power optimization goals.
A
A
Well,
it
happened
that
the
Telco
itself,
the
Telco
industry,
represent
about
two
percent
of
the
total
Humanity
consumption
of
energy.
So,
yes,
we
should
be
doing
something,
and
this
is
part
of
an
integration
work
that
we
created
and
what's
in
the
name,
is
following
so
smart,
because
we
want
to
make
sure
that
we
can
do
far
more
than
just
a
traditional
automations.
We
want
to
be
able
to
use
machine
learning
models
and
apply
AI
to
achieve
certain
goals.
A
In
this
particular
case,
we
are
talking
about
the
goal
of
Green
Computing
and
following
Cloud
native
principles,
and
it
should
work
for
experimentation
and
the
same
sexual
work
for
operations.
So
that's
that's
really
what
this
means
now.
Some
concept
here
goal-driven
workload
management.
Let's
start
with
that,
and
the
idea
here
is
that
we
don't
want
just
to
go
and
have
another
specialized
orchestrator.
We
want
to
be
able
to
Define
goals
today
we're
talking
about,
for
example,
the
goal
of
okay.
A
Let's,
let's
do
a
certain
amount
of
Maximum
certain
amount
of
CO2
on
cluster
upper
node
in
material
of
the
source
that
that
particular
node
or
cluster
can
be
being
powered
by.
So
you
see
there,
for
example,
on
carbon,
petroleum
and
the
others.
Each
one
of
them
has
certain
penalty
on
the
CO2
emission,
but
when
we
talk
about
a
goal,
driven
basically
orchestration,
so,
for
example,
the
traditional
way
here,
everything
will
pretty
much
be
spread
across
and
our
CO2
will
be
off
now.
A
Okay,
so
that
that's
part
of
the
the
idea
behind
the
goal
driven
workload
and
the
same
applied
to
any
other
goal
which
could
be
like
okay,
I,
want
to
run
this
on
more
power,
efficient
nodes
or
run
this
to
reduce
my
energy
consumption
or
to
have
higher
availability
or
lower
latency
to
our
customers.
So
that's
the
idea
here
and
it
applies
the
same
for
node
old
clusters.
A
Another
concept
is
the
idea
of
domain
aware
warlock
management,
and
these
two
needs
to
be
together,
because
not
all
the
workloads
can
be
treated
the
same,
so
understanding
the
goal.
We
can
also
map
that
to
the
realities
of
that
type
of
workload
and
again
being
from
Telco
world.
That
means
we
can
relate
to,
for
example,
what
happened
that
let's
say
during
cubecon
is
a
it's
a
big
conference
from
what
we
can
see
here
during
any
other
time
of
every
night.
A
There
would
not
be
too
much
activity,
so
basically,
you
can
really
dig
down
or
basically
remove
workload
that
will
help,
for
example,
on
the
caching
or
things
like
that
on
the
Telco
side.
Now,
once
it
starts
detecting
that
there's
people
coming,
then
it
can
go
and
increase
those.
So
all
of
that
knowledge
on
how
to
do
it
is
very
specific
to
that
domain.
The
same
thing
will
apply
to
streaming
or
other
specialized
workloads.
A
So
with
that,
so,
okay,
let's
make
sure
that
we
can
do
an
integration
that
is
composable
that
it
works
for
experimentation
and
the
operations
team.
These
are
problems
that
has
been
solved
already,
so
we
start
very
basically
with
adopting
the
getups
model,
because
that
give
us
basically
a
declarative
way
and
aversing
way
of
rebuilding
anything.
A
So
that
means
that
if
my
infrastructure
here
or
platform
highlight
there
by
letter
e
is
destroyed,
I
still
have
a
way
to
recover
that
and
redo
exactly
to
the
point
right
before
it
was
destroyed
without
having
to
worry
about
other
mechanisms
or
traditional
way
of
doing
this.
The
other
part
of
this
is
each
one
of
these
blocks
here
can
scale
independently.
A
So
even
when
they
have
certain
relationship,
each
one
can
scale
as
needed
by
an
organization.
So
that
means
I
can
do
this
from
a
centralized
location
or
I
can
have
this
highly
distributed
again.
It's
based
on
the
what
the
organization
will
meet
the
same
will
happen,
for
example,
on
F,
where
we
are
collecting
all
the
metrics
and
logs
yeah
on
certain
organization.
We
can
centralize
those
on
Central
certain
metrics.
A
We
can
do
that
as
well,
but,
for
example,
again
on
our
wall
of
the
Telco
world,
we
have
situations
where,
just
at
single
device,
for
example,
antenna
can
be
generated
around
10
gigabits
per
second
of
metrics.
Imagine
now
having
a
hundred
of
those
in
a
small
City
like
this,
how
much
bandwidth
you
will
need
to
have
if
you
want
to
move
that
outside
the
area.
So
that's
why
it
needs
to
be
able
to
have
those
blocks
in
a
way
that
can
be
scaled
and
distributed
when
whenever
needed.
A
So,
during
the
day,
if
they
are
using
renewable
energies,
they
can
be
using
more
solar
or
if
they
have
win
win,
but
there
will
be
that
that
will
be
changing
every
single
day
of
the
weeks
and
every
single
time.
So
that
means
we
need
to
bring
those
as
additional
metadata
and
metrics
to
enhance
those
those.
B
A
And
other
metrics
that
we
already
have
in
in
the
system.
So
that's
where
the
calculated
or
derivative
derived
one
will
come
in
that's
another
notion,
for
example
outside
that
there's
equipment,
there's
switches
and
routers,
so
we
also
want
to
account
for
their
consumption
of
power
consumption
and
how
that
impacts,
for
example,
when
we
communicate
to
services
in
the
same
location.
So
when
account
for
that
as
part
of
the
CO2
emissions
that
that
workload
is
generating
or
that
service
from
there,
we
have
to
evolve
into
okay.
So
GitHub
is
great.
A
You
have
a
declarative
mode,
but
at
scale
maintaining
those
declaration.
It
will
be
quite
painful.
Think
of
the
Declaration
I
have
all
these
yaml
files,
so
we
need
an
easier
way
for
let's
say,
executive
or
the
business
owner
to
declare
what
they
really
want
and
that's
where
the
intent
driven
identification
of
those
constraints
come
into
the
table.
Now,
there's
everything
else.
We
know
how
to
do
it
that
part
for
now
we're
gonna
integrate
it
here
when
we
bring
the
actual,
basically
all
the
envelopes,
its
area
or
stack
in
this
case.
A
That
also
have
the
governance
and
the
the
ability
to
do
experimentation
with
actual
data,
but
remember
since
we
have
to
design
this
in
a
way
that
it
works
for
centralized
or
highly
distributed
environment.
That
means
that
part
is
also
something
that
can
go,
centralized
or
can
be
highly
distributed
so
that,
for
example,
if
I
go
back
to
to
my
previous
example
of
when
everyone
is
here
in
kubecon,
so
those
models
if
I,
want
to
really
be
accurate,
I
have
to
run
those
models
in
this
area.
A
I
cannot
try
to
centralize
that,
because
it's
just
too
much
data
to
move
and
by
the
time,
I'm
finishing
moving.
It
is
useless
because
it's
a
time-bound
value
metrics
so,
and
in
this
case
the
internal
driven
constraints
basically
will
work
at
that
layer,
and
now
we
have
basically
goal
driven
versioning
system
that
we
can
do
all
these
operations.
A
So
at
the
end,
that's
the
the
full
stack.
Now
there
are
some
initial
use
cases
that
we
were
looking
to
to
tackle
with
this
there's.
There
are
some
that
are
completely
on
the
Telco
sides,
but
some
that
are
applicable
to
everyone.
For
example,
here
the
maximum
CO2
emission
per
node
upper
cluster
that
everyone
can
relate
to,
so
that's
the
one
that
we're
going
to
be
talking
today
and
Federica
will
be
doing
that
part.
But
let
me,
for
example,
explain
the
others.
A
You
see
that
auto
theory
of
services,
so
a
challenge
that
it
will
happen
to
any
organization,
doing
user
facing
type
of
services
at
scale
which
can
be
streaming
service.
It
can
be
any
of
those
is
that
when
we
start,
we
have
very
few
subscribers.
For
example,
it
makes
ton
of
sense
have
that
on
a
public
cloud
now,
if
you're
doing
a
streaming
service-
and
you
start
having
a
lot
of
subscribers,
just
the
egress
cost
will
affect
your
financial
models.
A
So
that's
when
you
want
only
under
those
circumstances,
to
bring
that
on
premise
and
now
think
having
that
stack
where
we
can
correlate
and
identify
all
these
patterns.
So
now
that
identify
the
patterns
or
okay
based
on
the
financial
model,
this
time
I
can
maintain
my
my
Revenue
or
the
profit
per
se.
Then
I
can
do
this
Auto
tiering
of
the
services
and
and
bring
Services
as
needed,
and
and
bring
them
back
out
again
based
on
what
it
makes
sense
at
that
moment
for
the
financial
model.
A
The
last
one
I
want
to
highlight
here
is
this
smart
maintenance,
and
what
happened
is
that
up
to
now,
a
lot
of
the
industries
still
consider.
Okay,
we'll
have
maintenance
windows
and
it
might
work
on
on
a
lot
of
the
industry
or
commercial
industry,
but
that's
not
working,
for
example,
for
the
world
that
I
have
to
work
with,
because
it's
the
Telco
a
regular
deployment,
for
example,
5G
antennas
that
can
range
from
50
000
to
100
000
of
those
each
one
will
have
one
to
ten
or
more
service,
directly
tiled,
plus
everything
else.
A
So
four
hours
maintain
this
Windows
to
just
be
able
to
serve
100,
000
location
and
catch
up
with
kubernetes
release.
Schedule
will
not
cut
it.
It
doesn't
matter
how
many
years
I
have
so
we
have
to
be
more
intelligent
and
that's
where
this
smart
maintenance
operations
comes
in,
because
if
we
can
detect,
for
example,
there's
a
lot
of
people
this
week
here.
So
I
will
not
touch
anything
I,
see,
there's
less
usage
next
week,
so
I
can
go
and
increase
the
coverage.
A
C
We
have
three
nodes
on
which
they
have
different
power
source.
Of
course,
we're
just
simulating
them
all
right,
one
for
call
petroleum
and
natural
gas,
and,
as
William
was
explaining,
every
single
power
source
comes
with
a
penalty
right.
There
are
apis
where
you
can
access
to
find
out
exactly
when
you
have
a
server
on
a
data
center
or
even
at
home,
depending
where
the
area
where
you
are
you
can
find
out
if
the
electricity,
the
power
that
you
receive,
is
powered
by
by
coal,
petroleum
or
natural
gas.
C
How
do
we
achieve
that?
Well?
This
is
the
high
level
solution
architecture.
There
are
quite
a
lot
of
components
involved.
This
is
the
high
level
and
we're
gonna
drill
down
on
every
single
one,
but
at
a
higher
level
we
have
a
governance
that
uses
a
policy
engine
and
the
reason
why
there
is
this
governance
is
because
we
want
to
enforce
every
workload
we
run
on
the
cluster
to
use
the
Visa
optimization
for
the
CO2
emissions,
and
that
means,
as
a
cluster
administrator
I
can
enforce
Everybody.
C
This
scheduling
part
that's
where
we
have
the
intelligence
and
we
will
drift
down
on
on
all
the
specific
components,
but
the
magic
happened
thanks
to
a
number
policy
system,
a
policy
that
controls
how
the
schedule
the
scheduler
decide
where
the
workload
needs
to
to
be
scheduled
to
be
placed
on
the
nodes
and
next
a
metric
pipeline
as
William
was
mentioning.
We
need
to
to
feed
the
system
with
metrics
without
the
data
we
don't
have
visibility
about,
what's
happened,
so,
let's
move
to
the
next
slide
all
right.
C
Now
the
next
component
is
the
analytics
am
AIML
engine,
so
we
take
all
those
metrics.
We
combine
them.
We
add
the
penalty
that
we
discussed
before
and
we
feed
it
to
a
model-
the
user,
in
this
case
the
XG
boost
it-
makes
the
machine
learning
model.
It's
paid
out,
an
output
that
values
for
our
for
us
are
the
new
metrics
that
we
call
Smart,
Ops
metrics.
C
Then,
what's
going
to
happen,
we
export
also
both
of
our
metrics
for
promatius,
so
we
have
another
service
Monitor
and
then
the
Matrix
goes
back
on
parameters.
So
you
see
the
flow
we
start
from
the
node
we
collect.
Metrics
goes
and
chromatous.
We
pull
it
from
promatius.
We
processes,
we
run
it
with
a
machine
learning
model,
we'll
spit
it
out.
We
we
get
it
by
compromatives
once
Varun
primates.
At
this
point,
it's
somehow
we
need
to
expose
those
metrics
for
consumption,
for
the
scheduler
to
do
this.
Intelligent
scheduling
that
we're
talking
about.
C
So
in
order
to
achieve
that,
we
have
a
component
called
Matrix
proxy,
that
all
it
is
is
just
a
simple
python
flask
that
it
does
a
a
query
on
the
back
end.
It
uses
API
service.
If
you
never
use
the
CR
for
API
service,
remembering
kubernetes
everything
as
an
API
and
with
API
service,
you
can
expose
metrics
okay.
C
So
this
is
the
metric
pipelines
all
of
this
to
just
expose
back
to
the
scheduler,
something
that
is
called
a
kubernetes
node
metric.
So
why
is
a
node
metric
because
the
scheduler
make
decision
where
to
schedule
based
on
nodes,
so
it's
filtered
on
nodes,
so
we
need
to
be
presented
to
the
scheduler
as
a
kubernetes
node
metric
all
right.
Now
we
get
into
the
interesting
part
the
governance
and
scheduling
that
I
was
talking
about
before.
C
C
If
you
don't
know
what
it
is,
look
at
it,
it's
awesome.
There
was
a
talk
also
yesterday
about
it
using
qvarna.
We
have
a
lot
of
granularity
of
how
you
can
manipulate
in
CR,
do
admission
generation,
mutation
and
so
on.
So
we
use
kivierno
to
pretty
much
every
part
of
the
deployment
stateful
set
that
gets
into
the
cluster.
It
gets
injected
the
spec
to
use
the
custom
scheduler.
C
Okay.
Now,
let's
get
the
label
on
the
scheduling,
because
you
know
we
have
our
kubernetes
scheduler
right,
but
you
can
see
right
here.
We
have
something
called
secondary
scheduler.
The
reason
is
we
did
this
proof
of
concept
on
openshift
and
let's
say
that
you
don't
have
much
flexibility
if
you
want
to
make
changes
to
the
scheduler
on
openshift
you're
not
going
to
be
able
to
do
it.
We
provide
some
profiles,
but
you
cannot
make
changes
at
the
cube,
scheduler
configuration,
so
we
have
a
component.
C
So
let's
look
at
this
flow.
We
deploy
the
Manifest.
It
goes
to
the
secondary
scheduler
because
we
injected
the
information
to
use
this
additional
scheduler
and
Intel
task
is
implemented
at
the
an
extender
right.
So
when
you
get
the
when
the
schedule
get
the
message,
you
can
see
that
there
is
the
extender
configure
on
the
cube,
scheduler
configuration
so
a
trigger
a
request,
a
filter
request
to
the
task.
C
So
the
task,
it's
always
reconciling
policy.
Those
policy
are
based
on
strategies.
You
can
Define
several
strategies,
for
example
a
schedule
strategies
that
says:
okay.
If
the
CO2
emission,
it's
a
it's
less
than
100
units
schedule
allowed
to
schedule
on
the
Node.
Otherwise,
if
it's
greater
than
100
the
schedule.
Okay,
we'll
cover
the
schedule
part
because
the
the
scheduler,
of
course
it's
not
capable
of
the
scheduling.
There
is
another
components
here
that
is
called
with
the
scheduler
and
the
way
it
works
very
quickly,
pretty
much
when
the
policy
is
in
violation
right.
C
Think
about
I,
don't
know
any
kind
of
metrics.
It
doesn't
need
to
be
CO2.
It
could
be.
You
know,
CPU
memory.
It
could
be
a
query,
an
API
for
for
the
weather
in
that
area,
I,
don't
know
you
have
a
multi-cluster
and
you
want
to
distribute
the
workloads
based
on
the
weather.
You
know
it's
in
some
area:
the
temperature
is
higher.
That
means
the
power
implants.
Are
you
know,
working
harder
to
provide
electricity
or
even
if
it's
too
colder
actually?
C
And
so
you
want
to
do
this
kind
of
things,
but
not
only
that,
since
now
the
scheduler
on
kubernetes
get
the
intelligence
to
make
decision
based
on
metrics
this
open
a
world
of
opportunities,
your
application,
your
workload
or
web
application,
or
whatever
you're
running
on
the
cluster,
could
become
aware
of.
You
know
condition
externally
to
the
cluster,
so
you
could
get
an
holistic
view
of
a
network
and
your
application
knows
what's
happening
around
and
in
the
network,
and
you
can
make
scheduled
decision
based
on
that.
C
C
C
C
So,
let's
go
again
from
the
flow
all
right,
manifest
you
deploy.
It
uses
the
secondary
scheduler
secondary,
scheduler
sender.
Filter
request
to
the
task.
Betas
is
already
reconciling
this
policy.
They
use
specific
metrics
embody
here
by
the
way.
Here
you
see
just
a
simple:
if
metric,
Grid
or
or
less
band,
you
have
much
more
flexibility,
you
can
do
if
and
or
you
can
do
combinations,
so
it
doesn't
have
to
be
only
one
metric.
You
can
combine
the
metrics
right
and
then
using
this
information
of
the
metric.
C
The
tasks
Sent
send
the
API
request
to
the
custom
metrics
API
that
we
expose
with
that
service
called
the
metrics
proxy
and
it
gets
the
information,
and
at
this
point
we
know
that
the
node,
for
example,
node
4
as
a
37.75
unit.
So
it's
not
evaluating
the
policy
and
we
will
tell
the
scheduler
that
yes,
you're
good
you
can
you
can
deploy
that
workload
on
node,
4.,
simple
right
now.
C
We
created
a
little
dashboard
just
to
demonstrate
this,
so
we
have
the
three
nodes:
the
different
power
source,
and
then
we
have
a
meter
bar
for
the
CO2
emissions
and
what
you
see
below
below
here,
it's
a
node,
234
and
both
boxes
is
for
the
pods
placement.
So
when
there
is
the
green
box,
that
means
the
Pod
has
been
placed
there.
C
C
So
it
has
no
workload,
nothing
that
runs
on
that.
But
as
soon
as
the
node
one,
where
we
have
our
workload
running,
you
get
some
violation
of
the
policies
or
the
CO2
emission
go
over
the
value
that
we
Define
at
the
point
it
gets
rescheduled
on
in
this
case
on
node
four.
That
is
not
in
violation
of
a
policy,
so
this
is
for
CO2,
but
again,
but
solution
architecture
that
you
saw
and
the
com
the
components
put
it
all
together.
A
To
do
some
finish
so
some
of
the
projects
that
were
used
here
so
Kepler,
so
that's
the
URL
is
yes.
What
is
mentioned
is
it's
a
it's,
a
really
cool
project
that
allows
us
to
measure
or,
on
her
part,
energy
on
millijoules,
there's
a
POC
that
we
also
did
with
Keira
with
the
Keta
team
Upstream
in
Microsoft
and
the
real
city
of
office
and
that
use
CO2
to
do
scaling
per
se.
So
it's
a
CO2
aware
scaling
Intel
task.
That's
the
link!
A
The
data
science
Hub.
There
is
based
on
open
data
hub.
So,
as
you
can
see,
all
of
these
are
elements
that
already
exist
from
we
just
are
using
them
in
an
integrated
fashion
and
as
the
next
evolution
of
this,
this
project
that
precisely
was
announced
by
Intel
last
week
and
and
they
have
been
showing
this
week.
Basically
that's
part
of
what
we
intend
to
to
work
on
a
future
iteration
of
this.
B
Thank
you.
It's
a
very
nice
talk.
I
just
have
questions
about,
like
are
the
policies
if
you
set
it
to
like
these
schedules,
so
you
do
some
erections
of
the
pot
right
for
the
note
that
while
later
so
do
you
have
consider
like
priorities
to
descendo
the
part
or
you
treat
every
part
the
same,
which
part
are
you
going
to
Des
schedule.
A
Yeah,
okay,
so
right
now
this
we
are
treating
every
single
part
the
same.
So
what
we
did
is
all
the
all
the
workload
where
we
were
testing
this,
because
this
is
actually
a
multi-tenant
cluster.
So
we
were
only
doing
the
calculation
based
on
a
single
mainspace
and
the
parts
there.
So
we
did
not
account
for
the
total
across
the
whole
node,
otherwise,
we'll
have
to
descale
far
more
than
our
patch
and.
C
C
A
I
can
tell
you,
for
example,
I
know
that
AWS
does
have
a
tool
for
that.
I
know
that
gcp
is
also
providing
something
and
and
Azure
outside
those
three
I
don't
know,
and
the
level
of
granularity
on
each
one
is
very
dependent
on
on
the
cloud
providing
there
are
some
that
is
really
about
what
happened
three
months
ago,
not
what's
happening
now,
and
there
are
others
that
are
far
more
accurate.
Actually,
that
lady
over
there
Nikki
she's,
far
more
familiar
with.
C
C
A
And
if
anyone
really
is
interested
on
anything
related
to
Green,
Computing
I,
don't
know
if
you're
aware
that
cncf
just
created
and
approve
the
environmental
sustainability
tag,
and
we
will
have
several
working
groups
there.
Okay,
hopefully
one
of
the
working
groups
will
be
on
the
technical
side.
Some
of
them
will
be
on.
How
do
we
fix
the
apis
once
and
for
all?
So
because
we
need
accurate
data,
and
that
is
really
really
hard
and
the
places
where
it
is
available
is
usually
behind
a
paywall.