►
From YouTube: Don't Be Greedy: Rightsize Your Kubernetes Cluster... - Jesus Angel Samitier & David Lorite Solanas
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2023 in Amsterdam, The Netherlands from April 17-21. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Don't Be Greedy: Rightsize Your Kubernetes Cluster With Prometheus - Jesus Angel Samitier & David Lorite Solanas, Sysdig
Speakers: David Lorite, Jesus Angel Samitier
Setting the proper limits and requests to a Kubernetes cluster is one of the most challenging tasks engineers have to face. Not doing this could have critical consequences, like outages or astronomical cloud bills. Sometimes, when engineers don't have the full picture of the cluster they're working on, they tend to use more resources than actually needed for their workloads. This is called the "greedy developer" problem. To avoid this, you'll have not only to identify the unused resources, containers without limits, or very tight ones. But also to set the right requests and limits. In this talk, Jesus and David will walk you through this process. You might have to add new nodes, or just resize your workloads. Since every cluster is different, also measure the optimizations made, to help our audience with their own clusters.
A
A
Yeah,
well-known
people?
I've
been
one
of
them,
though.
So
it's
no
big
deal
and
how
many
of
you
say
it
sometimes
higher
requests
just
in
case,
because
you
want
your
application
to
run
smoothly.
You
want
you
don't
want
problems,
there's
a
deadline,
you
don't
want
surprises,
okay,
so
we
might
have
some
greedy
greedy
developers
here.
A
My
name
is
Jesus
and
my
name
is
David.
We
work
at
systic,
we
help
our
customers
understand
their
kubernetes
clusters
and
we
are
also
the
maintainers
of
promcad.io,
which
is
an
open
source
project
that
gives
you
a
curated
lists
of
Prometheus
exporters
that
are
just
ready
to
go.
You
can
go
there,
search
for
the
technology
that
you
want
to
Monitor
and
it's
very
straightforward.
A
It
helps
you,
configuring,
your
primitive
jobs,
your
exporter
Etc,
so
there
is
a
problem
with
the
real
developers
right
and
what
what
happens
here
is
that
knowing
the
exact
amount
of
resources
that
your
application
needs
to
use
is
a
really
complicated
matter
right,
so
we
don't
normally
know
how
to
do
it.
So
in
this
talk
we
are
going
to
to
show
you
guys
how
we
right
size.
The
request
of
our
clusters
for
a
lot
of
benefits
that
we'll
discuss
later
so
first
normally
agree.
Developer,
will
set
higher
limits
and
higher
requests.
A
So
what
happens
when
you
set
higher
limits?
Well,
first,
if
you
set
lower
limits,
you
might
kill
your
application.
Your
CPU
might
throttle
you,
you
might
have
some
boss
kills
by
the
out
of
memory,
but
on
the
other
hand,
if
you,
if
you
set
two
high
limits,
you
might
be
a
starving
other
applications
in
the
cluster.
If
the
usage
are
Rises
right,
something
really
similar
happens
with
requests.
If
you
set
low
requests,
is
the
same
right.
A
Cpu
throttling
memory
kills,
but
if
you
set
higher
requests
at
first,
your
obligation
runs
smoothly,
but
you
might
be
doing
core
harder
for
kubernetes
to
allocate
more
pots
and
also
you're
wasting
a
lot
of
resources.
You
are
buying
for
more
CPU
and
memory
that
you
won't
use.
So
you
are
you,
you
might
be
wasting
some
money
right.
A
B
The
limits
are
the
amount
of
the
maximum
amount
of
resources
that
your
application
can
use,
but
this
resources
are
not
guaranteed.
This
resources
depends
on
the
available
resource
that
the
node
has
and
what
are
the
requests?
The
requests
are
the
minimum
resources
that
your
application
have
Reserves,
so
how
it
starts
in
a
really
scenario.
A
Yeah
well
for
this
presentation,
we
are
going
to
right
size,
a
real
application,
well,
not
real
as
in
real,
but
this
application
existed
in
an
actual
cluster,
so
we
we
have
set
one
gigabyte
and
one
on
one
course
CPU
for
the
limits
and
the
same
for
the
requests,
and
we
are
going
to
start
from
here.
Okay,
of
course,
right
now
the
application
runs
perfectly.
A
B
Need
to
explain
what
these
are
qos
classes.
Quality
of
service
depends
of
the
memory
or
request
that
you
set
in
your
workloads.
If
you
set
the
same
amount
of
requests
and
limits,
you
will
have
a
qis
of
guarantee.
That
means
that
your
posts
will
be
the
last
of
all
parts
addicted.
If
you
set
the
limits
on
the
request,
but
the
limits
are
not
the
same,
and
then
you
will
have
a
qos
of
a
boost
cover.
B
A
Okay,
so
let's
put
it
an
example
here:
if
you
have
one
port,
your
application
runs
in
one
port,
for
example,
a
database
it
does
is
just
one
stateful
set.
You
don't
want
by
any
means
that
that
bot
gets
evicted
right.
You
don't
want
your
database
to
be
down.
You
just
have
one
port,
so
the
strategy
here
is
obvious.
You
need
to
set
the
same
limits
on
request.
So
if
governance
have
has
problems,
your
database
will
be
the
last
part
that
that
the
cluster
will
evict.
A
But
sometimes
you
need
to
optimize
your
limits
and
requests,
and
maybe
maybe
there
are
applications
that
you
can
take
some
risk,
for
example,
an
API
Gateway.
If
you
have
an
API
Gateway,
you
have
say
10
ports,
10
engine
exports
and
if
two
parts
are
evicted,
it's
not
a
big
deal
we'll
talk
more
about
this
later,
but
you
can
take
the
risk
because
the
benefits
are
most
efficient,
more
optimized
limits
and
requests.
Okay.
So
you
could
go
with
the
personal
strategy
here
and
the
best
effort.
A
B
It's
not
a
big
deal.
Actually,
if
your
application
is
running
in
multiple
paths
or
you
don't
have
just
a
single
boat,
it's
it's
normal
I
I
mean
kubernetes
works
like
that.
It's
the
natural
thing
in
kubernetes
and
after
any
things
you
do
in
the
in
the
request
or
in
the
limits
of
our
Warlock.
The
scheduler
will
try
to
arrange
your
ports,
so
it
will
be
a
bit
somber.
So
don't
worry
about
where
your
pot
is.
A
B
It
for
limit
right
size.
We
can
use
two
extra
things
phase.
One
is
the
conservative.
That
means
that
we
trust
more
in
the
request
than
the
limits,
so
in
this
case,
for
example,
a
database
that
has
several
Bots.
We
want
to
assure
that
we
are
using
the
request
more
than
the
limits,
and
that
way
we
will
have
enough
room
to
to
move
it
and,
in
the
other
hand,
on
in
that,
in
that
one
we
could
use
like
a
25
percent
more
than
the
request.
B
A
So,
let's
get
started
with
the
right
size
strategies,
but
first,
why
are
we
doing
this?
What
are
the
key
benefits
that
that
have
right
size
in
your
request?
What
first
of
all
you
you'll
have
a
better
understanding
of
your
application?
You'll
understand
how
your
application
performs
and
what
what
your
application
can
do
in
the
cluster.
A
Also,
you
might
discover
some
issues
that
your
obligation
have
related
to
performance
that
you
didn't
know,
because
your
application
was
running
with
a
lot
of
resources
available.
So
maybe
your
application
have
some
peaks
of
users
because
of
a
misconfiguration
or
a
software
problem,
and
now
that
you
have
limited
the
request,
you'll
see
it
so
you
can
fix
it.
A
Also
you'll
make
the
most
of
the
resources,
because
maybe
you're
sharing
your
cluster
with
another
team,
and
maybe
your
if
you
start
resizing
your
workloads,
the
team,
the
other
team,
will
have
more
room
for
their
application
and,
of
course,
you'll
save
money.
That's
that's
obvious,
because
you'll
be
buying
less
resources
for
your
applications,
so
we
have
a
plan
here.
A
A
Then
you
need
to
know
what
are
the
the
requests
that
you
have
to
set?
You
have
to
discover
this
then
we'll
do
this
results
right,
we
will
perform
the
size,
but
then
you
need
to
monitor
it
again
because
right
sizing,
your
request,
isn't
something
that
you
do
once
this
is.
This
is
something
you
you
have
to
do
quite
regularly,
because
your
application
is
a
living
thing.
So
maybe
today
is
one
number,
but
tomorrow
or
the
next
month
is
another
number
because
usage
can
vary.
So
please
monitor
your
applications.
A
We
can
tweak
it
and
then
we
back
to
four
for
this
monitoring.
So
so
we
need
to
monitor
what
what
are
we
going
to
use
here?
Of
course,
Prometheus
we
love
Prometheus
Prometheus
is
de
facto
standard
for
monitoring
kubernetes.
We
all
use
Prometheus
I
guess.
So
what
information
are
we
going
to
retrieve
from
Prometheus?
First,
we
need
to
know
what
is
happening
inside
the
containers.
We
need
to
know
how
many
resources
are
your
containers
using
right
now
and
for
that
we
are
going
to
use
the
C
advisor
exporter,
but
also
we
need
to
do.
A
B
B
A
Okay,
so
what
if,
instead
of
using,
if
instead
of
aggregating
by
by
workload,
namespace
and
cluster,
what
if
we
upgrade
this
just
by
namespace,
then
we
might
are,
we
are,
might
creating
the
cube
CTL
blame
command,
because
this
is
like
a
picture
of
what
are
the
themes
or
the
projects
that
are
wasting
more
resources.
I'm
just
kidding,
don't
do
this
because
you
don't
want
to
fight
anyone?
Okay,
so
we
monitored
we.
We
have
a
lot
of
information.
How
do
we
use
this
information
now?
The
next
step
was
to
calculate
the
the
request.
A
So
there
are
two
ways
of
doing
this.
The
first
one
is
the
conservative
way,
the
conservative
strategy
this
is
using
the
max.
This
is
calculating
the
max
number
of
resources
that
your
application
is
using
and
that's
your
request.
These
have
some
benefits
into
drawbacks.
The
benefits
is
that
you'll
be
full
of
room
for
your
application.
A
The
drawbacks
is,
you
might
be
wasting
some
money
anyway,
because
what
if
your
application
have
peaks
of
usage,
if
you're,
if
you
set
the
maximum
their
request
as
the
max
there
will
be
some
some
picks
that
that
you'll
be
paying
for.
But
you
won't
be
using
that
when
your
application
isn't
in
one
of
the
of
the
speak,
but
this
could
be
an
a
nicer
starting
point
for
a
start
to
tweaking,
because
there
is
not
a
formula:
did
you
there's
some
some
trying
here?
Well,
there
is
another
strategy,
maybe
a
more
aggressive
one
yeah.
B
You
can
use
some
more
an
aggressive
strategy
that
is
calculating
the
average
of
the
consumption
of
your
application,
and
that
way
you
application
might
use
some
pics
of
memory,
but
that's
okay.
The
limits
are
there,
so
the
limits
absorb
that
Peak
and
if
you
need
to
do
guitar,
so
you
have
to
increase.
That's
okay,
because
it's
a
living
things
as
you
say,
so
we
can
increase
it.
Okay,
we
need
it.
So
so,
let's
recite
the
application
that
we
saw
before
in
the
previous
slide.
B
B
In
that
panel
we
can
see
the
unused
memory
of
your
application.
What
is
the
amount
of
unused
memory?
We
can
see
that
now
we
are
requesting
less
resources
and
in
that
one
we
can
see
the
percentage
of
their
resources,
the
memory
versus
the
request.
So
now
we
have
increased
the
percentage
of
the
of
the
memory
that
we
are
using.
So
we
are
fine,
but
we
are
not
in
in
our.
A
Standpoint,
it
was
a
good
starting
point
right
now
we
are
wasting
less
resources,
which
is
nice,
but
we
have
some
room
for
for
improvement
here.
Let's
we
monitor
it
again
and
we
we
could
tweak
it
again.
So
what?
If
we
help
it
again,
we
go
from
512
to
256.
what
happens?
Let's
check
the
impact
so
in
this
screenshot
in
the
in
this
chart,
we
we
went
even
further.
A
Now
we
that
we
said
128,
we
are
using
a
bow,
a
bow
100,
because
we
have
the
limits
there
so,
and
this
is
where
we
want
to
be.
We
want
to
use
all
the
resources
that
we
are
reserving,
because
that
way
we
won't
be
wasting
any
resources
or
any
money.
This
is
the
same
image,
but
the
other
way
around
this
is
the
quantity
of
resources
that
were
wasted.
We
went
from
a
lot
of
percent
to
almost
zero
or
even
zero.
B
B
A
A
We
also
saved
75
dollars
a
month
for
yes,
one
wordload,
which
is
great,
but
the
most
important
thing
from
my
perspective
is
that
we
are
starting
a
workflow
that
is
easily
repeatable
to
monitor
the
resource
usage.
We
created
some
dashboards
depending
on
its
use
case
with
those
queries
with
those
metrics.
A
B
Yeah,
that's
why
we
do
our
navigates
for
over
time.
So
you
do,
you
don't
have
to
take
an
even
if
you
use
a
lot
of
resources.
Affairs,
that's
why
the
limits
are
so
the
limits
will
be
absorbed
at
amount.
So
you
have
to
assure
the
limits.
In
that
case
the
problem
is
the
limits,
so
we
have
to
put
the
limits
there.
You
know
what
are
the
big.
So
then
you
have
to
request
about
that.
So
I
do
know.
B
D
D
A
Yeah,
your
salary
is
higher
yeah.
That's
a
good
question.
Sorry,
the
question
is
what
what
if
we
are,
you
are
saving
a
lot
of
money,
but
you
are
spending
a
lot
of
time
on
our
efforts
right
right,
sizing,
yeah,
that
this
is
a
good
question.
As
we
said
we
worked
in
this
dashboard
once
and
then
we
are
using
it
for
all
the
attachments
for
the
workloads.
So
yes,
at
first
it's
some
you,
you
need
to
be
some
time,
writing
queries
and
knowing
how
your
applications
work.
A
A
B
B
A
Are
for
that
yeah,
so
you
using
100
it's
a
good
idea,
because
you
are
taking
the
best
of
your
requests
and
if
there
is
something
that
makes
your
application
go
even
further,
you
have
the
limits
to
cover
that
to
absorb
that,
and
of
course,
if
that
go
even
further
and
you
have
memory
issues
as
you
are
monitoring
this
you'll
know
this
and
you
you'll
be
able
to
fix
it.
So,
yes,
that's
a
good
question,
because
at
first
I
was
thinking
the
same,
so
I
100.
G
Hi,
actually,
my
question
is
related
to
the
same
question.
He
asked
so
I
got
something
like
a
five
six
different
environments
and
each
environment
I
got
like
probably
more
than
100
applications
running.
If
someone
is
in
SRE,
if
he's
sitting
around
and
thinking
that
okay
I've
said
quite
a
lot
of
money
by
tweaking
my
apps
and
I
go
to
bed
and
then
my
ports
get
evicted,
so
I
have
to
make
a
decisions
like
do
I
make
my
life
easy
or
do
I
save
money
or
just
vice
versa.
G
G
Then
you
constantly
have
to
keep
and
watch
on
where
the
metrics,
where
the
load
is
flowing
through
it
might
become
a
bit
quite
a
cumbersome
job
to
constantly
have
to
tweak
your
apps
and
then,
if
you
have
hundreds
more
than
100
applications
and
then
each
has
got
and
I've
got.
Four
or
five
environments,
where
do
I
go.
B
A
G
Not
actually
a
question,
it's
like
it's
a
little
bit
difficult
to
actually
it's
good
to
start
with
this
kind
of
right
sizing,
but
if
you
have
say
100
workloads,
100
applications
and
I
got
four
to
five
different
environments.
That
means
four
times
hundred
four
hundred.
How
can
I
maintain
this
thing?
It
starts
off
from
the
beginning
and
over
a
period.
A
A
B
A
Continue
with
that
yeah
it
depends
it's
it's
the
the
the
other.
This
other
person
asks
something
like.
Is
it
worth
it?
Well,
you
can
start
with
what
what's
the
name
space
that
is
wasting
more
resources,
you
can
start
there
and
you
don't
you
don't
need
to
do
it
at
in
one
night.
So,
yes,
it
depends
on
on
any
on
each
scenario.
G
H
H
B
B
If
you,
if
you
have
an
idealist
scenario,
where
all
applications
can
down
can
be
down
or
can
be
evicted,
we
found
any
problem
yeah
you
can
put
No
Limits,
so
every
application
that
needs
that
CPU
will
really
will
receive
the
the
CPU
that
is
for
CPU,
mainly
not
for
memory
for
memory.
It's
not
not
spelled,
but
for
CPU
there,
sometimes
it's
it's,
but
it
depends
of
all
teams
or
of
your
class,
because
sometimes
you
have
a
lot
of
teams
that
don't
tell
each
other
yeah.
You.
A
B
I
I
A
B
A
You
do
yeah
if,
if
there's
a
problem,
if
you
go
above
your
100
of
your
of
your
limit
of
the
request-
and
you
have
problems
with
that-
you
should
set
the
limit
yeah,
because
you
might
need
to
tweak
the
limit
a
bit.
If
you
need
a
guaranteed
State,
you
might
want
to
up
to
increase
both
limit
and
request,
but
but
it
depends
on
any
scenario.
Sorry
because
there's
no
formula
for
this,
but
but
if
you
monitor
this
you,
you
could
have
some
some
answers.
J
A
The
question
he
asked
is:
is
there
any
automated
way
of
doing
it
of
doing
this,
instead
of
being
manually
tweaking
the
queries
and
and
the
limits
on
the
requests
like?
This
is
a
very
tough
question
because
you
need
this
is
a
very
Craftman
operation,
because
it's
a
scenario
it's
application,
even
even
two
two
teams
could
be
using
nginx
and
yeah
limits
and
requests
of
each
team
could
be
completely
different.
Maybe.
A
Missing
so
I
I'd
go
for,
for
example,
if
you
go
to
the
systic
booth
in
in
in
the
in
the
here
in
the
salon,
you
can
you
can
see
how
CSD,
for
example,
do
this
we
we
have
this
thing
called
okay,
I'll,
kubernetes
dashboards,
that
that
gives
you
this
information
and
can
help
you
finding
what,
without
you
doing
a
thing
finding.
What's
the
right
request
with
the
right
limit,
but
like
I,
don't
think,
there's
an
algorithm
that
you
could
deploy
in
your
cluster
and
I.
K
A
A
K
A
L
L
A
Yeah
you,
you
could
use,
for
example,
Kira.
No,
sorry!
No,
because
that's
that
would
be
with
the
with
the
BPA
yeah
we
could
yeah.
We
could
use
the
current
vpa
for
right
sizing,
the
limits
on
the
requests,
but
that,
but
that
wouldn't
be
very
like
did
that,
wouldn't
be
the
most
efficient
way
of
doing
this
because
you'll
have
some.