►
From YouTube: Getting the Optimal Service Efficiency That Autoscalers Won’t Give You - Mauro Pessina, Moviri
Description
Don’t miss out! Join us at our upcoming hybrid event: KubeCon + CloudNativeCon North America 2022 from October 24-28 in Detroit (and online!). Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Getting the Optimal Service Efficiency That Autoscalers Won’t Give You - Mauro Pessina, Moviri
A challenge when tuning a Kubernetes microservices application is identifying the container size (CPU and Memory), due to frequent application changes and varying traffic patterns. Kubernetes autoscalers are the standard solution to automatically adjust Kubernetes container resources for service efficiency. We present the results of an extensive tuning activity we successfully conducted on a Kubernetes application delivering business-critical financial services to SMB customers. Our goal was to minimize cloud cost without compromising on performance of this application. The unexpected result was that configurations minimizing the service cost were not recommended by the autoscaler. Indeed, autoscalers work by adjusting resource sizing wrt the historical usage, without being aware of the actual cost of cloud resources and of the impact on application performance. In our session, we illustrate how you can use our exploratory testing approach we leveraged to get these results.
A
A
We
will
then
introduce
the
new
approach
we
implemented
in
moviri,
which
leverages
ai
to
automate
the
optimization
process
and
we'll
do
by
they
do
that
by
providing
real
world
examples.
Finally,
we
will
conclude
with
the
some
take
away
the
quiz
before
proceeding.
Please
allow
me
to
introduce
myself.
My
name
is
mauro
and
I'm.
The
head
of
performance
engineering
services
align
in
mobility
and
performance
engineering
has
always
been
my
passion,
since
I
graduated.
A
A
A
Speaking
of
kubernetes
itself,
there
are
more
and
more
stories
about
how
difficult
it
is
to
ensure
performance
and
stability
for
kubernetes
application,
as
you
may
have
seen
also
on
speeches
before
mine.
The
kubernetes
fellow
stories,
for
example,
is
a
website
that
was
specifically
created
to
check
kubernetes
incident
reports
with
the
goal
to
learn
how
to
prevent
them.
A
We
will
go
through
the
six
main
key
facts
and
their
implication
starting
from
the
first,
which
is
resource
request.
You
may
have
heard
many
talks
before
mine
talking
about
this
topic,
but
let's
make
a
quick
recap:
when
developers
define
a
pod,
they
have
the
possibilities
to
specify
also
resource
requests.
Those
are
the
amount
of
cpu
and
memory
which
are
granted
to
the
pod
or
orbit
or
better
a
container
within
the
pod
and
kubernetes
will
schedule
the
pod
on
a
node
where
the
request
requested
resources
are
actually
available.
A
In
the
example,
you
can
see
the
pod
day,
which
requires
two
cpus
and
is
scheduled
on
a
four
cpu
node
when
a
new
part
b
of
the
same
size
is
created,
it
can
be
also
scheduled
on
the
same
node.
This
is
not
this
node
now
has
all
of
the
four
cpus
requested.
If
a
pod
c
is
created,
kubernetes
won't
schedule
it
on
this
node
and,
as
its
capacity
is
full.
A
This
means
that
those
numbers
a
developers
puts
in
a
yaml
file
are
accurately
used
by
kubernetes
to
manage
a
real
cluster
capacity.
A
quite
surprising
fact
is
that
kubernetes
there
is
no
other
commitment
on
request.
You
cannot
request
more
cpus
than
those
available
in
the
cluster.
This
is
very
different
from,
for
example,
virtualization,
where
you
could
create
vms
with
many
more
cpus
than
the
real
physical
ones,
and
another
very
important
fact
to
notice
is
that
the
resource
requests
are
not
equal
to
utilization.
A
The
second
important
concept
is
resource
limits.
Resource
requests
are
the
guaranteed
resources;
a
container
will
get,
but
the
usage
can
be
higher.
Resource
limits
is
the
mechanism
that
allows
you
to
define
the
maximum
amount
of
resources.
A
container
can
use
like
two
cpus
or
one
gigabyte
of
memory.
A
All
this
is
great,
but
what
happens
when
resource
usage
is
the
limit
kubernetes
tweets
in
cpu
and
memory
differently
here?
So
when
cpu
usage
approaches
the
limit,
the
container
gets
throttled.
This
means
that
the
cpu
is
artificially
restricted
and
this
can
result
in
application
performance
issues.
Instead,
the
web,
when
memory
usage
is
the
limit.
The
container
could
get
terminated,
so
there
is
no
applica,
no
no
applications
slow
down
due
to
paging
or
swapping
or
so
on,
but
with
kubernetes
your
body
will
simply
disappear
and
you
may
face
serious
application
stability
issues.
A
The
third
factor
is
about
an
important
and
less
known
effect.
Cpu
limits
have
on
application,
performance,
cpu
limits
work
by
throttling
cpu
performance,
and
you
may
think
that
this
happens
only
when
cpu
usage
hits
the
limit.
But
surprisingly
the
reality
is
that
the
cpu
throttling
starts
when
the
cpu
usage
is
well
below
the
limit.
We
did
quite
a
bit
of
research
ourselves
and
we
found
out
that
the
cpu
throttling
can
start
when
cpu
usage
is
as
low
as
30
percent
of
the
cpu
limit.
A
A
A
Well,
it
depends.
The
answer
could
be
no,
because
cpu
limits
is
exist
for
the
purpose
of
ensuring
that
application
run
fine
and
consists
with
other
applications.
So
if
cpu
limits
are
removed,
a
single
runway
application
can
completely
disrupt
the
performance
and
expected
availability
of
your
most
critical
services.
A
A
A
A
What
is
the
lesson
learned
here?
Vpa
autoscaler
is
based
on
resource
usage
and
does
not
consider
application
level
metrics
like
response
time.
For
example,
we
need
to
evaluate
the
the
effects
on
of
the
recommender
settings
as
be
as
they
might
be
somewhat
aggressive
and
cause
service,
severe
service
performance
or
reliability
degradation,
but
we
do
also
have
hpa
the
horizontal
pad
auto
scaler.
A
What
about
it?
In
this
example,
we
wish
a
situation
where
hbr
hpa
scales
out
when
specific
memory
or
cpu
consumptions
are
met
as
a
result
from
one
pod,
we
get
two
identical
pods
that
provide
the
same
service.
This
is
great,
so
we
have
two
times
the
resources
we
had
before
to
complete
the
service.
What
could
possibly
go
wrong?
A
Well,
things
may
go
wrong
because
it's
if,
for
some
reason,
you
didn't
properly
set
up
your
memory
limit,
for
example,
you
undersized
it
and
you
will
simply
get
into
a
situation
where
you
get
twice.
The
out
of
memory
kills
you
had
before,
but
also
on
the
other
hand,
you
may
face
a
situation
where
you
set
your
cpu
limits
to
height,
but
your
pod
can
find
such
resources
directly
impacting
your
service
latency
because
of
throttling.
A
A
A
A
new
approach
is
required
to
successfully
solve
this
problem,
and
this
is
where
ai
can
help.
Air
have
revolutionized
the
entire
industries,
and
the
good
news
is
that
they
can
be
used.
Also
in
the
performance
tuning
process,
ai
can
automate
the
tuning
of
the
many
parameters
we
have
in
the
software
stack
with
the
goal
to
optimize
application,
performance,
resiliency
and
cost.
A
What
the
results
were
almost
impossible
to
achieve
with
the
tuning
practice
they
had
in
place
since
they,
it
was
manual,
and
it
took
almost
two
months
to
tune
one
single
macro
service
with
mixed
results,
obviously,
because
it
was
always
too
difficult
to
to
find
the
right
trade-off
between
performance,
resiliency
and
obviously
cost
by
design.
The
approach
would
was
the
good
old
over
provisioning,
but,
as
a
result,
what
did
they
have?
The
result
was
a
huge
of
a
spending
and
a
lack
of
performance,
operational
efficiency
and
business
agility.
A
So
how
did
we
address
this
customer?
I
would
like
to
introduce
quickly
our
ai
power
and
optimization
methodology.
In
practice,
the
process
is
fully
automated
and
it
works
in
five
steps.
The
first
step
first
step
is
to
apply
the
new
configuration
ai
suggested
to
our
target
system.
This
is
typically
done,
leveraging
kubernetes
apis
and
to
set
the
new
value
for
the
parameters.
For
example,
the
cpu
of
the
request
of
the
container.
A
The
second
step
is
to
apply
a
workload
to
the
target
system
in
order
to
assess
the
performance
of
the
new
configuration.
This
is
typically
done
by
leveraging
performance
testing
tools.
In
this
case,
we
we
use
the
geometer
test
that
was
previously
built
to
stress
the
application
with
a
realistic
workload,
we'll
see
it
later,
and
the
third
step
is
to
collect
kpis
related
to
the
target
system.
The
typical
approach
here
is
is
to
leverage
observability
platforms
and
in
this
case
we
integrated
elastic
apm,
which
is
the
monitoring
solution
used
by
the
customer.
A
A
So
I
I've
said
that
we
have
to
to
have
a
goal.
We
can
define
a
goal
in
this
scenario.
The
goal
was
to
reduce
the
cloud,
the
cost
required
to
run
the
authentication
service
on
kubernetes.
As
you
can
see,
the
optimization
glo
is
declarative.
Okay.
At
the
same
time,
you
also
wanted
to
ensure
that
the
service
would
always
meet
its
reliability
targets
which
are
expressed
as
latency
throughput
and
error
rate
slos.
A
So
how
can
we
leverage
ai
to
achieve
this
high
level
business
goal
in
the
ai
powered
optimization
methodology?
The
ai
will
change
the
parameters
of
the
system
to
improve
a
metric,
a
metric
that
we
have
defined.
In
this
case,
the
ai
goal
is
simply
to
minimize
the
application
cost.
This
is
a
metric
that
represents
the
cloud
cost.
We
will
pay
to
run
the
application
on
the
cloud
which
depends
on
the
value
of
cpu
and
memory
sources
that
are
allocated
to
the
microservices.
A
The
the
methodology
that
we
used
also
allows
to
set
constraints
to
define
which
configuration
are
acceptable.
In
this
case,
we
state
that
the
system,
throughput
response
time
and
error
rate
should
not
degrade
more
than
10
than
the
baseline,
so
to
assess
the
performance
and
cost
efficiency
of
a
new
configuration
suggested
by
the
ai
optimizer.
We
stress
test
the
siemens,
the
system
with
a
load
test.
A
A
Before
looking
at
the
results,
it's
worth
commenting
on
how
the
application
was
initially
configured
by
the
customer,
we
called
that
the
baseline
configuration
let's
look
at
the
kubernetes
setting.
First,
this
container
powering
the
application
was
configured
with
the
resources
of
resource
requests,
sorry
of
a
1.5,
cpus
and
3.4
gigabytes
of
memory,
and
the
team
also
specified
their
source
limits
of
two
cpus
and
4.4
gigabytes
of
memory.
A
Remember
the
requests
are
the
guaranteed
resources
that
kubernetes
will
use
for
scheduling
the
capacity
management
to
the
cluster.
In
this
case,
requests
are
lower
than
the
limit,
which
is
a
common
approach
to
guarantee
resources
for
the
application
to
run
properly,
but
at
the
same
time
allow
some
room
for
unexpected
growth.
A
Besides
looking
at
the
container
settings,
it's
important
to
also
see
how
the
application
runtime
is
configured.
The
runtime
is
what
ultimately
powers
our
application
and
for
java
application.
We
know
that
the
jvm
settings
might
play
a
big
role
in
application
performance,
but
the
same
and
happens
for
other
languages,
so,
for
example,
golang
and
so
on.
A
A
Okay,
we
have
covered
how
the
applications
is
configured.
Okay,
let's
now
look
at
the
behavior
of
the
application
when
subject
to
the
load
test
that
we
have
shown
before
with
the
baseline
configuration
in
this
chart,
you
can
see
the
application
throughput
the
response
time
and
the
number
of
replicas
that
were
created
by
the
autoscaler
two
facts
are
important
to
notice.
A
First,
one
when
load
increases,
the
autoscaler
triggers
a
scale
out
event
which
creates
a
new
replica.
This
event
causes
a
big
spike
on
response
time,
which
impacts
service,
reliability
and
performance.
This
is
due
to
the
height
cpu
usage
and
cpu
throttling.
Obviously,
during
the
jvm
startup
phase,
when
the
load
drops,
the
number
of
replicas
that
does
not
scale
down
despite
the
container
cpu
usage
may
be
idle,
as
in
this
case,
it's
interesting
to
understand
that
why
this
is
happening.
A
A
A
First
of
all,
it's
interesting
to
notice
how
our
optimization
increased
the
both
memory
and
cpu
request
limits,
which
is
not
all
obvious,
especially
for
kubernetes,
because
kubernetes
is
often
considered
well
suited
for
a
small
and
highly
scalable
applications.
We
will
deep
dive
on
this
into
a
minute.
The
other
notable
changes
are
related
to
the
jvm
options.
Ai
peaked,
the
maxip
was
increased
by
20
percent
and
is
now
well
within
the
container
memory
request,
which
was
increased
to
five
gigabytes.
A
So
let's
now
see
how
the
application
performs
with
the
new
configuration
identified
and
how
it
compares
to
baseline.
There
are
two
two
important
differences
here:
autoscaling
is
not
triggered
in
this
configuration,
as
the
full
load
is
sustained
by
one
pod.
This
is
clearly
beneficial
in
terms
of
cost
response.
Time
always
remains
within
the
response
time
slos
and
there
are
no
more
peaks,
so
this
configuration
not
only
improves
on
cost,
but
also
is
beneficial
in
terms
of
performance
and
resilience.
A
A
This
configuration
has
the
effects
as
the
effect
of
triggering
the
auto
scaler
less
often
okay,
as
we
have
seen
but
interestingly
well,
this
implies
a
kind
of
fixed
cost.
Considering
the
pricing
of
the
container
resources,
it
turns
out
being
much
cheaper
than
a
configuration
where
auto
scaling
is
triggered,
and
these
also
avoid
performance
issues.
A
A
Let's
now
have
a
look
at
another
configuration
found
by
ai
at
the
experiment
number
14
at
after
about
eight
hours.
We
level
we
labeled.
This
configuration
high
resilience
for
a
reason
that
will
be
clear
in
a
minute.
The
score
of
this
configuration,
while
not
as
good
as
the
best
configuration,
also
provides
about
16
percent
of
cost
reduction.
A
So
this
can
be
considered
also
an
interest
configuration
with
respect
to
the
cost
efficiency
goal
as
regarded
parameters.
What
is
worth
noticing
is
that
this
time,
ai
picked
different
settings
that
significantly
change
the
shape
of
the
container
it
is.
It
is
now
has
a
much
smaller
cpu
request
than
the
baseline,
but
the
memory
is
still
pretty
large,
which
is
interesting.
A
A
But
let's
now
compare
the
behavior
of
this
configuration
with
respect
to
the
baseline
there.
There
are
two
important
differences
here.
The
peak
on
the
response
time
upon
the
scaling
out
is
significantly
lower.
It
is
still
higher
than
the
response
time
slows.
However,
the
peak
is
less
than
half
the
value
for
the
baseline
configuration.
A
A
A
In
terms
of
customer
result,
what
did
we
get?
The
red
sizing
of
the
service
spots
allowed
a
huge
cost
reduction
in
the
kubernetes
environment,
but
where's
more.
We
allowed
them
to
automate
the
tuning
of
their
services
or
services
in
a
matter
of
hours
instead
of
months,
enabling
not
only
the
cost
reduction
but
also
enabling
an
improvement
in
the
latency
of
our
application
and
overall,
a
better
user
experience.
Experience
for
the
customer.
A
C
A
Yeah,
so
we
could
talk
our
about
the
the
approach
we
had
I'll
just
share
some
insights
so
base.
Basically
that
is
based
on
referencement
learning
and
we
provide
as
input
the
parameters
that
we
apply,
so
the
value
of
the
parameters
that
we
apply
and
the
score
so
the
score
of
the
last
iteration
that
we
had
then
after
analyzing
the
such
score
of
the
iteration,
the
parameters,
the
algorithms,
the
algorithms
that
we've
used,
learns
how
it
has
to
behave
and
propose
the
new
iteration.
D
A
A
So
I
got
it
so
in
this
particular
case.
What
we
did
is
that
was
focused
on
on
the
pod
sizing,
not
on
the
let's
say
on
the
node
sizing,
or
we
didn't
think
about
auto
scaling
of
the
nodes.
But
what
we
usually
see
is
that
the
experimental
approach
allows
us
to
monitor
how
the
system
behaves
in
terms
of
node
capacity.
While
we
run
so,
we
are
able
to,
let's
say,
identify
if
we
have
a
shortage
in
the
cpu
and
memory
utilization
on
specific
nodes
and
doing
this
experimental
approach.
A
A
F
E
E
A
A
Basically,
it
may
take
longer
to
find
a
suitable
situation,
but
we
can
test
multiple
services
together
also,
I
can
add,
in
this
specific
case,
constraints
related
to
the
sizing
of
the
node.
So
if
I
want
let's
say
to
be
safe
in
terms
of
not
allocation,
I
can
add
more
constraints
in
terms
of
okay.
You
don't
use
much
more
of
let's
say
the
30
percent
of
the
capacity
on
the
load,
but
I
can,
I
can
add
it
as
a
constraint
in
order
to
let's
say,
define
a
limit
in
the
space
that
you
have
to
search
against.
B
A
Yeah
yeah,
the
idea
is
to
predict
the
possible
configuration
that
could
lead
to
an
improvement
okay,
so
that
is
an
iterative
and
predictive
approach.
If,
if
you
have
followed
also
the
the
talk
before
mine,
they
were
talking
about
predictive
approach.
This
is
the
case
I
mean
this
is
a
predictive
and
iterative
approach.
Okay,.
G
Hey
in
one
of
your
slides,
you
said
that
you
found
that
if
you
disabled
the
cpu
limits
the
response
times
actually
reduced
yeah.
As
far
as
I
remember,
there
was
a
linux
linux
kernel
back
in
2020.
If
I'm
not
mistaken,
and
there
was
already
a
bug
fix
about
that
and
afterwards,
presumably
setting
the
limits
again
would
actually
eliminate
the
bug
and
fix
the
bug.
Actually,
so
my
question
would
be:
do
you
still
think
that
turning
off
the
limits
is
the
way
to
go
or.
A
Okay,
in
my
opinion,
I
mean-
I
know
that
in
some
cases
many
kernels
fix
that
that
issue
on
the
throttling.
The
idea
is
that
this
argument
of
removing
the
request
and
the
resource
limits
is
a
some
kind
of
it's
by
the
way.
It's
something
like
a
a
war
between
the
people
who
wants
to
remove
the
meets
and
the
people
who
says.
Oh,
no,
you
have
to
keep
that
limit.
The
nice
part
of
what
we
do
is
that
we
don't
need
to
take,
let's
say
a
position,
since
our
approach
is
experimental.
A
F
Hi,
I
have
a
question
on
the
problem
of
overfitting
to
the
data,
so
you
know
in
machine
learning.
You
typically
have
this
problem
that
you
can
you
know
you
can
your
classifier
or
whatever
can
you
know,
become
too
sensitive
to
the
training
data
and
not
perform
well
on
the
real
data?
So
I
see
you're
using
we
are
using
performance
tests,
so
some
artificial
tests
right
to
to
set
the
parameters
so
like
how
does
it
behave
well
in
production?
F
A
Yeah,
basically
I
mean
I,
I
come
from
a
long
practice
of
performance
testing,
so
performance
testing
is
what
I
do
for
a
living
since
started
working
and
basically
the
requirement
to
to
this
approach
is
to
find
a
way
to
shape
a
proper
mod
to
shape
a
proper
model
of
of
the
of
the
let's
say
of
the
performance
test.
So
you
do
have
obviously
to
use
the
same
version
of
application.
The
same
version
of
the
operative
system,
comparable,
let's
say
sizing
in
the
production
environment,
where
I
test
the
configuration.
A
So
that
is
everything
that
you
have
to
take
into
account
and
for
sure
that's.
The
problem
of
of
how
significant
significative
is.
That
is
your
performance
test.
So
our
this
approach
is
well
suited
for
a
customer
for
for
people
who
are
very
into
performance
engineering
and
know
how
to
shape
your
traffic
well.
But
we
are
also
investigating
the
field
of
doing
it.
Let's
say
in
production
with
the
real
data.
H
A
So
basically,
what
we
did
with
the
with
the
jvm
topic
is
that
in
this
particular
case,
because
it's
not
let's
say
a
one
fit
for
all
solution.
Okay,
in
this
particular
case
case,
obviously
the
best
configuration
was
the
one
we
didn't
trigger
the
replica.
But
if
you
see
into
the
other
configuration
which
led
to
a
some
kind
of
15
reduction
in
cost
and
that
configuration,
we
managed
to
find
a
sizing
of
both
the
jvm
in
terms
of
mini
pen,
money
and
maxip,
which
were
increased
where
the
the
gvm
wasn't
suffering.
A
Let's
say
in
terms
of
throttling
at
the
startup
of
the
jvm,
obviously
ijvm
always
suffers
in
terms
of
cpu
and
starts,
but
that
specific
configuration
in
terms
of
minip,
maxip
and
memory
request
on
on
on
the
pod
allowed
us
to
cut,
let's
say
the
the
spike
in
half.
But
that
is
something
you
will
have
to
face
ever
I
mean
in
every
situation.
If
you
don't
want
to
let's
say
over
on
over
oversize
your
environment,
you
will
have.
You
will
have
some
kind
of
spike
when
a
gvm
starts.