►
Description
Don’t miss out! Join us at our upcoming hybrid event: KubeCon + CloudNativeCon North America 2022 from October 24-28 in Detroit (and online!). Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
How Lombard Odier Deployed VPA to Increase Resource Usage Efficiency - Vincent Sevel, Lombard Odier SA
Container orchestrators have become the de-facto standard to deploy a wide variety of workloads. As the number of deployments increases, so is the pressure on resource usage, and hardware costs. Container runtimes and Kubernetes come with a set of tools that help make the most out of your infrastructure such as cgroups with resource usage limitation and prioritization, requests and limits on cpu and memory, quality of services. Even with those tools, it can be challenging to understand how they work, and how to use them. In this talk, the speaker will offer a review of the available mechanisms, how they map at the orchestrator and runtime levels, and introduce the Vertical Pod Autoscaler as a mean to optimize resource tuning at scale. He will share some of the lessons the company learned since starting this effort. And finally he will describe where they are in the deployment phase, and give some perspective on the direction where they are headed.
A
Today,
we're
going
to
talk
about
resource
efficiency
in
the
context
of
allocation
in
a
kubernetes
cluster
hi.
My
name
is
vincent
saval.
I'm
working
for
a
company
called
lombardier,
I'm
an
architect,
I'm
I'm
involved
in
a
cross-unit
team
called
platform
ops
which
aims
at
delivering
the
platform
as
a
product
and
in
that
context,
I'm
involved
in
different
projects.
A
A
A
A
So
the
story
starts
probably
a
year
and
a
half
ago
when
we
were
starting
to
ramp
up
our
workload
on
our
kubernetes
cluster
and
we
started
seeing
this
kind
of
message
a
little
bit
too
often,
then.
Well,
basically,
there
were
two
reasons
for
this.
The
first
reason
is
the
was
the
the
the
cluster
we
were
using
was
pretty
small
and
the
the
workload
was
starting
to
pile
up,
but
the
other
reason
was
we
did
not
have
a
good
understanding
about
how
resource
allocation
works
inside
the
kubernetes
cluster.
A
A
So
at
that
point
you
could
be
wondering
okay.
What
is
I
mean?
What's
particular
about
all
of
this
I
mean
I've
been
doing
capacity
planning
on
traditional
workload
for
many
many
years,
and
I
guess
the
big
difference
is
that
we're
moving
from
static
workload
to
dynamic
workload
where,
as
opposed
to
treat
your
vms
as
pets,
now
you've
got
your
worker
nodes
that
are
catalyzed
and
you
standardize
the
configuration
for
your
worker
node.
So
you
have
to
be
better
at
tuning
your
your
workload
underneath
all
right.
A
So
at
lumbarod,
we've
got
currently
three
main
clusters,
one
for
development
and
integration.
One
is
tnt
uat
and
one
for
production.
We've
got
another
smaller
one
for
production
for
different
use
case
and
we're
expanding
the
number
of
cluster
when
studying
so
we're
working
on
the
increasing,
and
we
are
in
the
process
of
deploying
another
cluster
into
the
the
public
cloud,
I'm
just
showing
the
applicative
part
so
that
that's
the
parts
for
the
components
that
we
are
writing
ourselves
actually
are
our
worker
nodes.
A
Our
cluster
has
something
like
starting
1300
or
1400
pods.
When
you
take
everything
into
account,
including
the
third
party
and
all
the
supporting
supporting
workloads,
we're
running
openshift,
four
eight
and
on
the
virtualized
vmware
infrastructure
with
size,
our
worker
nodes
and
one
thing
on
the
hardware
at
the
hypervisor
level-
is
that
we're
using
a
cpu
overcommit
the
ratio,
one
two
five.
A
So
at
the
end
of
the
gx
project,
we'll
probably
be
running
more
than
twenty
thousand
pods
across
all
clusters,
and
I
mean
by
definition
we
won't
be
able
to
do
fine
tuning
on
all
those
workloads.
With
this,
this
amount
of
of
containers
and
parts
running
in
our
platform,
all
right.
So
when
we
talk
about
resources
on
the
container,
we
need
to
take
into
account
two
different
notions.
A
A
A
Then
we
have
cpu,
which
is
said
to
be
a
complete
compressible
resource,
which
means
that
in
in
this
two
situation,
when
the
usage
reaches
the
limit,
assuming
one
limit
has
been
defined
for
your
pod
or
the
nod,
which
is
100
percent
of
its
capacity.
Then
the
pod
is
going
to
be
slowed
down.
It's
not
going
to
crash
it's
going
to
be
throttled,
but
your
position
to
memory,
which
is
a
non-possible
resource.
A
Okay,
so
we
need
to
to
find
the
right
balance
between
oversizing
the
request,
which
gives
you
high
behavior
predictability,
because
you're
going
to
over
reserve
resources
for
your
workload,
but
it's
going
to
give
you
low
density,
so
a
low
efficiency
to
some
extent,
if
you're,
using
overcommit
at
the
vitalization
layer
you'll
be
able
to
compensate
some
of
it,
but
not
not
everything.
So
we'll
say
this
is
a
performance
performance,
optimized
approach
and
you
need
to
balance
that
with
the
opposite,
which
is
undersizing.
A
The
request
in
your
cluster,
where
you're
going
to
have
low
b
of
your
predictability,
because
you
put
a
high
density,
so
you'll
be
gambling
the
usage
of
your
resources
against
the
performance
of
your
workload.
If
we're
talking
about
cpu
starving,
then
you'll
get
some
throttling
on
your
worker
on
your
on
your
pods.
A
If
you're
talking
about
memory,
then
you
may
have
some
evictions,
so
we
could
say
that
there
is
a
high
density,
so
high
efficiency,
but
if
you
start
having
some
throttling
or
some
evictions
you're
going
to
lower
your
efficiency,
so
you're
going
to
use
on
you're
going
to
sacrifice
the
behavior
of
your
of
your
pod.
So
we'll
say
in
that
situation:
it's
a
cost
optimized.
So
so
how
do
we
sit
right
in
the
middle,
where
we
assess
precisely
the
the
request
we
need
to
set
up
on
our
on
our
workload?
A
Well,
one
idea
is
to
look
at
past
behavior.
So,
let's,
let's
take
a
small
example:
we've
got
four
pods
running
on
an
ideal
worker
node
and
we've
got
resources
ranging
from
25
to
1100,
with
workloads
behaving
very
differently,
some
with
peaks
and
low
activity,
some
with
a
lot
of
activity
all
the
time.
A
So
the
first
question
is:
what's
work
or
not
that
you
need
to
run
this
this
workload.
So
when
you've
got
the
least
amount
of
waste-
and
you
don't
have
you
don't
sacrifice
on
the
behavior,
but
actually
the
better
question
is:
what
do
I
need
to
reserve
for
each
of
my
pods
right
because
you're
all
going
to
configure
them
independently
from
one
another,
because
each
part
doesn't
know
where
it's
going
to
sit
and
we're
going
to
be
the
neighbors?
A
Another
way
to
look
at
this
graph
is
to
look
at
the
the
cumulative
conception
of
a
time,
so
it's
very
easy
to
see
that
the
optimal
host
would
be
the
one
which
is
sitting
at
the
peak
of
this
graph,
so
in
in
that
situation.
79
now,
if
you
took
the
average
consumption
of
each
pod
and
you
send
that
what
would
happen
is
that
you
would
get
a
a
host
with
1200
capacity,
so
you've
got
a
lot
of
density
because
you're
taking
a
lot
of
the
resources
that
are
available
on
the
now.
A
A
Now,
if
you
took
the
max,
you've
got
the
opposite
situation,
where
you
over
reserve
resources
for
your
workload,
so
you've
got
lots
of
extra
capacity,
so
you're
not
going
to
have
any
throttling
or
any
eviction,
but
you're
going
to
waste
a
lot
of
resources.
So
we
need
something
in
the
middle,
which
is
what
vpi
is
doing.
A
A
A
So
there
is
a
little
bit
of
gambling
and
then
you
benefit
from
the
extra
capacity
that
is
provided
and
reserved
for
the
others,
but
but
not
used,
and
that's
basically
doing
the
exact
same,
overcoming
that
we
were
doing
initially
at
the
utilization
layer,
but
we're
doing
it
at
the
pod
level,
all
right
so
vertical
pad
auto
scaler.
This
is
a
kubernetes
tester
project
and
it
can
provide
a
recommendation
on
resource
usage
and
specifically
on
the
amount
of
resources
to
reserve
for
your
pod.
So,
basically
that
helps
you,
configuring.
A
A
Interestingly,
it
can
upscale
or
don't
scale
the
resources
that
has
been
that
have
been
configured
depending
on
the
situation.
Maybe
you
know
you
need
more
cpu.
Maybe
you
need
less.
You
can
work
on
memory
and
cpu
and
optionally.
It
can
apply
the
recommendation
on
your
workload
right,
it's
available
on
the
public
cloud
through
different,
offering
and
at
combi
we're
using
it
through
a
dedicated
and
supported
openshift
operator.
That's
what
we're
doing
all
right!
So,
let's
take
an
example:
here's
the
main
cr
that
you
would
create
for
a
specific
workload.
A
A
A
After
watching
your
container
working
for
some
time,
vpa
will
calculate
a
recommendation
that
appears
directly
into
the
cr
in
the
status
section,
and
there
are
four
metrics.
The
most
important
one
is
the
target
which
is
based
on
the
p90
like
we
saw
like
we
saw
before
for
all
the
the
control
resources.
A
A
The
controller
will
replace
at
startup
those
results
with
the
the
calculated
values
in
terms
of
use
cases
we
using
vpr
chromebody
for
stateless
workloads
and
jobs
and
current
jobs,
but
there
is
also
there
is
also
an
additional
use
scale,
which
is
interesting,
which
is
a
stateful
workload
where
scaling
horizontally.
Stateful
workload
might
be
specific
to
the
technology
that
you're
using.
So
in
that
situation,
vpa
is
a
very
good
option
to
grab
more
resources.
A
There
are
so
a
few
limitations,
however,
when
you're
using
vpa,
it
is
said
in
the
documentation
that
you
should
be
using
a
vpa
to
control
memory
on
jvm
based
workload
it.
This
is
essentially
related
to
the
way
the
jvm
manages
memory
and
the
limited
visibility
it
provides
on
the
heap
to
the
to
the
vpa,
or
I
mean
the
metrics
that
are
coming
from
prometheus
ano.
Another
thing
to
look
into
is
is
what
you've
got
to
be
careful
is
using
vpn
with
hpe,
and
typically
you
cannot
use.
A
A
A
You
have
to
be
careful,
and
you
have,
if
you,
if
you
give,
I
mean
if
you're,
using
the
initial
or
even
the
recreate,
auto
mod.
You
have
to
be
careful
about
potential
excessive
recommendation.
That
vpa
would
be
doing
because
it
doesn't
understand
fully
your
workload,
and
you
want
to
make
sure
that
the
sum
of
all
the
requests
that
will
be
calculated
by
vpa
do
not
exceed
your
your
cluster
capacity.
A
One
thing
one
pinfall
is
that
there
is
only
one
vp
object
allowed
per
workload,
so
if
you
create
2vp
object
for
the
same
deployment,
vpa
is
going
to
select
only
the
the
all
that
all
this
one
is
going
to
ignore
the
the
most
recent
one,
and
the
last
thing
is
that
there
are
some
distributions
that
limit
the
number
of
vp
objects
you
can
have
in
a
single
cluster.
So,
for
instance,
gke
allows
only
500
objects
into
the
cluster.
A
A
So
where
are
we
in
in
lombardy,
so
we've
deployed
vpn
on
all
clusters,
we're
using
the
request
only
mod
that
was
based.
That's
basically
an
answer
to
the
initial
problem
that
we're
having
and
we're,
also
starting
to
experiment
with
the
operative
highlighting
on
the
dev
cluster,
and
that
explains
some
of
the
numbers
on
the
dev
cluster,
which
are
pretty
low,
nothing
to
do
with
vpn,
but
it's
another
tool
in
the
toolbox
for
capacity
planning.
A
A
We
still
have
a
bit
of
this
discrepancy
on
the
cpu
request
versus
usage.
They
might
be
well
we're
still
analyzing
this.
It
might
be
related
to
the
other
type
of
workload
that
are
running
on
on
a
worker,
node
and
that
are
not
covered
by
by
vpn,
but
basically
the
next
challenge
that
we
have.
We
kind
of
solved
the
cpu
situation
and
the
next
challenge
is
going
to
be
about
memory
so
for
cpu
we're
running
initial
on
all
nodes.
A
So
so
we
it's
kind
of
deployed
progressively
on
each
workload
in
a
different
cluster.
We
start
we're
starting
actually
to
see
the
the
services
in
terms
of
vcp
use.
It
doesn't
change
the
usage,
it's
it
changes
the
perspective
of
the
allocation
or
the
the
the
you,
the
usage
that
you
have
on
your
cluster
and
what
you
reserve
for
memory.
It's
very
new.
I
mean
it's.
We
we
started
doing
that
a
few
weeks
ago,
so
it's
deploying
on
the
cluster
on
the
dev
cluster.
A
Only
and
what
happens
is
that
we,
the
development
team,
are
going
to
set
a
value
which
is
going
to
be
used
by
default
for
the
request
and
the
limit,
so
basically
we're
working
in
guaranteed
mode
by
default.
Initially,
and
what's
going
to
happen,
is
that
vpa
is
going
to
watch
your
activity
and
basically
downscale
the
request
on
the
memory.
So
so
far
we
were
able
to
save
18
gigs
of
ram
of
request.
A
All
right
so
listen
learns.
You
have
to
be
careful
about
namespace
or
object,
deletion
and
recreation
in
our
situation.
We're
choosing
we're
not
getting
the
metrics
out
of
prometheus,
which
means
they're
being
stored
in
a
compacted
format
in
a
specific
cr.
If
you
remove
checkpoint
cr,
then
you
lose
your
matrix
right
and
you
have
to
start
over
with
your
recommendation.
A
The
other
thing
is
that
recommendation
is
stored
as
a
status
in
the
vpa
object.
If
you
remove
the
vpa
object
or
you
replace
it,
the
next
time
you
deploy,
then
for
30
seconds
you're
going
to
lose
the
recommendation.
Your
pod
is
going
to
start
with
whatever
is
configured
right.
So
if
you
want
to
make
modification
on
this,
vpn
object,
you've
got
to
update
it
and
not
replace
it.
A
A
One
thing
to
to
kind
of
not
forget
is
that,
with
reserving
resources
based
on
the
p90,
for
instance,
in
the
context
of
vpa,
is
basically
doing
overcoming
it
at
the
pod
level
and
if
you're
using
overcommit
at
the
vitalization
layer,
they
could
conflict
with
ways
with
one
another.
So
you
have
to
be
very
conservative
with
your
recommend.
You
you're
doing
a
divitization
layer
and
one
very
interesting
thing
that
we
got
out
of
using
vpa.
A
Was
that
not
only
we
were
able
to
do
fine
tuning
from
one
application
to
another,
but
we
were
able
to
get
a
different
recommendation
and
tuning
for
the
same
application
in
the
different
environment,
and
it's
it's
not
unusual
when
you
start
using
vpa
to
not
have
the
same
value
in
production
test
and
integration.
You've
you've
got
kind
of
the
best
value,
a
few
things
to
to
be
careful
about
in
the
context
of
gvm
workload.
One
mistake
we
did
at
the
beginning
is
that
we
wanted
to
protect
the
startup
of
critical
application.
A
The
issue
with
application
java
applicative
framework
is
that
they
tend
nowadays
to
consume
a
lot
of
cpu
at
startup,
so
you've
got
to
pick
an
initial
peak.
If
you
size
your
request,
based
on
that
you're
going
to
over
reserve
an
overconsume
request,
so
do
not
oversize
to
cope
with
startup,
you
would
have
a
better
efficiency.
A
The
issue,
however,
is
then
the
soldering
hub.
What
we
call
the
soldering
hub
problem,
which
is
when
you've
got
a
massive
amount
of
pods
that
are
starting
are
going
to
do.
A
a
big,
huge
peak
of
of
of
consumption.
Vpa
is
not
helping.
You
with
that,
so
that
that's
an
issue
okay,
so
next
we
want
to
like
I
said
I
mean
the
next
challenge
for
us
is
keep
keeping
memory
at
a
sustainable
level.
So
we
will.
A
We
want
to
continue
working
on
that
and
so
get
some
feedback
on
the
development
cluster
and
start
actually
going
up
in
the
other
cluster.
We
want
to
continue
adding
some
density
on
our
kernel.
I
mean
the
ultimate
goal
would
be
at
some
point
that
we
don't
need
the
vitalization
of
a
commit
at
I
mean
at
the
utilization
level,
and
we
can
go
to
bare
metal
right.
You
don't
want
to
go
to
bare
metal
if
you've
got
a
low
density
on
on
your
work
or
not.
A
A
few
issues
to
consider
so
the
first
one
is
being
pushed
by
a
colleague
of
mine,
matthias
bertie,
which
aims
at
providing
the
ability
to
configure
a
target
personal
value
different
from
the
default
90,
which
is
hard
coded
today.
So
we
could
choose
with
a
more
cost
optimized
approach
or
a
more
performance,
optimized
approach,
depending
on
the
on
the
workload.
A
It
would
give
us
more
flexibility,
just
if
you
go
back
in
the
the
small
example
I
was
showing
earlier,
the
perfect
value
was
p78
right,
so
pt
was
already
very
conservative
and
we
could
be
saving
some
value
by
actually
a
little
bit
with
more
gambling
toward
a
more
custom
cost
optimized
approach.
The
second
issue
is
very
important.
That's
the
that's!
A
The
idea
is
to
provide
kubernetes
with
the
ability
to
replace
requests
and
limits
without
restarting
the
pod.
This
would
open
up
use
cases
like
the
auto,
which
requires
that
and
might
help
also
as
the
basis
for
implementing
something
smarter
for
the
centering
health
problem
about
that
particular
issue.
D
A
That's
a
good
question:
no,
we
didn't
look
at
this
integration.
There
would
be
one
one
good
thing
about.
It
is
that
you
wouldn't
rely
on
internal
storing
with
the
cr,
so
you
would
have
you
wouldn't
have
the
limitation.
I
was
talking
about
where,
if
you
remove
this
object,
you
lose
your
metrics,
but
the
good
thing
about
not
depending
on
it
is
that
you
don't
have
a
dependence
on
on
permit
series.
So
if
it's
not
available,
basically
vpa
is
standalone
and
it
can
continue.
A
Okay,
so
yeah.
My
understanding
is
that
so
we
haven't
studied
that
work,
but
my
understanding
is
that
you
can
use
hpl
with
vpn,
but
you
can
you
you
need
to
use
it
on
a
metric
which
is
not
controlled
by
vpa.
So,
for
instance,
you
could
take
requests
per
second.
If
you,
which
is,
I
mean
the
basic
notion
into
in
serverless,
you
could
even
use
the
cpu
control
resource
on
vpn
and
use
memory
for
scaling,
but
you
don't
want
to
actually
have
both
metrics
used
in
in
the
two
in
these
two
features.
E
A
A
There
is
a
way
to
size,
the
hip
in
relation
to
ratio
on
the
the
container
memory.
So
if
you
start
actually
having
vpn
just
scaling
up,
I
mean
the
problem
is
with
ip
committed
versus
hip
usage
in
in
the
gvm,
so
we're
not
getting
getting
into
the
limits
today.
So
I
mean
my
understanding.
Is
that
we're
fine
with
recursion
limit.
C
A
My
yeah
very
good
point:
a
lot
of
people
have
been
complaining
about
the
the
I
mean
throttling
the
startup
because
of
jvm
workloads.
So
so
it's
a
common
issue.
The
solution
could
come
with
the
the
other.
The
cap
I
was
showing
earlier
with
the
in
place
update
of
resources.
So
the
idea
was
that
if
we
had
that,
maybe
we
could
startup
over
provision
request
to
cover
the
startup
and
once
the
pod
is
ready,
downgrade
the
the
request.