►
Description
Don’t miss out! Join us at our upcoming hybrid event: KubeCon + CloudNativeCon North America 2022 from October 24-28 in Detroit (and online!). Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
How Adobe is Optimizing Resource Usage in Kubernetes - Carlos Sanchez, Adobe
Moving to Kubernetes opens the door to a world of possibilities, the amount of workloads that can be run and the flexibility it provides. However this comes at a cost on managing the resources used by many applications and teams. At Adobe we make extensive use of standard Kubernetes capabilities to reduce resource usage and we have also built some solutions at several levels of the stack to improve it. From autoscaling to workload hibernation, from automated resource requests to Kubernetes Jobs, we have experimented with and implemented several features that decrease our resource usage and lower the cost of running many Kubernetes clusters at scale. Both at workload resource level and also at achieving higher density clusters that reduce the number of clusters we need and the operating costs.
A
Thank
you.
So
I'm
here
to
talk
to
you
about
how
we
are
optimizing
resources
in
kubernetes.
How
can
you
reduce
costs?
How
can
you
improve
utilization
on
your
kubernetes
clusters
and
I'll
show
you
what
you
can
do
with
open
source
kubernetes
I'll
show
you
what
we
have
built
on
top
further
business
use
cases,
and
hopefully
you'll
get
some
ideas
that
you
can
go
back
and
use.
I
used
yourself,
so
that's
what
you
already
said.
I
one
thing
that
people
know
me
for
for
good
or
bad
things
is.
A
I
restarted
the
junkies
kubernetes
plugin
and
but
I'm
happy
that
you
all
made
it
here
right
after
lunch
instead
of
going
and
take
a
siesta
which
would
be
the
proper
thing.
I
hope
you
got
coffee
because
we're
going
to
be
here
for
the
next
two
hours
just
kidding,
so
I
work
at
one
team
at
adobe.
We
use
kubernetes
at
adobe.
A
lot
of
people.
Ask
me,
oh
I
didn't
know.
Adobe
use
is
kubernetes.
A
A
A
It
has
a
huge
marker
for
extensions
developers
that
write
their
own
extensions,
plugin,
well,
components
that
run
on
adobe
experience
manager
and
they
write
modules
that
that
run
in
process
on
am
so.
This
is
an
interesting
use
case
because
we
run
customer
code
in
our
multi-tenant
clusters,
so
on
kubernetes
on
the
cloud
service
am
cloud
service
specifically
there's
also
on-prem
and
managed
services,
but
on
on
on
the
cloud
service,
we
have
more
than
25
clusters
and
we
keep
growing
every
month,
because
this
is
a
content
management
system.
A
A
A
I
like
to
refer
it
as
a
micro
monolith,
because
it's
not
our
use
case
is
not
the
the
typical
use
case.
I
have
one
deployment
and
I
scale
up
to
200
pods
or
something
like
that.
We
have
thousands
of
deployments
that
are
very
similar,
but
they
also
scale.
But
it's
not
one
deployment
that
scales
a
lot
is
all
thousands
of
deployments
that
are
smaller.
A
We
have
multiple
teams
building
services
on
top
of
kubernetes
and
on
the
application
level,
and
this
is
also
important,
because
we
want
a
way
to
to
make
these
optimizations
and
these
ways
to
to
scale
that
are
like
orthogonal
to
the
developer
teams.
If
we
don't
want
to
chase
people
in
each
team
to
do
something,
sometimes
it
is
way
better
to
have
a
way
that
applies
to
the
whole
cluster
or
or
we
or
we
don't
have
to
require
each
team
to
do
something.
We
do
that
for
them,
so
those
solutions
are
way
better.
A
On
am,
this
is
a
specific
case
for
java
applications.
Anybody
here
running
java,
kubernetes,
yes,
a
lot
of
people.
So
if
you
probably
know
java,
the
jvm
is
going
to
take
all
the
memory
on
a
startup
and
manage
it.
So
you
set
the
heap
sizes
and
if
you
say
I
don't
know,
75
percent
of
the
memory
as
a
hip
size
and
you
have
one
gig,
it's
gonna
take
those
750,
megs
and
use
it
all.
The
time
and
kubernetes
doesn't
have
any
visibility
on
what's
actually
used.
A
What's
not
the
jvm
just
takes
it,
and
so
that
kind
of
makes
it
a
bit
harder
to
get
visibility
on
what
happened
on
the
jd
case
over
11,
the
jvm
will
detect
how
much
how
many,
how
much
limits
is
set
on
the
memory
level
and
the
jvm
is
never
going
to
try
to
use
math
ma
more
than
that
in
the
previous
versions.
It
was
just
using
by
default.
It
would
take
the
host
memory
and
that
would
always
typically
cause
crashes.
A
So,
to
start
on
the
things
you
can
do
to
improve
usage
first
thing,
obviously
kubernetes
cluster
autoscaler,
who
is
using
the
cluster
router
scaler.
Everybody
is
using
clutter
scaler.
Unless
you're
running
on
bare
metal,
you're
gonna
be
use
the
cluster
autoscaler
to
increase
and
reduce
the
the
cluster
size.
A
You
can
base
it
on
cpu
a
memory
request.
We
always
leave
room
for
the
spikes,
because
you
don't
want
new
paths
to
require
notes
and
the
time
it
takes
for
a
node
to
come
up
and
so
on,
and
we
have
multiple
scale
sets
for
different
regions
for
different
reasons.
Specifically,
we
want
a
different
availability
zones.
A
A
So
things
that
you
can
see
these
are
examples,
real
examples.
So
you
can
see
the
cluster
of
the
scalar
going
up
and
down
in
the
number
of
nodes
in
a
cluster,
some
spikes
that
could
be.
I
don't
know,
maybe
it's
a
day
of
the
week,
or
maybe
it's
business
hours
or
something
like
that
for
our
customers
and
we
have
these
typical
patterns.
A
Sometimes
you
will
see
other
patterns
that
are
a
bit
more
scary
right.
So
this
going
to
the
to
the
limit
of
this
cluster
size-
and
this
was
because
of
a
bug
that
triggered
out
the
scalar
to
to
scale
up.
So
you
see
how
at
some
point,
the
two
scalar
went
crazy
and
because
we
have
the
limit
set
up,
this
didn't
keep
going,
but
you
see
once
we
figure
out
what
was
happening.
A
The
other
typical
option
you're
going
to
have
is
the
horizontal
polarity
scale
who's
using
our
horizontal
button,
scaler
a
lot
of
people
using
so
you
this
is
basically
creating
more
parts
of
a
deployment
when
you
need
them,
and
we
have
two
ways
to
metric
set
up
to
the
to
the
hpa
of
the
scale
on
cpu
and
http.
Requests
per
minute
for
cpu
is
a
bit
problematic
because
you
could
have
periodic
tasks
or
startup
tasks
that
spike
the
cpu.
A
So
imagine
somebody
makes
a
mistake
or
a
customer
makes
a
mistake
and
only
start
up
the
cpu
spikes,
especially
with
java.
Maybe
the
garbage
collection
things
like
that.
This
would
cause
a
cascading
effect.
So
if
every
time
a
pot
starts
there's
a
spike
on
cpu
that
would
trigger
another
pot
to
start
up
that
will
also
spike
on
cpu
and
so
on.
A
A
As
long
as
customers
don't
have
expensive
requests
that
you
have
one
request
that
takes
a
lot
of
resources,
then
the
number
of
requests
is
not
an
indicator
to
scale
up,
and
this
probably
save
us
like
50
to
75
percent
of
then
running
at
full
scale,
eight
balls
temples
or
whatever
for
each
customer
right.
So
this
allows
us
to
run
from
two
parts,
for
we
use
two
parts
for
for
production
environments.
A
We
have
one
pot
for
other
environments,
but
we
don't.
We
don't
go
to
the
to
the
run
on
the
limit
all
the
time-
and
here
it's
another
example
how
you
see
that
the
number
of
spots
more
or
less
matches
the
requests
per
minute
that
we
get
and
the
request
per
minute
is
again
it's
a
very
typical
example
of
business
hours.
Night
number
of
requests
that
you
get
and
the
pods
just
match.
A
Another
option
you
have
on
kubernetes
is
the
vertical
polarity
scaler
anybody
using
the
vertical
scaler,
less
people?
Okay.
So
this
is
a
basically
increase
and
decrease
the
number
of
resources
used
for
each
spot
or
each
deployment.
So
you
don't
scale
with
motherboards.
You
just
take
one
scale:
increase
or
decrease
the
number
of
resources.
A
Something
that
is
problematic
is
that
it
requires
the
the
restart
of
the
pots.
So
if
you
have
something
that
is
very
fast
to
start,
that
may
not
be
a
problem.
If
you
have
something
that
takes
a
long
time
to
start,
then
that's
a
problem.
This
could
be
said
automatically
or
on
the
next
start.
So
you
don't
are
you're,
not
killing
pots
continuously.
A
Now
some
things
that
you
that
are
outside
of
kubernetes,
that
we
build
ourselves
for
a
use
case.
The
first
one
is
hibernation
and
is
very
similar
to
the
scale
to
zero
problem
that
you
can
solve.
Today,
with
the
with
the
horizontal
pull
out
the
scalar
and
custom
matrix,
you
could
do
a
scale
down
to
zero
or
if
you
use
k
native
or
something
like
that
or
functions,
then
it
is
something
like
that,
but
this
has
a
twist.
We
don't.
First,
our
pods
take
a
while
to
start
some
minutes
to
start.
A
A
So
when
a
customer
environment
doesn't
have
any
resources
coming
in
for
x
amount
of
hours,
we
just
scale
it
down
and
we
scale
the
whole
environment,
which
is
several
deployments,
and
we
also
delete
ingress
routes
and
other
objects
that
may
limit
cluster
scale.
So
we
hit
a
limit
where
we
have
a
lot,
because
we
have
a
lot
of
environments
running
thousands
of
environments
in
each
cluster.
We
have
thousands
of
ingresses
and
at
some
point
that
becomes
a
problem
for
reprogramming
the
ingress
controller.
A
This
is
implemented
very
easily,
very
simple:
it's
a
cron
job
that
just
goes
to
prometheus
checks,
the
number
of
requests
in
the
last
n
hours
and
if,
if
it
was
not
accessed
it
just
scales,
it
hibernates
it
and
for
the
hibernation,
because
we
we
change
the
ingress
route
that
the
customer
with
you
or
the
user
would
use
and
just
point.
We
point
them
to
an
to
a
website
where
they
can
click
a
button
and
be
hibernated,
and
this
we
are
playing
between
16
80
percent.
A
A
A
Obviously,
if,
if
for
whatever
reason,
many
pods
that
use
a
lot
more
resources
than
their
requests
happen
to
be
in
the
same
node,
then
you
could
get
cpu
throttling
on
some
side
effects.
But
then
that's
what
I
mean,
what
we
analyzed
right.
We
look
at
the
utilization
and
what
are
the
chances
of
high
resource
using
paths
being
in
the
same
node
at
the
same
time,
so
that's
risk
benefit
thing.
A
A
A
A
A
A
Some
questions
that
I
anticipate
is,
why
do
you
use
arc
another
vpa
recommender,
and
this
is
a
team
that,
where
joe
is
working
so
the
arc
allows
them
to
to
do
the
full
control
of
the
recommendation
engine
and
in
the
implementation.
As
I
said,
it
was
at
the
cluster
level,
so
you
don't
have
to
deal
with
specific
deployments.
It
just
applies
to
everything
running
in
the
cluster.
A
Okay,
so
to
sum
it
up,
we
have
a
few
optimizations.
I
didn't
mention
phenox
in
the
whole
talk.
Maybe
I
should
or
I
shouldn't,
because
that's
all
the
trend
now
so
we
have
from
the
kubernetes
ecosystem.
We
use
the
cluster
of
the
scalar
hpa
vpa
and
there's
some
new
things
coming
like
the
hpa
down
to
zero.
That
was
very
interesting.
That
was
already
out
there
some
releases
ago
vpa.
A
C
A
D
C
B
A
I
think
that's
all
tunable
at
the
autism
how
to
cluster
out
the
scaler.
If
you
want
to
do
that
or
not,
we
have
a
lot
of
things
that
come
and
go
so
we
don't
have
very
long-lived
pods.
So
we
don't
have
that
problem
that
much
we
I
don't
know.
Probably
we
don't
have
anything
more
than
a
few
days
old.
So
it's
continuously.
A
C
E
A
We
started
with
some
numbers:
the
different
teams
building
applications.
They
come
up
with
those
numbers
and
we
just
help
them
see
what's
happening
and
have
providing
them
their
grafana
dashboards
and
things
like
that
and
each
team
looks
at
it
and
says
you
know:
well,
I'm
using
more
I'm
using
less,
I
need
more,
a
little
less
and
they
keep
refining
it.
The
initial
numbers.
E
A
A
A
Enforce
that,
on
the
pull
request
level,
we
use
regal
policies
with
conf
test
and
we
have
a
set
of
policies
typical
ones
like
security
policies
like
you
should
not
mount
secrets
as
environment
variables.
You
should
not
do
some
things
mount
things
from
the
host
and
then
you
have
to
put
requests
and
limits.
G
A
A
If
you
use
like
machine
learning,
workloads
right,
those
are
going
to
be
like
90,
95,
cpu,
all
the
time,
otherwise
you're
just
wasting
money.
You
want
them
to
be
up
there.
In
our
case.
We
we
cannot,
because
it
doesn't
depend
on
us.
It
depends
on
how
much
traffic
we're
getting.
We
cannot
make
them,
be
there
all
the
time
and
because
it
has
to
handle
the
spikes.
A
H
A
Use
a
and
we
don't
use
a
spot
instances
because
pricing
reasons
or
whatever
is
not
worth,
I
guess,
but
I've
done
it
in
the
past-
just
use
spot
instances
for
things
I
don't
know
like
jenkins
builds.
You
can
just
on
kubernetes
just
use
spot
instances
if
you
don't
care,
if
sometimes
a
bill
fails
because
there's
no
spot
instances,
you
can
wait
a
bit
more.
Then
that's
fine
and
the
types
of
nodes.
A
We
use
standard
ones,
we
look
at
that.
We
look
at
the
proportion.
Basically,
what
we
measure
is
the
proportion
between
cpu
and
memory
that
our
application
uses,
and
then
we
went
and
looked
at
vms
that
have
that
proportion,
because
horse
is
pretty
normal.
I
said
but
yeah.
If
you
have
like
high
cpu
usage,
then
you
would
go
with
a
hype:
cpu
to
range
ratio,
cpu
to
memory
ratio,
and
so
so
it
depends
on
your
use
case
and
also
depends
if
whether
you
mix
workloads
or
not,.
I
So
I'm
wondering
if
you
could
go
into
more
detail
about
your
jvm
configuration
and
the
memory
usage.
I
think
the
newer
right,
the
newer
garbage
collectors
are
able
to
release
memory
to
the
operating
system
yeah.
We
didn't
have
a
really
good
experience
with
that.
So
I'm
wondering
if
you
could
share
your
insights.
A
A
A
So
I
wouldn't
recommend
sticking
with
the
defaults
because
then
you're
gonna
get
surprises,
maybe
you're
using
pods
with
less
memory
and
then
now
the
jvm
suddenly
changes
your
garbage
collection
algorithm.
It
changes
your
your
amount
of
hip
and
things
like
that.
So
it's
better.
If
you
always
set
it
explicitly
to
what
you
need
and
yeah,
it
was
mostly
memory
setting
up
the
right
memory
in
this
application
case.
We
want
the
heap
to
be
always
the
same
amount.
We
don't
want
it
to
go
up
and
down.
K
A
We
were
looking
at
the
different
vm
sizes,
but
we
kind
of
left
it
not
for
now
it
looks
good
enough
for
us,
as
we
start
mixing,
maybe
different
workloads.
We
may
have
to
revisit
that
and
probably
increasing
the
cpu
usage
making
sure
that
people,
because
our
cpu
usage
is
very
low.
In
average,
then
we
have
times
where
there's
a
lot
of
traffic
black
friday
whatever
and
then
yeah
I
mean
we're
looking
now
more
at
other
costs
that
come
on
is
not
so
much
about
the
infrastructure
level,
but
it's
on
how?
A
J
I
J
A
A
The
problem
is
when
you
have
pods
that
request
a
huge
amount
of
mem
of
cpu
like
things
for
customers,
so
they
have.
I
don't
know
four
cpus
or
hcps
whatever,
but
then
you
have
cluster
services
that
you
want
always
running
that
require,
but
that
only
leads
like
operators
or
something
that
only
need
0.5
cpus.