►
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon North America 2021 in Los Angeles, CA from October 12-15. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Groupless Autoscaling with Karpenter - Ellis Tarn & Prateek Gogia, Amazon
Autoscaling is a powerful technique to improve reliability and save costs. However, configuration of node autoscaling systems is a complex art. Karpenter is an open source, vendor neutral project that enables node autoscaling without the need for configuration and discovery for node groups in any cloud provider. With direct provider integration and out-of-the-box support for common node parameters like architecture, operating systems etc., taints, and labels, Karpenter gets users up and running quickly, while extending cleanly to more complex use cases.
In this session, we'll explore the history of autoscaling and our latest approach to low latency node autoscaling.
A
A
A
A
A
A
A
A
There
are
some
challenges
with
this
approach,
though.
Let's
see
what
are
those
challenges?
Crystal
autoscaler
assumes
that
the
instance
types
are
all
identical
in
a
given
group,
and
if
you
want
to
use
multiple
instance
types,
you
need
to
create
separate
groups
and,
furthermore,
if
you
need
to
spread
parts
across
acs,
multi
multiply
the
number
of
groups
by
each
a
z.
A
They
improved
how
cluster
autoscaler
generates
template
nodes
for
groups
to
decide
what
kind
of
node
to
expect
when
scaling
up
they
added
reliable
back
of
logic,
in
case
of
in
case
a
node
group
phase
to
scale
up
nodes,
and
they
made
some
more
additional
changes
and
some
of
their
efforts,
which
have
been
upstreamed
to
cluster
order
scaler,
which
makes
it
more
robust
and
production
ready.
But
one
thing
to
observe
here
is
all
these
different
approaches
are
trying
to
solve
around
node
groups.
A
B
Some
of
them
were
just
code
bugs
in
our
cloud
provider
implementation
things
that
we
could
fix.
You
know
things
that
we
did
fix.
Sometimes
it
was
just
customer
configuration
since
these
things
are
really
hard
to
get
right.
It's
really
easy
to
misconfigure
node,
auto
scaling.
It's
one
of
the
things
that
we
we
decided.
We
really
wanted
to
focus
on
to
make
easier
and
finally
part
of
it
was
just
architectural
by
nature
of
having
node
groups.
B
You
know
we
can
pick
an
m5,
large
or
or
any
instance
that
makes
sense
for
those
those
pods
whether
or
not
it's
been
seen
before,
and
additionally,
the
the
other
node
properties
like
the
availability
zone
and
the
operating
system.
All
of
these
things
we
can
compute
dynamically,
so
we've
shifted
from
this
static
template
world
to
this
dynamically
generated
template
world,
which,
which
reduces
the
configuration
complexity
pretty
significantly
so
once
we've
decided
what
instance
we
want.
We
just
tell
our
cloud
provider.
Please
give
us
this
instance.
B
B
Also,
the
api
load
is
pretty
significantly
reduced.
Since
for
every
single
one
of
those
asgs,
you
have
to
ask
the
cloud
provider.
You
know
what
is
what
is
the
current
status
of
this
thing?
What
is
the?
What
is
the
value
and
those
api
requests?
Really
they
really
stack
up
and
you
can
run
into
api
limits
and
and
the
whole
system
grinds
to
a
halt.
So
instead,
we
just
make
calls
when
we're
creating
and
deleting
capacity,
and
otherwise
we
rely
on
the
kubernetes
api
server,
which
can
support
much
higher
throughput.
B
It
also
reduces
the
simulation
complexity.
We
no
longer
have
to
to
look
at
such
a
large
number
of
templates.
We
can
collapse
the
simulation
space
to
just
instance
types,
because
we
can
prescribe
the
availability
zone
and
scheduling
properties
which
really
simplifies
things.
For
example,
instead
of
looking
at
for
at
all
the
possible
groups
and
saying
which
one
has
this
zone,
we
can
just
tell
our
cloud
provider
run
an
instance
in
this
zone.
B
So
we
ended
up
with
this
architecture,
which
should
seem
fairly
familiar
if
you're
familiar
with
the
cluster
auto
scaler.
We
start
up
at
the
top
left,
where
you
have
pods
that
were
created
and
haven't
been
scheduled
yet,
and
the
coupe
scheduler
gets
the
first
crack
at
it.
So
it
looks
at
the
existing
capacity
and
the
and
the
pending
pods
and
it
tries
to
find
places
to
schedule
those
pots
and
if,
for
some
reason,
their
resources
aren't
available
to
schedule
them.
B
We
try
to
make
sure
that
capacity
is
as
efficiently
packed
as
possible
and
that
we
can
bring
it
up
as
quickly
as
possible
and
then
that
capacity
joins
the
cluster.
So
your
existing
capacity
and
your
provision
capacity
just
now
or
your
new
cluster
state
over
time,
you
can
get
you
can
get
holes
in
the
capacity
you
can
get
inefficient
utilization.
So
we
have
another
controller
that
looks
globally
at
the
cluster
and
attempts
to
optimize
it
over
time.
B
So,
where
we
end
up
with
is
two
controllers,
one
is
a
fact
fast,
acting
latency,
sensitive
controller,
the
allocator
and
the
other
is
a
slow,
acting
cost
sensitive
controller
called
the
reallocator
and
together
these
work
to
both
schedule
pods
as
quickly
as
possible
and
optimize.
The
resource
utilization
of
your
cluster.
B
B
Now
we've
removed
asg,
and
so
we
want
to
use
ec2's
direct
capacity
launch
apis.
There's
two
of
these
one
is
called
fleet
and
the
other
is
called
run
instances
they're
roughly
the
same.
We
could
actually
use
either
of
them,
and
I
imagine
that
other
cloud
providers
they
might
have
multiple
options
as
well.
We
know
that
every
cloud
provider
does
have
the
ability
to
run
instances
in
general,
but
we
for
our
use
case
and
for
our
cloud
provider.
We
select
the
fleet
api
because
we
think
that
it
gives
us
the
most
flexibility.
B
So,
for
example,
once
when
deciding
what
node
we
want
to
create
for
a
given
set
of
pods,
we
don't
actually
limit
ourselves
to
one
availability
zone
and
one
instance
type,
and
this
is
constrained
to
our
cloud-
provider's
implementation.
You
could
have
another
implementation
that
selected
the
specific
instance
type
in
the
specific
availability
zone,
but
we
actually
can
get
it
some
additional
value
by
being
flexible.
B
So
we
tell
ec2
these
pods
will
fit
in
any
of
these
instance
types
and
given
their
topology
constraints,
it'll
run
in
any
of
these
availability
zones
and
by
giving
it
that
flexibility
ec2
can
actually
pick
the
cheapest
instances.
Given
the
constraints,
the
more
flexibility
you
give
easy
to
the
better
of
a
job
that
can
do
to
pick
the
right
instance
type,
given
your
constraints.
B
So
we
get
pretty
massively
increased
node
flexibility.
Here
we
get
this
really
nice
failover
behavior.
If
a
particular
instance
type
is
out
of
stock,
it
also
pretty
significantly
reduces
the
node
provisioning
latency,
so
systems
systems
like
auto
scaling
groups
and
and
other
group
systems,
there's
there's
a
little
bit
of
overhead
to
them.
So
there's
some
reconciliation
loop,
that's
happening
in
the
implementation
of
that
thing
and
you
know
maybe
it's
really
fast,
or
maybe
it's
not
very
fast,
but
we
found
that
with
asg.
B
B
Scheduling
is
also
critical
to
the
provisioning
workflow.
So,
like
I
mentioned
earlier,
we
work
in
tandem
with
the
cube
scheduler.
We
give
the
coupe
scheduler
the
first
shot
and
when
it
fails,
we
kick
in
afterwards.
But
one
of
the
corollaries
to
this
is
that
is
the
realization
that
provisioning
decisions
are
scheduling.
Decisions
by
nature
of
picking
a
new
node
for
pods
to
schedule
onto
you
are
making
a
scheduling
decision
whether
or
not
you
like
it
or
whether
you
like
it
or
not.
It's
it's.
B
B
We
get
an
instance
id
back,
we're
able
to
create
the
node
object
immediately
and
then
bind
the
pods
immediately
and
so
we've
enforced
a
scheduling,
decision
kind
of
outside
of
the
loop
of
the
kube
scheduler
and
that's
a
fairly
interesting
pivot,
from
the
way
that
the
cluster
autoscaler
works,
which
is
it
brings
up
the
new
capacity
and
then
waits
for
the
kube
scheduler
to
do
the
scheduling.
But
we
decided
that
if
we're
already
making
a
scheduling
decision,
we
might
as
well
enforce
it,
and
there
are
some
benefits
to
that.
B
So,
for
example,
because
we've
already
bound
the
pod,
the
second
that
the
node
comes
online,
you
can
start
pulling
the
image
because
you
already
know
the
binding
decision.
You
don't
have
to
wait
for
the
node
to
become
ready.
You
don't
have
to
wait
for
the
cube
scheduler
to
schedule
it,
so
that
was
just
kind
of
a
nice
dual
optimization
we
did
so
one
is
we
enforced
that
the
decision
that
we
made
is
actually
in
is
actually,
it
actually
happens
in
the
cluster
and
we
also
get
some.
B
The
other
thing
that
we
realized
is
that
the
cluster
auto
scaler
is
limited
in
its
version
compatibility,
so
they
copy
this
coupe
scheduler
directly
into
their
code
base
and
use
it,
and
so
because
of
this
there's
a
recommendation
that
the
coupe
scheduler
must
be
the
same
in
the
cluster
auto
scalers
code
base,
as
as
you
have
running
upstream
or
as
you
have
running
in
your
masters
rather,
and
so
this.
This
creates
a
tight
version,
coupling
between
those
two
things
because
they're
not
enforcing
the
the
binding
decision,
and
so
by
doing
this
binding
binding
decision.
B
We
now
free
ourselves
from
this
version
compatibility,
because
the
the
coupe
scheduler
is
its
own
system
that
has
its
own
set
of
responsibilities
and
the
carpenter
allocator
scheduler
is
is,
is
the
same.
It's
it
also
has
its
own
responsibilities,
and
so
you
no
longer
have
this
type.
Coupling
and
versions
of
carpenter
can
work
across
many
versions
of
kubernetes.
B
Another
core
layer
we
have
here
is
the
scheduling,
api
conformance
and
so
by
nature
of
writing
a
scheduler
or
writing
logic
that
decides
new
nodes
to
provision.
We
have
to
support
all
the
common
things
like
resource
requests
and
selectors
and
affinity
and
topology,
and
so
this
is.
This
is
just
kind
of
a
burden
that
we
take
on
as
part
of
trying
to
play
in
the
salt.
In
this
problem.
Space
and
specifically,
the
resource
requests
need
to
are
get
a
little
bit
complicated.
B
So
when
you
have,
I'm
sure
you
guys
are
familiar
with
running
gpus
or
running
potentially
other
custom
instance
or
custom
resource
types,
your
cloud
provider
effectively
needs
to
have
knowledge
of
how
to
launch
instances
with
those
particular
instance
types
or
resource
types.
Sorry,
and
so
there's
kind
of
an
ongoing
burden
here
of
the
cloud
providers
to
ensure
that
the
the
particular
properties
of
the
instances
that
the
pods
are
requesting
are
actually
able
to
be
launched
and
we've
built
a
bunch
of
machinery.
B
B
We
also
operate
based
off
of
this
concept
of
well-known
labels,
so,
for
example,
this
is
how
we
communicate
that
a
pod
needs
to
run
in
a
particular
zone
or
in
a
particular
instance,
type
there's
this
concept,
that
if
in
your
pods
node
selector
you
use,
for
example,
topology
dot,
kubernetes
dot,
io
zone
and
you
state
that
zone.
That's
a
signal
to
carpenter
that
the
the
node
that
you
need
must
be
constrained
that
then
it
must
have
a
label
of
that
zone.
B
So
we
use
these
well-known
labels
to
apply
additional
constraints,
there's
also
a
set
of
defaults,
so
that,
if
you
don't
you
don't
define
these
or
you
don't
have
to
define
these
in
all
cases.
But
if
you
do
need
to
do
apply
additional
constraints,
you
would
do
that
via
node
selectors
and
well-known
labels.
B
So
once
we've
sorted
out
our
scheduling,
you
end
up
with
kind
of
groups
of
pods.
Imagine
you
have
a
hundred
pods
that
need
to
be
scheduled
and
some
of
them
have
one
node
selector
and
some
of
them
have
another
node
selector.
You
end
up
with
these.
What
we're
calling
constraint
groups-
and
these
are
pods
that
that
can
be
scheduled
together
according
to
their
scheduling
constraints,
but
we
still
we
have
another
problem
here,
which
is
how
do
we
pick
the
instance
that
can
fit
all
of
those
pods?
B
So
we
sum
up
all
of
the
resource
requests
of
the
pods,
and
then
we
do
what's
called
bin
packing.
This
is
a
classical
computer
science,
np
complete
problem.
There's
a
couple
variants,
the
one
that
this
is
is
called
online
bin
packing,
which
means
we're
looking
at
the
subset
of
the
overall
pods
in
the
cluster
and
we're
attempting
to
pack
them
into
the
existing
capacity.
B
There's
a
bunch
of
research
in
this
space
and
we're
actually
super
excited
for
all
the
innovation
we
can
do
here.
But
the
algorithm
that
we've
picked
right
now
is
called
first
fit
descending
and
what
that
basically
means
is
we
sort
the
pods
by
size
and
we
try
to
pack
the
pods
biggest
to
smallest,
and
what
this
does
is
it
kind
of
has
this
nice
effect?
It's
called
first
fit
descending.
B
B
You
might
ask?
Well,
how
do
you
order
the
pods,
because
it's
not
a
single
dimensional
problem,
there's
a
couple
approaches
to
this:
you
can
either
prioritize
cpu
or
memory
or
do
the
euclidean,
which
is
the
the
sum
of
or
the
root
of
the
sum
of
the
squares,
and
so
we
found
that
the
euclidean
is
is
kind
of
a
best
effort,
simulation
of
kind
of
weighting
memory
and
cpu
equally,
when
when
sorting
them
and
that
it
tends
to
work
fairly
well
for
bin
packing.
B
B
B
We
also
respond
to
other
events
that
might
cause
instances
to
be
terminated.
So,
for
example,
when
ec2
notices
that
your
virtual
machine
is
unhealthy
for
some
reason,
it'll
send
an
event,
and
so
we
respond
to
those
and
we
we
drain
the
node
as
quickly
as
possible
and
try
to
try
to
heal
the
situation.
B
Termination
happens
after
the
rebalance,
so
you're
not
guaranteed
to
get
a
rebalance
request,
but
you
typically
do
you
are
guaranteed
to
get
a
termination
signal
and
that
termination
signal
means
you
have
two
minutes
to
drain.
So,
ideally,
we've
we've
managed
to
terminate
the
nodes
and
reschedule
everything
by
the
rebalance,
but
if,
for
some
reason
it
happens
so
quickly
or
something's
getting
gummed
up,
then
it
has
to
be
done
before
that
that
node
termination.
B
The
other
thing
that
we've
we've
realized
is
that
you
can
apply
a
node
ttl
of
a
long
time
frame.
So,
for
example,
there's
there's
some
value
in
saying
I
want
this
node
to
live
for
90
days,
and
this
is
super
useful
in
the
upgrade
case.
As
long
as
our
allocator
is
creating
nodes
on
the
latest
kubernetes
version
and
as
long
as
the
nodes
only
live
for
90
days,
your
cluster
is
basically
always
up
to
date.
B
Given
the
kubernetes
three-month
release
cycle,
there's
some
other
compliance
use
cases
where
you
might
care
about
how
long
the
node
lives.
Maybe
you
want
to
roll
your
fleet
every
month
or
so,
but
this
is
another
feature
that
we've
built
in
that
that
we
think
will
be
super
useful
for
upgrade.
You
know,
potentially
weird
memory
leaks,
potentially
compliance
cases.
B
This
is
actually
a
really
challenging
problem,
because
pods
are
immutable,
they
can't
be
moved.
You
have
to
be
really
careful
about
deleting
a
pod
and
waiting
for
the
new
one
to
be
created
and
then
enforcing
your
repacking.
So
this
is
still
very
much
in
the
design
phase.
We're
still
trying
to
figure
out
how
to
do
this,
but
we're
committed
to
the
idea
that
you
shouldn't
be
stuck
with
the
the
initial
state
that
things
were
provisioned
in.
You
should
have
another
controller.
That's
always
monitoring
always
trying
to
improve
the
situation,
so
we're
exploring
different
algorithms
here.
B
Also,
one
of
the
most
important
pieces
of
this
problem
is
one
that
doesn't
really
come
up
very
often,
but
we
realized
is
absolutely
critical.
It's
how
quickly
are
these
nodes
launched,
there's
a
whole
bunch
of
steps
they're
all
listed
here
in
in
the
critical
path
of
a
node
coming
online
and
right
now
in
aws?
That's
about
two
minutes
in
different
clouds.
I
think
it's
it's
actually
fairly
common
for
it
to
be
about
two
minutes
and
when
it's
this
slow,
you
end
up
with
this
use
case
where
you
you
actually
want
to
over
provision.
B
You
can't
wait
those
two
minutes
so
you'd
like
extra
nodes
to
be
in
the
cluster,
so
that
the
the
pods
can
be
more
scheduled
more
rapidly,
and
so
we've
decided
to
really
attack
this
problem.
We
think
we
can
get
it
down
to
maybe
50
to
30
or
15
seconds
if
we
do
a
ton
of
work,
and
so
we're
really
excited
to
to
attack
this,
this
latency
problem,
which
we
think
will
mitigate
the
need
to
do
over
provisioning.
B
I
also
want
to
mention
quickly
that
we
do
this
using
kubernetes
custom
resources.
This
is
kind
of
the
modern
standard
way
to
write
controllers
and
kubernetes.
We
found
that
using
command
line
arguments
and
a
single
global
operator
is
kind
of
untenable.
So
all
of
these
configuration
parameters-
the
taints,
the
labels,
the
cluster,
that
the
new
nodes
connect
to
all
of
these
things
are
encoded
in
a
custom
resource.
B
This
project
is,
of
course,
open
source
and
vendor
neutral.
It's
currently
incubating
in
the
aws
labs,
github
repo
and
we're
excited
to
figure
out
where
it's
going
to
go.
After
that,
we
have
a
roadmap
that
you
should
come
check
out
to
see
our
0.2
0.3
and
0.4
releases
that
we
have
planned.
We're
planning
on
executing
about
the
rate
that
kubernetes
does
so
releasing
as
the
crew
as
kubernetes
releases
and
the
other
thing
is.
A
Thanks
alice,
so
let's
go
ahead
and
see
some
of
the
concepts
that
alice
has
explained
in
action.
I
have
already
provisioned
an
eks
cluster,
it's
running
a
single
node
and
I
have
also
deployed
carpenter
onto
these
on
this
one
node.
So
at
the
moment
we
have
carpenter
running
some
of
the
components
which
carpenter
needs
and
some
of
the
system
parts.
A
So
as
a
next
step,
let's
go
ahead
and
keep
a
watch
on
all
the
nodes
on
this
cluster.
At
this
point,
we
are
ready
to
configure
carpenter
to
be
able
to
talk
to
the
api
server.
So
these
are
the
configurations
that
we
need
to
provide
to
carpenter
to
be
able
to
talk
to
the
api
server,
and
the
other
configurations
I
have
here
are
optional.
A
We
provide
a
list
of
instance
types
to
limit
the
num,
the
instance
types
carpenter
can
use
when
requesting
for
the
capacity.
We
are
also
limiting
the
zones
in
which
carpenter
can
create
the
capacity
and
detail
seconds
tell
carpenter
for
how
long
to
wait
when
it
detects
an
un
underutilized
nodes
before
it
terminates
the
node.
So
let's
go
ahead
and
create
the
crd
object,
which
carpenter
can
watch
and
and
using
all
this
information
carpenter
will
be
able
to
talk
to
the
api
server.
A
So
at
this
point
we
are
ready
to
add
some
workload
into
the
cluster,
but
since
we
have
these,
this
node
tainted
and
it's
only
running
system,
critical
components.
No
other
workload
can
be
scheduled
on
this
one
node.
We
will
need
additional
capacity
added
to
the
cluster,
so
let's
go
ahead
and
add
a
deployment
into
the
cluster.
A
A
And
carpenter
detected
that
there
is
one
part
which
is
pending
and
it
requested
a
node
from
ec2
and
it
binded
this
one
part
on
this
particular
node.
So,
let's
give
it
few
seconds-
and
this
is
the
node
which
is
coming
up
and
in
another
few
seconds
we
should
be
able
to
see
the
version
instance
type
zone
and
architecture.
A
A
So
in
about
49
seconds,
we
had
the
container
running
on
this
node,
which
was
requested
by
carpenter,
because
there
was
a
pending
point:
let's
try
to
add
two
more
parts
of
this
in
the
same
deployment
and
see
what
happens
so
once
we
add
two
additional
parts.
What
happened
was
because
this
extra
large
node
had
had
sufficient
capacity.
These
parts
went
into
running
state
and
they
were
never
detected
by
carpenter.
A
A
So
at
this
point
we
have
all
the
seven
parts
running:
let's
also
go
ahead
and
check
how
bin
packing
works
in
carpenter.
So
this
time
I'm
going
to
add
100
parts
in
the
same
deployment,
and
once
these
parts
go
in
pending
state
carpenter
detected
that
there
are
55
pending
parts
and
it
tried
to
bind
all
these
parts
onto
the
same
node
and
then
it
detected.
There
are
35
parts
on
the
second
and
try
to
bind
these
35
parts
on
the
second
node
it
requested.
A
A
A
At
this
point,
we
can
quickly
check
how
many
are
in
the
running
state
out
of
100,
so
we
have
95,
give
it
few
more
seconds
and
see
once
we
reach
100.
So
we
have
100
parts
and
we
can
go
ahead
and
watch
the
parts
again.
So
it
took
about
two
minutes
for
all
these
parts
to
be
created
across
two
nodes,
which
is
which
are
12x,
large
and
24x
large.
A
Next,
let's
see
how
scale
down
works,
so,
let's
say
the
load
has
degrees
and
the
replica
count
has
been
set
to
0.
Now
at
this
point,
all
the
parts
are
being
terminated
across
all
the
four
instances
that
we
just
created
and
once
all
the
parts
are
being
completely
removed,
carpenter
will
detect
that
these
nodes
are
under
utilized
and
carpenter
will
wait
for
10
seconds.
That
was
the
ttl
seconds
we
configured
earlier
and
remove
these
nodes
by
draining
these
nodes
and
then
and
then
terminating
these
nodes.
A
A
So
now
we
are
back
to
where
we
started
with
a
single
node
and
no
workload
at
all.
Let's
say
as
a
use
case.
You
want
to
schedule
some
of
your
parts
on
a
different
architecture
for
a
node.
Let's
say
you
want
to
use
an
arm
64.,
so
let's
repurpose
our
deployment
and
try
to
run
the
parts
on
the
on
in
this
deployment.
A
Now
on
an
arm64,
let's
keep
in
mind,
we
never
created
any
auto
scaling
group
or
a
node
group
for
auto
scaling,
based
on
particular
architecture,
or
instance,
type
and
let's
go
ahead
and
create
two
replicas
for
these
parts
and
they
should
come
up
on
an
arm.
64
type
architecture
node,
so
they
both
went
into
the
pending.
As
expected,
carpenter
detected
two
two
provisionable
parts
which
need
capacity
and
it
requested
a
node
from
ec2
and
binded
two
parts
on
on
this
particular
node,
and
in
this
case
the
architecture
it
requested
was
an
arm
64..