►
From YouTube: Vertically Scaling Kubernetes Clusters
Description
At Intuit, we’ve been building our Kubernetes Platform for several years now. We are running our kubernetes clusters on Amazon EKS. As more services are onboarded into Kubernetes clusters we realized we were running out of Ingress subnets as we provision Alb Ingress for every service. As we were reaching thresholds on the Ingres subnets instead of creating new clusters we thought of scaling the clusters vertically by expanding VPC's. This process of scaling Kubernetes clusters vertically by expanding VPC's should be done seamlessly so that we do not run into any issues with services already running on them.
We adopted a gitops framework where we update the VPC's using a configuration file that gets translated to cloudformation stacks and then gets executed on the target account where we want to expand VPC. Once VPC is expanded then we use a Gitops framework to update the kubernetes cluster by passing in the expansion name so that the cluster can then consume the new subnets created and assign the subnets to ALB's that have least used subnets this helped us reduce creation of additional kubernetes clusters and hence the overhead.
A
A
So
the
fundamental
problem
statement
we
have
is
that
for
every
service
we
have
a
load
balancer
associated
with
it
and
in
our
kubernetes
clusters
that
we
have.
We
are
reaching
a
limit
on
our
ingredient
subnets
in
this
cluster.
As
you
can
see,
we
have
hit
a
limit
of
about
eighty
percent
already
and
when
we
scanned
and
looked
into
all
the
other
clusters
we
have
seen
a
couple
of
clusters
are
on
the
words
of
keeping
this
limit,
and
some
of
them
are
a
little
away
from
between
this.
A
So
there
are
multiple
options
for
us
to
choose:
one
we
could
scale
horizontally,
which
is
like
creating
new
classes
all
the
time,
but
we
need
to
pay
for
all
the
cloud
resources
such
as
spac
volumes,
vpcs
knackers,
and
we
have
to
create
new
pairings
or
transit
gateways
and
then
the
pcm
plants
etc.
Right,
and
then
we
see
that
only
ingress
subnets
are
getting
exhausted
and
we
have
a
lot
of
room
in
other
segments.
So
we
thought,
if
it's
only
for
expanding
english
subnets,
it's
better.
We
scale
vertically,
rather
than
doing
horizontal.
A
A
A
Now
the
challenges
with
different
types
of
load
balancers
are
once
an
nlb
is
created.
We
cannot
change
the
subnets
on
them.
Aws
doesn't
support
that,
and
the
other
challenge
is
enb
is
created
by
the
controller
running
in
the
control
plane
right,
because
there
are
some
controllers
that
the
cloud
providers
run
in
the
control
plane
to
create
these
resources
and
elv
is
one
such
resource.
A
The
thing
that
we
notice
peculiar
with
elb
is
that,
even
though
we
can
update
the
subnets
from
the
console
when
we
try
to
update
that
from
kubernetes
object,
the
english
subnets
that
we
wanted
to
change
are
not
reflected
on
database
console.
So
primarily,
the
challenge
here
is
that
from
kubernetes
object
we
cannot
update
here
the
subnets
on
the
inside.
A
Even
though
we
can
update
the
subnets,
we
have
to
ensure
nothing
breaks
right.
So,
even
though
change
is
submitted
from
one
old
subnet
to
probably
in
your
subnet
that
we
want
to
create,
we
were
skeptical
if
that
would
pose
any
challenges
like
services
might
see
some
errors
and
stuff
like
that,
so
we
want.
We
don't
want
to
run
into
those
problems.
So
these
are
some
challenges
that
we
face
with
different
types
of
load
balances.
A
A
If
you
pass
multiple
of
them,
it
will
choose
only
one,
so
we
have
to
ensure
the
one
that
we
pass
has
sufficient
eyepiece
now,
in
order
to
solve
this,
we
have
done
something
called
vpc
expansion
and
we
followed
the
process
of
githubs
so
interior.
We
manage
all
the
kubernetes
based
aws
resources,
where
the
githubs
to
manage
the
cloud
resource
lifecycle.
A
A
If
you
have
chosen
ingress
subnets,
it
will
create
this
additional
english
subnets,
but
we
can
also
choose
other
subnets,
but
our
primary
problem
here
is
on
the
ingredient
subnet.
So
so
here
we
only
solve
for
image
sequence,
so
this
is
in
a
dsl
custom
dsl
that
we
write
later
it
gets
translated
into
custom
resources.
A
So
with
this,
we
follow
this.
The
githubs
pattern,
where,
where
we
use
arco
cd
to
read
the
changes
from
the
github
and
targo
cd
once
reads
these
changes,
it
will
pass
it
on
into
the
controller
that
runs
in
a
centralized
cluster
that
has
access
to
other
clusters
or
aws
accounts,
and
it
can
manage
those
resources.
A
So
once
we
pass
in
this
change,
it
will
read
the
desired
state
and
it
will
check
the
current
scheme
and
it
will
bring
the
current
state
to
the
design
state.
So
by
running
this
reconciliation
loop
and
the
logic
that
is
passed
into
it
right,
so
it
will
essentially
create
this
new
vpc
expansion
for
us
when
we
passing
the
aws
change,
we
also
use
gitops
process
for
our
cluster
creations
and
cluster
updates.
A
So
once
this
cluster
is
already
available,
so
we
need
to
let
the
cluster
know
that
there
are
new
subnets
that
are
available
and
whenever
we
have
created
this
new
ingress
subnets,
we
pass
and
we
inject
a
name
into
this
subnets
right,
so
that
name
contains
exp1.
If
it's
the
first
expansion,
if
it's
a
second
expansion,
it
will
continue.
Exp2
instance
like
that,
so
in
our
cluster
conflict
that
we
use,
we
have
written
this
block
such
that
we
can
pass
in
the
short
code
or
to
identify
the
subnets.
A
So
it
it
is
an
array.
So
it
need
not
be
one
expansion
it
can.
If
you
have
more
than
one
expansion,
also
it
can
come
somewhere.
So
we
pass
this
expansion
name
in
the
cluster
country
and
it
again
follows
the
gitops
model
and
it
you
know,
runs
the
cluster
as
the
cluster
updates
are
run.
What
happens?
Is
it
detects?
The
new
subnets
and
cluster
understands
that
there
are
new
subnets
available
right.
A
So,
with
the
github's
process,
we
are
able
to
achieve
vpc
expansion
and,
with
the
tops
clustered
updates,
we
were
able
to
let
the
cluster
know
that
there
are
new
subnets
that
are
available
now.
The
challenges
after
expanding
the
vpcs
is
that
we
have
to
ensure
the
cluster
upgrades
that
we
run
goes
through
smoke
and
nourishes
right
and
the
other
thing
is.
A
We
should
also
ensure
the
the
subnets
that
that
we've
created
earlier
can
be
reused
right
or
we
need
to
ensure
that
we
don't
waste
resources
just
because
we
have
expanded
our
subjects.
So
that's
another
thing
and
then,
whenever
the
existing
services
try
to
change
something
on
the
english
objects,
we
shouldn't
suddenly
start
injecting
the
new
subnets,
because
that
could
cause
some
failures
and
customers
might
see
rs
when
the
traffic
is
right,
for
example,
they
might
want
to
change
just
the
health
check,
endpoint
or
the
port
or
other
things
right.
A
For
those
reasons,
if
you
start
making
changes
on
the
subnets
on
which
this
ingress
runs,
there
could
be
a
potential
chance
of
error
right.
So
these
are
some
challenges
that
we
want.
You
know
that
we
foresee
and
we
need
to
solve
for
and
the
solution
that
we
came
up
with
is
once
the
cluster
starts,
understanding
that
there
are
new
subnets.
A
We
will
let
the
cluster
know
what
are
the
least
used
subnets
when
I
say
least
few
subnets.
It
is
the
subnets
that
have
more
number
of
ips
available
right.
So
these
least
use
subnets
are
are
put
into
a
map
and
the
cluster
is
made
aware
of
the
existing
subnets
and
the
least
used
subnets
right.
So
when,
when
the
cluster
knows
that
these
use
subnets
the
advantage,
then
that
we
get
is
we
can
inject
these
subnets
whenever
a
new
ingress
objects
are
created,
because
we
do
not
want
to
change
the.
A
Subnets
injection,
we
don't
want
to
inject
the
subnets
at
the
time
of
updates.
We
only
want
to
enjoy
at
the
time
of
creation.
With
that
what
happens
is
will
not
disrupt
anything
that
is
running
successfully,
but
when
we
create
new
objects,
these
english
subnets
are
used.
So
we
can
optimally
used,
use
the
newly
created
english
subnets,
and
we
run
this
cluster
upgrades
frequently
because
of
our
code
changes
or
some
new
features
that
we
bring
in
and
stuff
like
that.
A
So
every
once
in
a
while
like
every
week
or
every
two
weeks,
we
whenever
we
run
the
cluster
updates
at
that
point
in
time,
whichever
are
the
least
using
the
subnets,
those
are
populated
in
the
cluster
and
those
are
the
subnets
that
would
be
injected
into
the
english
analysis
and
how
we
inject
this
english
annotations
is
using
open
policies.
We
use
opera
mutation
webpro
so
at
the
time
of
creation,
oppa
checks.
A
If
this
is
a
creation
of
the
ingress
object,
either
it
is
alp
or
it
is
nlp
or
it
is
clb,
and
it
would
correspondingly
inject
the
appropriate
annotation
with
the
least
used
set
of
subnets.
So
let
me
show
you
how
this
work,
so
this
is,
on
the
left
hand,
side.
You
see
one
cluster,
where
the
expansion
is
not
done.
So
if
I
go
ahead
and
create
an
inverse
object,.
A
So
I'm
just
filtering
the
annotation,
because
that
is
what
we
are
interested
in.
If
you
see
in
this
one,
because
it
is
not
expanded,
it
uses
the
regular
subnets
that
we
that
we
have
created
at
the
time
of
initial
cluster
when
we
created
the
cluster
now,
on
the
right
hand,
side
there
is
a
cluster
which
had
run
out
of
ips
the
threshold
data
admin
and
we
have
expanded
the
cluster.
Now,
if
I
go
ahead
and
run
this
creation
of
new
service
in
that
expanded.
A
A
So
this
is
how
we
achieve
optimal
use
of
ingress
subnets,
as
well
as
we
could
expand
the
clusters
and
vertically
scale
the
kubernetes
clusters
to
solve
the
problem
that
we
have
now
coming
to
the
results
as
we
expanded.
These
clusters
solved
the
problem.
With
the
first
listed
and
unique
month.
We
have
seen
a
couple
of
more
additional
clusters
and
into
the
same
problem,
but
we
have
the
solution
ready
now
and
we
were
able
to
solve
for
it.
A
So
this
is
how
we
solved
it,
and
you
know
it
is
we
also
experimented
with
other
subnets,
but
which
is
not
of
priority
for
now,
like
private
subnet,
expansions
and
stuff,
like
that,
we
we
just
played
around
with
it,
and
we
saw
that
also
working,
but
we
didn't
have
to
use
it.
So
we
didn't
explore
all
the
potential
challenges
and
stuff
like
that,
but
definitely
for
english
supplements.