►
From YouTube: Scaling to 5000+ Unique K8s Deployments, How We Did It [I] - Nicole Hubbard, WP Engine
Description
Scaling to 5000+ Unique K8s Deployments, How We Did It [I] - Nicole Hubbard, WP Engine
Most organizations only need to run a couple deployments of their application in Kubernetes. In these situations, deploying onto Kubernetes clusters is relatively straightforward. What happens when you need to simultaneously deploy 5,000 unique instance of your application to different Kubernetes clusters at different providers worldwide?
Over the last year, we have worked to move over 60,000 of our customers' unique workloads from virtual machines onto Kubernetes. I will share our experiences on how to automate and simplify managing unique Kubernetes workloads at scale.
About Nicole Hubbard
Nicole Hubbard is an Architect at WP Engine where she focuses on building container based infrastructure, automation and helping teams deploy their applications.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
Yupi
engine
so
WP
engine
today
we're
undergoing
a
shift
in
our
infrastructure
working
to
migrate
from
VMs
to
kubernetes.
We
currently
run
over
6000
VMs,
which
we
use
to
power
over
500,000
domains,
which
are
on
300,000
WordPress
installations.
We
estimate
that
5%
of
the
online
world
visits
at
least
one
site
we
host
every
single
day.
A
So
for
those
of
you
that
aren't
familiar
with
WordPress,
it
was
open
sourced
in
2003
and
founded.
Then
it's
traditionally
ran
on
a
lamp
sack,
and
so
you
normally
run
that
on
a
VM
or
with
a
VPS
provider,
your
classic
use
case
with
that
is
generally
a
pretty
low
utilization
website.
So
you
have
a
lot
of
customers
that
run
their
personal
blogs.
Things
like
websites
for
law
firms,
engineering
firms
and
apartment
complexes,
and
so
forth
that
run
on
WordPress.
But
then
you
also
have
customers
who
run
extremely
high
performance
sites.
A
So
some
large
instances
of
WordPress
that
are
out
there
today
are
like
the
CMAs
for
the
Country
Music
Awards.
The
TechCrunch
web
site
runs
on
WordPress,
as
does
the
New
Yorker.
So
with
WP
engine
we
allow
our
customers
to
install
any
plugins.
They
want
in
their
WordPress
installation,
and
they
can
add
any
custom
code
that
they
have
so
this
creates
some
of
our
difficulties
and
with
WordPress,
all
of
your
workers
require
access
to
the
same
file
system
so
that
if
something's
changed,
they
all
see
the
exact
same
file
system.
A
So
what
we're
trying
to
do
is
we
have
today
our
virtual
machines,
which
have
engine
acts
a
caching
layer,
PHP,
and
then
we
have
our
storage
and
our
databases
and
they
all
live
in
that
VM.
So
what
we're
trying
to
do-
and
one
of
the
problems
we
face
with
this-
is
the
domains
we
host.
They
live
on
that
single
VM.
So
if
that
instance
goes
offline,
their
entire
site
is
offline.
So
the
first
thing
we're
trying
to
do
is
move
the
PHP
piece
of
this
out
of
the
VM
and
into
the
kubernetes
cluster.
A
This
will
allow
us
to
eventually
move
all
of
these
pieces
off
of
the
single
VM
and
allow
us
to
provide
a
more
reliable
infrastructure
for
our
customers,
but
we're
doing
it
one
piece
at
a
time
as
to
keep
the
process
as
simple
as
we
can.
So
we
first
started.
Looking
into
how
we
were
going
to
do
this,
and
our
first
thought
was
we'll
just
give
every
installation
that
we
host
their
own
PHP
fpm
instance
in
kubernetes,
will
deploy
those
will.
Let
kubernetes
manage
all
of
this.
A
For
us
everything
will
go
great,
we'll
just
kubernetes
all
the
things
it
will
do
everything
for
us.
We
can
even
do
multiples,
we
can
do
replica
sets
and
everything
will
be
great
yeah.
It
turns
out.
Not
so
much
remember
has
said
we
have
three
hundred
thousand
installations
that
we
have
to
power.
So
kubernetes
has
some
limits.
Those
limits
in
1/8
are
150,000
pods
per
cluster.
We
have
300,000
installations
and
we
want
to
run
multiple
instances
of
those.
So
if
one
fails,
we
still
have
reliability.
A
So
that's
way
more
than
150,000,
not
to
mention
you
can
only
have
a
hundred
pods
per
node
websites,
especially
WordPress
sites.
Their
utilization
is
extremely
volatile,
so
one
minute
it
will
be
doing
nothing
then
it
might
for
the
next
10
minutes,
use
all
the
CPU
and
memory
as
and
then
do
nothing
for
the
rest
of
the
day.
A
That
creates
a
problem
when
you're
trying
to
schedule,
and
you
can
only
put
a
hundred
pods
per
node,
so
we
needed
something
that
allowed
us
to
go
more
dense
than
that
which
led
us
to
go
back
to
the
drawing
board.
So,
with
these
limits
in
mind,
we
took
our
domain
expertise,
which
is
around
serving
WordPress
and
built
our
own
PHP
application
server.
A
So
what
this
does
is
the
solution
we
found
was
to
develop
a
custom
server
written
and
go
which
runs
PHP
fpm
workers.
We
then
mount
the
site's
content
into
an
isolated
worker
gel
for
every
request
as
they
come
in,
so
this
is
very
similar
to
what
you
get
with
a
VM
or
any
other
PHP
fpm
process,
but
we
in
real-time
for
each
request
as
they
come
in.
A
A
A
What
happens
is
that
application
server
looks
at
its
available
workers
that
are
free
which
are
all
running
inside
their
own
namespace
to
isolate,
and
then
it
will
mount
the
files
for
that
specific
request
in
and
send
the
request
to
the
worker
once
the
worker
finishes
processing
that
request.
That
will
then
send
the
response
back
out.
We
unmount
the
files
and
the
response
goes
back
to
the
VM
and
back
out
to
the
customers
that
made
the
request,
and
then
we
make
the
worker
once
everything
has
been
cleaned
up
out
of
that
worker.
A
So
this
has
brought
us
down
from
300,000
deployments
to
only
6,000
now
so
now,
how
are
we
gonna
manage
6000
deployments?
It
sounds
a
lot
more
manageable,
at
least
so
we
like
to
use
helm
for
all
of
our
internal
deployments
and
so
helm
requires
three
pieces
of
information
to
do
a
deploy.
You
first
have
to
have
the
chart,
which
is
all
of
your
templates,
that
you're
going
to
deploy
for
your
application,
and
then
you
have
the
values,
and
these
are
the
things
that
are
going
to
be
different
between
the
different
deployments
of
this
application.
A
So
these
could
be
database
connection.
These
could
be
passwords.
Anything
like
that.
That
happens
to
vary
between
your
deployments,
and
then
you
have
what
helm
calls
a
tiller
and
the
tiller
is
something
that
runs
inside
of
your
kubernetes
cluster
and
it's
responsible
for
actually
deploying
and
creating
the
kubernetes
objects
that
you've
asked
helm
to
create.
So
the
local
helm
command
will
actually
pass
everything
to
the
tiller
and
it
will
then
process
it
and
create
that
for
you,
inside
of
your
cluster
and
we'll
see
some
examples
of
how
that
works
in
a
little
bit.
A
So
now
we
only
have
6,000
workers
we
have
to
manage
and
our
initial
we
like
to
use
helm,
but
we
have
6,000
unique
value
files
that
we
have
to
manage
for
helm.
So
this
gets
to
be
really
hard
to
maintain.
We
also
have
to
remember
which
of
our
seven
regional
clusters.
We
have
to
deploy
each
of
those
values,
files
to
so
6000,
spread
across
seven
clusters.
What,
if
we
have
to
add
a
new
fill
to
that
file,
sounds
like
a
lot
of
work
if
we're
actually
having
to
manage
these
files.
A
For
us,
though,
we
needed
a
way
to
make
all
of
the
existing
data
we
had
about
our
VMs
available
to
that
operator
in
the
kubernetes
cluster.
So
to
solve
this,
we
built
a
tool
that
watches
our
cloud
providers
api's
and
creates
a
custom
resource
inside
of
our
kubernetes
cluster.
That
represents
the
VMS
that
are
in
the
same
region
as
that
kubernetes
cluster.
A
So
for
those
of
you
that
aren't
familiar
with
custom
resources,
they
basically
allow
you
to
extend
the
kubernetes
api
I
personally
like
to
think
about
them
as
a
way
to
put
whatever
data
I
care
about
into
the
kubernetes
cluster
and
work
with
them
just
the
same
way.
I
would
any
other
objects
in
you
can
store
any
data
you
want
in
the
meta
fields
of
a
custom
resource
and
they
were
released
in
kubernetes
1.7
and
they
replaced
the
now
deprecated
third-party
resources.
A
So
here
we
have
an
example
of
a
custom
resource
definition
so
to
be
able
to
create
custom
resources,
we
first
have
to
create
a
definition.
So
this
looks
like
if
you're
familiar
with
kubernetes
objects,
the
slips
very
familiar,
so
we've
got
our
API
version
and
our
kind
defined
and
then
for
our
name.
This
is
a
very
specific
name,
that's
required.
It
has
to
be
the
plural
name
of
the
objects,
you're
creating
and
then
the
group
that
this
is
going
to
go
in
so
then
down
in
our
spec.
A
We
have
our
scope,
and
so
this
one's
currently
scoped
to
a
namespace,
and
so
you
can
have
two
different
scopes
named
spaced
or
clusters.
Most
objects
in
kubernetes
are
named
spaced,
so
they're
only
available
in
the
same
namespace
you're
in,
but
you
can
also
create
cluster
available
custom
resources
so
for
our
VM
custom
resources,
those
are
actually
available,
no
matter
what
namespace
you're
in
for
the
entire
cluster,
and
then
you
define
your
group.
A
We
defined
our
kind,
which
is
character
and
the
plural
and
the
singular
of
that
which
are
characters
in
character
and
then
we're
able
to
create
a
actual
custom
resource
so
because
we
deployed
that
custom
resource
definition
in
we're
now
able
to
use
API
version
stable
that
nicole
renee
io,
/,
v1
and
kind
of
character.
This
allows
it
and
then
just
like
any
object
in
communities,
it's
required
to
have
a
metadata
name,
so
we
called
it
thing
one
and
then
in
spec
we're
able
to
add
any
fields
we
want.
A
So
we
created
thing
one
from
cat
and
the
hat
and
dr.seuss
created
thing
one
so
have
a
little
demo
here,
showing
the
use
of
custom
resources
and
how
you
can
interact
with
them.
So
if
you
see
this
is
our
custom
resource
definition
that
I
was
just
showing
a
minute
ago
and
it's
the
characters,
that's
table
got
Nicole
Rene,
and
so
we
can
just
cube
control
apply
this
just
like
we
could
any
other
kubernetes
file
and
we
can
see
a
custom
resource
definition
for
characters
was
created.
A
A
And
we
can
see
that
without
doing
anything
for
cute
control.
It
automatically
knew
how
to
tell
us
that
it
created
a
character.
It
didn't
tell
us
a
custom
resource
was
created,
so
this
character
thing
one
and
character
thing
two
were
created,
then
we're
just
gonna
go
ahead
and
create
one
more
that
we'll
call
Nemo
and
character.
Nemo
is
created,
and
so
we
can
work
with
these
objects.
Just
like
anything
else.
A
So
we
can
actually
do
a
kubernetes
get
characters
and
this
works
for
everyone
who
has
access
to
the
cluster
without
any
client-side
configuration,
and
you
can
see
all
three
characters
that
we
created
and
then
you
can
also
describe
these
just
like
all
other
objects.
So
we
can
actually
take
a
look
here
at
Nemo
and
we
can
see
all
the
fields
we
specified
that
it's
by
Disney
from
Finding
Nemo
and
the
names
Nemo,
but
for
those
of
you
familiar
with
Disney
characters,
you'll
know
that
Nemo
is
technically
a
Pixar
character.
A
A
And
we're
able
to
see
that
it's
now
by
Pixar,
so
one
of
the
interesting
cases
with
custom
resources
is,
if
you
name
your
custom
resource
the
same
thing
as
a
built
in
kubernetes
object:
you'll
have
to
access
it
using
the
full
path
name,
including
the
DNS
piece.
So
in
this
case
we'll
cube
control,
get
characters,
dot,
stable,
dot,
Nicole
Renee
IO,
and
it
will
print
those
for
us.
So
if
you
decide
to
name
yours
something
like
deployment
or
along
those
lines,
then
you
have
that
problem.
A
So
now
that
we've
got
custom
resources
for
all
of
our
VMs,
we
built
an
operator
that
we've
made
available
open
source
on
github
called
list
ramos,
and
what
Louis
Ramos
does
is
it
watches
custom
resources
in
Corona
DS
and
then
will
deploy
a
home
chart
for
you
passing
in
the
values
that
are
defined
in
the
custom
resource
to
the
helm
chart.
So
we
run
a
version
internally
that
monitors
our
VMs
and
we'll
deploy
our
PHP
worker
for
every
VM.
A
We
have
automatically
when
the
new
VM
gets
created
with
Google
or
Amazon,
the
custom
resource
gets
provisioned
and
then
the
PHP
worker
gets
automatically
deployed.
No
manual
intervention
is
required,
so
we've
got
a
little.
You
know
demo
here
showing
some
of
this.
We
create
Nemo
and
that
creates
the
character.
Nostromo's
gets
the
notification
that
it
was
created,
which
then
will
call
out
to
tiller
and
create
anything.
That's
in
our
helm
chart
and
the
same
thing
works
for
deletions.
A
A
So
what
we
have
first
is
a
really
simple
demo:
application
written
and
go
all
it's
going
to
do
is
print
out
that
the
name
first
appears
in
where
they're
from
and
who
created
it.
So
by
and
it's
using
environment
variables
to
get
those
so
we're
going
to
use
resource
name,
resource
from
and
resource
by,
so
that
we
can
pass
those
in.
A
So
then,
let's
take
a
look
at
our
sample
chart
that
will
actually
deploy
our
deployment,
and
so
most
of
this
is
pretty
stock.
But
down
at
the
very
bottom,
we
pass
in
the
environment,
variables
and
right
here
the
part
that
matters
is
if
you're
not
familiar
with
helm.
You
have
values
which
is
all
of
your
values,
but
we've
created
values
not
resource
by
passing
in
all
the
fields
of
the
resource.
So
then
we
can
call
spec
name
SPECT
up
from
spec
top
by
you
can
nest
these.
They
can
be
arrays,
they
can
be
maps.
A
A
A
A
A
So
now
the
nice
thing
is
less.
Your
mouse
is
still
running
up
there
at
the
top.
And
so,
if
you
watch
you
can
see
here
that
we
can
go
down
at
the
bottom
and
we
can
apply
to
create
Nemo
again
and
you'll,
see
here
that
when
this
runs
up
at
the
top,
let's
we'll
clear
that
so
you
can
see.
But
as
soon
as
we
submit
this
lush
most
will
notice.
Oh
yeah
I
hope
if
you
type
the
file
name
right.
A
A
And
then
we'll
change
by
will
use
Nicole
and
from
will
set
to
cube
con,
and
so,
if
you
watch
as
we
save
this
file
up
at
the
top
you'll
see,
it
immediately
got
the
update
unless
you're
us
and
redeployed
the
helm
chart.
And
so
now.
If
we
come
down
here
at
the
bottom
and
we
curl
thing
a
which
used
to
be
thing
one,
we
can
actually
see
the
new
response,
which
is
thing
a
first
appeared
in
Cuba
and
created
by
Nicola,
and
so
this
works
and
allows
us
to
also
handle
deletions
of
custom
resources.
A
A
A
We
knew
it
couldn't
be
that
slow
had
to
be
faster,
so
we
shot
for
a
target
of
within
an
hour
to
be
able
to
deploy
to
all
of
these,
we
found
a
few
bugs
that
were
working
through
I'm
working
with
the
helm
community
on
around
concurrent
deployments
with
helm.
So
when
you
pass
the
tiller
concurrent
deployments,
it
will
actually
sometimes
mix
the
values
up
between
deployments,
so
we're
working
on
figuring
out
how
to
resolve
that.
I
actually
had
some
good
discussions
about
that
this
week.
A
Another
thing
we
wanted
the
ability
to
do
phase
deploys
so
that
we
could
do
Canaries
and
different
stages
of
deploy.
Instead
of
hitting
all
of
these
at
once
so
we've
added
the
ability
to
create
lower
most
instances
that
filter
and
only
watch
certain
custom
resources.
So
you
can
apply
an
annotation
to
your
custom
resource
and
that
can
exclude
it
from
lush
promise,
and
so
another
piece
was
monitoring.
How
do
we
monitor
all
of
this?
So
low
stress
itself
has
a
Prometheus
metrics
endpoint
that
exposes
all
of
the
data
about
what
it's
doing.
A
So
it
includes
the
number
of
added
resources,
the
number
of
deleted
resources,
updated
resources
and
the
number
of
ones
that
have
felt.
It
also
includes
a
total
number
of
resources
that
are
being
managed
by
listeners
so
that
you
can
tell
if
you
know
you
have
300
custom
resources
and
less
remote
says
it's
only
managing
298,
you
know
have
a
problem
and
then
there
was
reconciliation.
A
What
happens
if
we
miss
an
event,
because
los
números
is
offline,
say
we're
deploying
a
new
version
and
a
custom
resource
gets
recreated
or
changed
or
created
for
that
matter,
and
then
what
happens?
If
someone
changes
something?
So
what
happens?
Is
lush
mas
has
an
option
to
do
a
resync
and
you
can
specify
any
duration
of
time
and
at
that
duration
it
will
resync
every
custom
resource
and
redeploy
the
helm
chart
to
ensure
that
it's
the
correct
version
and
undo
any
changes
that
have
manually
been
made
in
your
cluster.
A
We
also
at
startup,
read
in
every
custom
resource
that
exists
and
update
the
state
in
the
tiller
to
match
the
state
that
it
should
be
so
some
other
use
cases.
Besides
our
simple
VM
use
case
is,
you
could
create
databases
for
your
application,
create
custom
resource
called
database
and
every
time
you
needed
a
database
for
your
application.
You
could
just
create
a
database
custom
resource
and
that
would
deploy
your
database
for
you
using
the
home
chart.
A
So
if
you
wanted
to
actually
create
the
cloud
sequel
database
or
RDS
instances,
you
could
just
create
those
using
kubernetes
objects
and
have
them
created
with
your
cloud
provider
so
summer.
It
is
we
have
for
where
we
want
to
go
with.
Lesser
mos
includes
supporting
watching
resources
beyond
custom
resources,
so
for
every
deployment
we
have,
we
want
to
be
able
to
also
deploy
this
other
thing
or
anything
along
those
lines
or
for
every
service
in
our
cluster.
We
want
to
and
then
support
for
additional
deployment
mechanisms.
A
A
So
that's
a
good
question.
Question
was:
if
we
allow
our
customers
to
create
any
files,
they
want
what
stops
them
from
creating
a
PHP
info
file
and
dumping
the
environment
variables
that
could
include
database
passwords.
So
in
our
case,
we
actually
limit
the
environment
variables
that
we
pass
in
to
our
customers
to
remove
any
that
we
use
on
our
side,
which
aren't
very
many,
and
then
we
also
in
this
case
the
customer
would
only
be
leaking
their
database
password,
which
luckily
we
don't
actually
pass
in
passwords
for
those
such
as
the
database
via
environment
variables.
A
A
A
A
A
So
you
can
pass
in
the
chart,
so
it
if
you
looked
at
that
config
file
for
list
that
was
actually
a
path.
I
was
running
it
locally
on
my
machine
normally.
So
in
our
case,
we
actually
deploy
it
as
a
config
map.
So
we
add
the
chart
as
a
config
map
and
then
mount
that
as
a
volume
in
our
list
for
most
deployment
and
if
you
were
using
a
public
chart.
Unfortunately,
that
doesn't
work
so
you'd
have
to
pull
it
down
and
push
it
up.
B
A
Yeah,
so
so,
if
we
look
here
at
our
entire
stack
for
a
VM
wow,
that
was
all
the
way
back
here.
All
of
those
things.
Potentially,
we
haven't
done
all
the
research
research
on
moving
the
database
and
the
storage
in,
but
we
know
after
we
successfully
complete
moving
all
of
the
PHP
n.
We
do
want
to
start
on
the
engine,
X
and
caching
layers
and
then
once
that's
complete,
we
want
to
start
researching
what
we
can
do
for
storage
and
database.