►
From YouTube: How Atlassian Built Our Own Kube Clusters and Why You Shouldn’t Do the Same - Nick Young, Atlassian
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
How Atlassian Built Our Own Kube Clusters and Why You Shouldn’t Do the Same - Nick Young, Atlassian
Three years ago, deciding between building or buying k8s infrastructure was simple. Few vendors existed and none that could support Atlassian's self-imposed requirements as we sought to replace our hand-rolled PaaS. Fast forward to today and our internal Kubernetes now run both batch and stateless service workloads. If we repeated our build vs buy decision, the outcome is no longer clear. Come along to hear about how we designed and built our Kubernetes clusters to run on our cloud platform and why we think you’re probably better off starting on a managed service today.
To learn more: https://sched.co/GrS7
A
So
hi
everyone
just
to
make
sure
you're
in
the
right
place.
You're
here
for
do
as
I
say
not
as
I
do
how
we
build
our
cute
clusters
and
why
you
shouldn't
I,
am
Nikki,
oh
I,
to
introduce
myself.
I
have
been
a
sysadmin
for
20
years
now,
I've
been
on
a
lysines,
cube
team
or
then
our
last
in
for
five
years
and
on
a
licensed,
cube
team.
Since
it
started
three
years
ago
and
yeah
we've
been,
we've
built
a
whole
bunch
of
stuff
in
that
time.
So
it's
some
important
things
as
we
get
started.
A
The
name
of
our
team,
which
will
be
relevant,
is
the
kubernetes
infrastructure
technology
team.
You
may
notice
there's
a
redundant
T
there.
It
solely
exists
so
that
we
can
make
Nightrider
pants
for
all
very
pose
and
stuff.
Yeah
I
firmly
believe
that
whimsy
is
important
in
your
name
is
games
so
what
I'd
like
to
you
to
walk
away
with
today?
Firstly,
what
we've
been
trying
for
we're
doing
all
around
cube
clusters,
what
we
actually
did,
how
it
went
and,
lastly,
and
the
meat
of
the
thing
why
you
shouldn't
do
the
same
as
us?
A
Okay,
so
what
we
were
trying
for
here
is
what
we
were
trying
for
from
the
start.
We
wanted
to
build
a
set
of
clusters
that
invention
could
run
95
percent
or
more
of
compute
inside
of
us
in
so
you
know
the
numbers,
the
numbers,
not
a
hundred
percent,
but
it's
pretty
close,
and
so
that's
the
key
thing
here.
We
wanted
to
make
make
the
whole
thing
fit
as
many
use
cases
as
possible
inside
a
Lyceum.
A
Why
did
we
build
our
own?
Firstly,
the
tools
weren't
ready
three
years
ago,
the
you
was
just
the
stand
me
up.
A
cluster
tooling
was
very
incomplete
to
build
our
clusters.
We
literally
started
from
kubernetes
the
hard
way
and
you
know
built
from
there.
So
our
clusters
are,
you
know
it's
close
to
artisinal,
as
you
can
get
I
would
say.
A
Secondly,
compliance.
We
knew
we
needed
a
compliance
regime.
We
know
it's
not
that
long
ago
that
we
up
you'd
know
the
company
we've
got
sauce
controls
the
same
as
everybody.
We
wanted
to
make
sure
that
we
could
meet
Alison's
internal
compliance
regime,
which
has
a
couple
of
nifty
things,
but
we'll
talk
about.
Is
that
that
is
for
another
day?
Lastly,
if
just
everything
was
still
pretty
new
at
the
time,
and
so
we
wanted
to
make
sure
that
we
knew
the
system
as
well
as
possible,
and
that
was
why
we
started
for
all.
A
That
was
why
we
started
from
basically
cube.
The
hard
way
was
just
to
make
sure
that
we
knew
this
whole
platform.
You
know
every
which
way
upside
down
back
to
front,
so
what
we
actually
did
well
myself
and
one
other
guy
were
the
ones
who
actually
did
the
initial
design
and
so
I
started
from
my
rule,
zero
design
out
the
biggest
problems
you
know
about,
so
you
can
find
all
the
new
and
interesting
ones
later
the
this
certainly
applied
we
picked.
A
We
picked
some
problems
that
we
knew
would
be
a
problem
based
on
our
experience
running
the
on-demand
platform.
A
lot
of
our
team
came
from
running
the
lysine
on-demand
platform,
which
used
to
run
hosted
confluence
at
hosted
JIRA.
It
was
a
pretty
big
platform
that
used
an
early
form
of
containerization
in
openvz,
and
so
we
had
experience
at
running
quite
a
big
fleet
of
stuff,
and
we
wanted
to
make
sure
that
we
took
as
many
of
the
learnings
that
we
had
from
that
as
possible
and
built
them
into
this
new
platform.
A
So
the
problems
we
decided
to
solve
were
firstly
to
manage
blast
radius.
This
is
the
number
one
thing
that
we
had
learned
running
a
big
platform
earlier,
that
you
really
needed
to
make
sure
that
when
you
make
changes,
you
can
make
changes
to
some
subset
of
your
stuff.
If
you're,
if
you're,
managing
a
full
stack
like
we
are,
then
the
having
that
subset
be
manageable
is
super
important
and
it
lives,
and
it
frees
you
up
keeping
it
orthogonal
frees
you
up
from.
Actually
you
know
stepping
on
each
other's
toes
when
you're
working
together.
A
We
wanted
to
really
manage
dependencies.
We
knew
that
eventually
95%
of
computing
would
be
running
on
us,
so
it
was
really
important
to
us
that
we
didn't
have
anything.
We
didn't
depend
on
anything
run
by
a
lassie,
and
that
would
end
up
running
on
us,
because
cyclic
dependencies
are
really
hard.
They
give
you
really
difficult
bootstrap
problems,
so
we
tried
our
best
to
keep
them
cut
out
the
whole
time.
A
Lastly,
cattle
no
pets
we
want
to.
We
embraced
immutable
infrastructure
as
much
as
we
could.
This
I
mean
don't
want
to
preamp
too
much,
but
this
has
been
a
real
saving
grace
for
us,
so
the
layer
cake
so
the
first
layer
we
called
the
flag.
That
is
a
kit
that
is
a
Knight
Rider
acronym
in
Knight
Rider.
It
was
the
foundation
for
law
and
government
who
funded
the
whole
cool
truck
and
everything.
A
So
the
important
part
there
is
that
that's
the
base,
AWS
config,
so
it
has
like
VP-
sees
the
relationships
to
vgw
security
groups,
stuff
that
we
need
to
be
there
to
build
the
next
layer,
but
that
we
don't
anticipate
changing.
Very
often,
you
know
changing
this.
Changing
this
bottom
layer
is
a
bit
of
a
big
deal.
It's
a
bit
riskier
than
the
other
ones,
because
you
know
it
affects
everything
all
the
way
up.
The
stack
next
up
was
the
car
one.
A
If
you
go
watch
and
I
rode
I
shouldn't
need
to
explain
that
one
to
you,
so
this
one
has
all
the
compute
the
control
plane,
the
eightydaze.
Basically
all
of
the
ec2
instances
and
associated
shenanigans
is
all
in
this
layer.
The
important
defining
part
of
this
layer
is
that
it
stands
up
an
API
server
in
point.
Nothing
else
that
API
server
is
a
completely
unconsidered
vanilla,
api
server,
because
we
have
all
the
are
back.
Things
turned
on.
A
That
means
that
when
you
first
stand
up
one
of
these
clusters,
this
API
server
can
do
nothing,
not
as
if
you
can't
even
get
nodes,
because
the
nodes
haven't
even
checked
in
yet
because
they're
not
allowed
to,
because
the
arbok
hasn't
been
done.
That
is
the
responsibility
of
the
Goliath
layer.
'life
was
the
truck,
so
the
this
contains
all
config
that
actually
runs
inside
cube.
A
So
that
includes
our
back
the
PSP's,
the
initial
cube
network,
config
all
of
the
sort
of
a
basic
stuff
that
paints
a
cluster
and
makes
it
a
cluster
that
we
are
prepared
to
manage
is
in
here
plus
all
of
the
load
all
of
the
system,
services
that
we
provide
out
to
our
users.
Because,
again,
the
point
of
this
platform
is
to
be.
A
You
know,
a
platform
that
other
Atlassian
users
can
use
to
deal,
what
they're
going
to
do,
and
so
we're
trying
to
be
as
opinionated
as
we
can
up
to
our
demarcation
point,
which
is
this,
which
is
this
Goliath
layer.
And
on
top
of
that
we
try
to
be
relatively
unup,
Union
ated
and
let
people
do
they
think
they
can
come
talk
to
us
and
say:
hey
you
did
this
stuff
all
the
time.
What
do
you
think,
but
we're
not
going
to
say
you
know?
No,
don't
do
that
with
some
exceptions.
A
Ok,
so
let's
talk
about
cattle
markets,
so
the
controllers
and
the
nodes.
We
are
an
AWS
shop,
so
we,
the
controllers
and
the
nodes
of
both
created
in
OS
cheese.
The
controllers
are
cycled
automatically
when
we
update
the
launch
config
by
the
ASG
and
terraform.
Sorry
I
should
have
mentioned
that
the
first
two
layers,
a
terraform,
the
top
layer
Goliath,
is
ansible
rig.
A
So
this
one
yeah,
so
this
one
when
we
say
need
to
upgrade
the
way
that
we
upgrade
the
control
plane.
Is
we
update
the
cube
version
inside
the
terraform
and
push
there
from
that's
it?
Terraform
handles
are
creating
a
new
ASG
mate
they'll,
be
health
checks
confirm
that
the
API
servers
have
come
up
correctly
and
then
it
deposed
is
the
old
one.
It
destroys
it.
So,
for
us,
it's
a
relatively
atomic
operation
to
do
a
cube.
Upgrade
that's
been
really
good.
A
When
we
have
things
like
the
recent
severe
meant
that
we
needed
to
deploy
changes
pretty
quickly,
it's
really
straightforward
to
us
to
do
the
other
thing.
That's
good!
Is
that
the
other
nodes
there
cycle
by
the
otoscope
by
an
auto
scaler.
So
if
we
want
to
upgrade
their
version,
we
just
changed
their
launch
config
and
then
the
next
time
that
they
die
and
come
back
and
you
get
new
nodes.
Then
those
new
nodes
are
the
right
version.
So
there's
never.
We
do
because
of
the
cattle.
A
No
pets
thing
we
do
is
zero
in-place
upgrades
zero,
touching
the
instance
touching.
Yes,
this
is
100%
any
pattern.
The
only
reason
you
go
on
to
the
instance
at
all
is
to
figure
out
some
curly
problem
that
you
don't
have
in
your
logs
or
or
some
Dockers
gone
weird
or
something
like
that,
and
that
has
really
really
saved
us
so
much
time
and
heartache
in
the
long
run.
It
is
not
funny
a
CD
servers
now
in
terms
of
cattle,
not
pets,
I
like
to
say
they
like
milk
cows.
A
You
are
right,
the
and
so
the
yeah,
because
the
point
there
is
you
do
got
to
be
a
little
bit
careful
with
the
air
CDs
because
they
are
the
source
of
truth,
but
trying
to
keep
them
immutable
is
really
really
helpful
because
it
means
it
because
they're
the
sort
of
truth
you're
never
going
to
accidentally
shoot
yourself
in
the
foot,
and
so
you
again
this
is.
This-
has
actually
been
one
of
the
hardest
parts
for
us
to
get
right
in
our
setup.
It
is
really
tricky
to
run
a
CD
nodes
in
an
immutable
fashion.
A
So
yeah
rebuilding
originally
our
plan
was
that
any
significant
cluster
change
we
will
burn
it
down
to
the
ground
and
rebuild
it
from
scratch,
and
so
the
tooling
is
designed
to
allow
us
to
do
that
in
about
30
minutes.
So,
basically
there's
a
build
that
we
can
do.
That
is
a
destroy
build
that
will
destroy
the
at
the
car
level
that
will
destroy
all
the
compute
and
then
we
do
a
fresh
deploy
and
we'll
get
all
the
compute
back.
A
That
means,
however,
all
the
stuff
that
was
in
that
CD
is
gone
so,
as
we've
started
to
take
on
loads
that
were
less
stateless
and
more
stateful
and
start
nave
started
to
persist,
data
and
NCD
that
they
don't
want
to
lose.
We
needed
to
it's
harder
to
do
that
now,
right,
we've
got
to
do
a
bit
more
shenanigans
about
a
CD
restores.
It
is
possible,
though
so
secrets
here.
A
This
is
talking
about
managing
dependencies
for
these
secrets,
I'm,
not
talking
about
cube
secrets,
I'm
talking
about
the
initial
secrets
that
you
need
to
actually
stand
up,
the
cubes,
so
I
would
like
things
like
the
see,
a
search
for
the
cluster.
The
new
this,
the
search
for
the
serving
search
for
NCD,
because
the
client
search
that
you
use
to
do
a
TD
authentication
because
you
better
be
doing
entity
authentication
at
this
point.
If
you
are
not,
you
could
possibly
be
in
for
a
very
bad
time
the
it
is
off
by
default
as
well.
A
So
you
know
things
to
remember
and
look
at
that
so
yeah
we
saw
the
secrets
in
private
s3
buckets.
There
are
only
accessible
to
the
nodes
and
obviously
to
ask
this
admins
and
then
there's
a
there's,
a
net
job
that
runs
at
startup
on
the
nodes
and
it
pulls
all
the
secrets,
put
them
in
the
right
place,
and
then
it
is
a
system
to
you
that
has
that
everything
else
is.
Has
a
requirement
has
that
as
a
requirement,
so
that
yeah
nothing
happens
until
it's
finished.
A
A
That's
that
supply
chain
integrity
is
a
really
really
important
security
hole
that
everybody
needs
to
plug,
so
how
it
went
well.
I
cost
us
go
pretty
well,
our
biggest
cluster
size
so
far
is
about
14,000
three
CPUs
and
50
terabytes
of
RAM,
and
we
currently
have
about
twenty.
Most
of
them
are
not
usually
that
big,
that's
the
biggest
one
that
runs
a
whole
bunch
of
the
elastin
internal,
see
I.
A
For
now,
most
of
our
workload
is
batch.
Jobs
ci
builds
from
one
internal
customer
or
another
at
in
October.
Our
average
was
177
K
pods
a
day,
which
is
a
little
bit
more
around
about
2
per
second.
So
it's
a
pretty
it's
a
pretty
respectable
amount
of
pods
I.
Think
overall
the
and
what
we
find
is
that
the
internal
workloads
are
super
bursty.
As
I
said,
a
lot
of
the
time
it'll
be
at
5
a.m.
Sydney.
Time
will
often
get
two
and
a
half
to
three
thousand
builds
shared
you'll
double
need.
A
You
know
a
couple
hundred
nodes
added
very
quickly.
So
that's
one
of
the
reasons
why
we
built
escalator,
which
is
the
batch
optimized
ortus
cluster,
autoscaler,
that
all
the
order
scales
nodes
for
us,
that's
a
github
home,
slash
or
less
in
flash
escalator.
If
you
want
to
check
it
out,
it
is
definitely
built
for
jobs
that
run
to
completion,
though
so
it's
completely
not
useful
for
state
bliss
server.
A
They'll
just
say:
I
want
a
docker
image,
a
Postgres
database
and
I
want
it
to
be
public.
You
know
I
wanted
to
exposed
on
the
internet,
please
in
a
really
simple
declarative
file,
and
then
there
is
a
whole
bunch
of
magic
that
the
poster
team
are
building.
That
will
then
translate
all
of
that
through
a
variety
of
translations
into
actual
cube
things.
Why
aren't
all
up
for
you
generate
your
Postgres
database?
A
However,
some
stuff
has
nothing
great.
Managing
at
today
is
hard
really
hard.
The
we've
had
heaps
of
problems
with
running
at
TD
yeah
consensus
problems
with
raft,
io
problem
running
out,
I
Oh
problems
can
cause
a
election
storms
that
you
know
are
not
a
good
time
if
you
ever
get.
If
you
ever
hit
the
2.1
gig
limit
on
the
size
of
your
database,
then
your
cluster
will
flip
into
read-only
mode,
and
you
will
have
a
pretty
bad
day.
A
A
The
instable
is
really
good
for
early
prototyping,
but
the
deployment
model
we
originally
use
just
in
scale.
What
one
of
our
guys
had
to
do
was
to
paralyze
the
build
state
to
build
jobs
in
bamboo
the
CI
tall
we
use,
obviously,
since
it
sails
and
when
we
once
we
paralyze
the
build
jobs.
You
know
we
admin's
that
now
we
wait.
20
minutes
rebuild
not
20
minutes
for
each
cluster
times
the
number
of
clusters
we're
currently
at
20
clusters
and
adding
more
so.
A
That
was
a
really
important
step
for
us
not
to
be
spending
three
or
four
hours
a
day
waiting
for
built,
but
that
is
mostly
fixed.
Now.
Yeah
security
is
a
challenge
like
I
said
there,
it's
it's
not
really
tied
to
design.
It's
gonna
happen,
no
matter
what
you
do,
but
it
is
really
worth
noting.
Security
is
really
hard,
the
other,
the
other
main
workload
Bachelor
clothes
that
we
use
is
bitbucket
pipelines.
Epochal
pipelines
is
bit
buckets,
CI
CD
in
the
cloud
for
you,
I
like
to
say
to
people
as
the
people
who
run
the
platform.
A
A
There
are
great
options
available
for
you
in
Cuba
noddies
to
tune
all
this
stuff,
but,
like
a
bunch
of
people
have
said
in
the
keynotes
and
other
places
that
defaults
are
not
very
secure
yet
so
tips,
sorry,
so
here's
just
some
general
notes
and
stuff
on
things
that
I
learned
from
our
design
that
really
really
worked.
Firstly,
manage
the
dependencies,
be
careful
about
those
dependency
cycles.
If
you
are
going
to
have
dependency
cycles,
you
need
to
make
sure
that
you
are
knowingly
choosing
them
and
that.
A
Sorry,
you
need
to
be
careful
that
you
are
choosing
knowingly
to
have
dependency
cycles,
and
then
you
know
how
to
bootstrap
yourself
out
of
them.
If
everything
is
down,
we
chose
to
not
have
to
worry
about
that
by
making
sure
we
didn't
have
dependency
cycles,
it's
okay
to
have
dependency
cycles.
You
just
have
to
know
that
they're
cattle,
not
pets.
The
immutable
infrastructure
practices
will
make
your
life
easier
Abel.
A
Lastly,
you
should
percent
get
involved
I,
you
know,
I
do
I'm
a
sysadmin,
not
you
historically,
not
even
a
poll
that
has
changed
since
I
started
using
cube,
but
the
you
I've
got
involved
in
the
kubernetes
community,
literally
just
a
shop
to
meetings
and
be
like
hey.
I
use
this
thing
and
I
have
found
that
you
know
as
an
end
user.
People
are
desperate
to
hear
what
I
have
to
say
about
things
and
so
I
would
strongly
recommend.
Even
if
you
are
just
a
clock,
cluster
operator
I
guarantee
that
any
seek
call
you
dial
into.
A
Manesh
cube
is
pretty
great.
Now,
I'm.
Sorry,
the
you
know
when
we
started
there
was
no
such
thing
as
managed
cube.
Really
there
was
very
early
gke
there
wasn't
even
google
kubernetes
engine
back
then,
but
now
you
know
between
the
three
major
club
providers
and
everyone
else.
You
know
the
manage
cube.
A
Clusters
are
100%
usable,
especially
if
you're,
just
starting
out
now,
if
you're
just
starting
out
now,
there's
really
no
reason
for
you
not
to
use
one
of
them
at
whichever
manage
cloud
provider
you're
closest
to
or
to
you
know,
whichever
your
cloud
provider
of
choice.
You
know
if
there
is
a
managed
cube
solution
for
you
and
you're
just
starting
out.
It
is
almost
certainly
going
to
be
better
for
you
to
get
started
with
a
managed,
cube
solution
and
then
think
about
doing
your
own.
A
If
you
find
you
need
to
yeah
I
mean
the
the
managed
platforms,
don't
claim
to
be
all
things
to
all
people
and
they're,
not
you
know,
there
is
definitely
stuff
that
you
just
cannot
do
I
mean
for
us,
we've
twiddled
every
API
server
knob
you
can
twiddle,
and
so
for
us
we
can't
go
back
to
not
being
able
to
twiddle
those
things.
You
know
you
get
addicted
to
the
twiddling
once
you
once
you
start
actually
doing
it.
A
You
don't
want
to
run
a
TD,
that's
the
other
reason
to
use
a
menu
one.
Is
you
really
don't
want
to
have
to
run
a
TD?
If
you
have
to
running
your
own,
NCD
is
like
running
your
own
database
thirty
years
ago.
There
is
not.
You
know,
the
software
is
great.
Definitely
the
social
is
excellent.
It
is
well
built,
it
runs
really
well,
but
even
the
documentation
for
the
software.
There
are
pages
and
pages
and
pages
of
caveats.
Things
you've
got
to
be
aware
of.
A
If
you
haven't,
if
you
don't
know
that
they're
there,
they
will
shoot,
they
will
catch
you,
and
so
that's
why
I
mean
by
it's
like
a
database
30
years
ago.
There's
no
sort
of
you
know,
using
years
and
years
of
best
practices
and
here's
the
things
you've
got
to
watch
out
for
and
all
those
sort
of
you
know,
industry
knowledge
that
is
just
sitting
around
waiting
to
be
used.
A
A
It
is
the
hard
way
you
know
and
from
the
problem
for
us
now
is
that
we
have
got
this
whole
tool,
train
working
and
it's
you
as
good
as
you
buy
em
for
us
for
our
specific
use
case
and
now
there's
a
huge
ingénue
effort
involved
in
actually
swapping
over
to
everything
else.
So
again,
it's
kind
of
a
bit
like
being
addicted
to
it,
because
we
can't
give
it
up
so
yeah.
That's
does
the
reason
why
I
think
you
really
should
use
manage
things
where
you
can
thanks.
I've
got
time
for
questions.
A
A
Yeah,
so
the
question
was
about
managed
services
and
where
I
feel
the
future
is
yes
for
us
absolutely.
We
we
reevaluate,
obviously
for
us
it's
eks,
because
we
are
mainly
we're,
enabled
a
WI
shop.
Are
we
reevaluate
BK
s
every
six
months
or
so
to
see
where
it's
at
we've
talked
with
Hawking
to
the
e
KS
team,
about
the
exact
features
we
want
and
that
sort
of
thing
I
would
strongly
recommend
that
ya?
A
B
B
A
Don't
have
much
experience,
I
started.
The
question
is
about
TD.
If
we
run
zookeeper
already,
is
there
TD
harder
than
that?
I?
Don't
have
much
experience
with
the
zookeeper
personally,
but
yeah
all
consensus
algorithm,
just
kind
of
similar.
You
know
like
if
you're
used
to
running
a
consistent
algorithm
and
you
have
a
good
sense
of
like
how
many
nodes
you
need
and
all
those
sort
of
things
it's
probably
gonna,
be
easier
for
you
than
it
would
be
for
other
people.
That
said
it
C
D
does
have
a
bunch
of
you
know
of
its
own
stuff.
A
Yeah
sure,
so
the
question
is
about
Costas
eyes
and
look
sounds
like
we
have
lots
of
smaller
clusters
and
was
that
a
decision
during
the
design?
So
yes,
and
no
during
the
design,
the
original
intent
was
actually
to
have
very
small
clusters
to
have
no
class
to
span
and
AZ
boundary
in
OWS.
So
they
would
all
be
single
quite
single
AZ
clusters.
It
would
the
intent
that
different
AZ
had
problems
that
went
away.
You
know
you
treated
the
clusters
as
as,
though
they
were
nodes.
You
know
super
cluster.
A
The
intent
was
to
use
a
original
Federation
to
do
that.
We
found
that
original
federation.
Just
it
didn't
move
fast
enough
for
us,
and
we
had
you
know
greater
reliability
requirements.
Then
the
solution
could
deliver,
and
so
what
we
ended
up
doing
was
changing
our
clusters,
design
so
that
it
spanned
three
AZ's
all
the
time,
so
that
your
main
three
young,
main
important
control,
plane
parts
all
had
a
node
in
haz
and
that's
how
we
sort
of
met
our
availability
requirements,
the
cluster
size.
We
have
built
them
to
scale.
A
I
think
the
number
of
nodes
we
can
have
is
about
four.
Thousands
in
each
cluster,
so
the
thing
that
one
of
the
things
that
we've
found
is
one
of
the
primary
determinants
of
the
scale
of
your
that
your
clusters
can
get
to
is
actually
your
IP
allocation
strategy.
You
know,
because
if
you,
if
you're
running
an
SDN,
each
node
needs
to
have
a
subnet
carved
out
for
it.
A
The
size
of
your
note
of
your
pool
of
for
your
SD
n,
divided
by
the
number
of
nodes
you
want
to
lay
out,
is
how
many
pods
you
can
have
running
on
each
node.
So
there's
a
real
sort
of
tight
coupling
between
those
three
numbers
as
to
exactly
how
much
you
can
scale.
Sorry,
oh
they
were
then
into
your
couldn't
yeah
thanks,
I
think
I'll
go
back
to
front
of
those
okay.
A
Right
yeah,
so
if
we
were
running
containers
before
uncle
II,
we
did
Amazon
asked
us
to
run
release
yes.
So
the
containers
that
we
were
running
before
were
not
containers
in
the
modern
sense
of
the
word.
They
were.
They
used
the
same
C
group
container
functionality,
but
they
were
openvz
containers
where
it's
kind
of
a
bit
more
like
a
truth.
A
A
Yes,
your
so
the
question
is
about
our
the
number
of
individual
clusters
and
what
the
use
case
we
have
for
each
cluster
so
the
way
that
we
designate
so
when
I
talked
about
the
bottom
layer,
the
flag,
the
way
that
we
break
up
it
divided.
The
clusters
is
via
a
three
tuple
of
information:
there's
the
internal
cost
customer
name,
an
internal
customer
environment
and
the
AWS
region,
and
so
the
intent
being
that
we
could
have.
You
know
we
have
the
say
tobacco
placards
clusters.
A
We
can
have
a
big
pocket
pipelines,
customer
that
has
a
dev
environment
in
u.s.
East
one
and
a
dev
environment
in
u.s.
West
and
a
pride
environment
in
u.s..
This
one
on
a
protein
or
a
man
in
u.s.
was,
and
then
we
can
have
a
different
customer,
the
payers
customer
that
also
have
their
own
set
of
all
the
levels
that
they
want
in
all
the
ridges
that
they
want.
A
And
so
that's
one
of
the
reasons
we've
ended
up
with
a
proliferation
of
things
is
that
we've
got
a
few
internal
customers
now
and
they
all
want
different
regions
or
different
levels
or
different
environment
names.
It's
not
actually
that
each
of
them
is
very
big.
It's
that
these
are
administrative
sort
of
distinctions
between
the
clusters
and
it
the
good
part
about
that
is
that
it
lets
you
do
you
know
it
makes
the
clusters
more.
A
Sure
I'm!
Yes,
so
the
question
is
about
exactly
how
we're
bringing
up
the
clusters
see
the
the
the
flag
layer,
just
as
I
said,
does
the
initial
VPC
creation
and
stuff
that's
with
terraform
and
then
we're
terraform
in
in
a
car
layer.
It
literally
just
blacked
out
or
look
all
of
the
compute
and
the
good
part
about
terraform
is
that
it
deduces
a
lot
of
the
dependencies
based
on
how
you
use
outputs
from
tariffs
or
modules,
and
so
we've
set
up
dependencies
and
stuff.
So
there's
a
CDS
come
up.
A
First,
API
servers
come
up
after
that,
so
we
use
controller
nodes
which
run
the
API
server,
the
controller
manager
and
the
scheduler.
And
then
we
have
separated
city
nodes
that
handle
the
actual
secret
storing
and
then
there
are
multiple
pools
of
worker
nodes
that
can
scale
to
different
sizes.
Sorry,
does
that
answer
your
question.
A
So
for
those,
so
the
question
is
out
of
the
layers
in
the
cake,
which
ones
does
our
team
manage
and
which
ones
do
we
delegate
out
of
him?
Those
are
the
three
layers
that
we
that
we
actually
manage.
We
are
uncalled-for
all
three
of
those
the
delegation
to
other
teams
is
usually
about
what
they
build
after
Goliath
runs
and
so
part
of
what
we
do
with
Goliath
is
say:
if
someone
says
hey
I'd
like
a
no
space
in
a
cluster,
then
we
will
create.
A
We
will
create
the
namespace
for
them
and
ensure
that
the
you
that
super
basic
stuff
is
wired
up
and
make
sure
that
they
have
a
namespace
admin
service
account
and
then,
after
that,
it's
on
them
what
they
do
with
it.
We
sort
of
say:
hey
it's
up
to
you
but
come
and
come
and
talk
to
us.
If
you
just
say
what
you
want
to
do.
Yes,
sorry.
B
A
Sure
sure
so
the
question
is
about
you
know:
are
we
concerned
about
turning
over?
Are
we
concerned
about
turning
over
control
to
other
people?
The
the
in
turn
is
actually
that
we
will
never
turn
over
control
of
those
things
to
other
people
when
other
people,
when
other
customers
are
using
the
platform
we
are
still
on
the
hook
for
the
operational
platform,
so
our
SLA
s
are
all
about
availability
of
the
API
servers
and
whatever
metric
the
customer
cares
about.
A
So
in
the
case
of
the
CI,
the
metric
is
actually
time
to
schedule
for
pods
in
the
case
of
sort
of
stateless,
long-running
service
customers.
Their
metric
is
like
how
the
uptime
of
the
services
that
are
running
on
the
platform.
Does
that
answer
your
question?
Yes,
cool,
so
I
think
why
not
the
back
day.
A
A
Do
we
retire
clusters?
We
don't
retire
clusters
very
often,
unless
we
determine
it's
no
longer
a
need
for
them.
That
said,
we
found
that
one
of
the
dead
clusters
we
stood
up
originally
recently
that
no
one
was
using
it,
so
we
have
retired
that
one,
but
that's
the
only
one
that
we've
actually
retired
since
we
started
standing
them
up
yeah
most
of
the
time
people
want
to
use
them.
A
A
Yes,
oh
right,
so
the
question
is
about
failover
and
are
we
running
clusters
active-passive?
So
yes,
what
we
try
and
do
is
make
it
so
the
customer
gets
to
decide
exactly
how
they're
using
the
clusters.
As
far
as
we're
concerned,
they
are
active,
active
active
everything
always
got
to
be
up.
You
know
we're
in
our
internal
sort
of
priority
classification.
We
are
at
e0
at
service.
A
You
know
we're
four
or
five
nines
and
reliability,
depending
on
which
customer
you're
talking
to
so,
but
the
intent
being
that
the
customers
need
to
have
a
way
to
be
able
to
flip
to
another
cluster
if
they,
if
they
need
to
I'm
the
way
that
we
actually
do.
That
is
we
have
cluster
Registry
running
and
with
there's
ways
that
we
can
poke
that
closet
registry
to
say
hey
for
this
customer,
you
should
direct
your
traffic
over
there.
A
Only
some
customers
are
started
using
it
because
it's
still
pretty
new,
but
we're
hoping
that
that
will
be
the
way
that
we
can
sort
of
guide
traffic
without
people
needing
to
be
woken
up.
Even
ideally,
no
no
so
people
only
schedule
most
customers
that
we
have
now
we'll
only
schedule
load
in
one
cluster,
and
so
there
is
a
problem.
It's
literally
just
we
ask
them
to
you
know.
Currently
we
have
to
ask
them
to
change
their
to
linked
up
to
flip
to
another
one.
A
Yes,
yeah
and
that's
what
we
found
I
mean
generally,
if
you're
having
a
bad
enough
problem
that
you'd
even
consider
failing
over
the
you
know,
people
would
prefer
to
have
some
workload
working
then
be
too
concerned
about
the
stuff
that
is
broken
already
well.
We
we
had
found
I
think
there
was
one
other
person
behind
there
and
then
your
next
min.
A
Okay,
so
the
question
is
about
what
are
the
requirements?
We
have
that
that
we
can't
run
on
managed
service,
yet
most
of
them
are
around
messing
with
the
API
server.
Turning
on
experimental
features
for
a
while,
we
needed
to
add
a
bunch
of
settings
about
Web
books
and
things
like
that.
Most
of
that
stuff
is
defaulted
on
now,
so
some
of
that
has
gone
away,
but
one
of
the
things
about
having
app
as
having
people
developing
app
has
on
us
is
a
lot
of
the
time.
A
B
A
I
know
that
is
a
specific
sort
of
contract
to
the
platform.
Is
that
you
know
if
you,
if
you
store
stuff
in
the
cluster,
it's
up
to
you
to
make
sure
that
it
ends
up
in
both
places
since
you're
up
to
your
pipeline,
cool
I
think
we're
out
of
time.
I
am
happy
to
take
some
more
questions.
Just
I'll
I'll
be
just
outside
once
I
packed
up.