►
From YouTube: The Myth of the Monocluster - Matt Silverlock, Google
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
The Myth of the Monocluster - Matt Silverlock, Google
Building out a single monolithic Kubernetes cluster and trying to migrate all the things rarely, if ever, works out, and Kubernetes doesn't change that. It becomes harder to gather non-conflicting requirements, or avoid scope creep as new teams have what seem like reasonable asks (to them). Not to mention the technical challenges & increased blast radius of a big cluster. How can we start with smaller teams, help them migrate and operationalize their clusters, learn from the inevitable mistakes, document the shortcuts, and use that as the framework for future teams? Let's talk through what we need to ask ourselves in order to migrate to Kubernetes, how to divide & conquer (our clusters), and some lessons learnt from working with large organizations.
https://sched.co/Uady
A
Thank
You
room
for
coming
I
met
and
I'm
gonna
talk
about
the
myth
of
the
mana
cluster,
so
as
a
quick,
intro
I'm,
a
p.m.
at
Google
I'm,
not
gonna,
talk
about
anything
Google
related
because
I'm
sure
you're,
probably
tired
of
that
at
this
point
and
I'm
still
pretty
unsure
whether
I
should
keep
writing
to
conference
talks.
It's
always
like
a
last
minute
thing.
A
So
this
talk.
This
talk
is
about
the
mana
clusters.
The
mana
foster
is
the
idea
that
you
have
this
one
big
kubernetes
cluster
in
your
organization
right,
you
might
have
Devon
staging,
but
you
have
this
one
big
production
cluster.
You
had
datacenter
neighbor
cluster.
It's
this
like
ideal
world
to
sort
of
deploy
into
you
know
its
gets
controlled.
You
know
the
surface
of
it,
that's
great,
but
this
talks
about
reducing
the
blast
radius
of
all
of
your
communities
clusters.
It's
about
understanding
the
failure
modes.
It's
also
the
considerations
as
you
scale,
communities
in
organization
level.
A
So
you
know
what
are
these
myths
that
I'm
talking
about
at
least
like
mono
cluster?
It's
that
one
large
cluster
makes
sense.
It's
like!
Isn't
that
what
like
Google?
Does
we
bog,
and
instead
of
similar
sort
of
cost
dimension
tools?
You
know,
is
it
that
managing
more
masters
is
bad?
Is
it
that
centralized
controls
great
and
that's
often
true?
A
You
can
control
all
of
that
sort
of
auto
scaling
with
a
single
cluster
much
more
easily.
So
you
know
the
problems.
Are
you
know?
Building
a
single
company
y
cluster
sounds
like
a
good
idea,
and
the
biggest
problem
of
that
is
like
teams
work
at
different
paces.
They
all
have
different
risk
tolerances
and
what
you
end
up
doing
is
with
a
single
mono
cluster.
You
end
up
operating
at
the
pace
of
your
slowest
team.
So
if
your
slowest
team
is
your
core
billing
team,
your
use
a
database
see
make
if
you're
a
ecommerce
site.
A
Maybe
it's
your
comm
front-ends
right,
they're
gonna
be
slow,
they're,
not
gonna,
deploy
every
week.
They're,
not
gonna
want
to
be
on
the
latest
version
of
Cuban
Eddie's.
Just
cuz
you
want
these
features.
They
don't
want
those
features
right,
they're,
giving
away
their
work
risk.
When
you
have
this
big
cluster,
you
kind
of
suss
to
ossify,
and
you
start
to
not
upgrade
you
start
to
not
think
about
security.
You
know
security
patches
often
involve
upgrades.
They
often
involve
you
know,
potential
sort
of
rolling
restarts.
A
You
start
to
add
all
this
risk,
because
you're
too
scared
to
sort
of
actually
do
anything
it.
Actually,
it
was
obvious
in
the
inverse
right
at
some
point,
you're,
like
I'm
gonna
de-risk,
by
having
one
big
cluster,
it's
all
fully
controlled
everyone's
on
the
same
version
of
kubernetes,
great
feed,
debugging.
Everyone
has
the
same
experience
and
then
at
some
point
the
table
turns-
and
you
say,
actually
wait.
All
of
these
teams
have
beholden
to
like
one
particular
team.
This
probably
reminds
you
a
lot
of
like
micro
services
in
general.
A
Right
is
a
very
similar
sort
of
sort
of
pattern
here
and
sort
of
paradigm.
We've
had
monolithic
applications.
We
started
to
break
them
down
because
we're
all
sort
of
you
know
deploying
them
once
a
year
in
the
bad
old
days
or
realistically,
you
know
once
a
month
or
so,
and
it
was
scary
and
like
only
one
person
knew
how
to
sort
of
deploy
and
we
kept
flowing
and
flowing
slowing
down.
A
So
what
are
the
challenges
with
you
know?
A
not
gonna
cost
the
design
right,
it's
kind
of
avoiding
following
the
Oakshott.
It's
you
know
with
the
mono
cluster.
It's
also
like
how
big
is
too
big
with
individual
clusters.
It's
how
small
is
too
small
like
do
you
have
you
know
in
a
large
organization?
Do
you
just
have
a
hundred
clusters
of
ten
nodes
each
or
something
like
that?
And
you
know
stuff
like
multicast
orchestration,
which
is
it's
pretty
NASA
and
pretty
hard
to
do
so?
A
Let's
talk
a
little
bit
about
sort
of
double
back
on
the
mono
cluster
pod
right,
which
is
you
know,
I've
said
it's
bad,
but,
like
I
haven't
proved
to
you
that
it's
bad.
Some
of
you
probably
have
experience
with
this
before,
but
you're
like
what
are
the
ways
that
it
can
fail
and
others
ways
bad
enough
to
do
something
else
about
right,
dude,
maybe
just
sort
of
like
push
through
it
solve
those
problems.
Are
they
even
reasonably
solvable
problem,
either
tomorrow
or
next
year?
A
Obviously,
there's
a
lot
of
changing
with
communities
in
general
and
a
lot
of
the
tapping
sort
of
this
week.
But
you
know
what
are
these
failure
modes
that
we
want
to
get
away
from,
so
you
know
think
about
it
in
these
sort
of
three
ways:
what
components
you
know
scale
with
the
cluster
and
where
their
failure
counts
actually
like
scale
proportionally
or
even
worse,
with
the
cluster
shot
at
size
in
shape
right,
so
what
components
fail
when
the
clusters
are
likely
to
fail
in
the
clusters?
A
50
nodes-
and
you
know
2,000
pods
versus
500
nodes
and
you
know,
or
at
magnitude,
more
pods,
so
on
and
so
forth.
Right.
What
are
the
things
that
start
to
get
worse
and
worse
and
have
a
much
lighter,
budget-wise
radius
just
inside
the
cluster,
let
alone
the
cluster
actually
impacting
the
organization?
A
What's
the
impact
of
those
failures
on
your
services
right?
Is
it
like
complete
failure?
Is
it
just
partial
failure?
Is
it
some
kind
of
degradation
of
service,
and
can
you
manage
around
that?
That
might
be
okay
right,
you
might
just
say:
hey
I'm,
okay,
with
having
one
cluster.
If
failure
is
like
a
partial
degradation,
I
can
sort
of
work
around
it
right
and
that
sort
of
speaks
to
you
know
can
I
avoid.
Can
we
instead
of
avoid
those
problems
in
general,
so
there's
a
few
parts,
DNS
DNS
always
fails.
A
It's
nearly
always
the
bottleneck
right.
It
is
doubly
so
if
you're
actually
sinking
that
cluster
DNS
to
some
other
kind
of
DNS
that
you
need
to
sort
of
rely
with.
Oh
it's
your
cloud
provider,
whether
it's
on
Prem
right.
You
have
some
infrastructure
somewhere
else
that,
like
services
outside
kubernetes,
if
most
of
us,
probably
haunt
you,
know
100%
communities
everywhere,
let's
do
a
sort
of
very
new
or
very
into
it
right.
So
there's
more
failure
points
there
as
well.
A
If
the
cluster
dns
fails
all
the
controller
that
syncing
DNS
fails,
you've
lost
even
more
you've
lost
all
connectivity
in
and
out
and
when
DNS
performance
sort
of
actually
starts
to
take
a
hit
right.
That
impacts
like
all
your
services,
service,
lookups
as
well
right
and
for
the
most
part,
TTLs
and
kubernetes,
are
typically
pretty
short
to
deal
with.
You
know
auto-scaling
challenges
as
well
right.
Do
you
want
those
names
to
be
reasonably
fresh
that
other
services
know
where
to
go?
A
So
if
you
see
you
know
if
you've
ever
seen,
sort
of
the
communities,
failure
stories
and
github,
which
I
highly
recommend,
is
a
a
list
of
a
long
long
number
of
like
how
its
failed
for
many
organizations
in
many
different
ways.
You
notice
that
I
think
there's
about
thirty
or
so
seventeen
of
them
feature.
Dns
right.
Dennis
comes
up
a
lot
as
a
finally
might
write
and
again
with
that
DNS.
What
are
you
gonna
do
when
your
cluster?
How
a
service
is
supposed
to
communicate?
A
What
else
so
the
API
server-
and
you
know
by
extension,
instead
of
like
at
CD
right.
That's,
that's
your
control,
plane
right
with
DNS.
That's
that's!
The
core
part
of
the
control
plane
scaling
your
masters
vertically
only
gets
you
so
far
right.
You
can
keep
scaling
them
up,
but
inevitably
in
the
actual
API
server,
binary
itself
is
potentially
going
to
have
issues
right.
It's
still
not,
actually
you
know
reducing
the
blast.
Radius.
You've
just
actually
packed
more
on
right,
32
cores
under
28
cores
as
much
RAM
as
you
want
to
throw
at
it
right.
A
You
can
keep
scaling
this
up
and
we
live
in
a
world
of
you
know
scheduled
virtual
machines,
but
that's
only
gonna
take
this
phone.
It
can
also
be
really
hard
to
fix
problems
once
you
actually
start
having
API
server
issues.
So
you
can't
roll
back
or
you
can't
scale.
A
lot
of
things
are
on
the
Lord.
Do
you
say
era
I'm,
just
gonna
Scout
a
little
bit
more
to
sort
of
like
deal
with
the
load
issue
and
actually
debug
why
these
services
are
overloaded
individually
later
right?
A
So,
of
course,
I
think
you
know
what
to
do
a
talk
here
without
some
kind
of
like
Kelsey
Hightower
queen-like
quote
tweet
right,
but
they
really
it's
a
really
key
part.
Is
your
control
pain
should
be
sort
of
scoped
to
the
smallest
fairly
domain
you
can
afford
and
that
is.
Can
you
afford
the
whole
business
value?
Because,
although
services
you're
running
are
on
the
same
cluster,
probably
not,
could
you
afford
your
customer
facing
api's
or
comm
failing
because
they
were
all
on
the
same
cluster
as
well?
A
Probably
not
as
well
right
this
like
really
out
of
place,
you
want
to
be
I.
Think
failure
is
inevitable.
We
should
assume
that
at
some
point,
even
with
sort
of
all
the
magic
and
and
sort
of
orchestration,
that
kubernetes
provides
that
things
will
fail,
and
we
should
have
a
plan
to
sort
of
deal
with
that
failure
right
so
figuring
out
what
the
control
plane
is
around
that
is
it
the
same
CI
pipeline?
Maybe
you
can
deal
without
that.
You
have
a
great
glass
for
deploying
manually.
A
A
The
whole
point
of
us
like
having
these
sort
of
orchestration
platforms
is
to
get
away
from
the
hardware
and
be
at
a
sort
of
abstract
that,
but
if
we're
coming
back
to
these
singular
control,
planes,
we've
just
manufactured
another
form
of
sort
of
failure,
and
you
know
the
other
part
of
this
is
you
think
about
how
do
other
workloads
write,
noisy
neighbor
is
still
a
problem
in
Cuba
Nettie's
as
well
right
again,
one
big
cluster.
All
these
things
contending
you
can
have
the
smallest
node.
A
You
can
have
these
huge
nodes,
there's
tons
of
calls
and
RAM
and
there's
plenty
of
things
that
kubernetes
is
not
correct
for
right.
If
you've,
if
you've
ever
written,
like
a
like
a
pod,
manifest
right,
you
can
have
CPU
and
instead
of
rambling,
like
memory
limit
limits
in
the
in
sort
of
requests
right.
But
what
about
network?
What
about
disk
I/o
like
AI
ops,
like
you,
can't
reasonably
define
those
through
cribbens
directly
for
the
most
part
right
into
those
things,
can
totally
destroy
your
cluster
right.
They
can
completely
blow
up
workloads.
A
It's
very
easy,
in
fact,
I've
seen
many
folks
do
this
and
I'm
seeing
from
experience
they
define
everything
with
you
know.
Resource
limits,
and
it's
great
in
those
like
is
gonna,
be
awesome,
and
they
have
this
really
like
disk
I/o
heavy
application
and
it
completely
saturated
the
network
and
the
VM
as
well,
and
none
of
it
actually
works
right.
It
never
actually
uses
a
less
CPU
Ram.
A
It
never
gets
actually
rescheduled
anywhere
else,
because
it's
totally
within
its
limits,
but
I've
tried
to
pack
as
much
as
possible
onto
this
single
cluster
and
they've
blown
everything
up
right
at
some
point
and
network
interfaces
and
disks
are
only
scale
so
much
so.
The
way
to
sort
of
think
about
this
is
like
multi-tenancy
and
that's
really
the
core
of
having
a
mono
cluster.
Is
you
want
this?
A
A
multi-tenant
sort
of
system
that
deals
with
all
your
business
has
a
lot
of
dimensions,
right,
you're,
sort
of
dealing
with
again
you're
dealing
with
memory
resources,
you're
dealing
with
if
you're
dealing
with
network
you're
dealing
with
other
users
on
the
system,
who
maybe
don't
have
your
best
interest
in
mind
right?
You
might
work
at
the
same
company
and,
like
you
know
that
to
get
each
other
right,
but
like
they
have
their
things
to
get
done.
You
have
your
things
to
get
done.
A
You
might
not
be
talking
to
each
other
about
this
kind
of
stuff
and
when
you
have
a
large
cluster
right,
it's
not
like
individually.
Are
you
all
managing
or
thinking
about
how
much
load
am
I
putting
on
the
API
servers
like
am
I
making
enough
like
can
making
too
many
DNS
queries
to
sort
of
like
reach
out?
Would
it
make
maybe
rip
make
requests
outside
of
the
cluster
like
when
you're
all
done
together
and
you're
all
sort
of
sharing
this
system
right?
A
That
isolation
is
not
always
there
and
communities
only
provides
so
much
isolation
like
namespaces.
Don't
give
you
really
what
you
want
in
terms
of
actual
underlying
workload,
isolation
even
stuff,
like
gee,
visor
and
cutter
containers,
and
firecracker
right,
well,
isolate
the
actual
workload
running
in
the
container
from
ideally
breaking
out
and
actually
doing
things
on
the
host
right,
but
that
workload
can
still
interfere
with
everything
else
right,
so
you
know
see
that's
another
way,
right,
I!
A
Think
the
challenge
is
that
when
you
have
a
big
mono
cluster
you're,
trying
to
make
clusters
general-purpose
right,
you're
trying
to
say
hey
I,
can
build
a
thing
that
everyone
in
my
business
can
use
that
every
developers,
gonna
love
right,
is
sort
of
like
CTO
brain.
Maybe,
and
it's
gonna
solve
a
lot
of
our
problems
right.
A
We
all
this
nice
consistent
sort
of
deployment
target,
and
it's
actually
really
hard
to
do
that,
because
how
are
you
supposed
to
know
what
every
team
wants
up
front
on
a
system,
particularly
migrating
kubernetes,
which
is
still
the
case
in
most
organizations?
How
do
you
know
what
those
requirements
really
are
if
no
one's
ever
deployed
to
this
before
right?
If
no
one's
actually
run
this
truly
production
before
right?
A
You
you'll
never
gather
all
those
requirements
in
full
upfront
and
trying
to
pile
everybody
into
the
same
system
and
say:
oh,
you
know,
web
front-end
team
you'll
hear
billing
team
you're
here,
I'm
gonna
put
some
stateful
services
on
this
cluster
right.
They
all
had
very
different
requirements
and
you're
ending
up
with
this
just
sort
of
massive
scope.
Creep,
you'll
probably
never
build
a
glass
that
actually
meets
any
of
like
even
half
those
requirements
and
you're
exposing
yourself
to
more
and
more
and
more
failure
modes.
A
So
we
should
avoid
a
single
production,
cost
I
think
I
probably
made
that
opinion
fairly
cute,
pretty
clear
at
this
point
right.
But
what
do
we
do
next?
What's
the
sort
of
alternative
to
this
right?
What's
the
you
know,
many
ways
like
the
knee-jerk
reaction
to
having
one
big
cluster
is
that,
like
everyone
gets
a
cluster,
so
we
have
one
we
cluster,
we
break
it
down
into
a
series
of
clusters.
Right
like
one
of
these
is,
is
again
like
billing.
One
of
these
is
a
bunch
of
back-end
services.
A
Some
of
these
may
be
a
sort
of
internal
like
authentication
systems.
You
know
we've
like
broken
out
these
sort
of
failure
domains.
Maybe
some
of
these
services
aren't
running
across
multiple
clusters,
so
this
looks
like
okay
like
this
is
a
moderate
amount,
and
you
know
this
is
kind
of
a
good
mood.
Like
everyone
gets
a
cluster,
they
can
do
what
they
want.
They
can
sort
of
manage
their
own
requirements.
They
can
install
whatever
they
want
on
it.
They
can
keep
up
with
the
latest
stuff.
A
It's
like
hey,
you
know,
I
want
this
thing
in,
like
you
know,
in
one
one,
six
wait
I
want
these
features.
I
can
upgrade
my
cluster
with
my
team.
We
get
agreement,
I
don't
get
the
whole
ol
to
buy
in,
but
that
sounds
pretty
good,
but
you
know
we
kind
of
like
put
some
structure
to
these
a
little
bit
more
and
what
we
can
have
end
up
with,
like
a
bit
of
an
org
chart
where
every
team
has
a
cluster
and
that's
probably
not
actually
what
we
want.
A
There's
actually
a
name
for
this,
and
it's
called
like
Conway's
law
right
and
it's
the
idea
that
you
know
we
sort
of
produced
the
designs
that
actually
like
mirror
the
size
or
the
shape
of
our
organization.
And
again
this
goes
back
to
the
cult
sort
of
what
we
probably
seen
from
microservices,
where
that
was
actually
in
some
ways.
Pretty
good
right,
not
always
being
bound
to
other
team,
has
helped
a
lot
of
folks
move
at
their
own
sort
of
pace.
Right.
We
use
a
most
of
us.
A
You
sort
of
API
contracts
to
sort
of
at
least
agree
upon
some
sensible
boundaries
between
how
we
sort
of
interact
right.
That's
pretty
good
I,
don't
think
that
is
as
good
in
the
community's
case,
where
every
team
is
now
running
close
to
right
because
out
of
these
teams,
development
teams
or
are
the
kubernetes
operations
teams,
they're,
probably
development
teams
and
humanities
is
just
a
tool
to
get
the
job
done.
A
So
you
know
+14
Conway's
law,
I
kind
of
go
hand-in-hand,
but
it's
really
not
what
you
want.
So
you
know
I've
talked
about
this
a
little
bit
before.
Why
many
clusters
there
work
at
different
paces.
You
have
the
handful
of
clusters
to
measure
electability,
so
you
can
deploy
different
things
at
different
classes.
You
can
have
different
versions,
it's
way
easier
to
canary
properly
at
scale.
Right
teams
can
replicate
their
clusters.
They
can
do
things
at
their
own
pace
and
again
you
don't
have
to
keep
moving
at
the
pace
of
your
most
risk-averse
team.
A
That
is
not
what
we
want.
We
want
the
teams
that
need
to
be
able
to
sort
of
experiment.
We
need
the
teams
that
can
maybe
tolerate.
You
know
an
hour
of
downtime
if
they
really
screw
up
right.
Maybe
there's
like
a
batch
job
team
or
an
m/l
team
is
like
hey
look
if
we
loose
in
progress
that
sucks,
but
I
can
tolerate
an
hour
of
downtime.
We
have
to
roll
back
and
rebuild
our
cluster
from
scratch.
That's
not
so
bad!
Your
team,
that's
managing
the
authentication
services
for
your
customers,
not
so
much
right.
A
That's
probably
a
serious
outage
just
going
to
page
a
whole
bunch
of
folks
and
keep
you
up
at
night.
You
really
don't
want
that
right.
So
you
end
up
operating
at
their
pace
instead,
so
the
problem
is
instead
of
one
pet.
Now
you
just
have
many
pets.
Every
team
treats
his
clusters
like
they're
like
the
little
baby.
It's
like.
If
everything
in
the
world
they
have
customized
it
intensely
for
their
own
things
and
they've
ignored
everything
that
they
don't
have
to
think
about
it.
Doesn't
they
don't
care
about
right?
Oh
I.
A
Don't
need
like
security
policies;
they
don't
need
like
admission
controls.
We
know
everybody
in
the
team.
It's
fine
right,
but
now
that
we
can
start
to
resistant
to
change
themselves
right.
So
we've
come
back
all
the
way
through
and
we've
said,
one
big
cost
is
bad,
because
no
one
wants
to
change
it.
Now
we
have
say
50
classes
and
no
one
was
to
change
their
Kloster.
The
teams
are
now
platform
teams
and
realistically
right,
it's
it's
actually
kind
of
it's
not
the
way
you
want
to
operate
this
sort
of
business
right.
A
A
You
know,
and
the
cloud
providers
don't
make
that
a
ton
easier
today,
most
of
these
sort
of
existing
communities
distros,
don't
make
that
a
ton
easier,
there's
all
a
tooling
up
there,
but
it's
still
not
super
mature
right.
Like
the
idea
of
like
true
multi
costs,
the
Federation
is
still
really
growing.
So
there's
hard
things
just
a
little
bit.
Security
version
drift
is
real,
so
version
drift
of
the
cluster
itself
of
the
Masters
of
all
the
components
they're
running.
A
So
if
you
have
any
admission
controllers,
any
other
kind
of
custom
controllers,
any
kind
of
operators,
you
know
stuff
like
gatekeeper,
all
those
things
start
to
have
version
drift.
That
starts
become
a
real
problem
right
security
patches
again,
that's
also
an
issue.
It's
like
who's
watching
out
on
the
mailing
lists.
Do
you
have?
A
How
does
the
security
team
actually
manage
like
by
proxy
all
of
these
classes
and
make
sure
the
teams
upgrade
right
now
they
have
to
go
and
convince
52
sync
teams
to
go
and
prioritize
this
patch
right
and
in
many
or
given
with
the
best
security
teams
that
can
be
really
hard.
They
all
have
different
deployment
strategies,
because
they've
customized
how
they
generate
their
manifests
and
what
CS
CD
sei
CD
platform
they
might
be
using.
A
You
might
have
you
know,
circle,
CI
or
like
team
city
internally
right,
but
everyone's
like
the
point
of
they
cluster
in
a
slightly
different
way,
because
it
just
works
for
them.
Your
SRA
and
ops
teams,
especially
dev
teams.
This
becomes
a
huge
mess
there.
It's
like
well
I,
don't
really
want
to
own
this
because
they
didn't
build
it.
It's
all
custom
and
it's
truly
impossible
to
troubleshoot
right.
You
have
a
dev
that
moves
from
team
a
to
team
B,
it's
like
moving
to
a
different
company
in
the
worst
case
right.
You
have
this
cluster.
A
That
is
different
version.
Different
software
different
deployment,
different
risks.
All
of
it
is
very,
very
different
right
that
actually
becomes
really
hard,
and
you
know
in
most
most
companies
are
not
just
working
every
day
for
a
single
team
you're
doing
a
lot
of
cross-functional
work,
you're
helping
out
other
folks
in
neighboring
teams
to
solve
their
problems
right
if
you're,
an
ops
like
a
sorry
team,
that's
even
worse,
because
you
get
paid
you're
like
okay,
great
I,
think
we've
seen
this
before
it's
this
problem.
A
You
get
a
brand
new
cluster
you've
never
seen
before,
and
there's
another
like
49
of
them
out
there
in
the
company
that
you
haven't
had
touched
it
because
they
just
haven't
broken
or
you
haven't
even
realized,
because
you
haven't
actually
hooked
up
any
kind
of
monitoring
right
again
it.
How
do
you
kind
of
enforce
that
kind
of
like
nice
platform
when
you've
got
so
many
clusters?
A
So
what
do
we?
What
do
we
go
from
here,
like
one
cost
of
bad
50
class,
is
also
bad
like?
Is
there
a
middle
ground?
Is
there
some
kind
of
like
you
know,
formula
we
can
figure
out
and
say:
hey
like
X
many
people
in
our
company
or
as
many
teams
means.
You
know
why
many
clusters,
not
really
but
I,
think
you
know
we
can
think
about
in
terms
of
criteria.
We'll
hope
you
get
that.
So
this
goes
back
to
I
really
of
saying.
A
Like
the
whole,
like
you
know,
it's
cat
or
not
pets
thing
you
should
treat
those
classes
like
you
can
throw
them
away
right
and
that
doesn't
apply
in
the
motocross
the
case,
because
that's
all
you've
got
it
doesn't
apply
when
every
team
has
are
in
cluster
right.
It
applies
when
you
truly
think
about
clusters.
As
a
throwaway
thing
right,
you
should
be
able
to
throw
them
away.
It's
that
close
to
gets
deleted
and
you
can't
bring
it
back
up
in
a
reasonable
at
a
time.
It
reason
of
time
depends
on
your
own
sort
of
s.
A
Eliza
NASA
Lowe's
right
you
that
might
change
right,
but
you
might
say
look
my
goal
is
that
I
can
bring
a
cluster
up
from
scratch
in
an
hour
in
precisely
the
same
stated
as
it
was
when
I
deleted
it
or
destroyed
the
whole
thing
right.
That's
good,
so
use
stuff
like
the
cluster
API
or
cloud
providers
toolkits
to
like
define
a
base,
cluster
config.
That
is
shared
right
and
that's
where
your
platform
teams
can
have
the
most
influence
right,
they're,
going
to
define
the
baseline
controllers
based
on
versions.
A
First
right
stuff,
like
gatekeepers,
really
good
for
that
and
again
I
talked
about
this.
It's
still
pretty
hard,
but
definitely
try
to
take
a
multi-class.
The
ingress
approach
to
D
risk
you're
like
custom
world
facing
services.
So
again
you
can't
multi
cluster
everything
easily
without
a
lot
of
pain.
That's
kind
of
the
dream
state
to
have
this
like
lovely
Federation,
the
clusters
and
you
can
deploy
things
to
like
five
clusters
and
it
just
all
scales
magically
together.
A
That's
not
probably
gonna
happen,
but
at
least
thing
is
I'm
soon,
but
for
your
really
core
stuff
put
the
effort
in
to
do
that.
Right.
Put
that
effort
into
like
have
multiple
control
points.
It
may
be
even
okay
if
it
doesn't
sort
of
perfectly
auto
scale
on
both
classes
or
downscale
on
both
causes.
At
the
same
time,
right,
you
might
say:
look
I
really
want
to
focus
on
reducing
the
boss
radius.
Here
I
want
to
focus
on
having
you
know
a
case
where
I
discuss
the
files.
A
A
A
The
idea
that
you
have
teams
that
completely
risk-averse-
and
these
are
the
old
teams
that
start
to
slowly
a
sort
of
the
old
monolithic
cluster
designed
down
right,
where
you
start
to
sort
of
move
at
their
pace
and
not
everybody
else's
pace.
What
services
are
mostly
most
likely
to
impact
others?
A
lot
of
services
that
are
super
high
ops
use
the
network.
A
lot
isolate
those
isolate
those
from
your
critical
services,
don't
share
them
on
the
same
clusters,
even
right.
A
Yes,
you
could
probably
put
them
in
different
nodes
with
affinities
and
taints,
instead
of
go
that
way
too.
But
you
know
that's
only
like
one
dimension
of
this
little
again.
This
multi-tenancy
is
break
them
out,
put
them
somewhere
else
where
they
can
fail
and
impact
them
set
and
impact
only
themselves
and
not
other
services
in
your
business,
particularly
again
the
ones
in
that
really
sort
of
critical
risk
domain,
and
you
know,
last
but
not
least,
it's
all.
A
It's
all
good
to
also
to
talk
about
all
this
and
say
like
we
can
all
architect
this
in
like
a
vacuum.
But
you
know:
what's
your
actual
real
dollar
budget
like
how
much
time
you
want
to
go
and
spend
to
make
this
sort
of
the
most
fault,
tolerant,
reduce
blast
radius
system
in
the
world
right
at
some
point,
you
need
to
draw
a
line
and
say
look.
This
is
actually
what
we
think
it's
good
enough.
It's
probably
the
probably
a
subsequent
failure
where
things
going
wrong.
A
We
go
okay,
we're
gonna,
do
a
little
bit
more
right
and
it's
always
like
a
learning
process.
You'll,
never
get
it
right
the
first
time,
but
but
certainly
trying
to
engineer
this
like
perfect,
completely
fault-tolerant
system
from
day.
One
is
also
not
going
to
work
so
draw
a
line
and
say:
look
again
call
services.
You
know
these
things
have
to
be
multi.
A
Cost
I
need
to
be
able
to
deal
with
these
deal
with
an
adage
of
a
complete
data
center
deal
with
an
outage
of
like
a
complete
cluster
or
a
serious
bug,
and
you
know
this
patch
release
of
communities
right.
Think
about
all
those
things
that
could
go
wrong
just
by
having
a
single
cluster
and
all
of
its
dependencies
and
have
something
that's
a
little
different.
A
So
who
should
do
this
right?
I
strongly
believe
a
dedicated
platform
team
should
run
your
clusters
right,
a
team
that
has
experience
in
actually
doing
this,
whose
job
it
is
to
secure
the
cost
to
upgrade
the
class
to
maintain
the
components.
The
job
is
to
not
write
important
business
logic
for
the
company
right.
That's
what
a
lot
of
the
desn't
to
be
doing
right.
They
have
another
job,
that's
not
maintaining
clusters
and
again.
A
What
ends
up
happening
is
that
when
you
give
people
this
sort
of
joint
responsibility
and
again
instead
of
like
every
every
cost,
sorry
every
team
as
a
cluster,
you
get
this
incredible
tension
of
like
I,
have
all
these
tickets
in
my
backlog
and
I
have
the
annoying
PM's
and
everybody's
like
saying
we're,
gonna
ship,
this
stuff
and
less
and
less
people
are
probably
saying.
Oh,
we
need
to
make
sure
we
upgrade
to
the
latest
version
communities,
because
it
fixes
these
problems.
A
We
need
to
make
sure
that
we
shore
up
our
DNS
configurations
so
on
and
so
forth,
right
if
a
team
is
trying
to
sort
of
become
a
platform
team
and
actually
write
functional
code
for
the
company,
those
things
really
don't
work
in
I've.
Seen
that
fail
a
lot.
What
do
we
ended
up
whatever
actually
wants?
This
particular
case
was
a
team
when
it's
a
sort
of
write
their
own
operator.
A
This
is
just
much
much
better
and
again
those
platform
teens
get
to
see
and
generalize
all
the
problems
that
folks
have
right,
o
teams
having
a
hard
time
because
they're
using
like
the
raw
api's,
which
we
really
don't
want
to
expose
to
sort
of
like
most
development
teams.
They
can
shim
those
right
tooling
that
sort
of
understands
those
cases
they
can
also
look
at.
You
know
again
they
see
a
problem
on.
A
Even
if
the
the
ops
teams
managing
say
20
clusters,
50
clusters,
they
can
see
problems
over
here
and
map
them
to
problems
over
there
when
people
own
their
own
clusters.
That
doesn't
always
happen
right.
How
many
set
of
dev
teams
are
gonna
write
a
fully
fledged
post-mortem
for
their
individual
cluster
failure
and
share
it
with
the
rest
organization
every
time,
but
again
that
goes
back
to
the
fact
that
they
sort
of
tension
between
the
actual
day
job
and
managing
this
cluster,
which
is
really
just
the
means
to
an
end
right.
A
So
again,
you
know
this
or
keep
going
I
think
we
can
probably
be
pretty
honest,
like
there's
a
lot
of
growth
for
communities
to
do
and
a
lot
of
growth,
a
lot
of
folks
to
sort
of
figure
it
out
right
and
again,
just
throwing
it
a
sort
of
a
new
developers
doesn't
always
work
so
just
sort
of
put
a
wrap
on
this.
You
know
what
are
the:
what
are
the
takeaways?
How
do
we
we've?
You
know,
we've
said
like
the
myth
of
the
mono
cluster.
A
We
don't
want
the
same
across
the
design,
but
how
do
we
actually
like
get
to
a
world
where
that's
not
the
case
right?
How
we
convince
others
in
our
organizations
that
this
single,
like
whole
of
the
world
cost
of
design,
is
really
not
what
we
want
the
deer's.
You
should
start
small,
so
focus
on
teams
particular
feel
sort
of
movie
new
communities
early,
start
small
focus
on
team.
That
has
like
a
clear
use
case
that
like
really
works
well
in
communities
right,
it's
like
a
stateless
application.
A
It
makes
it
just
truly
impossible.
The
idea
that,
again,
you
can
have
this
whole
of
company
cluster
that
just
suits
everyone's
workload
perfectly
doesn't
exist.
You'll,
never
finish!
Writing
the
like
PID
for
this
or
the
functional
requirements,
the
mindful
the
blast
radius,
every
time
you're,
you
know,
adding
you
components
or
you're,
adding
things
in
a
share
cluster
think
about.
If
that
fails.
What
else
does
it
take
out?
Think
about?
A
If
it
fails,
if
the
whole
cluster
fails
like
what
is
the
impact
at
everything
else
beyond
that
right
again,
in
many
cases,
having
some
kind
of
degradation
where
you
have
services
split
across
different
clusters
is
probably
better
than
nothing
and
at
the
end
of
the
day
you
know,
committees
is,
might
walk
right.
Let
your
platform
teams
run.
It
I
think
it's
still
important
that
developer
teams
actually
understand
how
it
works.