►
From YouTube: To Infinite Scale and Beyond: Operating Kubernetes Past the Steady... Austin Lamon & Jago Macleod
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
To Infinite Scale and Beyond: Operating Kubernetes Past the Steady State - Austin Lamon, Spotify & Jago Macleod, Google
Operating large distributed systems at significant scale is challenging. Most discussions focus on scalability either at a single point in time under sustained load, or explore challenges related to changes in incoming traffic. But running distributed systems at scale is about more than steady states and transitions between them. What is equally challenging and tends to get overlooked are the operational challenges of running at scale: upgrading many and/or large clusters; deploying applications to and across multiple clusters in a reasonable way; balancing freedom and consistency across multiple teams. In this case study, Google and Spotify share some of the challenges of running Kubernetes at Scale, together with concrete solutions, patterns, and common pitfalls we have found together. Intended for cluster operators and developers from organizations of any size and on any provider.
https://sched.co/UaYk
A
Good
afternoon,
thank
you
all
for
coming
I
would
apologize
that
you're
all
missing,
sunny,
San
Diego,
but
you're,
not
so
going
conference
room.
The
conference
room
is
not
so
bad
today,
so
we're
really
glad
you
made
the
trek
over
to
the
Marriott
and
to
hear
about
how
we're
scaling
kubernetes
at
Google
and
Spotify
and
learning
some
of
the
pain
points
that
we've
had
in
tactics
that
we've
taken
along
the
way.
Just
to
start,
I'm
austin,
I
I
work
in
core
infrastructure
at
Spotify.
B
We
find
that
there's
a
sort
of
a
separation
of
roles
of
the
team
that
manages
the
platform
and
the
teams
that
build
the
applications
that
run
on
the
platform,
and
this
talk
is
largely
targeted
towards
the
platform
managers
or
operators.
There's
some
something
to
learn
for
the
application
developers
as
well,
but
really
this
is
about
when
you
scale
out
across
your
organization
how
to
maintain
some
control
over
that
the
platform
that
you're
supporting.
B
So
we
have
a
few
goals
for
today.
This
was
to
keep
ourselves
focused.
It's
a
big
topic.
So
our
goal
today
to
talk
about
why
it's
important
to
think
beyond
the
steady-state
and
we'll
define
that
and
talk
more
about
that
in
a
minute,
and
the
second
is
to
talk
about
the
the
real
operational
challenges.
There's
a
saying
that
quantity
has
a
quality
all
its
own
and
it
really
is
true.
Operating
at
scale
carries
some
unique
challenges
that
are
not
evident
when
you
spin
up
a
3-node,
kubernetes
cluster.
B
So
it's
important
to
qualify
what
we
mean
by
steady
state,
so
kubernetes
is
itself
about
elasticity
and
reaction
being
self-healing
self
scaling.
So
that's
all
within
the
requirements
of
this
system
and
included
within
the
steady-state.
So
after
you
provision
a
cluster,
you
have
a
few
nodes,
you're
running
some
workloads,
it's
scaling,
pods,
it's
replacing
things
that
die.
That's
all
expected
behavior!
It
is
don't
get
me
wrong.
A
marvel
of
modern
engineering,
it's
incredible
that
it
works
as
well
as
it
does.
But
it's
sort
of
expected
for
this
talk.
B
The
components
on
the
underlying
platform
in
the
steady
state
are
assumed
to
be
stable,
you're,
not
swapping
them
out
at
runtime,
and
so
that
platform
is
somewhat
stable
and
it's
important
to
distinguish
that
even
the
high
end
of
a
predictable
or
mostly
predictable
curve
is
included
within
the
steady-state.
Maybe
you
run
media
outlet
and
you
have
a
daily
curve
or
you
run
a
game
and
peak
users
should
be
on
the
weekends
but
actually
they're
on
Monday
morning.
B
Oddly,
no
one
can
figure
out
why,
but
even
that
even
the
the
peaks
of
Black
Friday,
whatever
you
can
generally
predict
and
be
prepared
for
those
are
included
within
the
steady-state.
And
interestingly,
when
we
talk
about
scalability,
we
tend
to
over
focus
on
this
steady-state.
How
many
nodes
can
a
kubernetes
cluster
scale
to
5000?
B
We've
done
other
talks
where
we
talked
about
scalability,
not
being
a
number
but
really
a
bunch
of
dimensions,
and
the
dimension
that
we
want
to
focus
on
today
is
really
the
dimension
of
time
evolving
over
quarters
and
years,
and
we
talked
about
that.
You
do
this
until
you're
brushing
up
your
resume
to
work
somewhere
else.
B
So
if
folks
see
that
over
your
shoulder,
they
should
think,
but
this
is
a
supporting
a
system
over
multiple
years
living
with
your
past
choices,
understanding
the
trade-offs
and
then
living
with
those
four
years
is
a
interesting
area,
and
so
we
tend
to
overlook
the
operational
challenges.
It's
not
just
about
five
thousand
nodes.
What
happens
when
you
need
to
upgrade
five
thousand
nodes
and
it
takes
an
hour
to
upgrade
every
node.
That
is
suddenly.
B
You
realize
that
what
works
for
three
nodes
is
drastically
different
than
for
many
many
nodes,
so
there's
a
laptop
sticker
bumper
sticker.
That
says
there
is
no
cloud,
it's
just
someone
else's
computer
I.
Don't
think
this
is
actually
well
enough
understood
still
today.
It
actually
is
one
of
the
things
that
kind
of
annoys
me
about
the
term
server
lists.
There
is
always
a
server
running
that
code.
It
is
occasionally
sometimes
often
useful
to
hear
what
that
computer
says
about
running
the
thing
that
you'd
like
it
to
run.
B
So
it's
not
truly
server
list,
it's
just
somewhere
else,
and
even
someone
else's
computers
are
computers
in
this
case.
Fail
they
fail
all
the
time
you
don't
see
it.
Gce
does
magic,
live
migration
of
VMs.
Those
of
you
who
are
pushing
the
boundaries
do
detect
the
infant
decimally
small
bit
there
so
kudos
to
you,
but
mostly
it's
invisible
to
end
users
and
kubernetes
like
we
said.
The
expectation
is
that
it
will
resolve
many
of
these
requirements
for
elasticity
and
self-healing
zones,
even
regions
they
go
down,
they
will
go
down.
We
embrace
that
at
Google.
B
B
So
the
core
question
for
this
session
to
focus
here
is
how
do
you
build
a
globally
available
service
in
a
collection
on
a
collection
of
demonstrably
demonstrated
to
have
been
unreliable
regions
and
zones,
and
that
that
is
the
question
that
you
should
drive
to?
You
should
have
an
understanding
of
that
and
it
will
take
time
to
evolve,
but
that
should
be
the
North
Star
and
you
need
to
think
beyond
these
study
States
over
quarters
and
years
in
order
to
get
there.
So
we
have
two
core
takeaways
that
we
wanted
to
focus
on
today.
B
B
Managing
multi
clustered
deployments,
we
take
it
as
a
given
that
if
you
have
highly
available
systems
globally
available
services
that
that
necessarily
includes
multiple
clusters,
the
most
multi
cluster
will
be
a
very
crowded
space
in
the
kubernetes
community
and
vendor
community
over
the
next
two
to
three
years.
So
there
are
lots
of
choices
but
think
about
your
strategy
and
what
you're
trying
to
get
out
of
it
over
time
before
you
jump.
B
Well,
let's
jump
in
okay,
so
this
is
the
first
part,
the
automating,
the
cluster
upgrades.
Okay,
this
will
take
some
explanation
at
--gk
we
use
a
concept
of
soak
time,
which
is
essentially
cluster
days
or
cluster
years
that
have
accumulated
as
used
by
some
end
user.
So
a
release
candidate
becomes
available.
The
blue
line
after
an
open
source,
kubernetes
version
is
cut
and
a
few
brave
customers
adopt
it.
They
try
it.
They
find
a
bug.
Okay,
we
patch
that
in
open
source,
Coburn
Eddie's
cut
a
new
patch
version.
B
We
make
a
new
release
candidate,
it
accumulates
a
little
more
soak
time
and
then
maybe
the
point
three.
Maybe
the
point
four
release.
It
accumulates
enough
soak
time
to
get
over
the
first
green
line,
which
means
that
it
can
progress
from
our
alpha
channels
or
rapid,
rapid
release,
channel
or
clusters
into
the
GA
or
regular
release
channel.
B
It
is
made
available
first,
not
auto,
upgraded
to
it
accumulates
more
soap
time.
Then
it
becomes
the
default
for
new
cluster
creation
and
then
it
becomes
the
upgrade
target
and
we
will
auto
upgrade
many
of
the
clusters
in
the
fleet
very
carefully
across
multiple
regions
and
zones,
and
so
this
is
the
idealized
soak
time
accumulation
of
a
version
in
a
perfect
world
this
there
would
be
exactly
three
release
candidates.
B
B
The
math
is
not
is
the
math
is
kind
of
against
us.
Those
are
the
lines
of
code
of
kubernetes
over
time.
There
are
a
massive
number
of
lines
of
code,
interesting
spike
here
and
kubernetes.
One
five
was
checking
in
generated
code,
which
we
quickly
realized
was
not
brilliant
and
stopped
doing
that
so
good
for
us,
but
this
was
sig
offs
state
of
security
slide
from
the
contributor
summit
on
Monday
and
it's
showing
that
there
is
just
a
tremendously
growing
number
of
lines
of
code.
B
Interestingly,
not
included
in
this
slide
are
all
the
other
things
that
get
packaged
together
in
a
distribution
of
kubernetes
by
some
vendor
or
yourself,
so
core
dns
or
cube
dns,
maybe
use
calico,
maybe
include
sto.
There
are
all
kinds
of
other
add-ons
extensions
ecosystem
stuff
that
you
include
in
it
and
so
the
if
you
all,
if
we
all,
would
write
all
of
this
code,
which
is
completely
bug
free
and
worked
perfectly
together.
There
would
be
really
no
issue.
B
We
would
be
back
at
the
idealized
version,
but
in
practice
that's
not
what
happens,
and
so
this
is
an
actual
view
of
accumulating
soap
time
for
gke
release.
Candidates
over
time,
and
we
backport
important
security
fixes
and
patents,
two
older
versions,
those
that
are
considered
more
stable
and
open
source
kubernetes,
and
this
includes
the
kubernetes
components
themselves.
There
are
all
these
add-ons.
There
may
be
an
issue
in
a
container
runtime.
There
might
be
a
base
OS
image.
There
are
kernel
regressions.
B
This
goes
all
the
way
down,
and
occasionally
the
automated
testing
that's
in
place,
doesn't
catch
things
in
tests
and
so
only
running
elasticsearch
at
scale.
Under
these
conditions,
with
this
kind
of
throughput
manifest
some
issue,
it's
an
issue
for
all
kubernetes
users,
but
somebody
discovers
it,
and
so
even
older
versions
get
patch
releases
fairly
frequently,
and
you
all
end-users
also
do
things.
We
didn't
anticipate.
It's
really
quite
amazing.
What
a
creative
group
end
users
are,
so
all
of
that
together.
What
I'm
trying
to
drill
into
is
that
the
frequency
of
upgrades?
B
It's
not
once
every
three
months
in
practice,
it's
typically
more
than
that
to
maintain
a
stable
and
secure
environment,
and
we
go
through
thought
exercises
at
Google
about
scalability
and
if
we
think
about
scaling
10x,
you
can
almost
see
how
the
super
effective
engineer
could
work
a
little
extra
and
solve
a
problem.
That's
growing
only
10x,
but
when
you
do
the
thought
exercise
with
a
hundred
X,
you
think.
Oh
okay,
we
have
to
fundamentally
assume
that
there's
a
different
way
of
operating,
so
cluster
upgrades
are
coming
and
things
that
are
frequent.
B
You
need
to
do
you
just
need
to
automate
before
we
move
on.
I
will
point
out,
but
a
very
effective
solution
here
is
to
assign
a
super
sarcastic
kind
of
lazy
but
brilliant
engineer
to
this
task
to
do
a
manual
task
once
every
week
for
four
weeks
in
a
row
and
I
promise,
they
will
automate
that
they
will
die
before
they
do
it
a
fifth
time.
This
has
been
super
effective.
For
me,
I
hope
it
is
for
you
too
okay,
but
there
is
some
good
news.
B
The
good
news
is
that,
historically,
we
have
been
changing
api's
fairly
frequently
over
the
last
five
years.
The
truth
is
adoption
really
outpaced.
Production
readiness
in
kubernetes
I
take
this
to
mean
that
we
created
something
very
useful,
useful
enough
to
assume
that
risk.
But
honestly
there
has
been
risk
and
it's
not
getting
to
a
v1.
Api
is
not
really
about
the
quality
of
the
code.
B
So
Brian
grant
put
this
diagram
together
in
2018,
and
those
things
that
are
read
were
not
v1
api's.
We
invested
pretty
heavily
this
year
and
bringing
forward
those
api's
that
had
stalled
out
at
beta
and
only
a
couple
left
I'm,
not
sure
why
we
didn't
highlight
this,
but
ingress
you
can
see
in
the
upper
right-hand
corner
still
has
a
couple
steps
to
go
and
cron
job
has
been
quite
stubborn
to
move
to
GA.
B
So
those
are
the
two
remaining
ones,
but
the
the
point
here
is
that
looking
backwards
in
time,
I
typically
talk
to
customers
who
are
still
on
1/9
and
they
are
holding
onto
1/9
because
they
were
burned
in
the
1
7
to
1/8
upgrade
because
we
moved
an
annotation
to
a
field
and
we
broke
a
bunch
of
things.
So
I'm
just
encouraging
you
not
to
base
your
upgrade
policy
on
2
year
ago.
Information
look
forward
and
understand
that
this
is
over
multiple
years
and
to
develop
a
strategy.
That's
looking
forward.
B
Okay,
so
to
summarize,
the
good
news
just
about
all
api's
are
GA
or
there's
a
plan
to
get
to
GA
in
the
next
quarter,
and
the
other
interesting
development
is
community
support
around
this
concept
of
supporting
four
instead
of
three
minor
releases
of
kubernetes
so
twelve
months.
So
we
find
that
retail
customers,
for
example,
don't
upgrade
during
q4
very
important,
and
then
they
do
sort
of
clustered
upgrades
at
the
beginning
of
the
year,
and
this
is
a
pattern.
I
would
expect
to
move
forward.
B
The
other
move
on
okay,
so
upgrades
very
frequent
if
it's
very
frequent
make
it
automated
find
some
way
to
automate
it
version.
One
is
just
about
done
and
final
point
being
too
early
or
too
late,
meaning
adopting
a
dot.
Zero
release
is
generally
not
encouraged.
We
found
a
couple
years
ago
before
we
had
release
channels
that
our
newest
customers
were
the
ones
that
looked
at
the
numbers
and
said
the
biggest
number.
It
must
be
the
best
one
and
they've
found
all
the
bugs
in
all
of
kubernetes,
which
was
not
very
fair
to
them.
B
B
So
we
have
a
concept
at
Google
of
haunted,
graveyards
and
haunted.
Graveyards
are
exactly
what
it
sounds
like
it's
a
thing
that
you're
scared
of.
So
you
avoid
it.
You
walk
all
the
way
around.
Instead
of
going
through
it
and
those
things
everyone
has
one
or
two
of
these
things
in
their
system.
Where
are
the
things
that
burned
you
last
time
anytime?
Anyone
touches
that
thing.
It
breaks.
We
just
do
whatever
we
have
to
do
some
smells.
That
might
mean
you
have
haunted.
B
Graveyards
is,
if
you
have
a
really
strong
front-end
team
and
a
back-end
team,
that's
just
getting
started
and
the
front-end
team
is
doing
all
kinds
of
crazy
things
to
get
around
not
having
the
right
API
in
the
backend.
This
is
an
example
of
a
haunted
graveyard,
so
upgrades
are
often
in
a
haunted
graveyard.
Just
if
something
is
scary,
do
it
all
the
time?
B
All
of
this
is
leading
up
to
cultural
changes,
and
the
cultural
change
is
really
dynamic.
Stability.
The
fact
that
smaller
releases
that
are
more
granular
easy
to
rollback,
they
tend
to
have
smaller
incidents
associated
with
them,
and
many
of
the
customers
I
talked
to
who
are
modernizing
to
their
environments
and
so
on.
B
It
really
is
critical
to
instrument
and
have
good
indication
of
health
so
that
when
you
make
these
small
changes,
you
can
correlate
a
change
to
this
thing
resulted
in
an
increase
in
latency
in
that
thing,
and
that's
really
important.
Okay,
so
we're
gonna
go
through
the
next
few
slides
really
fast.
This
is
a
very
deep
topic,
but
I
wanted
to
make
sure
to
put
in
some
very
concrete
guidance.
So
what
is
the
simplest
thing
you
can
possibly
do
to
automate?
B
Your
upgrades
is
just
enable
no
to
auto
upgrade
and
pick
the
rapid
or
the
regular
release
Channel.
It
will
happen
over
time.
Eventually
you
have
one
cluster
or
something
will
happen
to
your
one
cluster.
So
if
you
want
to
get
some
guardrails
around
that,
you
can
create
a
canary
cluster
and
we
publish
what
order
we're
going
to
roll
out
our
changes
across
Google
cloud
day,
1
day
2
day
3
day,
4,
regions
and
zones,
and
you
can
create
a
canary
cluster
in
a
day,
1
region,
our
upgrades
will
land
there.
B
First,
you
have
implemented
a
stack
driver,
uptime
check,
and
you
can
see
that
something
has
gone
wrong
in
your
canary
cluster
and
then
you
just
use
the
maintenance
windows
to
delay
the
upgrade
to
your
production
clusters.
This
is
very,
very
little
work
on
your
part,
but
it
gives
you
just
in
time.
Information
about
upcoming
changes
to
the
platform
also
make
sure
to
automate
the
alerts
too,
depending
on
yourself
to
remember,
to
check
the
canary
cluster
every
certain
day
of
the
week
is
not
from
experience
that
doesn't
work
well.
B
B
This
is
meaningfully
subtly,
but
meaningfully
different
than
configuration
as
code
code
implies
that
there
is
some
conditional
logic,
maybe
even
a
turing-complete
language
that
you're
running
against
your
config
template.
This
is
almost
always
non-deterministic
and
will
result
in
very
strange
outcomes.
So
thank
thinking
of
this,
even
if
your
implementation
has
some
of
that
thinking
of
configuration
as
data
is
really
important.
Then
you
check
that
data
in
to
get
and
you
have
a
source
of
truth
of
what
was
true
at
that
point
in
time,
terraform
is
great.
If
you're
going
to
configure
and
automate
cluster
creation.
B
In
addition
to
automating
your
upgrades,
please
read
terraform
up
and
running
or
some
similar
book.
It
is
a
very
powerful
tool
and
you
can
delete
clusters
by
accident.
I've
seen
it
happen,
it's
a
it's
scary,
but
it's
very
it's
powerful
and
if
used
correctly,
it's
great
use,
customized
or
something
that
does
configuration
overlay
instead
of
this
Turing,
complete
language
parson
and
modify
your
template
and
then
one
project
that's
coming
along
is
config
connector
to
use
kubernetes
resource
model
everywhere.
So
you
just
like
you
created
a
new
pod.
B
You
would
create
a
new
cloud
sequel
or
a
load
balancer
and
so
on.
So
you
can
use
the
same
tools
that
you
manage
your
community's
resources
to
manage
the
infrastructure
itself.
Okay,
quick
note
on
large
node
pools.
Large
node
pools
5,000
nodes,
one
node
at
a
time,
an
hour
it
can
take
week
or
more
to
upgrade
your
node
pool.
This
is
usually
completely
unacceptable.
B
So
a
couple
of
patterns
that
work
well,
one
is
surge
upgrades
where
you
can
specify
up
to
20
nodes
at
a
time
and
the
other
is
creating
a
new
node
pool
and
then
shift
traffic
to
it.
Instead
of
doing
a
rolling
update
to
the
node
pool
protip,
it's
usually
safe
to
skip
a
version
number
for
note
pool.
So
if
you
end
up
a
few
versions
behind
upgrade
your
control
playing
two
versions
and
then
you
can
skip
usually
the
interim
version
of
a
node
pool
it
that
generally
works.
B
B
Please
take
some
time
and
come
up
with
a
global
deployment
strategy
figure
out
how
many
regions
you
want
to
be
in
how
many
zones
it
will
take
you
quarters
to
get
there,
but
just
having
that
roadmap
of
where
you're
trying
to
go
will
truly
help
so
have
understand
the
implications
of
multiple
clusters:
high
availability
across
those
clusters,
how
you're
going
to
shard
users
or
use
those
multiple
clusters,
how
you
do
failover
disaster
recovery
and
special
note
on
networking.
As
a
not
networking
expert
before
this
networking
gets
complicated
quickly.
B
So
don't
leave
that
to
the
end
and
then
say:
oh
so,
once
you
get
to
this
advanced
stage,
you
need
to
do
something
to
simplify
the
experience
for
your
end
users
and
Spotify,
and
has
done
some
amazing
work
to
simplify
the
use
for
their
end
users.
The
customers
of
their
platform
that
they've
built
and
Austin's
gonna
talk
some
of
that
that
specifically
Thanks.
A
Thanks
chica,
oh
you
get
a
major
speak,
clap
very
good.
So,
as
you
said,
we've
been
doing
a
lot
of
work
to
simplify
that
deployment
process
across
many
of
these
clusters
that
we're
creating
so
jumping
in
here
to
our
second
takeaway,
create
multi-tenant,
ephemeral
clusters
across
regions
and
manage
cross
custer
deployments.
So
let's
unpack
that,
starting
with
some
of
the
sort
of
foundational
challenges,
independent
of
what's
possible
for
your
scale
and
and
what
clusters
will
support,
you
don't
want
one
cluster.
A
One
cluster
is
one
single
point:
failure
we've
listed
a
couple
things
here
that
can
happen.
I'll
certainly
call
out
the
last
one
as
Jago
mentioned
it,
and
it
was
actually
keynote
at
Cuba
and
Europe
around
having
terraform
blow
up
and
delete
clusters.
That
does
happen,
and
if
you
only
have
one,
that's
not
good
and
second
and
in
this
one
may
be
obvious,
but
as
you
start
to
have
more
and
more
clusters,
any
difference
in
configuration
of
those
clusters
becomes
really
bad
at
2:00
a.m.
A
when
your
room
is
completely
dark,
except
for
your
laptop
screen
and
you're
trying
to
figure
out
why
everything
is
not
working.
And
then
you
have
differences
in
your
clusters.
That's
a
bad
place
to
be
I'm,
so
automate
that
as
much
as
possible.
But
but
how
do
you
do
that,
as
you
continue
to
scale
and
have
more
and
more
clusters
so
perhaps
you're
bought
in
on
those
challenges,
but
Before
we
jump
to
solutions.
There's
a
couple
of
learnings.
A
Make
it
automated
to
deploy
to
whichever
cluster
makes
sense
and
don't
make
a
developer
deal
with
that
every
single,
but
give
them
the
tools
to
be
able
to
understand
where
a
workload
is
or
where
problems
occurring,
and
secondly,
and
Jagger
already
called
this
out,
but
network
is
complex
and
it's
really
really
hard
to
change
once
you're
at
scale,
and
this
is
a
pain
point
that
we
have
and
are
working
through.
But
it
is
definitely
something
if
you
have
the
the
flexibility
and
freedom
to
deal
with
now.
A
Definitely
do
it
because
you,
you
will
not
want
to
deal
with
that
later
awesome.
So
to
walk
you
through
a
little
bit
of
a
story.
Here
is
a
very
simplified
version
of
our
CI
CD
stack,
so
there's
a
build
system.
It
creates
an
artifact
it
then
that
artifact
is
then
passed
to
tugboat,
which
is
a
very
simple
API.
We
use
to
manage
deployments
if
you've
seen
others
Spotify
talks.
You
may
have
heard
of
this.
A
It
essentially
is
what
was
used
to
actually
migrate
us
to
kubernetes
a
number
of
talks
on
that
that
you
can
find
or
even
see
this
week.
Tugboat
then
passed
the
spinnaker
which
deploys
the
final
artifact
to
GK,
but
what
I
told
you
is
that
you
need
something
that
actually
manages
which
cluster
to
deploy
to
enter
compass.
So
compass
is
a
tool.
A
That's
been
built,
it's
very
similar
if
you
actually
attended
Airbnb
stock
yesterday
to
how
they
manage
deployment
across
clusters,
probably
something
that
should
be
open
sourced
somewhere,
but
it
compass
essentially
manages
where
a
particular
workload
is
deployed
so
just
to
sort
of
land
that
a
step
further.
This
is
a
deployment
UML
file,
which
is
again
if
we
didn't
show
what
we'd
probably
get
boot
out
of
here
this.
This
is
what
talks
to
tugboats.
So
here
you
see
a
simple
file
that
I'll
break
down
this
bit
here
specifies
where
you'd
like
a
service
deployed.
A
A
Then
there's
a
lot
of
deployments
and
things
can
get
a
little
bit
messy,
so
compass
manages
that
compass,
abstracts
and
manages
which
cluster
or
service
is
deployed
to
based
on
different
algorithms
in
one
important
thing
to
call
out
is
having
it
at
this
abstraction
level
allows
that
algorithm
to
change.
So
you
can
start
by
having
random
cluster
deployment
to
any
cluster.
A
That
makes
sense,
and
then
you
can
get
smarter
and
smarter
over
time
choosing
clusters
based
on
pod,
headspace
or
something
else,
but,
as
you
know,
abstractions
are
only
good
until
they
break
and
until
you're
in
that
dark
room
and
it's
2:00
a.m.
and
things
aren't
working.
So
for
that
we
provide
a
couple
of
different
tools
for
observing
service
state
across
clusters
and
allowing
developers
to
quickly
get
in
the
right
context.
A
So
what
I
mean
by
that
is
if
a
service
is
deployed
in
a
particular
region,
the
context
for
that
a
developer
doesn't
have
out
of
the
box
because
you
managed
which
cluster
to
deploy
to
so
here
are
a
couple
of
visualizations
from
our
developer
portal
called
backstage
at
the
top
you'll
see
a
snapshot
of
a
mock
services.
Current
deployment
across
regions
with
an
accompanying
number
of
pots
in
each
region,
so
a
simple
visualizations
they
sort
of
have
the
conceptual
model
for
where
they've
been
deployed
at
the
bottom.
A
You
see
a
bit
more
detail
for
each
pod,
including
its
current
state
and
links
into
the
cloud
console,
so
they
can
reason
more
deeply
about
a
particular
pod.
This
saves
developers
a
ton
of
time
and
avoids
them
having
to
manually
reason
about
where
something
went
wrong
and,
as
I
mentioned,
to
get
into
that
context.
A
So
that
seems
great.
There
are
other
techniques
beyond
what
we
did
here.
These
are
some
things
that
were
looked
at
you'll,
see,
I
mean
a
very
simple
answer:
is
that
can
potentially
lead
to
over
provisioning,
which
may
be
something
you
don't
care
about
at
all
is
deploying
every
service
every
cluster
and
just
sort
of
assuming
things
will
be
fine.
That
is
not
something
that
would
work
for
us.
A
Second,
there's
kubernetes
cluster
Federation
again
something
we
look
at
I
included
the
concept
diagram
on
the
right
to
show
sort
of
the
complication
there
we
again
aimed
for
simplification
as
much
as
possible
and
is
why
we
moved
away
so,
lastly,
building
on
those
alternatives.
These
are
a
couple
of
things
just
to
think
about
as
you're
you're.
Yes,
thinking
about
multiple
clusters
and
deployments,
but
even
as
you're
thinking
about
rollouts
and
in
particular,
cluster
upgrades,
this
Jago
set
start
simple.
A
Don't
over
engineer
your
clusters
in
rollout
machinery,
something
like
compass
and
one
big
bang
like
that
is
never
a
good
idea
and
highly
related
always
be
weary
of
introducing
any
complexity
in
what
you
decide
to
build.
If
your
company
is
anything
like
Spotify,
new
engineers
will
be
there,
different
engineers
will
be
trying
to
understand
systems
and
somebody
who
doesn't
understand
it
or
didn't
build.
It
should
be
able
to
reason
about
it
and
understand
it
or
it
is
too
complex.
A
So
to
recap,
or
two
key
takeaways
when
operating
kubernetes
at
scale,
think
beyond
steady
States
and
in
particular,
automate
your
cluster
upgrades
and
manage
multi
cluster
deployments
in
a
simple
way,
and
with
that
we'll
go
to
infinity
and
beyond.
You
were
definitely
not
getting
out
of
here
without
a
buzz
lightyear
Jeff.
Any
questions.
A
So
they
can
totally
see
where
they
are.
Oh
sorry,
so
the
question
was
for
development
use
cases
how
to
essentially
how
to
developers
see
where
things
have
been
deployed
to
be
able
to
to
debug.
They
absolutely
can
and
actually
what
we
showed
in
the
console,
the
visualizations
there.
They
actually
click
into
the
pods
and
allow
you
to
like
see
them
up
front
and
center.
We've
just
provided
a
light
abstraction
over
that
to
make
it
easier
to
get
into
that
context.
So,
yes,
that
is
100%
available
to
people
just
by
default.
A
B
The
question
was
about
surge
upgrades
and
how
big
a
set
can
you
upgrade
one
of
our
very
first
customers
to
adopt
this
before
it
was
made
public
and
beta
used
this
at
the
11th
hour
before
a
big
launch
and
upgraded
a
large
cluster
and
who
maybe
Kobe's
ear?
He
told
me
that
this
happened.
He
said
I
worked,
it
was
you
know,
2008-2009,
cluster
and
I
said
well.
How
many
did
they
do
at
a
time,
and
he
said
all
of
them.