►
Description
How Google Cloud Hosts and Manages Kubernetes at Scale [I] - David Aronchick, Google
A breakdown of what we do on Google Cloud to make running Kubernetes great, both managing large amounts of Kubernetes clusters, and what we do on behalf of users to ensure everything is running in the optimal configuration. Folks will be able to take away our best practices and apply them to their own installations.
About David Aronchick
Product manager, Google Container Engine, and member of the Kubernetes Product Management group
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
Okay,
I'm
David
Ronn
check
I'm
a
product
manager
for
a
Google
container
engine
I'm,
also
part
of
the
company's
open
source
product
management
group.
I
say
that
because
I
really
want
to
stress
this
is
not
a
Google
project
anymore,
as
you
saw
today,
just
a
litany
of
people
contributing
to
the
project
across
the
world
over
50%
of
contributors
today
come
from
not
Google
and
we're
very,
very
proud
of
that,
and
the
reality
is
that
you
know
by
us
being
part
of
this
open
source,
kubernetes
product
management
group.
A
It's
just
incredibly
important
for
us
to
say
that
we
are
merely
a
vote.
We
really
want
the
community
to
move
forward,
and
so
that's
why
I
stress
it
so
much
I
want
to
give
you
two
caveats,
as
you
start,
which
is
generally
a
terrible
sign
for
a
talk
when
the
person
has
to
give
a
bunch
of
caveats
first,
this
is
a
vendor
talk.
I
do
work
at
Google
I'm,
going
to
talk
about
Google
container
engine,
which
includes
things
like
pricing
and
other
Google
hosted
efforts,
so
that
makes
you
feel
uncomfortable.
A
You
know,
please
see
your
way
out.
The
second
thing
is
that,
and
again
this
is
a
very
funny
thing
to
Startalk.
With
most
of
this
stuff
is
totally
unnecessary
for
you,
Google
container
engine
spins
up
literally
many
thousands
of
clusters,
unless
you
plan
on
hosting
a
service
that
spins
up
many
thousands
of
clusters
which
I'd
love
to
help
you
do
you're,
probably
not
going
to
need
a
lot
of
it.
A
That
said,
we
did
solve
a
bunch
of
really
hard
problems,
help
think
get
things
scalable
in
the
way
that
we
did
it
and
we'd
love
to
share
it
with
you
to
really
show
you,
the
power
of
communities
and
the
power
of
this
scalable
set
up
system
that
we
have.
Okay
without
further
ado
containers,
apparently
they're
a
thing:
it's
pretty
sweet.
A
A
We
are
very,
very
proud
that
that
we
do
not
need
to
make
changes.
Kubernetes
is
so
scalable
as
it
stands
that
we
can
use
Google
container
engine.
We
can
use
vanilla
kubernetes
on
Google
container
engine
today
and
all
the
value
that
you
see
added
here
is
done
by
building
things
around
kubernetes
in
in
the
vanilla,
install
a
lot
of
people.
Ask
you
know
why
is
it
you
know
Google
container
engine
with
a
k',
not
because
we're
German?
It's
because
the
K
stands
for
delicious?
A
No,
it
is
because
it
stands
for
kubernetes
and
there
was
already
a
GCE
Google
compute
engine,
and
so
we
had
to
be
very
crafty.
I
think
I
saw
Joe
betta
there
in
the
back,
and
you
have
him
to
blame
or
crane
one
of
the
other
okay
I
mentioned
previously.
Google
container
engine
hosts
some
of
the
largest
workloads
in
the
world.
The
companies
behind
seven
out
of
the
top
ten
websites
in
the
world
use
kubernetes,
either
in
whole
or
in
part
to
run
their
business,
and
we're
very,
very
proud
of
that.
A
A
It's
a
really
cool
experience.
However,
when
Disney
inventiveness,
the
thing
they
were
most
proud
of,
oh
and
I
should
show
you.
This
is
roughly
what
it
looks
like.
It
doesn't
look
exactly
like
that,
but
this
is
the
box
kind
of
thing
that
that
it
looks
like
now
when
Disney
invented
this.
They
didn't
care
about
the
actual
ride
itself
when
it
came
to
patenting
things.
The
thing
that
they
valued
the
most
the
most
IP
around
this
ride
was
getting
people
in
and
out
of
the
ride.
A
How
do
you
get
40
people
into
a
room
and
simultaneously
get
the
40
people
who
are
already
in
that
room
out
within
30
seconds
now,
I,
don't
know
I'm
scanning
over
here,
maybe
we
have
I,
know
80
people
and
two
doors
like
I
guarantee.
If
I
asked
everyone
to
get
out
of
here
in
30
seconds,
there
would
be
murder
and
that's
not
ideal,
but
they
sold
it
and
it's
you're
kind
of
like
well.
That
seems
really
strange
why
you
know
why
are
they
so
proud
of
that?
A
It's
because
it's
a
really
hard
challenge,
but
it's
a
challenge
only
when
you
get
to
the
scale
of
someone
like
Disney,
where
you
have
to
move
people
in
and
out
every
30
seconds
you
got
to
move.
You
know.
The
flow
of
this
is
just
enormous
for
those
that
are
curious.
The
way
they
solve
it
is
with
three
things:
one.
They
have
a
lot
of
doors
right
so,
for
you
can
see
they're
about
eight
people.
They
have
one
door
for
them.
A
If
we
were
gonna,
do
that
in
this
room,
I
don't
know,
we'd
have
what
20
doors
or
something
like
that.
So
they
have
a
lot
of
doors
then.
Secondly,
they
have
prepotent,
they
don't
basically
precache.
They
have
exactly
as
many
people
as
gonna
fit
into
those
eight
rows
or
those
eight
seats
load
them
up
beforehand
before
the
door
even
begins,
and
then
once
the
door
opens
everyone
flows
in
and
then
they
have
even
more
doors
on
the
other
side,
so
the
people
who
come
in
leave
via
a
different
set
of
doors.
A
It's
a
really
kind
of
fascinating
thing,
and
so
these
people
over
here
come
in
one
door
and
then
the
other
people
go
out
the
other
side,
but
that's
how
they
solve
it,
and
this
is
all
like
IP
that
they
license
all
over
place
and
have
patents
on
it.
Okay,
why
have
I
told
you
this
story
well
in
a
lot
of
ways,
this
is
like
Google
container
engine
Google
container
engine
is
excuse
me,
kubernetes,
very
straightforward.
A
To
set
up
there's
processes
there,
documentation,
there's
the
absolutely
excellent
kubernetes,
the
hard
way
that
Kelsey
Hightower
has
set
up,
and
so,
if
you're
setting
up
one
cluster
or
even
several
clusters,
you're
gonna
solve
as
that
set
of
problems,
and
it's
gonna
be
really
well
Scout.
Now,
if
you're
Google
and
you
are
setting
up
as
I
said,
literally
thousands
of
container
or
clusters,
you
need
to
do
it
in
a
totally
different
way.
Just
like
they
did.
You
know
it's
not
just
two
doors
enter
and
exit
the
room.
You
got
to
set
it
up.
A
So
in
a
lot
of
ways,
go
look
at
an
your
engine
is
start
towards
for
kubernetes
I'm
trying
this
out.
Anyone
could
know
right
all
right,
so
how
did
we
do
this?
What
are
our
you
know,
seventeen
doors
and
pre
lying
lineups
to
set
up?
We
do
basically
three
things
one.
We
make
it
very,
very
easy
to
get
started
through
an
enormous
amount
of
automation
and
watching
things
and
I'll
show
you
how
we
do
that.
A
Second,
we
monitor
your
clusters
and
really
simplify
the
administration
of
your
clusters
on
an
ongoing
basis
and
third,
we
supercharge
your
applications.
So
what
does
this
look
like?
I'm
gonna
walk
you
through
this.
This
is
Google
or
without
Google
container
engine.
These
are
the
steps
you
might
walk
through
in
order
to
set
up
a
kubernetes
cost.
It's
quite
straightforward.
Like
I
said
there
are
many
documents
out
there
that
do
this.
You'll
set
up
a
set
of
resources,
you'll
exchange,
some
certs
and
other
things.
A
You'll
install
the
necessary
packages,
you'll
make
sure
everything
settles
and
join
the
things
together
and
then,
of
course,
you'll
go
and
you'll
tie
it
all
together
with
your
networking,
and
it
takes
you
know
a
few
minutes
and
and
in
particular,
I
really
want
to
call
out
the
work
that
the
folks
for
kube
ADM
have
done.
It's
really
transformative
and
made
even
this
much
much
easier.
However,
with
Google
container
engine,
this
is
what
it
looks
like
you
hit
a
button.
It
seems
pretty
straightforward
right
behind
the
scenes.
A
A
And
then
you
have
the
users
project,
that's
where
it's
all
yours
and
you
can
see
everything
in
there.
Okay,
so
number
one.
We
build
a
disk
image,
pretty
straightforward.
We
build
it
to
make
a
pre
cache
it
with
all
the
things
necessary,
for
you
know,
downloading
tools
and
also
strip
out
all
the
things
that
aren't
necessary
on
Google
compute
or
exceed
me
container
engine.
We
use
a
disk
image
called
costs
which
is
container
optimized
OS.
It
is
optimized
for
running
containers
on
Google
cloud.
It
is
fairly
bespoke
to
Google
cloud,
but
it
is
open-source.
A
It's
based
on
chromium
and
you're,
welcome
to
download
it
and
take
a
look
at
it.
Second,
we
create
a
set
of
credentials,
so
we
don't
do
the
credentials
as
the
master
comes
up,
whereas
the
notes
come
up,
we
pre
build
them
and
store
them
in
a
metadata
server
that
we
have
on
our
private
infrastructure.
Third,
we
go
and
we
create
a
master
VM.
Actually
I'm
gonna
fudge
a
little
here.
We
actually
do
a
bunch
of
things
in
parallel,
so
just
bear
with
me,
but
for
now
we're
just
gonna
say
we're
doing
them.
Serially.
A
We
asked
GCE
to
create
a
cluster,
it
goes
and
it
grabs
the
image
that
we
had
pre-built
and
it
spins
up
a
master
VM
that
master
VM
has
a
set
of
packages
on
it,
an
API
server,
docker,
cubelet
and
some
other
admin
stuff.
The
admin
stuff
is
some
of
the
stuff
that
we
use
you'll,
see
later
for
making
sure
things
are
up
and
running,
taking
fcd
snapshots
reporting
back
a
variety
of
things
like
that.
A
A
We
have
a
whole
bunch
of
flags
that
we
build
into
the
process
here
that
we've
pre-built
and
are
sure
that
they're
gonna
run
great
on
Google
container
engine
and
as
we're
spitting
that
up,
we
grab
that
metadata
or
the
assert
from
the
metadata
server
that
we
had
set
up
previously.
Okay,
fourth,
we
create
a
mig
who
knows
what
it
meg
is,
who
doesn't
know
what
a
mig
is.
All
right,
who
knows
what
an
auto
scale
group
is.
A
A
Now
on
that
I'm
gonna
blow
out
one
here,
you
create
a
node
VM
on
that
you're
gonna,
pull
from
this
same
image
that
you
use
to
create
your
excuse
me,
your
master
VM,
with
slightly
different
packages
involved
and
you're,
obviously
slightly
different
admin,
stuff
and
you're
also
going
to
pull
from
that
same
sir,
because
you
need
that
cert
and
all
those
credentials
to
talk
back
to
the
master,
and
then
you
do
that
for.
However
many
nodes
you
have
in
the
cluster
okay,
we
also
on
Google
container
engine
label.
A
The
reality
is
under
the
hood:
they're
just
implemented
using
labels,
but
it's
actually
really
really
powerful
stuff
and
I'll
show
you
why,
in
a
second
five,
we
do
networking
magic,
so
every
node
needs
to
be
able
to
talk
to
each
other.
We
do
that,
then
we
create
a
whole
set
of
ciders
for
each
of
the
pods
to
be
able
to
talk
to
each
other.
Then
we
create
a
whole
bunch
of
advanced
routes
for
the
master
to
be
able
to
talk.
So
obviously
it
gets
kind
of
complicated
6.
A
We
do
nothing,
we
sit,
we
let
things
settle.
We
let
that
settle
for
a
certain
period
of
time
make
sure
everything
has
come
up
and
is
starting
to
report
healthy
and
7.
We're
done
waste.
We
report
back
to
the
customer
that
we're
ready.
So
what
happens?
Next
you
see
a
green
checkmark.
Well,
that's
it
under
the
hood.
Everything
happened
there
and
that
usually
happens
about
3
to
5
minutes.
A
So
that
is
creating
a
cluster
and
again,
our
goal
here
is
to
make
it
so
easy
that
you
know
with
a
single
API
command.
You
can
spin
up
a
cluster
at
your
demand,
and
so,
if
you
want
to,
if
you
want
to,
you,
know,
spin
one
up
for
dev
test,
if
you
want
to
spin
one
up
just
for
you
know
to
test
out
a
new
version
of
kubernetes
or
you
know,
you
have
a
brand-new
zone,
it's
all
the
same.
The
same
API
works
everywhere.
A
Ok,
so
that's
creating
a
cluster
that
I
mentioned
was
admitting
your
cluster
and
it
basically
breaks
down
to
roughly
about
four
things.
One.
What
do
you
watch
and
how
do
you
watch
it
too?
How
do
you
upgrade
the
master
3?
How
do
you
upgrade
your
nodes
and
4?
How
do
you
repair
it
and
you
can
see
a
lot
of
the
steps
there.
You
know
monitoring
is
pretty
straightforward,
upgrading
masters
and
nodes
very
similar,
but
there's
some
steps
we
take
for
through
automation
and
finally
repair.
A
Unfortunately,
that's
basically
the
same
as
upgrade
for
now,
so
with
Google
container
engine.
On
the
other
hand,
we
do
it
all
for
you,
we're
gonna,
monitor
things.
We're
gonna
upgrade
your
mastered,
upgrade
your
nodes
is
a
one-click
operation
and
repair.
Unfortunately,
it's
complicated
I'll
get
to
that
in
a
second.
A
So
with
monitoring,
we
have
very
clean
endpoints
for
watching
all
the
elements
that
we
consider
to
be
part
of
a
healthy
cluster.
First
there's
a
wack
component
status
on
the
master
vm
you
can
use
this
today.
We
watch
that
we
watch
all
of
the
system,
services
and
record
whatever
is
going
on
there,
and
then
we
watch
all
of
the
individual
notes
and
the
MiG
which
has
its
own
health
point,
and
we
can
watch
that
as
well.
A
Then,
if
something
bad
happens,
an
alert
comes
in
Google
on
after
a
certain
period
of
time
and
I'll
get
to
this
in
a
second
Google
on
call
swoops
in
and
she
takes
a
look
sees
what's
going
on
figures.
It
out
fixes
the
server
and
off
it
goes.
So
that's
monitoring
your
server
and
again
we
take
care
of
it
for
you.
A
As
far
as
upgrading
your
masters
are
concerned,
it
looks
like
this,
so
the
first
thing
you
do
is
we
shut
down
your
master.
The
state
at
CD
is
stored
on
a
persistent
disk
external
to
the
master.
So
it
doesn't
matter
we
can
just
get
rid
of
it.
However,
we
do
one
more
thing
which
is
very
important,
which
is
in
Google
compute
engine
you
have,
the
the
master
has
a
static
IP
and
the
static
IP
is
saved.
A
It's
not
returned
to
a
pool,
it's
saved
off
for
reuse
later,
so
we
shut
down
the
master,
the
PD
is
there:
it's
got
the
state,
we
a
new
master
and
make
sure
that
it's
up
to
date,
we
reattach
the
PD,
and
then
we
restart
the
API
server.
Just
like
we
were
creating
a
brand
new
cluster,
including
pulling
down
the
same
cert
that
you
had
before
and
then
we're
done
so
pretty
straightforward.
We
do
this,
for
you
very
very
regularly.
A
Normally
we
track
kubernetes
open
source
release
within
you
know
three
to
five
days
and
it's
automatically
taken
care
of
for
you.
Now
you
may
be
asking
how
do
the
nodes
connect,
since
this
is
a
brand
new
master?
That's
why
we
save
the
IP
address
same
IP,
address
same
cert
must
be
the
same
master,
I'm
good
and
that's
how
we
take
care
of
that.
Okay,
by
the
way
I
should
say:
if
anyone
has
any
questions,
you
feel
free
or
I
guess
find
me
afterwards.
It's
a
pretty
big
room
or
you
could
I
guess
ask
you.
A
Yeah,
okay,
so
now
how
do
we
do
nose?
Well,
the
way
you
do
nodes
is
we
rely
on
the
MiG
as
our
first
step.
So
in
this
case,
what
we
say
is
we
change
the
MiG
template
so
remember,
the
MiG
is
just
a
description
of
what
a
vm
should
look
like
and
if
you
change
it,
but
the
VMS
are
still
running,
nothing
happens.
So
what
we
do
is
we
change
the
big
template
and
we
say:
well,
you
were
spitting
up
1.5
nodes.
Now
you're
gonna
spend
up
1.6
and
then
we
pick
a
node
at
random.
A
A
The
mid
says:
oh
I'm,
down
on
person,
I
got
a,
you
know,
come
back
and
it
decides
to
create
a
new
one
and
because
the
MiGs
template
was
updated,
it
now
creates
a
1.6,
and
then
we
repeat
that
a
lot
so
one
at
a
time
randomly
we
will
pick,
may
nodes
and
pull
them
out.
So
that's
the
standard
that
we
use
today,
however,
many
customers
want
to
do
a
more
gradual
process
where
they're
a
little
bit
more
in
control.
A
Now
we're
going
to
get
into
node
pools,
which
we
offer
as
a
feature
on
Google
container
engine.
So
in
this
case,
what
you
do
is
you
go
and
create
a
new
node
pool?
It's
a
one
line
command.
You
say
I'd
like
to
create
a
node
pool
and
I'd,
like
the
version
of
crewman
Eddie's
that
I'm
running
in
that
node
pool
to
be
in
this
case.
One
point:
six,
the
MiG
says:
oh
I'm
supposed
to
have
six
in
here:
I
have
none
so
I'm
gonna
create
six
new
nodes
and
then
we
go
to
the
old
end.
A
A
You
can
prevent
your
existing
workloads
from
ever
scheduling
to
something
like
that,
but
by
doing
this
you're
in
a
single
kubernetes
cluster-
and
it
gives
you
the
power
to
test
it
out,
you
can
do
Canaries,
you
can
do
in
a
sample
testing
or
you
can
just
do
straight
blue-green
and
split
between
the
two
and
just
watch
your
logs
once
you're
happy
and
everything
is
working
properly.
You
go
and
you
peel
off
those
nodes.
One
time
just
like
you
saw
before,
and
then
you
destroy
the
MiG
and
presto
you've
you're
up
with
no
down
time.
A
Okay,
so
that's
upgrades.
Let's
get
to
repairs,
repairs,
there's
nothing
I'm
afraid
until
now,
so
we
are
proud
to
announce
we're
proud
to
announce.
As
of
December,
we
had
auto
upgrades
so
for
your
nodes,
which
you
saw
there
and
as
of
just
three
weeks
ago,
we
launched
Auto
Repair
into
beta
and
I'll
walk
through
what
these
look
like,
the
the
two
are,
you
know,
like
I,
said:
I,
oughta,
whup
grades
simply
watches
your
nodes
ensures
that
on
a
regular
basis,
they
are
kept
up
to
speed
exactly
the
same
way
as
your
master
does.
A
It
is
a
fairly
coarse
tool
right
now.
It's
it's
new.
We
don't
have
things
like
windows
and
things
like
that
for
upgrades,
so
you'll
probably
want
to
use
use
auto
upgrade
in
states
and
places
where
they
aren't
stateful
apps
and
things
can
die
at
any
time.
That
said,
we've
heard
lots
of
feature
requests,
and
this
is
going
to
advanced
very
very
quickly
over
time
with
repair,
we're
gonna,
watch
your
nodes
and
take
repair
action
on
them
on
your
behalf,
so
enabling
auto
upgrade.
You
can
see
they're
incredibly
straightforward.
A
This
is
the
command
line
for
enabling
it
you
can
do
it
either
on
a
new
cluster
or
on
a
new
node
pool.
So
you
can
try
it
out,
as
I
was
showing
you
earlier
and
enabling
it
on
a
new
cluster
in
the
UI
is,
is
literally
a
drop
down
for
repair.
These
are
the
things
that
you
watch
while
we're
doing
repair
and
the
repair
communication
is
also
communicated
to
the
masters.
A
So
if
a
node
begins
to
misbehave,
the
new
master
can
take
action
and
say
like
I,
don't
want
you
to
schedule
anything
new
to
that
now
to
be
clear,
there
are
instances
where
we're
not
gonna
be
able
to
do
that.
For
example,
if
your
kernel
is
panicking
or
there's
a
network
partition,
and
we
literally
can't
communicate
with
the
node
there's
nothing
we
can
do,
but
for
many
cases
this
will
work
great,
so
enabling
auto
repair
looks
just
like
Auto
upgrade
very,
very
straightforward.
A
You
can
do
it
either
on
a
new
cluster,
and/or
node
pool
or
an
existing
one.
It's
just
an
attribute
and
then
for
the
UI
same
deal.
Okay,
so
we're
through
creating
clusters
through
monitoring
clusters.
Let's
show
about
running
your
applications
without
Google
container
engine
there's,
basically
four
big
things
that
you're
going
to
want
to
do
start
your
application.
That's
easy:
that's
taken
care
of
by
kubernetes,
but
there
are
things
that
you
can
do
there,
configuring
your
load,
balancer,
you
need
to
get
track
network
traffic
to
them.
A
You'll
need
to
wire
up
your
applications
for
logging
and
wire
up
your
applications
for
monitoring
with
Google
container
engine.
It's
the
same
as
doing
it
yourself,
except
you
get
a
an
enormous
amount
of
flexibility
here.
So
in
this
case
you
can
dictate
the
type
of
node
you
like
in
any
various
locations.
For
example,
persistent
disk,
you
could
ask
for
SSD,
you
can
use
preemptable
VMs
if
you
have
batchi
workloads
that
don't
mind
being
killed
in.
You
know,
30
seconds,
you
can
do
that.
A
You
can
custom
machine
types
where
maybe
you
have
an
enormous
amount
of
RAM
enormous
amount
of
CPU
that
shape
is
available
to
you.
You
can
use
multiple
node
pools
and
that's
literally
a
checkbox
for
selecting
how
many
node
pools
you
can
test
out
alpha
clusters.
Where
will
spin
up
a
cluster
with
alpha
api
is
enabled,
and
it
will
automatically
use
it
that
those
clusters
are
not
upgradeable
and
they
will
be
killed
in
30
days,
but
they're
perfect
for
testing
things
and
I
want
to
highlight
auto
scaling.
A
So
auto
scaling
comes
in
two
forms:
there's
auto
scaling
your
pods
and
there's
always
scaling
your
knows.
Auto
scaling!
Your
pods
is
something
that's
supported
today
and
open-source
through
the
horizontal
pod
autoscaler.
You
can
see
how
you
do
this
here,
using
the
straight
cube,
control
command,
you'd
like
to
auto
scale
based
on
CPU
right
now.
It's
just
CPU
and
memory.
Custom
metrics
are
coming
shortly.
On
additionaly
google
container
engine
does
publish
arseny
kubernetes
publishes
the
signal
for
adding
additional
nodes
to
your
cluster
and
it's
up
to
the
individual
cloud
provider
to
support
that.
A
Of
course,
we
do
surprise
that
support
that
on
Google
container
engine,
where
it
will
scale
up
if
you're
out
of
CPUs,
that
is
let's
say
you
have
1010
pods
that
are
trying
to
schedule
against
10
CPUs
and
you
add
an
xi.
The
system
will
report,
it
is
needs
an
additional
CPU
and
it
can
take
action
on
that
or
scale
down
and
again
down
is
a
very
gradual
one,
we're
very
very
sensitive
to
not
doing
that
too
quickly,
because
we
want
make
sure
that
things
don't
die.
A
So
if
you
go
and
you
test
this
and
you
leave
it
up
for
around
20
minutes
and
it
doesn't
scale
down,
you
think
it's
broken.
No,
it's
not
broken
just
be
patient
and
you
can
see
how
to
enable
that
there
and
then,
like
I,
said
custom.
Metrics
will
be
coming
soon,
so
you
could
scale
up
based
on
things
like
queue
length
or
you
know,
free
handles
on
your
disk
or
something
like
that.
A
Okay,
so
that's
creating
a
cluster
as
far
as
wiring
every
node
out
there,
your
your
network
traffic.
This
is
where
Google
Cloud
really
begins
to
shine
so
Google
container
engine.
Obviously,
as
you
publish
services
automatically
connects
to
Google
cloud
load
balancer,
we
create
services
on
your
behalf
on
the
port's
that
you
specify
take
care
of
all
that
for
you,
but
we're
now
getting
into
real
magic,
and
that
is
where
you
talk
about
federated
ingress.
A
So,
as
everyone
is
anyone
familiar
with
Federation,
not
a
ton
of
people,
the
idea
buy-in
Federation
is
that
you're
able
to
take
multiple
clusters
and
have
them
join
together,
such
that
you
can
control
them
all
via
a
universal
control
plane
that
sits
over
the
top.
That's
not
spreading
workloads
across
each,
but
you
have
a
single
API
that
allows
you
to
control
all
these
clusters
simultaneously.
This
is
incredibly
powerful.
This
is
how
we
operate
internally
to
Google
and
we'll
really
a
now
enable
you
to
work
against
things
like
multiple
zones.
A
Maybe
you
want
to
spread
your
workloads
between
an
Amazon,
Google,
Cloud,
digitalocean,
Azure,
so
on
and
so
forth,
so
that
you
have
more
than
one
cloud
available
to
you,
adding
your
on-prem.
These
is
all
possible
and
it
makes
it
much
much
easier
to
accomplish
that.
Federated
ingress
is
where
you
issue
a
single
command
across
all
of
the
clusters
under
your
control
and
have
it
have
them
all,
be
able
to
handle
the
inbound
traffic
and
route
them
properly,
and
let
me
show
you
how
crazy
magical
this
is
so.
A
A
Third,
you
create
a
Google
end
point.
This
is
how
you
do
it
on
Google
cloud.
You
simply
say:
I'm
gonna
create
this
IP
address.
It
is
global
irresolvable.
This
is
the
name
of
it.
In
this
case,
it's
called
my
ingress
and
I.
Add
a
flag.
Fourth
I
create
an
application
and
a
service
and
I
hand
that
to
the
Federation.
So
in
this
case
I
might
say
like
I
want.
You
know
five
nginx
pods
spun
up
across
all
of
these
various
clusters.
A
This
is
how
you
would
do
that.
Just
like
you
would
against
a
single
cluster.
You're
not
doing
to
getting
upset
Federation
endpoint
and
fifth,
you
create
the
ingress.
The
ingress
is
very,
very
easy.
You
can
see
here
it's
almost
exactly
like
creating
a
service
you
could
see
there.
All
you
do
is
create
the
endpoint
and
then
the
most
important
line
of
this
one
right
here,
you're
adding
an
annotation
to
say
this
should
use
the
global
static,
IP
address
name
and
that's
it.
You
now
have
a
single
global
IP
address.
A
It
is
resolvable
from
over
a
hundred
pops
that
are
googles
and
no
matter
where
a
user
contacts
that
IP
address
they
enter
via
that
pop
and
they
are
routed
via
Google's.
Don't
you
know
private
backbone
to
the
cluster
that
is
most
appropriate
for
you,
so
I
mean
it
looks
like
this.
It's
crazy
and
it's
magical
and
it
really
works.
A
Ok,
so
that's
federated
ingress,
that's
enabled
on
Google
cloud
are
consumed
me
on
Google
container
engine
and
the
reality
is
it's
nothing
about
kubernetes
or
Federation?
That
makes
the
difference
there.
It's
the
fact
that
it's
kubernetes
running
on
Google
cloud
that
makes
all
the
difference
and
then
finally
wiring
your
applications
up
to
logging
as
well
as
wiring
your
applications
up
to
monitoring
very,
very
straightforward.
You
can
see
they're
literally
two
check
boxes
to
enable
both
and
you're
using
stackdriver
monitoring
and
what
it
does.
A
Is
it
introspect
into
your
cluster
and
figures
out
a
collects
all
the
logs,
whether
or
not
it's
system
whether
or
not
its
default
logs
just
standard
out
out
it
understands
things
like
cluster
names,
so
you
can
do
multiple
clusters,
multiple
zones,
all
built
in
there
by
default
and
a
single
aggregation
point:
okay,
but
wait
there's
more
an
additional
thing
which
I'm
not
going
into
here
is
all
the
ways
that
we
actually
roll
these
things
out
and
manage
them.
Let
me
put
in
a
plug
for
the
Google
sre
book.
A
This
is
all
out
there
in
the
public
and
in
fact
it's
a
free
down
up.
You
don't
even
have
to
download
it.
You
can
just
read
it
online,
but
it's
it's
all
the
ways
to
do
things
like
safe
rollouts.
So
when
kubernetes
has
a
new
release,
how
do
we
actually
roll
that
out
to
all
these
clusters?
Well,
we
have
a
testing
and
a
staging
environment
for
those
clusters
internally
to
see
what
happens
when
we
roll
these
out.
We
do
canary
rollouts
of
single
clusters
to
ensure
that
things
are
operating
properly.
A
We
do
zonal
and
exponential
rollouts
and
that's
extremely
important
where
we
say:
okay,
we're
gonna,
do
it
to
0.1
percent
and
1
percent,
10
percent,
and
so
on,
and
so
forth
and
we'd
also
do
canary
pools,
so
not
just
single
clusters
but
in
fact,
entire
pools
of
clusters,
and
this
second
part
is
incredibly
important
when
you're
dealing
with
many
thousands
of
clusters
like
we
are,
you
have
to
come
up
with
an
entirely
new
alerting
scheme.
Probably
the
first
thing
that
we
have
to
do
is
not
panic.
A
When
we
get
an
alert
was
something
you
know
was
a
cluster
just
transiently
down.
Was
it
a
network
partition?
You
know.
If
you
see
a
single
master
that
fails,
you
don't
want
to
go
and
reboot
all
math
across
the
entire
world.
You
need
to
understand
exactly
what's
going
on,
you
also.
Do
it
look
things,
look
at
things
in
a
much
more
holistic
way
when
it
comes
to
how
you
group
things
together.
A
So,
for
example,
if
all
the
clusters
in
a
single
zone
stop
reporting
properly,
it's
actually
probably
not
your
fault,
it's
probably
something
due
to
with
networking
or
disk
or
compute
what
happens
if
it's
just
clusters
of
his
version
that
we're
having
trouble
you
know
we
can
go
and
investigate
that
too.
This
really
short
circuits
a
lot
of
the
like
noise
and
improves
the
signal-to-noise
ratio.
When
it
comes
to
reporting
and
monitoring,
you
can
see
an
example
here.
Oh
I
do
want
to
reiterate:
please
do
go
and
look
at
the
gosari
book.
A
A
They
were
able
to
handle
Black
Friday
with
zero
outages
down
times
or
interruption
Duggal
container
engine,
and
this
is
really
what
we're
hoping
for
more
than
anything.
If
we
fade
into
the
background-
and
you
pay
no
more
attention,
we
will
consider
ourselves
a
great
success,
and
all
this
can
be
yours
for
none
of
the
above,
though,
a
shiny
silver
dollar
do
whoever
figures
out
what
those
numbers
are:
the
low
low
price
of
fifteen
cents
an
hour
per
cluster,
and
that's
only
if
you
have
more
than
six
notes.
A
So
this
is
very,
very
amenable
and,
as
I
said
earlier,
you
know
it's
basically
how
much
your
time
is
worth.
I
should
note
here
for
some
important
point.
This
does
not
include
the
cost
of
running
your
nodes,
any
network
usage,
any
disk
usage
and
things
like
that
beyond
the
VM
itself.
But
obviously
this
is
all
you
need
for
creating
managing
and
running
your
cluster.
So
in
summary,
kubernetes
on
google
container
engine
you
get
pure
upstream
kubernetes,
no
Forks,
no
changes
with
operational
excellence
and
the
power
of
Google
cloud
platform.
A
The
key
is
by
using
Google
container
engine,
you
get
production
grade
kubernetes
with
one,
and
it
really
enables
you
to
focus
on
your
applications
rather
than
the
administration
of
your
cluster
and
and,
like
I,
said
all
this
plus
no
lock-in.
You
can
take
your
cluster,
you
can
move
your
applications
anywhere
else.
Obviously
we
can't
guarantee
the
same
performance
and
criteria
and
node
sizes
and
things
like
that.
But
as
far
as
kubernetes
is
concerned,
it
still
runs
great,
and
that
is
everything
I
love
saying
this
slide,
because
this
is
where
we
all
live.
A
A
That
is
a
deep
and
internal
thing
that
I
am
happy
to
talk
to
you
about
offline
I,
won't
say
your
name.
Please
come
up
here
and
I'll
talk
to
it
like.
Like
anything
you
see
here,
we
have
a
deep
set
of
internal
scripts
that
we
use
to
manage
the
thing
as
we
move
over,
and
it
looks
a
lot
like
you
saw
here.
So
you
know
it's
taking
things
offline,
it's
running
the
process
it's
reattaching
later
and
doing
that
in
an
elegant
way.
A
One
of
the
biggest
concerns
we
have,
of
course,
is
how
do
we
roll
back?
If
something
failed,
how
do
we
recover
something
failed?
And
it's
just
the
reality
of
running
production
systems
like
this?
Where
are
the
slides,
I'll,
post
them
somewhere?
Don't
know
where
GCE
virtual
instances
are
hosted
in
containers?
What
is
this
commingling
between
GC
and
GK?
You
look
like
okay,
so
that
is
true.
A
Everything
at
Google
runs
inside
a
container
now
they're,
not
containers
that
you
might
see
they're
our
own
internal
containers,
but
they
are
in
fact
running
in
containers
on
our
version
of
kubernetes,
which
is
called
Borg.
So
you
have
the
stack,
looks
like
you
know:
Borg
equivalent
container
GCE,
VM
kubernetes.
You
know,
then
additional
container
on
top
of
it.
A
A
What's
with
upgrading
cluster
breaking
standards,
for
example,
moving
pets
at
the
stateful
set
pets
set,
if
I'm
not
mistaken,
so
we
have
in
the
kubernetes
project
a
dip
dum
around
what
is
allowed
for
transitioning
between
versions
and
how
long
we
will
maintain
old
versions
if
you're
an
alpha
I
want
to
stress
this,
and
this
is
a
kubernetes
decision
if
you're
an
alpha.
Nobody
makes
second
any
guarantees
now
conveniently
unless
you're
running
an
alpha
cluster
on
kubernetes
or
excuse
me
on
Google
container
engine.
A
You
don't
have
to
worry
about
that,
because
Google
container
engine
does
not
support
alpha
clusters,
so
there
are
no
alpha
features
plus
or
minus.
There
are
no
album
features
running
on
Google
container
engine,
so
you
shouldn't
have
to
worry
about
things
that
are
that
large
and
breaking.
If
something
like
that
that
was
large
and
breaking
come
on
came
along,
we
would
follow
that
lead
of
whatever
the
kubernetes
community
decided.
For
example,
they
said
this
object
is
going
to
have
a
six-month
deprecation
policy.
A
So
you
shouldn't
way:
I
know
that
there
it
is
well
known
that
we
run
everything
inside
Google
container
engine
as
orcses
me,
everything
inside
Google
as
a
cluster
or
X's
mate
in
a
container,
but
you
can't
think
about
Google
container
engine
as
having
any
visibility
into
that
underlying
infrastructure.
It's
just
totally
totally
separate.
Think
of
the
fact
that
it's
running
inside
a
container
has
purely
an
implementation
detail.
It
could
be,
you
know,
gremlins
from
Mars
running
it
for
all
that
matters.
A
Have
you
thought
about
management
and
we
have
so
sorry
I'm
reading
it
and
answering
right
now.
Have
you
thought
about
running
the
master
inside
a
kubernetes
cluster
instead
of
a
dedicated
VM?
Yes,
we
have
very
deeply
investigated
doing
self-hosting
kubernetes,
as
you
see
here
so
kubernetes
on
topic
communities.
We
continue
to
investigate
it
and
you
know
I
still
think.
Then
we're
gonna
make
that
decision
eventually,
but
you
know
it's
again
just
think
of
it
as
an
implementation
detail
to
some
degree
it
would
just
be.
A
You
know,
for
our
own
satisfaction
that
we
were
running
through,
but
Nettie's
all
the
way
down
yeah
there
there
was
only
one
master
in
the
cluster.
Yes,
there
was
only
one
master
in
the
cluster.
This
is
you
know
what
we
call
regional
clusters
is
something
we
are
constantly
investigating.
If
you
have
requests
for
regional
clusters,
we
are
happy
to
talk
to
you
and
please
come.
Please
do
come
talk
to
me.
A
So
you
know,
regardless
of
whether
or
not
there's
one
or
multiple
clusters
we
commit
to
having
that
SLA
in
place,
and
so
you
shouldn't
be
worried
that
there's
only
one
we'll
take
care
of
it
for
you
and
I
think
that's
it
any
I
missing
anything
here.
Oh,
can
you
choose
the
container
engine
to
use
not
yet
we
were
waiting
for
1.6
to
support
CRI,
which
has
just
landed,
and
we
are
deeply
investigating
whether
or
not
to
use
rocket,
whether
not
to
offer
rocket
high
per
container
d
docker
all
these
various
things
as
options.