►
From YouTube: Kubernetes in Action: Examples from Google Container Engine by Aparna Sinha & Robert Bailey, Google
Description
Kubernetes in Action: Examples from Google Container Engine - Aparna Sinha & Robert Bailey, Google
Kubernetes deployments on Google’s cloud use the same underlying objects as on-premise, but with cloud provider specific networking, storage and services. This talk highlights the use of Kubernetes in Google Container Engine for different types of applications - Gaming, IoT and Analytics. We will walk through the generic architecture for each application type as well as specific implementations for illustration. Upgrading a Kubernetes cluster for greater scale, setting up multi-zone clusters for higher availability and using Ingress with L7 LB for global load balancing will be some of highlights.
About Robert Bailey
Robert has been working on Kubernetes for more than 2 years and was one of the founding members of the Google Container Engine team. Prior to Kubernetes, he was a Site Reliability Engineer helping teams around Google launch new products and services.
About Aparna Sinha
Aparna Sinha leads the product management team at Google for Kubernetes. Prior to Google, Aparna has worked in enterprise software for 15+ years. She was previously Director of Product Management for NetApp’s manageability software where she also led development of storage plugins for Oracle, VMware, Unix and Windows. She holds a PhD in Engineering from Stanford and has several publications from her research work as well as a patent at Google on Android IoT protocols.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
Good
afternoon
or
good
morning,
everyone
sorry
we're
getting
started
a
little
bit
late.
Our
microphones
disappeared
so
but
we're
ready
now
so
I'm,
a
pronoun
Sena
product
manager
for
kubernetes
and
Google
container
engine
at
Google,
and
we
have
Robert
Bailey
he'll,
be
joining
me
for
this
talk,
he's
a
staff
engineer
on
Google
container
engine
as
well.
A
So
I
think
you
know
everyone
here
knows
that
kubernetes
is
a
platform
that
runs
across
different
different
types
of
clouds
and
has
broad
industry
adoption.
Obviously
this
was
something
that
was
a
project
that
initiated
at
Google
and
is
open
source
and
has
a
booming
community
around
it,
but
also
there
is
a
hosted
version
of
kubernetes
at
Google
on
Google's
cloud,
and
that
is
what
we
call
Google
container
engine
and
it
kind
of
serves
as
a
reference
implementation.
A
A
So
the
first
version
of
kubernetes
was
released
in
July
of
last
year
and
I
actually
followed
after
17
business
days,
and
that
was
some
of
the
feedback
we
got
at.
That
time
was
Wow.
It
takes
a
little
bit
of
time
to
get
this
and
then,
if
you
look
at
the
sequence
of
releases,
this
is
work
that
we've
been
doing
with
the
community
to
accelerate
the
the
release
train
it.
You
know
after
the
the
first
release.
A
A
A
So
again,
the
goal
here
is
to
shrink
the
time
from
innovation
to
user
feedback.
So
one
question
is
you
know
what
is
the
value
of
this
container
engine
to
the
community
and
really
from
the
very
start,
the
the
value
has
been.
One
of
the
core
values
has
been
improving
the
product
quality.
So
even
in
the
beginning,
you
know
the
community
would
look
to
Google
container
engine,
be
a
signal
of
a
release,
being
you
know,
ready
for
production
and
ready
for
deployment.
A
A
A
But
the
that's
the
value
to
the
community.
What
is
the
value
to
the
user?
So
there's
kind
of
two
aspects
of
this
when
you
want
to
set
up
kubernetes,
there
are
a
number
of
different
choices
that
you
have
to
make.
You
have
to
choose
your
cloud
provider,
you
have
to
choose.
You
know
your
OS
image
and
a
number
of
other
things
with
container
engine.
A
lot
of
that
is
done
for
you,
and
so
it
makes
it
simpler
in
terms
of
choice.
A
Secondly,
the
installation-
this
is
done
automatically,
it's
just
a
click
of
a
button,
and
you
know
you
install
so
that's
the
other
value
and
and-
and
thirdly
perhaps
most
importantly,
is
the
ongoing
management.
So
the
ongoing
management
and
upgrade
and
repair-
and
that's
really
I-
would
say
the
the
value
of
I
managed
and
hosted
offering.
So
it
makes
it
so
that
developers
can
focus
on
the
app
and
developing
their
app
and
Google
can
focus
on
making
sure
that
the
that
the
system
is
running.
A
The
second
piece
of
the
user
value
is
really
kind
of
of
kubernetes.
It
comes
from
the
Google
cloud
platform
itself
and
what
we
try
to
do
is
combine.
You
know
the
the
differentiators
of
the
underlying
cloud
platform
with
kubernetes,
so
some
of
the
features
that
we'll
be
discussing
and
demoing
today
will
be
examples
of
this,
but
to
some
of
the
some
of
the
sort
of
differentiators
with
the
cloud
platform,
are,
you
know
faster
startup
times
for
the
VMS,
as
well
as
heterogeneous
types
of
different
types
of
VMS,
preemptable,
VMS
and
custom
custom
VMS?
A
Also,
the
Google
cloud
load
balancing
has
some
unique
properties,
its
global
at
an
l7
level,
we'll
be
demonstrating
some
of
that
as
well,
and
then
some
of
the
other
features
with
regard
to
bigquery
and
datastore.
The
customer
architectures
on
Google
cloud.
Make
extensive
use
of
these
as
well
as
I
am
another
point
here,
is
that
the
container
engine
binaries
can
actually
be
upgraded
on
a
weekly
basis
and
in
fact
they
are
so
a
lot
of
these
cloud.
Specific
features
are
pushed
out
constantly,
even
aside
from
the
3-month
train
of
kubernetes.
A
And
this
is
some
of
the
feedback.
These
are
some
of
the
customers.
This
is
not
a
complete
list,
but
you
can
see
that
there's
a
variety
of
different
companies
that
use
container
engine
and
the
feedback
that
we
get
consistently.
Is
that
hey?
It's
really
nice?
It's
easy
to
use.
Also.
You
know
with
you
managing
this
with
Google
managing
this
I'm
able
to
focus
on
my
on
my
application
and
in
fact
it's
nice
that
it
runs
so
so
you
know
with
high
performance
and
so
seamlessly.
So
this
is
what
it
looks
like
I
think.
A
Hopefully
most
of
you
have
tried
this
out,
but
it's
really
a
very
simple
UI
and
you
know
you
can
go
and
create
a
cluster
with
the
click
of
a
button,
the
create
cluster
button
and
there's
also
a
built-in
command
line
interface.
So
the
Google
Cloud
console
has
a
built-in
command
line
interface
which
launches
this
CLI,
which
has
both
cue
cuddle
and
g-cloud
already
installed.
A
So
this
is
where
the
whole
installation
process
is
just
greatly
simplified
and
allows
you
to
get
started
in
a
matter
of
minutes
without
having
to
worry
about
all
of
the
things
that
are
underneath
it.
But
in
this
talk
we're
going
to
talk
about
some
of
the
things
that
are
underneath
it.
What
do
you
get
when
you
set
up
a
container
engine
cluster,
so
the
whole
process
takes
about
three
minutes
to
setup
a
cluster.
You
click
that
button
in
three
minutes
you
get.
A
You
know,
however,
many
nodes
you
wanted
with
whatever
features
turned
on
you
know
you
can
turn
those
on
in
the
UI,
but
what
is
happening
behind
the
scenes?
There's
quite
a
bit
happening
and
that's
why
I
have
the
picture
of
the
of
the
iceberg.
First
of
all,
we
are
checking
to
make
sure
that
you
know
the
user
is
authenticated.
A
We
are
also
checking
the
IAM
policy
to
make
sure
that
you
have
the
credentials
to
actually
run
the
command
that
you
want
to
run
if
it's
to
create
a
cluster
or,
if
it's
to
add
a
load
balancer.
You
have
that
privilege,
and
all
of
that
is
done
behind
the
scenes
by
calling
the
Google
compute
engine
API
and
starting
starting
the
master
on
the
node,
so
that
is
really
the
technology
behind.
That
is
the
container
engine
API,
which
is
a
crud
API,
for
you
know
container
clusters.
A
It
also
is
what
we
use
to
upgrade
the
kubernetes
version
for
the
cluster
and
obviously
is
integrated
with
all
the
different
DCP
features,
and
we
manage
we
own
and
manage
the
master
and
we're
going
to
talk
about
how
we
monitor
the
health
of
the
mount,
the
master
and
repair
it
and
also
maintain
a
set
of
add-ons.
These
are
actually
default.
Add-Ons
most
of
them
are
what
you
would
use
in
a
reference
implementation
from
the
open
source
as
well.
A
So-
and
this
is
a
picture
of
you-
know
the
master
and
some
of
the
things
that
are
on
the
master,
so
it
looks
very
similar
for
those
of
you
who
are
familiar
with
running
a
kubernetes
cluster
either
on-premise
are
on
juice,
EE
or
on
AWS
I.
Don't
think,
there's
anything
here
that
will
surprise
you,
so
it's
just
that
this
control
plane
is
managed
by
Google
and
it
looks
like
a
black
box
API
endpoint
to
the
user,
so
in
addition
to
you,
know
we're
using
sed
and
API
server
and
the
scheduler
and
the
controller
manager.
A
These
are
all
standard
components,
purely
as
they
are
in
the
open
source.
In
addition
to
those
there's,
an
l7
ingress
controller
and
cluster
autoscaler
again
also
pure
open
source
versions
of
this.
But
these
are
the
things
that
interact
with
the
Google
cloud
platform.
So
the
ingress
controller
interacts
with
juicy
lb
Google's
cloud
load
balancer
to
set
this
to
set
up
the
global
load
balancing
and
also
that
master
is
backed
up
regularly
on
an
automated
schedule.
So
we
ensure
that
it's
always
available
and
then
some
of
the
things
that
are
done
behind
the
scenes.
A
Obviously,
cubelet
and
docker
so
people
that
is
the
agent
on
the
node
and
docker.
This
is
all
automatically
installed
queue
proxy
and
then
a
number
of
default
add-ons.
So
this
is
the
stuff
that
is,
you
would
do
it.
You
know
yourself
if
you
were
doing
it
in
an
on-premise
environment,
but
this
is
done
automatically
so
specifically,
the
the
add-ons
that
we
have
via
fluency
and
cube.
Dns
fluency
is
for
log
collection
and
it
runs
on
every
node.
A
It
collects
the
logs
and
exports
them
to
Google
Cloud
logging,
which
and
through
stackdriver
and
then
cube
dns
enables
the
service
discovery
within
the
cluster
and
then
the
additional
three
add-ons
hipster
is
what
we
use
for
aggregating
all
the
monitoring
and
event
data.
This
is
again
same
as
what
is
available
in
the
open
source
and
and
we're
going
to
talk
a
little
bit
more
about
monitoring
in
just
a
second
and
then
the
l7
load,
balancing
back-end
and
the
dashboard.
A
A
A
So
we'll
go
through
each
one
of
these
and
talk
about
how
users
are
using
it
and
what
feedback
they've
had.
So,
first
of
all,
for
monitoring
monitoring
has
two
purposes.
One
is
internally
container
engine
collects
metrics
from
the
master
to
make
sure
that
the
master
is
healthy
and
also
to
repair
it
in
case
it
is
not
healthy
and
to
scale
it
with
the
needs
of
the
rest
of
the
cluster.
A
Secondly,
in
terms
of
user
monitoring,
so
the
way
that
user
metrics
are
collected
is
through
the
heap
store
add-on
that
I
mention
earlier.
There's
a
sea
adviser
that
runs
on
every
single
node
inside
the
cubelet
agent,
and
it
gathers
statistics
from
all
the
containers
in
your
cluster
and
exports
them
via
a
REST
API
heaps
tur
runs
in
just
one
pod
per
cluster.
A
So
the
as
I
mentioned
stack,
driver
and
dashboard
UI
provide
these
metrics
to
you,
but
also
it's
a
fairly
open
system
and
users
can
plug
in
their
own
add-ons
for
specific
things
that
they
want
to
do
in
terms
of
monitoring,
and
this
is
an
example
of
where
the
open-source
ecosystem
is
really
available
in
Google's
cloud
and
many
of
our
customers
actually
do
this
and
use
these
add-ons
and
I'll
talk
about
so
cystic
Prometheus
data
dog.
These
are
some
of
the
common
ones.
They
they
have
different
deployment
methodologies.
Data
dog
is
deployed
as
a
daemon
set.
A
Assisting
is
actually
a
kernel
header,
but
I'll
talk
about
what
users
do
with
these,
and
this
is
some
of
the
feedback
that
we've
that
we've
received.
They
really
like
having
additional
options
and
the
flexibility
of
choosing
the
monitoring
solution.
I've
heard
all
of
these
actually
being
used
in
different
situations.
A
lot
of
the
common
requests
are,
you
know
they
would
like
to
see
a
particular
UI
presentation
or
would
like
to
cut
the
data
in
a
certain
way.
A
They
dog
does
a
really
good
job
of
providing
a
great
UI
oftentimes
they're,
looking
to
be
able
to
see
the
resources
in
the
cluster
in
a
certain
hierarchy
based
on
label
and
have
different
views.
Also,
customers
often
have
different
parts
of
the
organization
that
are
used
to
using
a
different
tool
or
would
like
to
use
a
different
tool
and
so
being
able
to
plug
that
in
as
a
diamond
setter
or
whatever
in
a
different
container,
allows
different
organizations
to
to
plug
in
their
specific
monitoring
monitoring
solution.
The
other
use
case
is
really
for
hybrid
cloud.
A
So
that's
monitoring
the
second,
the
second
sort
of
additional
feature-
that's
that's
actually
very
popular-
is
ingress,
l7,
lb
and
actually
I
would
say
it's
also
very
powerful.
So
many
of
our
customers
who
are
global-
and
you
know
global
gaming
companies
or
companies
that
have
you-
know
IOT
solutions
that
that
are
available
in
multiple
geographies,
like
using
the
the
ingress
l7
load
balancer.
So
the
ingress
resource
itself
is
no
different
than
it
is
in
the
open
source.
A
Essentially,
you
know
customers
want
to
be
able
to
send
traffic
based
on
the
URL
to
the
appropriate
part
of
the
cluster,
and
that's
what
that's
what
ingress
enables
it
does
the
mapping
between
you
know
the
l3
l4
service
and
the
the
URL
based
on
where
it
should
go,
based
on
a
map
that
the
user
provides,
and
it
is
supported
by
nginx
and
H
a
proxy
and
also
has
SSL
support.
So
this
is
identical
to
what
is
available
in
the
open
source.
A
What
is
kind
of
different
in
Google's
cloud
is
really
the
the
cloud
load,
balancing
capabilities
which
we're
very
proud
of
these
capabilities,
but
essentially
the
Google
cloud
load.
Balancer,
you
know,
has
the
the
forwarding
rule
the
URL
map
and
in
the
backend
service,
and
we
have
a
controller,
the
l7
ingress
controller
pre-installed
as
part
of
them
as
part
of
the
master,
to
set
up
this
to
set
up
this
Google
cloud
load.
A
Balancing
some
of
the
additional
features
that
are
unique
is
you
know
you
can
get
round-robin
or
you
can
get
such
an
affinity
and
I
think
maybe
the
most
unique
piece
is
that
it
can
handle.
You
know
scaling
from
0
to
millions
of
QPS
and
that's
something
that
obviously
has
been
very
important
for
some
of
the
larger
global
customers
also
GCL
be
offers
cross
region
load
balancing,
and
this
is
kind
of
unique
amongst
cloud
providers.
A
It's
it's
actually
not
as
usable
by
the
simple
ingress
object,
but
where
it
is
really
usable
is
with
the
new
1.4
release
of
federated
ingress
and
federated
ingress
makes
it
possible
so
that
you
can
use
a
single
API
endpoint.
Sorry,
are
you
a
single
service
endpoint
to
distribute
your
load
across
multiple
clusters,
multiple
regions
and
multiple
clusters,
and
that's
currently
only
possible
on
Google
cloud
because
of
juicy
all
these
capabilities?
A
B
B
You
know
it's
currently
in
beta:
it's
it's
very
conservative
in
terms
of
how
quickly
it
scales
up
and
down
your
cluster,
but
it's
already
proving
really
valuable
to
our
customers
and
it's
it's
actually
proving
so
valuable
that
people
have
ported
it
to
also
work
on
Amazon,
and
so
one
of
the
advantages
of
us
having
built
it
in
the
open-source
community.
Is
that
now
it's
it's
being
taken
advantage
of
across
the
wider
community
and
it's
not
just
specific
to
a
container
engine.
B
It
also
it'll
scale
up
based
on
pending
pods,
which
is
really
nice,
because
if
you
just
look
at
CPU
utilization
across
your
cluster,
you
can
very
easily
get
into
a
state
where
you
have
very
low
CPU
utilization,
because
pods
are
scheduled
based
on
resource
reservations
and
not
current
resource
usage
and
it's
impossible
to
schedule
new
pods
into
your
cluster.
But
when
you
want
to
schedule
a
new
pod,
there's
nowhere
to
place
it,
but
a
autoscaler
is
just
looking
at
resource
utilization
will
not
actually
scale
up
your
cluster.
B
So
if
you
look
at
resource
requests,
then
you
can
tell
that
a
new
pod
would
be
able
to
schedule.
If
you
add
more
nodes
to
your
cluster,
you
can
dynamically
add
a
node,
and
then
you
can
see
that
that
pod
gets
placed
on
a
new
node
that
you
added
so
some
of
the
things
that
customers
say
is
it?
You
know
they
really
like
the
feature.
They've,
you
know
obviously
requested
it.
They
were
concerned
about
how
conservative
the
future
is
right.
Now
you
know
we.
B
So
the
next
thing
we're
going
to
talk
about
is
node
pools.
So
no
pools
are
sort
of
another
interesting
set
of
feedback
that
we
got
from
customers
we're
at
internally
at
Google.
Pretty
much
everyone
who
wants
to
write
a
service
doesn't
care
what
type
of
machine
they're
running
on.
Like
you
write
your
code,
you
give
it
to
Borg,
and
it
just
runs,
and
you
don't
worry
about
the
underlying
infrastructure,
and
this
is
sort
of
what
we
started,
building
with
kubernetes
and
with
Google
container
engine,
and
so
you
could
launch
a
cluster.
B
You
tell
it
what
you
want.
Your
underlying.
If
a
structure
to
look
like
I
want,
two
cores
per
machine
or
four
cores
per
machine
and
user
is
sort
of
very
quickly
turned
around
and
said
that
they
wanted
more
flexibility.
You
know
they
wanted
to
be
able
to
change
the
type
of
machine
they
were
using.
You
know
they
they
guessed
wrong
when
they
created
their
cluster
and
now
they
decide.
They
need
eight
cores
instead
of
four
or
they
decide
that
you
know.
I
want
to
have
labels
on
different
types
of
machines.
B
I
want
to
isolate
my
workloads.
I
want
to
be
able
to
put
labels
so
that
some
of
my
users
can
run
stuff
on
some
nodes
and
other
users
can
run
stuff
on
other
nodes,
or
they
might
have
a
use
case
where
they
need
some
nodes
that
have
a
lot
of
memory,
but
not
much
CPU
and
sometimes
that
have
a
lot
of
CPU,
but
not
as
much
memory.
B
So
we
sort
of
consolidated
all
of
this
feedback
and
built
a
feature
that
we
call
node
pools,
which
is
sort
of
related,
a
simple
layering
on
top
of
the
basic
kubernetes
concepts
where
you
can
have
multiple
groups
of
homogeneous
nodes
instead
of
a
single
group
of
muchness
nodes,
and
this
sort
of
allows
you
to
get
all
of
those
sort
of
requested
features
that
I
just
mentioned.
You
can
have
multiple
node
pools
which
can
give
you
different
types
of
machines.
B
You
can
create
new
node
pools
which
can
allow
you
to
transition
the
type
of
machine
that
you're
using
you
can
add
custom
labels
to
those
machines,
and
you
can
also
use
note
pools
for
things
like
upgrades,
which
allows
a
better
upgrade
story.
So
there's
a
great
example
of
us
sort
of
listening
to
users
and
sort
of
taking
that
feedback
into
our
product,
and
we
did
it
in
such
a
way
that
it
actually
sort
of
works
across
other
providers.
B
Also
because
really
note
pools
as
a
concept
is
us
applying
special
labels
to
notes
in
specific
groupings,
and
you
can
do
the
same
thing
if
you're
running
on
Prem
or
on
AWS,
if
you're
running
on
Prem,
you
might
want
to
label
all
of
your
nodes
with
which
failure
domain
they're
in
which
rack
or
which
data
center
nodes
already
report
their
resource
type.
You
might
want
to
give
them
custom
labels.
Like
you
know
this
should
be
a
dedicated
machine
type.
B
So
in
here's
sort
of
a
couple
examples
of
how
no
pools
work,
you've
got,
you
know
one
node
pool
with
one
version
of
kubernetes
and
then
another
node
pool
with
different
size.
Vms
I
went
through
all
of
that
already
and
then.
Lastly,
we
have
a
couple
of
examples
of
user
feedback
from
people
that
are
actually
using
node
pools
and
their
production
systems.
B
So
now
we're
gonna
switch
and
I'm
gonna
show
a
quick
demo.
The
demo
again
I
mentioned
earlier.
Auto
scaling
is
a
bit
slow,
so
in
for
the
demo
I'm.
Actually
I
made
a
recording
last
night
and
I'm
just
going
to
play
that,
instead
of
trying
to
go
through
auto
scaling,
because
otherwise
it
would
take
about
20
minutes.
B
All
right,
so
many
of
a
sort
of
a
voice
over
here
is
above
the
demo,
so
the
setup
is
we're.
Gonna
do
a
very
small
scale
demo.
So
it's
not
actually
the
scale
demo,
but
this
is
a
really
good
sort
of
sample
application.
You
can
run
the
code
for
all
of
this
is
open
source.
It's
in
the
contributory.
Basically,
you
run
some
nginx
servers
and
then
you
run
some
load
testers
and
you're,
sending
a
load
to
your
internet
servers.
B
So
what
I've
done
to
set
up
is
I've
created
a
cluster
with
a
single
node
pool
now
I'm
actually
launching
the
scale
demo.
So
we
launched
a
couple
of
different
services
and
controllers
and
then
we're
gonna
scale.
Both
those
up
and
you're
gonna
see
we're
gonna
send
about
10,000
requests
per
second
to
a
number
of
backends.
B
B
And
then
what
we're
gonna
do?
You
can
see
here
that
the
node
version
is
1.3
out
of
10,
which
is
not
the
current
node
version?
There's
an
update
available.
So
one
option
is
to
click
that
button
and
to
say
you
know,
update
my
nose
to
the
latest
version.
Some
people
want
a
little
bit
more
control
than
that,
and
so
one
way
you
can
have
more
control
is
you
can
create
a
second
node
pool
at
a
different
version.
B
So
let
me
do
is
I'm
going
to
create
a
new
node
pool
you
you
may
not
have
caught
this,
but
the
first
node
pool
had
four
core
machines.
The
new
node
pool
I'm
going
to
make
have
two
core
machines,
so
this
no
pools
can
have
slightly
solve
all
our
machine
types
I'm,
actually
integrating
in
one
of
those
use
cases
of
changing
the
machine
type
because
I
guessed
wrong
when
I
created.
My
cluster
I've
also
enabled
the
autoscaler
for
my
new
cluster.
B
All
right,
so
we
see
two
new
nodes
are
now
in
the
cluster
and
now
what
the
first
thing
we
do
is
we're
going
to
upgrade
from
the
old
node
pool.
The
new
node
pool
is
we're
going
to
cordon
all
of
the
old
nodes,
and
what
this
does
is
it
marks
all
the
old
nodes,
it's
unschedulable
and
what
that
will
do
is
it
means
that
when
you
try
to
reschedule
your
work,
when
you
drain
the
nodes
in
the
next
step,
it
will
force
the
the
pods
to
be
moved
over
to
the
new
node
pool.
B
If
you
skip
this
step,
then,
when
you
drain
a
node,
the
pods
can
be
rescheduled
on
to
other
nodes
in
the
old
node
pool,
which
means,
when
you
drain
that
note
they'll
get
rescheduled
again.
So
this
is
really
sort
of
an
optimization
to
make
sure
your
workload
isn't
rescheduled
more
often
than
necessary.
Next
we're
going
to
migrate
the
workload
over
by
draining
and
after
each
drain
step,
you
might
have
missed
that
after
each
strain
stuff
were
actually
going
to
delete
to
the
old
node.
B
B
If
you
look
over
here,
we're
running,
get
pods
and
you
can
see
the
node
on
which
each
pod
is
running
and
all
of
the
the
pods
not
anymore
all
of
them.
A
couple
of
them
is
switched
over
to
the
new
node
pool,
and
many
of
them
are
still
running
on
at
the
old
node
pool
and
then
down
here
you
can
see
one
by
one.
All
the
nodes
in
the
old
note
pool,
become
not
ready
and
then
disappear
as
we
delete
them
and
you'll
start
to
see
new
notes
get
added
at
the
bottom.
B
So
we
started
with
two
we're
already
up
to
three
and,
as
we
moved
through,
this
sped
up,
part
of
the
demo
you'll
see
new
nodes
show
up,
is
not
ready
and
then
flipped
to
ready
and
right
after
they
flip
to
ready.
You'll
see
up
here
that
one
of
the
nginx
instances
that
was
previously
unschedulable
because
there
was
no
space
in
the
cluster
switches
to
be
container,
creating
it
than
in
the
running
state.
B
B
Great,
so
we're
now
back
back
up
to
six-inch
next
instances
we
still
have
our
ten
load
BOTS
running.
We
were
serving
all
10,000
requests
the
entire
time,
and
you
can
see
if
you
drill
into
the
cluster
that
the
old
node
pool
is
now
shrunk
all
the
way
down
to
zero,
because
my
shell
script
was
deleting
one
note
at
a
time,
and
the
new
node
pool
has
scaled
itself
up
automatically
to
seven
notes.
B
So
that's
sort
of
an
example
of
how
you
can
combine
this
concept
of
no
pools
with
cluster
auto-scaling
to
do
sort
of
seamless
upgrade,
obviously
for
the
demo
I
did
the
upgrade
much
faster
than
you
want
to
do
in
production.
You
know
there
was
a
very
short
sleep
between
each
drain.
You
won't
do
that
much
more
slowly.
You
can
do
it
at
whatever
sort
of
level
of
granularity.
You
want
right,
you
can
do
it
manually.
You
can
automate
the
process,
but
it
gives
you
a
lot
of
control
over
exactly
how
you
upgrade
your
cluster.
B
Alright
and
lastly,
we're
going
to
talk
about
alpha
clusters,
so
a
partner
mentioned
earlier
that
alpha
clusters
are
something
that
we
recently
added
to
kubernetes
again.
This
was
based
on
user
feedback,
where
users
would
see
something
being
developed,
an
open
source
oftentimes
when
you're
you're
launching
a
service
as
a
large
company.
B
Is
we've
created
a
way
for
people
to
launch
clusters
that
have
all
the
alpha
features
enabled
so
they
can
try
out
the
alpha
features
before
they're
ready
for
production,
and
this
turned
out
to
be
a
really
great
way
to
get
feedback
for
those
features
before
they
become
sort
of
locked
into
their
final
API
state
and
so
for
pet
sets
and
init
containers
and
scheduled
jobs.
We're
getting
a
lot
of
feedback
from
the
community,
because
people
can
go
into
gke
click.
A
button
have
a
cluster.
B
Yes,
that's
a
really
a
question.
The
question
was
the
no
pools
underneath
use,
something
we
call
managed
instance,
groups
in
GCE,
and
you
can
actually
turn
on
a
GCE
autoscaler
on
the
managed
instance
group,
and
we
actually
tried
this.
So
this
is
what
I
was
referring
to
when
if
you
just
turn
that
on-
and
you
tell
it
to
look
at
the
CPU
utilization,
you
get
really
terrible
behavior,
because
the
autoscaler
is
assuming
that
all
of
your
nodes
are
effectively
the
same
that
any
of
them.
B
If
you
add
a
new
one,
it's
gonna
have
the
same
behavior
the
existing
one.
So
it
sees
the
overall
utilization
go
up.
Adding
in
other
nodes
should
make
the
overall
utilization
go
back
down.
It
turns
out
with
kubernetes,
since
people
schedule,
pods
and
pods
don't
land
exactly
evenly
across
all
nodes,
and
some
pods
have
higher
resource
reservations.
That
offended
breaks
down
really
quickly,
and
so
we
realized
that
we
couldn't
just
flip
on
the
autoscaler
and
that's
why
we
ended
up
building
sort
of
the
custom
cluster
autoscaler
for
kubernetes.
B
Right,
so
the
question
is
the
the
heap
sir
pod
is
running
as
a
single
pod.
It's
actually
running
underneath
a
replication
controller,
so
it's
a
replication
controller
with
one
instance.
So
there
is
a
single
pod
running,
but
there
is
a
controller
making
sure
there
is
always
a
single
pod
running
so
if
it
gets
killed
or
if
that
node
gets
removed,
then
a
new
instance
of
peeps
will
get
started
right
and
we
also
monitor
that.
B
So
if
the,
if
the
note
actually
disappears,
we
can
usually
detect
that
pretty
quickly,
because
we
can
talk
to
the
Google
Google
cloud
platform,
compute
engine
API
and
find
out
that
the
note
is
gone
and
we
can
kill
it
immediately.
But
if
we
just
stopped
getting
status
about
that
pod,
then
yeah
I
can
take
five
minutes
before
anyone
starts.
That's
the
default
threshold.
B
Right
so
to
actually
configure
like
the
underlying
machines,
we
use
all
the
scripts
that
are
open-source.
So
if
you
look
in
the
clustered
directory
and
open
source,
there's
a
bunch
of
cluster
GCE
machine
configuration
scripts
that
are
used
to
configure
all
of
the
nodes
that
run
the
cluster.
You
can
then,
on
top
of
that
run
like
a
daemon
set
that
tries
to
reconfigure
all
the
nodes
in
your
cluster.
If
you
need
to
make
customizations.
B
Right,
so
the
question
is:
there's
upgrade,
follow
the
same
path.
So
what
happens
with
upgrades
is
when
an
upgrade
comes
out?
We
upgrade
the
master
machine
for
you
and
then
it's
up
to
you
to
upgrade
your
notes,
and
the
reason
for
this
is
that
we
found
that
that
note
upgrades
can
be
disruptive
right
and
we
want
that
to
be
under
your
control
of
when
that
happens
and
the
speed
at
which
it
happens.
B
So,
as
I
mentioned
before,
when
you
want
to
upgrade
your
note
pool,
you
can
click
a
button
and
will
upgrade
your
note
pool
for
you,
depending
on
your
application.
That
can
be
perfectly
fine,
but
it
also
might
break
your
specific
application.
So
you
can
also
create
a
second
note
pool
and
you
can
migrate
your
workload
over,
which
is
what
I
showed
during
the
demo.
B
B
You
know
kernel
that
comes
out
that's
incompatible
with
a
tweak
that
you've
been
making
to
your
notes
and
again,
that's
why
it's
sort
of
on
you
in
terms
of
scheduling
when
you're
gonna
upgrade
your
notes,
because
then
you
are
breaking
yourself
instead
of
us
breaking,
you
know,
maybe
it
at
2
a.m.
and
your
time,
son.
A
And
the
the
the
two
features
that
Robbie
mentioned
with
regard
to
pod
reception
budget
and
graceful
termination.
Those
are
work
in
progress
for
the
upcoming
release,
so
you
can
read
about
them
on
github,
but
they
will
allow
at
least
four
nodes
with
the
default
note
image,
which
is
Google
container
image
GCI
for
you
to
opt
into
you
know,
node
upgrades
managed
through
Google.
That's
a
that's
not
yet
available,
but
it's
coming
with
those
features.
B
Yeah,
it's
just
the
two:
we're
still
trying
to
figure
out
how
to
make
it
possible
to
feel
how
people
bring
customized
images.
It's
a
very
difficult
problem,
because
everyone
has
sort
of
their
own
pet
image.
It's
not
sort
of
clear
exactly
how
that
will
work
with
the
rest
of
the
system,
and
we
want
to
make
sure
that
the
experience
people
have
is
a
functioning
kubernetes
system
we.
B
B
Please
launch
that
for
me
and
we'd,
be
like
oh,
this
cluster
didn't
create
I,
wonder
why,
like
oh
they're,
trying
to
launch
Windows
so
they're
trying
to
launch
you
know
Ubuntu
or
red
hat
or
some
other
system
where
the
machine
configuration
part
didn't
actually
work
right,
and
so
it
was
impossible
for
us
to
give
them
a
functioning
cluster,
which
is
a
pretty
terrible
user
experience.
So
we
want
to
make
sure
the
user
experience
is
really
good
before
we
allow
that
to
be
more
customized.
That's.
A
Specifically
for
the
node
base
image,
and
so
there's
only
the
two
options,
which
is
the
Debian
based
container
image
and
GCI
the
Google
container
image.
If
you
have
specific
requirements,
then
you
can
use
GCE,
you
can
always
set
up
a
kubernetes
cluster
in
compute
engine,
and
then
you
know
it
can.
It
doesn't
have
all
the
automation,
but
then
it
has
100%
flexibility.