►
From YouTube: Building a Cluster Management API using Kubicorn [A] - Robert Bailey, Google & Kris Nova, Heptio
Description
Building a Cluster Management API using Kubicorn [A] - Robert Bailey, Google & Kris Nova, Heptio
Kris Nova (Heptio) and Robert Bailey (Google) join forces and begin the difficult task of looking into the future of the infrastructure layer of Kubernetes. We start the talk with a brief summary of the state of infrastructure today and explain the differences between “infrastructure as code” and “infrastructure as software”. We look at how the lack of definition in the most fundamental layer of the stack has fragmented our community and caused problems with adoption of Kubernetes.
We propose a new way of representing infrastructure (the cluster API) for the Kubernetes community and take a deep dive into its implementation in kubicorn. We look at the structure of the cluster API and share valuable insight on how we took lessons from other areas of Kubernetes to form what it is today. Furthermore we look at the power of having a declarative approach to infrastructure as we start to treat the infrastructure layer the same as the application layer.
The audience will walk away with a clear understanding of the infrastructure layer, as well as a new way of thinking about the infrastructure in the future via the cluster API.
About Robert Bailey
Robert is a lead for the cluster lifecycle SIG and has been working on Kubernetes for more than 3 years. He was one of the founding members of the Google Container Engine team. Prior to Kubernetes, he was a Site Reliability Engineer helping teams at Google launch new products and services.
About Kris Nova
Kris Nova is an Advocacy Boss for Heptio with an emphasis in containers and the Linux operating system. She lives and breathes open source. She believes in advocating for the best interest of the software, and keeping the design process open and honest. She is a backend infrastructure engineer, with roots in Linux, and C. She has a deep technical background in the Go programming language, and has authored many successful tools in Go. She is a Kubernetes maintainer, and the creator of kubicorn, a successful Kubernetes infrastructure management tool. She organizes a special interest group in Kubernetes, and is a leader in the community. Kris understands the grievances with running cloud native infrastructure via a distributed cloud native application, and is authoring an O'Reilly book on the topic called Cloud Native Infrastructure.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
Hi
everyone
today
we're
gonna
talk
about
building
a
managed
API
using
cubic
horn,
and
this
journey
that
Robert
and
I
went
on.
You
want
to
go
to
the
next
slide
so
to
get
started.
I'm
gonna
tell
everybody
about
myself.
I
co-authored
this
book
called
cloud
native
infrastructure.
We're
doing
a
book
signing
later
today.
Just
in
my
co-author
is
a
really
awesome.
Dude.
If
you
get
a
chance
to
go
talk
to
him,
he
wrote
a
lot
of
the
the
stuff
in
the
book
about
containers
and
orchestration
and
he's
really
awesome.
A
A
I
did
a
lot
of
work
in
cops
with
Amazon,
doing
some
private
topology
stuff
that
now
you
can
see
in
Amazon's
eks
solution
that
they
released
last
week
at
reinvent
I
wrote
this
program
called
Cuba
corn
one
weekend
and
it's
turned
into
a
fairly
successful
open-source
project.
We're
gonna
look
at
it
and
talk
more
about
it
later
on
and
I
already
mentioned,
cops
and
then
also
the
most
exciting
part
is
I
just
started
a
new
role
as
a
senior
developer
advocate
at
hep.
A
Do
let
me
know
I
can
do
you
two
if
you
want
yeah?
Ok,
this
is
exciting.
I'm,
just
gonna
keep
talking.
So
this
is
Robert.
Bailey
he's
a
I'm.
How
long
have
we
been
working
together
like
a
year
and
a
half?
Now
probably
so,
he
works
at
Google,
he's
up
in
Seattle
he's
a
colleague
of
sig
cluster
life
cycle,
so
show
of
hands.
Who
here
knows
what
sig
cluster
life
cycle
is?
Who
here
has
been
on
its
sig
cluster
life
cycle?
Call
before
awesome,
yeah
I
recognize
most
of
those
people.
A
So
what
we
do
is
we
sort
of
manage
this
idea
of
a
kubernetes
cluster,
beginning
middle
and
end
I,
like
to
think
of
it
as
credence
for
kubernetes
clusters?
How
do
they
get
created?
How
do
we
manage
them
over
time,
upgrade
them
et
cetera
and
then,
ultimately,
how
do
we
delete
our
set
and
dying
kubernetes
cluster?
So
he's
a
found,
a
member
of
the
Google's
kubernetes
June,
which
is
pretty
cool
like
this,
is
one
of
the
people
who
helped
write
gke.
A
Okay,
who
has
used
one,
that's
not
on
here
yeah,
so
there's
even
more
out
there
I've
worked
on
a
couple
of
bees.
I've
used
most
of
them
cops
and
keeper
corner
like
the
big
two,
but
there's
one
sort
of
like
underlying
weird
problem
with
all
of
these,
which
is
as
a
user.
When
you
walk
up
to
one
of
these
tools
for
the
first
time,
it
always
kind
of
looks
and
feels
a
little
bit
different
like
you,
don't
really
have
to,
or
you
do
really
have
to
go
through
this
exercise
of
learning.
A
Okay
for
this
specific
implementation,
what
do?
What
are
the
expectations
for
me
as
a
user?
What
what
do
you
need
for
me?
What
do
I
have
to
give
you
and
what
do
I
have
to
configure?
Maybe
I
have
to
like
write
a
file
or
change
them
configuration
values
or
like
edit,
some
yam?
Well,
we
all
love
doing
that.
But
the
point
was:
is
it
was
a
different
experience
for
every
one
of
these,
but
they're
all
trying
to
solve
the
same
problem
and
that's
what
we're
hoping
to
solve
with
this
cluster
API.
A
A
A
Whether
that
way
is
I
want
to
upgrade
the
control
plane
or
maybe
I
want
to
change
some
configuration
or
maybe
I
even
want
to
adjust
the
amount
of
nodes
that
are
running
all
of
my
lovely
application
containers
that
are
doing
all
those
things
that
make
me
so
much
money
as
a
software
engineer,
so
reinventing
the
wheel,
oh
also,
real,
quick.
All
of
these
pictures
were
looking
at
our
pictures
that
I
took
from
the
top
of
mountains
that
I've
climbed,
so
we've
gotten
some
feedback
that,
like
this
is
like
hard
to
read
right
here.
A
Like
sorry,
not
sorry,
look
at
my
mountains
so
just
deal
with
it
so
anyway.
This
is
not
democrat.
It
wasn't
my
favorite
ones
decline.
If
we're
looking
at
reinventing
the
wheel,
you
can
see
all
of
these
like
similar
things,
that
all
of
these
cluster
tools
usually
end
up
coming
to
as
they
mature
as
a
piece
of
software.
The
first
one
is
installation.
What
does
it
look
like
to
stand
up
the
control
plane?
What
does
it
look
like
to
stand
up
nodes?
The
configuration
of
these
components?
A
We
usually
try
to
make
it
easy
for
humans
to
like
define
something
that's
going
to
mutate,
how
the
cluster
is
going
to
look
and
feel
after
it
gets
up
and
running.
The
problem
again
is
those
shortcuts
that
we're
writing
they're
all
different
based
on
the
installation
tool,
we're
looking
at
cluster
upgrades
and
component
upgrades.
Maybe
that's
rolling
your
app,
maybe
that's
rolling
resources
in
your
cluster
and
the
kubernetes
components
themselves,
or
even
adding
a
node
like
this
is
actually
a
pretty
challenging
thing
to
do.
A
If
you've
ever
actually
looked
at
what
needs
to
happen
in
order
to
bring
up
a
new
virtual
machine
that
is
working
as
a
worker
node,
you
have
to
get
TLS
configuration
in
there
somewhere.
If
you're
using
TLS,
you
got
to
get
a
cubelet
up
and
running.
Usually
that's
watched
with
system
D
and
we
all
know
how
much
fun
system
D
is
so
there's
a
lot
of
noise
that
goes
into
that
and
then
Auto
scaling,
which
is
again
kind
of
a
fancy
way
of
rebranding,
adding
a
node
to
your
cluster.
A
A
So
right
now,
if
we
like
walked
on
this
journey
of
like
I'm,
a
DS
cops
and
I'm
gonna
go
install
a
cluster
and
Amazon
cops,
has
an
API
you
when
you
want
it's
declarative,
just
like
kubernetes
and
you
go
and
you
get
this
really
awesome
cluster
in
Amazon
and
then
it's
great
and
it
works
and
it
scales
and
it's
production
ready.
It
runs
Nha,
it's
baller,
but
then
what?
If
you
want
to
like
move
that
to
digitalocean
or
what?
If
you
want
to
move
that
to
Google
or
what?
A
If
you
want
to
move
half
of
it
to
Google
or
digitalocean
and
the
other
half
to
bare
metal?
Or
maybe
you
want
to
move
one
of
your
nodes
to
a
Raspberry
Pi
that
on
top
of
the
mountain
and
the
rest
can
go
in
Microsoft's
pager?
The
point
is:
is
we
don't
really
have
a
good
way
of
doing
that
and
last
I
checked
the
whole
point
of
kubernetes
was
to
be
cloud
agnostic
and
to
liberate
us
from
these
decisions,
of
which
cloud
should
I
be
running
in
right
now,
so
next,
so
the
birth
of
Kube
admin?
A
Who
here
has
used
cube,
admin
or
even
know
what
it
means?
Okay,
so
most
of
you
so
cube
admin
is
this
tool
that
we
started
at
about
a
year
ago
and
it
was
sort
of
designed
to
do
the
same
thing,
but
with
a
different
like
part
of
the
stack
right
like
right
now
we're
talking
about
how
we
would
represent
infrastructure
in
kubernetes,
Kyubey
admin.
It
solves
the
problem
of
like
first
okay,
let's
unify
bootstrapping,
so
we
don't
care
if
you're
on
Ubuntu,
we
don't
care.
If
your
unsent,
we
don't
care.
A
So
that
again
we
created
one
bootstrapper
to
rule
them
all,
but
it
did
not
do
machine
provisioning
and
so
in
our
next
slide.
We're
gonna
look
at
this
really
awesome
diagram,
but
it's
really
important
to
sort
of
have
this
understanding
of
these
two
fundamental
layers
of
kubernetes
one.
Is
this
virtual
machine
underlying
like
resource
stuff?
These
are
like
fake
things
that
I
imagine
you
could
touch
and
the
other
one
is
like
the
actual
kubernetes
components
themselves.
Who
knows
what
a
the
kubernetes
components
are
off
the
top
of
their
head?
A
This
is
like
your
quiz
for
the
day,
I'm
gonna
make
everybody
rattled
enough.
It's
the
API
server
it's
at
CD,
it's
a
scheduler
manager.
It's
all
of
the
things
that
you
run
the
bits
of
software.
You
run
that
make
kubernetes
do
what
it
needs
to
do.
Cube
admin
gets
those
up
and
running,
but
it
does
not
get
the
infrastructure
that
that
runs
on
top
of
in
place.
So
we're
gonna
look
at
this
slide
here.
Well,
that
was
a
really
great
transition.
A
So
we
can
see
that
we
have
these
three
layers
over
here
on
the
right
layer,
3,
layer,
2
and
layer
1,
the
cube
add
many
layer.
If
you
look
it's
kind
of
right
here,
I'm
trying
to
point
I
wish
I
to
laser,
and
you
can
see
that
runs
on
top
of
masters
and
nodes,
but
either
way
underneath
all
of
these
logos
at
the
bottom.
That's
our
infrastructure
layer.
A
This
is
the
stuff
I
as
a
DevOps
user
as
an
infrastructure,
engineer,
care
about
and
a
lot
of
software
engineers
kind
of
don't
really
want
to
deal
with
that,
and
so
that's
why
I
make
the
money
I
do
right
so
anyway
to
keep
admins
up
there.
But
what
solves
this?
How
do
we
unify
this
layer?
How
do
we
say
infrastructure
you're
the
same?
We
want
to
hit
the
next
slide,
rather
so
here's
an
example
of
cubic
corn.
This
is
a
tool,
I
wrote
and
we're
gonna.
A
What
use
it
as
an
example
of
one
tool
that
solves
provisioning
clusters
and
we're
gonna,
compare
it
to
another
tool
that
effectively
does
the
same
thing,
but
it's
just
we
think
about
it
differently.
So
if
you
want
to
go
like
in
this
example,
we're
gonna
take
cubic
corn
and
we're
going
to
using
the
cubic
or
an
API
that
only
matters
to
Cuba,
corn
or
cubic
ORN
and
we're
gonna
create
some
resources
in
Amazon.
These
resources
could
be
virtual
machines,
load,
balancers,
subnet
security
groups,
whatever
also
we
want
to
go
and
create
some
stuff
in
GCP.
A
This
is
another
exercise
of
learning
and
cuba,
corn
and
creating
stuff
in
GCP,
using
the
cubic
or
an
api
that
only
matters
to
this
one
piece
of
software
and
we're
gonna,
look
at
the
autoscaler.
It's
gonna
do
effectively
the
same
thing
and
one
more
and
then
we're
gonna,
create
resources
and
GCP
and
AWS,
but
here's
the
problem.
A
Everything
over
here
is
fragmented
and
segmented
in
its
own
independent
way
of
creating
these
resources.
The
autoscaler
is
another
way
of
saying
creating
a
resource,
because,
if
you
think
about
it,
going
from
zero
to
one
shouldn't
really
be
any
different
from
going
from
one
to
many.
So
an
auto
scaler
effectively
is
another
recreation
of
kubernetes
infrastructure
management
in
general.
This
is
the
whole
point
of
sig
clustered
lifecycle.
We're
not
called
sig
cluster
creation
right,
let's
go
ahead
and
hit
the
next
one.
A
So
what
have
we
had
a
way
to
create
a
declarative
representation
of
a
cluster
and
a
comply
and
apply
it
consistently
across
the
clouds?
We
would
let
kubernetes
provision
this
infrastructure,
which
this
is
always
an
interesting
problem
for
me
for
us
to
solve
we'll
get
to
there.
You
know
sack
o
things,
because
in
order
for
us
to
get
kubernetes
up
and
running,
we
need
an
infrastructure
in
place,
but
if
we're
using
kubernetes
to
create
infrastructure
like
which
one
comes
first,
how
do
we
solve
that
bootstrap
problem?
A
So
we
software
engineers
we
get
to
start
looking
at.
How
are
we
going
to
create
infrastructure
and
then
simultaneously
bootstrap
it,
and
we
get
to
kind
of
like
ReWalk
the
steps
of
our
forefathers
who
tried
to
compile
the
first
compiler
for
the
first
time
without
a
compiler
like
how
did
how
did
that
happen?
So,
anyway,
number
three:
let
kubernetes
handle
changes
to
the
cluster
in
a
cube
native
way,
so
kubernetes
the
cube
native
way
when,
when
I
say
this,
what
it
means
to
me
is
I.
A
B
Excellent
all
right,
thank
you
Chris!
So
what's
next
the
cluster
management
API,
and
so
the
cluster
management
API,
is
we've
sort
of
set
up
as
bienes
talk,
is
a
declarative
way
to
create,
configure
and
manage
your
cluster
I
think
Chris
sort
of
mentioned
this,
but
it's
not
called
sig.
Cluster
creation
like
creating
a
cluster,
is
relatively
easy.
There
are
more
than
40
getting
started,
guys
like
linked
from
the
official
kubernetes
documentation.
B
There
were
a
number
of
them
mentioned
on
an
earlier
slide
and
a
lot
of
those
guides
sort
of
get
you
up
and
running,
like
even
Kelsey
Hightower's
communities.
The
hard
way
gets
you
up
and
running,
but
doesn't
really
talk
about
the
rest
of
the
lifecycle
of
your
cluster,
like
what
happens
when
a
new
kubernetes
release
comes
out.
B
If
there's
a
security
patch,
if
there's
a
software
fix,
you
need
to
keep
using
your
cluster
over
time,
and
so
the
cluster
management
API
is
really
designed
not
just
to
create
your
cluster
but
to
manage
your
cluster
over
time.
So
if
you
look
at
the
very
high
level
view
of
what
a
kubernetes
cluster
is,
you
know
what
people
think
about
a
cluster
is
having
a
master
and
some
nodes,
but
really
what
we
have
is
a
control
plane
and
some
machines.
B
The
machines
become
nodes
by
virtue
of
the
fact
that
they're
running
cubelets,
but
they're
really
they're
just
machines.
In
addition
to
machines
and
the
control
plane,
we
have
some
cluster
wide
properties,
so
some
generic
things
that
need
to
be
sort
of
spread
across
the
whole
cluster
that
aren't
really
part
of
the
control
plane
and
aren't
really
part
of
the
machines
themselves,
but
our
properties
they
need
to
be
sort
of
distributed
across
the
different
parts
of
kubernetes.
So
they
all
work
together.
B
So
we
think
about
configuration
there,
these
three
sort
of
different
pieces
that
we
need
to
configure
to
have
a
declarative
way
to
describe
what
a
kubernetes
cluster
is
so
for
the
control
plane,
we're
only
in
pretty
heavily
on
something
called
component
configuration
which
is
using
kubernetes
primitives,
like
config
maps,
to
configure
kubernetes
resources.
So
all
of
the
control
plane
components
are
themselves
pods
that
run
inside
of
kubernetes,
and
so
we
can
use
cube
renée's
primitives
to
configure
those
as
we
can
figure
any
application
kubernetes.
B
But
the
machines
in
the
cluster
properties
are
sort
of
outside
of
what's
running
inside
of
cubed
easing.
We
need
a
different
way
to
describe
those.
So
the
first
thing
when
you
describe
is
the
cluster
wide
properties,
and
so
we
took
a
look
at
many
of
the
different
cellars
in
the
ecosystem
and
try
to
figure
out
what
are
the
generic
things
that
need
to
be
sort
of
filtered
into
the
entire
cluster,
and
it
turns
out
there
aren't
really
very
many
and
the
ones
that
we
came
down
with
are
really
how
you
define
networking.
B
So
you
need
to
define
how
what
your
pod
network
looks
like
what
your
service
network
looks
like
and
how
you
want
to
describe
DNS,
because
those
parameters
get
distributed
out
to
different
parts
of
your
cluster.
They
go
to
the
qubit,
they
go
to
Cordy
and
S,
or
cube
DNS.
They
go
into
the
API
server,
so
those
need
to
be
defined
at
a
high
level.
So
they
can
be
distributed.
We
also
when
we
looked
at
this.
B
There
are
many
other
properties
we
could
have
elevated
for
this
level,
but
we
decided
to
go
as
minimal
as
possible
because
it's
really
easy
to
add
things
later
and
it's
really
difficult
to
take
things
away.
So
if
we
put
things
in
here
that
don't
really
belong
at
this
high
level,
it's
gonna
become
hard
for
us
to
then
move
forward
and
take
them
out
in
a
future
version.
The
other
thing
you
need
to
find
is
what
your
machines
look
like.
B
So
machines
are
the
underlying
physical
or
virtual
machines
that
will
turn
into
nodes
in
your
cluster,
and
so
you
know,
if
you
guys,
are
familiar
with
kubernetes.
We
already
had
this
notion
of
node
in
kubernetes.
A
node
has
a
spec
in
a
status,
but,
unlike
other
objects
or
kubernetes,
the
spec
of
a
node
is
not
actually
declarative.
There's
no
way
to
create
a
node
object,
Interbrew
Nettie's
and
put
a
speck
in
there.
B
What
this
node
should
look
like
and
then
have
a
node
show
up
right,
so
nodes
are
always
reactive
nodes
show
up
and
they
give
you
status
of
the
machine
that
the
node
is
running
on.
So
one
of
the
principles
we
started
with
with
this
project
was
that
we
didn't
think
we
could
modify
core
kubernetes.
You
know
a
couple
of
years
ago.
B
It
was
decided
that
cluster
infrastructure
management
was
outside
of
the
core
of
kubernetes
we've
sort
of
taken
that
to
heart
and
as
we
as
we
define
this
API
we've
kept
it
outside
of
kubernetes,
and
so
we've
defined
as
a
machine
and
a
machine
is
basically
the
declared
a
specification
for
what
a
node
should
look
like
when
it
shows
up.
But
since
it's
a
different
object
itself
has
a
spec
and
a
status.
B
The
status
of
the
machine
just
points
to
a
node
when
that
node
shows
up
and
the
spec
tells
you
what
type
of
node
you
want.
And
again
we
looked
at
sort
of
common
things
that
we
would
like
to
define
across.
You
know
the
different
installers
that
we
have
today
and
we
decided
there
there's
some
really
basic
things.
We
we
know
we
want
to
have
so
we
know
we
want
to
happen.
B
Names
like
machines
need
to
be
named,
so
we've
used
like
the
typical
kubernetes
metadata
mechanism
for
naming
machines,
and
you
can
see
in
this
example
here
we're
actually
using
a
generate
name.
So
we're
letting
the
kubernetes
api
machinery
actually
pick
the
name
for
us.
We
don't
really
care
what
the
name
is,
but
we
wanted
to
follow
this
pattern.
You
can
also
specify
it.
You
know
labels
and
other
sorts
of
things
in
the
normal
kubernetes
fashion,
and
you
can
define
desired
versions.
So
we
know
we
want
to
run
this
version
of
the
cubelet.
B
We
want
to
run
this
version
of
the
container
runtime
doesn't
show
here,
but
you
can
also
specify
a
specific
version
of
your
control
plane
if
you
set
the
role
to
be
a
master
and
then
finally,
we
have
roles
which
are
right.
Now
you
could
be
a
node,
you
can
be
a
master
or
you
can
actually
be
both,
and
this
can
be
extended
in
the
future.
If
you
want
to
have
different
types
of
machines
in
your
cluster,
you
can
imagine
wanting
to
have
a
machine
that
just
runs
at
TD
right.
B
You
want
to
have
a
separate
at
CD
cluster,
outside
of
the
machines
that
run
your
API
server,
your
controller
and
your
scheduler,
and
then
lastly,
you'll
see
this
section,
the
middle,
which
is
a
provider
config.
It
turns
out
that
trying
to
make
a
API
abstraction
across
lots
of
clouds
is
very
difficult.
You
know
many
people
have
tried
this
and
many
people
have
failed
in
the
past,
and
so
rather
than
try
to
reinvent
that
wheel.
B
What
we
decided
to
do
was
to
allow
a
generic
string
to
be
placed
in
here
and
in
this
example,
the
string
itself
is
actually
a
kubernetes
api
type,
so
you
can
see
the
common
Fields
of
API
version
and
kind,
and
then
some
GCP
specific
fields
underneath
that
that
allow
us
to
actually
provision
the
underlying
machine,
and
this
allows
us
to
actually
harness
sort
of
the
full
power
of
the
underlying
cloud.
So
if
you're
running
on
Amazon,
you
could
do
things
like
you
spot
instances
right
even
though
spot
instance
isn't
a
common
thing.
B
You
can
get
across
all
infrastructure
providers.
You
can
still
take
advantage
of
the
power
of
that
particular
cloud,
and
so
what
does
this
look
like
in
practice?
So
we
have
this
cluster
management
API,
which
abstracts
away
the
sort
of
the
basic
infrastructure,
and
you
create
your
installer
you.
You
know
you
have
to
your
installer
tools,
so
you
know
use
cubic
corn.
You
pass
it
these
amol
definitions
of
what
my
cluster
should
look
like
it
talks
to
the
underlying
infrastructure
and
provisions
are
clustered
for
you.
You
know
likewise
with
cops.
B
You
take
that
same
yamo
configuration
you
can
give
it
to
cops
and
you
end
up
with
a
cluster
or
you
can
give
it
to
GK
or
you
can
give
it
to
a
ke
or
you
can
give
to
ek
e,
and
you
know
in
the
end,
you
end
up
with
a
cluster
that
exposes
this
API.
That
is
consistent
and
what
you
can
do
with
that
API
is,
then
you
can
write
generic
tools
that
will
work
across
a
cluster
created
by
any
of
these
different
cluster
creation
tools.
B
So
for
things
like
cluster
auto-scaling,
the
cluster
autoscaler
can
target
the
cluster
management
API
instead
of
the
underlying
cloud
and
pressure.
Ap
is
to
allow
you
to
make
your
cluster
I/o
scalar
portable
across
clusters
and
across
environments,
and
likewise
you
can
take
other
functionality
that
deals
with
machines
like
checking
the
health
of
your
machines
and
fixing
them
when
they
break
or
running
upgrades
across
your
machines,
and
you
can
build
that
as
common
tooling,
that
works
across
multiple
environments
rather
than
re-implementing
it.
B
So
right
now,
if
you
want
to
do
upgrades
with
cops
versus
upgrades
of
cubes
for
a
or
upgrades
in
gke,
they're,
all
completely
different
api's,
and
this
allows
us
to
have
a
single
mechanism,
two
upgrades-
you
can
still
write
different,
upgrade
policies
and
different
upgrade
tools,
but
they
can
all
be
reused
and
they
can
be
shared
across
a
much
larger
community
of
kubernetes
operators.
So
with
that,
we're
gonna
switch
into
a
demo
and
sort
of
show
you
what
this
actually
looks
like
in
practice.
B
So
I've
pre
created
a
cluster
using
a
cluster
API.
It
takes
about
seven
and
a
half
minutes
to
come
up
and
we
have
a
relatively
short
talk
here.
So
I
wanted
to
get
that
out
of
the
way
Oh.
Can
you
not
see
that
at
all?
Okay,
that's
C,
legible,
not
so
much!
Okay!
Alright,
let's
create
I
thought
your
enum
black
would
look
better
up.
There
is
white
better!
You
see,
white.
C
B
B
B
You
can
then
inspect
a
single
machine.
So,
let's
see
what
that's
machine
looks
like,
and
this
is
very
similar
to
the
machine
that
I
showed
in
a
slide,
except
in
this
case
we've.
We
have
extra
metadata
I'm,
not,
hopefully
that's
going
to
fast,
because
the
cube
Randy's
API
machinery
adds
additional
metadata
to
all
of
the
kubernetes
objects.
So
you
get
creation
timestamp,
we
have
the
generate
name,
but
we
actually
get
an
actual
name
generated
by
the
server.
B
We
added
some
labels
in
this
case
and
we
get
a
couple
of
extra
fields.
As
I
mentioned,
we
have
the
spec,
and
so
this
is
votes
from
the
slide.
Where
we've
got
where
this
machine
should
live
and
what
role
and
what
versions
we
want
on
the
machine.
And
finally,
so
it
does
it
communities
object.
We
also
have
a
status,
and
in
this
case
the
status
is
a
reference
to
a
node.
B
So
again,
machines
and
nodes
are
two
different
types,
but
they're
linked
through
this
field
called
node
breadth
where
you
can
look
at
a
machine,
especially
if
you're
writing
a
tool.
You
can
look
at
the
machine
with
a
tool
and
then
find
the
node
reference
and
then
go
describe
that
node.
So
in
this
case
it's
just
as
easy
as
running
queue,
control
described,
node,
pacing,
that
node
name
and
then
curate
says
great.
Here's
your
node
right,
and
so
this
was
the
node
that
was
created
from
that
machine
definition.
B
And
so,
if
I
want
to
create
a
new
node,
I
can
have
a
new
demo
file,
so
I'm
gonna
create
a
new
node,
that's
gonna
be
called
GCE,
singleton
and
all
I
have
to
do
is
cube.
Ketel,
create
dash
F,
my
machine,
yellow
file
and
it
says,
create
a
new
machine
and
then,
if
I
get
machines,
I
will
see
now
there's
a
new
machine
created.
So
what
does
actually
mean?
So?
What
happens
when
I
create
a
new
machine?
Is
that
there's
a
controller
in
this
case?
B
So
if
I
look
at
the
VMs
that
are
running
in
my
project,
you
can
see
the
two
beams
that
were
already
there
and
in
this
there's
a
new
VM
that
showed
up
that
VM
gets
provisioned
I
starts
up,
it
runs
cube,
admin
join
and
after
about
two
minutes,
you'll
have
a
new
node
running
in
your
system.
So
if
we
run
cube,
cuddle
get
nodes
right
now
we
still
just
have
to.
If
we
wait
another
minute
or
so
another
node
will
show
up
in
a
system.
B
So
this
is
great
so
now
I
can
easily
manually
scale
my
cluster
right.
So
if
I'm,
a
cluster
operator
and
I
say
I
want
to
add
more
nodes
to
my
system,
I
can
say
great
cube.
Cuddle,
create
a
chef,
cube,
cut
a
crate
Jeff
that
gets
tiring,
pretty
quickly
creating
nodes
in
that
fashion,
and,
if
you
think
about
kubernetes,
what
we
like
to
do
is
build
sort
of
higher
level
primitives
on
top
of
our
lower
level
primitives.
B
So
what's
the
obvious
thing
you
want
to
build
on
top
of
the
machine
is
a
set
of
machines
right.
So
in
gke
we
call
these
node
pools
where
you
can
have
a
bunch
of
nodes
that
have
the
same
sort
of
template
right.
They
all
look
the
same
and
you
can
scale
them
up
and
down
cops
introduced.
A
very
similar
notion,
I,
don't
think
they
call
it
node
pools,
but
it's
node
sets
or
node
groups
and
Chris
has
instant
sets
something
like
that
and
and
sort
of
it's.
This
is
a
pattern
that
we've
seen
promulgate
throughout.
B
So
what
we've
done
is
is,
rather
than
make
this
first
class
API
I've,
actually
written
a
client-side
tool
to
sort
of
pretend
to
be
a
set
of
machines,
and
so,
if
I,
if
I
check
out
my
client-side
tool,
I
can
say
I
want
to
get
all
of
my
machines
that
are
in
my
set,
node
and
set
node.
Is
a
label?
We've
applied
a
machine
so
we're
just
doing
label
matching
here,
and
we
see
that
there
is
one
machine.
That's
an
ascent,
that's
great!
B
So
if
I,
you
know,
when
I
did
a
cube
call
to
got
machines,
there
were
three
machines:
there
was
the
master.
There
was
this
one
machine
in
the
set,
and
then
there
was
a
third
machine
that
I
created
separately
and
if
I
want
more
I
can
say
scale.
So
let's
say
I
want
five
and
it
says
greats
I
found
one
machine.
That
means
I
need
to
create
four.
It's
gonna
tell
you
that
the
names
of
the
four
machines
that
it
creates
and
then
those
would
get
provisioned,
let's
say
I
accidentally.
B
Do
that
again,
like
that's
fine,
it
says
great
they're
already
five
right.
So
it's
an
idempotent
request.
It
says
we've
already
done
that
work
for
you
we're
not
going
to
turn
the
system
and
if
I
look
at
my
machines,
it's
now
I
see
five
machines
right.
So
I've
got
the
original
machines
that
I
got
when
I
create
the
cluster.
I
had
the
machine
that
I
created
a
couple
of
minutes
ago
and
then
the
four
new
machines
that
were
created
through
scaling,
my
cluster
and
this
client-side
tooling,
is
very,
very,
very
simple
to
write.
B
So
it's
effectively
doing
what
I
did
originally,
where
I
was
saying
queue
petal
creates
with
a
memo
file,
except
in
this
case
it's
inspecting
machines
that
have
this
label
and
just
making
extra
copies
of
them,
and
so
this
works
great,
so
you've
now
you've
got
a
larger
set
of
machines.
So
we
can
see
this
machine
has
come
up
the
one
that
I
created
individually
and
next
we
will
get
four
more
machines
to
join
the
cluster
as
well.
Okay,
so
we
talked
about
manual,
expand
linear
cluster.
B
We
talked
about
scaling
our
cluster
through
a
little
bit
of
automation
through
client-side,
tooling.
That
allows
you
to
scale
up
and
down
really
easily
some
other
things
that
we
mentioned.
That
would
be
really
useful
to
do.
Are
upgrades
right
so
upgrades
a
sort
of
everybody's
favorite
topic.
It's
something
that's
very
difficult
to
get
right
and
kubernetes.
We've
spent
a
lot
of
time
on
the
Google
container
engine
system
trying
to
implement.
You
know
better
upgrades
that
are
less
disruptive
that
aren't
gonna
bring
your
application
down.
B
That
are,
can
do
things
like
just
respect,
paw
disruption
budgets,
but
it's
something
that
has
to
be
re-implemented
in
every
cluster
management
system.
What
we
like
to
do
is
we
like
to
sort
of
consolidate
a
lot
of
the
knowledge
around
how
to
do
really
good
upgrades
in
kubernetes
and
make
that
available
to
everyone?
And
so
here
what
we've
done
is
we've
written
a
tool
that
will
upgrade
your
system,
so
we're
gonna
wait
for
Raleigh's
notes
to
show
up.
It
looks
like
they're.
Almost
there
we've
got
four
that
are
that
are
just
joining
it.
B
Taking
about
maybe
like
15
more
seconds
to
become
ready,
then
we're
gonna
upgrade
so
all
of
our
nodes
right
now
we're
running
174.
So
I
was
the
initial
version,
that's
what
was
specified
in
our
yellow
file.
When
you
looked
at
the
version
of
the
cubelets,
if
we
actually
look
at
the
control
plane,
we
also
specified
one
seven
four.
So
this
is
our
master
node.
You
can
see
that
it's
got
the
rolls
master.
So,
in
addition
to
saying
the
desired,
cubelet
version
we're
also
setting
the
desired
control
plane
version
so
see
if
everybody's
ready
all
right.
B
So
now
we're
gonna
upgrade
so
we're
gonna
upgrade
our
cluster
from
174
up
to
183
and
what
the
upgrader
does
is.
It
is
inspects
the
current
version
of
the
cluster
and
decides
what
to
change
it
to
it.
First
upgrades
the
control
plan
for
the
cluster,
so
it's
gonna
go
and
modify
declaratively
the
control
plane
version
in
the
machine
that
represents
the
master
in
this
case,
and
it's
gonna
set
that
to
be
183.
And
then
it's
gonna
wait
for
the
controller
to
come
in
and
actually
upgrade
the
master
from
one
seven
four
to
one.
B
Eight
three
in
this
case
we're
doing
that
using
a
cube,
a
bin
upgrade,
which
is
a
new
feature
that
was
added
with
kubernetes
1.8
and
so
on.
The
master
machine
we're
running
cube
at
an
upgrade
that
will,
you
know,
upgrade
all
the
control
plane
components
using
cube
admin,
which
is
great
because
that
means
we
don't
have
to
go
and
figure
out
ourselves.
What
does
it
mean
to
upgrade
from
one
seven
to
one?
B
Eight,
like
we've,
figured
that
out
once
for
cube
admin,
we're
reusing
all
the
work
we've
done
there
and
not
reinventing
that
wheel,
and
it
turns
out
a
lot
of
times
when
you're
moving
between
minor
versions
of
kubernetes.
It's
not
as
simple
as
just
installing
the
new
software
version.
You
often
have
to
do
things
like
add
new
auerbach
roles
or
change.
What
system
components
running
to
actually
be
successfully
running
the
new
version?
So
now
we
can
see
that
we've
finished
upgrading
the
control
plane.
B
If
we
look
at
the
version,
we
see
that
sir
version
is
our
running
183
right,
so
the
control
plane
has
been
upgraded.
So
what
it's
doing
next
is
it's
gonna
upgrade
all
of
the
nodes.
So
there
were
six
nodes
in
my
cluster.
There
was
the
one
I
created
I
scale
it
to
up
to
5,
plus
the
singleton,
and
it's
gonna
just
upgrade
all
of
those
in
parallel,
so
I
guess
I
should
have
left
that
window
open.
So
we
could
watch.
B
There
we
go
okay,
so
all
of
our
notes
have
gone
into
the
not
ready
state.
This
is
because,
for
demo
purposes
we're
trying
to
upgrade
as
fast
as
possible,
because
we
have
a
very
short
time
segment
and
what
we're
doing
is
when
we
see
that
we
want
the
desired
version
to
be
different,
we
immediately
kill
all
of
the
VMS
and
create
new
VMs
underneath
them.
So
in
this
case,
all
of
the
nodes
are
not
ready.
B
If
we
watch
our
VMs,
we
should
see
that
we
have
new
VMs
provisioned
for
all
of
those
nodes,
so
we
immediately
kill
VM
as
we
create
new
VMs,
the
VMs
come
up,
they
run
their
startup
script,
the
nodes
will
join
the
cluster
and
then
we
will
have
new
nodes
that,
instead
of
running
one,
seven,
will
be
running
one
eight.
If
you
looked
at,
if
you
were
watching
closely-
and
he
looked
at
the
output
of
the
first
tool
that
we
ran,
it
was
scaling
all
it
did
was
very
quickly
modified.
B
That
declared
state
of
the
cluster
and
exit
right
now
with,
and
then
we
were
sort
of
going
back
and
like
looking
to
see
what
had
changed.
In
this
case,
we
decided
to
make
the
tooling
a
little
bit
smarter
in
this
case.
The
tooling
is
actually
watching
and
waiting
and
telling
an
output
and
telling
us
when
it's
done.
So
it's
watching
the
machine
objects,
the
machine,
knobs
lecture,
recreated
very
quickly.
B
It's
following
the
links
in
the
status
of
the
machine
objects
to
the
corresponding
nodes
and
checking
to
see
when
those
nodes
are
ready
and
running
the
desired
version.
So
this
sort
of
shows
you
different
ways
that
you
can
write
tooling
around
the
cluster
API.
You
can
write
tooling,
that
a
sort
of
fire-and-forget
and
just
expect
the
system
to
reconcile
to
the
desired
State
or
you
can
write
tooling,
that's
actually
following
along
and
watching
what's
happening,
which,
in
you
know
in
some
cases,
can
be
better.
B
B
D
B
B
C
B
A
Yeah
I
would
say
the
one
thing
that
is
important
to
realize.
There
is
because
this
is
a
controller
model.
You,
as
a
software
engineer,
flexible
to
make
that
controller.
Do
virtually
anything
you
want.
You
can
make
that
read
from
other
data
sources
or
make
that
do
literally
anything
you
want
anyway,.
E
B
The
bootstrapping
we're
doing
right
now
is
client-side
we're
creating
the
master,
we're
creating
a
VM,
we're
running,
cube
admin
in
it
and
then
we're
installing
a
CR
D
and
we're
installing
a
controller
form
from
there.
I
think
there
are
a
couple
of
changes.
We're
planning
to
make
in
the
near
future.
One
is
to
switch
from
using
crts
using
API
aggregation,
because
we
believe
we've
heard
from
other
people
that
if
you
see
our
DS,
that
with
API
irrigation,
you
get
much
better
support
for
forward
and
backward
versioning
compatibility.
B
If
you
want
to
mutate
your
API,
and
it
also
allows
us
to
run
the
I
rated
API
server,
either
inside
or
outside
of
the
cluster.
So
then
you
can
stand
up
your
aggregated
API
server
and
you
can
actually
run
the
machine
controller
when
you
don't
have
a
cluster
running
or
if
your
clusters
API
server,
is
broken.
You
can
still
run
your
machine
controller,
which
is
really
nice.
B
We've
also
talked
about
improving
the
bootstrapping
process,
because
the
code
that
actually
is
is
instantiating
that
initial
cluster
with
the
master
is
very
very
similar
to
the
code
that
creates
more
nodes
in
your
cluster
and
it
really
should
just
be
the
same
code.
So
you
defected
Lee,
run
the
machine
controller
locally
to
bootstrap
into
a
cluster
and
then
promote
that
either
into
the
cluster,
to
keep
sort
of
self
manage
or
next
to
the
cluster,
to
manage
outside
of
the
same
failure.
Domain
awesome.