►
Description
Self-Hosted Kubernetes: How and Why [I] - Diego Pontoriero, CoreOS
How Kubernetes is deployed and managed has changed since the first release of the project. From configuration management systems and unit files to deploying Kubernetes using Kubernetes, a lot has changed. Self-hosted Kubernetes has many benefits as a deployment option, and this talk will highlight those benefits, as well as explain the history and nuances of making self-hosted Kubernetes possible.
In this talk I will describe what self-hosted Kubernetes means, why it exists, how it came into existence, and what you need to know if you're running a self-hosted cluster. Many tools now deploy self-hosted clusters including bootkube and kubeadm, so knowledge of how this works can be very important for anybody running a Kubernetes cluster.
What are the benefits of self-hosting? How does it work? What do I need to know if I'm administering a self-hosted cluster?
All those questions and more will be discussed in detail in this talk. In addition, I will discuss how various projects and products take advantage of the many benefits of self-hosting, such as Tectonic.
About Diego Pontoriero
Diego Pontoriero is a Software Engineer on the Tectonic team at CoreOS, where he works on software that deploys, manages, and upgrades self-hosted Kubernetes clusters. Prior to CoreOS Diego worked at Google building a video-based learning platform, a mobile phone carrier, and a petabyte-scale data warehouse.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
Okay
know
a
lot
to
cover,
so
we're
gonna
get
started
hi.
My
name
is
Diego
I
work
at
core
OS
and
I'm
here
to
talk
to
you
about
self,
hosted,
kubernetes,
so
core
OS,
we
use
self
hosted
kubernetes
in
our
enterprise,
kubernetes
distribution,
which
is
called
tectonic.
We've
been
doing
this
for
quite
a
while
now
in
production,
with
lots
of
customers.
Customers
doing
this
and
we
think
it's
the
best
way
to
run
to
run
kubernetes.
So
this
talk
I'm
just
going
to
kind
of
try
to
prove
that
to
all
of
you.
A
So
who
is
this
talk
for
well?
A
lot
of
different
people.
First,
be
cluster
operators.
If
you're
running
a
cluster,
maintaining
a
cluster
multiple
clusters,
upgrading
them,
self-hosted
kubernetes-
is
one
approach
you
can
take
that
we
think
is
a
good
one
that
will
make
your
life
easier
and
a
lot
of
dimensions
I'll
get
into.
Also,
if
you're,
a
kubernetes
contributor,
you're,
just
interested
in
kubernetes
self,
hosted
communities
touches
on
a
lot
of
interesting
aspects
of
kubernetes
api
surface
and
things
like
that.
We
had
a
lot
of
kind
of
corner
cases
we
had
to
solve.
A
So
it's
it's.
It's
cool
from
a
community's
point
of
view
just
to
secret
is
running
itself
and,
lastly,
people
who
enjoy
clever
hacks,
so
we
had
to
come
up
with
a
couple
of
neat
tricks
in
order
to
make
career
a
cell
phone
secure,
Mary's
work
that
I'll
get
into
during
this
talk,
and
so
I
think
they're
kind
of
cool,
and
hopefully
you
do
too
so
first
to
get
a
run
on
the
same
page.
I
want
to
talk
about
what
is
self
hosted,
kubernetes.
Well,
it's
somewhat
self-evident,
but
it's
kubernetes
running
on
kubernetes.
A
Specifically
all
the
communities,
control,
plain
components
are
running
as
native
kubernetes
objects,
namely
deployments
daemon
sets
they're
using
things
like
secrets
and
that's
about
it
in
terms
of
what
it
actually
is,
but,
of
course,
the
Devils
in
the
details
right.
So
this
talk
is
gonna.
Come
in
three
parts,
first
I'm
going
to
talk
about
why
you'd
want
to
use
self-hosted,
kubernetes
I'm
gonna
make
the
case
for
why.
We
think
this
is
the
best
way
to
run
kubernetes.
A
Today
it's
work
great,
but
it
also
unlocks
a
lot
of
possibilities
for
what
you
can
do
once
you
have
a
self-hosted
cluster
in
doing
some
pretty
sophisticated
stuff
in
terms
of
managing
your
cluster
scaling,
it
up
that
sort
of
thing
and
so
I'll
touch
on
that
in
the
end.
So
first,
let's
go
over.
Why
you'd
want
a
self
host
a
kubernetes
cluster?
A
Well,
first
of
all,
you
want
to
leverage
Cabrini
strengths.
We
have
kubernetes
it's
great
at
running
highly
available
resilient
software
wouldn't
be
great
if
we
could
actually
run
kubernetes
taking
advantage
of
all
of
that
useful
stuff,
that's
encoded
in
kubernetes.
Secondly,
it
really
simplifies
your
node
management
story,
so
it
turns
out
once
you're
using
self-hosted
kubernetes.
Your
node
requirements
are
very
simple
and
I'll
describe
that
a
little
bit.
A
Lastly,
it
really
makes
your
cluster
lifecycle
management
a
lot
easier
because
you're
managing
using
the
same
lifecycle
management
tools
that
you
use
when
you're
managing
any
application
on
kubernetes.
You
know
coop,
cuddle
and
friends
like
that,
so,
if
you're
running
a
kubernetes
control
plane,
what
might
be
some
properties
that
you'd,
like
this
control
plane
to
have
well
for
one
thing
you
might
want
it
to
scale
it
up
and
down
automatically.
Let's
say
your
cluster
is
growing
you're
getting
more
users.
More
people
are
hitting
the
kubernetes
api.
A
You
might
want
to
scale
up
the
number
of
API
servers,
you're
running
or
controller
managers
or
so
on.
You
also
might
want
to
handle
node
failures,
gracefully
so,
let's
say
one
of
your
masters
goes
down.
You
want
to
quickly
bring
up
another
master,
to
run
the
components
that
we're
running
on
that
failed
node,
so
that
you
don't
have
any
downtime
so
that
you'd
you'd
your
Roccat
workloads,
keep
running.
A
A
What,
if
you
set
a
flag
that
it
turns
out,
breaks
some
part
of
your
control
plane,
you
won't
be
able
to
roll
back
to
the
last
known,
good
state
that
you
had
and
while
we're
at
it
listing,
you
know
making
our
wish
list
here.
What
about
advanced
networking?
What,
if
you
want
to
run
some
sort
of
network
policies
that
can't
constrain
how
your
control
plane
talks
to
each
individual
component
into
other
parts
of
your
cluster?
What
if
you
want
to
have
role-based
access
access,
control
and
auditing
to
control
and
keep
track
of
your
control
plane?
A
Who
is
talking
to
what
it's
allowed
to
do?
What
about
health
checking
monitoring
if
you're
deploying
Prometheus
in
your
cluster,
if
you're
looking
at
live?
'no
stand
health
checks?
You
want
your
control
plane
to
have
that
to
make
sure
that
you
know
very
quickly
if
there's
a
problem.
Lastly,
what
about
resource
allocation
and
accounting?
You
don't
want
to
waste
compute
by
having
snowflake
master
nodes
that
aren't
optimal
using
the
resources,
and
you
also
might
want
to
know
how
much
is
my
control
plane
costing
to
run
how
many
resources
is
actually
using.
A
So
if
you
think
about
it,
what's
really
good
at
doing
all
this
stuff,
I
mean
the
obvious
answer,
is
kubernetes
itself
right
wouldn't
be
great
if
we
could
just
run
our
control
plane
this
way,
something
else
that
self-hosting
unlocks
is
simplified,
node
management.
So
if
you
think
about
kubernetes
worker,
so
forget
about
masters
for
a
second.
What
do
you
really
need
to
run
kubernetes
worker?
Well,
you
need
the
couplet
right.
That's
what's
gonna
talk
to
the
API
server
and
figure
out
which
pod
should
be
running.
You
need
to
contain
a
runtime.
A
You
know
docker
container,
D,
rocket,
etc.
That
is
actually
gonna
run
your
pods.
Now
you
need
some
credentials
to
talk
to
the
API
server.
You
know,
let's
say
a
coop
config,
and
that's
it
really
most
workers,
that's
what
they.
What
they
run
is
a
very
minimal
subset
of
things
compared
to
a
master
where
you
might
be
running
specific,
API
server,
controller
manager,
etc,
system,
D
units,
if
you
will,
or
whatever
else
so
in
a
self-hosted
world.
This
is
all
you
need,
because
everything
is
running
in
kubernetes.
A
Everything
is
really
a
worker,
there's,
no
distinction
between
a
master
and
a
worker
from
a
compute
point
of
view.
So
in
that
case,
how
do
you
select
a
master
now?
Well,
it's
actually
just
as
simple
as
applying
a
label.
The
only
difference
between
master
and
non
master
nodes
is
that
they
happen
to
have
labels
on
them
and
so
again,
using
simple
using
the
kubernetes
built-in
primitives,
it's
simpler,
it's
more
uniform
and
more
flexible
and
I'll
get
into
that
a
little
bit
later
as
well.
A
Lastly,
what
about
lifecycle
management?
What
about
upgrading
your
cluster,
changing
Flags
in
your
cluster,
adding
new
rotating
your
certs,
let's
say
from
your
API
server!
Well,
all
you
do
is
coop
cuddle
apply,
coop
cuddle
edit,
and
you
can
go
in
and
change
anything
and
then
kubernetes
will
roll
out
those
changes
in
the
way
we
all
know
and
love.
Now,
realistically,
you
know
your
control
plane
is
your
most
important
part
of
your
community's
cluster
from
you
know,
in
one
point
of
view,
so
you
don't
really
want
to
be
using
coop
cuddle
in
production.
A
A
So
when
you
want
to
create
a
self
hosted
clusters,
it's
three
main
areas
that
you
want
to
address
in
order
to
have
something:
that's
production
ready.
That
is
something
you
could
actually
use
to
run
real
workloads.
The
first
is
bootstrapping.
How
do
you
actually
create
a
cluster?
It's
not
it's.
If
you
start
to
think
about
it's
actually
not
that
easy,
and
then
what
about
upgrades
I
touched
on
it
a
little
bit,
but
in
practice,
how
does
it
work?
How
do
you
actually
upgrade
your
holcomb
control,
plane
and
then
disaster
recovery?
A
Inevitably
something
will
cantle
things
can
will
go
wrong.
How
do
you
make
sure
that
self
hosted
cluster
is
recoverable,
so
I'll
go
through
each
of
these
one
by
one
and
I'll
actually
jump
over
the
terminal
and
make
a
cluster
and
then
break
it
for
all
of
you
and
we
can
see
how
this
works.
So
the
first
step
that
we
need
to
figure
out
is
bootstrapping.
How
do
you
actually
create
a
cluster
from
scratch?
A
If
you
think
about
it,
the
control
plane
as
a
illustrator
earlier
is
running
as
daemon
sets
and
deployments
right,
but
you
need
a
control
plane
to
create
Dennis
sets
and
deployments
right.
You
know
how
can
you
say,
could
cuddle
create
if
there's
no
API
server
to
talk
to
so
here
we
have
clever
hack
number
one,
which
is
that
we're
gonna
create
a
temporary
static
control
plane
to
bootstrap.
A
How
do
you
actually
do
this
in
practice?
There
are
a
few
few
projects
that
are
doing
this.
We
want
I'm
going
to
talk
to
you
today
is
called
boot
cube.
It's
a
kubernetes
incubator
project
that
I
work
on
a
few
other
people
here
and
the
way
it
works
is
you
give
it
temporary
control
plane
manifests,
so
these
are
pod,
manifests
I'd
purely
describe
pods
because
they
need
to
run
on
the
couplet
there's
no
control
plane.
Yet
the
couplet
only
really
knows
how
to
run
pods
seas
are
gonna,
describe
your
API
server
controller
and
then
scheduler.
A
You
also
need
your
self
hosted,
control
plane
manifest.
So
this
is
actually
gonna
describe
your
permanent
self
hosted
cluster
once
it
gets
running
and
then,
lastly,
you
need
an
initial
master,
node
and
I
put
master
here
in
parentheses
because,
as
I
said,
there's
nothing
really
special
about
the
master
node.
It's
just
the
node
that
we
so
happened
to
choose
to
be
our
initial
node
that
we're
going
to
bootstrap
on
so
boot.
A
Coop
is
going
to
take
these
three
inputs,
they're
just
files
on
disk
and
your
node
and
it's
gonna
stand
up
a
temporary
control
plane
and
then
it's
gonna
pivot,
that
into
the
self
hosted
control
plane.
So
let's
actually
launch
a
cluster
and
take
a
look
at
that
so
here,
can
everyone
see
this
okay
cool?
A
So
I've
got
a
few
nodes
sitting
around
here
on
my
laptop,
so
let's
look
at
the
actual
manifest
we're
gonna
use
for
a
sec,
so
fukube
actually
provides
a
nice
little
render
tool
that
can
create
manifests
that
are
good,
they're
kind
of
baseline
to
run
a
self-hosted
cluster.
If
you
want
to
go
run
your
own,
you
can
just
download
buku
play
with
this.
In
practice.
A
You
really
want
to
cuss
these
assets
to
match
your
the
specific
configuration
you
want,
but
let's
so,
if
we
look
at
what
we've
got
so
we
can
see,
we've
got
a
coop
config
up
here.
We've
got
these
bootstrap
manifests
that
I
was
talking
about,
so
we've
got
an
API
server,
controller
manager
and
scheduler
and
those
are
just
pods.
Then
we've
got
a
bunch
of
other
manifests
and
this
is
actually
our
control
plane.
That's
going
to
run
so.
We've
got
API
server,
controller
manager,
scheduler
we've
also
got
coop
DNS,
which
is
an
add-on.
A
We
like
it's,
got
coop
flannels
we're
actually
deploying
our
network
overlay
as
part
of
this
self
hosted
cluster
and
then
a
few
other
things,
our
back
rules
and
so
on,
and
there
go
our
PKI
material
that
we're
gonna
need
to
have
a
secure
cluster.
So
if
I
hop
over
to
my
master
node,
you
can
see
here
I've
copied
over
the
assets
already
and
I
have
buku.
So
what
I'm
gonna
do
is
I'm
just
gonna
say
so.
A
A
What
it's
done
is
it's
copied.
These
bootstrap
manifests
over
to
the
couplet
manifest
directory,
so
it's
starting
them
up
as
static
pods
and
once
that's
once
that's
done
it's
going
to
use
that
temporary
control
plane
to
create
the
self-hosted
assets.
So
since
looking
at
text
is
not
actually
that
enlightening,
I'm
going
to
illustrate
this
with
pictures,
and
thanks
to
my
coworker
Aaron,
who
originally
made
these
really
good
slides.
So
here
here
is
just
gonna
reflect.
What's
going
on
in
my
terminal
right
now,
we've
got
the
coolant.
We've
got
boot
Kubb
and
we've
got
at
CD.
A
The
first
thing
boot
cube
does:
is
it
creates
static
pods
as
I
show
it?
It's
just
by
copying.
These
static
manifests
over
to
the
Kubla
directory,
and
so
we
have
an
API
server,
scheduler
controller
manager
once
they
get
up
and
running
they
talk
to
at
CD.
You
effectively
have
a
functioning
control
plane.
The
couplet
is
configured
to
look
at
this
API
server
address,
and
so
it
actually
joins
the
cluster.
So
we
have
this
temporary
control
plane
and
we
have
the
Kubla.
It's
a
one.
A
Node
cluster
fukube
detects
that
this
cluster
is
up
and
running,
and
then
it
creates
a
self-hosted
components
effectively
by
calling
coop
code
will
create,
and
so
the
couplet
since
is
talking
to
api
server,
sees
that
it
should
be
running
these
pods.
So
it's
going
to
start
up
the
actual
self-hosted
api,
server,
scheduler
and
controller
manager,
but
really
it's
just
functioning
as
a
Kubla
normally
does
running
a
daemon
set
deployment,
etc.
A
Once
a
self-hosted
control
planes
running
boot,
cout
tears
down
the
static,
pods
they're
not
needed
anymore.
It
sees
that
the
self
hosted
cluster
is
running
and
then
the
cube
itself
exits
boot
kube's
done.
We
never
use
boot
cube
again
in
an
ideal
world
and
the
self
hosted
cluster
is
now
talking
the
same
at
CD
instance.
So
that's
the
kind
of
trick
that
allowed
us
to
use
the
temporary
control
plane
in
the
final
control
plane
and
the
pivot,
and
the
couplet
now
talks
to
this
API
server
and
we
have
a
one
node
cluster
from
there.
A
We
can
start
joining
other
nodes
and
expand
our
cluster
accordingly.
So
let's
go
back.
Look
our
cluster!
Ok
cool!
It's!
It's
all
done!
We
have
it
all
self
hosted,
control
plane
is
successfully
starts,
aboot,
Kubb
exited
we're
done
with
it.
So
let's
actually
look
at
our
cluster,
so
you
can
see.
We
got
two
nodes,
we
got
our
master
and
then
I
had
a
worker
sitting
around.
That
was
just
waiting
for
an
API
server
to
appear,
and
then
we
actually
have
our
control
planes.
A
So
if
we
say
coop
system
get
deployments,
got
our
controller
manager,
scheduler,
etc.
Our
daemon
sets
we've
got
cube,
API
server,
flannel,
coop,
Roxy,
etc.
Right
cool,
alright!
So
now
we've
got
a
cluster,
but
as
we
know
in
kubernetes
land
things
are
moving
very
fast,
so
you
want
to
upgrade
it
right.
How
did
we
upgrade
it?
Well?
This
is
probably
the
most
boring
slide
of
the
talk.
Actually,
we
just
go
and
change
the
image
in
our
daemon
set.
It's
actually
that
easy,
so
I'm
gonna,
say
coop
cuddle
edit
I'm
gonna
have
to
put
that
first.
A
A
B
A
Alright,
so
a
cheat
we're
saying
well
in
185
came
out
what
yesterday
two
days
ago,
that's
let's
upgrade
to
that.
You
want
to
be
on
the
bleeding
edge
cool.
Now
it's
get.
Damon
sets
a
wide,
so
kinda
hard
to
see
when
it's
blown
up,
but
you
can
see
that
a
coup,
baby,
Ice
Cube
API
server,
has
been
upgraded
to
185
and
thanks
to
a
daemon
set
rolling
update
strategy.
A
It's
gonna
do
this
in
a
safe,
safe
manner,
and
as
I
mentioned,
we
don't
want
to
really
be
using
coop
cuddle,
edit
our
production
clusters,
possibly,
but
you
can
in
a
pinch
or
in
an
emergency,
you
can
go
in
and
you
can
fix
something
this
way.
Alright.
The
last
piece
I
want
to
cover
here
is
des
step
disaster
recovery.
So
thank
you.
Things
can
and
will
do,
go
wrong
with
your
cluster
either
due
to
bugs
operator
error.
What
have
you
so?
What
kind
of
failure
modes
might
you
see
in
a
self-hosted
cluster?
A
First,
you
have
a
partial
control,
plane
loss.
Let's
say
that
someone
is
editing
their
Kubb
scheduler
deployment
accidentally
scales
it
down
to
zero.
Well,
you've
got
a
problem
now
because
you
can,
you
can
scale
it
back
up,
but
there's
no
scheduler
there
to
decide
which
no,
the
the
it
should
be
running
on.
So
how
do
you
deal
with
that?
Well,
in
that
case,
you
have
to
recover
recover
the
scheduler
itself.
What
about
if
you
lose
the
whole
control
plane?
A
Let's
say
you
know:
I
have
a
bare-metal
cluster
in
my
basement,
I
accidentally
go
and
trip
the
fuse
and
I
started
back
up.
We
need
to
restart
things
right,
so
we'll
have
to
recover
the
entire
control
plane
and,
lastly,
you
might
lose
a
cluster
completely.
Let's
say
I
accidentally
delete
my
auto
scaling
group
in
AWS.
Well,
hopefully,
you
have
backups
and
you
can
recover
from
a
backup.
A
I'm
gonna
make
a
brief
interlude
to
talk
about
a
pod
check
pointer,
it's
another
thing
that
we
use
when
self
hosting.
So
if
some
of
you
were
I
know,
I
could
went
kind
of
fast,
but
a
keen
observer
might
have
noticed
a
trick
during
the
upgrade
demo.
I
said
it
was
easy
is
not
actually
that
easy.
How
do
you
up
create
an
API
server?
A
How
do
you
actually
handle
master
node
reboots,
your
API
server,
when
I
updated
it
I
have
a
single
master
cluster
here,
it's
not
actually
what
you
want
to
do
in
the
real
world,
but
my
little
laptop,
that's
all
I
can
handle.
So
when
I
updated,
the
API
API
server,
you
think
about
a
daemon
set.
What
it's
gonna
do
is
it's
going
to
terminate
the
old
pod
and
once
it's
fully
terminated
start
up
a
new
pod.
Okay,
we
terminated
the
API
server,
then
what
there's
no
API
server?
No
one!
A
No
one's
telling
us
what
to
do
next
right.
So
how
did
this
work
well?
Welcome
to
clever
hack
number
two
when
we
run
a
checkpoint
or
daemon,
and
this
is
actually
running
on
all
the
masters-
it's
a
pretty
simple
daemon,
but
what
it
does
is
for
certain
critical
pods.
It
creates
local
checkpoints,
it
copies
down
the
static
manifests
and,
in
the
case
of
a
control
plane
outage,
it
will
actually
deploy
that
manifest,
wait
for
the
cluster
recover
and
then
on.
We
go,
and
so
during
an
upgrade.
A
This
is
actually
a
mini
outage
that
we're
creating
on
purpose
and
the
checkpoint.
Er
is
what
allows
us
to
do
that
upgrade
by
temporarily
running
a
local
API
server
waiting
for
the
new
API
server
to
start
back
up
and
then
decommission
the
checkpoint,
so
I
can
illustrate
through
some
pictures.
So
let's
say
we
have
our
couplet
and
our
API
server.
The
check
pointer
is
watching
both
and
it's
basically
to
try
and
reconcile
the
API
server
says
that
pod
one
and
two
should
be
running.
A
The
couplet
says
that
pod
one
and
two
are
running
so
we're
good.
This
is
our
steady
state.
The
check
pointer
will
create
these
inactive
check
points
that
are
just
on
standby.
Let's
say,
pod
two
disappears
in
the
Kubla.
Let's
say
it's
the
API
server.
It's
like
this
upgrade
example
you're
talking
about
and
it's
not
able
to
start
a
new
one,
because
there's
no
API
server.
If
you
have
this
chicken-and-egg
problem,
the
check
pointer
will
see
this.
It
will
activate
the
static,
manifest
a
check
point.
This
pod
will
start
the
it's
an
API
server.
A
So
now
the
couplet
can
talk
to
it
and
decide.
Oh
I
should
be
running
an
API
server,
so
it'll
actually
run
the
real
API
server
pod.
The
check
pointer
sees
ok,
API,
server
and
Kubler
matching
again
and
it'll
just
retire.
The
static
pod,
so
pretty
simple,
but
not
obvious,
and
it
helps
us
both
in
upgrades
and
in
reboot
situations
and
we're
back
to
our
steady
state.
A
A
We
need
a
bigger
tool,
so
if
only
there
was
a
way
to
jumpstart
a
cluster
jumpstart
sounds
kind
of
like
bootstrap,
so
clever
hack
number
three
comes
in,
which
is
we
actually
use
boot
cube
to
extract
manifest
from
the
cluster
and
then
run
another
temporary
control
plane.
So
just
like
when
we
started
the
cluster
the
first
time
we
can
do
the
same
trick
to
create
a
control
plane,
that's
pointing
at
our
same
at
CD,
fix,
whatever's,
broken
and
then
on.
A
C
A
So
we're
gonna
scale
our
coupe
scheduler
to
zero
replicas,
because
why
not?
So
if
we
say
you
know
get
pods,
we
see
got
two
schedulers
that
are
terminating
that's
bad
news,
so
we
can
do
go
away.
Okay,
they're
gone
well,
oops
I
didn't
really
mean
to
schedule
it
to
two:
maybe
I
didn't
at
ten
or
something
sorry:
zero
I'm
at
ten.
So
let's
scale
it
back
up.
A
Okay,
that
worked
right,
API
server
said
sure:
that's
you
want
to
do,
but
then
it's
gonna
ask
the
scheduler
to
schedule
those
pods
and
oh,
we
don't
have
a
scheduler.
All
we
have
is
pending
pods
they're,
not
gonna,
get
scheduled
right.
So
what
do
we
do?
We
can
use
boot
kübra
cover,
so
let's
hop
to
the
server
back
to
our
master,
really
quick,
and
so
we
can
say
boot
Kubb
recover
and
then
we
say:
recovery
derp
is
home
core
recover
and
then
coop
config,
because
it
needs
to
talk
to
the
API
server.
A
In
this
case
we
still
have
an
API
server,
Etsy
kubernetes,
good
config,
all
right.
So
this
talk
to
the
API
server
and
it
recreated.
These
bootstrap
manifests
because,
if
you
think
about
the
ones
we
used
to
start
the
cluster,
that's
a
one-time
thing.
This
could
be
a
year
later.
Our
cluster
is
very
different.
We
can't
reuse
those
assets,
but
this
one
is
gonna,
create
boot,
cube,
start
friendly
assets,
so
we
boot
could
start
it.
That's
boot
could
start,
and
this
time
we
say
the
asset
der
is
home.
A
Sorry
Oh,
cluster
recover
I
was
sorry
core
recover.
Thank
you
all
right.
So
it's
the
same
process.
It's
gonna
create
a
temporary
control
plane.
As
soon
as
that
happens,
we're
gonna
have
a
scheduler.
It's
gonna
be
able
to
schedule
two
missing
schedule,
pods
and
we're
good
again.
That's
just
gonna.
Take
a
second
for
the
couplet
to
take
care
of
this.
A
So
well,
that's
recovering
and
I'll
check
on
in
a
second
I
want
to
talk
about.
What's
next
for
self
hosting
cell
posting
is
a
good
place,
as
I
mentioned.
Tectonic
is
core
West
distribution,
we're
using
self
hosting
in
production.
Today
lots
of
clusters,
it's
stable
and
great,
and
has
all
these
nice
properties
I
talked
about,
but
it
also
unlocks
a
lot
of
interesting
things.
We
can
do
to
kind
of
take
our
cluster
management
to
the
next
level.
So
one
is
automated
operations.
A
There's
a
lot
of
things
you
can
do
now
that
these
are
kubernetes
objects
that
you
can
manage
using
the
cube
Nettie's
api.
One
is
cluster
upgrades,
as
I
said
before,
you
don't
really
want
to
be
using
coop
cuddle
apply
right.
You
want
to.
Maybe
you
use
client,
go
and
code
some
logic
and
specifically
want
to
have
kind
of
fine-grained
control
over
rollout
ordering
it
turns
out
you're.
You
really
want
to
upgrade
your
API
server.
First,
then
your
scheduler
controller
manager
and,
lastly,
your
couplets.
A
Once
everything
else
is
okay,
you
also
want
to
maybe
handle
some
pre
and
post
upgrade
operations.
Turning
on
a
new
flag,
updating
some
credentials,
things
like
that
you
can
encode
this
in
an
operator
also
couplet
upgrades.
So
how
do
you
actually
upgrade
a
couplet?
Well,
it
that's
one
thing
that
is
running
statically
on
each
machine,
but
what
why
we
just
deploy
a
daemon
set
that
can
actually
run
on
every
machine
and
make
the
necessary
changes
it
could
you
know
rpm
or
Deb,
install.
A
And
lastly,
a
configuration
management,
as
we
know
things
change
over
time.
You
might
want
to
change
our
cluster.
Ideally,
we
don't
want
to
have
to
tear
down
our
cluster,
make
a
new
one.
So
one
thing
we
can
do
is
actually
have
our
operator
go
in
and
change
flags
and
do
the
right
coordination,
it
could
be
might
might
be
a
nuance
process,
but
it's
something
you
can
encode
in
software
and
it's
much
less
error-prone.
That
way
we
can
even
do
things
like
deploying
a
new
network
overlay
or
changing
your
network
settings.
A
A
Similarly,
we
talked
about
node
management,
how
that's
simpler,
but
it
also
by
having
these
unified
nodes,
it
unlocks
a
lot
of
power
as
well.
One
is
self-healing,
so
I
just
did
a
boot
kübra
cover
manually,
that's
also
not
ideal.
I
had
to
SSH
the
Machine
I
had
to
run
some
fans.
What
if
there
were
no
dajun's
that,
if
the
operator
senses
that
there's
a
control
plane
problem,
could
remotely
invoke
the
kübra
cover
or
something
equivalent
to
heal
your
control
plane
or
what
about
auto
scaling?
A
You
could
increase
the
number
of
masters
if
you're
experiencing
higher
load
or
if
your
requirements
increase
over
time
or
what
about.
If,
since
all
nodes
are
the
same
anytime,
your
autoscaler
adds
a
new
node.
It
talks
to
the
operator
and
says:
hey
I'm
in
your
node.
What
should
I
be?
Should
I
be
a
master
or
should
I
be
a
worker,
should
I
have
some
other
specialized
role
right,
since
all
nodes
are
the
same,
they
can
be
provisioned
as
anything
and,
lastly,
node
identity.
A
So,
in
recent
grades
versions
something
called
TLS
bootstrapping
was
added
which
allows
every
node
to
have
a
unique
identity
by
performing
a
challenge
and
response
with
the
API
server
when
it
joins
the
cluster.
And
so
since
our
nodes
are
simple
and
have
very
little
state,
they
can
use
this
very
nicely
and
we're
very
close
to
actually
merging
this
into
boot.
Coop
should
be
in
hopefully
next
week,
so
let's
just
take
check
bag
on
our
cluster
sec
yeah,
it's
back!
A
So,
let's,
let's
look
at
our
pods
here
and
yes,
our
scheduler
is
back
we're
happy
cool,
so
boo,
cube
is
a
a
Cabrini's
incubator
project
and
tectonic
is
core
os's
specific
distribution
of
communities,
but
we
are
trying
to
also
get
this
support
actually
in
upstream,
so
that
more
commodious
users
can
access
it.
So
cube
ATM,
which
is
the
upstream
tool
for
deploying
clusters,
is
adding
community
support
a
self-hosted
create
support.
It's
almost
done,
I
think
you
can
actually
launch
clusters
now
and
upgrading
is
coming
next.
A
The
check
pointer
instead
of
being
a
standalone
pod,
is
actually
being
added
to
the
couplet
itself,
and
that
is
already
done
for
pods
and
we're
just
adding
support
for
secrets
which
is
necessary
for
the
API
servers.
The
way
we
run
them,
but
as
always,
we
need
help
if
you're
interested
in
this
kind
of
stuff.
If
you're
interested
in
coming
up
with
more
clever
hacks
check
out
cig
lifecycle,
you
can
join
us
on
slack
or
our
mailing
list
or
and
come
to
our
weekly
meetings.
So
that's
all
I've
got.
We
have
a
few
minutes
for
questions.
B
A
Natively
self
hosting
at
CD
is
something
we've
worked
on.
It.
We've
actually
uncovered
a
lot
of
bugs
in
the
API
server
as
a
result
in
terms
of
connections,
because
when
you're
running
the
self
hosted
at
CD
on
the
pod,
Network
and
they're
moving
around
a
lot,
it
turns
out.
The
API
server
is
not
actually
resilient
to
reconfiguring
these
connections.
A
It
just
was
something
that
we'd
never
exposed
before,
so
we're
working
on,
trying
to
kind
of
fix,
ha4
self-hosted
at
CD
and
then
we'll
reevaluate,
whether
that's
something
to
do
I
think
it's
something
that
makes
a
lot
of
people
nervous
because
that's
it's
at
CDs
kind
of
the
baseline
building
block.
But
on
the
other
hand,
you
can
launch
the
self
host,
add
CD
clusters
for
use
not
for
your
control
plane
and
that
works
really
nicely.
A
E
What
are
the
bare
requirements
that
you
need?
You
might
have
done
it
and
I
missed
it.
In
order
to
get
you
need
to
run
the
couplet
directly
on
the
node
right,
that's
not
obvious!
You
can't
sell
foes
to
cool
it.
In
the
couplet
we
tried
that
one
is
a
little
little
too
tricky
and
adding
new
nodes
to
the
collect
I.
Guess
it
doesn't
matter
where
the
control
pain
runs
or
its
daemon
set.
So
I'm
also
confused
about
the
the
proxy
good.
A
Flamel
also,
yes,
so
I
kind
of
lighted
this
for
time,
but
the
so
we
run
the
control
plane
on
nodes
that
are
labeled
as
master
nodes
right.
So
we
said,
we
label
the
nodes,
we
set
selectors
on
our
control
plane
and
we
set
taints
on
those
nodes
so
that
other
workloads
don't
run
on
them
and
then
for
the
API
server
and
coup
proxy.
Yes,
those
need
to
have
network
connectivity.
So
that's
why
they
run
as
daemon
sets
using
host
networking,
but
the
controller
manager
and
the
scheduler
we're
able
to
run
as
deployments
since
that.
A
E
F
A
The
checkpoint
runs
conservatively,
it's
basically
trying
to
reconcile
the
state
between
the
last
time,
I
talked
to
the
API
server
and
what
it's
seeing
on
the
couplet,
which
I
can
always
see
because
it's
running
on
the
couplet
itself.
So
if
the
API
servers
down
or
if
there
is
a
network,
a
network
partition
or
something
it'll
do
the
conservative
thing,
but
in
kubernetes
land.
That's
what
you
want
right,
it's
if
you're
running
an
extra
pod.
Theoretically,
that'll
be
okay!
F
A
F
C
B
A
A
But
if
you're
maintaining
a
permanent
control
plane,
then
you
lose
you.
You
have
double
the
overhead
and
you
don't
have
the
benefits
of
self-hosted
control
plane
in
terms
of
management
and
stuff
right.
So
if
you're,
having
main
to
maintain
both,
it's
almost
asked.
What's
what's
the
point
in
that
case.