►
Description
KubeCeption! A Story of Self-Hosted Kubernetes - Aaron Levy, CoreOS, Inc.
"Self-hosted Kubernetes is here, and with it you can upgrade Kubernetes using nothing but Kubernetes, manage the full cluster lifecycle using nothing but kubectl, and run Kubelets with Kubelets.
Wondering what a self-hosted Kubernetes cluster actually is? This talk will go over how we can vastly improve and simplify management of Kubernetes clusters by getting out of its way, and letting Kubernetes manage itself. At the end of the day, Kubernetes is oftentimes better at managing applications than the tools we build to manage Kubernetes.
From the initial node bootstrap, to a scaling a highly-available Kubernetes installation - we will cover how it works through each stage of deploying a self-hosted cluster, and what it looks like to use Kubernetes to manage Kubernetes."
About Aaron Levy
Aaron Levy is a software engineer working on all things Kubernetes at CoreOS.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
I'm
gonna
go
again
head
and
get
started
in
custards,
quite
a
bit
of
stuff
to
go
over.
So
thanks
everyone
for
joining
me
on
the
early
session.
On
the
second
day,
my
name
is
Aaron
Levie
I'm,
a
software
engineer
at
core
OS
and
one
of
the
things
I
want
to
talk
about
today
is
self
hosted
kubernetes.
So
when
I
say
self
hosted,
kubernetes
kind
of
show
of
hands.
Has
anyone
heard
this
phrase?
A
Does
it
mean
anything
awesome,
sweet
so
just
to
start
off
a
quick
explanation
of
you
know
what
exactly
is
so
postal
kubernetes
when
I
say
this?
What
I?
What
I
mean
is
that
you
know
communities
is
responsible
for
managing
its
own
core
components
and
that
those
core
components
are
actually
deployed
as
native
API
objects.
So
that
means
things
like
the
API
server,
the
scheduler,
the
controller
manager.
These
are
modeled
as
deployments
and
daemon
sets
and
secrets
and
services.
A
So
if
we
were
to
actually
look
at
a
self-hosted
control
plane,
this
is
what
it
would
look
like.
If
we
said
coop
control
get
deployments,
get
daemon
sets
get
secrets.
We
can
see
that
for
deployments
we
have
our
controller
manager
and
our
scheduler
and
we're
running
a
couple
copies
for
daemon
sets.
We
have
the
API
server
and
the
proxy,
and
then
we
also
distribute
some
of
the
assets
that
these
components
need
as
secrets
so
for
the
API
server.
That
would
be
things
like
the
TOS
assets,
the
controller
manager.
A
That
might
be
things
like
the
service
account
keys.
So
that's
just
like
really
brief,
like
well,
actually
one
one
other
point
that
I
wanted
to
make.
When
I
say
it's
self
hosted
communities
and
it's
responsible
for
managing
its
own
components,
what
we're
thinking
of
as
a
flat
cluster.
So
this
isn't
like
a
control
plane
in
kubernetes
than
managing
multiple
other
kubernetes
inside
of
it.
This
is
actually
the
control
plane
of
kubernetes
managing
itself
in
a
single
flat
cluster.
A
So
if
we
have
a
self-hosted
cluster
or
what's
important
about
this,
why
do
we
want
this?
One
of
the
reasons
is
that
you
know
we've
already,
you
know
drunk
the
container
kool-aid
like
we
all
believe
in
container
izing
our
applications.
We
think
that
that's
the
right
way
forward.
So
why
don't
we
do
the
same
thing
for
kubernetes?
It's
you
know,
there's
many
ways
that
we
can
actually
deploy
the
kubernetes
components.
A
We
could
run
them
as
binaries,
but
if
we
already
kind
of
believe
in
this
this
method
of
running
our
applications,
why
not
do
the
same
for
kubernetes
and
then
two
other
really
big
reasons
and
I'll
go
into
more
detail?
Are
that
by
self-hosting
kubernetes
we
can
vastly
simplify,
simplify
how
we
bootstrap
our
nodes
and
then
also
greatly
simplify
how
we
actually
manage
the
lifecycle
of
the
cluster
so
going
over.
How
we
actually
can
simplify
the
the
node
bootstrap.
A
One
of
the
main
things
is
that,
when
we're
saying
that
all
of
the
components
in
kubernetes
are
going
to
be
run
as
as
communities
objects,
we're
essentially
saying
that
we're
going
to
be
stacking
them
on
top
of
the
Kubla
and
removing
that
necessity
of
configuring
them
on
the
host.
So
what
we
get
the
point
that
we
get
to
is
that
our
on
host
requirements
essentially
become.
A
We
have
a
couplet,
we
have
a
container
run
time
and
we
have
a
coop
config
with
enough
information
of
how
to
contact
an
API
server
and
that
that's
essentially
what
you
need
to
configure
for
a
node
and
so
no
matter
how
you're,
actually
configuring
them.
Those
are
essentially
the
three
things
that
you
need,
and
this
is
true
for
all
nodes,
including
master
nodes,
so
for
a
core
OS
node.
What
this
might
look
like
in
a
very
simplified
way
is
like
the
other
cloud
config.
A
The
two
things
that
we
actually
need
to
set
up
are
a
Kubla
service
and
the
coop
config.
You
know
you
would
actually
want
to
add
the
flags
that
you
wanted
for
your
equivalent
there,
but
core
OS
itself,
ships
with
both
docker
and
rocket
as
part
of
part
of
core
OS
Linux.
So
that's
already
available
for
you,
and
this
becomes
your
generic
every
node,
config
you're,
just
throwing
compute
out
your
cluster
without
really
having
to
configure
that
much
on
the
host.
A
But
ultimately
this
isn't
just
a
core
OS
specific
thing
when
we're
abstracting
away
all
of
our
on
host
requirements.
What
that
means
is
that
for
any
distribution
we
get
these
same
benefits
because
we're
layering
it
on
top
of
kubernetes
and
the
process
again
just
becomes.
You
know
we
need
to
install
the
couplet.
We
need
to
install
it
container
run
time.
We
should
have
a
coop
config
and
then
we
start
the
couplet.
Now
this
is
compute
that
is
added
to
our
cluster.
A
So
I
also
said
this
should
be
all
nodes
and
not
just
master
nodes.
So
if
you
know
what
makes
a
master
node
is,
is
certain
components
are
running
on
that
that
host
and
that's
something
that
we
get
for
free
from
kubernetes,
because
we're
saying
well,
these
components
are
pieces
of
software
that
managing
inside
of
kubernetes.
So
we
can
give
you
breeze
the
information
that
needs
to
know
how
to
schedule
those
components.
So
we
could
do
something
as
simple
as
adding
a
particular
label
selector
to
something
that
we
deploy,
so
we
can
say,
master
equals
true.
A
We
could
add
this
via
to
control
just
labeling
a
particular
node,
or
we
could
precede
this
information
on
the
couplet
as
a
flag
so
that
when
the
node
registers
it
already
knows
that
it
should
have
this
label.
So
if
we
actually
go
back
to
this
slide,
we
can
see
here
for
the
API
server.
We
have
master
equals
true
as
the
node
selector,
and
we
see
we
only
have
one
copy
of
it
running
but
right
below
it.
There's
the
coop
proxy,
where
we
have
three
copies
running
and
no
node
selector.
A
So
this
is
a
three
node
cluster,
where
we've
told
the
API
server
that
we
only
want
it
running
on
a
particular
node,
and
we
might
want
this,
maybe
because
of
network
address
ability
or
at
CD
address
ability
or
something
of
that
sort,
but
rather
than
a
special
case
in
our
nodes
themselves.
We
add
this
into
the
scheduling,
mechanics
of
kubernetes,
because
there's
self-hosted
components
it
will
figure
out
where
it
should
run
for
us.
A
A
Kubernetes
is
really
really
good
at
managing
software.
Life
cycles
like
if
we
have
our
application
and
you
want
to
do
a
rolling
update
of
that
application
across
our
cluster.
This
is
something
that's
already
built
into
kubernetes.
You
know
you
have
a
deployment,
you
add
a
new
deployment
and
it
manages
the
rowing
update
in
between.
A
Take
advantage
of
all
of
these
strengths
that
it
has
about
managing
our
actual
application
life
cycles
and
then
have
it
manage
itself
as
well,
and
so,
when
I
say,
can
building
on
the
shoulders
of
kubernetes.
What
I
mean
is
that
like,
for
example,
upgrading
your
cluster
rather
than
building
this
complex,
doing
external
to
kubernetes
itself,
to
then
go
and
upgrade
the
components
and
know
things
about
rolling
these
updates
and
how
your
nodes
are
managed,
which
ones
are
masters
which
ones
are
not?
A
Ultimately,
we
should
be
able
to
get
to
a
point
where
these
are
like
the
four
commands
in
a
manual
upgrade
process.
We
say
coup
control,
apply
our
new
manifests
and
what's
happening.
Underneath
is
that
you
know
kubernetes
is
doing
these
rolling
updates
of
these
updated
deployments,
and
these
updated
objects
for
us
and
our
interface
to
it
is
just
the
kubernetes
api
and
I
have
a
little
note
down
here.
That
says:
we're
not
quite
to
this
point
of
running.
A
A
A
We
have
the
API
server
in
the
proxy
if
we
were
to
look
at
the
version,
so
my
local
coop
control
client
is
1,
4
or
5,
but
my
actual
server
is
1,
4
4,
and
so
that's
what
we're
going
to
be
upgrading
today
and
we're
gonna
upgrade
the
control
plane
using
nothing
but
coop
control,
because
there's
a
self-hosted
cluster
and
it's
actually
fairly
straightforward.
So.
A
Tell
me
if
this
gets
too
small.
Okay,
I
can
see
it
though
okay,
so
the
first
thing
that
we
want
to
do
is
well.
We
have
one
API
server,
one
scheduler
one
controller
manager
and
what
a
good
operator
would
do
is.
If
we're
going
to
do
a
rolling
update
of
this
cluster,
we
shouldn't
rely
on
Singleton's
of
these
components.
We
should
actually
have
a
highly
available
control
plane,
so,
while
we're
doing
the
upgrade,
it
actually
keeps
working.
So
the
first
thing
we
should
probably
do
is:
let's
label
a
node
as
a
master.
A
A
A
A
So
I'm
just
gonna
change
the
image
field
and
until
I
want
to
use
one
four
five
instead
of
144.
Now
that
note
that
I
had
before
that
said,
you
know
we're
not
quite
to
that
point
where
we
can
just
apply
these
new
manifests
and
have
it
have
it
upgraded.
This
is,
is
kind
of
what
I'm
talking
about
so
deployments
are
sorry
Damon
says:
don't
support
rolling
updates
natively
in
kubernetes,
yet
there's
actually
an
open
PR
about
adding
this
functionality.
So
this
is.
A
This
is
one
of
the
pieces
where
I'm
gonna
save
this
manifest
and
the
daemon
set
has
been
updated,
but
it's
not
going
to
roll
these
pods
for
us,
so
I
actually
have
to
go
through
and
now
I'm
going
to
tell
the
API
server
that
I
want
to
roll
the
pods
now
down
below
I
just
have
a
watch
command
running.
So
this
is
just
every
two
seconds.
A
That's
hitting
the
API
server
and
I'm
gonna
do
what
a
rolling
update
underneath
would
actually
do,
which
is
you
update
the
daemon
set,
and
then
you
kill
one
of
the
pods.
The
daemon
set
will
see.
Well,
I
actually
should
be
running
one
on
one
of
those
pods
on
that
node.
So
I'm
gonna
start
a
new
one
with
the
new,
with
the
new
manifest.
A
Okay,
so
now
I
happen
to
have
hit
the
API
server
that
I
just
terminated,
but
the
next
request
hit
the
other
API
server.
So
we
can
see
that
the
one
that
I
just
killed
is
now
is
now
terminating
there's,
actually
a
pretty
long
grace
period
on
that,
so
that'll
take
a
second
for
it
to
determinate,
but
I'll
get
ready
with
this
other
one.
So
we're
gonna
do
the
same
thing
on
this
other
API
server.
A
We
want
to
wait
for
this
one
to
come
back
because
we're
good
operators-
and
we
don't
want
to
take
the
whole
control
plan
down
at
once.
Alright,
so
we
have
that
that
new
API
server
we
can
see
by
the
the
suffix
here
that
the
new
pod
has
launched,
and
we
can
also
see
that
it's
only
nine
seconds
old.
So
let's
go
ahead
and
delete
the
other
one.
A
Sometimes
if
you,
if
you
delete
the
pod
that
you're
contacting
you,
get
that
error
in
the
file
so
we'll
see
see
the
API
server
now
also
terminate
well
line
up
the
next
command
cuz
again,
the
grace
periods
gonna
take
a
second,
but
this
is
actually
the
more
interesting
one.
So
when
you
update
deployments
it
actually
does
it
for
you,
and
so
it's
it's
super
fast
when
it
happens.
A
Want
that
to
be
running
four
or
five
all
right,
so
when
I
save
this
what's
gonna
happen
is
the
new
manifest
is
going
to
go
in
place
and
the
deployment
controller
in
kubernetes
is
going
to
see
that
now
it
needs
to
do
a
rolling
update
of
those
components
and
we
have
two
of
them
running
and
it
also
right
now
has
a
I
think
by
default
it
converts
to
one
extra
copy.
So
what
we'll
probably
see
is
like
four
copies
of
a
controller
manager.
We
it
are
it
already.
Did
it
like
it's
that
fast.
A
A
A
And
this
is
probably
gonna
happen,
pretty
quick,
again
yeah.
So
that's
already
done
so
we
just
we
created
an
H,
a
control
plane
from
a
single
node,
and
then
we
upgraded
the
control
plane
and
we
didn't
use
anything
except
for
control.
So-
and
this
is
just
me
as
a
person
doing
this
manually
like
this-
wouldn't
be
that
difficult
to
script
right.
Something
else
that
is
more
intelligent
about
this
of
greatest
single
component
have
some
kind
of
exit
criteria
of
saying
yeah.
That
component
is
successfully
upgraded.
I
can
move
on
to
the
next
one.
A
So,
coming
back
to
you
know
some
of
the
ways
that
we
simplify
the
lifecycle
management.
Why
we
want
to
do
this?
One
of
the
main
reasons
is
the
any
improvements
that
we
see
upstream
directly
translate
into
improvements
in
managing
kubernetes
itself,
so
Damon
said,
rolling
updates
are
great
for
our
applications
and
as
soon
as
that's
an
upstream,
we
can
use
that
for
kubernetes
and
I.
A
Don't
have
to
go
and
roll
those
API
server,
pods
anymore
and
then
also
another
big
point
is
that
you
know
your
expertise
in
managing
your
software
in
kubernetes
then
also
directly
translates
into
your
expertise
and
managing
kubernetes
itself,
rather
than
having
these
two
kind
of
concepts
in
your
team
of
well.
We
have
this
whole
set
of
tools
that
maybe
some
people
understand
about
how
to
manage
kubernetes
and
deploy
communities.
Then
we
have
people
that
are
familiar
with
how
to
manage
their
applications
within
kubernetes.
A
This
is
just
all
cross
over
and
it
all
makes
sense
in
the
same
exact
way.
The
way
that
you
would
introspect
and
debug
your
applications
is
the
way
that
you
would
introspect
and
debug
your
actual
kubernetes
installation
itself,
so
I
kind
of
want
to
go
over
a
little
bit
about
how
it
works,
and
so
some
some
pieces
I
want
to
touch
on.
Are
you
know
how
we
launch
a
self-hosted
cluster,
some
of
the
self
hosted
components
and
which
ones
are
self
hosted
and
which
ones
are
not,
and
disaster
recovery
to
your
questions.
A
Right
now,
in
this
cluster,
it's
not
self
hosted,
but
I'm
actually
gonna.
So
the
question
was:
how
do
you
run
a
CD
right
now,
so
in
that
cluster,
that
I
was
just
on?
It
is
not
so
fast,
but
I'll
come
back
to
that
and
we'll
talk
about
kind
of
next
steps
there
as
well,
so
in
terms
of
launching
a
cellphone
closer.
This
is
kind
of
an
interesting
thing,
because
you
need
it's
a
kind
of
a
chicken
and
egg
situation
where
you're
asking
the
control
plane.
A
That's
telling
you
that
you
need
to
be
running
a
control
plane,
but
if
there's
no
control
plane
there
to
tell
you
that
you
you're
not
running
anything,
and
so
what
we
needed
to
do
is
is
we
need
to
build
a
tool
that
can
act
as
this
temporary
control
plane
just
long
enough
that
a
self-hosted
control
plane
can
be
launched
and
then
that
initial
control
plane
can
disappear.
So
we
have
a
tool
called
boot
cube
it's
actually
a
community's
incubator
project,
but
essentially
it's
just
a
binary
that
starts
up.
A
Acts
like
a
control
plane
replaces
itself,
and
then
you
don't
need
it
anymore,
and
there
should
only
be
run
on
the
very
first
node
of
your
cluster
and
you
don't
need
it
after
that
point,
because
after
that
point
it's
just
kind
of
self
assembling
of
a
self-hosted
cluster.
Add
more
nodes
to
it,
have
work
scheduled
to
them,
but
fukube
doesn't
actually
come
into
it,
so
how
Bill
coop
works
so
similar
the
the
question
that
was
just
asked
in
this
case:
let's
consider
that
at
CD
is
network
addressable.
It
could
be
on
the
same
host.
A
It
could
be
a
remote
cluster,
it
could
be
of
any
size,
but
for
the
sake
of
keeping
this
relatively
simple,
let's
assume
that
CD
is,
is
network
addressable
somewhere
and
that
we
have
a
couplet
that's
available
on
a
node
that
we
can
start
up.
So
the
first
thing
we
do
is
we
start
boot.
Tube
and
buku
is
just
a
single
binary
that
contains
the
community's
control
plane
inside
of
it.
So
you
start
via
Kubb
runs
an
API
server
runs
a
scheduler
runs
a
controller
manager.
A
A
A
The
daemon
sets
the
secrets,
the
services
that
make
up
our
self-hosted
control
plan,
the
thing
that
we
want
to
replace
ourselves
with,
so
the
couplet
gets
scheduled
those
pods,
because
it's
registered
as
as
a
component
that
can
be
scheduled,
and
now
we
have
a
replacement
control
plane,
but
this
replacement
control
plane
isn't
really
doing
anything
right
now.
The
API
server
can't
bind
on
four
four
three,
because
it's
already
in
use.
A
So
it's
just
kind
of
sitting
there
at
this
point,
but
buku
knows
that
it's
it's
launched
a
replacement
and
it's
actually
checking
and
seeing
this
season.
Ok,
those
pods
have
started.
I
can
go
away
now,
so
we
killed
boot
cube
and
what
happens
here
is
that
from
the
coup
boots
perspective,
the
only
thing
that
changes
is
that
it's
seen
a
network
blip
on
it
can't
connect
to
443
anymore,
so
it
was
connecting
to
443
boot
coop
dies,
it
can't
anymore.
This
other
control
plane
starts
up,
and
so
the
API
server
can
now
bind
on
443.
A
All
the
components
are
running
together.
It's
still
referencing
the
same
data
store
oops
and
then
the
Kubik
can
now
connect
to
the
API
server
on
443.
So
for
the
mokuba's
perspective,
it's
just
seen
a
network
blip
and
we've
actually
pivoted
from
boot,
cube
to
a
self-hosted
cluster
and
as
far
as
our
data
is
concerned,
we're
actually
coordinating
coordinating
around
at
CD.
So
the
Kubla
sees
oh
I
should
be
running
these
three
pods
I
am
running
these
three
pods
all
as
well
with
the
world.
A
So
the
question
was
the
version
that
gets
launched.
The
self-hosted
cluster
does
that
need
to
be
the
same
version
as
we've
coop
itself
and
right
now
it
should
be
close,
and
the
reason
for
that
is
that
boot
cube
actually
compiles
in
most
of
the
communities
control
plan
like
you
saw,
and
so,
if
we're
ven
during
in
kubernetes-
and
it's
like,
you
know
one
for
something,
but
you
want
to
launch
a
one
three
cluster
that
version
skew
may
not
actually
be
viable.
A
So
one
of
the
ways
for
that
we
can
do
is
rather
than
comparing
those
components
into
buku.
We
instead
just
use
static,
manifests,
and
so
all
boot
coupe
would
be
doing
is
doing
a
shuffle
of
static
manifests
temporarily.
That
could
be
of
any
version
as
long
as
you
can
read
those
files.
So
that's
something
we've
talked
about
changing,
but
for
now
you
should
probably
launch
a
cluster.
That's
within
the
same
minor
version.
It
probably
wouldn't
be
anything
wrong.
If
you
didn't
but
yeah
and
then
do
you
have
a
question.
A
Right
yeah,
so
essentially
what
it
doesn't
the.
What
we're
doing
is
we.
We
need
a
temporary
starting
point
and
that
temporary
starting
point
could
be
binaries.
It
could
be
static
files,
it
could
be
an
all
compiled
in
binary
like
we
just
need
a
temporary
control
plane.
After
this
point,
we
can
throw
it
out,
so
it
doesn't
even
need
to
be
that
resilient.
It
just
needs
to
be
there
for
a
short
period
of
time.
Any
question.
A
So
the
question
was:
is
there
some
kind
of
like
upgrade
contract
like
how
the
ordering
of
versions
that
kind
of
thing-
and
there
is
so
there's?
Actually,
you
should
read
the
the
upstream
documents
on
this
because
there's
it's
even
between
there's
certain
supposedly
supported
version
skews,
but
it's
it's
kind
of
undefined
at
this
point
right
now.
Essentially,
you
want
your
control
plane
to
be
a
higher
or
equal
to
all
of
your
other
components.
So
you
don't
want
to
be
upgrading
your
couplets
before
you
upgrade
your
your
API
servers.
A
This
has
actually
bitten
me
in
the
past,
so
there
are
some
concerns
in
that,
and
so
when
I
went
through
and
I
said,
API
server
first
and
then
the
rest
of
the
control
green
components,
that's
kind
of
a
very
simple
process,
but
in
the
future,
what
kind
of
version
skew
is
actually
supported
in
terms
of
you
know,
maybe
you
have
you
know
if
I'm
jumping
from
some
minor
and
patch
version
up
to
a
new
minor
version.
Is
that
supported
how
many
of
those
and
I
think
right
now?
It's
saying
that
there
should
be
within
three.
A
A
Yeah
I
mean
so
the
same
kind
of
thing.
Where
we're
you
know
you
consider
these
operators
as
kind
of
smart
tools
that
know
how
to
manage
the
lifecycle
of
a
particular
component.
There's
no
reason
that
if
our
contract
is
the
kubernetes
api,
that
we
couldn't
build
an
operator
that
is
aware
of
kubernetes
and
these
kinds
of
restrictions
and
how
it
should
and
should
not
be
upgraded
or
you
know
the
order
that
didn't
need
so
current
or
like
the
exit
criteria
for
or
upgrade
to
the
scheduler.
A
Now,
how
do
we
actually
exercise
that
to
a
point
that
we
feel
good
about
that?
It's
not
just
running,
but
it's
actually
functioning
normally
so
yeah
I
definitely
think
that
there's
room
there,
so
the
couplet
so
I
took
out
this
slide
because
I'm
probably
gonna
be
pretty
pressed
to
time
on
this
one.
But
so
you
can
self
host
the
couplet
as
well,
but
ultimately,
even
if
your
self
hosting
the
cupola
there's
an
on
host
coup,
but
there's
a
turtle
at
the
bottom,
so
four
core
OS.
A
So
we
have
a
daemon
set
that
we
call
the
node
agent,
which
is
just
sitting
there
watching
annotations
on
the
node
object,
and
if
it
sees
that
we
say
desired,
version
of
the
the
couplet
is
a
new
version.
Then
it
will
pull.
It
will
doesn't
actually
do
any
of
this.
It
just
sets
the
environment
variable
on
the
service
file,
and
then
it
tells
system
D
to
restart
the
cupola.
And
what
ends
up
happening
is
that
service
restarts.
It
pulls
down
the
new
image.
It
starts
the
couplet.
A
So
even
that
it's
you
know
still
on
the
host.
It's
the
one
last
component
on
the
host
and
it's
not
truly
self
hosted.
We
are
still
pushing
those
updates
via
the
API
via
annotations,
on
the
node
object
and
the
new
agents
itself
is
actually
pretty
straightforward,
like
it's
set
an
environment
variable
and
then
restart
a
system
d-unit,
but
let's
say
you're,
not
on
core
OS
that
could
be.
You
know:
DNF
install
or
apt-get
install
a
new
version
of
the
Kubler.
That's
what
the
node
agent
does,
but
there's
other
interesting
things.
A
We
should
start
doing
there
as
well,
where
you
could.
You
know
you're
going
to
be
upgrading
the
kula,
so
it's
actually
drain
the
node
first,
because
maybe
this
has
changed
some
kind
of
interaction
with
how
it's
going
to
launch
and
manage
manage
the
pause
all
right.
So,
let's
see
one
of
the
things
I
wanted
to
go
over
is
like,
when
you
say
self
host
and
kubernetes,
there's
a
potentially
a
spectrum
of
what
that
actually
encompasses.
So
you
could
be
self
listening
to
google
it
you
may
not
be.
A
You
may
just
suppose
some
of
these
components
and
not
others.
So
when
I
talk
about
it
generally,
what
I'm
thinking
of
is
the
control
plane
and
the
proxy.
Some
of
these
other
pieces
are
a
little
bit
more
experimental.
So
if
you
were
to
self
host
the
couplet
there's
still
some
things
to
be
worked
out
there,
but
essentially
the
API
server.
We
run
as
a
daemon
set,
because
that
makes
Network
addressability
much
easier
because
it's
going
to
potentially
be
on
a
fixed
set
of
nodes.
A
This
could
be
a
deployment
in
some
cases
and
there's
some
some
upstream
work
around
improving
API
server
discovery
so
that
as
the
API
server
moves
around
your
cluster,
if
it
is
a
deployment-
and
it's
just
running
as
a
pod
anywhere
that
we
can
still
discover
where
it's
actually
living
and
then
the
controller
manager
and
scheduler
running
as
deployments,
this
works
really
well
pretty
straightforward
to
proxy
is
a
perfect
fit
for
the
daemon
set.
We
want
this
pretty
much
on
every
node
and
then
we
set
our
TLS
assets
and
service
account
information
as
secrets.
A
So
if
the
controller
manager
dies
on
a
particular
node
and
moves
to
another,
it's
not
up
to
us
to
be
like.
Well,
you
need
these
service
account
keys
to
also
be
on
that
host
as
well
for
it
to
function
properly.
As
that
pod
moves,
that
secret
is
going
to
be
available
to
it
as
well.
So
when
I
say
self-hosted
kubernetes
like
this
is
I
feel
like
completely
sane
and
stable
and
reasonable,
and
then
we
can
kind
span
from
here
into
into
new
components.
A
So
one
of
those
is
self
closing
the
pod
network
so
deploying
the
pod
networking
tools
actually
as
a
daemon
set.
This
has
been
kind
of
a
difficulty
in
a
lot
of
installations
because
it
means
you
need
to
get
when
you're
deploying
a
pod
networking
tool.
You
need
to
install
it
correctly
on
the
host.
You
need
to
interact
with
docker
correctly.
You
need
it
to
interact
with
kubernetes
correctly.
So
there's
a
lot
of
knobs
there
that
you
know
kind
of
can
make
this
a
difficult
thing.
A
So
what
we
can
do
now
actually
is
that
we
can
coordinate
around
the
container
network
interface
configuration
the
cni
configuration.
So
what
the
couplet
will
do
is
it's
looking
for
this
configuration
file
and
if
it
doesn't
exist,
then
it
thinks
and
you've
told
it
that
you're
using
CNI,
then
it
knows
I
can
only
start
pods
that
use
hosts
networking.
So
they
don't
need
to
know
about
the
pod
Network.
So
that's
fine
I
can
start
them
anything
that
needs
the
pod.
Network
I
can't
start
yet
because
I
don't
know
how
to
configure
it.
A
So
what
this
means
is
that,
rather
than
us
having
a
pre
configure
the
pod
network
on
every
one
of
our
hosts
and
complicating
our
on
host
configuration
that
we're
still
at
this
very
simple
on
hosts
configuration
and
then
we
will
tell
it
how
to
configure
the
pod
network
via
you
know,
native
kubernetes
objects,
and
so
what
happens
is
you
could
say
you
know
coop
control
create
the
daemon
set
that
can
be
tuned
to
your
pod
networking
tool.
That
tool
will
drop
the
cni
configuration
now.
The
couplet
we'll
see.
A
A
A
So
similar
picture
to
what
we
had
before
is
that
CD
now
we're
not
considering
as
this
external
component
that
we're
managing
and
it's
just
network
addressable,
we
are
adding
as
a
piece
of
boot
cube
as
well.
So
we
now
have
a
full
control
plane
inside
of
boot,
coop,
where
we
have
the
main
communities
components,
and
then
we
have
our
data
store
as
well,
and
so
the
couplet
is
contacting
that
and
when
boot
cube
chests
are
launched,
the
replacement,
self-hosted
control
plane.
We
have
an
extra
piece
here
that
we've
deployed
the
SED
operator.
A
So
what
this
we're
going
to
do
with
is
we're
going
to
say.
Okay,
let's
see
the
operator,
we
want
to
start
a
self-hosted
at
CD
cluster,
but
we
wanted
to
seed
with
this
previous
data
store.
We
want
it
to
actually
migrate
that
data
into
this
new
cluster,
and
so
when
we
told
it
to
seed
it.
What
it's
going
to
do
is
it's
gonna,
say:
okay.
Well,
let's
expand
this,
and
so
we
need
to
do
an
ad
member
operation,
we're
going
to
start
a
pod
and
we're
going
to
add
a
number.
A
It
knows
it
can
die
away
and
the
NCD
operator
needs
to
just
say:
remove
members,
so
we've
gone
from
a
single
Etsy
denote
our
seed
node
and
then
we've
expanded
it
to
a
two
node
cluster
and
then
we've
dropped
it
back
down
to
a
one
node
cluster
and
at
this
point
buku
can
die
and
then
now
we
have
a
fully
self
hosted
control
plane,
including
at
CD,
and
the
same
way
that
you
saw
before
where
I
said.
You
know,
coops
ETL,
scale
and
I
wanted
to
actually
expand
this
into
multiple
nodes.
A
We
can
do
the
same
thing
by
the
EDD
operator
so
now
that
we've
migrated
into
a
fully
self
hosted
control
plane.
We
tell
the
NCD
operator
well,
let's
make
this
a
three
node
cluster
now
and
it'll
actually
schedule
new
pods
and
then
add
those
as
members.
So
this
is
being
integrated.
Now,
it's
not
quite
there
yet.
But
this
is
our
life
structural
plan
at
a
high
level.
A
Okay.
So
this
is
all
awesome
stuff
and
it's
amazing,
but
we
should
talk
a
little
bit
about
disaster
recovery
in
in
some
failure
situations.
So
situations
I
want
to
go
over
node
failure
in
h8
appointments,
partial
loss
of
control,
plane,
components,
power,
cycling,
the
entire
control
plane
or
just
permanent
loss
of
the
control
plane.
So
in
the
case
of
node
failure
and
h8
appointments,
we
get
this
for
free
kubernetes
will
deal
with
this.
For
us,
you
know
and
no
dies.
It
had
control
thing
components
on
it.
They
will
be
scheduled
elsewhere.
A
You
know,
we've
we've
deployed
multiple
copies
of
them,
so
if
one
or
a
couple
of
copies
dies,
we
should
still
be
fine,
partial
loss
of
the
control
plan.
So
let's
say
we
lose
all
schedulers
or
all
controller
managers,
or
both
we're
essentially
in
a
situation
where
we
need
a
scheduler
to
schedule
the
scheduler
their
schedulers.
That
makes
any
sense,
but
this
actually
can
be
recovered
still
just
using
cubes
ETL
commands
and
the
recovery
method
is
scheduling
a
single
copy
pod
onto
a
particular
node
and
saying
on
this
node.
A
A
A
Okay,
so
I
mean
this
is
okay,
so
this
is.
This
is
enough
of
a
demo
like
if
we
have
more
time,
I
would
delete
the
controller
manager
too.
But
what
we
see
here
is
that
we
did.
We
did
the
scheduler,
but
the
controller
manager
is
actually
what
takes
a
deployment
and
then
creates
new
pods
from
it.
So
the
the
controller
mannequin
for
a
deployment
ways,
so
the
controller
manager
is
saying
well.
I
should
be
running
one
copy
of
this
and
we're
not
anymore
because
we
just
deleted
it.
A
A
A
A
A
Let's
call
it
cover
scheduler
and
what
we
need
to
do
is
we
need
to
give
it
a
particular
node
to
go
on.
So
we
want
to
bypass
the
scheduler
because
we
don't
have
one,
and
this
is
going
to
to
be
our
recovery.
So
let's
get
a
node,
let's
put
it
on
this
one
okay.
So
this
is
now
a
scheduled
pod
that
we
can
just
force
into
the
cluster
and.
A
They
reel
so
start
a
scheduler,
and
it's
like
okay,
these
things
need
to
be
running
somewhere,
let's
find
somewhere
for
them
to
run,
and
one
of
those
things
is
the
actual
real
scheduler
that
is
managed
by
a
higher-level
object.
So
we
no
longer
need
this
pod
anymore,
so,
let's
just
delete
it
and
so
we've
recovered
from
like
a
complete.
You
know,
partial
control,
plane,
failure
and
the
same
exact
thing
works
for
controller
managers.
Given
an
API
server
and
Etsy
D
to
interact
with
you
can
recover
these
states
using
coops
ETL.
A
A
And
so
this
is
back
to
that
chicken
and
egg
issue
where
in
some
cases
you
know
if
you
have
a
control
plane
to
talk
to,
even
if
you
like
power,
cycled
most
of
your
nodes,
if
you
still
had
an
API
server
to
talk
to
that's
enough
for
it
to
tell
you,
hey,
go
run
these
other
components,
but
if
we
have
no
API
server
to
talk
to
anymore
we're
innocent
we're
back
in
that
chicken
in
an
egg
state.
So
for
the
case
where
we're
not
actually
losing
any
data,
this
is
just
a
power
cycle.
A
A
So,
in
the
absence
of
an
API
server,
I
should
continue
running
whatever
I
was
running,
and
this
is
true
today,
just
normally
like
your
coolant
is
running
if
the
API
server
goes
down,
for
whatever
reason
it's
going
to
continue
running
what
it
was,
and
if
your
pods
have
a
restart
policy
of
always
and
those
pods
die,
it's
still
going
to
restart
them,
even
in
the
absence
of
an
API
server.
So
what
we
need
is
that
same
kind
of
recovery,
but
in
in
a
power
cycle
sense.
A
A
So
then
it
will
be
started
up
by
the
couplet,
so
I
have
to
reboot.
What
it
would
do
is,
it
would
say:
well,
I
was
running
a
API
server
and
I
have
this
cold
copy
of
it.
So
I
should
move
that
over
into
the
static
pod
location.
Kuba
starts
it
up.
Eventually,
it
sees
well.
The
actual
API
server
has
come
down
just
like
we
saw
with
that
recovery
pond.
We
can
now
move
that
back
out
into
cold
storage.
Ideally,
this
is
something
that
we
can
eventually
get
into
the
coolant
itself.
A
A
So
it
is
a
fair
amount
of
work
and
it
it
can
be
complex,
so
it's
just
kind
of
figuring
out
like
how
do
we
start
to
build
this
kind
of
functionality
and
it's
starting
that
discussion,
because
it's
really
useful.
We
can
get
around
it
with
the
user
space
stuff,
but
it's
a
little
bit
more
fragile
than
if
the
kulit
already
knows
its
state
and
it's
able
to
checkpoint
that
speed.
A
So
this
is
the
case
of
just
power,
cycling
and
not
actually
losing
any
data,
so
we
figured
to
go
over
a
permanent
loss
of
the
control
plane.
So
this
is
a
similar
situation
to
the
initial
node
bootstrap,
we're
back
in
a
place
where
you
know
we're
in
that
chicken
and
egg
scenario
it's
slightly
different
in
that
we
either
have
existing
MCD
state
or
we
have
an
MCD
backup
in
this
case.
A
Ideally,
you
are
backing
up
your
data
store
and
CD
is
not
magic,
it's
pretty
close
to
it,
but
you
should
still
be
doing
backups
so
again.
What
we
essentially
need
to
do
at
this
stage
is:
we
need
to
start
a
temporary
replacement,
API
server.
We
just
need
to
get
enough
so
that
we
can
contact
the
API
server
and
this
could
be
a
binary
static
pods.
We
could
create
a
new
tool
that
does
simple
recovery.
A
We
could
make
boot
Kubb
operate
in
slightly
different
form
where
it
just
starts
an
API
server,
but
once
we
have
at
CDE
and
an
API,
then
as
you've
seen,
we
can
recover
the
rest
of
the
state
just
using
coop
CTL.
So
there's
a
lot
of
options
here
and
even
in
the
case
of
like
everything,
goes
wrong.
That
stand
up
process
is
in,
isn't
actually
that
crazy,
so
I
just
want
to
quickly
touch
on
some
of
the
things
that
you
know.
A
There's
still
work
to
be
done
here,
and
some
of
the
high-level
things
are
things
like
the
daemon
set
rolling
update
and
there's
an
open
pull
request
about
adding
this
functionality.
Pod
checkpointing,
which
I
just
talked
about
or
something
so
posting
at
CD.
This
is
a
work
in
progress
right
now
and
then
I
think
that
you
know,
as
I
showed
I
I
can
recover
these
pods
using
cube,
CTL
and
copying
out
the
pod
spec.
A
But
that's
a
fragile
process
to
explain
to
someone
I
go,
get
the
deployment
and
take
this
one
section
of
the
spec
and
paste
it
into
a
pod.
So
we
could
build
simple,
tooling.
That
just
does
that
for
us,
you
know
we
could
just
have
a
command
that
says
recover
deployment
target
node
equals
in
it
asks
for
that
deployment.
It
extracts
the
the
pod
spec
and
it
just
assigns
it
to
a
node.
We
can
have.
You
know
simple.
A
You
know
complete
failure
case
tooling,
where
it
knows
how
to
take
a
net,
CD
backup
and
then
start
a
temporary
API
server
and
then
stand
up
the
rest
of
your
control
plane
after
that.
But
these
are
the
kinds
of
things
we
start
to
need
to
build
through.
So
that's
all
I
have
for
you
guys.
Thank
you
very
much.