►
Description
Kubernetes Day 2: Cluster Operations [I] - Brandon Philips, CoreOS
How do you keep a Kubernetes cluster running long term? Just like any other service, you need a combination of monitoring, alerting, backup, upgrade, and infrastructure management strategies to make it happen. This talk will walk through and demonstrate the best practices for each of these questions and show off the latest tooling that makes it possible. The takeaway will be lessons and considerations that will influence the way you operate your own Kubernetes clusters.
About Brandon Philips
Brandon Philips is helping to build modern Linux server infrastructure at CoreOS as CTO. Prior to CoreOS, he worked at Rackspace hacking on cloud monitoring and was a Linux kernel developer at SUSE. As a graduate of Oregon State's Open Source Lab he is passionate about open source technologies.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
So
what
we're
going
to
do
is
I'm
going
to
go
through
all
the
content,
but
what
I've
done
is
I've
put
everything
up
on
github
as
well,
so
for
some
of
them
I'm
going
to
have
to
skip
over
the
demo
portion
and
just
talk
about
things
in
abstract.
So
the
purpose
of
this
talk
around
kubernetes
want
to.
Oh
I
gave
a
talk
called
real-world
kubernetes,
which
went
through
how
to
manage
kubernetes
and
how
kubernetes
fails,
and
this
talk
is
kind
of
updated
for
2017
kubernetes.
We
now
have
actually
pretty
good
monitoring
systems.
A
If
you're
not
familiar
with
core
OS,
we
build
a
bunch
of
the
open
source
projects
in
this
ecosystem,
introduce
things
like
at
CD
and
container
linux,
and
then
we
have
products
for
large
organizations
and
enterprises
like
tectonic
and
clay.
If
you
want
to
learn
more,
you
can
find
out
at
core
Escom
and
so
in
this
presentation,
what
I'm
using
and
getting
a
little
bit
of
like
echo
or
feedback
on
stage,
what
what
I'm?
What
I'm,
using
to
set
up
these
clusters
is.
A
We
have
a
thing
called
the
tectonic
installer,
which
installs
kubernetes
clusters
in
a
very
particular
way.
This
can
all
be
replicated
with
upstream
kubernetes
because,
in
fact,
tectonic
is
simply
upstream
kubernetes,
but
just
as
you,
if
you
want
to
follow
along
at
home
later,
that's
that's
what
I'm
doing
so.
A
The
outline
of
the
architecture
or
the
discussion
is
I'm,
going
to
talk
through
cluster
architecture,
how
things
fail
and
how
the
design
for
cell
failure,
the
monitoring
and
alerting
once
you've
designed
out
the
cluster
architecture,
backup
and
disaster
recovery,
primarily
focusing
on
a
CD
how
customer
upgrades
are
approached,
particularly
with
how
we
install
kubernetes
with
Peconic
and
then
scaling
a
management
of
the
cluster
over
time.
This
is
going
to
be
quite
a
bit
and
on
a
couple
of
these
topics,
particularly
the
scaling
and
management
and
disaster
recovery.
Those
are
quite
in-depth.
A
Topics
solve
the
skipping
over
most
of
the
details
there,
so
across
your
architecture,
I
like
to
kind
of
describe
kubernetes
like
this,
you
could
of
course
manage
a
lot
of
machines
yourself
by
writing
like
a
while
loop
with
SSH
and
then
just
kind
of
doing
the
stuff
and
landing
the
containers
within
what
happens.
When
you
know
machines
fail,
there's
no
monitoring,
there's
no
state
recovery,
and
so
we
have.
A
That's
what
those
master
machines
are
doing
is
is
telling
everyone
what
to
do
cool.
So,
of
course,
as
we
all
know,
those
three
those
three
machines,
there
are
actually
the
kubernetes
control,
plane
or
better
known
as
master
nodes,
and
then
all
those
master
nodes
there's
a
number
of
control
plan
components.
This
includes
things
like
the
kubernetes,
scheduler,
API,
server,
controller
manager
at
CD,
etc.
A
So
a
better
way
that
I
would
argue
is
that
we
should
be
separating
out
the
infrastructure
from
kubernetes.
So
the
tools
like
the
tectonic
installer
just
lay
down
infrastructure,
and
then
we
have
tools
in
particular.
Kubernetes
api
is
themselves
to
manage
kubernetes,
which
means
that
we're
able
to
use
as
I'll
show
you
later.
The
kubernetes
api
is
to
say
upgrade.
A
The
kubernetes
api
is
totally
reasonable
and
I'll
tell
you
why
that's
totally
reasonable
later
and
the
advantage
to
those
dip
sauce
is
that
we
can
have
really
consistent
experiences
across
lots
of
different,
a
bunch
of
different
cloud
providers
and
bare-metal
includes
in
etc.
Ok.
So
if
you
want
to
try
installing
a
self-hosted
cluster
that
is
using
kubernetes
to
manage
kubernetes,
you
can
find
it
here.
A
This
installer
runs
on
all
these
different
platforms.
Ok,
all
the
demos
and
stuff
that
I'm
about
to
dive
into
are
at
this
github
URL,
my
name
spelled
with
1l
and
you
can
find
kubernetes
day
too.
Ok.
So
the
first
thing
that
we
want
to
talk
about
when
we're
thinking
about
day
2
operations
with
kubernetes
is,
of
course
failure
because
day
3
inevitably
is
going
to
bring
failure.
So
you
should
probably
start
planning
pretty
immediately
inside
of
failure.
There's
a
number
of
topics.
First,
we'll
walk
through
the
master,
node
failures
of
the
control
plane.
A
A
The
big
difference
here
is
that,
instead
of
that
application
say
rendering
a
blog
or
something
the
application,
which
is
the
API
server,
tells
computers
other
computers.
What
to
do,
there's
a
bunch
of
different
components:
I'll
talk
through
some
of
them
in
depth,
but
that's
the
rough
architecture,
primary
data
store
and
a
primary
API
server.
A
Okay,
so
the
first
demonstration
they'll
go
through
is
walking
you
through
a
cluster
that
is
self
hosted
where
we're
actually
using
kubernetes
to
manage
two
vanities,
so
I'm
serious
a
little
bit
of
latency.
So
hopefully
this
isn't
too
painful.
But
inside
of
this
cluster
there
is
a
number
of
components
that
may
look
familiar.
A
There's
a
qv-
deployment
for
managing
the
controller
manager,
physic
unity
deployments
for
managing
two
BNs,
there's
a
kubernetes
deployment
for
managing
the
kubernetes
scheduler,
and
so,
if
you
actually
click
in
here,
you
can
start
to
do
super
useful
things
like
Oh
figure
out
from
the
kubernetes
api.
Where
is
the
pod?
A
So
that's
the
big
idea
is
that
we're
managing
all
these
things,
whether
it's
the
daemon,
the
API
server,
which
runs
a
daemon
set
for
a
variety
of
reasons
or
the
API
or
the
scheduler,
or
the
networking
system,
which
runs
as
a
daemon
set,
etc.
All
these
things
are
managed
as
part
of
kubernetes
on
kubernetes.
Everyone
get
it
I
just
showed
hands
that
it
makes
sense,
what's
going
to
show
them
so
far:
okay,
cool,
okay!
A
So
that's
the
high-level
concept.
Now
the
the
one
little
like
fly
in
the
ointment
here
is
that
we
also
have
to
run
a
database
and
who
can
I
get
a
show
of
hands
and
whose
ran
databases
on
kubernetes,
okay,
uh-huh,
so
a
lot
less
confidence
there.
So
what
what
we've
built,
we'll
all
walk
through
at
CD,
so
FTD
is
a
little
bit
special.
A
A
lot
of
applications
that
are
really
easy
to
deploy
on
top
of
kubernetes
are
stateless
apps,
but
this
is
a
stateful
app,
which
means
that
all
the
unjust
states
is
unique
to
the
node
that
it's
running
on
sed
is
useful
and
the
reason
that
exterminators
uses
it
is
because
it's
clustered
and
fully
replicated.
You
can
think
of
it.
A
Sort
of
like
braid
for
database
in
that
every
right
into
ends
up
getting
written
out
to
a
bunch
of
different
replicas
at
the
same
time,
and
then
it
requires
of
50%
plus
one
availability
to
do
any
writes
into
the
datastore.
So
what
that
means
for
us,
those
people
who
have
to
operate
this
databases,
that
we
need
to
ensure
that
it's
bonded
gird
and
that
we
recover
if
any
of
the
machines
go
down
as
quickly
as
possible.
A
This
is
generally
why
the
recommendation
is
of
5
nodes,
because
you
can
have
one
I
tripped
over
the
power
table
moment
and
then
also
a
hardware
randomly
failed
moment
and
still
have
50%
plus
one
ok.
So
what
we've
done
is
we've
built
this
piece
of
technology
to
help
operate
at
to
be
on
top
of
kubernetes
called
the
sed
operator?
A
Thing
latency
is
killing
me,
so
you
see
here
that
inside
of
our
consider
our
namespace
of
cube
system,
we
have
a
cube
SPD,
which
has
a
number
of
events
and
then
a
cluster
size
and
a
particular
version.
And
so
what
does
that
do
the
operator
does?
Is
it
sits
there,
sort
of
like
the
controller
manager
for
other
components
in
kubernetes,
and
it
observes-
and
it
essentially
runs
this
reconciliation
loop
of
state?
Is
the
cluster
in
the
states
that
it
should
be
I'll,
observe
the
current
cluster
state,
I'll
analyze?
A
Well,
the
user
said
to
run
five,
but
only
four
running
and
then
act
it'll,
say
well
in
order
to
get
back
up
to
five
I
need
to
add
another
one.
This
whole
thing
is
open
source
and
is
really
slick
for
running
SED
on
top
of
kubernetes.
What
I'm
going
to
show
you
next
is
that
we're
actually
running
it,
both
on
top
of
and
underneath
kubernetes
and
I'll.
Show
you
why
that
space
as
well.
A
So
in
order
to
survive
a
failure
of
SPD,
you
need
to
do
a
few
things
you
need
to
NAT
CD
for
network
latency
activity
relies
on
being
able
to
know
that
a
given
round
trip
they
need
its
peers
happens
within
some
time
bound.
You
need
to
maintain
that
50%
plus
one
availability.
You
ensure
that
it's
persistent
disk
if
a
node
goes
away
and
comes
back,
the
same
data
should
be
there.
This
is
true
and
pretty
much
all
cloud
environments,
at
least
on
a
single
machine
and
that's
sufficient
for
STD's
for
disaster
recovery.
A
So
I
want
to
show
you
this
demo
that
we
that
I
did
last
night
the
reason
that
I
had
to
record
it
is
because
I'm
using
Amazon,
auto
scaling
groups
and
I'd
rather
not
spend
15
minutes
of
us
just
watching
a
spinner,
even
though
it
might
be
entertaining
the
last
day
of
the
conference.
Maybe
that's
all
the
brainpower
we
have,
but
so
what
I
have
here
is.
This
is
the
exact
same
cluster
that
is
running
here.
It's
a
like
an
8
member,
so
cluster
on
AWS
yeah.
A
I'll
roll
it
back.
Just
a
smidge
STD
operator
has
said:
please
run
another
copy
of
that
CD
I'm,
currently
out
of
spec,
but
kubernetes
has
a
constraint
that
says:
I
can't
run
FTD
on
the
same
machine
and
so
it'll
just
sit
here
unschedulable
until
the
new
machine
comes
up
and
then
once
the
new
machine
comes
up,
it
goes
into
readiness.
It
gets
scheduled
to
the
machine,
the
machine
downloads,
a
new
copy
of
@cv
software.
A
It
gets
replicated
the
data
from
the
existing
cluster
and
we're
back
into
a
healthy
cluster
state,
and
this
is
all
possible
through
two
reasons.
One
is
that
the
sed
operator
is
essentially
acting
and
in
acting
out
all
this
operational
knowledge
of
how
to
add
and
remove
machines,
members
from
the
f2p
cluster
and
it's
a
single
piece
of
software-
that's
running
on
top
of
kubernetes,
this
leader
elected,
so
we're
essentially
relying
on
its
ability
to
make
decisions
cool,
so
that
is
kind
of
the
vision
that
we
have
for
the
near
future.
Here.
A
A
A
So
when
we
recover
a
node,
the
way
SPD
works
is
that
when
the
nodes
come
up,
they
essentially
tell
everyone
what
what
the
latest
version
of
the
data
store
they
have
and
then
one
of
the
machines,
not
the
leader,
will
catch
the
other
node
up.
It
works
very
much
like
raid,
essentially
great
question.
Yes,.
B
B
A
Yeah,
so
the
reason
that
we
marked
the
question
was:
is
there
any
reason
to
mark
the
nodes
as
master?
So
we
have
this
label
that
we
put
all
of
those
that
says
master
equals
true.
The
reason
that
we
do,
that
is
because
we
want
particular
workloads
to
land
on
the
master
nodes,
and
it
allows
us
to
do
like
things
like
anti
affinity
calculations.
A
Correct,
yes,
there
are
ways
of
essentially
ensuring
certain
workloads
land.
This
is
mostly
about
operate,
Li
operationalizing
it
knowing
which
nodes
the
monitor
having
them
of
different
size,
etc.
Cool.
So
that's
that's
node!
Failure
in
discovery,
there's
a
couple
other
demos
that
are
recorded
one
is
about
scaling
it
up.
So
this
cluster
started
off
as
a
single
member
cluster
and
then
I
scaled
everything
up
to
a
five
member
cluster
and
the
SDV
operator
helped
me
scale
it
up.
You
can
also
scale
it
back
down,
etc.
A
Okay,
so
the
next
piece
of
thing
that
you
need
to
worry
about
is
the
API
server.
This
one's
a
lot
easier,
it's
stateless!
It
has
to
be
back
at
horizonte
scalable.
You
can
just
add
more
to
get
more
reach
capacity.
It
does
require
a
load
balancer,
and
this
is
for
a
few
reasons.
One
is
that
as
a
command-line
tool
or
if
you're
interacting
with
a
graphical
console
you
you
need
a
single
point
to
interact
with
it.
The
other
reason
is
it's.
A
A
You
want
to
run
any
API
servers
generally
to
match
your
sed
count,
because
a
CI
server
is
essentially
relying
on
a
2d
as
its
primary
data
store
and
if
you
think
of
the
API
server
is
like
a
recap
in
trying
to
in
front
of
that
CD
you
kind
of
want
to
have
a
one-to-one
mapping.
You
also
want
to
match
failure
domains.
A
Generally.
All
these
systems
are
designed
for
single
machine
failures
to
be
okay,
and
so
you
want
STD
and
HIV
over
the
wrestling
B,
because
there,
because
STD
is
useless
without
the
API
server
in
the
context
of
a
kubernetes
and
the
API
server
is
useless.
Without
that
CD.
There
really
the
require
each
other
and
then
monitor
and
maintain
load
balancers
over
time.
So
the
cool
thing
that
we
can
do
with
the
cell
phone
stuff
is,
we
can
actually
just
come
in
and
sale.
Individual
API
server
notes,
so
I
can
come
and
then
just
leave.
A
This
pause,
it'll
end
up
getting
terminated
and
then
kubernetes
will
recover
from
it.
Now.
One
of
the
questions
that
I
often
get
asked
about
this
is
this
is
really
recursive.
I'm
a
system
administrator
I
want
everything
to
be
stable,
as
in
this
really
unstable
the
way
the
whole
thing
is
designed
is
that
just
like
any
other
system
that
is
executing
processes
under
Linux,
we
persist
things
to
disk.
A
It's
no,
no
more
or
less
different
than
you
know.
Having
like
an
init
Rama
fest
that
has
system
D
services
then
are
launched
later.
That
makes
sense,
okay
cool.
So
what
are
the
risks
like
one
of
the
risks?
If
you
don't
maintain
the
Khachaturian
sed?
If
your
ad
CD
cluster
goes
below
a
certain
threshold
below
50%
plus
one,
it
will
go
read-only,
so
you
won't
be
able
to
do
any
post,
which
means
that
your
cluster
is
fine.
Kubernetes
clusters
will
continue
to
run
their
workloads
if
they're
unable
to
talk
to
the
control
plane.
A
Well,
you
will
lose.
Is
cluster
visibility
because
a
lot
of
things
like
events
and
stuff
get
pushed
through
at
CD
and
you
may
lose
the
ability
to
find
and
do
service
discovery.
However,
DNS
should
be
cached.
So
as
long
as
your
DNS
instances,
your
coop
dns
instances
remain
running.
You
aren't
at
risk
of
the
app
starting
to
fail
for
enter
clusters.
Service
discovery,
okay,
so
the
next
bit
are
the
scheduler
and
the
controller
manager,
so
the
scheduler
and
troller
manager
is
stateful,
but
they
back
their
state
with
the
API
server.
So
to
a
large
degree.
A
What
happens
is
that
if
you
lose
a
scheduler
controller
manager,
they
will
take
a
few
seconds
to
come
back
up,
so
they
need
to
rebuild
internal
state,
and
then
they
rebuild
that
internal
state
from
the
API,
their
leader
elected.
So
it's
totally
safe
to
have
multiple
copies
of
these,
but
they're,
not
horizontally
scalable,
adding
more
schedulers
or
more
controller
managers
does
not
make
the
scheduling
faster.
I
wish
that's
the
way
that
worked,
but
that's
just
simply
not
the
case
and
it's
totally
possible
to
recover
from
the
states.
A
So
we
think
back
to
API
server
and
sed
I
said
that
we
persist
these
things
to
disk.
We
don't
persist
any
other
other
kubernetes
components
to
disk,
and
but
it's
okay,
because
the
scheduler
and
controller
manager,
if
you
screw
up
an
update
or
you
accidentally
scale
your
scheduler
down
to
zero,
you
can
you
can
recover
from
that
and
I'll
show
you
how
so
you
can
run
like
I
say
you
run
two
more
copies,
but
their
lead
elected.
You
need
to
have
anti
affinity
to
ensure
that
you
survive
any
single
machine
failure.
A
So
imagine
that
you
have
the
scheduler.
You
have
two
copies
for
they're
running
on
the
same
post:
that's
not
actually
going
to
help
you
when
that
host
goes
down
okay.
So
this
is
a
pretty
quick
demo.
So
I'll
show
you
this.
What
we'll
do
is
both
scale,
the
tube
scheduler
down
to
zero,
which
means
that
essentially
we're
simulating
a
outage
where
we
lost
all
the
machines
where
the
scheduler
was
running
on
and
now
no
new
forward
progress
can
happen
against
kubernetes,
okay,
so
the
kubernetes
deployment
for
the
scheduler
has
been
scaled
down.
A
Now
we
have
a
problem:
how
do
we?
How
do
we
schedule
our
new
scheduler?
Anyone
have
an
idea
about.
We
might
do
that.
So
what
what
yeah?
Exactly
so
you?
What
you
do
is
you
give
the
deployment
of
the
scheduler
so
and
then
you
act
like
you,
scheduler
yourself,
I
think
I
can
figure
out
the
name
of
a
node
to
place
the
scheduler
on
I'm,
a
pretty
smart
guy.
I
can
choose
randomly
from
a
list
of
six
options.
So
I'll
show
you
how
that
works.
So
you
take
the
scheduler.
A
You
get
this
appointment
object,
so
the
API
server
is
still
running
so
I
go
in
and
I
look
at
the
scheduler
object.
The
first
thing
I
have
to
do
is
have
to
change
the
kind
from
deployment
pod.
I
can
delete
most
of
this
stuff.
I'll
keep
the
name
space
and
the
name,
and
then
the
spec
mostly
deals
with
the
deployment
stuff.
But
inside
of
here
is
the
pod
template.
A
So
if
I
come
in
here
and
just
delete,
essentially
everything
else,
that's
not
the
pod
template
I
move
this
around
and
now
I
have
pod
a
single
pod,
and
the
last
thing
that
I
need
to
do
is
just
choose
a
node.
So
if
I
go
in
and
put
a
node
name
and
I
may
get
the
name
of
my
nodes
choose
one
about
true,
so
I
need
to
find
a
node.
A
The
other
thing
to
keep
in
mind
about
the
scheduler
and
the
controller
manager
is
that
there's
failure
risk
the
controller
manager.
If
it
goes
down,
you
will
lose
cloud
provider
integrations.
You
won't
be
able
to
spin
up
load
balancers
or
attach
persistent
disks.
If
the
controller
manager
and
the
scheduler
go
down,
you
won't
be
able
to
do
deployments.
You
won't
be
a
little
scale,
as
I
just
demonstrated
and
then
failed.
Node
workloads
won't
reschedule.
A
So
if
that
auto
scaling
group
like
I'd
shown,
if
I
had
nukes
that
machine,
when
the
new
machine
comes
up,
it
won't
actually
get
scheduled
new
workloads
and
the
old
workloads
won't
get
unscheduled.
So
that's
your
risk,
and
generally,
that
means
that
you
can
have
like
anywhere
from
five
minutes
to
hours,
to
respond
to
these
components
inside
of
kubernetes
going
down.
The
last
bit
is
DNS
I
like
to
tie
two
really
well
I
used
to
work
on
DNS
early
in
my
career.
It's
not
be
enough.
There's
no
way
of
CNS.
It
was
DNS.
I.
A
Think
we've
all
experienced
this
for
ourselves,
so
kubernetes
dns
is
a
little
tricky
pretty
much.
Everything
else
in
kubernetes
can
use
service
discovery,
but
the
reason
this
I
is
so
applicable
is
because
everything
relies
on
service
discovery,
and
so
the
kubernetes
dns
service
lives
at
a
predefined
service
IP,
and
so
you
have
to
run
this
service
at
that
IP
all
the
time,
but
the
risk
is
really
low
of
the
beam
odds
going
down.
It's
used
by
many
applications
through
Venus,
but
it's
not
used
by
cube
itself,
so
it
goes
down.
A
Your
applications
may
stop
responding
that
you
still
start
to
recover
your
cluster
and
reschedule
a
new
DNS
server.
All
right.
The
last
bit
is
around
nodes,
the
places
where
you're,
the
actual
user
applications
work.
Labels
are
really
flexible.
So
with
labels,
we're
able
to
say
you
know,
make
selections
in
a
variety
of
ways,
maybe
front
end
and
back
end
or
based
on
user
or
based
on
environment.
A
These
same
labels
can
be
used
against
nodes,
and
this
is
really
useful
where
we
can
start
to
say
this.
Node
lives
in
this
particular
rack
or
there's
no
list
in
this
particular
region,
or
this
node
lives
on
this
particular
switch.
The
way
that
I've
been
demonstrating
all
this
stuff
with
MCD
in
the
API
server
etc.
A
So
this
is
a
feature
called
failure,
domains
which
is
in
beta
and
it
uses
labels
like
the
rest
of
the
system
to
decide
where
to
land
mushiya
land
workloads.
So
again
you
can
start
to
break
up
your
system
based
on
node
labeling.
So
a
quick
demo
that
I
wanted
to
give
was
cuckoo
kettle.
Has
this
really
great
thing
called
JSON
path?
So
it
allows
you
to
query
through
the
cluster,
so
you
can
get
nodes.
You
sleep
at
night.
There
we
go.
A
So
what
we'll
do
is
we'll
look
at
this
thing
called
the
failure
domain
region
which
is
automatically
populated
by
the
cloud
provider
integration.
So
in
this
case
it
lists
all
the
AWS
regions
and
then
there's
also
failure
domain
zone
which
lists
the
actual
able
availability
zone.
So
you
can
start
the
partition,
and
this
is
how,
like
the
FTD
operator,
partitions,
stop
by
default
cool
and
then
it's
all
just
wrapped
up
in
this
node
selector
and
then
a
topology
key.
A
So
in
the
case
of
FTD,
we
say
ensure
that
things
that
are
labeled
F,
2d
cluster
equals
cube.
Ftd
don't
end
up
in
the
same
topology
like
each
node
doesn't
end
up
in
the
same
topology
where
the
hostname
matches,
because
we
don't
want
them
to
land
on
the
same
machine
ever
all
right.
So
I
have
five
minutes
left
and
five
sections.
This
is
going
to
go,
super
well
monitoring
and
alerting.
A
We
of
course
have
built
this
thing
called
a
Prometheus
operator
which
is
also
open
source.
You
find
it
a
Korolev,
slash
Prometheus
operator
on
github
one
of
the
things
that
a
kind
of
a
lasting
I'll
demo
here
is
the
thing
called
cou
Prometheus,
which
folks
from
SoundCloud
have
built
a
bunch
of
rules
that
allow
you
to.
A
A
So
we
can
do
things
like
look
at
the
node
status
until
you
like
a
bunch
of
nodes,
are
down
and
give
you
an
alert
to
example.
This
query
will
say:
if
there's
more
than
10%
of
my
notes
that
are
in
non
node
ready,
send
me
an
alert.
I
could
go
to
page
or
duty
or
email
or
whatever,
because
and
the
other
piece
that
it
has
is
it'll
start
to
scrape
metrics
CPU
network
storage
metrics
from
individual
nodes,
so
that
you
can
start
to
alert
on
a
ram
usage
is
at
80
percent
coal
or
whatever.
A
So
this
is
an
example
of
a
query
that
is
provided
in
queue.
Previous
to
say,
give
me
the
use
percentage
of
kubernetes
memory
versus
currently
consumed
memory
in
our
pods,
and
you
can
also
start
to
drill
down
and
say
give
me
the
container
memory.
That's
used
by
individual
applications
like
node
exporter.
A
So
the
most
interesting
thing,
I
think,
is:
how
do
you
monitor
the
control
plane
so
in
this
repo
there's,
there's
rules
around
ensuring
that
FTD
is
remaining
up
with
that
50%
plus
one
checks
for
high
turnover,
your
election,
perhaps
you're
having
a
network
problem
and
then
also
API
response
times.
The
key
metrics
that
you
need
to
look
for
for
thinking
about
the
API
server
is
essentially
latency
and
the
sed
latency
between
the
API
server
and
also
scheduling
latency
our
notes.
A
A
Alright,
so
disaster
recovery,
you
can
read
the
reading
on
the
notes
that
I
have
on
github
cluster
upgrades.
This
is
where
really
where
self-hosted
shines.
So
there's
two
vulnerabilities
that
happen
in
october,
20th
2016,
there's
dirty
cow,
who
is
affected
by
dirty
cow,
everyone
I
hope.
Probably
everyone's
phone
is
still
unpatched
and
then
there's
also
kubernetes
clients
or
validation
problem,
which
I
gave
a
name
it
didn't
catch
on
called
mask
capitalized.
A
That's
currently
at
one
five,
four
and
bump
it
up
to
one
five
size
and
then
watch
and
use
the
rolling
upgrade
capabilities
of
kubernetes
to
handle
that
operational
concern
all
right
last
fit
is
a
management
of
nodes
not
going
to
cover
it,
but
essentially
you
can
use
with
all
the
technology,
auto
scaling
groups
or
terraform
to
kind
of
stateless.
We
manage
the
nodes
and
then
rely
on
kubernetes
to
manage
the
state
of
the
kubernetes
application.
A
All
right
so
try
to
get
one
more
demo
in,
but
if
you
want
to
try
this
stuff
out
again,
go
to
github.com,
slash
tulips,
slash,
kubernetes,
day
2
and
try
out
some
hose
demos.
If
you
want
to
get
a
cluster
running
check
out
the
tectonic
installer
and
kind
of
play
with
everything
that
showing
up
today,
I
want
to
thank
you,
I
want
to
try
to
get
renewed,
applaud
thing
and
I'll
try
to
get
one
more
demo.
Okay,
thanks.
A
A
Yes,
it
you're
actually
in
disaster
recovery
mode,
and
you
have
to
restore
from
backup
the
backup
will
be
really
up-to-date,
it'll
be
because
everything's
been
read-only
for
such
a
long
time,
but
you
have
to
actually
go
through
and
take
manual
steps,
because
only
a
human
being
can
make
decisions
at
that
point,
it's
sort
of
like
the
scheduler
problem.
Once
you
go
below
quorum,
the
computer
can't
make
a
reasonable
decision,
so
human
has
to
step
in,
but
there's
a
disaster
recovery
guide
on
FCB
for
handling
that
yeah.