►
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Operating kube-apiserver Without Hiccups - Stefan Schimanski & David Eads, Red Hat
Kube-apiserver is the central component of every Kubernetes cluster. Keeping it available at all times is crucial for reliable operation. At the same time we want to restart kube-apiserver, for version upgrades, key rotation or just configuration changes. How can this be done without dropping a single request on the floor? Outline: - overview and problem statement - graceful termination - health and readiness checks - rolling upgrades - reliable use of informers in client apps - self-hosting done right, without self-references. David and Stefan are both the builders and the consumers of core apiserver infrastructure used in kube-apiserver, maintaining most of the code behind those features.
https://sched.co/MPcV
A
Hello,
everybody,
my
name
is
Tom
I'm,
not
giving
this
talk,
I'm
just
introducing
them,
but
first
make
sure
you're
in
the
right
room,
because
there's
been
a
swap.
So
if
you
were
here
for
the
day
in
the
life
of
a
Cuban
eTI's
contributor,
that's
next
door,
but
the
slides
been
up
for
10
minutes
now,
so
hopefully
you've
got
that
so
I'm
here
to
introduce
David
and
Stefan
both
from
Red
Hat
I'm.
A
B
Hi
so
I'm
David-
and
this
is
Stefan
and
we're
talking
about
how
to
manage
a
cube,
API
server
and
one
of
the
ways
that
a
lot
of
people
have
tried
to
do.
It
is
by
running
the
platform
on
the
platform
we
are
here
today.
Let's
talk
about
how
we
started
our
experiences
from
there,
how
we
started
a
second
time
and
where
we
actually
ended
up.
So
we
started
out
with
these
goals.
B
We
want
to
be
able
to
make
config
changes,
we
want
to
be
able
to
do
upgrades
and
we
want
to
be
able
to
handle
certain
all
without
having
hiccups
on
the
cube
API.
So
we
looked
at
the
existing
technology
was
out
there.
Boot
cube,
existed,
it
looked
like
a
really
good
idea.
They
start
out
having
a
set
of
static
pods
that
they
use
for
bootstrapping
in
their
cluster.
So
you
have
a
static
pod,
which
is
a
pod
that
runs
on
the
cubelet
without
any
other
resources
being
present,
no
active
cube,
API
server
at
all.
B
It
runs
effectively
a
container
for
you
and
the
container
runs
whatever
process
you
want.
So
you
start
out
with
a
static
pod
for
a
cube,
API
server,
and
you
have
one
for
a
queue,
controller
manager
and
a
scheduler
and
from
there
you
create
real
workload
resources.
So
you
create
resources
for
the
QIO
server
as
a
daemon
set,
for
instance,
and
it
rolls
out
using
the
control
plane
that
you
have.
This
works
very
well,
it
starts
up,
it
runs
the
pods
are
roughly
equivalent
and
you
then
remove
your
static
pods.
B
So
you
started
out,
you
had
your
table
set
and
then
you
just
pull
the
tablecloth
out.
Everything
stands
up
and
keeps
running.
The
idea
is
that
you're
gonna
use
the
workload
resources
we
developed
in
Cube
to
roll
out
your
future
changes.
So
you're
gonna
make
a
change
to
a
daemon
set
name
and
set
rolls
out
onto
all
the
nodes
nodes,
recycle
and
you
have
upgraded,
and
this
this
works
pretty
well
until
you
crash,
but
cube
thought
of
that.
So
we
have
a
static,
pod
check,
pointer
right.
B
So
in
case
you
crash,
there's
gonna
be
something
for
you
to
start
back
up
with
now
static
pod
check
pointer.
How
does
it
work?
It
looks
at
the
config
maps
and
the
secrets
and
the
other
resources
you
have
for
your
daemon
set
pod
and
creates
mirrors
of
those
on
disk.
You
never
actually
run
them,
but
you
have
a
potty
animal
that
you
think
is
gonna
work
when
you
replay
it
right.
So
the
idea
is
that
everything
goes
down.
B
You
find
your
static
pod
yamo,
you
start
over
again:
reboot
strap
repin
it
so
one
day,
I
had
a
bad
day.
We
were
working
on
this
project.
I
had
my
three
nodes:
3q
controller
managers
they're
all
running
happily,
and
if
you
know
much
about
the
cube
controller
manager,
you
have
a
health
check
which
you
must
pass
or
the
qubit
will
kill
you,
and
then
you
have
a
lease.
The
lease
is
what
the
cue
controller
manager
does
to
make
sure.
Only
one
is
running
at
a
time,
and
it's
very
important.
B
Bota
means
is
that
you
actually
have
idle
processes
running.
So
you
have
three
nodes.
Only
one
is
doing
work,
two
are
just
idling
and
waiting,
but
they
are
reporting
healthy,
so
I
go
to
roll
out.
The
next
version.
I've
made
a
change
to
the
cue
controller
manager
and
I
start
rolling
it
out
with
my
damn
ascent,
and
things
are
going
well
right.
First,
odd
rolls
out
goes
in
the
node
says
that
it's
healthy
doesn't
acquire
the
lease.
One
of
the
old
ones
are
still
running
it.
B
Dayna
set
controls
are
still
running,
I
get
my
second
PI.
Second
pod
comes
up
fine
reports
healthy,
it
also
gets
checked,
wanted
it
doesn't
have
the
lease
third
one
updates,
and
now
my
very
first
one
that
are
rolled
out
acquires
the
lease
and
oh,
oh,
my
it
dies.
It
had
a
bug
where
now
doesn't
matter
what
it
was.
It
panicked
and
it
died,
and
at
this
point
the
next
lease
gets
acquired.
B
Don't
worry,
I've
got
this
and
oh
dear,
and
we
end
up
in
this
state,
so
I'm
in
this
state
and
I
say:
oh,
that
was
bad.
Well,
I'm,
just
gonna
roll
back,
so
I
roll
back,
my
demons
that
change
and
nothing
happens
and
being
a
bit
slow.
I
sat
there
and
I
stared
at
it
for
a
while.
This
is
weird
what
has
happened
and
it
turns
out
that
without
a
Damons
that
controller,
the
name
said
to
put
my
diamond
said,
backed
and
it
works.
So
well
so
I
go
okay,
don't
worry!
B
The
solution
to
my
problem
is
actually
to
have
static,
pods
static,
pods
that
are
always
running
as
static
pods,
and
this
lets
me
avoid
a
pivot.
I
never
have
to
figure
out
how
to
go
from
a
static
pod
to
a
daemon
set
back
to
a
static
pod
and
then
how
to
restore
it,
because
that
process
usually
works,
but
it's
not
foolproof,
like
I
just
found
out
then
using
static
pods.
You
have
your
tolerant
to
cube
failures.
You
don't
have
any
cyclical
dependencies
right.
B
So
the
main
point
here
is
that,
when
I
experience
this
problem
in
the
future,
I
don't
have
to
worry
about
my
daemon
set
being
up
in
order
to
fix
the
daemon
set
for
my
cube
controller
manager.
So
how
do
we
do
it?
We
have
an
operator
that
manages
static
pods
for
us
and
the
way
it
does
it
is.
It
has
this
idea
of
a
revision.
There
is
a
set
of
config
maps
and
secrets
and
they're
an
immutable
set.
So
what
happens?
Is
you
have
a
live
set?
You
observe
a
change.
B
You
copy
them
over
and
make
an
immutable
set,
and
we
call
that
a
revision,
and
then
we
have
a
cheat.
We
have
an
operator
that
runs
and
creates
an
installer
pod.
That's
how
we
actually
get
the
files
down
on
to
the
node
that
we
want
and
when
we
do,
that
we
create
this
installer
pod
and
we
force
it
directly
as
a
pod,
explicitly
labeled
for
a
given
master.
B
What
that
means
is
that
we
don't
depend
on
the
cube
controller
manager
and
we
don't
depend
on
the
cubes
scheduler,
so
once
the
installer
pod
is
there,
it
copies
the
secrets
and
config
maps
down
onto
disk
and
then
creates
a
pod
yamo
and
places
it
into
the
cubelets
manifest
directory.
So
the
static
pod
then
starts
to
run
before
the
operator
moves
on
to
another
node.
It
makes
sure
that
you're
healthy.
B
So
what
then
happens
for
the
deployment
of
doom?
What
would
happen
in
this
scenario?
Everything
would
still
get
updated.
All
those
pods
would
still
crash.
I
would
still
end
up
with
that.
Has
everything
crashing
but
because
I
have
this
installer
pod,
which
is
for
scheduled
and
because
I
have
an
operator
which
is
still
running
I
can
make
a
change
and
deploy
a
new
version
and
recover
my
cube
controller
manager
or
in
place.
B
This
is
much
easier
for
us
to
manage
over
all
the
other
property
that
you
get
from.
This
is
that
by
carefully
naming
your
static,
pod
resources
you're
actually
able
to
have
the
previous
revisions
that
works.
For
you
know,
5
6
10,
whatever
you
want
number
of
successful
revisions
located
on
the
cube
with
that.
You
have.
C
So
I
highlight
our
seam
here
without
hiccups,
so
what
does
it
mean
to
to
1.he
gaps
without
ever
was
for
the
user,
so
the
user
shouldn't
notice
that
we
hole
out
in
you
have
prohibition,
and
so
we
sum
that
up
and
clients
don't
get
timidly
dropped.
So
clients
keep
wanting
the
requests
getting
answers.
They
don't
notice
at
all
that
says,
a
warning
update
going
on,
for
example,
for
a
certain
age
in
Reverse
you
later
orphan
population
changes
or
upgrades,
make
sure
you
are
not
ready,
but
healthy
I
will
show
details
in
a
second.
C
C
They
don't
like
to
see
those
errors
and
when
they
as
a
protection,
refuse
they
complain
to
us.
What
I
doing,
why
is
the
PI
server?
Not
very
not
answering
my
requests,
so
the
first
one
is
certainly
something
we
have
to
to
fix
to
claim.
We
don't
have
hiccups,
the
second
one
go
away.
Everybody
might
have
seen
that
already
it's
an
HTTP
to
put
a
coil
message.
We
see,
but
it's
about
watches.
So
we
have
stream
watch
here
somewhere
and
the
lock
switch
is
a
good
hint.
It's
it's
about,
watches
and
watches
can
go
down.
C
We
have
code
for
there
to
cope
with
closed,
watch
this
and
restarts
them.
It's
not
nice,
but
it
happens
and
we
can
cope
with
it.
We
don't
have
to
investigate
anything.
The
second,
the
third
one
is
end
of
file.
So
there's
a
request
going
on
it's
not
really
answered
properly
and
in
the
middle
of
the
request.
Connection
goes
down,
something
we
don't
want
and
I
highlighted
here
already
a
term.
It's
called
long,
long
running
and
non
long
running
or
short
running
requests.
The
watch
is
by
definition,
long
running,
so
it's
infinite
if
everything
goes
well.
C
A
post
like
in
this
short
example
yeah
it's
written
post,
it's
short
running,
so
we
expect
an
answer
within
at
most
60
seconds
and
the
first
one.
It's
also
enter
fire.
But
again
it's
about
watch
us,
so
we
don't
care
they
go
down
when
an
ApS
over
we
terminate
at
the
end.
So
this
is
completely
normal,
of
course,
where
nobody
really
knows
know
which
will
host
it's
a
network
issue
probably
not
really
related
to
terminating
cube,
API
servers.
C
So
we
have
two
cases
we
have
to
investigate
here
and
that's
why
we
come
to
grace
for
termination
so
for
some
time
to
prepare
our
server.
We
have
this
concept,
so
we
can
gracefully
terminate
and
grace
for
determinate
means
yeah.
We
have
requests
going
on,
so
yes,
the
middle,
so
from
left
to
right.
At
some
point,
we
get
a
sick
term
from
couplet.
We
are
asked
to
terminate
and
there's
a
request.
What's
going
on
the
blue
line,
and
it
goes
up
to
the
end
to
end
the
termination
really
finishes.
C
It's
fine
I
should
we
bubble
here.
So
we
accept
that
all
right,
there's
a
list
so
I'm
very
early.
It
takes
ages
like
60
seconds
after
60
seconds.
It's
killed
anyway.
So
if
a
request
takes
so
long,
something
is
wrong.
It's
this
under
too
much
load.
We
accept
that
it's
also
file,
but
here
are
short
running
requests.
I
could
get
from
inside
Jocasta
this
green
one
and
the
second
one
here
it's
from
noble
answer.
I'm
there
answered
until
certain
points
when
you
get
ever
saw
those
red
bubbles.
C
The
critical
time
here
is
the
yellow
one.
So
we
receive
our
signal.
The
API
server
removes
itself
from
the
endpoints.
It
doesn't
have
to
wait
for
for
the
couplet
to
do
that
or,
for
example,
you
are
to
do
that
it
does
it
automatically
it's
a
that's
fine
and
it
stops
listening
and
stop
stopping
listening
means.
Every
request
which
comes
in
now
gets
a
connection
refused.
C
Not
so
good
because
Q
proxy
takes
time
to
not
Eve
to
realize
that
external
open
answers
need
time
to
realize
that
if
this
server
goes
down
so
in
this
timeframe
we
get
at
worst,
we
don't
want
them.
So
graceful
termination
is
not
enough.
Actually,
the
first
idea
is:
why
do
we
stop
listening
so
early?
Let's
move
that
far
to
this
place
here
far
after
Q
proxy
and
open
answers
noticed
yeah.
We
can
do
that.
We
cannot
do
it
now,
so
Q
API
server
doesn't
allow
it,
but
we
added
something
called
minimum
shutdown.
Duration.
C
That
was
a
quick
fix.
It's
not
much
yet,
but
it
helps
it.
Delays
termination,
so
repairs
have
a
nose
as
a
signal
I
have
to
shut
down,
but
it
keeps
running
and
answering
everything
was
for
30
seconds,
for
example,
so
it
takes
out
the
IP
from
the
endpoint,
so
the
few
parts
you
be
noticed
and
internal
cluster
clients
will
eventually
was
in
those
30
seconds
not
sent
traffic's
in
traffic
anymore,
external
appearances,
yeah.
C
You
have
to
tell
them
somehow
more
in
a
second
what
this
means
and
after
30
seconds,
no
traffic
comes
in
anymore,
so
you
can
shut
down
and
stop
the
the
connection
not
shut
down.
Actually
I'm
grateful
termination
means
that
we
still
possess
all
requests
which
are
in
the
stack
so
every
request
which
went
in
a
cheapy
server
in
in
bowling
still
possessed,
and
when
there's
no,
what
a
no
request
anymore
in
the
pipe.
We
really
terminate
that's
wasteful
chatter,
and
if
you
do
that,
thus
creating
a
face
here
becomes
great,
so
it's
not
critical
anymore.
C
We
stopped
listening
after
our
system
around
has
adapted
I'm.
Just
one
point
missing:
how
does
it
open
answer?
I
really
know
about
that.
First
idea
we
have
held
C's
or
the
health
check.
Endpoint
helps
means
couplet
will
fill
us
if
it's
everything,
so
we
cannot
really
send
anything
else
and
200.
Otherwise
we
are
killed
earlier.
So
we
need
something
else
and
there's
something
yeah
well
known
in
other
areas.
It's
a
second
endpoint
for
for
hairiness
read
easy
again.
We
don't
have
said
today.
We
might
want
it
because
otherwise
we
cannot
implement
something
like
that.
C
Develop
an
answer
should
listen
on,
read
easy,
and
this
goes
to
an
error
code
immediately,
but
API
server
is
healthy
and
keeps
answering
for
30
seconds,
but
the
local
answer
has
a
chance
to
notice
it
and
adapt,
and
then
we
can
do
brace
for
termination
and
that
way,
yeah.
We
realize
just
way
box
really
for
all
situations
yeah.
So
why
do
we
do
all
that
without
hiccup
hiccups?
The
same
conviction,
jizz
upgrades
it's
clear
for
everybody
and
we
want
to
talk
about
search
rotation
a
bit
more.
C
B
So,
sir
TRO
tation,
everyone
loves
cert
rotation
day.
I'm
sure
none
of
us
have
ever
had
an
issue
where
I
forgot
to
rotate
those
and
oops
it's
down
for
the
whole
company.
I
was
gonna,
go
ahead
and
start
this
section
by
looking
at
the
flags
that
were
available,
and
so
I
I
pulled
them
up
and
with
a
little
description.
They
they
don't
fit
so
well,
and
that
is
that
is
actually
the
smallest
font.
B
Google
slides
will
allow
you
to
use
and
they
still
don't
fit
so
so,
rather
than
starting
there,
we
will
talk
about
the
how
individual
chains
are
managed.
So
kubernetes
has
a
lot
of
chains,
lots
of
flags,
and
it's
not
always
obvious.
Why
we
have,
and
what
you
see
here
is
a
mutual
TLS
connection,
and
what
that
means
is
that
the
server
verifies.
The
client
and
the
client
verifies
the
server
and
I'm
gonna
very
briefly,
go
through
how
that
works.
B
So
a
client
has
a
cert
and
a
key,
and
he
uses
that
to
identify
himself
to
a
server
who
has
a
client
see
a
bundle
which
he
uses
to
verify
the
clients.
Identity
server
then
responds
back
with
a
serving
certificate,
certain
key
that
the
client
has
to
be
able
to
trust
using
a
server,
see
a
bundle
now
what
you'll
note
about
that
is
that
if
you
make
that
into
flags,
that's
a
bunch
of
flag.
B
So
you
have
to
get
all
these
flags
right
and,
as
you
rotate,
you
have
to
be
sure
you
get
them
updated
in
the
correct
order.
So
if
we
look
at
a
specific
example,
one
that
you're
going
to
have
to
manage
when
you're
trying
to
manage
your
control
plan
of
static,
pods
and
manage
the
cert
rotation
is
the
cube
controller
manager
speaking
to
the
queue
API
server
and
all
these
flags
exist.
B
You
have
the
cert
key
for
the
client,
you
have
a
CA
bundle
and
when
you
update,
you
always
have
to
remember
to
update
the
C
a
bundle
first
and
you
have
to
be
able
to
orchestrate
the
change.
So
you
have
to
handle
the
fact
that
all
the
cube,
API
servers
have
to
respect
the
new
client,
sir,
before
you
give
it
to
the
cube
controller
manager.
If
you
accidentally
get
that
wrong,
your
cue
controller
manager
gets
four.
Oh
ones
and
think
bad
things
happen
right
same
thing
happens
on
the
server
see
a
bundle
side
right.
B
So
now
that
we
know
all
that
in
research
that
we
have-
and
we
repeat
this
chain-
I,
don't
know
four
or
five
times
four
different
components.
You
also
have
to
understand
the
relationships.
So,
as
you
go
to
manage
a
relationship,
you
can't
just
change
a
flag
in
a
vacuum,
one
of
the
really
common
ones
that
you
want
to
turn
on.
B
Well,
it
it
starts
and
it
waits
and
then
it
crashes
because
it
can't
make
forward
progress
because
the
controller
manager
that
it
expected
to
sign
it
isn't
signing.
So
you
then
had
to
know
well.
I
have
to
set
this
flag
an
acute
controller
manager,
so
you
go
ahead
and
set
the
flag
and
a
cue
controller
manager.
You
get
a
signed
certificate
back
to
your
cubelet,
you
launch
your
cubelets
and
you
never
get
any
pots.
Not
a
single
pod
goes
to
that
cubelet.
You
look
in
the
logs.
B
B
So
we're
looking
at
all
these
flags
thirty-some-odd
flags
that
have
to
get
set
that
all
have
to
be
coordinated,
and
this
is
why
we
look
at
it
and
say
what
we're
gonna
have
an
operator
manage
this,
because
in
the
end
you
set
all
these
flags.
You
have
all
these
relationships.
You
had
to
make
sure
that
the
bundles
were
set
correctly
and
you
orchestrated
the
update
so
that
you
did
everything
in
the
correct
order
and
what
are
the
cluster
admin
care
about?
He
cared
about.
B
One
thing
he
wanted
to
be
able
to
bring
the
one,
sir,
that
he
paid
for
that
assigned
by
let's
encrypt
whoever
it
was
so
his
QAPI
server
could
actually
be
trusted.
That's
the
one
flag
he
wanted
to
set,
and
so
we
can
take
over
management
of
all
the
others,
and
it
makes
it
much
easier
and
safer
to
roll
out
changes
overall.
So
where
do
we
end
up?
B
We
managed
to
create
a
flow
that
has
a
cube,
API
server
or
any
other
process
managed
by
static
pods
roll
out.
There
changes
over
time
changes
to
the
configuration
and
to
the
upgrades
and
to
the
cert
rotation
all
happen
without
dropping
connections
on
the
floor
and
rolling
out
cleanly
on
a
self-hosted
environment.
A
B
So
trust
for
the
operator
when
the
operator
comes
up,
it
runs
as
a
regular
pod,
so
you
have
static
pods
to
launch
and
then
the
static
pod
is
equivalent
to
the
very
first
static
pod
you're
going
to
use.
But
once
you
have
that
we
have
a
signer
in
the
cluster
that
we
can
then
distribute
to
any
other
pod
with
the
service
account
see
a
bundle,
so
the
operator
trusts
based
on
that
and
then
any
additional
trust
the
cluster
admin
can
choose
to
configure,
but
they
usually
don't
have
I.
B
So
rotation
for
that
piece
happens
the
same
way
so
there's
actually
a
signer
located.
You
can
drop
a
new
signer
in
the
secret
and
then
when
the
new
signer
is
available,
it
will
immediately
get
added
to
the
sea.
A
bundle
see
a
bundle
then
ripple
out
across
all
the
pods.
We
then
sign
the
new
serving
certs
serving
cert
ripples
out.
Thank
you.
D
B
B
It
is,
this
is
an
operator
in
the
OpenShift
org.
It's
the
open
shift.
Cluster
queue,
cluster
cube
api
server
operator.
It's
a
mouthful.
We
have
them
for
each
of
the
control,
plane,
components.
You
can
look
at
them
there.
You
can
use
them
if
you
wish,
they
all
make
use
of
the
same
library
for
managing
static,
pods
right,
because
we
had
to
manage
multiple
different
static
pods.
We
have
a
little
library
to
develop
an
operator
that
can
do
that
and
handle
the
rollout
that
we
showed
at
the
beginning
right.