►
From YouTube: Kubernetes SIG API Machinery 20230803
Description
[lavalamp, deads2k, liggitt, munnerz] Discovery when apiservers are at mixed versions
- Description of current behavior and some problematic scenarios: https://docs.google.com/document/d/1wMst-2R7Zr0ADrJ_fr40zwkDanxwihHdL7RKSLK_D_s/edit?resourcekey=0-L-Yljgf99s70jmWmRiiHRw#
- Problems to solve (or verify solutions for)
ensure unhandled local APIService endpoints return 503s, not 404s
https://github.com/kubernetes/kubernetes/pull/104748 did this for unready apiservers
server response GC depends on for GET of individual items
type known
list with no objects == 200 with empty list
list inside missing namespace == 200 with empty list?
get missing object == 404
type unknown
list/get == 404 (404 on get is problematic for GC controller)
type in discovery but not known locally
list/get = 404, but maybe should be 503? will this break API clients that use 404 as signal to fall back to other API versions?
(servers don't have discovery-level type-level info for types not known locally today)
recording/informing existence (and info like verbs, namespacedness) of resource types so older servers can serve more complete/correct discovery
would benefit clients who would ignore some types of resources (like namespace controller wouldn't care about metrics server because it wasn't writable, or about cluster-scoped resources)
related to StorageVersion API, which has an entry per resource
related to aggregated discovery
improve namespace controller behavior when discovery permafails (avoid storms)
improve GC behavior when discovery permafails (avoid locking)
[ichekrygin] Question/guidance request about controller extensibility for core k8s types.
A
B
B
So
this
means
that
there's
types
in
etcd
corresponding
to
the
set
of
built-ins
that
the
most
recently
started,
API
server
knows
about
which
may
have
built-ins
that
other
API
servers
don't
know
about.
However,
all
API
servers
serve
the
serve
the
discovery
information
directly
out
of
this
data
in
SED,
which
means
that
if,
when
you
go
to
Discovery,
you
may
see
types
like
like,
no
matter
which
you
see
the
same
set
of
types,
no
matter
which
set
which
API
server
you
get
Discovery
from,
but
depending
on
which
API
server
you
actually
connect
to.
C
Question
on
that,
I
would
expect
that
a
server
that
is
informed
that
a
new
group
exists
would
show
the
new
version
for
that
group,
but
I,
don't
think
it
would
know
the
types
that
were
contained
in
that
new
group
version.
So,
for
instance,
when
we
introduce
our
Auto
scaling,
V2
I
wouldn't
expect
an
old
API
server.
To
be
able
to
tell
me
that
a
horizontal
pod,
Auto
scaler
was
in
that
group.
D
B
D
D
D
So
I
think
either
way
we
solve
this.
If
you
have
discovery
on
one
server
advertising,
something
that
it
doesn't
know
how
to
actually
serve,
you
run
into
one
set
of
problems.
If
you
have
local
servers,
avoid
publishing
Discovery
information
for
things
which
other
servers
are
actually
serving,
you
run
into
a
separate
set
of
problems
which
in
some
ways
are
maybe
worse,
it's
it's
hard
to
judge
so.
D
D
D
I
could
buy
a
503
from
qbaggregator
if
the
local
routing
to
something
an
API
service
exists
for
hits
a
dead
end.
I
could
buy
that
like
at
least
that
gives
Discovery
driven
controllers
like
a
hint
that
there
is
an
API
like
that,
tells
them
that
your
knowledge
is
incomplete,
like
some
controllers
react
badly
to
that
some
controllers
that
actually
puts
them
into
like
a
safe
mode.
D
Where
they're
saying
my
job
is
to
clean
up
and
if
I
don't
have
complete
information,
then
I
can't
be
sure
I
cleaned
up,
so
I'm
gonna,
like
I,
guess,
fail
safe
and
so
anyway,
that
the
current
behavior
is
the
top
left
corner
of
this
table,
which
is
old,
server,
publishes
existence
of
the
version
and
the.
But
if
a
controller
is
talking
to
that,
server
like
it
fails
to
get
to
complete
Discovery
and
hangs
sorted
okay.
D
B
Okay
I'll,
it's
all
propose
two
different
ways
that
we
could
possibly
solve
this
problem
and
the
first
one
is
we
could
make
it
work
the
way
I
thought
it
worked,
which
it
doesn't
work,
which
is
the
aggregator
reads.
The
set
of
bulletins
reads:
all
the
API
services
from
etcd
ignores
all
those
with
a
null
back
end
also
gets
the
list
from
its
local.
It
merges
this
with
the
list
that
it
gets
from
its
local
API
server
right,
we
modified
the
controller.
That's
currently
writing
these
things.
B
We
make
that
available
to
the
aggregator,
so
it
merges
those
two
things
and
uses
that
for
serving
Discovery
rather
than
directly
using
etcd.
Then
you
have
the
case.
Then
it
would
be
the
case
that
API
server
advertises
all
and
only
the
built-ins
that
it
can
serve
locally
so
that
that
is
probably
not
too
bad.
B
C
B
Yeah
yeah,
okay,
that
could
certainly
be
that
could
certainly
be
a
problem.
Okay,
so
my
the
second
idea
is,
we
extend.
This
is
going
to
rely
on
the
API
server
identity
effort.
We
extend
the
API
Services
field
so
right
now
it's
got
like.
If
the
back
end
is
null,
then
you
know
it's
built-in,
so
we
extend
it
and
add
an
a
list
of
alternate
alternates
and
an
API
server.
B
That
knows
how
to
serve
a
certain
built-in
puts
itself
in
this
list
of
alternates
right
and
a
API
server
that
is
attempting
to
server
resource
that
it
doesn't
host,
looks
at
the
list
of
alternates
and
proxies
the
request
over
to
the
API
server
that
can
serve.
It
I
think
it's
and
then
it's
more
complicated
because
we
have
to
look
at.
We
have
to
match
up
the
alternates
with
API
server
identity
and
remove
ones
that
don't
exist
anymore.
B
B
B
A
C
B
That's
another
yeah
that
ties
into
the
the
feature
thing.
That
would
be
easy
with
the
feature
thing
that
I'm
working
on
a
proposal
for
but
yeah,
that's
that's.
Another
solution
is
if
each
API
server
knew
what
other
API
servers
could
serve,
and
then
they
exposed
the
set
of
resources
that
all
API
servers
were
able
to
serve.
D
So
that
I
think
moves
you
to
the
bottom
right
of
that
table
and
I
was
I
was
trying
to
figure
out
like
what
the
implications
of
that
would
be
if,
during
an
upgrade,
the
new
API
versions
aren't
available.
Yet
because
they're
so
old
servers
in
the
cluster,
then
any
client
we
write
has
to
be
able
to
deal
with
absence
of
those
versions
and
I
guess
still
right
to
the
previous
version.
That
gets
really
awkward
around
like
the
beta
to
GA
transition,
where,
like
all
right,
it
actually
an
API
reached
GA.
D
A
D
Today,
like
the
way
the
system
works
today,
if
controllers
only
talk
to
their
local
API
server,
then
they
can
use
the
versions
released
in
their
release
and
they're
available
and
it
sort
of
works.
The
way
you
would
expect
the
cross
server
stuff
and
a
client
that
talks
through
a
load
balancer
to
arbitrary
servers
runs
into
problems,
but
a
lot
of
deployments.
D
A
D
C
Were
it
just,
it
sounds
like
the
the
deployments
that
do
that
need
to
be
sure
that
the
server
they're
contacting
in
addition
to
now
they
already
have
to
check
Readiness
and
then
have
to
have
a
way
to
kill
their
controllers
when
they
become
not
ready.
They
also
have
to
have
an
additional
check.
Are
you
serving
I
mean
that
doesn't
seem
entirely
unreasonable?
It's
it's
an
unusual
deployment.
C
D
Assuming
a
controller
talks
to
his
local
server
waits
for
it
to
be
ready
and
then
talks
to
it
it
if
that
server,
even
once
ready,
is
not
serving
new
apis
and
will
at
some
later
Point
dynamically.
That's
just
really
different
from
today's
behavior
and
seems
harder
to
write
a
controller
against
to
me.
A
B
B
D
B
C
B
B
C
A
C
D
B
A
C
C
B
C
D
I'm
talking
about
serving
like
you're,
not
required
clusters
are
not
required
to
serve
the
beta,
that's
true,
okay,
so
yeah
storage
version
is
going
to
stay
beta
because
you,
you
might
have
been
serving
it
previously.
I'm,
not
talking
about
the
internal
storage
version
stuff
I'm
talking
about.
Actually
what
is
our
Now
default
progression,
which
is
you
know,
new
API
shows
up
hits
Alpha
hits
beta
not
sort
of
by
default.
D
Hopefully
people
are
trying
it
giving
feedback,
but
if
they
don't
like,
then
it
goes
to
GA,
and
now
we've
we're
going
to
cut
a
release
where
the
API
went
to
GA.
What
version
do
we
make
the
controller
speak
today?
We
say
a
controller
can
assume
that
it's
talking
to
an
API
server,
at
least
as
new
as
it
that
is
able
to
serve
the
API
versions
that
that
controller
knows
about.
B
So
I
I
think
the
I
think
the
deal
is
if
you've
got
a
new
controller
that
only
understands
the
new
version
and
an
old
controller
that
only
understands
the
old
version.
The
new
controller
needs
to
wait
until
the
new
version
appears
and
until
then
it
can
be
either
uncontrolled
or
the
old
controller
can
control
it
depending
on
who
has
the
lock.
B
D
B
B
B
E
Maybe
to
clarify
something
you
mentioned
earlier,
so
the
use
case
when
the
controller
takes
talks
locally
to
API
server
in
the
context
of
upgrade
that
can
exist
between
two
controllers
and
different
versions.
Only
in
a
transient
period
of
time
right,
it's
at
some
point:
all
controllers
should
go
out
of
scope.
E
B
C
D
If
you
scroll
down
to
scenario,
two
there's
actually
a
worst
case,
which
already
exists
today,
where
we
add
a
new
resource
to
an
existing
group
version,
and
so
in
this
case
server
a
and
server
B
both
know
about
the
group
version
both
are
happy
serving
Discovery
for
it,
but
server
B.
The
new
server
like
has
a
new
resource
and
so
clients.
D
Talking
to
the
new
servers
like
oh
I've,
got
this
brand
new
resource
I'm
going
to
go,
create
some
of
those
namespace
controller
running
against
the
old
server
says:
I
need
to
clean
up
a
namespace.
Let
me
see:
what's
in
this
group
version,
okay
I've
got
my
list
of
old
resources,
I
successfully
clean
them
up,
namespace
deleted.
C
I
think
we
saw
that
back
when
we
were
using
localhost,
because
the
Ripple
start
would
always
prefer
the
old
Cube
controller
manager
for
cleanup,
which
always
got
the
old
list
yeah
and
and
we
faced
issues
I
did
link
I
did
provide
dates
for
when
those
meetings
were
August,
25
and
November.
Whenever,
if
you
want
to
go
back
I
think
in
the
end,
we
did
make
a
503
or
a
fall
through
case
yeah
until
you
were
ready.
D
B
B
C
A
A
C
B
B
C
B
D
D
D
The
the
Gap
that
I
talked
about
in
the
second
scenario
where
a
new
resource
shows
up,
even
if
we
want
a
server
to
be
able
to
proxy
resources,
requests
for
resources.
It
doesn't
have
to
other
servers
who
do
have
those
in
order
to
do
that.
It's
going
to
need
to
know
that
those
resources
exist
so
figuring
out
how
to
record
the
existence
of
those
resources
and
which
servers
know
about
them.
E
C
D
I
don't
know:
I
I
was
surprised.
The
name
service
controller,
like
was
spamming,
so
badly
I
I
would
have
expected
if
it
was,
if
it
had
100
namespaces
to
clean
up,
and
they
all
got
deleted
10
minutes
ago,
like
it
should
be
able
to
coalesce
those
into
saying
like
I
need
Discovery,
that's
at
least
successful
within
the
last
10
minutes,
like
it
shouldn't
be
repeating
all
of
that.
Every
time
so
I
think
there's
just
tactical
things.
We
can
do
to
improve
our
two
Discovery.
C
Controllers,
it
was
an
API
behavior
that
I
was
interested
in
trying
to
list
out
all
objects
of
a
certain
type
in
a
given
namespace
using
a
URL
path
of
apis,
something
something
you
know:
group
version,
namespace
Foo
slash
my
resource,
pretty
clear
that
with
404
and
probably
should
a
a
request
that
came
in
for
apis
group
version
namespace
and
then,
instead
of
namespace,
said
my
resource,
which
would
normally
list
all
of
them.
And
then
it
passes
a
query,
parameter
that
says:
I
want
them
confined
to
namespace
equals
Foo.
D
B
C
C
A
D
C
B
B
D
Let's
fill
out
in
the
notes
like
what
this
individual
things
we
want
to
verify
and
or
fix,
are,
and
then
we
can
start
like
putting
names
to
them
and
saying
who's
going
to
verify
this
or
file
an
issue
with
details
of
what
we
want
to
fix
and
the
bigger
and
more
complex
fixes,
like
you
know,
stuff,
with
API
implications,
record
existence
of
resources
and
proxy
between
API
servers.
We'll
need
keps,
I.
Think
some
of
the
earlier
ones
could
probably
just
be
issues
or
about
fixes.
B
D
B
Don't
know
yeah
I'm
saying
there
should
be
two
cases
there
to,
but
today
there's
only
one:
it's
like
API
server
doesn't
have
the
type
you
get
a
404,
but
it
should
if
the
type
is
present
in
Discovery,
but
not
in
the
local
API
server
that
should
503
right
like
like
I,
can't
serve
this,
but
some
other
API
server
should.
Could
we
shouldn't
give
a
404
for
that?
That's
that's
lying.
That
would
require
information.
C
B
All
right
on
this
list,
next
yeah
I
thought.
If
we
had
time
left,
we
would
do
stuff
from
the
backlog.
It
looks
like
we've
already
got,
one
from
the
backlog.
That's
been
moved
down.
Is
this
the
same
as
the
question
in
slack
a
few
days
ago
about?
Yes,
maybe
there's
a
proposal
is
like
put
a
like
a
field
or
annotation
or
something
in
on
objects,
so
the
object
can
select
which
controller
controls
it.
E
Exactly
so
yeah
I
think
that's
generally
broadly
to
describe
this
question.
Is
that
today
essentially
very
hard
to
extend
built-in
controllers
and
if
the
depends
on
the
object,
sometimes
it's
not
easy
to
even
reproduce
the
behavior
with
your
own
object.
Let's
say
you
create
your
own
type,
because
How
Deeply,
some
built-in
types
can
be
entrenched
across
multiple
surfaces
and
I
made
an
example
with
blood
disruption
budget.
B
E
But
I
think
that's
what
what
I
was
proposing
is
that
the
meatball
trying
to
entertain
the
value
in
it.
So
in
the
past
we
had
this
issue
where,
let's
say
we
could
have
multiple
Ingress
controllers,
which
were
driven
by
initial
annotation
and
then
by
the
Ingress
class
and
even
on
the
Pod
level.
We
do
have
an
analog
of
scheduler
name
where
you
can
actually
plug
in
different
scheduler,
which
will
schedule
the
spot
in
different
with
different
Behavior.
E
So
you
can
see
that
how
let's
say
I
could
imagine
that
if
I
would
Implement
my
own
pdb,
it
would
say
disruption
controller
which
does
not
change
the
behavior
happy
to
be
handled
by
eviction
API
or
scheduler.
But
it
changes
the
how
the
available
instructions
get
calculated.
Then
I
could
potentially
calculate
slightly
differently
for
pdbs,
which
I
care
about,
or
you
experimenting
with.
E
That
notion
can
be
applied
across
the
board
for
all
kubernetes
built-in
types,
slash
controllers
where
you
want
to
experiment
in
it
could
be
in
the
path
of
proposing
as
a
cap
to
update
existing
types.
What
could
be
purely
experimentations
and
hey
I
want
to
do
something
playing
it
and
just
maybe
provide
orthogonal
solution
today.
The
only
meaningful
way
to
do
it
with
the
schedule.
Specifically.
Oh
sorry,
with
the
disruption
control,
is
to
disable
the
controller
manager.
E
You
can
run
your
own,
which
again
not
very
comfortable
path
for
multiple
reasons,
because
again,
I
will
have
to
maintain
full
for
parity
between
functionality
of
the
Upstream.
This
disruption
controller
myself
my
controller
and
then
also
I,
cannot
do
that
in
the
environments
like
third
party
cloud
or
so
forth,
where
I
don't
have
access
to
control
plane.
So
I
was
thinking.
Oh
first
of
all,
what
is
the
guidance?
How
to
even
approach
that
is
it?
E
B
I
I
think
so
to
begin
with
I
I
think
this
is
the
wrong
Sig,
for
this,
like
oh,
like
you're,
actually
proposing
a
project,
wide
guidance
for
controllers
and
types,
so
Sig
architecture
is
a
place
to
go,
but
I
I
see
it
I,
see
this
as
being
pretty
disruptive
and
very
dangerous
for
like
existing
objects
and
making
sure
that
the
defaulting
works
correctly
so
that
they
there's
no
disruption.
If
you
add
something
like
this
in
I,
don't
have
any
problem
at
all
with
individual
controllers.
B
E
B
B
E
B
B
Actually,
the
first
time
I've
encountered
someone
who
wants
to
make
a
controller
Behavior
change
without
a
course
like,
like
usually
people
want
to
add
a
field
and
then
make
the
controller,
adapt
and
respect
the
field,
and
so
to
solve
that
problem.
We
there's
an
idea
floating
around
that
we
haven't
actually
seriously
proposed
anywhere,
which
is
to
make
overlay
type
like
a
crd
but
describing
fields
to
be
inserted
into
an
existing
built-in
schema
and
like
we're
going
to
get
to
that
approximately
never.
But
it's
it's
a
something
that
has
been
considered.
B
E
Agree
if
they
type,
if
there's
a
change
required
for
the
type,
then
it's
a
little
bit
different
ball
game,
but
I
can
also
see
other
use
cases
which
could
involve
node
cell
changing
type
but
change.
It
feels
like,
for
example,
I'm
in
spitballing.
If
I
would
like
to
introduce
different
taint
key
support
again:
I,
don't
change
in
schema,
but
I'm
changing
the
behavior
and
the
value.
So
I
would
like
to
maybe
play
with
alternative
controller
for
10
controller
yeah.
B
Own
challenges,
so
there
is
another
option
which
is
you
could
fight
with
the
with
the
default
controller
right
like
if
you,
if
you
want
to
do
something,
that's
not
incompatible
with
the
default
controller.
You
can.
You
can
just
go
ahead
and
modify
the
object
anyway,
and
some
control
like
if
it's
the
case,
that
the
built-in
controller
will
resist
your
change
and
write
it
back.
Then
you
can't
really
do
that,
but
there
may
be
cases
like
if,
like
adding
a
condition,
is
probably
fine.
B
B
If
you,
if
you
have
one
particular
controller
that
you
want
to
make
this
work
with,
then
I
would
go
talk
to
the
people
who
own
that
controller.
If,
if
you
feel
strongly
about
making
every
controller,
do
it
in
a
consistent
way,
which
has
you
know,
has
some
appeal
than
cigarchitecture
and
the
API
API
review?
Probably
subgroup
is
the
place.
I
would
go.
B
C
Like
saying
like
we're
sitting
there
and
stick
Arch,
what
would
I
say?
I
would
probably
say
something
like
how
about
we
try
to
demonstrate
that
this
works
with
one
thing
before
we
try
to
make
a
rule
about
how
this
should
work
for
all
things
and
eviction
or
pdbs
are
interesting
because
of
how
they
interface
with
the
eviction
API
endpoint
and
as
I
recall.
There
was
significant
logic
in
there
as
well.
E
There
is
in
terms
of
demonstrating
I
think
we
already
kind
of
have
examples
today,
as
I
mentioned
earlier.
Ingress
controllers,
like
were
the
cases
where
you
can
have
multiple
Ingress
controllers
brought
us
in
the
same
type,
requests
or
Ingress.
So
it's
kind
of
not
unprecedented
in
the
sense
with
the
pdb,
specifically
I,
think
it's
less
of
a
logic
of
world
change
in
type
it's
rather
than
Computing
the
status
value
of
it.
How
many
available
structures
you
can
provide,
or
you
can
tolerate
or
whatever
it
is.