►
From YouTube: Kubernetes SIG API Machinery 20230906
Description
- [logicalhan, jpbetz, liggitt] [external, public] Safer Kubernetes Upgrades
- [nilekh/mo] KEP to move SVM in-tree
https://hackmd.io/@azure-container-upstream/H14Q8R2T3
- [benluddy] Binary Data Format Questions
B
A
Api
Machinery
by
weekly
meeting
today
is
September
6th
2023
we
are
in
September.
Can
you
believe
that
it's
amazing
how
the
time
flies-
and
we
have
a
very
nice
agenda
today?
Thank
you
for
the
people
that
has
been
contributing
to
that.
So
without
any
more
introductions,
let's
start
with
Hanjo
and
Jordan.
If
he
joins.
C
C
C
We
can
upgrade
a
component
so
that
its
binary
version
is
like,
let's
say,
let's
say
our
cluster's
on
1.28.
We
upgrade
a
component
to
1.29.
We
want
to
be
able
to
start
that
component
in
a
mode
that
is
compatible
to
1.28,
and
this
means,
like
all
apis,
like
all
resources
that
are
being
served
all
feature.
C
Flags
that
are
enabled
would
be
consistent
with
the
set
that
was
enabled
for
1.28
and
the
reason
why
we
would
want
to
do
this
is
we
want
to
make
the
steps
for
upgrading
more
granular
right,
because
when
we
fail
during
an
upgrade
depending
on
when
we
fail,
we
have
more
information
about
like
what
around
actually
failed
right.
If
we
have
smaller
steps,
there's
also
less
to
rewind
if
we
need
to
roll
back.
C
C
There
are
more
applications,
obviously,
so,
if
we're
able
to
start
with
compatible
com,
if
we're
start,
if
we're
able
to
start
a
binary
on
1.29
in
compatibility
mode
1.28,
if
we
change
some
of
our
internal
policies,
it
also
becomes
possible
to
start
these
binaries
on
compatibility
versions
that
extend
past
n
minus
one.
For
instance,
we
could
start
a
binary
on
N
minus
3..
C
This
would
theoretically
enable
skip
level
upgrades
and
then
what
we
could
do
is
we
could
upgrade
a
binary
to
n
plus
three
with
compatibility
mode
and
-3,
and
then
we
could
do
like
a
stepwise
bump
of
the
flags
compatibility
mode
Flags
in
order
to
do
like
a
skip
level
upgrade.
That
would
be
sort
of
the
the
high
level
overview
of
how
how
that
could
possibly
work.
A
C
No
I
think
each
controller
would
like
the
thing
that
we
are
considering
is
like.
Joe
is
actually
working
on
this
component
resource
proposal,
which
is
somewhat
rounding
and
parallel,
but
that
would
allow
us
to
model
component
life
cycles.
I
have
not
really
been
thinking
much
about
cross
component.
Skew,
though
I
suppose
we
could
build
that
in
once.
We
have
these
component
registration
things
and
Joe
has
a
hand.
E
Yeah
I
think
I
think
the
answer
is
not
for
this
particular
sub
feature,
but
in
general
I
think
that's
the
direction
we're
headed
towards,
because
this
allows
us
to
be
very
clear
on
you
know
what
our
compatible
version
is
and
so
like,
like
Khan
mentioned,
we're
gonna
We
want
to
build
up
more
more
visibility
onto
what
all
the
versions
of
things
are.
As
you
know,
API
objects,
and
once
we
do
that,
I
think
detecting
versions.
E
A
F
F
G
To
David's
question
about
interlocks,
this
is
actually
something
I
thought
about,
even
if
we
didn't
add
any
of
this
stuff
like
we,
we
say
things
about
you
know.
This
component
must
not
be
newer
than
that
component
or
the
cubelet
must
be
within
the
destinations,
like
I
kind
of
think.
It
would
make
sense
to
already
add
some
of
those
checks
and
like
when
the
cubelet
starts
up.
If
it's
running
against
an
API
server
out
of
SKU
like
by
default,
the
people
should
say
no
I
can't
do
this
or
a
controller
running
against
an
API
server.
G
That's
out
of
supported,
SKU
by
default
should
say
no
I
can't
do
this.
Maybe
we
could
let
people
like
ignore
versions,
Q
checks
or
something
I,
don't
know,
but
taking
some
of
what's
already
in
our
supported,
SKU
Doc
and
making
it
safer
or
more
obvious
when
people
are
doing
things
out
of
bounds,
I
don't
know.
D
More
obvious
than
what
I'm
going
for,
because
I
think
this
is
going
to
get
more
and
more
people
trying
to
run
skewed,
who
maybe
haven't
thought
about
many
of
the
edges,
particularly
around
downgrades,
I've
seen
I've
seen
people
screw
up
downgrade
already
and
I
know
here
we're
going
to
get
faced
with
questions
like
if
I
run,
a
newer
cubelet
in
an
older
compatibility
mode
can
I
run
that
against
my
my
older
API
server,
because
in
a
downgrade,
that's
like
exactly
what
people
want.
Like
my
my
control
plane
went
bad.
D
G
Yeah,
that's
right,
even
if
the
checks
are
simplistic
in
nature,
like
cubelet,
starts
up
and
checks
the
version
of
the
API
server
it's
running
against
and
make
sure
it's
within
bounds.
A
simplistic
check
like
that
is
way
better
than
what
we
do
today,
which
is
no
check
run
and
then
maybe
fail
in
weird
ways.
In
edge
cases,
foreign.
G
C
E
Yeah,
if
people
aren't
sure
what
feedback
to
give
one
thing
I
found
useful,
is
to
read
this
and
then
try
and
imagine
which
upgrade
paths
you
would
want
to
use
with
this
and
see
if
you
think
it
supports
them,
I
like
to
try
and
simulate
different
types
of
upgrades
with
this,
because
there's
there's
more
that
become
possible
and
some
of
them
are
much
safer
than
others.
Yeah.
C
That
kind
of
thought
experiment,
if,
if
you
really
want
to
look
at
the
upgrade
flow,
like
the
testing
Matrix,
that
we
have,
we
have
a
little
small
section
with
the
testing
Matrix
and
that
pretty
much
gives
you
like.
The
envisioned
flow
right.
Like
start
binary
version,
that's
compatibility
version
X.
Then
the
control
plane
ends
up
in
a
mixtape
where
we're
upgrading
like
one
of
the
control
plane,
instances
to
X
plus
one,
but
only
one
of
them.
So
then
we
we
have
a
mix
of
binary
versions,
but
the
compatibility
version
is
the
same.
C
So
everything
in
the
cluster
should
theoretically
be
thinking
that
it's
on
1.28.
Even
though
it's
a
mix
of
1.28
1.29,
then
we
upgrade
all
the
control
plane
instances
to
1.29,
but
it's
still
pretending
to
be
a
1.28
and
then,
and
then
is
the
real
part.
This
is
probably
where
we
would
expect
to
encounter
some
failures,
and
this
is
also
where
we
expect
uvip
and
stuff
to
come
into
play,
which
is
control
plane
at
binary
version,
X,
plus
one
and
mix
compatibility
versions.
D
Yeah
I
was
going
to
point
out
that
one
of
the
sharp
edges
that
we
have
hit
is
we're.
Turning
on
and
off,
Flags
individually
is
that
you
have
to
adjust
the
runtime
config
at
the
same
time
that
you
modify
feature
dates
and
the
runtime
config
has
a
granularity
of
group
version,
but
the
the
feature
gate
has
a
logical
granularity
of
of
resource
in
that
group
version.
D
D
G
B
C
D
C
A
A
A
B
B
F
H
H
I
guess
so,
my
my
thought
process
was
basically
what
is
like.
The
smallest
thing
we
can
do
here
right
so
I,
My
Hope
was
that
that
could
be
a
logical
follow-up
where
we
have
the
capability
to
drive
storage
migration
entry,
and
then
we
start
adding
capabilities
to
automatically
drive
it
based
on
criteria,
whether
the
criteria
be
schema,
changes
or
key
rotation
or
whatever.
D
Yeah,
so
by
bringing
it
in
tree,
it
seems
more
likely
that
we
will
grow.
It
seems
likely
that
we
will
grow
adoption
and
if
part
of
the
thing
we
have
difficulty
with
is
an
API
structure
that
doesn't
work
well
with
API
servers.
I
think
we'd
be
better
served
to
correct
that
during
this
move,
with
a
V1
Alpha
too,
then
then
leaving
it
broken.
H
H
D
H
B
A
H
C
H
The
pr
from
Roy
that
was
going
to
fix
it
so
storage
version
API
actually
was
correctly
honored
for
custom
resources
and
that
never
made
it
and
hasn't
progressed
so,
but
but
the
gist
of
that
bullet
point
is
saying:
I
don't
want
to
try
to
solve
all
of
those
problems.
Now,
if
there
are
problems
within
the
custom
resource
apis
that
exist
out
of
tree
today,
then
yes
most
definitely
we
should
fix
those
before
moving
them
entry
or
as
process
of
moving
inventory.
Okay,.
D
H
H
H
So
nominally
the
high
level
goals
of
this
cup
right
now,
as
stated,
are
there
will
exist
something
inside
of
controller
manager
that
will
make
it
so
that
a
human
being
or
an
automated
entity
can
ask
for
storage
migration
to
happen,
and
then
it
will
happen
how
the
which
apis
that
are.
It
doesn't
really
super
matter,
but
it
basically
says
that
we
will
not
automatically
do
it
on
like
random
kubernetes
schema
changes.
We
will
not
automatically
do
it
on
like
kubernetes
encryption
arrest
changes,
we
will
not
automatically
do
it
on.
H
D
We
killed
the
exposed
served
pieces,
but
we
still
have
to
be
able
to
decode
them
because
there's
no
guarantee
that
they
are
not
stored
in
Sea
today,
and
one
of
the
reasons
for
this
tool
is
to
allow
us
to
permanently
retire
them.
We
wouldn't
be
able
to
do
it
willy-nilly,
because
you
don't
want
to
make
it
impossible
for
people
to
use
your
new
clients
talk
to
your
old
server,
but
on
some
cycle
we
would
finally
be
able
to
remove
the
old
serializations.
H
Yeah,
so
right
now
as
sort
of
written
there
isn't
because
there's
no
automatic
triggering
of
it.
It
doesn't
give
us
that,
yet
if
we
want
that,
I
wasn't
exactly
sure
what
the
automatic
triggering
would
be
on
to
give
us
that
guarantee
that
we
can
finally
make
extensions.
B1
beta
1
go
away,
we're
good.
H
H
H
H
H
D
I
would
really
like
to
see
this
API
use.
The
new
streaming
watch
capability
that
we're
building
I,
don't
think.
I
would
predicate
that
I,
don't
think
I
predicate
a
move
on
it,
but
as
a
ga
criteria,
I
think
that
would
be
really
helpful
for
the
memory
footprint
of
a
q
API
server
when
this
is
running
automatically
and
trying
to
do
those
conversions.
We've
definitely
hit
spikes
even
with
pagination
trying
to
do
this.
D
G
F
G
D
B
D
G
D
B
E
Right
it
might
work
because
when
it
first
starts
streaming,
it
starts
to
streaming
like
the
listing,
and
that
gives
you
a
marker
to
tell
you
that
the
list
is
done.
So
if
your
only
goal
was
to
like
get
the
list,
then
you
would
you
would
get
those
as
watch
events,
but
then
you
get
a
marker,
basically
saying
you're
done
with
the
list
and
you
could
just
stop.
G
Yeah
I'm
just
trying
to
figure
out
like
what
happens
if
it
can't
finish
processing
the
full
set
before
I.
Don't
know
whatever
expires
like
it.
You
know
if
it's
doing
a
no-op
right
to
everything
and
that
takes
a
while
or
it
gets
throttled
or
it
goes
slow
to
avoid
flooding
the
server
with
rights.
Maybe
it
doesn't
handle
the
full
initial
list
in
time
and
gets
dropped
so
then
it
restarts,
but
it
restarts
with
the
full
initial
list
again.
So
maybe
it's
doing
internal
bookkeeping
to
make
sure
it
can
skip
stuff.
G
G
D
D
H
H
H
B
H
G
It
would
be
good
to
Define
if
that
is
a
guarantee
or
if
that
is
just
an
implementation
detail,
especially
around
some
of
the
initial
watch
stuff.
We've
seen
a
lot
of
like
implementation
as
spec,
where
especially
well-considered,
behavior
and
like
ordering
on
initial
synthetic
watch.
Events
was
an
example
that
I
know
was
not
super
well
specified.
I
I.
It
may
be
different
for
the
streaming
watch.
So,
if
that's
the
case,
that's
good,
but
we
need
to
make
sure.
H
I
know
list
is
lexographical
order.
I
just
didn't
know
about
watch,
certainly
don't
know
anything
about
streaming
a
lot
of
stuff.
Okay,
let's
see,
could
you
scroll
up
to
the
nun
goals
for
a
second,
so
I
can
remember
what
I
said
was
a
knock
go?
Do
we
all
agree
that
we
don't
have
to
integrate
with
encryption
arrested
anyway?
D
H
B
H
G
H
C
E
There's
also
we
earlier,
we
were
mentioning
when
we
were
talking
about
the
feature
Flags
about
the
versions
you
problems,
I
think
it
could
also
be
useful
for
that
right,
like
you,
shouldn't
really
upgrade
Beyond
a
certain
point.
If
you
haven't
done
certain
storage,
migrations,
I
think
we
might
have
enough
information
to
kind
of
do
this
in
more
thing
here,.
A
E
Want
to
get
two
I'm,
not
sure
exactly
it's
kind
of
a
night,
it's
kind
of
more
a
general
idea
like
I,
like
Jordan's
idea
of
like
we
shouldn't,
have
to
wait
on
the
feature
Flex.
This
is
where
I
have
enough
information.
Now,
I,
don't
know
exactly.
We
want
to
do
it,
but
I
would
like
to
get
there
as
part
of.
G
G
D
G
To
answer
your
question,
though,
I
think
starting
with
it
being
manual,
is
easier
to
reason
about,
and
we
know
we're
going
to
have
things
driving
it
that
we
won't
have
visibility
to
so,
starting
with
that
surface
area
gives
us
like
the
building
block
and
then,
if
we
want
to
hook
things
into
it,
to
drive
it
automatically.
We
can
consider
that
case
by
case.
H
B
H
Okay,
so
that
makes
sense
to
me
would
would
in
the
implementation
of
this
cap.
Would
we
then
include
any
any
flow,
that's
automatic,
for
example,
if
if
the
storage
version
API
is
enabled
and
the
hashes
change
for
what
API
servers
agree
on,
would
that
trigger
a
creation
of
one
of
these
migration
thingies?
H
G
Until
we're
sure
that
automatically
doing
a
bunch
of
stuff
is
what
we
want,
I
wouldn't
do
it
automatically
like
hey
I,
just
got
the
last
cluster,
the
last
server
in
my
cluster
upgraded.
They
all
agree
and
now
bam,
but
now
I'm
getting
a
ton
of
Rights
or
like
I'm,
not
I'm,
not
confident,
that's
what
we
want
so
until
we
are
confident
I
wouldn't
do
it
automatically,
but
maybe
maybe
there's
ways
to
be
sure.
That's
what
we
want
and
but
I
I
wouldn't
do
it
in
the
first
iteration
I,
don't
know
David
Joe.
D
G
I
think
it's
more
a
question
of
like
when
the
default
state
to
be
for
people
who
are
just
upgrading,
kubernetes
versions
who
aren't
paying
attention
to
this
feature.
Are
we
envisioning
at
some
point?
They
will
do
an
upgrade
and
without
them
taking
any
action
or
changing
any
invocations.
They
start
getting
storage
migrations.
D
D
Have
envisioned
it
being
the
case
where
a
at
a
certain
level
of
kubernetes,
when
this,
when,
when
this
feature
was
stable,
a
user
would
change
their?
The
crd
schema
to
say.
I
no
longer
want
to
serve
this
version,
and
this
controller
would
get
triggered
in
some
manner
to
start
migrating
their
custom
resource,
and
then
the
crd
status
would
eventually
be
updated
to
say
this
version
is
no
longer
stored.
D
Would
be
nice,
and
so
that
means
you
do
these
things
well
for
built-in
types
it
depends
on
how
it's
driven
right.
So
if
you
drive
it
based
on
a
crde
yeah,
it
probably
wouldn't
happen.
If
you
drove
it
based
off
a
storage
version.
Well,
maybe
it
would
you
have
something
stored
that
doesn't
match.
What's
served,
you
say,
oh
it's
unnecessary!
For
this
to
be
stored.
This
way,
let
me
trigger
that.
H
G
G
H
A
H
E
Yeah
I'm
a
little
torn
I
I,
really
like
the
idea
of
getting
crds
done.
Eventually,
I
would
like
to
get
built-ins
fully
automated,
but
I
don't
want
to
like
stop
progress
in
the
effort
for
Perfection
and
so
I.
Would
you
know
as
long
as
we're
as
long
as
we
have
like
a
North
star
and
we're
working
towards
that?
Even
if
one
cat
finishes
the
other
one
has
to
do
that
either
still
is
acceptable
and
like
the
sense
that,
like
somebody,
has
a
task
they
complete
and
we
get
closer
towards
the
goal.
F
I
So
today,
right
with
with
Jason,
we
keep
the
last.
We
produce
non-fail
errors
that
enumerate
all
the
duplicate
keys
that
we're
seeing
and
you
know,
depending
on
field
validation,
options
or
I.
Think
recognizing
encoder
was
the
other
place.
We
can
interpret
these
strict
errors
and
do
different
things.
I
You
can
reject
them
or
not,
whereas
in
protobuf
all
the
generated
on
Marshall
code
is
for
duplicated
Fields.
This
last
field
wins
and
it's
the
same
thing
for
map
map
elements.
The
last
map
element
where
the
key
wins
so
clearly
in
Json.
There's:
ambiguity
right,
API,
duplicate,
Keys,
it's
also
a
text
format.
I
I
I
I
I
I
The
shelf
and
rates
that
have
evaluated,
and
then
the
third
is
sort
of
what
Proto
does
say:
hey
we've
defined
up
front
that
we're
only
going
to
consider
your
last
map
item.
So
it's
not
an
error,
but
except
that
we're
going
to
ignore
any
other
duplicates
which
is
supported.
You
know
by
virtually
every
encoding,
Library
I
looked
at
and
doesn't
have
as
much
performance
cost
so
I'm
curious
to
hear
about
or
huge
misses
on.
Why
we're
doing
this
with
Jason
and
I
see
Joey
everyone.
E
D
I
G
I
think
we
have
a
similar
issue
actually
with
yaml
and
Json.
So
if
you
give
Cube
control
yaml
today,
it
first
actually
converts
it
to
Json
and
then
sends
Json
to
the
server,
and
so
it
loses
duplicate
information
when
it
does
that
transformation
and
when
we
added
the
server-side
validation,
we
captured
that
duplicate
information
at
that
point
client-side
and
made
use
of
it.
So
so
in
Cuba
control,
what
you
say:
Cube
control
apply,
you
give
it
a
gamble,
file,
duplicate
stuff.
G
I
E
And
if
I
mean,
if
there's
a
client
that
needs
to
have
backwards,
compatible
Behavior,
they
could,
when
they
do
their
conversion
to
see
more,
they
could
you
know
Implement
their
behavior
there
right.
So
if
it's,
if
they
want
to
pick
last
thing,
wins
they
do
that
before
I'm
riding
the
C
board,
and
then
they
just
read
the
last
one.
Can.
G
You
go
up
Ben
I'm,
maybe
I,
misunderstood
the
middle
issue.
Duplicate
keys
are
never
acceptable
and
produce
fatal
errors
on
decode.
That's
what
I
would
have
expected
the
default
to
be
for
seaboor
and
I
would
have
expected
that
to
be
way
cheaper.
Actually
because
they
don't
have
to
maintain
a
list,
am
I.
Misunderstanding
I
thought
option
two
would
be
the
default
and
cheapest.
I
D
Can
we
get
a
list
of
people
who
are
actually
interested
in
going
through
the
others,
because
they're
likely
to
take
a
longer
period
of
time?
We
can
find
time
to
discuss
them.
We
have
I,
think
a
better
shot
at
resolving,
so
perhaps
assignment
for
Ben
to
reach
out
to
everyone,
via
slack
and
figure
out
whether
they
are
interested
to
try
to
put
together
one
off.