►
From YouTube: SIG API Machinery Deep Dive - App... Abu Kashem & Stefan Schimanski, Joe Betz & Federico Bongiovanni
Description
Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
SIG API Machinery Deep Dive - Applying What We've Learned - Abu Kashem & Stefan Schimanski, Red Hat; Joe Betz & Federico Bongiovanni, Google
We will have at least 2 deep dive topics in this session. It will be mostly focused around Server Side Apply and the how can client-go and other clients use apply server side. We might cover also several other topics around extensibility using CRDs and Webhooks, touching on the deprecation of v1beta1 and related issues. Additionally, we will remind everybody attenting about the structure of the sig, our areas of coverage, our regular meetings, the places where you can find us to chat, and how to become a contributor.
A
A
A
A
B
B
To
give
a
little
bit
of
context.
Let's
talk
about
how
you
can
access
apply,
both
as
a
human
and
control
and
as
a
controller
in
the
original
client-side
implementation
of
apply.
You
could
access
it
through
coupe
cuddle
apply
and
you
would
provide
a
file
that
included
the
subset
of
fields
of
the
object
that
you
cared
about.
Whenever
you
change
any
of
those
values
and
called
apply
again,
it
would
say
it
would
tell
the
server
to
change
just
that
particular
value.
B
There
was
no
support
for
controllers
for
this.
There's
no
reasonable
way
to
use
it
right.
First
of
all,
all
the
the
applied
logic
was
bundled
directly
into
cuddle.
The
only
way
to
even
theoretically
access
it
would
have
been
to
shell
out
to
cuddle,
but
cuddle's
a
stateful
thing
and
controllers
sometimes
used
to
be
it
just
really
wasn't
a
good
fit.
B
B
All
the
fields
are
tracked
on
the
server.
So
ownership
is
very
clear.
There's
a
bunch
of
conflict
detection
and
resolution
that
goes
on
on
the
server
it's
a
much
better
implementation
of
apply.
I
would
recommend
everybody
switch
to
it
and
nice
conveniently
it's
accessible
to
controllers.
You
don't
need
coop
cuddle
to
access
it.
You
can
access
it
through.
The
client
go
api,
and
today
I'm
going
to
focus
on
a
bunch
of
things
that
we've
done
to
make
this
really
convenient
to
use
from
controllers.
B
To
help
motivate
the
example.
Let's
look
at
what
was
available
in
the
client
in
1.20
and
earlier
and
then
we'll
show
how
we
improved
on
this
so
in
kubernetes
120.
This
is
what
the
api
looks
like
you,
access
server
side
apply
through
the
patch
operation
by
setting
the
patch
type
appropriately,
and
then
you
can
make
your
call.
B
B
B
B
B
B
B
This
is
pretty
bad,
so
what
this
tells
the
server
to
do
is
whatever
service
this
auto
scaler
controls
selling
to
downscale
to
zero
pots.
If
this
is
a
production
service
running
hundreds
of
pods
across
the
cluster,
we
just
downscaled
that
to
zero
pages
were
definitely
gonna
go
off.
This
could
be
a
major
outage,
and
this
is
particularly
worrying
because
there
was
nothing
in
our
request
that
said
anything
about
this.
This
field,
there's
nothing
here
about
max
replicas.
B
B
This
is
pretty
surprising
and
you
should
get
a
couple
takeaways
from
this.
The
first
takeaway
is
that
the
ghost
trucks
are
absolutely
not
safe
to
be
used
with
apply
they're
perfectly
fine
for
create
an
update.
They
were
designed
for
those
use
cases,
but
apply
as
an
additional
constraint
that
you
need
to
only
send
the
fields
that
you
care
about.
So
all
the
defaulting
and
zero
valuing
that
can
happen
in
the
ghost
trucks
which
is
safe
for
create
an
update
is
not
safe
for
why
you
cannot
use
them.
B
B
The
other
takeaway
you
should
hopefully
get
is.
This
is
totally
a
fine
way
to
send.
You
know
a
yaml
file
that
you've
handwritten.
You
can
load
that
file
from
disk
and
then
you
can
send
it
to
a
server
side
apply
through
the
patch
operation.
That's
fine!
It's
just
the
ghost
trucks
here
that
are
the
problem,
all
right.
So
what
I'm
going
to
do?
Instead,
we
still
need
some
kind
of
support
for
controllers
and
the
go
structure
in
work,
so
in
121.
B
What
we
introduced
is
a
bunch
of
packages
bundled
under
this
apply
configurations
package
in
client
go
and
there
is
a
a
new
go
struct
that
we've
generated,
which
we
call
an
apply
configuration
where
all
the
fields
are
optional
and
they're
all
pointers.
So
if
you
don't
explicitly
include
a
field,
it's
just
not
going
to
be
sent
to
the
server
at
all
this
just
kind
of
solves
the
immediate
problem
that
we
described.
B
We
added
some
conveniences
around
these
newly
generated
types.
So
each
one
has
a
constructor,
so
you
can
very
quickly
construct
an
object.
It
automatically
sets
the
name,
the
kind
in
the
api
version
for
you
here.
We
also
have
these
builder
functions,
since
all
the
fields
are
pointers.
We
provide
these
utility
functions
to
set
the
fields
and
you
don't
have
to
provide
the
pointer
value.
You
just
provide
an
actual
reference,
and
this
is
a
lot
more
convenient,
especially
for
primitive
types.
I'll
show
some
examples
of
why
that
would
be
difficult.
B
Otherwise,
in
just
a
moment-
and
then
of
course
the
last
thing
you
do
is
you
call
apply,
we
generated
the
new
apply
function
right
next
to
create
and
update
and
delete,
so
you
no
longer
have
to
go
through
patch
to
do
server
side
apply.
Also.
This
takes
the
apply
configuration
type
as
the
argument,
so
you
don't
have
to
do
any
serialization
and
it's
really
clear
what
type
you're
expected
to
construct
and
send
in
here
all
right.
So
I
mentioned
that
we
needed
these
kind
of
builder
utility
functions
to
get
around
some
problems
with
pointers.
B
Let's
break
that
down
a
little
bit
so
one
there's
a
limitation
and
go
where
you
cannot
do
an
inline
pointer
to
a
literal.
So
ampersand
zero
is
is
not
a
legal
expression
income
you
can
assign
a
variable
to
zero,
and
then
you
can
take
the
a
reference
of
that.
But
then
you
would
have
to
in
this
example:
you'd
have
to
like
put
a
line
up
above
your
actual,
like
inline,
go
struck
declaration,
saying
every
single
literal
that
you're
going
to
then
include
in
your
in
your
in
your
actual
struct,
that's
inconvenient.
B
B
Unfortunately,
this
doesn't
really
fully
solve
the
problem.
Kubernetes
has
a
lot
of
idiom
types
and
you
can't
pass
them
to
pointed
out
string
because
they're,
their
own
type
they're
some
type
ref
to
string
but
they're.
Actually,
you
know
in
this
case
the
protocol.
One
is
core
v,
one
dot
protocol,
that's
its
type
and
since
go
doesn't
have
any
generics
there's
no
way
to
write
one
of
these
pointer
libraries
for
it.
B
Now,
if
you
still
don't
like
the
builder
functions-
and
you
would
rather
you
you
prefer
the
use
of
ghost
drugs-
that's
fine!
You
can
actually
use
the
structs
when
we
implemented
these.
When
we
generated
these
apply
configuration
types
expect
we
exported
all
of
the
fields,
so
you
can
still
use
the
fields
directly.
If
that's
what
you
prefer,
but
I'll
warn
you,
every
field
is
optional.
B
There
are
a
lot
of
pointers
to
deal
with
and
you
might
end
up
changing
your
mind
if
you
go
that
route
and
actually
find
the
builders
are
pretty
helpful,
the
other
thing
I'll
say
is
in
the
future.
This
could
change
right,
go
could
add
support
for
generics.
It
could
fix
the
support,
for
you
know
inline,
pointer
literals,
in
which
case
this
might
become
a
lot
more
convenient
and
the
builders
might
not
be
as
useful.
B
If
any
of
that
happens,
since
we've
already
exposed
all
the
all
the
fields,
you
could
just
start
using
that
new
functionality.
So
that
way,
we've
kind
of
made
our
implementation
future
purif,
while
still
at
the
same
time
trying
to
make
things
as
convenient
as
possible
for
people
today
all
right.
So
now
that's
how
you
use
controllers
in
the
basic
case,
let's,
let's
talk
a
little
bit
about
migrating
controllers
to
server
side
apply.
B
So
in
the
basic
case,
you've
got
a
controller
that
just
has
a
single
reconciliation.
That's
actually
really
easy
to
migrate
to
apply.
Usually
you
just
find
the
one
update
call
you
made
and
instead
of
calling
update,
you
call
apply
and
sell
the
same
fields
that
you
are
changing
when
you
call
to
update
you.
Do
it
unconditionally
right,
even
if
the
field
hasn't
changed
in
the
current
state
of
the
object,
you
still,
you
still
call
it
pi.
B
One
of
the
properties
of
apply
is
that
if
you
don't
include
a
field
in
your
apply
request,
then
the
server
interprets
that
as
you're
not
carrying
with
that
field
anymore
and
if
you're,
the
owner
of
that
field-
and
you
don't
care
about
it
anymore.
The
server
believes
it's
safe
to
delete
it.
So
in
code,
where
you're
doing
an
update,
if
you
have
some
something
like
if
the
field
is
changed,
then
set
it
to
something
you
wouldn't
do
that
you
would
always
set
the
field
to
whatever
value.
B
You
want
that's
kind
of
the
only
caveat
here
other
than
that,
it's
really
straightforward.
Now
there
are
much
more
complicated
use
cases.
We've
seen
a
number
of
controllers
that
have
multiple
code
paths,
whereas
in
some
cases
they
update
some
set
of
fields
and
another
code
path.
They
update
a
different
set
of
fields
for
that
we've
added
a
special
feature
to
try
and
make
those
easier
to
migrate.
I'm
going
to
walk
through
that.
B
If
you
want
to
convert
those
types
of
things
to
apply,
this
is
a
simple
approach.
You
can
take.
The
first
thing
you
do.
Is
you
read
the
object,
just
like
you
did
before
the
next
thing
you
do?
Is
you
do
something
called
extract?
There's
an
extract
function
on
every
single
apply
configuration
type
and
you
provide
in
the
object
that
you've
read
plus
the
name
of
the
field
manager
that
that
controller
identifies
itself
as
and
what
this
function
does.
Is
it
reconstructs
and
apply
configuration
based
on
the
field
manager
state
from
the
server?
B
So
if
previously,
you
applied
field
a
and
b,
and
then
you
call
this
extract
function,
it's
going
to
reconstruct
you
and
apply
configuration
with
a
and
b
again
as
long
as
no
other
you
know,
field
manager
claimed
ownership
of
those.
That's
going
to
be
your
pi
configuration
so
now
you
can
go
ahead
and
just
change
any
fields.
You
want
and
then
call
apply
again.
This
works
great.
B
The
next
thing
you
do
is
you
set
any
fields
that
you
care
about
just
the
same
exact
way
that
you
would
do
if
you're
doing,
update
and
then
the
last
thing
you
do
is
you
call
apply
you
set
force
to
true
because
you're
unconditionally
changing
the
fields
to
the
values
you
want
them
to
be.
This
is
typical
for
controllers.
B
You're
gonna
have
to
read
again
try
reapplying
the
the
changes
you
wanna
make
a
second
time
and
then
try
and
write
again
and
you
could
have
to
loop
and
do
that
as
many
times
as
you
get
conflicts
with
apply,
you
don't
have
to
do
that.
All
the
merging
happens
on
the
server,
so
you
just
send
the
the
request
once
and
you're
done.
You're
also
sending
the
minimal
request
about
the
information
you
need
to
change,
whereas
update
sends
the
entire
state
of
the
object.
So
there's
some
advantages.
B
B
C
C
Priority
unfairness
is
the
new
self-protection
mechanism
for
the
kuberca
server.
Self-Protection
implies
that
we
need
to
prioritize
cluster
critical
requests
for
self-maintenance.
We
also
need
to
prevent
a
flood
of
inbound
requests
from
overloading
and
potentially
crashing
the
api
server
priority
and
fairness
is
available
since
118
and
it
has
replaced
the
max
in-flight
filter.
C
The
maxim
flight
filter
limits
the
total
number
of
executing
requests
at
any
time.
The
kuberpi
server
provides
two
command
line
options
through
which
a
cluster
operator
can
configure
the
concurrency
limits
of
the
api
server
max
requests
in
flight.
This
is
the
maximum
number
of
read-only
requests
in
flight
allowed
at
any
given
time,
and
then
we
have
max
mutating
requests
in
flight.
It
is
the
maximum
number
of
mutating
requests
in
flight
allowed
at
any.
Given
time
when
any
of
these
two
limits
is
exceeded,
the
server
will
reject
the
request
instructing
the
client
to
retry.
C
C
C
C
C
The
cluster
operator
can
allocate
a
concurrency
limit
for
a
priority
level
and
also
a
specified
queuing
configuration
a
vanilla
queries.
Cluster
ships
with
a
set
of
bootstrap
apf
configuration
objects.
This
chart
shows
what
percentage
of
the
server
concurrency
limit
is
allocated
to
each
priority
level.
For
example,
we
reserve
about
12
of
the
server
concurrency
limit
for
cubelet
traffic
via
the
system
priority
level.
C
C
C
We
know
that
cubelet
liveness
checks
are
anonymous
and
further
investigation
revealed
that
all
anonymous
requests
are
assigned
to
the
global
default
priority
level
that
has
a
lower
concurrency
limit
due
to
degradation.
The
global
default
priority
level
was
already
saturated
and
consequently,
cubelet
healthcheck
was
being
throttled
by
apf
and
resulted
in
429..
C
C
C
Ideally
b
should
have
higher
priority
than
a
since
a
is
already
executing.
For
now
we
will.
We
work
around
priority
inversion
by
first
identifying
the
these
spont
requests
originating
from
the
aggregated
api
server,
and
we
then
create
apf
rule
to
mark
them
as
exempt.
For
example,
if
you
have
delegated
authorization,
the
aggregated
api
server
will
send
a
subject
access
review
to
the
kubercase
server
in
order
to
determine
whether
a
should
be
allowed
or
denied.
C
C
This
one
is:
we
allow
the
cluster
operator
to
set
the
server
concurrence
limit
via
the
maxim
5
settings,
what
if
it
is
set
at
a
much
higher
threshold
than
the
actual
server
capacity?
For
example,
the
number
of
cpu
cores
allocated
to
the
api
server
may
not
be
enough
to
sustain
the
load
at
the
concurrency
limit.
C
C
If
the
cpu
usage
goes
beyond
the
yellow
line,
we
will
start
experiencing
cluster
degradation.
The
blue
line
depicts
a
relationship
of
how
cpu
uses
relates
to
increasing
load
where
the
blue
line
meets.
The
yellow
is
the
saturation
point.
Ideally,
the
server
concurrency
limit
should
be
below
the
saturation
point,
since
the
actual
server
concurrency
limit
set
by
the
cluster
operator
is
well
above.
The
saturation
point
degradation
starts
well
before
apf
has
an
opportunity
to
protect
the
api
server.
C
C
The
graph
you
see
in
this
slide
is
an
actual
prometheus
query,
screenshot
taken
from
an
aja
cluster
with
three
qba
server
instances.
It
shows
the
number
of
registered
watches
over
time
per
api
server
instance.
Each
api
server
is
represented
by
a
color
label.
The
gap
on
the
line
indicates
a
downtime
for
an
api
server.
The
cluster
was
degraded
for
some
reason.
C
C
C
Okay,
here
on
the
left,
you
have
an
overloaded
priority
level
requests
are
being
rejected,
since
the
concurrency
limit
has
been
exceeded
on
the
right.
You
have
an
under
utilized
priority
level.
The
number
of
currently
executing
requests
is
far
below
the
limit,
even
though
the
server
has
more
capacity
to
spare
the
over-utilized
priority
level
cannot
borrow
from
the
one
that
is
under-utilized.
C
C
C
A
A
A
There
is
no
need
to
be
an
expert
to
join
either
of
the
meetings.
Everybody
is
welcome.
The
triage
meetings
are
a
great
concrete
way
to
get
familiar
with.
What's
going
on
in
the
day
by
day
of
the
sea,
the
sig
meetings
are
recorded
and
they
are
available
in
our
youtube
playlist
to
get
all
these
invites,
you
have
to
join
the
mailing
list
and
then
you
will
get
them
automatically.