►
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Deep Dive: API Machinery SIG - David Eads, Red Hat & Daniel Smith, Google
This Deep Dives will cover the recent developments of Sig API Machinery. Sig API Machinery covers API registration and discovery, admission control, storage migration, conversion, defaulting, OpenAPI, CustomResourceDefinition, garbage collection, quota, and client libraries. We will have the chance to dive into the topics and to answer technical questions around them. One main theme will be server side apply which is supposed to ship as as alpha in 1.14. This will have a big impact on the ecosystem and therefore we want to take this deep dive as chance to reach developers who will have to adapt their operator logic to work well as an apply actor.
https://sched.co/MPkO
A
A
First
up,
I'm
gonna
have
my
colleague
Jenny
talk
about
apply
since
she's
been
working
on
that
then
we'll
go
to
control
playing
fairness,
I'll
say
something
and
then
David's
gonna
talk
about
certs
and
then
that's
probably
enough.
So
I'll
probably
take
questions
there
and
say
the
series,
and
in
case
everybody
is
like
really
ready
for
more
stuff.
So,
ok
with
that
here,
Jenny.
C
B
Just
gonna
talk
really
briefly
about
server-side
apply
because
it's
an
alpha
right
now
and
we're
gonna
be
moving
into
beta
soon,
thanks,
so
yeah
server
side
apply,
if
you
don't
know
is
we
are
moving
the
logic
from
Q
control
apply
into
the
server,
so
won't
use
strategic,
merge
patch
anymore,
and
most
of
that
is
pretty
straightforward.
The
if
you
send
a
resource
to
the
endpoint,
it's
gonna
do
the
same
thing
as
if
you
had
run
the
client-side
apply
on
it,
but
there's
a
few
differences
and
I'm
just
gonna
talk
about
those
for
a
little
bit.
B
This
is
all
the
stuff.
That's
the
same.
It
accepts
a
partially
specified
object
and
it
knows
the
structure
of
the
resource
from
open
API,
but
we
we
actually
built
a
different
schema
format
for
server
side
apply,
so
it
won't
be
tied
to
open
API,
v2
or
anything,
and
that's
all
housed
in
a
separate
repo,
so
for
client
side
apply.
The
way
one
thing
apply
needs
is
a
sort
of
history
of
the
requests
that
have
been
made
in
the
past
and
for
clients
that
apply
the
way
we
did.
B
B
So
we
have
some
basic
methods
that
convert
an
object
into
field
sets
or
multiple
objects
into
field
sets,
and
then
some
other
things
that
operate
on
field
sets
themselves
and
you
could
see
how
these
operations
could
be
combined
into
some
apply
behavior
pretty
easily.
But
there's
one
more
thing:
that's
different
between
client,
side
apply
and
server
side
apply,
that's
pretty
complicated
and
it's
multiple
versions.
So
when
you
convert
between
versions,
what
actually
happens?
One
thing
that
could
happen
is
the
field
name
can
change,
and
this
is
an
example
of
that.
B
Just
changing
the
field
name
changing
the
value
that
would
be
pretty
easy
to
handle,
but
it's
not
that
simple.
We
could
also
have
a
field
turn
into
multiple
fields
or
the
other
way
around
or
you
could
have
two
fields
turn
into
three
fields
and
then
turn
back
into
two
fields,
but
in
like
a
little
bit
of
a
different
way.
So
this
two
fields
actually
yeah
they
get
all
mixed
up.
It's
end-to-end,
but
it's
even
worse,
because
you
could
also
have
a
field
value
become
a
list
item
which
is
like
can
be
or
a
map
key.
B
So
since
conversion
is
a
black
box
for
apply
we're
gonna
well,
what
we
do
is
we
have
a
convert
function,
that's
exposed
to
the
apply
repo,
and
all
it
knows
is
that
it
can
convert
full
objects.
You
can't
convert
the
field
sets
and
you
can't
convert
those
partially
specified
objects
and
you
can-
and
you
have
to
be
in
the
same
version
for
two
field
sets.
So
it
gets
really
complicated
to
do
the
logic
for
apply
now,
but
luckily
it's
in
a
different
repo,
so
we
could
test
it
really
well
and
and
it's
decoupled
from
kubernetes.
A
Cool
thanks,
Jenny
all
right.
That's
awesome,
we'll
take
some
questions
afterwards,
but
right
now,
I
wanted
to
talk
about
flow
control,
which
is
basically
the
the
problem
is.
If
you
make
a
bunch
of
requests
of
API
server,
it
will
fall
over
and
that's
not
good.
So
we
want
to
fix
that
because,
like
right
now
you
basically
can't
you
can't
drive
to
the
edge
of
what
your
API
server
is
possible
because
there's
no
offense
and
you'll
just
fall
over.
A
So
we
want
to
put
up
a
fence
so
that
you
can
increase
your
utilization
like
provision,
you're
your
master
or
your
your
control
plane
nodes
provision,
those
better
yeah.
So
we
have
the.
We
have
a
few
goals
with
this
effort.
A
prioritization.
Some
requests
are
more
important
than
others,
and
we
want
to
make
sure
that
those
get
more
of
the
throughput
to
examples.
Our
system
requests
might
be
more
important
than
user
requests,
but
even
within
system
requests.
Some
requests
are
more
important
than
others
like
leader.
Election
is
more
important
than
adding
a
random
event.
A
Right
fairness.
That's
the
other.
Major
property
is
like
within
a
priority
band.
If
two
people
or
two
agents
are
sending
requests,
we
want
them
to
get
honored
about
the
same
so
and
then
I'd
say
like
our
other
goals
are
like:
let's
protect
the
CPU,
let's
protect
RAM,
at
least
while
actively
processing
a
request,
there's
a
yeah.
So
in
order
to
get
something
out
in
a
reasonable
amount
of
time
we
took
on
some
non
goals.
A
This
is
not
a
solution
for
a
hostile
das
for
prevention
because
it
doesn't
like
the
the
it
doesn't
protect
the
SSL
handshaking
or
like
the
the
queue
itself.
So
we're
not
quite
all
the
way
there
to
the
to
solving
that
problem.
But
hopefully
you
don't
have
your
yeah
hopeful.
You
don't
have
that
problem
yet
eventually,
eventually,
we've
all
and
yeah
yeah,
so
we're
not
protecting
like
the
RAM
from
queuing.
The
requests
we're
not
protecting
CPU
from
SSL
handshaking,
because
you
gotta
answer
the
SSL
handshake,
whether
or
not
you're
gonna
answer
the
request.
A
You
don't
know
at
that
point,
so
maybe
in
the
future,
but
not
for
the
first
take
also
not
addressing
the
like
of
a
load.
Balancing
like
problems
like
you
could
make
requests
of
you've
got
three
API
servers
running
to
get
all
your
throughput.
You're
gonna
have
to
talk
to
all
three
of
them.
Sorry,
maybe
next
next
take
will
will
add
a
like
a
balancing
layer
or
something
and
a
future
goal.
I
think
an
important
future
goal
is
to
make
it
automatically
tune
itself.
A
So
it
like
is
monitoring
like
how
much
CPU
am
I
using
increase
the
concurrency
yeah.
How
does
it
work,
so
this
is
sort
of
the
the
design
that
we've
sketched
out,
and
this
has
been
the
people
working
on.
This
have
been
min
who's
sitting
right
there
and
David
EADS
and
myself
and
Michael
sprites
are
and
a
few
other
people,
but
that's
been
that's
been
the
most
of
the
participants
in
the
conversation
oh
yeah
and
my
teammate
Aaron,
so
we're
adding
two
API
objects,
a
flow
schema
and
a
priority
level
configuration
and
basically
a
yeah.
A
D
A
Up
at
the
top
there's
a
request,
all
right
and
well.
Let
me
back
up
the
basic
idea
here
is
shuffle
charting,
so
we're
gonna
make
a
number
of
queues
and
the
idea
is
we're
gonna
hash,
the
request
in
some
way
that
we
select
a
few
queues.
We
take
the
shortest
one
and
put
the
request
there
and
then
separately.
We
DQ.
A
That's
that's
the
overall
scheme,
let's
zoom
in
just
a
little
bit,
so
we
have
a
request.
We
compute
some
sort
of
distinguisher.
The
distinguisher
is
part
of
the
fluor
schema
it's
the
way
of
computing
it.
We
use
the
distinguisher
to
hash,
so
the
distinguisher
is
going
to
be
like
the
namespace
or
the
user
and
we
might
add
some
more
in
the
future.
Are
you
gonna,
you're
gonna,
take
this
hash
it
and
that
lets
you
decide
how
some
number
of
queues,
the
level
configuration
gives
you
a
hand
size
and
a
total
number
of
queues
right.
A
So
you
might
have
like
128
queues
and
you're.
Gonna
pick
a
hand
size
of
like
8
8
cards,
so
you
so
the
the
the
shuffle
sharding
step
gives
you
eight
choices
of
queues,
and
then
you
pick
the
one:
that's
the
shortest.
So
what
what
this
does
is
if
you've
got
you've,
got
someone
who's
or
some
agent
who's,
sending
requests.
Let's
say
it's
by
namespace
and
it's
this
one
namespace
is
sending
you
gobs
and
gobs
of
requests.
A
This
will
confine
them
to
just
eight
queues
and
and
they'll
fill
up
those
queues
completely,
but
that
leaves
so
many
other
queues.
And
if
you,
if
you
do
the
math,
it's
like
really
good,
like
the
percentage
of
queues,
that'll
be
free
and
the
the
chance
of
those
those
eight
queues
completely
overlapping
with
with
agent.
That's
just
trying
to
send
a
few
requests
is
very
low,
so
yeah.
This
is
I'm,
really
excited
about
this.
A
This
is
my
contribution
to
the
design
so
yeah,
so
that's
great,
and
then
then
then,
let's
talk
about
the
the
D
queuing
step,
which
I
think
I
have
one
more
slide:
yeah
yeah.
So
the
D
queueing
step
here
is
like
each
priority
level
has
its
own
set
of
cues
right
and
so
they're,
more
or
less
full.
The
D
queuing
step
is
using
a
thing
called
fair
queuing.
A
Micro
sprites
are
helped
us
adapt
this,
and
so
it's
basically
going
to
go
on
a
loop
and-
and
this
is
where
the
prioritization
comes
in,
because
the
the
priority
level
has
a
number
of
shares
so
yeah.
So
if
one
has
like
30
shares
and
the
other
one
has
like
ten
shares
and
then
you
know,
you
get
a
certain
percentage
of
the
over
of
the
overall
throughput
yeah
so
and
then
the
the
D
queuing
just
basically
goes
in
the
circle.
A
While
it's
got
the
capacity
to
handle
another
request
like
pull
it
out
and
service,
it
yeah,
so
I'm
not
sure
if
I'm
gonna
make
alpha
for
this
release,
but
soon,
hopefully
we're
still
sketching
out
the
yeah.
We're
still
sketching
out
the
API.
We're
close,
but
I'm
excited
that
we're
gonna
arrive
at
a
pretty
great
design
here.
So
with
that
I'm
gonna
turn
it
over
to
David
and
we're
can
I
hear
about
shirts.
E
E
All
you
end
up
with
a
chart
like
this
that
doesn't
fit
even
at
the
smallest
possible
font
size,
and
you
may
think
these
are
some
some
mad
mad
scientist's
and
and
he
is,
but
the
rest
of
us
are
trying
to
figure
out
how
to
handle
em
TLS
connections
between
multiple
sources.
So
when
you
actually
break
it
down,
it's
not
as
bad
as
it
initially
seems
right
it.
E
It's
a
few
specific
kinds
of
connections,
so
I've
gone
ahead
and
listed
them
here
and
I'm
gonna
go
ahead
and
get
into
some
detail,
so
you
have
at
C
D
and
you
need
to
be
able
to
have
the
cube.
Api
server
know
that
it's
speaking
to
the
correct
that
C
D
and
you
have
to
have
at
C
D
be
sure
that
it's
the
cube,
API
server,
trying
to
write,
and
not
just
some
other
agent
that
actually
takes
several
flags
to
do
right.
E
You
have
the
same
kind
of
connection
with
the
queue
API
server
when
you're
trying
to
serve
a
quick,
show,
hands
cluster
admins
people
who
actually
try
to
configure
this
right.
So
you
guys
are
familiar
problem
than
was
setting
up
SNI.
You
know
that
you
have
to
set
several
flags
to
pass
us
that
cert
that
you
bought
and
you
have
to
set
several
others.
E
You
may
not
care
about,
but
then
you
have
client
certificates
right,
and
this
is
a
way
for
the
queue
API
server
to
terminate
the
connections
and
identify
users
from
them
and
the
two
most
important
ones
that
you
see
are
for
end-user
connections
where,
if
you
have
a
PKI
infrastructure,
your
signing
certificates
for
users-
and
you
want
them
to
be
able
to
use
their
certain
key
to
talk
and
the
other
one
that
is
really
common
is
cubelets.
Right.
Cubelets
should
use
a
short-lived
certificate
that
uniquely
identifies
which
particular
couplet
is
speaking.
E
You
have
to
be
able
to
terminate
both
the
next
one
is
a
front
proxy
certificate,
so
there
are
actually
two
uses
for
this.
One
is
to
set
up
your
own
front
proxy
actually
of
the
cluster
admin.
How
many
of
you
guys
actually
do
set
up
an
identification
front
proxy
in
your
clusters?
Okay,
then,
the
other
use
of
it,
since
you
guys
don't
do
that
is
for
aggregation
right,
so
you're
familiar
with
the
metrics
server,
probably
the
most
commonly
used
one.
E
You
have
to
be
able
to
authenticate
users
and
the
way
it
works
as
a
request
comes
in
to
the
queue
API
server
and
it's
forwarded
with
identity,
and
you
again
have
to
have
that
mutual
TLS
connection
cubelets
serving
has
to
be
trusted
by
the
cube
API
server
cubelet
clients
have
to
be
have
to
be
trusted,
and
the
cubelet
has
to
be
able
to
terminate
connections
from
the
cube
api
server
that
one's
a
little
less
obvious
people
don't
always
know
that
one.
The
last
two
are
also
related
to
are
also
flags
for
the
controller
manager.
E
E
It's
gonna
have
a
default
cert
and
it's
gonna
have
sni
certs
so
that
you
can
specify
different
names
and
get
the
correct
certificate
serve
with
it,
and
then
the
client
uses
the
server
see
a
bundle
to
trust
on
the
other
side.
So
you
can
see
here
we
have
what
one
two
three
four
five
six
flags
you
have
enough
of
these
connections
and
they
do
start
to
add
up.
E
E
And
the
answer
is
that
you
do,
but
for
every
one
of
you
there
are
somebody
else
who
doesn't
and
they
just
want
their
connections
to
work
right
and
they
want
them
to
be
secure
and
they
don't
want
to
worry
about
it
and
so
having
distinct
chains
of
trust.
It's
not
a
perfect
way,
but
it
limits
the
blast
radius.
Should
something
go
wrong,
so
I'm
gonna
give
a
few
examples
and
I've
labeled
all
these
pictures.
So
if
anyone's
looking
later,
they
can
actually
see
all
the
different
flag
names
related
to
the
connections
they
have.
E
So
you
can
see
you
have
cube
control
and
talking
to
the
cube,
API
server
and
those
flags
on
the
queue
API
server
are
the
actual
ones
you
set
for
this
purpose,
so
I'm
not
going
to
go
through
all
of
them
because
you
can
read
them,
but
you
have
cube
control
to
the
queue
API
server
cube,
API
server
to
add
CD,
a
duplicate,
the
cubelets
to
the
cube
api
server.
So
this
is
when
cubelet
is
asking
for
pods
and
services
volumes
PVS.
It
asks
for
a
bunch
of
stuff.
E
Then
you
also
have
the
cube
api
server
back
to
the
cubelet.
This
is
the
one
that
happens
when
you're
trying
to
do
a
cube,
control
logs
right,
that's
actually
a
proxied
call,
so
you're
you're,
making
a
call
to
the
QB
API
server
queue.
Api
server
then
makes
a
different
request
on
your
behalf
to
the
cubelet,
because
you
don't
necessarily
have
network
connectivity
and
your
cert
isn't
good
to
terminate
there
and
it
has
to
be
able
to
verify
it's
talking
to
a
real
cute
lit.
E
Now
we
have
a
slightly
different
case:
it's
not
just
a
straight
request
to
the
queue
API
server
of
a
certificate
that
one
will
normally
terminate
and
give
a
user
identity
on
the
request
that
is
related
to
the
cert
that
you
used
there's
another
case
where
you
have
a
front
proxy
set
up-
and
this
is
what
I
was
talking
about
with
aggregation.
So
you
have
a
front
proxy
which
terminates
TLS
and
it
decides
which
user
you
should
be
and
makes
another
request
proxied
through.
That
asserts
the
identity
on
that
request.
E
So
if
deads
2k
makes
a
request
to
the
server,
he
would
make
a
request
to
the
cube
aggregator,
which
is
built
into
the
queue
API
server
for
metrics
the
queue
API
server
says:
yep
I
recognize
this.
Somehow
this
is
deads
2k
and
I'm
going
to
assert
it
to
the
metric
server,
and
so
the
metric
server
has
to
confirm
not
that
it's
dad's
to
K,
because
des
Duque
didn't
make
the
request
right.
The
one
that
it
sees
is
actually
from
the
queue
API
server.
It
needs
to
see,
isn't
really
the
queue
API
server.
E
Talking
to
me
does
the
cube.
Api
server
have
the
power
to
assert
identity:
okay,
yes,
it
does
and
it
is
deads
to
que
speaking
to
so
there's
two
common
ways:
you'll
see
this
in
clusters,
someone
apparently
not
uncommon
Leigh
will
use
Apache
with
semat
off
and
they
will
have
Apache
terminate
the
connection.
E
Apache
will
decide
which
user
it
is
and
then
send
it
on
to
the
queue
API
server,
asserting
identity
and,
in
the
other
case,
which
is
common
if
you're
going
to
be
using
if
you're
going
to
be
using
metrics,
is
to
have
the
cube
aggregator
send
it
through,
and
there
are
a
few
other
aggregated
API
service
service
catalogs
an
example:
OpenShift
has
an
aggregated
api
server
as
well
all
right
so
search
CSRs
and
cubelets.
It's
important
to
always
remember
the
relationships
that
you
have
between
the
components
and
this
can
be
kind
of
difficult.
E
As
in
this
example,
we
have
a
set
of
three
binaries
total
of
six
flags
and
they
are
all
related.
They
have
to
move
together.
If
you're
gonna
turn
on
cubelet
rotation,
you
need
to
be
sure
that
your
cube
controller
manager
is
actually
going
to
sign
those
certificates.
So
you
have
to
pass
the
flags
for
the
cube
controller
manager
or
you
get
cubelets
that
don't
go
anywhere.
They
get
stuck
waiting
for
their
cert
to
be
signed
and
then,
even
after
you've
done
that,
you
need
to
make
sure
that
your
queue
API
server,
has
the
CSR
serve.
E
They
use
to
sign
from
the
controller
manager
in
its
clients,
EA
bundle
and
inside
of
the
Hulett
certificate
authority.
I
had
to
look
I,
don't
remember
that
flag,
the
Cubitt
certificate
authority
and
if
you
forget
any
of
those
pieces,
if
you
forget
the
cube,
API
server
pieces,
you
can
have
cubelets
that
can't
get
a
list
of
their
pods
that
run
on
them.
They'll
get
401's
back
or
if
you
forget
the
cubelet
certificate
authority,
you'll
get
a
cubelet
that
you
can't
get
logs
from
I.
E
A
E
No,
so
the
question
was:
is
this
thing
that
I've
discussed
here
the
situation
where
you
have
six
flags
that
are
all
interrelated
that
span
multiple
binaries?
Is
this
the
objective
of
the
component
config
to
solve,
and
the
answer
is
no?
The
component
config
is
trying
to
solve
the
problem
of
how
do
I
represent
the
configuration
for
an
individual
binary,
but
it's
not
going
to
be
opinionated
about
how
that
configuration
is
distributed.
Not
all
distributions
or
configurations
are
going
to
agree
on
exactly
how
that
says.
So
the
way
I
want
to
config
is
the
common
denominator.
E
A
E
A
A
A
E
Can
we
can
talk
about
Co
DS
up
just
before
I?
Do,
though
this
is
similar
to
Stefan's
the
end
of
Stefan's?
Talk
about
open,
API
publishing
what
percentage
you
guys
were
here
for
that
a
fair
percentage,
I'll
buzz
through
it
real
quick.
So
there
are
a
few
pieces
that
are
going
to
be
important
for
CRTs
to
go
to
GA.
We
want
the
behavioral
changes
related
to
CRTs
to
all
land.
Before
we
make
a
GA
and
Beyond
by
default.
That
doesn't
necessarily
mean
GA
themselves.
E
Some
of
them
will
probably
still
be
beta,
but
on
by
default
for
behavioral
changes,
and
so
on
this
slide,
those
behavioral
changes
are
open,
API,
publishing,
pruning
and
defaulting.
Those
are
the
ones
that
I
can
think
of
right
off.
The
top
of
my
head,
so
pruning
seems
a
little
weird.
When
you
first
see
it.
If
you
have
a
non-existent
field,
then
you
submit
it
to
any
normal
API
resource.
It
will
actually
be
removed
on
its
path
to
Etsy
D.
It
gets
removed
because
I
think
very
originally.
E
And
so
it
has
a
couple
side
effects,
one
good
and
one,
maybe
not
as
good,
so
we'll
go
not
as
good
first
veggies.
First,
what
happens
is
if
you're
trying
to
do
a
create
of
a
cool
new
feature-
and
you
happen
to
use
this
manifest
against
an
older
server.
We've,
probably
all
done
this.
It
gets
created
without
your
feature.
E
E
So
if
you
have
some
special
new
field,
maybe
it
didn't
exist
in
release
on
a13,
but
you
added
an
it
containers
and
you
had
someone
who
was
smart
enough
to
create
an
entire
piece
inside
of
of
their
pod
before
you
upgraded
the
cluster,
you
upgrade
the
cluster
and
suddenly
there's
an
an
it
container
that
gets
created
that
shouldn't
be
it's
super
privileged
somehow
in
it
container
and
it
owns
your
cluster.
You
would
be
very
unhappy.
This
is
the
good
part
about
pruning
right.
E
A
E
It
doesn't
matter
so
much,
but
the
other
thing
that
is
going
to
be
a
behavioral
change
is
defaulting,
so
we're
all
familiar
with
the
easy
case
of
defaulting.
You
create
an
object.
You
want
to
default
value,
you
write
to
the
server,
it
gets
written
debt
CD
with
the
defaulted
value.
You
do
the
read
and
it
comes
back
out.
This
is
the
case
that
people
see
and
they
all
think
well.
I
can
just
do
defaulting
in
my
own
mutating
admission
web
hook
right
and
for
this
case
you
are
correct.
E
E
It
does
not
have
this
neat
defaulted
field
right,
but
when
we
upgrade
and
we
have
defaulting
when
we
do
the
read,
we
will
get
that
fancy
new
default
of
field
on
the
read
we
don't
have
to
rewrite
the
resource
at
CD
still
has
it
empty
and
it
happens
on
the
read
side.
There's
no
admission
on
reads:
you
can't
reproduce
this
functionality.
E
Arbitrary
sub
resources
is
something
that
I'd
like
to
see
on
the
roadmap
and
I
do
not
believe
it
is
required
for
GA,
though,
and
the
reason
I
don't
think
it's
required
for
GA
is
that
it's
not
going
to
change
the
behavior
of
existing
functionality
right.
If
you
look
at
pruning
it
changes
what's
actually
stored.
If
you
look
at
the
faulting,
it
changes
what
you
actually
read,
but
this
is
just
an
additive
change
that
allows
you
to
have
more
power
right
and
the
idea
here.
So
you
could
do
something
like
logs
or
port
forwarding
right.
E
If
you
know
how
a
logs
requests,
work
and
I
sort
of
mentioned
it,
you
had
go
through
a
queue.
Api
server,
which
has
a
bunch
of
logic
to
decide,
can
I
proxy.
Where
do
I
proxy
to
then
I
use
my
identity
to
proxy
it
using
some
of
the
flags
that
you
just
talked
about?
Yes,
anyway,
does
all
those
things
you?
We
can't
realistically
build
this
in
to
the
queue
API
server
that
logic
we
can't.
We
can't
allow
someone
to
write
that
program
and
have
us
run
it
for
them,
but
maybe
something
like
API
aggregation.
A
E
I
think
that
would
be
good
too
yeah
and
then
the
last
one
that
I
came
up
with
was
priority.
If
you've
worked
a
lot
with
CRTs,
you
may
have
hit
this.
Some
names
are
just
popular
policy:
it's
not
policy,
no
matter
what
it
is
is
not
policy,
because
what
happens
is
somebody
else
will
have
policy.
You
will
try
to
get
a
policy
and
you
won't
get
the
one
you
want.
So
the
example
I
have
here
is
a
foo
and
it's
in
two
different
groups
and
you
do
a
cube,
control,
get
foo
and
which
foo?
E
Should
you
get
right
now?
The
results
is
predictable,
lays
the
same,
but
for
the
life
of
me,
I
can't
remember
what
it
is
and
and
I
think
we
need
to
come
up
with
a
way
for
people
to
wait
and
balance
them.
Part
of
the
problem
is
that
everyone
will
believe
it.
There's
is
the
most
important
I
mean.
Clearly
you
want
my
policy,
not
Daniels
policy.
E
So,
like
the
one
consistent
rule
that
absolutely
everyone
agrees
to
no
CRD
should
ever
get
to
override
a
core
kubernetes
resource.
So
we're
not
going
to
allow
that
to
be
possible,
but
I
think
we'll
be
trying
to
add
functionality
for
this.
At
some
point
still
I
would
say
in
your
scripts.
The
standard
advice
is
always
there
always
specify
your
entire
resource
version
group
when
you
specify
it
so
that
you
get
consistent
serialization
as
you
try
to
do
something
like
output,
the
JSON
yeah.
That's.
A
A
D
F
F
C
E
Suspect
that
so
so
there
was
a
the
overall
conversation.
Just
for
the
record
was
about
I
think
what
project
would
we
use
to
try
to
create
something
to
serve
CRD,
sub-resources
or
similar
api
server?
I
suspect
that
once
we
actually
create
the
ability
to
serve
CRT
sub
resources,
there
will
be
more
interest
in
creating
compatible
servers.
To
that
end,
we
actually
created
the
kate's
io
api
server,
repo
with
primitives.
That
would
make
this
possible.
It's
not
necessarily
easy
today,
but
I
think
that
once
the
CRT
feature
comes,
we're
gonna
see
more
so.
A
So
speaking
of
sub
resources,
there's
two
categories
of
sub
resources:
there's
things
like
the
scale
sub
resource
which
is
like
I,
took
some
fields
and
I
manipulated
them
in
some
way,
and
now
I've
got
this.
This
type
that
I
can
serve
from
multiple
places
and
multiple
multiple
people
can
understand
it.
The
other
kind
of
sub
resources
like
logs
is
exact,
attach
port
forward,
where
it's
like
some
completely
non
kubernetes,
specific
or
kubernetes
specific
and
strange
protocol,
which
is
not
like
a
declarative
thing.
So
those
two
cases
are
quite
different:
yeah.
E
A
I
think
the
the
namespace
name
space
finalize
errs
are
the
same
yeah
yeah.
Thank
you
again.
So
certainly
the
logs
attached
port
forward
that
that
sort
of
stuff
I
think
that
needs
to
be
implemented
in
the
front
door
like
the
aggregator.
If
we
want
to
customize
that
so
it'd
be
like
you're
registering
for
it
in
the
API
service
object
I.
We
haven't
actually
designed
this
yet
yeah.
The.