►
From YouTube: Kubernetes SIG API Machinery 20180620
Description
For more information on this public meeting see this page: https://github.com/kubernetes/community/tree/master/sig-api-machinery
A
B
Yeah
I'm
happy
to
talk
about
this
one,
so
I
think
this
got
added
to
the
to
the
agenda,
because
who
are
we
trying
to
serve
CRTs
consistently
or
something
anyway?
The
the
I've
noticing
a
pattern
where
there's
no
good
way
for
API
servers
to
do
things
sort
of
at
the
same
time
right
like
if
we
want
to
sync
up
discover
like
like
so
so.
The
scenario
here
is
you're
in
an
aka
situation
and
we
have
to
consider
like
the
maximally
hard
case
where
you're
doing
a
rolling
upgrade
of
API
server.
B
It's
very
similar
problem
to
I've
turned
on
a
CR,
D
and
I'm,
not
sure
if
all
of
the
API
servers
have
turned
it
on
yet
or
not.
Additionally,
there's
like
suppose,
I
do
a
rolling
update
and
toggle
in
API
server
flag
like
I,
earn
a
group
on
or
off
by
runtime
config
and
there's
also
say,
I.
Add
an
API
service
like
a
aggregated
API
server.
B
How
do
I
know
when
that
has
been
added
to
API
server,
so
there's
sort
of
I
think
two
components.
The
problem
one
is
for
some
things:
users
need
to
know
when
we're
done
where,
when
it's
active
and
there's
also
like
to
support
that
I
think
API
server
itself
needs
to
have
some
information
about
the
state
of
other
API
servers
in
the
system.
B
B
C
C
B
We
need
some
sort
of
function
forcing
function
to
make
this
better.
I
am
okay
with
additive
things
that
don't
go
in
a
wrong
direction
and
also
don't
change
this
behavior
I'm,
not
okay,
with
adding
things
that
solve
this
piecemeal
without
considering
the
the
whole
thing
is
that
that
makes
the
problem
larger.
Instead,.
C
B
C
B
I
mean
I,
guess
it
sort
of
depends
on
what
you're
gonna
put
in
the
config
Zee
end
point
like
yeah
I.
Another
aspect
of
this
is
we
need
to
look
at
the
flags
that
API
server
takes
on
the
command
line.
I
think
there's
two
categories.
One
is
like
specific
things
about
this
API
server
on
this
host
and
the
other
is
flags
that
need
to
be
synchronized
between
all
API
servers
in
order
to
make
sense
and
and
really
like
flipping
a
flag
in
the
latter.
Category
requires
like
a
rolling
like
a
rolling.
C
Update
or
something
of
that
nature
in
that
case,
though
I
don't
know
so
so,
I
see
something
like
the
the
dynamic
auditing
it's
dynamic
config
for
auditing,
as
analogous
to
dynamic
admission
where
you
want
to
have
a
uniform
set
of
audit
rules
across
your
cluster
right
and
I,
see
that
as
distinct
from
something
like
I
want
to
set
a
particular
Phi
on
the
cube,
API
server
to
I,
don't
know
change
its
host
bind
parameter
or
something
right.
So,
yes,.
B
D
C
D
C
B
Yeah
I
I
agree
that
we
should
let
people
continue
to
add
things.
I
think
we
shouldn't
let
them.
We
should
make
sure
that
they
add
things
that
still
have
this
problem.
Basically,
rather,
unless,
unless
they're
offering
a
solution
that
can
be
generalized
because
I
don't
want
like
one-off
solutions
for
each
of
these
things,
that
seems
like
that
seems
like
a
poor
state
to
end
up
in
yeah
and
and
David
mentioned
admission
web
folks.
B
D
I
think
I
just
got
scopes
down
to
kind
of
strip
out
a
lot
of
the
multi
level.
Cluster
scope,
namespace
scoped
bits
to
not
try
to
subdivide
like
policy
around
what
kinds
of
audit
you
should
be
able
to
set
like
basically
letting
the
dynamic
version
have
the
same
ability
as
the
currently
provided
static
version
and
yeah.
C
D
B
Like
even
like
dynamic
auditing
should
be
like,
we
should
be
careful
with
that.
Api
like
if
it's
referencing
say
a
file
on
the
file
system
as
a
destination
for
dynamic
auditing
that
should
not
be
in
an
API.
That's
shared
among
all
API
servers
right
because
that
file
could
be
there
different,
different
hosts
right.
So
I.
D
Think
the
the
remaining
questions
were
is
the
dynamic
that
a
singleton
initially
or
does
it
allow
for
multiple
destinations
with
different
policies?
The
former
is
much
a
much
easier
path
from
where
we
are
today,
but
pretty
limited.
The
latter
requires
pretty
big,
possibly
requires
changes.
Big
changes
to
the
audit
pipeline,
which
today
is
optimized
around
short-circuiting
and
not
doing
any
more
work
once
the
policy
says
it
doesn't
it
no
more
detail
is
needed,
so
I
think
those
were
the
last
two
kind
of
questions
left
singleton
versus.
F
F
B
B
F
B
H
F
B
B
D
J
B
F
Don't
think
there's
a
use
for
having
multiple
backends
and
maybe
different
policies
for
those.
So
it's
is
tricky.
I
mean
we're.
Trying
I
guess
skip
it
down
to
just
talking
about
the
web.
Folks
I
definitely
think
there's
a
use
for
for
multiple
web
folks,
but
I
do
heed
the
warning
that
that
may
be
difficult.
F
B
Yeah
I
think
it
makes
sense
to
have
multiple
web
folks,
like
I
I,
can
imagine
like
people
making
standard
consumers
of
or
servers
for
these
web
hooks
for
various
audit
backends,
and
maybe,
if
you
want
to
roll
over
from
one
to
the
other,
it
makes
sense
to
enable
both
for
a
while,
so
I
I
think
I
think
that's
nice
to
users.
That
doesn't
mean
it's
worth
the
cost
sure,
and-
and
it's
also
as
long
as
you
don't
make
the
API
too
horrible.
B
F
B
B
K
K
K
Probably
having
side
effects
el
padre
island
space,
if
you
keep
for
us,
for
example,
and
also
changing
the
storage
layer,
so
that
who
would
avoid
stoning
the
object
in
case
of
a
dragon,
the
goal
is
to
go
as
deep
as
we
can
in
the
stack
so
that
we
can
return
an
object
that
is
going
to
be
as
close
as
possible
as
it
would
have
been
stopped.
The
goal
is
to
get
an
idea
of
what
would
have
happened
if
it
had
happened
without
having
it
happen.
C
Yeah,
so
it
looks
fairly
straightforward,
but
there
are
a
couple
things
worth
noting.
We
talked
in
the
comments
about
admission
controller
config.
It
seems
like
a
good
idea
to
force
an
admission.
A
web
hook,
admission
piece,
a
yes
I
support
dry
run
because
not
supporting
dry,
run
and
being
called
is
kind
of
a
big
deal,
and
it
can
default
to
false
now
and
to
true
later
in
a
later
version,
because
we
know
we're
gonna
have
more
versions
to
come
when
it
goes
GA.
The
other
was
inside.
C
Of
our
actual
storage
stack,
the
there
are
rest
interfaces
for
things
like
there's
like
rest
update,
graceful
deletion,
creator.
Things
like
that
I
think
is
gonna
be
worth
having
a
new
interface
there
for
things
that
support
a
dry
run,
so
that
someone
and
there's
a
new
level
of
duty.
They
will
actually
get
a
compile
failure
when
they
try
to
use
it
and
I
am,
and
so.
C
And
there
was
one
other,
oh,
the
behavior,
of
something
like
Florida,
the
Oh
Daniel
just
typed.
Yes,
different
want
to
do
different
things,
so
I
think
it's
worth
noting
that
the
major
quote
I
think
we
all
agree.
Quotas
should
not
bump
usage,
but
whether
or
not
quarter
should
reject
is
less
obviously
I
want
to
use
dry
run
to
see
if
they
have
enough
quota.
Other
people
want
to
use
dry
run
to
see
what
the
object
would
look
like,
and
so,
if
quota
rejects,
it
makes
it
impossible
to
fulfill
that
latter
use
case.
D
D
E
C
C
E
Other
cases
here
is
like
there's
different
types
of
UI
is
a
CLI
is
not
the
same
as
a
web.
Ui
UI
I,
don't
necessarily
need
to
validate
a
controller
that
somebody's
building.
An
experience
around
like
a
deployment
doesn't
need
the
same
thing.
The
deployment
controller
needs
when
it
needs
to
validate
or
when,
when
you're
running
a
creating
a
pod
directly,
so
I'd
be
like
even
at
the
highest
level,
we
just
say:
don't
put
googly
incident
general
level.
I
think
that
might
even
just
be
big
people
need
a
hood.
B
G
K
D
E
I'm
not
saying
bypass
our
back
I'm
saying
there
is
a
valid
use
case,
which
is
I.
Want
a
system.
Look
a
bunch
of
objects,
I
don't
have
a
namespace.
Am
I
as
a
client
asking
the
question?
Is
this
object
valid
for
this
server
namespace
or
valid
for
my
rules,
plus
that
I'm
not
saying
that
to
solve
that
in
this
particular
API
flow,
but
like
like?
E
C
D
C
Information
I
was
the
same.
Are
you
for
Clayton's
point?
I
was
trying
to
explain
me.
Why
do
I
have
some
things
that
wanted
run
quota
and
be
enforced
and
some
things
that
don't,
as
opposed
to
the
no
namespace
thing,
I,
think
that
just
representing
the
the
query
parameter
as
a
string
and
threading
the
string
through
and
then
having
individual
mission
webhooks
can
look
and
say
do
I
support
this
train.
Do
I
know
what
it
means.
My
like
I,
don't
reject
kind
of
hey.
We
we
probably
need.
C
E
Well
under
if
we
just
want
tight
list
of
phases,
like
logically
like
there's
in
the
validation,
I
got
to
go
back
and
look
at
Andy's
original
doc,
but
we
actually
talked
about
the
different
student
different
types
of
validation,
Optima,
its
internal,
like
it's,
the
object
internally
consistent
is
the
object
consistent
within
the
other
resources
that
might
be
in
that
namespace
is
the
object,
consistent
with
limits
that
are
applied
to
you,
I
think,
there's
one
other
one.
Other
distinction
again
like
I,
don't
know
that
we
need
to
implement
all
of
them.
K
D
C
A
G
So
hi
I'm,
Jenna
I'm,
usually
in
sick
apps
and
I,
worked
on
workloads
api's.
So
in
C
caps
we've
observed
some
limitations
of
garbage
collection.
So
that's
why
I
prepared
this
proposal
to
expand
the
garbage
collections
and
I
plan
to
work
on
this
on
in
Kanto,
so
I
don't
plan
to
discuss
the
details
of
the
design
today
because
I
just
publish
it
yesterday,
I
love
everyone
to
take
a
look
and
give
me
some
input,
maybe
discuss
it
in
the
next
meeting.
G
A
L
Yes,
I'm
here,
I
just
wanted
to
pick
your
brains
and
and
develop
some
understanding
of
well.
My
original
question
was
scaling
in
the
garbage
collector,
but
actually
Clayton
gave
an
answer
that
really
answered
the
real
question,
which
is
really
I,
need
to
I'd
like
to
understand
everything
that
has
some
need
to
to
load
all
the
objects.
I
mean
that's
a
and
others
can
sort
of
like
to
understand
that.
L
So
did
you
follow
up
on
Clayton's
ants
or
let
me
just
pick
much
so
in
the
API
servers
there
is
I
may
help
me
with
the
terminology
right.
There's
this
thing
called
hoob
aggregator
and
what
I
think
of
as
the
main
API
servers
is
built
on
the
KU,
bagra
Gator,
and
then
something
that
provides
the
the
entry
types
and
someone
without
of
tree
types
can
create
additional
API
service
objects.
That
register
where
to
find
the
anchors
yeah.
L
D
B
C
It's
that
way,
yes,
the
Informer's
should
be
shared
amongst
them
all
and.
D
L
K
L
To
ask
about
both
and
thank
Clayton
for
answering
about
both
I
was
just
trying
to
discuss
them
one
at
a
time,
so
I
think
I've
gotten
most
of
the
answer,
then
about
the
API
service.
So
each
one
is
gonna
load,
every
object
of
every
type
that
it's
responsible
for
and
I
think
Clayton
mentioned
that
if
there's
multiple
versions
in
play,
if
the
watch
cache
is
gonna
hold
a
copy
to
each
of
those
versions,
right,
yeah.
C
C
E
Believe
only
stores
metadata,
yes,
the
grant,
but
you
saw
the
informants
backing
it,
which
store
the
whole
direct
web
engine.
It
has
a
few
cases
where
we
have
started
to
have
reflector
driven
caches
that
only
cache
a
subset
of
the
object
defined
by
the
cache
like
keep
like
you
know,
the
resource
version,
the
name
and
the
namespace
and
a
couple
other
fields.
E
B
L
Okay,
so
yeah,
so
let's
go
on
to
the
garbage
collector,
so
the
garbage
collector
runs
in
the
controller
manager
right,
okay
and
the
garbage
collector
loads,
everything
except
events.
So
it's
loading
the
internal
version
or
some
which
version
external
approval,
which
external
version
yeah.
That's
actually
yeah,.
M
D
L
B
D
D
B
D
H
D
B
So
I
I
have
a
somewhat
related
thought,
which
is
eventually
we're.
Gonna
have
to
think
about
charting
our
controllers
by
namespace.
I.
Think
is
the
only
way
to
do
it,
because,
eventually,
we're
gonna
have
clusters
that
are
big
enough,
that
we
don't
want
to
run
all
the
controllers
on
one
system
for
all
resources,
yeah.
L
B
So
the
way
the
only
way
I
could
think
of
that
will
make
charting
work.
Is
you
have
to
shard
you
can't
charge
within
a
day
in
space
you
have
to
shard,
you
have
to
include
entire
namespaces
and
the
reason
I
think
it
has
to
be.
That
way
is
because
there's
a
lot
of
controllers
that
depend
on
the
visibility
guarantees
so
like.
If
your
deployment
controller,
you
have
to
look
over
all
the
replicas
sets
in
the
namespace
to
see
if
any
of
them
are
owned
by
the
deployment
you're
trying
to
control.
So
because
of
this
visibility.
B
Concern
I,
don't
think
that
our
system
I,
don't
think
it
makes
sense
to
shard
our
system
into
pieces
smaller
than
a
namespace.
So
I
think
we
need
to
think
about
charting
groups,
including
entire
namespaces
groups
of
namespaces.
We
we
don't
have
right
now
from
API
server.
We
support
watching
all
namespaces
or
an
individual
namespace.
We
go
have
a
option
where
you
can
watch
several
namespaces
I
mean
you
could
set
that
up
with
this
series
of
watches
yourself,
but
we
don't.
B
E
So
I'm
gonna
push
back
a
little
bit,
which
is
I,
don't
want
to
do
sharding
so
the
next
year,
because
I've
got
a
bigger
than
everybody
else's
clusters,
except
maybe
huawei's,
and
it
hasn't
been
a
practical
problem.
Yet
except
David
keeps
making
things
more
inefficient,
but
that's
not
David's
problem.
It's
like
that's
just
our
architecture.
E
So
like
I
right
now,
the
biggest
clusters
I
know
of
peaked
at
about
20
gigs
for
the
controllers,
which
is
like
it
was
like
that
was
like
it
wasn't
quite
the
5x
load,
but
it
was
something
like
3
to
4
X
3
on
forex.
What
prefix,
3
4x
overlap,
do
two
redundant
cashes
garbage
collector,
cashing
things
good
and.
G
E
E
At
we've
bounced
off
the
limit
several
times
on
eight
yeah,
it's
kind,
one
of
those
things
where
I
just
I
want
to
I
want
to
get
everybody
in
the
community
to
kind
of
buy
off.
Before
we
go,
do
something
complicated,
which
is
there's
a
lot
of
work
to
do
sharding
in
practice.
Do
we
really
expect
clusters
to
get
past
five
or
ten
million
key
boundary.
L
E
I
mean
I
know
like
most
of
the
high
cardinality
by
namespace.
Resources
are
our
back
quota,
anything
that
you
need
one
of
per
name
space
because
name,
space
roughly
equals
tenant,
and
that's
about
a
million
with
20,000
I
mean
if
you
want
to
get
you
want
to
do
that
by
an
order
of
magnitude.
It's
within
reason.
G
E
B
I
know
what
it's
gonna
ask:
are
those
27
billion
keys
gonna
be
in
one
namespace
or
are
you
gonna
use
multiple
namespaces
yeah
that
all
these
political
money,
namespaces
okay,
so
I
mean
you
could?
If
you
want
to
manage
watching
each
namespace
individually,
you
could
actually
shard
controller
manager.
Today
it
wouldn't
be
easy.
The
the
the
thing
that
you
can't
do
is
get
API
server
to
or
tell
it
I
want
to
watch
this
range
of
a
big
case
of
namespaces.
You
can't
get
that.
L
Right
so
I'm
thinking
about
something
you
know
I'm
talking
about
making
an
infrastructure
while
control
plane
using
the
Korean
API
machinery,
so
India
knew
an
infrastructure
cloud
will
typically
have
a
large
number
of
tenants
all
right.
So
we
want
to
be
able
to
handle.
You
know
thousands
of
tenants,
so
I
hate
to
make
a
shard
for
each
tenant.
That
would
be
too
small
and
I.
Think
that's
where
you're
going
right.
You
want
handles
as
some
kind
of
aggregation
there
yeah
I
I
can
tell
you
like.
B
Like
there's
sort
of
two
ways
you
can
do
multi-tenancy
right,
you
can
make
things
really
really
big
or
you
can
make
things
really
really
small,
so
so
right
like
if
it
were
basically
cheap
or
free
to
operate
an
API
server,
then
it
wouldn't
be
so
bad
to
to
run
one
per
user.
But
it's
not
so
you
could
consider
a
middle
ground
which
is
run
multiple,
multiple
clusters
that
has
the
additional,
like.
B
L
B
L
L
E
E
We
can
certainly
a
couple
would
get
an
order
of
magnitude,
improvement
and
efficiency
there,
but
I
think
they'd
be
about
the
bar
without
more
expensive
stuff,
I
mean
I,
don't
know,
I,
don't
disagree
that
some
form
of
sharding
would
be
useful.
I
would
want
to
leave
that
from
sharding
based
on
what
makes
organizational
tendencies
full
or
like
security
boundaries
like
the
difference
between
a
cluster
admin
and
every
other
user
on
the
cluster,
trying
to
think
through
those
cases
there,
but
there's
several
proposals
on
those
lines
like
yeah.
B
E
B
E
I
mean
I'm,
just
I'm
not
trying
to
be
condescending
or
anything.
It's
like
we're
running
really
big,
and
it's
not
even
our
main
problem,
like
our
problem,
is
none
of
the
controller's
deal
well
at
all
with
any
failure
modes
and
so
lightning
at
a
million
keys
like
these
are
the
big
data
bases,
but
the
problems
are
not
getting
to
two
million
or
three
million
or
4
million
keys.
The
problems
are
all
most
of
the
stuff
we
work.
We.
B
E
Ok
is
not
efficient
when
you
have,
and
in
while
we
hit
this
like
I,
I
kind
of
feel
like
the
Huawei
use
cases
and
some
of
ours,
like
another
thing,
they're
an
absolute
upper
bound,
but
they
kind
of
feel,
like
you
know,
there's
probably
like
six
or
seven
people
who
care
about
this
scale.
I
would
much
rather
spend
our
time
on
everybody
who
said
Oh
order
of
magnitude.
Smaller
scale
make
their
lives
easier,
yeah.
L
L
E
G
E
Damon
said
controller
like
it's,
some
of
the
edge
cases
for
failure
modes
like
if
a
particular
node
plagues
out
impacts
the
entire
Damon
SEC
controller.
If
a
particular
namespace
has
an
insane
request
and,
like
you
know,
you're
getting
air
back
offs
because
of
omission,
plugins
that
don't
allow
certain
things
going
through
the
controller
machinery
and
generalizing.
E
The
lessons
we've
learned
on
controllers
would
probably
benefit
anybody
who's
at
a
high
cardinality,
tennis
tenancy
and
we
haven't
spent
a
ton
time
like
you
know,
make
the
generic
control
or
the
work
that's
going
on
and
queue
builder
be
resilient
to.
If
a
particular
namespace
is
erroring
out.
Maybe
we
should
just
take
that
namespace
out
of
the
QB
for
awhile,
it's
stuff
like
that.
So
it's
not
that
it's
not
that
we're
100%
broken
it's
just
that
for
well
a
lot
of
what
we've
seen
the
mixed
use.
Clusters
tend
to
fail.
E
C
L
C
B
M
Okay,
so
kinky
me
yeah,
yes,
okay
great,
so
it's
about
recommendation,
so
I've
been
talking
with
with
Frederick
from
instrumentation
and
right
now
they,
the
IP
I
library,
here's
a
struct
which,
when
you
register
metrics
Prometheus,
you
can
define
an
endpoint
which,
when
you
send
a
delete,
request
to
tweet
it
basically
use
as
a
matrix
or
just
or
part
of
them
and
according
to
him
this
is
a
very
bad
anti
pattern
and
we
should
probably
get
rid
of
it.
And
my
thing
my
question
is:
if
we
should
get
rid
of
it,
tweet-tweet
booth.
E
Exists
was
for
six
scalability,
so
you
need
to
talk
to
six
scalability
high
scalability
had
an
inconsistent
in
point.
They
they
had
added
this
originally,
so
they
could
reset
metrics
during
runs,
and
then
we
moved
it
to
delete,
which
is
where
it
is
today.
But
then
in
general
this
was
a
six
scalability.
E
M
B
M
C
E
I
mean
this
use
case
was
only
because
of
something
that
six
scalability
testing
was
doing
in
their
testing
just
to
make
something
easier.
I'm
I!
Don't
really
think
that
this
is
a
requirement
for
us
they
for
their
use
case.
The
reason
it
was
put
in
they
could
probably
just
at
this
point,
parse
the
metrics
and
do
a
delta
calculations,
not
that
difficult.