►
Description
Extending Kubernetes: Our Journey & Roadmap [I] - Daniel Smith & Eric Tune, Google
What is the vision for Kubernetes Extensibility? Do you know the difference between initializers, cloud providers, and the CRI? In this talk we will describe how extension points in Kubernetes have evolved and go over the options today, and what they let you do. As we go over the extension points, we’ll give our vision for how they will evolve in the future, and talk about the sorts of things we expect the broader Kubernetes ecosystem to build out of them.
About Daniel Smith
Currently TL of Kubernetes’ API Machinery sub-team, Daniel has been working on Kubernetes since before it was open sourced, and contributed enough in the early days that he’s still one of the top contributors overall. Before that, Daniel worked on Google’s borg and AppEngine. He lives in Mountain View with his wife and two children.
About Eric Tune
Eric has worked on Kubernetes since before the first public release. He has contributed to Security, Tenancy, Application Controllers, Charts, Jobs, documentation, and more. Before Kubernetes, he worked on Google's Borg system, on datacenter-scale efficiency and performance measurement, and on web search.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
So
before
I
start
talking
about
extensibility,
let
me
address
who
I
think
cares
about
this,
so
cluster
operators
are
gonna
want
understand
the
components
that
are
deployed
in
their
clusters.
Even
if
they're,
you
know
not
writing
extensions
and
they
will
they
may
want
to
create
site
specific
policy
distribution.
Creators
want
to
know
how
to
integrate
with
infrastructure
as
a
service
cloud
providers
especially
will
want
to
integrate
with
their
own
infrastructure
as
a
service.
A
Hardware
and
software
vendors
may
want
to
add
new
api
is
to
kubernetes
or
expose
unique
storage
or
network
features.
Most
importantly,
if
you're
a
core
contributor
or
any
kind
of
kubernetes
user,
you're
gonna
benefit
from
stabilization
to
the
core
of
the
core
of
the
kubernetes
project.
So
right
now
we
have
a
lot
of
pressure
to
add
new
features
constantly,
and
yet
our
users
are
asking
us
for
a
more
stable
product.
Less
changes
released
notes
are
ridiculously
long.
A
I
tried
to
come
up
with
like
a
short
definition
of
what
it
means
to
be
be
using
extensibility
in
kubernetes
or
what's
an
extension
and
what's
not
an
extension
and
actually
couldn't
come
up
with
one
that
I
could
fit
in
a
sentence.
So
let
me
first
start
by
talking
about
what
kubernetes
is
and
and
what
it
is
not
what
isn't
extending
kubernetes,
so
kubernetes
is
open
source,
so
you
always
have
the
option
of
forking
it
and
feel
free.
That's
hard.
It's
projects
moving
really
fast.
A
The
codebase
is
really
big
and
it's
hard
to
figure
out
where
the
point
is.
You
want
to
extend
things,
there's
constant
refactoring
going
on
and
finally,
if
you're
a
lot
of
Kuen
a's
users
are
using
kubernetes
in
a
hosted
environment,
I
think
at
least
the
top
four
clouds
probably
have
hosted
kubernetes
I'm,
not
sure
exactly.
You
can
argue
over
exactly
what
the
ranking
is,
but
there's
a
lot
of
hosted.
Q&Amp;A
is
out
there
and
I,
don't
think
any
other
people
want
to
support
a
fork.
A
So
another
way
you
can
use
kubernetes
is
by
what
I
call
automating,
which
means
just
taking
this,
that
what
I
call
the
normal
api's
things
that
you
would
normally
post
resources
to
and
then
using
our
client
to
automate
that
process
once
you
have
a
you
know,
need
to
do
things
multiple
times.
We
have
clients
and
I
think
seven
different
languages.
So
that's
easy,
but
automation
doesn't
cover
everything.
It's
not
good
for
things,
it's
asynchronous,
so
it's
not
good.
A
When
you
need
to
hook
you
into
every
event,
make
sure
that
you
don't
miss
anything
block
things
before
they
happen.
I
can't
add
api's
and
it
can't
change
the
behavior
of
existing
api's.
So
when
you
need
to
be
synchronous,
we
need
to
add
api's
or
change
them
or
add
new
features.
Then
you
need
an
extension
now,
kubernetes
I!
Think
if
you
maybe
somebody
went
to
a
friend's
keynote.
A
This
morning
she
mentioned
there
are
like
12
different
extension
points
in
kubernetes.
Guess
what
I'm
gonna
exhaustively
list,
those
for
you
with
the
help
of
Daniel?
In
fact,
there's
so
many
ways
to
extend
that
it
can
be
a
bit
overwhelming.
So
part
of
this
is
kind
of
this
talk
is
going
to
give
you
a
guide
of,
maybe
which
ones
you
want,
how
to
combine
them
and
how
to
think
about
our
roadmap,
whether
you
should
sort
of
trust,
this
morass
of
extension
possibilities.
A
So
I
like
to
say
that
kubernetes
is
really
two
separate
things
to
me:
it's
an
abstraction
over
infrastructure
and
it,
but-
and
it's
also
a
framework
for
declarative,
API
and
distributed
control
and
those
are
actually
pretty
separate.
Like
a
lot
of
people
are
talking
coming
to
me
and
say:
hey
I
want
to
use
Cuban
a
style
api's,
but
actually
don't
need
pods
and
services
and
nodes,
so
I
think
in
the
year
or
two
we're
gonna
start
to
see
more
of
that,
that's
not
what
this
is
talk
is
about.
A
So
we
got
a
dozen
extension
mechanisms
and
for
me
it's
useful
to
categorize
them
into
two
rough
groups.
One
is
what
I
call
infrastructure
extension
extension
mechanisms
which
matches
my
first
definition
and
the
second
one
is
API
extension
mechanisms,
I'm,
Daniel,
I,
gonna
divide
this
up,
I'll
talk
about
infrastructure,
extensions
and
then
Daniel
dive
into
the
API.
So.
A
It's
December
for
me:
I'm
getting
excited
I'm
thinking
about
skiing.
Maybe
some
of
you
are
pretty
excited
too.
So
then
you
know
what
this
is.
I
didn't
hear
woods
from
everyone,
so
maybe
I
should
explain
this
for
a
second.
These
are
like
the
difficulty
codes
you
get
when
you
go
skiing
and
to
tell
you
how
hard
to
run
is
a
green
circle
means
you
know,
beginners
can
go
there.
A
Double
black
diamond
is
the
hardest
and
sort
of
in
this
order,
and
so
trans
I
tried
to
translate
the
Tsukuba
Nettie's
Green
met
like
I,
have
high
confidence
that
you
can
find
examples
documentation.
You
have
to
write
a
small
amount
of
code.
You
get
a
good
choice
of
languages
and
you're
unlikely
to
bork
your
cluster
by
trying
to
write
one
of
these
extensions
or,
if
you
do
you'll
be
able
to
understand.
What's
going
on,
it
doesn't
affect
everything
in
the
cluster.
It
only
affects
what
you're
extending
and
the
project
is
likely
to
extend
it.
A
So
it
says
port
support
this
extension
mechanism
going
forward
or
not
plenty
of
deprecated
or
anything
like
that.
So
the
negation
of
all
those
would
be
like
a
double
black
diamonds.
So
we're
gonna
see
all
these
ones
as
we
go
some
of
these
route,
or
some
of
these
runs-
are
gonna
get
groomed.
You
know
over
the
next
year,
so
they'll
get
easier,
but
for
now
this
is
the
current
state.
A
So
let
me
go
on
infrastructure.
Extensibility
I
put,
if
you
guys
are
the
type
of
person
who
likes
to
actually
like
get
way
more
data
while
I'm
talking
and
ignore
me
and
there's
a
URL
up
there
for
some
Docs
that
actually
just
merged.
They
give
a
lot
more
detail
about
all
the
extension
mechanisms
in
the
Cooper
days
on
Io,
Docs
and
I
will
I
see
a
people
taking
pictures.
That's
fine,
please
tweet,
whatever
I
promise
that
will
upload
the
current
PDS
afterwards,
so
you
can.
A
If
that's
what
you're
doing
you
can
relax
for
cameras,
so
I'll
start
with
storage
extension.
So
the
storage
extensions
allow
you
to:add
new
kinds
of
volumes
to
your
pods
things
that
get
mounted
as
a
POSIX
I'll
system.
This
is
not
intended
to
like
handle
API
based
storage.
Like
bucket
storage,
it's
meant
ins
handle
things
that
have
a
UNIX
file
file
API,
so
there's
actually
two
different
mechanisms.
Flex
volumes
came
first.
These
are
super
easier
right.
A
I
think
I
saw
someone
write
a
volume
plug-in
and
I
plug-in
by
the
way
means
a
binary
that
gets
called
out
by
the
couplet
or
another
component
of
the
kubernetes
system.
Someone
wrote
one
in
bash
with
like
100
lines,
including
a
really
long
copyright
editor,
so
I
thought
was
pretty
amazing.
The
plan
for
that
I
just
talked
to
sada
Lee
who's,
one
of
the
leads
in
cig
storage
and
he
said,
expect
flex
volume
support
to
stick
around
indefinitely,
but
we're
not
gonna
make
it
better.
A
But,
oh
to
me,
it's
super
easy,
so
I
think
that's
great.
This
new
thing
that
just
came
out
in
1.9
is
called
container
storage
interface.
It's
really
open.
Docker
may
sews
and
kubernetes
all
support
this
type
of
plug-in
it's
easier
to
upgrade
and
deploy
on
top
of
kubernetes.
Instead
of
having
to
like
go
into
your
node
file
system
and
like
upgrade
a
bash
script,
you
can
actually
deploy
the
CSI
plugins
as
a
daemon
set,
which
I
think
is
really
cool.
I,
don't
expect
this
one
to
stick
around
and
grow
its
alpha
in
1.9.
A
A
A
Device
plugins
that's
a
way
to
like
allow
discrete
hardware
resources
to
be
surfaced
from
your
cubelets,
so
things
I
could
go
an
iGoogle
for
this.
I
saw
people
talk
about
GPUs,
FPGAs,
QR,
ngz,
I,
hadn't,
even
heard
of
a
parent.
That's
like
a
random
number
generator
I
know
that's
for
like
financial
traders
or
or
something
I,
don't
know.
Does
anyone
know
that's
for
okay,
so
that
was
cool
like
it's
being
used
to
extend
resources,
I
hadn't,
even
heard
of
so
so.
This
is.
A
Let's
see
first
Network
plugins,
this
there's
an
open
standard
called
CNI
and
there's
like
a
couple
dozen,
these
plugins
and
they
I
think
they
started
with
docker,
but
we
had
to
tweak
it
a
little
bit
to
get
it
to
work
with
everywhere
with
kate's,
because
we
have
kubernetes
because
of
host
port,
so
I
mean
there's
a
couple
that
support
kubernetes,
fully
and
I
expect
to
see
more
of
those
over
time.
This
is
a
little
bit
trickier,
like
I
think
this
uses
G
RPC,
instead
of
being
a
binary,
plug-in
I
know
the
least
about
this.
A
So
I
should
stop
talking
all
right.
Oh
wow,
that
did
not
format,
so
those
diamonds
are
supposed
to
be
like
side
by
side,
but
so
this
one's
one
that
scares
me
the
most.
This
is
like
and
I
like
Double
Diamond
runs,
but
this
like
I
broke
my
leg
snowboarding.
This
is
like
this
is
the
scary
run
for
me
is
replacing
the
scheduler.
A
You
can
replace
the
scheduler
with
your
own
scheduler,
but,
like
I,
think
that'll
Bork,
your
cluster
personally
I
would
talk
to
the
sig
they're
doing
a
lot
of
refactoring
I
would
I
would
think
twice
before
doing
that.
The
easiest
thing,
although
it
didn't
render
on
the
slides
this
one's
called
scheduler
extender,
is
actually
the
easiest
one
to
do.
You
can
add
your
own
prioritization.
So
if
you're
like
oh
yeah,
those
machines
on
like
rack,
20
I
really
want
to
use
those,
because
I
can't
wait
for
them
to
burn
up.
A
A
So
there's
like
a
data,
encryption
key
and
a
key
encryption
key
and
like
right
now,
I
think
the
key
encryption
key
is
like
on
the
like
in
a
file,
but
the
plan
is
for
1.10
you'd
be
able
to
actually
put
that
key
encryption
key
in
like
a
kms
like
vault
and
that'll,
be
an
extension
where
it
can
like
call
out
to
your
kms
and
so
that'll
make
people
happier.
It's
that's
been
a
law
passed
for
for
a
long
time
and
I
talked
to
one
of
the
leads
on
that
and
said.
B
It's
a
work
better
felt
here.
Let's
talk
about
API
extensibility,
so
I
broke
down
a
little
okay,
you
can
sorta
see
it.
I
broke
down
a
little
spectrum
of
some
various
API
level.
Integration
extension
points
going
from
easiest
to
hardest.
We
have
CR
DS.
How
many
of
you
know
what
a
C
or
D
is?
Okay,
great,
most
of
you.
B
Next
we
have
a
C
or
D
and
you
can
put
a
schema
in
there
or
we
have
a
C
or
D
plus
a
schema
plus
a
validation
web
book.
That's
getting
a
little
bit
more
difficult
and
then,
finally,
if
you
really
want
a
challenge,
you
can
use
an
aggregated,
API
server.
All
of
these
are
mechanisms
for
adding
your
own
API
object
into
the
system.
B
So,
let's
talk
a
little
bit
about
those
in
more
detail
in
2018
I
expect
C
R
D's
to
go
to
GA
along
the
way.
I
expect
us
to
improve
completeness.
We
already
have
a
schema.
We're
gonna,
make
it
easy
to
make
validation.
Webhooks
we're
gonna,
make
it
easy
to
set
permissions
yeah,
so
so
it
should
integrate.
With
our
back
and
the
author
of
the
CR
D
should
be
able
to
tell
what
permissions
what
roles
doesn't
make
sense
to
operate
on
the
CRT.
B
We
shouldn't
force
the
cluster
administrator
to
understand
that
and
I
expect
to
improve
aspects
of
the
API
wide
consistency
like
we
have
a
status
pattern.
We
have
a
scale
pattern.
We
have
a
few
common
sub
resources
like
that
I
expect
to
add
mechanisms
to
CR
D
so
that
you
can
seamlessly
integrate
those
so
so
that
you
can.
We
can
have
HP
a
horizontal
pod,
autoscaler
work
on
your
CR
D,
so
you
can
scale
up
your
CR
D.
B
So
when
people
hear
about
CR
DS
and
they
hear
about
aggregated
API
servers,
the
question
is
invariably
how
do
I
know
which
one
to
do
what
with
or
if
you
look
at
a
C
or
D.
It's
pretty
easy.
You've
submitted
Yama
file
to
the
system.
It
gives
you
a
new
resource,
but
if
you
look
at
a
it
making
an
aggregated,
API
server,
it
doesn't
look
so
easy
because
you
have
to
compile
stuff
and
import
a
library
and
there's
a
API
server
builder
program,
but
you've
got
to
do
a
bunch
of
extra
steps.
B
So
why
do
we
have
both
I
thought?
It
might
be
useful
to
say
why
exactly
you
need.
You
would
ever
need
to
use
a
aggregated,
API
server
instead
of
a
C
or
D
and
the
the
really
there's.
Actually,
the
number
of
things
are
shrinking
over
time.
So
the
one
thing
that
I
expect
to
never
change
is
with
a
with
an
aggregated,
API
server.
B
You
can
completely
change
the
storage
mechanism,
so
if
you
are
doing
like
our
metrics
API
server,
if
you
are
fabricating
data
for
somebody
who
wants
to
watch
or
or
pull
an
aggregated
API
server
is
really
the
only
option
and
that's
the
the
metrics
is
high
volume
data.
It's
manufactured.
It's
it's
a
aggregated
together,
so
yeah
it
doesn't
make
sense
for
that
to
be
in
a
C
or
D,
something
that
is
true
today
and
will
be
less
true
in
the
future.
B
As
soon
as
1.9
goes
goes
out,
live
the
admission
chain
you
can,
if
you
haven't
aggregated
API
server,
you
can
put
you
know
whatever
random
business
logic
you
want
in
the
admission
chain
and
something
that
is
true
today
and
might
not
be
true
at
the
end
of
2018.
Is
that
the
aggregated,
API
server
allows
you
to
write
arbitrary
version
transformations.
B
B
We
had
someone
in
mind
extension
developers,
but
we
had
in
mind
our
selves
a
lot.
The
kubernetes
repo
is
very
large.
It
is
twisty
and
turny
and
full
of
passages
all
alike,
and
it
is
getting
to
be
difficult
to
change,
to
make
changes
that
don't
have
effects
that
ripple
out
through
the
entire
thing.
So
the
idea
is
that
we
split,
we
give
API
authors
their
own
binaries
and
they
have
there's
less
of
a
chance
of
them
interacting
poorly
with
another
component
yeah.
B
It
is
everything
between
the
authorization
check,
the
permissions
check
and
before
we
actually
commit
the
request.
So
everything
between
the
permissions
checking
the
storage
layer-
and
this
is
where
we
do
all-
of
the
all
the
fun
stuff
like
checking
quota
or
making
sure
that
you're
making
sure
that
your
pod
image
pool
policy
is
is
what
the
what
the
your
system
administrator
wants.
B
This
is
the
ideal
place
for
policy
enforcement.
There's
a
problem
it's
all
compiled
in.
So
if
you
wanted
to
use
this
before
1.9,
your
option
was
to
fork
api
server
and
write
a
compiled
in
admission
control
plugin
if
you've
contemplated
forking
api
server
you'll
be
very
happy
that
we
have
produced
web
hooks.
So
you
can
now
dynamically
register
a
web
hook.
The
api
you
have
to
implement
is
relatively
simple.
B
B
B
Another
option
for
a
similar
purpose
as
initializers
we've
sort
of
been
working
on
both
initializers
and
and
web
folks.
Initializers
are
in
alpha
and
at
some
point
in
the
course
of
2018.
We
expect
to
make
a
decision
about
whether
we
will
go
all-in
on
web
ebooks
or
if
there
is
a
place
for
initializers,
also
yeah.
B
Yeah,
so
what
else
are
we
gonna
do
in
2018,
I
expect
to
be
in
the
business
of
moving
flags
into
config
files
for
for
kube,
API
server,
and
ideally,
I
also
expect
to
move
config
files
into
api's,
where
it's
appropriate
III.
Don't
like
the
fact
that
to
change
things
about
API
server,
you
have
to
twiddle
with
the
flags
and
restart
it
right
now.
B
I
think
that's
very
inconvenient
for
cluster
operators
and
so
as
a
concrete
example
of
the
sort
of
thing
the
permissions
or
Ozzie
webhook
right
now,
it's
a
command
line
flag
and
it's
a
config
file
and
you
can
only
have
one
I
think
we
need
to.
Is
that
that's
that's
a
great
example
of
something
else
that
I
could
see
lots
of
people
wanting
wanting
to
add
the
difference
between
the
permissions
check,
webhook
and
the
admission
check
webhook.
B
Is
that
the
admission
check
any
one
of
them
can
fail
the
request
and
you
can
also
modify
a
request
with
the
admission
of
a
book,
but
for
permissions.
It
only
takes
a
guess.
You
can
now
they've
they've
they
fixed
it.
You
can
now
deny
a
request,
but
it
must
pass
everybody
yeah
and
my
goal.
My
ultimate
goal
is
fully
portable
extensions.
You
write
an
extension
on
that
runs
on
one
cluster.
You
should
be
able
to
move
it
to
another
cluster.
Just
like
the
the
goal
of
kubernetes
is
portable
workloads.
I
think
your
your!
B
A
That's
like
we
call
the
operator
pattern.
You've
probably
heard
that,
especially
if
it's
like
running
and
applicate
like
a
database
or
something
we
call
that
operator
pattern,
and
then
Daniel
mentioned
that
when
you
can
use
the
mission
webhooks
+
CR
DS,
to
get
better
validation
on
them.
I'm
really
curious
to
see
what
sort
of
interesting
combinations
people
will
come
up
with
like
combine.
A
Like
a
storage
plugin
with
you
know
an
aussie
plugin
or
something
I
don't
know
so,
look
next
year
will
tell
so
Daniel
talked
a
little
bit
we're
heading
I
want
also
share
my
aspirations
for
the
next
year.
I
want
to
see
automatic,
rich
seat
like
if
you
write
a
CRT
or
any
kind
of
customer
source,
and
it's
just
automatically
in
the
CLI
and
the
GUI
GUI.
Give
you
a
good
experience.
A
A
So
that's
somewhat
aspirational,
but
that's
the
direction
we
want
a
head
also
want
like
right
now
you
can
build
customer
sources
on
top
of
the
platform,
but
you
can't
build
take
existing
things
and
have
them
come
on
top
of
the
customer
sources
as
well.
Daniel
mentioned
this.
Adding
the
scale
resource
will
allow
you
to
use
HPA.
Also,
let's
use
disruption
budgets
which
I'm
talking
about
tomorrow.
A
A
So
please
find
me
if
you
have
ideas
on
how
to
do
that
and
the
Daniell
suggested
like
a
cluster
introspection,
90
API,
so
one
thing
we
do
in
gooeys
is
when
you
create
a
typical
pattern
with
like
a
customer
source,
is
that
it
makes
more
resources
and
those
resources
make
more
resources,
but
as
a
viewer,
you
typically
only
want
to
see
the
top-level
thing
that
you've
created,
except
when
it
breaks.
Then
you
wanna,
debug
and
expand
it.
A
So
I
want
this
to
be
a
pervasive
pattern
in
you
eyes
where,
like
you,
just
see
the
top-level
resource,
but
then,
if
you'd
want
to
drill
down,
you
can
find
the
things
and
if
you
see
an
error
message
from
a
pond,
you
can
tie
that
back
up
to
its
parent
and
the
API
server
actually
has
all
that
information
in
a
graph
used
by
the
garbage,
collector
but
I
think
there's
a
lot
more.
We
could
do
with
that.
So
that's
that's
one
more
thing.
A
You
could
see
that
from
the
number
extension
points
we
have
the
number
of
talks
about
extensibility
keynotes
that
mentioned
it
completing
these
is
a
large
part
of
the
work
for
several
SIG's
in
the
next
year.
You
can
also
see
a
commitment
to
open
standards
where
they
apply.
For
example,
CSI
and
CNI
are
used
by
other
orchestrators
and
CR
DS
use,
JSON,
schema
and
I'm
very
interested
in
more
opportunities
to
reuse,
open
standards
with
kubernetes,
especially
extensibility,
and
without.
C
The
question
was:
if
we
have
to
use
communities
for
orchestrating
non
containerized
services
along
with
container
service,
do
we
have
to
write
our
own
Orkut
and
our
own
extensions?
Is
that
a
pattern
at
all
that
you've
seen
Kafka
and
others
that
are
not
contained
raised?
But
there
are
the
containers
services
so.
A
The
pattern
I
see
is
some
people
are
starting
to
use
kubernetes
style
api's,
maybe
even
using
crts
and
an
operator
to
manage
resources
that
actually
don't
have
pods
under
them.
So
it
could
be
like
serverless,
which
might
be
on
top
of
kubernetes
or
might
not
I.
Think
Red.
Hat
openshift
has
an
extension,
allows
you
to
manage
your
VMs
using
a
custom
resource,
so
I
would
say
it's
finding
a
custom
resource
to
manage
infrastructure.
That's
not
containers,
hey
I'm,
happy
to
follow
up
on
that
and.
B
A
D
Had
first
of
all,
I
won't
put
my
vote
on
initializers
yeah
initializers
second
it'd
be
super
helpful
when
I
want
to
use
an
initializer
I'm
a
bit
concerned,
putting
it
on
some
of
the
core
resources
because,
like
I,
want
to
put
it
on
pods,
but
it
should
be
kind
of
cool.
In
my
mind,
you
have
initializers
target
based
on
labels.
So
it's
like
oh
yeah.
B
B
F
E
A
There
wasn't
complete
agreement
on
exactly
what
the
pod
preset
API
should
be,
so,
instead
of
putting
it
in
core
decision
for
now
is
to
put
it
in
the
service.
Catalogs
resource,
api
groups
and
I
think
the
intent
is
to
use
in
a
future
version
to
use
the
mutating
web
hooks
has
what
service
catalog
builds
the
pod
preset
on
top
of,
and
then
they're
gonna
run
with
that
for
a
while
and
get
feedback
on
with
an
API
is
good,
and
then
it
can
be
in
a
conversation
whether
that
needs
to
pull
in
the
core
or
what.
A
A
So
the
cloud
provider
manages
like
IP
address
ranges
for
services,
I,
think
it
might
help
with
ingresses.
It
manages
like
the
lifecycle
of
nodes
and
detecting
one
like
the
the
backing
instance
of
a
node
has
disappeared.
It
probably
some
other
stuff.
I,
don't
know
anymore,
so
it's
possible
that
might
get
broken
up
into
more
like
individual
web
hooks.
If
there's
a
demand
for
that,
but
right
now
the
low-hanging
fruit
is
just
getting
it
to
be
a
separate
release,
train
and
I.
Think
we'll
just
let
that
soak
for
a
while
before
we
decide.
A
F
A
Those
are
my
personal
opinions
and
not
just
that
run
looks
scary
to
me.
Maybe
you're,
like
an
expert
skier
you've
already
been
doubted.
It's
not
that
bad,
but
I
do
think
that
there's
I
think
with
dual
schedulers.
It
will,
if
you
go
too
far
down
the
dual
scheduler
path,
it'll
limit
the
ability
in
the
future
for
us
to
add
better,
like
resource
sharing
primitives
that
we
would
like
to
add
like
a
year
from
now
so
I
think
at
some
point
that
might
get
broken,
but
maybe
it
solves
your
media
need.
We.
G
I
actually
share
everything
that
Eric
said.
So,
as
we
were
going
forward,
it
seems
harder
and
harder
to
support
other
schedulers,
especially
competing
schedulers,
but
which
is
sometimes
the
case
in
some
of
the
clusters.
So
people
run
multiple,
the
schedulers
at
the
same
time,
but
as
scheduler
is
advancing
and
adding
more
features,
it
becomes
harder
and
harder
to
keep
that
promise
that
we
are
completely
scheduler
agnostic
feel
free
to
replace
our
skills,
and
that
was
that
was
sort
of
like
the
promise
initially,
but
it
becomes
harder
and
harder.
So
it's
pitiful.
G
So
basically
nobody
prevents
you
from
replacing
this
scheduler
but
you're
sort
of
like
giving
you
a
heads
up.
I,
don't
want
to
use
warning
but
you're,
giving
you
a
heads
up
that
it
may
become
harder
and
harder
and
maintaining
a
custom
scheduler,
which
completely
replaces
the
schedule
may
become
very
hard.
So
what
we
encourage
people
is
to
keep
running
a
default
schedule.
A
default
schedule
is
very
configurable.
You
can
pretty
much
disable
everything
that
it
does
almost
and
by
everything
I
mean
all
the
predicates
and
priorities
that
it
supports
and
it's
pluggable.
G
A
Yeah
I
might
summarize
it
as
like,
like
a
go
package,
might
export
an
interface
and
they're
committing
to
like
not
adding
new
functions
or
changing
the
signatures
that
interface
and
like
web
hooks.
When
that
becomes
GA
like
that'll,
be
we're
committed
to
not
changing
that
and
we
already
don't
think
we're
gonna
change
it
a
lot.
The
scheduler
does
not
have
a
well-defined
interface,
it
has
a
hundred
responsibilities
and,
we
might
add
50
more
in
the
next
year.
So
you
you
you'll
be
running
to
keep
up
is
that
yeah.