►
Description
Don’t miss out! Join us at our upcoming hybrid event: KubeCon + CloudNativeCon North America 2022 from October 24-28 in Detroit (and online!). Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Building a Nodeless Kubernetes Platform - William Denniss, Google Cloud
Can Kubernetes be nodeless, and is nodeless Kubernetes still really Kubernetes? That is the question the Google Kubernetes Engine team asked themselves before embarking on a project to create a new operating mode for their 6-year-old Kubernetes platform, GKE. Go behind the scenes of the creation of GKE Autopilot, a fully managed "nodeless" Kubernetes platform, and hear from one of the creators how it was built, and why various decision decisions were made. Topics covered will include the trade off between operational support, security and Kubernetes compatibility, and why it actually makes sense for nodes to retain their semantic meaning on a nodeless Kubernetes platform. The community has seen several approaches for offering serverless/nodeless Kubernetes to operators, including with technology like Virtual Kubelet. This presentation will break new ground, providing an alternative path to achieving the goals of serverless Kubernetes without removing node-level APIs like affinity.
A
First,
a
little
background
about
me:
I'm
a
product
manager
in
google
cloud
where
I
work
on
kubernetes
engine
in
2019,
I
co-founded
gke
autopilot
and
that's
what
I
currently
work
on.
This
is
a
new
mode
of
operation
for
gke,
which
I'll
be
talking
a
bit
about
today,
I'm
a
very
big
supporter
of
the
kubernetes
open
source
project
and
in
2017
I
also
co-founded
the
certified
kubernetes
conformance
program,
which
is
still
used
today
to
ensure
portability
between
vendors
distributions
of
kubernetes.
A
Firstly,
let
me
do
a
quick
audience
poll,
so
I'd
like
to
know,
raise
your
hand
who
believes
that
it's
possible
to
have
a
noble
kubernetes,
where
there
is
still
technically
a
node
object,
show
hands,
got
a
couple,
not
too
many
who
doesn't
everyone
else,
yeah
and
third
option
who
believes
I
was
just
adding
a
bit
of
controversy
to
my
kubecon.
Talk
to
you
know,
spruce
up
the
marketing
a
bit
all
right.
A
Well,
the
truth
probably
lies
somewhere
in
the
middle
of
of
of
those
three,
but
let's,
let's
dig
in
so
what
does
you
know
as
we
look
at
building
a
fully
managed
kubernetes
platform,
I
think
it's
worth
taking
a
step
back
and
looking
at
a
traditional
kubernetes
platform
and
I'll
be
using
gke
as
the
example
here.
So
a
traditional
kubernetes
platform
basically
consists
of
two
apis
that
you
need
to
interact
with
as
the
developer.
The
first
one
above
is
the
kubernetes
api.
That's
what
you're
here
for
right!
A
A
I
believe
that
a
an
ideal,
fully
managed
kubernetes
platform
would
just
be
the
kubernetes
api
where
you
interact.
Everything
at
that
level
using
you
know,
cube
ctl
using
yaml
files.
Basically,
so
what
does
it?
What
does
it
mean
to
build
a
fully
managed
kubernetes
platform?
Well,
when
I
was
looking
at
this
problem
in
2018,
I
wrote
a
position
paper
and
I
had
three
headings.
Let
me
just
quickly
share
them
here.
The
first
one
was
that
a
nodeless
or
a
fully
managed
kubernetes
platform
should
still
be
kubernetes.
A
Secondly,
the
containers
should
be
out
of
utilize
unused
reserved
capacity,
in
other
words,
allow
for
bursting
people
come
to
kubernetes.
One
of
the
reasons
they
come
to
kubernetes
is
to
be
able
to
pull
their
resources
and
and
potentially
burst
when
when
needed,
and
the
third
thing
is,
I
wanted
to
make
sure
that
we
priced
this
in
such
a
way
that
it
supported
continued
usage.
A
I
didn't
like
the
idea
of
creating
like
a
kind
of
a
toy
version
or
something
that
you
wouldn't
want
to
run
100
for
100
percent
of
your
workloads,
100
of
the
time
and
on
node
visibility.
I
had
a
suggestion
just
to
make
them
visible,
while
still
hiding
certain
bits,
like
maybe
the
vm
I'll,
be
digging
into
that
in
a
bit.
A
So
back
to
that
first
point,
though,
the
first
point
being
that
nodeless,
kubernetes
or
fully
managed
kubernetes
should
still
be
kubernetes.
Why
is
that
important?
So
when
we
approach
this
problem
of,
let's
build
a
simpler
to
operate
as
simpler
to
use
kubernetes
a
lot
of
people
came
to
me
and
said
well,
william
kubernetes
is
hard.
There's
there's
a
lot.
You
have
to
learn
to
get
started,
there's
a
lot.
A
These
are
professionals
like
they
might
be
running
a
massive.
You
know
website
for
a
fortune,
500
company,
for
example,
and
they're
a
professional
right.
They
they
need,
they
need
power,
they
need
flexibility.
You
know
one
day,
someone
might
approach
them
and
say
hey.
I
need
to
run
a
reddit
database
in
this,
or
I
have
some
like
legacy
state
for
workload
that
you
have
to
you
have
to
run.
Kubernetes
can
handle
that.
So
I
believe
that
the
the
power
behind
kubernetes
is
is.
A
A
Now,
when
the
team
looked
at
this,
we
had
a
couple
of
different
operational
models
to
consider
where,
when
thinking
about
like
how
to
actually
run
pods,
how
to
run
pods
on
this
fully
managed
product-
and
maybe
I'll-
take
a
pause
there
and
just
say
that
what
this
product
does
basically
what
our?
What
our
aim
was
is
that
you
just
create
a
kubernetes
workload
and
you
don't
have
to
configure
nodes.
You
don't
have
to
manage
anything
else,
will
provision
all
the
infrastructure
for
you.
A
The
first
one
was
actually
to
use
borg.
So
I
don't
know
how
many
people
are
familiar
with
borg,
it's
it's
a
container
orchestrator
developed
by
google
and
used
internally
and
was
the
subject
of
an
academic
paper
which
you
can
find
online
and
it
actually
formed
the
inspiration
for
kubernetes
itself
many
years
ago.
A
A
You
wouldn't
be
able
to
do
something
like
a
you
know,
5.25
cpu
pod,
on
this
system,
because
we
don't
have
a
vm
of
that
of
that
size.
The
final
option
was
just
be
like
gke.
The
existing
product
that
we're
looking
at
just
be
like
gke
standard,
run,
run
multiple
pods
per
node
and
yeah.
Basically,
don't
change
it
too
much.
A
A
So,
in
the
end,
the
team
chose
to
make
it
just
like
gke
one
of
the
other
nice
things
about.
This
is
that
we
can
offer
a.
We
were
able
to
offer
like
a
wide
range
of
pod
sizes,
scaling
from
a
quarter
of
a
core
all
the
way
to
28
cores
in
quad
core
increments,
so
we'd
be
able
to
support
something
like
a
17.75.
A
A
If
we're
trying
to
create
a
fully
managed
kubernetes
platform,
one,
that's
so
called
nodeless
like
what
does
it
mean
when
you
go
cube,
ctl
get
nodes,
should
it
list
all
the
nodes
that
we
actually
have
there?
Should
we?
You
know
just
group
that
together
in
one
and
just
return,
you
know
autopilot,
node
and
just
sort
of
hide
all
that
detail.
A
In
the
end,
we
went
with
just
list
the
nodes
be
transparent,
show
the
user
what's
actually
happening
under
the
hood,
even
if
they
most
of
the
time
don't
have
to
care
about
it.
Second
second
decision
we
looked
at
was
what
about
the
actual
like
inner
workings
of
those
nodes?
Should
we
be
transparent
about
what
what
shape
those
nodes
are?
Should
we
tell
you
if
it's
like
an
intel
or
an
amd,
or
this
this
type
of
machine
or
that
type
of
machine?
Should
we
disclose?
You
know
how
much
of
the
allocatable
table
is.
A
A
The
reason
is
that
we
want
you
to
be
a
we.
We
want
you
to
trust
us
that
we're
going
to
do
the
right
thing
with
your
pods
when
you
schedule
them
in
in
this
platform,
but
we're
going
to
let
you
verify
what
we
did
as
well.
So
if
you
want
to
poke
under
the
covers
and
and
see
exactly
how
things
landed,
it's
all
there
fully
transparent,
I
don't
think
we
hired.
We
literally
don't
hide
a
single
field
compared
to
the
gkd
standard
product.
A
The
other
question
was:
can
users
access
the
vm
object
separately
outside
of
kubernetes?
So
at
this
point
it's
probably
worth
mentioning.
Actually
I
showed
a
diagram
before
that
developers
used
two
apis
to
interact
with
this
system.
It
turns
out,
there's
actually
a
third
one
right
at
the
bottom
right,
which
is
the
vm
api.
A
The
reason
I
didn't
show
that
before
is
typically
people
using
kubernetes,
don't
have
to
care
about
the
vm
api.
It's
100
managed
for
you,
but
it
happens
to
be
there
and
you
can
interact
with
it
like,
for
example,
you
can
ssh
into
a
node.
So
going
back
one
of
the
one
of
the
problems
we
had
there
was,
you
know:
should
we
allow
access
to
that
object
decision
was
no.
In
this
case.
A
The
way
gke
works
and
and
the
platform
that
we
were
dealing
with
is
that
nodes
of
the
same
configuration
are
grouped
into
a
semantic
grouping
called
a
node
pool.
So,
for
example,
up
on
screen
there.
I
have
a
node
pool
with
eight
vcpu
cores
and
16
gigabytes
of
memory
and
there's
another
one
with
four
and
four,
and
so
what
happens
is
if
there's
existing
space
in
one
of
those
nodes,
the
pod
will
just
get
placed.
But
what
about
when
there
isn't.
A
There's
a
component
called
the
cluster
autoscaler,
which
will
actually
look
at
existing
node
pools
and
extend
that
node
pool
to
include
to
add
an
extra
node
that
can
that
can
handle
that
pod
and
therefore
that
pod
will
be
scheduled
in
the
case
of
the
pod
that
didn't
fit
in
any
of
the
existing
node
pull
definitions.
We
have
a.
A
A
And
how
this
system
works
and
and
how
it
actually
actuates
on
the
on
the
user
input
is
because
we've
pretty
much
eliminated
the
actual
like
node
api
and
the
no
pull
api.
So
there's
no
way
for
users
to
specify
those
things.
We
derive
everything
from
the
pod
spec.
So
one
of
the
simple
examples
is
the
the
resources
needed,
so
things
like
the
cpu
and
the
memory
we
derive
that
from
the
resource
request
of
the
pod.
A
A
That
was
a
spot,
node
pull,
and
then
you
would
target
that
spot
notable
with
your
pods
with
autopilot
we
flipped
the
script
there
and
you
actually
specify
the
requirement
just
in
the
pod
spec,
and
we
will
actuate
on
that
and
provision
a
node
that
can
then
handle
that
pod
spec,
which
I
think
is
actually
really
nice
design,
because
it
means
that
all
of
the
configuration
of
the
hardware,
essentially
right
of
like
hardware
properties
like
this,
should
be
a
spot
node
and
in
the
future,
potentially
other
things
like
you
know.
This
pod
needs
a
gpu.
A
A
A
Another
aspect
of
the
implementation
of
this
platform
was
an
admission
webhook,
so
we
had
to
achieve
two
things
here.
One
is
that
I
mentioned
that
we
have
a
we
built
this
system
to
have
a
very
wide
range
of
resources
for
the
pods,
but
there
were
still
some
limits
you
need
to
the
pod.
Cpu
needs
to
be
between
a
quarter
of
a
core
and
28
cores
and
there's
a
ratio
of
cpu
to
memory.
A
A
What
that
means
is
we
can't
really
offer
you
the
users
direct
access
to
those
nodes
at
a
root
level,
because
then
people
can
potentially
go
in
modify
the
kernel,
change,
bits
and
pieces,
and-
and
we
essentially
just
lose
the
confidence
to
actually
manage
that
thing
for
you,
because
we
don't
know:
what's
happened
to
the
node,
so
it's
important
for
us
that
all
the
nodes,
kind
of
look
the
same,
or
at
least
you
know,
have
very
well
known
properties.
So
we
have
an
emission
controller
there
to
enforce
those
policies.
A
A
A
Interestingly,
though
many
linux
capabilities
are
actually
still
offered
in
this
product,
so
p-trace
was
actually
requested
by
one
of
our
security
partners.
They
wanted
to
be
able
to
use
p-trace
to
inspect
running
running
processes.
We
looked
at
that
we
felt
like
it
was
actually
quite
fun
to
offer.
So
we
we
added
that
into
the
list
that
you
can
actually
use
other
things.
We
clamp
down
on
stuff
that,
like
directly
relates
to
the
node.
A
So
we
had
to
limit
that
limited
host
networking,
which
is
also
fairly
fairly
highly
privileged
host
path,
where
we've
also
restricted,
although
you
can
have
you
can
mount
var
logs
in
in
read-only
mode,
which
means
that
you
can
actually,
as
a
user,
just
have
like
a
damon
set
like
scraping
logs.
That's
totally
fine.
A
A
One
thing
we
didn't
restrict
was
the
ability
to
run
a
container
as
the
root
user.
You
can
yourself
restrict
that
using
the
the
open
source
pod
security
mission,
we
didn't
actually
need
to
restrict
that,
because
the
the
security
boundary
of
this
product
is
still
the
vm.
It's
not
actually
a
multi-tenant
system
at
all
the
vms
are
still
100
percent.
A
So
so
far
as
I've
been
talking
about
this
implementation,
one
of
the
interesting
things
is
you
could
have
actually
done
every
single
thing.
I've
described
yourself,
you
can
use
the
cluster
auto
scaler.
You
can
use
the
node
order.
Provisioner
component,
you
can
write
your
own
mutating
web
hook,
admission
controller.
You
can
do
all
that
you
can
literally
build
exactly
what
I
just
described
yourselves
today
on.
You
know,
gk
standard.
So
if
that's
the
case
like,
why
do
we
even
need
this
other
product?
A
Aside
from
the
fact
that
it's
a
bit
challenging
creating
these
mutating
web
hooks
and
so
on?
Well,
obviously,
one
of
the
benefits
of
us
doing
it
is
that
it's
all
pre-configured
in
a
nice
package,
but
that
alone
is
probably
not
enough.
There's
a
couple
of
things
that
we
add
with
this
product
that
is
sort
of
hard
to
do
yourselves
and-
and
that
is
the
billing
model-
is
different.
It's
it's
request-based!
So
we
we
charge
based
on
the
pod
request
rather
than
the
nodes.
Obviously,
that's
not
something
that
you
can
change
as
a
user.
A
Essentially
that's
kind
of
the
the
bargain
that
you
have
when
you
use
this
product,
if
you're
willing
to
give
up
that
little
bit
of
control
about
not
being
able
to
modify
the
node,
which
I
think
hopefully
for
most
workloads,
is
totally
fine
to
give
up
you,
don't
you
know
if
you're
running
mariadb,
you
shouldn't
shouldn't,
have
root
access?
What
you
get
in
return
is
a
more
fully
managed
system
with
with
us
being
the
sre.
A
The
other
thing
is
that
we,
as
I
mentioned,
we
eliminated
completely
the
the
vm
api.
So
there
is
no
visibility
on
these
vms,
the
vms,
actually,
still
there
they're
exactly
the
same.
They
have
a
different
prefix.
So
if
you,
if
you
look
at,
if
you
look
at
the
cube,
ctl
get
node
output,
you
might
notice
that
the
autopilot
node
has
the
prefix
gk3.
Instead
of
gke
the
node,
the
the
virtual
machines
are
actually
still
there,
they're
still
there
in
the
product,
but
the
the
api
is
removed.
A
A
A
So
at
the
beginning,
I
showed
this
diagram
of
the
the
two
apis.
Essentially,
users
have
to
interact
with
in
order
to
get
kubernetes
in
order
to
use
kubernetes,
and
even
this
diagram
with
that
extra
vm
api
at
the
bottom,
which
you
typically
don't
have
to
use,
but
it's
there
and
you
might
have
to
care
that
it's
there
so
with
autopilot,
what
we're
able
to
achieve
is
we
essentially
shrank
the
entire
gke
api
surface
area
down
to
one
command
which
is
create
you
just
create
it.
You
connect
it
to
cube
ctl
from
that
point
on.
A
You
are
100
just
interacting
using
the
kubernetes
api,
so
it's
really
a
kind
of
a
I
like
to
think
of
it.
As
like
a
very
pure.
You
know,
kubernetes
platform,
the
api
you're
using
on
this
platform
is
just
the
kubernetes
api.
There's,
there's
no
node
pool
api.
You
don't
have
to
configure
scaling,
auto
scaling
anything
like
that.
You
just
interact
with
it
through
kubernetes
from
a
ui
standpoint.
We
you
know,
obviously
the
ui
representation
of
that
of
that
api
is
also
pretty
nice.
A
A
Now
for
the
for
the
meat
of
this
talk,
I
guess
what
are
the
benefits
realized
from
all
this
from
this
design
of
you
know,
building
this
fully
managed
platform,
which
actually
looks
a
lot
like
the
traditional
gke
right
it
has.
It
has,
you
know,
pretty
much
the
same
nodes
under
the
hood,
the
same,
multiple
pods
per
node,
a
lot
of
the
same
things
other
than
the
fact
that
that
you
know
potentially
made
it
a
little
bit
faster
for
us
to
build
like
what's
the
benefit
to
the
user?
A
I
think
the
first
benefit
is
that
it
enables
really
granular
pod
sizes
because
we're
bin
packing
pods
onto
machines.
We
don't
have
to
shoehorn
the
pods
into
vm
sizes,
so
I
did
cover
that
earlier.
But
you
know
you
can
basically
create
like
a
you
know:
21.25
core,
pod
and
and
and
just
slot
it
in
there
we'll
run
that
just
fine
and
then
you
know
you
can
add
a
quarter
of
a
core
another
one
called
whatever
whatever
you
want
to
do.
It's
it's
very,
very
flexible
and
I
think
that
kind
of
matches
what
users
want.
A
Other
nice
things
about,
keeping
the
the
node
object
as
a
or
the
kind
of
the
node
scheduling
concept
in
this
design
is
that
things
like
pod
affinity
and
andy
affinity
continue
to
work.
These
are
important
concepts
that
come
from
the
kubernetes
api
right
and
if
you
remember
back
to
my
original
proposal,
I
really
wanted
this
to
look
like
and
be
a
fully
capable
kubernetes
platform.
A
Well,
it's
not
fully
capable
if
you
eliminate
things
like
pod
affinity
and
infinity,
pod
affinity
might
be
used.
For
example,
if
you
have
like
a
front
end
pond
and
a
back
end
pod,
and
you
want
to
say,
I
want
these
pods
to
always
be
together
on
the
node.
Well,
it's
hard
to
do
that.
If
you
don't
have
nodes
in
a
fully
nervous
system,
you
can't
do
that.
A
This
actually
works
quite
fine
with
our
design.
We
will
ensure
that
that
constraint
is
satisfied,
similar
story
with
node
affinity.
So
we,
you
know,
we
offer
zonal
affinity.
So
if
you
have
a
zonal
resource-
and
you
want
your
pods
in
that
same
zone,
you
can
use
that
one.
One
slight,
I
guess
drawback
of
this
system
is
occasionally
you
might
want
to
separate
workloads
in
in
a
one
pod
per
node
system.
You
don't
have
to
separate
them
because
they're
always
separated,
but
for
that
you
can
actually
just
use
the
kubernetes
constructs
of
tolerations
and
node
selection.
A
So
again
the
kubernetes
api
already
has
the
you
know
the
language,
the
syntax
to
describe
these
things,
and-
and
we
can
honor
that
in
this
managed
product
that
still
has
notes,
pod
spread,
topologies
work
and
the
last
one
here
is
daemon
sets.
Daemon
sets
is
often
under
often
overlooked.
I
think
when
it
comes
to
fully
managed
platforms,
but
they're
actually
really
important
daemon
sets
don't
make
much
sense.
A
If
you
only
have
one
pod
per
node,
because
the
whole
point
is
you
want
to
run
an
agent
on
the
node,
the
good
thing
is
with
this.
With
this
proposal
with
this
design,
you
can
still
have
a
name
instead,
because
there
are
still
nodes
with
one
catch
and
the
catch
is
that
daemon
sets
are
typically
used
to
actually
modify
the
node.
So
how
does
that?
How
does
that
work
in
this
system?
A
The
compromise
we
reached
here
is
that
we
looked
at
well-known
products
and
solutions
out
there
in
the
community
out
there
you
know
commercially
available
and
we
decided
to
allow
list
specific
solutions,
so
they
could
continue
to
work
on
autopilot,
so
many
security,
logging
and
monitoring
solutions
that
are
commercially
available.
Like
all
the
people
you
see
out
there
on
the
on
the
booth
floor,
most
of
those
solutions
still
work
and
if
they
don't
work,
they
can
probably
come
and
chat
to
me
and
we
can
make
it
work.
A
The
reason
we
could
do
that
is,
as
I
said
earlier,
the
security
boundary
for
this
product
remains
the
vm.
So,
even
though,
by
allowing
listing
certain
partner
workloads
to
have
elevated
privileges,
the
security
boundary
remains
the
same.
It's
still
a
vm.
This
is
not
a
multi-tenant
product,
so
we
didn't
have
that
concern
the
main
concern
when
we
were
building
this
from
from,
like
a
design
perspective,
was
we
had
to
make
sure
those
nodes
are
supportable?
We
didn't
we
didn't.
A
We
didn't
just
want
to
like
open
the
flood
gates
and
let
all
kind
of
node
modifications
happen.
But
if
it's
a
known
set-
and
you
know
professionally
supported
software,
we
felt
like
that
was
safe
enough
to
offer
from
a
supportability
point
of
view,
so
it
still
allows
damages,
and
I
think
that's
important
because
you
know
if
you
have
a
workload
that
you're
running
on-prem
or
you're
running
you
know
on
another
cloud
provider
or
on
gk
standard,
and
you
want
to
migrate
it
around.
A
You
probably
have
in
fact
most
of
our
customers
have
at
least
one.
You
know:
damage
set
workload.
They
just
want
to
run.
If
we
didn't
offer
this,
essentially,
you
would
have
to
go
in
and
add
that,
whether
it's
security
or
logging
or
whatever
you
would
have
to
add
that
functionality
to
every
single
pod,
which
I
think
is
a
really
really
big
burden
on
developers.
So
that
is
actually
one
of
the
really
overlooked
benefits.
A
I
I
think
of
of
still
supporting
nodes
and
still
offering
a
multi-node,
a
multi-pod
per
node
system,
other
benefits
and-
and
this
mostly
kind
of
helps
us
as
we
you
know,
work
on
the
product
further.
But
other
benefits
of
this
design
are
the
fact
that
the
infrastructure
features
it's
very
similar
to
gke.
So
a
lot
of
things
just
work.
One
really
good
example
is
stateful
set
so
right
out
of
the
gate.
A
A
Hardware
like
gpu
local
ssd
again,
because
these
are
just
regular
vms
under
the
hood.
It's
a
lot
simpler
for
the
team
to
add
support
for
those
features
and
finally,
it's
possible
to
offer
burstable
class
pods.
So
I
mentioned
this
right,
the
very
beginning
as
one
of
my
three
kind
of
objectives,
and
that
was
the
ability
to
utilize
unused
capacity
in
the
class
to
bipods.
Why
do
I
think
this
is
important?
A
A
Now,
if
a
request
comes
in
to
one
of
those
pods,
you
ideally
don't
want
to
constrain
it
to
just
the
resources
that
that
pod
requested
now,
certainly
that
pod
needs
to
be
able
to
handle
that
request
within
whatever
slo
you
have
with
the
resources
it
requested.
But
wouldn't
it
be
nice
if
opportunistically
it
could
just
burst,
use
the
capacity
you're
already
paying
for
on
the
node
and
serve
that
request.
A
little
bit
faster
delight,
the
user
a
little
bit
more.
I
feel,
like
that's
a
really
important
feature,
and
it
is
possible
with
this
design.
A
It's
really
only
possible
with
this
design,
because
if
you
don't
have
multiple
pods
on
the
node,
if
you
don't
have
this
concept
of
the
node,
then
you
can't
really
pull
that
capacity
for
bursting.
We
don't
actually
offer
this
yet
in
in
the
product
that
we
built,
but
it's
something
that
we're
looking
at
and
I'd
like
to
share
a
quick
kind
of
design
of
how
this
might
work.
A
So,
if
you
look
at
guaranteed
class
pods
today,
the
cpu
requests
equal
the
limits,
which
means
there's
no
bursting
traditionally
in
kubernetes
you
could.
You
could
set
a
much
higher
limit
than
the
request
and
thus
create
a
burstable
pod
that
can
scale
up
when
there's
extra
resources.
The
problem
with
the
design
that
we
came
up
with
is
take
this
example
of
three
pods
that
totally
have
requested
five
and
a
half
cores
and
that
are
running
on
an
eight
core
machine.
The
problem
is
in
a
traditional
kubernetes
model.
A
Those
pods
would
be
able
to
use
up
to
eight
cores,
which
could
be
a
bit
of
a
problem
for
us.
It
might
allow
for
some
gaming
of
the
system
where
you
could
sort
of
try
and
convince
autopilot
to
create,
like
an
eight
core
node
run
a
run,
a
one
core
workload
on
it.
You
know,
try
and
prevent
other
pods
getting
on
there
in
with
you
know,
using
some
technique,
which
is
potentially
possible
and
thus
kind
of
you
know,
get
like
8x
your
resources.
A
That
would
be
a
problem
for
us,
so
the
ideal
design
would
be
well
what
if
the
user
could
burst
within
the
paid
for
capacity
on
the
node
at
any
given
time.
So
if
we
clamp
the
bursting
in
this
case
to
five
and
a
half
cores,
then
we
would
be
giving
you
you
know
the
capacity
that
you're
paying
for
which
would
be
ideal.
A
A
A
What
about
the
downsides?
I
did.
I
did
mention
that
there
would
be
pros
and
cons
a
couple
of
the
downsides.
I
believe
of
the
design
that
we
came
up
with
here
is
that
there
is
a
potential
for
allocatable
inefficiency
in
this
system
right,
it's
possible
if
the
user
is
creating
and
deleting
a
lot
of
pods
of
all
different
shapes
and
sizes
that
you
could
end
up
with
very
much
underutilized
nodes,
and
that
requires
you
know
the
team
to
build
additional
features
like
defrag
to
kind
of
correct
that.
A
A
Obviously,
by
you
know,
running
two
containers,
two
pods
on
on
the
one
node
you
can
potentially
have
resource
contention,
although
we
do
have
a
way
to
solve
that
for
users,
they
can
still
separate
their
workloads
when
needed,
and
a
couple
of
downsides
of
using
a
multi-single
tenor
platform
like
we
do
as
opposed
to
a
fully
multi-tenant
platform.
Is
that
it's
harder
for
us
to
add
hot
standby
capacity,
so
one
of
the
really
nice
benefits
of
a
multi-tenant
system
is
you
typically
have
you
know
a
massive
resource
pool
shared
by
everyone?
A
You
can
just
individuals
can
scale
up
and
down
very
quickly.
We
don't
have
that
ability
with
with
this
design.
So
I
just
want
that
is
one
drawback,
adding
a
new
pod
if
it
needs
a
new
node,
can
take
between
60
and
80
seconds
and
there's
a
little
bit
of
greater
operational
complexity
on
the
platform
side.
I
think,
if
any
time
you
operate,
a
multi-single
tenant
system
there's
a
little
bit
extra
ops
complexity,
but
again
most
of
these
downsides
by
the
way
are
kind
of
just
a
burden
on
us.
A
I
I
think
you
know
just
makes
a
couple
of
different
problems
a
little
bit
harder
to
solve,
but
hopefully
it's
solvable
okay.
So
in
summary-
and
you
know
this
is
the
the
takeaway
here-
I
guess
that
I'd
like
to
try
and
convince
you.
I
hope
I
did
convince
you
of
this-
is
this
nodeless
and
does
that
even
matter?
Well,
you
know
when
I
started
this
project
back
in
2018,
novus
was
kind
of
synonymous
with
fully
managed
kubernetes,
and
I
guess
maybe
the
point
I'm
trying
to
make
here
is
that
it's
not
a
good.
A
I
synonym
that
this
design
is
is
operationally
nodeless.
It's
nervous
in
the
sense
that
you
don't
care
about
the
nodes.
You
don't
have
to
think
about
them,
but
there
is,
but
I
do
believe
there
is
a
benefit,
as
I've
hopefully
outlined,
of
having
nodes
existing
as
a
scheduling
concept
when
needed
and
when
relevant.
A
So
that
was
our
journey.
I
hope
that
was
useful.
I
hope
I
hope
yeah.
I
hope
it's
of
interest
to
to
learn
how
the
team
went
about
and
built
this
thing.
Maybe
maybe
it
can
inspire
you
as
you're
building
your
own
products
and
services
with
kubernetes,
and
with
that
I'd
love
to
take
any
questions.
We
have
a
couple
of
minutes
also
happy
to
continue
the
debate
on
twitter
over
beer
or,
however,
you
want
to
do
it.
A
B
A
C
Well,
I
hope
you
can
hear
me
well.
The
company
we
were
I
work
with
is
subject
to
a
lot
of
audits
and
they
require
us
to
split
nodes
in
different
subnets
in
order
to
you
know,
segregate
them
on
a
network
level
and
also
they
require
us
to
install
certain
software
and
binaries
on
these
nodes.
Basically,
we
have
to
be
in
control
of
the
machine
image
that
we
use
to
spin
up
the
nodes,
and
I
wanted
to
ask
if
this
is
capable.
If
you
are
capable
of
doing
this,
yeah.
A
I'm
just
both
of
these
things.
Actually,
I
I
don't
know
if
the
subnet
separation
thing
is
possible,
that's
something
I'd
have
to
probably
take
offline
and
do
a
deep
dive
with
you.
With
the
with
the
security
components
that
you're
running.
I
would
like
to
think
that
it
is
possible.
We
do
work
with
a
lot
of
security
partners
to
make
sure
that
their
solutions
work.
So
unless
it's
like
a
homegrown
home
built
container,
I
would
say
the
answer
is:
probably
yes:
it
either
works
or
can
be
made
to
work
with
this
system.
A
C
Yeah,
I
don't
know
about
that,
because
they
were
pretty
reluctant
of
allowing
us
to
go
to
the
cloud
in
the
first
place
right.
So
I'm
not
really
sure
how
you
know
cool
they
will
be
with
that,
but
yeah,
then.
The
other
question
that
spikes
up
is
is
our
rdav,
for
example,
softwares
you
know
enabled
through
daemon
sets.
D
A
D
And
apart
all
the
good
things
of
benefits
of
the
dki
develop,
we
are
active
users
of
happy
uses
of
the
ukee
autopilot
but
and
the
restrictions
we
come
up
with
the
with
not
being
able
to
deploy
things
like
secret
install
and
this
kind
of
drivers.
Do
you
have
any
plan
to
overcome
those
issues
that
implicitly
this
kind
of
designs
imposes
because
yeah
we
want
to
use
bolt,
for
example,
for
secrets,
and
we
weren't
able
to
do
that
by
using
standard
csi
drivers
instead
of
creating
direct
connections
to
those
bulk
clusters
and
so
on.
A
Yeah,
I
think
like
when
it
come
when
it
comes
to
these
like
restrictions.
Basically,
we
have
to.
We
have
to
either
if
it's
a
kind
of
a
partner
like
a
well-known
container,
a
well-known
workload
we
can,
we
can
take,
we
can
potentially
allow
list
it.
The
other
option
we
have
is,
if
it's
like
a
technique
we
have
to
enable,
we
essentially
have
to
kind
of
productize
it.
A
So
if
you
need
a
driver,
it
might
be
the
case
where
we
just
have
to
offer
the
driver
as
a
feature
right
where
you
can
just
turn
on
that
driver.
So
we
should.
We
should
connect
and
continue
this
conversation,
but
but
I
I
do
believe
vault
should
actually
work.
I
know
when,
when
we
first
launched
the
product
we
didn't
offer
mutating
web
hook,
support
which
broke
like
half
of
the
you
know
the
kind
of
types
of
workloads
like
vault
and
things
like
that
which
need
to
mutate
workloads.
A
A
A
Good
question:
it's
it's
it's
a
decision.
I
think
we
should
actually
revisit
the
you
know.
The
plan
was
to
start
a
little
bit
conservative.
I
guess
because
it's
much
easier
to
relax
these
restrictions
over
time
and
and
I
I
believe
that
the
theory
was
and
like
I
was
part
of
the
decision
and
I'm
still
trying
to
remember
kind
of
like
like.
Why
did
we
actually
do
that?
A
I
think
it
was
just
so
that
when
we
packed
the
pods
on
the
on
on
the
nodes
that
are
kind
of
if,
if
they're
known
kind
of
sizes,
like
you
know
the
little
tetris
kind
of
diagram,
you
know
we
can
kind
of
slide
it
in
a
little
bit
better.
In
hindsight,
I'm
not
convinced
it's
actually
needed
to
be
honest,
so
yeah
we
might.
We
might
take
another
look
at
that
and
yeah
I
mean.
Would
you
like
to
just
have
like
just
anything
like
between?
We
probably
need
a
minimum,
but
yeah
all
right.