►
From YouTube: Clusterless - Utilizing a Cluster of Clusters Seamlessly - Fei Guo, Alibaba & Chris Hein, Apple
Description
Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Clusterless - Utilizing a Cluster of Clusters Seamlessly - Fei Guo, Alibaba & Chris Hein, Apple
Multi-cluster support in Kubernetes becomes more and more important for application efficiency, better resource utilization and breaking the geographic boundary. While existing open source projects like Karmada, Tensile-Kube, Liqo, Admiralty, and more have addressed challenges in multi-cluster management such as configuration, lifecycle management, they either introduce new APIs/management workflows or have scalability limitations in terms of scheduling throughput. In this talk, a new Clusterless design is introduced to reduce the management burden of multi-cluster users. Basically, multiple clusters are abstracted as Pod resource providers, connecting to a management cluster which is exposed to the users. There are no new APIs introduced hence the consistent user experiences are preserved. The Pod dispatching is done by a highly efficient two-level scheduling algorithm which can achieve O(n) scheduling throughput, where n is the number of managed clusters.
A
So
hi
everybody
welcome
to
our
talk
about
clusterless
architectures.
This
is
a
new
design,
that's
coming
out
of
the
virtual
cluster
space
and
it
allows
you
to
utilize
clusters
of
clusters
seamlessly
to
run
workloads
wherever
wherever
you
need
to
faye-
and
I
we'll
be
talking
to
you
about
this
and
we're
both
the
maintainers
for
the
project,
along
with
some
other
folks.
Faye
works
at
alibaba
and
I
work
at
apple.
A
So
why
we're
talking
about
this
and
why?
This
is
important.
So
one
of
the
things
that,
as
you
start
to
use
kubernetes,
you
start
to
realize
that
when
you
started,
you
initially
started
with
at
least
a
single
cluster,
and
you
started
to
build
out
your
architectures
and
you
and
you
deployed
workloads
into
that
single
environment.
You
started
to
see
some
problems.
This
is
something
that
I
feel
like
everybody
experiences
in
their
kubernetes
journey,
and
so,
where
folks
usually
go,
is
they
turn
to
a
multi-cluster
management
style
and
the
whole
idea
behind
this?
A
Is
it
allows
you
to
actually
scale
out
your
architecture
so
taking
from
a
single
cluster
which
has
scale
targets
that
are
defined
by
sig
scalability
and
moving
that
into
how
do
we
make
it
so
that
we
could
run?
You
know,
4x
the
type
of
capacity
that
we
currently
have
and
and
being
able
to
do
that
and
scale?
A
If
you're
thinking
like
bachelor
workloads
where
you're
processing
a
lot
of
data,
you
don't
want
to
run
that
from
completely
opposite
ends
of
the
country,
for
example,
in
this
environment,
you
also
get
to
a
place
where
you're
really
taking
you're,
bringing
in
a
lot
more
compute
resources
and
everything
gets
more
difficult
to
actually
manage
and
aggregate
across
those.
So
what
you
end
up
doing
is
start
to
look
at
that
multi-cluster
space
and
then
bring
in
things
like
aha
strategies.
How
would
you
actually
increase
the
availability
of
your
workloads?
A
How
do
you
set
up
regions
that
you
can
actually
fail
over
into
when
something
goes
wrong?
I
imagine.
Everybody
here
has
experienced
issues
with
etcd
having
key
space
problems
and
you
get
a
corrupted
key
and
you
end
up
with
an
entire
cluster
that
just
doesn't
function
anymore,
and
how
do
you
actually
recover
for
those
things?
On
top
of
this,
we
get
into
a
new
world
of
how
do
you
actually
build
out
the
security
compliance
when
you
have
now
distributed
workloads
across
different
environments,
as
well
as
the
autonomy
of
those
workloads
going
forward?
A
How
do
you
actually
go
and
create
new
clusters?
Are
you
using
public
cloud
tools
like
eks
and
gke,
and
ak
ack
to
actually
deploy
those
things?
Are
you
using
open
source
tools
like
what
faye
and
I
work
on
with
the
cappy
space
or
are
using
open
container
management?
Are
you
using
any
of
the
off-the-shelf
vendor
tools
going
in
going
on
top
of
that?
There's
then
the
governance
of
those?
So
now
that
you
have
distributed
workloads
across
many
clusters,
how
do
you
do
centralized
management?
A
How
do
you
make
sure
that
the
security
policies
are
set
up
and
they
abide
by
what
your
security
teams
within
your
companies
actually
need
and
really
making
sure
that
they're
they're
customizable,
but
they
give
their
they
customizable,
but
they
don't
expose
you
to
anything
too
far.
On
top
of
that,
you
have
to
look
into
a
a
vast
space
here
of
monitoring
and
tracing.
There
are
many
tools
that
we
have
to
work
with
nowadays,
dashboards
to
visualize
across
those
clusters
where
the
data
is
actually
going
to
live
and
how
that
functions.
A
A
We
really
wanted
to
look
into
how
to
make
this,
how
to
abstract
this
and
how
to
make
it
so
that
you
can
have
better
workload
management
but
still
in
essence,
give
you
the
same
sort
of
experience.
We
have
now
the
tools
that
are
currently
existing
and
that
folks
have
been
using
things
like
cube,
fed
and
argo
cd.
A
They
bring
on
that
same
level
of
abstraction,
but
they
kind
of
given
they
give
you
some
trade-offs
in
terms
of
what
you're
actually
going
to
be
doing
at
a
granular
level,
and
then
the
last
thing
is
scheduling.
Now,
if
you
have
multiple
clusters,
you
need
to
figure
out
how
you're
going
to
actually
schedule
those
workloads
across
them.
Is
that
something
manual
where
everybody
knows
about
every
single
cluster
that
is
potentially
accessible
to
them
and
they
go
and
manually
decide.
A
I
want
to
deploy
to
this
cluster
in
this
region
because
of
this
specific
thing,
or
is
there
an
automated
system?
That
kind
of
does
those
does
that
work
for
you,
thinking
about
things
like
the
get
ops
world
and
trying
to
deploy
to
some
cluster
based
on
some
decision
in
a
in
a
declarative
script
so
enter
in
clusterless
now
in
a
multi-cluster
environment.
A
The
idea
here
with
clusterless
is
that
we
want
to
achieve
really
the
single
cluster
user
experience,
so
everything
that
everybody
knows
and
loves
about
kubernetes
today
being
able
to
take
that
and
say
what
does
this
look
like
across
many
clusters
without
really
introducing
new
apis
and
making
it
so
that
every
single
off-the-shelf
tool
that
you
currently
use
could
potentially
be
deployed
into
that
same
exact
experience?
You
deploy
some
operator
to
do
maybe
prometheus.
A
A
A
And
what
we
wanted
to
do
is
really
introduce
no
new
workload
clusters,
new
workload
controllers
into
those
worker
clusters,
so
taking
a
look
at
the
lowest
level,
abstraction,
that's
in
kubernetes
things
like
pods,
and
how
would
you
actually
spread
that
across
across
those
clusters-
and
this
is
something
that
we
already
are
seeing
other
open
source
tools
doing
so?
Tensile
cube
and
likio
are
basically
doing
this
already,
but
they
do
this
using
tools
like
virtual
cubelet
as
their
abstraction.
So
they
can.
A
They
can
run
a
virtual
cubelet,
which
then
addresses
one
or
many
clusters
under
the
hood
for
actually
deploying
those
workloads.
Now,
in
our
world,
we
started
with
with
the
tool
chain
that
we've
been
working
on
most
heavily,
which
is
virtual
cluster
and
trying
to
bring
that
same
level
of
abstraction
to
that
environment.
So
it's
something
that
you
already
are
seeing
and
we're
just
extending
this
into
tools
that
we
have
today.
A
Now,
to
take
it
aside
for
a
second
there's,
an
important
there's,
an
important
conversation
that
we
need
to
have
here
about
the
abstraction
level
that
we're
talking
about
and
that's
specific
to,
pods
versus
workloads.
So
as
you
look
at
the
the
entire
space
of
this,
not
just
the
tensile
cube
and
those
kind
of
workloads,
there's
other
things
in
here.
There's
ocm
or
yeah
ocm.
There's
argo,
there's
carmada!
A
So
I,
as
a
user,
might
be
using
one
of
those
tools
and
deploying
hdr
to
deploy
a
full
workload
which
defines
everything
and
it
goes
to
a
specific
cluster
and
gets
scheduled,
and
so
there's
the
difference
between
pod
and
workload
granularity
there.
So
things
like
with
pods,
you
get
no
new
apis.
You
get
to
use
the
standard,
off-the-shelf,
kubernetes
resources
that
we
all
know
and
love
and
have
been
using
for
many
years.
A
It
does
require
you
to
run
those
extra
extra
controllers
that
go
and
operate
on
those,
but
it
gives
you
the
benefit
of
distributed
orchestration.
So
if
something
goes
wrong
with
the
top
level
user
control
plane,
that
underlying
cr
could
continue
to
run.
Even
if
something
happens,
if
the
cluster
has
a
failure
and
it
comes
back
up
to
a
healthy
state,
it
can
continue
to
run
those
workloads.
A
But
you
get
the
negatives
of
having
to
run
those
new
apis
to
assist
with
the
actual
propagation
of
the
resources
and
it's
kind
of
inconvenient
to
actually
debug
those
workloads.
Because
now
you
have
a
difference
between
what's
in
the
what's
in
the
user
cluster
and
what's
in
the
actual
workload
cluster,
that's
the
worker
cluster.
That's
actually
running
those
things.
A
Basically,
if
you
have
single
tier
scheduling,
which
very
much
looks
like
what
the
kubernetes
scheduler
has,
you
have
all
of
the
nodes
allocated
to
a
single
scheduler
and
it
and
it
will
pick
and
choose
where
the
workloads
are
going
to
run
along
those
nodes.
This
is
also
this
also
limits
your
abilities.
A
Where
does
the
workload
actually
get
scheduled
and
then
once
that,
once
it
picks
that
it
drops
it
on
to
those
lower
level
worker
clusters
to
go
and
decide
what
node
is
it
going
to
do
so?
We
kind
of
distribute.
We
distribute
the
the
decision
making
to
make
it
a
faster,
faster
scheduling,
but
it
does
introduce
some
failure
modes
that
need
to
be
accounted
for.
So
if
you
go
through,
you
can
imagine
a
workload.
A
You
deployed
a
pod
into
your
worker
cluster
and
that
now
tried
to
get
scheduled
to
a
cluster
that,
at
the
same
exact
time,
had
a
job
that
was
scaling
up
and
it
ran
out
of
capacity,
and
maybe
you
didn't
have
a
cluster
auto
scaler
setup,
or
maybe
those
controls
aren't
something
that
you
have
available
to
you
and
now
that
pod
can't
actually
schedule
in
the
new
cluster.
So
those
are
kind
of
the
the
downfalls
of
that
type
of
architecture.
A
And
then
the
last
thing
here-
and
this
is
the
the
last
decision
kind
of
making
that
had
to
be
done-
is
distributing
that
scheduling,
decision
to
many
schedulers,
so
coordinated
schedulers
talking
to
each
other
and
deciding
and
doing
resource
requests
and
reservations
against
those
sub
schedulers.
And
this
is
a
really
a
really
interesting
architecture,
because
it
pretty
much
reduces
the
the
chance
of
any
of
those
race
conditions.
But
it
requires
a
lot
of
coordination.
A
A
So
enter
where
we,
where
we
mostly
work
again,
which
is
virtual
cluster
and
trying
to
basically
come
in
at
from
the
multi-tenancy
side
of
things,
if
you're
deploying
into
a
multi-cluster
world
you're
likely
to
dealing
with
a
lot
of
different
tenants
and
trying
to
make
sure
that
that
all
of
their
workloads
get
scheduled
into
somewhere.
So
problem
being
multi-tenancy
is
a
legitimate
concern.
When
utilizing
a
large
amount
of
aggregate
cluster
resources,
we
have
tons
of
cpu
and
memory.
A
So
if
somebody
wants
to
deploy
a
crd
into
that
cluster,
they
can
do
that
and
it's
not
going
to
affect
the
the
lower
level
workloads
and
so
that
originally
started
with
being
scoped
specific
to
one
supercluster,
but
it
kind
of
became
a
natural
progression
to
well.
Can
we
make
this
run
on
multiple
clusters,
using
that
same
exact,
syncing
mechanism
that
we
have
in
place.
A
So
has
actually
functioned.
It
functions
by
adding
another
interface
in
in
the
middle,
so
we
talked
about
the
two
two
layer
scheduling
and
how
that
could
function.
This,
in
essence,
is
building
out
a
top
tier
scheduler
which
operates
off
of
the
resources
that
we
already
in
most
multi-tenant
environments,
use
today,
so
things
like
name
spaces
and
resource
quotas.
So
a
top
level,
a
top
level
scheduler
can
go
through
and
look
at
the
amount
of
resources
that
you're
requesting
for
a
specific
name
space,
and
it
can
start
to
make
those
first
decisions
so
phase.
B
In
virtual
cluster,
we
have
synchro,
which
synchronizes
object
between
the
canon
control,
plane
and
analyzed
super
crosstalk.
As
the
natural
extension
you
will
see
that
we
will
assign
we
will.
We
will
create
a
sinker
for
each
worker
clusters
and
we
have
a
new
scheduler
which
watch
for
the
work
across
the
capacity.
Changing
capacity
changes
and
another
object
will
be
creating
a
technical
direction
again
in
the
vc
model,
the
synchro
can
watch
for
multiple
canon
clusters,
so
the
same
are
applied
for
the
scanner
as
well.
B
So,
overall,
if
you
look
at
the
the
the
new
architect,
is
that
we
on
top
of
the
virtual
cluster,
we
have
modified
sinkers,
and
then
we
have
a
new
first
level
schedulers
in
practice.
Both
the
thinker
and
the
scheduler
will
normally
manage
by
separate
meta
clusters,
but
for
simplicity,
I'm
not
going
to
show
that
in
this
this
figure.
B
All
right
next,
I'm
going
to
talk
about
some
of
the
implementation
details
to
realize
the
costliest
curl
height,
as
I
mentioned
before,
for
the
first
thing.
If
we
need
to
enhance
the
thinker
to
support
selective
object,
synchronization,
why
we
want
to
do
that
because
in
theory
you
can
simplify
the
solution
by
just
copying
all
the
tenant
objects
with
the
namespace
in
all
the
underlying
work
clusters,
but
not
the
part
object
just
to
prepare.
B
The
part
will
be
scheduled
to
any
of
the
water
clusters,
but
this
will
introduce
some
unnecessary
overheads
of
storing
the
objects.
If
the
part
is
not
running
those
clusters,
they
are
completely
waste
so
to
to
resolve
that
problem.
We
enhancing
the
sinker
to
support
selective
civilization
and
the
the
the
the
decision
is
is
determined
by
the
placement
results
which
is
the
which
is
which
is
specified
by
the
scheduler.
B
So
if
you
look
at
the
right
figure
in
this
example,
assuming
the
namespace,
a
in
cannon
control,
plane,
e1
has
the
quarter
and
the
quarter
size
config,
so
that
we
have
two
slices
and
the
scheduler
has
decided
the
the
scheduling
decision.
With
the
placement
results.
We
put
one
slice
in
c1
one
size
in
c2.
B
B
Out
of
all
the
implementation,
the
most
challenging
part
is
to
implement
a
scheduled
cache.
Unlike
the
traditional
kubernetes
scheduler,
you
only
need
to
watch
for
one
ap
server
in
this.
In
this.
For
this
scheduler,
you
have
to
watch
for
the
status
of
multiple
clutters,
both
in
both
tenant
control,
plane
and
mla
worker
clusters.
There
are
a
lot
of
failure
points
in
in
this
settings.
B
B
We
put
a
lot
of
efforts
to
make
sure
it
happens,
and
another
general
problem
is
that
whether
the
scheduler
should
watch
for
the
all
the
node
events
coming
from
underlying
water
clusters,
it
can
be
a
huge
amount
of
manual
traffic
if
you
really
want
to
do
that,
because
kubernetes
really
generate
known
harvest
and
will
be
kind
of
released,
no
least
api
calls
we
to
reduce
the
overhead.
We
choose
to
periodically
scan
the
clusters
to
collect
the
node
status
to
and
compute
the
available
classic
capabilities.
B
The
downside
of
doing
that
is
that
there
are
some.
There
is
a
certain
delay
if
the
underlying
cluster
capacity
change,
we,
the
scheduler,
will
be
aware
of
those
changes
these
with
some
delays,
which
which
can
cause
some
raw
scheduling
decision,
but
we
have
some
remediation
process,
can
accommodate
that
effect.
B
B
The
reason
is
that
we
want
to
reduce
the
the
the
amount
of
classes
that
synchronize
the
non-part
objects
from
those
space
and-
and
we
will
just
use
a
simple
phosphate
algorithm
to
pick
the
the
clutch
clusters
for
some
phase,
two
part
wise
schedule
among
candidate
clusters.
The
algorithm
is
really
simple.
We
just
use
a
simple
first
feed
or
round
robin
to
find
the
target
clusters
note
that
there
are
large
room
cleaning
costs
in
the
scheduling
domain.
B
For
example,
we
can
leverage
a
lot
of
from
the
upstream
scheduling,
schedulers
scheduling,
capabilities
such
as
support
affinity
and
anti-affinity,
or
even
spread
policy,
so
we've.
So
if
we
have
that
implementation
online,
we
can
pretty
much
support
some
scenario
that
will
be
currently
not
that
we
cannot
support
for
now.
Let's
say
for
the
redundancy
of
living
the
kind
of
aspect,
but
this
area
is
to
be
enhanced,
although
the
arizona
are
simple,
but
we
still
can
get
some
insights
from
the
design
such
that
the
namespace
quarter.
B
Slice
schedule
is
not
in
a
pod
creation
grid
path
and,
and
our
scheduling
always
make
sure
for
part
of
scaling.
The
number
of
candidates
clusters
should
be
small,
so
this
scheduling
overheads
can
be
measurable.
Overall.
By
doing
this,
we
can
roughly
achieve
can
system
wise,
positive
throughput
with
with
all
the
unstability
which
is
the
so
potentially
you
can
leverage
all
the
schedulers
running
inside
the
underlying
worker
clusters
to
schedule
all
the
paths
that
are
sending
to
the
top
level
user
facing
attendance
enterprise.
B
I
want
to
talk
about
some
other
features,
which
is
very
specific
to
this
multi-class
scheduling
the
first
one
is
rescheduling
because
the
the
the
placed
candidate
clusters
for
the
name
services
can
be
offline
and
time
and,
unlike
the
traditional
kubernetes,
they
have
no
computer
to
edit
the
path
in
case
of
node
fader.
We
don't
have
that
controller
ready
yet
for
this
multi-cluster
domain.
B
Instead,
for
now,
we
are
just
allowing
people
to
manually
revoke
the
scheduled
decision
by
removing
the
scale
results
from
the
namespace
annotation
the
schedule.
We
are
kicking
and
rescheduling
the
namespace
to
find
the
online
cluster
to
support
those
workload,
and
in
case
there
are
some
any
stating
objects
stays
in
the
work
clusters.
The
sinker
will
do
the
gc
another
aspect
of
in
terms
of
the
application
workload
runtime,
it
isn't
service
support.
B
B
Then
we
have
implemented
the
features
that
I
mentioned
before,
and
it
come
with
a
prototype
due
to
the
time
limitation.
I'm
not
going
to
give
a
full
live
live
demo,
but
you
can
find
the
demo
in
this
link,
essentially
in
this
demo.
What
I
do
is
I
create
one
tenant
control
plan
and
with
two
worker
clusters,
and
I
manipulated
the
quarter
as
specified
in
the
default
name.
Space
and
the
quarter
slide
size
so
that
we
have
two
slices
and
the
scheduler
scheduled
each
one
slice
in
one
worker
clusters.
B
Now
I
create
a
deployment
with
two
replicas
called
gold
line
and
the
scheduler
rear
schedule
rows
parts
in
two
separate
worker
clusters,
and
if
you
look
at
the
screenshot,
you
can
see
that
from
the
user
face
control
plan,
you
check
the
the
path
the
usdq
process
is
running
and
if
you
check
each
two
across
the
name
name,
the
root
one
root.
Two
only
one
part
is
running
one
worker
clusters
again
for
more
details.
Please
go
through
this
link.
B
You
see
a
full
demo,
so
in
summary,
so
the
multi
cluster
management
solution
becomes
popular
and
they
they
usually
choose
different
workload,
abstract
models
and
choose
different
scheduling,
strategies
to
schedule
those
workloads
and
normally
they
will
bring
new
apis.
So
really,
integrating
those
solutions
with
existing
solutions
will
be
non-treat
will
be
a
a
non-trigger
effort
and
for
road
solution
normally
requires
manual
capacity
planning,
which
means
the
users
have
to
have
full
knowledge
about
the
resource
usage
of
the
underlying
worker
clusters,
so
that
they
can
make
sure
they
are
scheduling.
B
B
We
implement
the
class
view
by
extending
the
virtual
cluster
framework
and
we
can
achieve
the
scalable
scheduling
throughput
and
the
entire
solution
is
pretty
easy
to
integrate,
because
we
don't
introduce
any
new
api
commandment
to
manage
the
workloads
all
right.
This
is
the
end
of
the
plantation
I'm
happy
to
take
any
questions.
Thank
you.