►
Description
Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
McK8s: Container Orchestration in Kubernetes Multi-Clusters - Mulugeta Ayalew Tamiru, University of Rennes 1 / Elastisys AB
Today many companies deploy their workloads across multiple Kubernetes clusters in private and public cloud data centers or in emerging computing paradigms such as Fog Computing. Recent efforts in the community such as KubeFed address some of the multi-cluster deployment challenges. However, KubeFed currently offers only manual or fully load-balanced placement. Other challenges such as autoscaling and inter-cluster network routing are solved by other projects separately. In this context, we believe there is a need for a more integrated and autonomous approach for managing geo-distributed Kubernetes clusters at scale. In this talk, we will present mck8s -- an extension and partial replacement of KubeFed for geo-distributed computing environments which aims to bring placement, autoscaling, bursting, inter-cluster routing, and cluster provisioning into one package. We will also emphasize its usability and easy adoption by using manifest files very much similar to vanilla Kubernetes.
A
A
So
today
I
would
like
to
talk
about
the
evolution
of
cloud
deployments
in
the
past
few
years
and
some
of
the
challenges
in
multi-clusters
management
in
terms
of
resource
management
and
application
deployment,
and
then
I'll
briefly
discuss
kubernetes
federation,
which
is
the
foundation
of
our
work,
and
then
I
will
briefly
discuss
the
architecture
and
controllers
of
our
platform.
This
and
lastly,
show
you
a
demonstration
of
our
platform
in
action.
A
So
in
the
last
few
years
we
have
seen
an
increasingly
geographically
distributed
cloud
deployments.
The
major
cloud
providers
now
have
data
centers
in
many
regions
of
the
world.
We
can
identify
three
main
geography
deployments,
namely
hybrid
cloud,
multi-cloud
and
fog
computing
in
hybrid
cloud.
We
deploy
applications
on
data
centers
from
private
and
public
cloud
providers.
In
the
multi-cloud
case,
we
deploy
applications
on
multiple
public
clouds
or
data
centers
from
multiple
regions
of
the
same
a
public
cloud
provider.
A
The
last
one
for
computing
is
an
emerging
geo-distributed
computing
paradigm,
where
we
have
resources
from
private
public
as
well
as
micro
data
centers
that
are
distributed
in
vast
geographical
areas
in
the
aims
to
be
closer
to
end
users.
Much
of
this
evolution
is
driven
by
the
increasing
demands
of
modern
applications.
Some
of
the
non-functional
requirements
are
low
latency
the
desire
to
provide
fast
services
to
end
users
by
placing
our
applications
in
regions
where
most
of
our
users
are
located.
A
High
bandwidths
and
reliable
connectivity
in
the
case
of
iot
and
video
analytics,
for
example,
would
like
to
have
high
bandwidth
and
reliable
connectivity
so
that
we
can
upload
a
vast
amount
of
data
to
the
analytics
framework.
We
have
other
requirements
also
such
as
high
availability,
disaster
recovery,
scalability
security
and
compliance.
A
So
managing
this
duration
environment
is
not
easy.
There
are
a
lot
of
challenges
we
are
talking
in
terms
of
hundreds
and
thousands
of
clusters
which
become
almost
impossible
to
manage
manually,
so
we
need
to
have
automated
ways
of
deploying
applications
and
managing
resources.
This
means
solving
many
challenges,
some
of
which
are
resilience
in
terms
of
hardware
and
network
failures,
providing
various
placement
policies,
automated
placement
policies,
solving
our
scaling
problems
at
various
levels
and
in
various
degrees,
in
order
to
provide
a
persistent
performance.
A
Even
though
our
workload
and
user
traffic
changes
all
the
time,
we
also
have
to
solve
user
traffic
coding
and
load
balancing
issues
so
that
we
can
route
user
traffic
from
one
cluster
to
another.
So
we
believe
we
try
to
address
these
problems
in
mckay
days
and
we
believe
that
state-of-the-art
container
orchestrators
such
as
kubernetes
serve
as
a
basic
foundation
and
building
blocks
because
of
their
portability,
interoperability
and
extensible
nature.
A
We
are
not
the
first
ones
to
address
these
problems.
For
example,
the
kubernetes
sync
multi-cluster
group
has
been
working
on
the
kubernetes
variation
project
for
the
last
few
years.
This
project
provides
control,
plane
and
concepts
and
obstructions
necessary
to
manage
multi-cluster
kubernetes
and
in
the
same
spirit
as
kubernetes.
The
architecture
has
a
host
cluster
where
the
controllers
are
deployed
and
much
of
the
decision
takes
place
and
member
clusters
where
applications
are
deployed.
A
It
also
provides
an
automated
mechanism
called
replica
scheduling,
preference
which
provides
fully
load
balanced
or
weight-based
placement
across
your
clusters,
but
we
believe
that
in
many
of
the
geo-distributed
environments,
such
as
for
computing,
you
need
more
automated
policies,
such
as
resource-based
policies,
in
order
to
utilize
the
clusters
fully
or
locality,
aware
and
proximity
our
place
and
policies
so
that
we
can
deploy
our
applications
on
the
clusters
where
most
of
our
users
are
located.
We
would
also
need
auto
scaling,
resource
provisioning
and
network
loading
policies.
A
We
also
aim
to
provide
inter-cluster
network
routing
and
load
balancing.
The
architecture
is
pretty
much
similar
to
that
of
kubernetes
federation.
In
our
case,
we
have
a
management
cluster
that
has
our
controllers,
as
well
as
workload
clusters
where
our
applications
are
deployed
in
mckates,
we
rely
on
other
open
source
software,
such
as
kubernetes
federation,
which
we
use
for
managing
membership
to
the
federation,
adding
and
removing
of
clusters
to
the
federation.
We
use
cluster
api
for
transparent
provisioning
and
removal
of
kubernetes
clusters
on
supported
cloud
providers.
A
We
use
prometheus
for
monitoring
our
workload
clusters
for
resource
use
and
network
use.
We
also
have
serv
and
celium
on
our
called
cluster
service
used
for
measuring,
inter
cluster
network
latency,
which
we
rely
on
when
making
offloading
and
bursting
decisions.
Psyllium
is
used
for
intel
clusters,
network
clouding
and
large
balancing.
So
on
top
of
this,
we
introduced
four
new
controllers,
namely
multi-cluster
scheduler,
multi-cluster,
horizontal
power,
scalers
rescheduler
and
cloud
cluster,
provisioners
and
autoscaler.
I
will
discuss
the
details
of
these
controllers
in
the
coming
slides.
A
So
in
mckay
this
we
have
introduced
some
six
custom
resources.
The
first
three
are
similar
to
that
of
kubernetes
federation,
so
multi-cluster
deployment
is
similar
to
federal
deployments.
Multi-Class
search
jobs
is
similar
to
federated
jobs,
multicultural
services
similar
to
a
federated
service,
but
we
have
also
introduced
multi-cluster
horizontal
power,
scalars
cloud
cluster
provisionary
multi-cluster
scheduler.
So
these
are
the
resources
that
our
controllers
deploy,
update
and
remove
on
the
right.
You
can
see
the
custom
resource
definition
for
one
of
our
resources.
The
multi-cluster
deployments
is
a
simple
definition.
A
The
first
controller
is
a
multi-cluster
scheduler,
so
this
one
is
responsible
for
creation,
deletion
and
updating
of
three
of
our
custom
resources,
namely
multi-cluster
deployment,
multi-cluster
service
and
multi-cluster
jobs.
We
provide
manual
and
automated
policy-based
placement
capabilities.
The
first
one
is
the
manual
cluster
affinity
capability,
where
you
can
deploy
your
applications
on
selected
clusters,
similar
to
kubernetes
federation.
So
you
can
specify
I
want
to
deploy
on
this
and
this
and
this
cluster,
and
then
our
scheduler
takes
care
of
that.
We
also
have
automated
resource
based
and
network
traffic.
Aware
policies.
A
For
example,
we
have
a
worse
feed
policy
that
deploys
applications
on
the
clusters
that
have
the
most
resources
available.
We
also
have
a
best
fit
policy
that
uses
a
bin
packing
algorithm
to
deploy
applications
on
those
clusters
that
have
been
used.
The
most
we
have
a
traffic
aware
policy,
which
is
which
comes
in
handy
in
the
case
of
for
computing
that
deploys
applications
on
those
clusters
that
have
that
are
receiving
the
most
traffic.
So
network
traffic
is
used
here
as
an
indicator
for
the
for
the
presence
of
most
most
users.
A
What
this
means
is,
if
your
selected
clusters
do
not
have
sufficient
resources
for
deploying
your
applications,
our
scheduler
will
deploy
or
upload
the
application
to
another
cluster
that
have
sufficient
resources,
but
is
closer
to
the
selected
clusters
in
terms
of
network
clearancy.
We
also
provide
bursting
capability
where,
if
a
cluster
cannot
place
all
the
replicas
of
a
deployment,
the
extra
replicas
are
deployed
on
neighboring
clusters
have
the
sufficient
resources
on
the
right
side.
We
can
see
the
manifest
file
for
a
sample
multi-cluster
deployment.
A
As
you
can
see,
this
manifest
file
is
pretty
much
similar
to
that
of
vanilla,
kubernetes
deployment.
The
only
difference
is
the
api
version
kind
and
certain
certain
fields
under
the
spec
section.
For
example,
here
we
have
specified
on
the
number
of
clusters
or
locations
where
we
want
our
application
deployed
and
the
placement
policy,
our
multi-cluster
service
works
in
collaboration
with
celium,
to
provide
inter-cluster,
routing
and
load
balancing
the
service.
A
Manifest
is
pretty
much
similar
to
that
of
kubernetes
service
and
we
add
the
annotation
for
psyllium,
so
that
psyllium
can
do
the
intercostal
network
clouding
android
balancing
for
us
this.
For
this
we
need
to
have
psyllium
and
celium
cluster
which,
on
the
workload
clusters,
the
multi-cluster,
horizontal
port
autoscaler,
also
similar
to
kubernetes
horizontal
port.
Auto
scanner
adjust
the
number
of
replicas
of
our
deployments
across
all
clusters,
and
this
decision
is
passed
to
the
scheduler,
which
makes
which
are
just
the
deployment
or
placement
of
our
replicas.
A
A
The
cloud
cluster,
provisional
and
autoscaler
periodically
checks
the
status
of
our
multi-cluster
deployments
and
transparently
provisions
given
this
cluster
on
a
supported
cloud
provisioner
via
cluster
api
and
then
joins
these
clusters
to
the
federation,
the
scales,
the
number
of
worker
nodes
as
necessary
and
finally
removes
the
cloud
cluster.
If
it's
not
needed
anymore
after
a
certain
amount
of
time,
the
manifest
file
is
shown
at
the
right
here.
We
have
to
specify
the
credentials
for
our
cloud
provider
and
additional
information
such
as
cip
of
the
load,
balancer
and
so
on.
A
The
implementation
of
mckay
tests
was
done
on
using
cop,
which
is
kubernetes
operators
framework
from
zalando
based
on
python.
Now,
let's
go
to
a
short
demonstration
of
mckates
in
action.
In
this
demo
we
have
one
management,
cluster
and
five
workload
clusters
in
the
grid,
5000
experimentation
test
paid.
We
have
a
management
cluster
in
rain
and
five
worker
workload
clusters
in
ren,
nant,
lil,
luxembourg
and
granola.
We
also
have
an
openstack
cluster
in
north
c,
which
acts
as
the
cloud
cluster
on
which
we
will
transparently
provision
the
kubernetes
cluster
when
necessary.
A
Each
of
our
five
clusters
have
a
master
and
five
worker
nodes
for
the
sake
of
heterogeneity
clusters.
One
and
five
their
nodes
have
four
cpu
cores
and
16
gb
of
ram
each
and
the
nodes
of
clusters.
Two
three
and
four
have
two
cpu
and
four
gb
of
ram
each
the
table
shows
the
inter
cluster
network
latency
between
these
clusters.
The
prerequisites
for
d
for
this
demo
are
the
management,
cluster
and
few
workload
clusters,
as
shown
here.
A
We
also
need
kubernetes
federation
prometheus
cluster
api
on
our
management
cluster
and
our
core
clusters
need
to
have
celium
and
celium
cluster
mesh
serve
as
well
as
prometheus.
We
also
need
the
credentials
for
a
cloud
provider.
In
our
case
we
have
used
openstack,
but
for
that
matter
we
can
use
other
card
providers
such
as
aws,
google
cloud
and
so
on,
and
if
we
like
to
have
internet
for
intel
cluster
network,
latency
and
load
balancing,
we
need
physical
or
virtual
networks
such
as
vpn
across
our
clusters.
So
now,
let's
go
to
the
demo
in
this
demo.
A
A
So
to
check
that
I'll
run
the
command
cube,
cuddle,
get
cube,
fit
clusters
in
the
cube
federation
system,
namespace
on
the
management
clusters,
and
we
see
that
we
have
five
clusters
that
form
the
federation.
Next,
let's
check
the
status
of
psyllium
cluster
mesh.
To
do
this,
I
will
face
my
cucu
command
into
one
of
the
clusters
cluster,
one
in
this
case
and
first
check
the
presence
of
the
celium
parts.
A
We
will
use
cluster
api
to
provision
a
cloud
cluster
from
openstack
in
this
case,
but
we
can
use
different
providers
for
that
matter.
So,
let's
check
the
presence
of
the
openstack
cluster,
so
I'll
run
the
command
open,
openstack
catalog
list-
and
I
see
the
details
of
my
openstack
cluster,
but
we
can
see
that
we
don't
have
any
servers
at
the
moment.
A
So
this
should
be
our
pc
and
we
see
that
there's
a
serv
cluster
as
well.
So
now
it's
time
to
deploy
our
custom
resources.
We
have,
in
this
demo
we'll
deploy
four
custom
resources
which
are
multi-cluster
deployment,
multi-cluster
service,
multi-cluster
jobs
and
cloud
provisioners
and
auto
scanner.
So
these
are
the
custom
resources
that
our
multi-cluster
scheduler
and
cloud
provisioners
use
later
on.
So
first,
I
will
deploy
the
first
three
custom
resources,
so
crt
multi-cluster
deployment,
multi-cluster
service
and
multi-cluster
jobs.
A
So
this
is
deployed
as
a
normal
kubernetes
deployment,
as
we
can
see
in
the
specification
file
here,
we
would
like
to
deploy
this
on
the
master
node
of
the
management
cluster.
So
for
that
reason
we
specify
a
node
selector
and
the
necessary
tolerations.
We
would
also
need
a
service
account
and
the
necessary
role-based
access
control
so
that
this
controller
has
the
necessary
privileges
to
do
its
functions,
so
we
deploy
the
r
back
and
the
deployment.
A
A
A
Great,
so,
let's
check
now
whether
the
parts
are
running
so
the
multi-cluster
scheduler
is
running
and
the
part
for
the
cloud
provisioner
is
being
created
as
we
speak,
so
it
looks
great
now,
so
what
we
can
do
now
is
we
can
deploy
multi-clusters
applications,
so
I'll
show
three
scenarios.
So
the
first
scenario
is:
I
will
try
to
use
our
multi-cluster
scheduler
to
deploy
applications
on
a
specific
cluster
and
then
I'll
show
the
horizontal
offloading
capability.
A
What
this
means
is,
if
the
we
don't
have
sufficient
resources
in
one
cluster,
our
scheduler
is
able
to
offload
the
applications
to
a
nearby
cluster
by
estimating
the
network,
latency
and
third
I'll
show
the
bursting
capability.
So
what
this
means
is
if
a
cluster
cannot
cannot
place
all
the
replicas
of
a
deployment
we
will.
Our
scheduler
will
deploy
the
extra
replicas
to
a
nearby
cluster
and
then
I'm
also
show
bursting
into
the
cloud.
A
A
So
one
thing:
first,
we
have
to
create
a
cloud
provisioner,
so
we
have
the
manifest
file
here
that
contains
the
necessary
information
about
the
cloud
cluster
in
this
case,
openstack
the
credentials
and
other
important
information,
such
as
cip,
address
for
the
load,
balancer
and
so
on.
So
we
create
this
cloud
provisioner.
A
All
right,
this
is
created
great,
so
now,
let's
deploy
our
first
application.
So
this
is
a
simple
application
that
prints
just
hello.
So
we
would
like
to
deploy
this
application
on
one
of
our
clusters,
in
this
case
cluster
2,
and
as
if
you
notice
the
manifest
file
for
a
multi-cluster
deployment
is
very
much
similar
to
that
of
a
normal
kubernetes
deployment.
The
only
difference
is
the
api
version
and
kind,
and
there
are
a
few
additional
fields
under
the
spec
section.
A
A
A
A
So
for
some
reason,
if
we
want
to
increase
the
resource
request
for
for
our
for
our
parts
in
this
case,
let's
increase
it
to
three
cores.
We
know
that
clusters
two
does
not
have
nodes
that
have
three
cores,
so
our
scheduler
will
try
to
find
another
cluster
that
has
three
or
more
cores
and
we'll
try
to
deploy
this
this
deployment.
A
So
let's
verify
that
so
I
will
apply
the
changes
so
now,
let's
check
the
status
of
our
multi-cluster
deployment
resource
and,
as
you
can
see
here
on
the
update
section
of
the
status,
we
now
see
that
the
deployment
is
now
running
on
cluster
one
and
not
cluster
two,
and
we
have
five
replicas
running.
So
let's
verify
this
so,
as
you
can
see
the
deployments,
the
parts
are
now
running
on
cluster
one.
So
the
reason
is
that
clusters
two
does
not
have
a
sufficient
resources.
A
So
our
multi-cluster
scheduler
has
now
deployed
this
deployment
on
the
nearest
cluster
two
clusters:
the
original
cluster
two,
which
is
cluster
one
next
I'll,
show
the
bursting
capability.
So
again,
let's
edit
the
manifest
file.
This
time,
let's
increase
the
number
of
replicas
from
5
to
10,
and
since
cluster
2
only
has
5
nodes,
it
cannot
place
on
these
10
replicas
of
the
application,
so
our
scheduler
will
try
to
deploy
the
extra
5
replicas
on
another
cluster
that
has
the
sufficient
resources
and
is
also
closer
to
cluster
2
in
terms
of
network
latency.
A
A
A
A
And
let's
see
what
what
happens
now
so,
what
happens
now
is
we
don't
have
sufficient
resources
on
our
five
fixed
clusters?
So
when
the
schedule
our
multi-cluster
scheduler,
cannot
deploy
all
these
replicas,
it
will
update
the
status
of
the
multi-cluster
deployment
with
a
message
saying
that
kind
of
deployed
and
it
needs
to
provision
a
cloud,
a
cloud
cluster.
A
So
that's
where
our
cloud
provisional
controller
comes
in,
so
it
will
check
the
status
of
the
multicluster
deployment
and
it
will
create
a
new
kubernetes
cluster
on
openstack
joins
it
to
the
kubernetes
federation
and
then
once
the
new
cluster
is
ready.
Our
multi-cluster
scheduler
will
deploy
the
extra
replicas
on
that
new
cluster.
So
let's
apply
this
change
and
let's
check
the
status
of
our
multi-cluster
deployment.
A
So,
as
you
can
see
here,
there
is
a
status
up
update.
So
there's
this
message
that
says
the
application
could
not
be
deployed
on
the
fixed
clusters
and
we
need
to
provision
a
cloud,
a
cloud
cluster
and
this
this
message
to
to
cloud
so
now
our
cloud
provisioner
will
create
the
kubernetes
cluster
on
openstack.
So
let's
go
to
openstack
and
check
if.
A
A
A
So
so
we
see
that
our
cloud
provisioners
has
indeed
created
a
cloud
cluster
and
joined
it
to
the
federation
when
our
scheduler
realized
that
it
did
not
have
sufficient
resources
on
the
fixed
clusters
to
place
our
our
deployments.
Our
cloud
provisioner
can
also
auto
scale
the
worker
nodes
of
the
cloud
clusters
and
even
remove
the
cloud
cluster
when
the
workload
has
decreased
and
that
cloud
cluster
is
not
needed
anymore.
After
a
certain
amount
of
time.
A
Next
I'll
show
the
automated
placement
capability
of
our
multi-cluster
scheduler.
For
this
I
have
another
deployment.
So
this
time
we'll
use
the
best
fit
placement
policy,
so
the
best
fit
placement
policy
tries
to
deploy
applications
on
those
clusters
that
have
been
used
the
most.
So
this
is
a
bin
packing
algorithm,
so
it
tries
to
utilize
resources
as
much
as
possible.
A
So,
as
you
can
see,
the
deployment
has
been
deployed
on
clusters,
one
and
five.
These
are
the
clusters
among
all
our
clusters
that
have
been
used,
the
most
and
list.
Let's
verify
so
we
can
see
that
the
parts
have
been
have
been
created
on
clusters,
one
and
five.
Another
placement
policy
is
the
westfield
policy,
so
what
this
does
is
it
will
try
to
deploy
applications
on
the
cluster
stuff
that
have
the
most
free
resources
available.
Similarly,
in
this
case,
we
would
like
to
deploy
on
two
clusters
using
the
worst,
a
policy.
So
let's
try
this.
A
A
And,
let's
verify
once
more
so
as
you
can
see
the
rest
with
the
deployments
using
the
wasp
policy
have
been
created
on
clusters,
two
and
three.
So
the
last
thing
I
want
to
show
you
is
how
to
deploy
the
multi-cluster
service
controllers
and
how
we
can
and
how
we
can
access
the
back
end.
The
first
back-end
application
using
a
front-end.
So
for
this
we
first
need
to
create
a
multi-cluster
service
corresponding
to
our
multi-cluster
deployment
named
hello.
A
So
this
in
this
case,
we
don't
have
to
specify
the
clusters,
because
our
scheduler
will
find
the
corresponding
cluster
deployments
in
mdl
and
it
will
create
the
corresponding
community
services
on
all
the
clusters
that
have
the
deployment.
So
let's
apply
this
and
let's
verify
whether
these
services
have
been
created.
A
So,
as
you
can
see,
the
hello
services
have
been
created
on
clusters,
one
and
five.
We
can
also
check
on
the
cloud
cluster
whether
the
services
have
been
created,
so
it's
created.
Now
we
need
a
front-end
deployment
and
service
to
access
the
application.
So
we
have
content
multi-cluster
deployment
that
we
we
need
to
deploy
this
front
end,
at
least
on
one
of
the
clusters
that
contain
our
back-end
multi-cluster
deployment.
A
A
And
let's
check
the
status,
so
the
front-end
service
have
been
created
on
cluster
one.
Let's
check
the
part,
the
front
end.
Front-End
deployment
have
been
created
on
clusters,
one
and
two.
This
is
because
we
have
specified
five
replicas
and
since
our
scheduler
could
not
deploy
all
the
five
replicas
of
cluster
one,
it
has
created
only
three
of
the
replicas
and
has
a
burst.
The
extra
two
cl
two
replicas
to
cluster
two,
so
this
is
okay.
A
A
So
in
this
demo
I
have
shown
you
how
you
can
deploy
the
custom
resources
and
controllers
of
mc
kats
and
how
using
these
controllers,
you
can
deploy
multi-cluster
applications
and
services.
I
hope
you
have
enjoyed
this
demo
and,
if
you'd
like
to
see
more
demos,
you
can
take
a
look
at
our
github
repository.
Thank
you.