►
Description
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects - Learn more at https://kubecon.io
Turtles All the Way Down: Inception for Kubernetes - Hardik Dodiya, SAP
Kubernetes has evolved into the de-facto platform for managing cloud-native workloads with consistent development and operational experience across infrastructure providers. But there is one workload which Kubernetes doesn't normally manage: Kubernetes itself! We study the design, benefits, and challenges of the INCEPTION model: treating Kubernetes as yet another application for Kubernetes to run. This is accomplished with standard means (eg, StatefulSets, Services, Machines via Cluster-API) and is productively used in SAP's open-source project Gardener. By enabling Kubernetes as an INCEPTION platform, we drastically reduce the management complexity by avoiding imperative toolsets, whilst extending the capabilities and scope of Kubernetes itself. Lastly, we show a live demo of how this works in practice. We also discuss opportunities for this work in the sig-cluster-lifecycle.
https://sched.co/YVxW
A
Hello,
everyone
I
hope,
no
I'm
horrible
right
perfect.
Finally,
soon
the
last
tag
of
the
kubernetes
forum,
Bangalore
I,
can
see
the
places
Irene
has
gone
to
the
containers.
Everyone
has
gone
to
the
kubernetes
inside
the
cave
where
it
is
outside
of
the
communities,
but
this
last
dog
is
gonna,
be
slightly
above
all
this
layer.
It's.
A
A
I
hope
these
listen
okay.
So,
let's
start
with
the
introduction,
so
my
name
is
Karthik
I
work
for
ICP,
since
last
three
and
half
years
I've
been
part
of
the
curators
community,
also
from
since
since
they're
half
years
and
I
work
for
a
project
which
is
called
Project
Cardinal.
So
how
many
people
have
heard
of
the
project
Carter
before
okay,
interesting
you're,
gonna,
you're
gonna
have
some
cool
stuff
to
know
today.
So
before
we
get
into
the
details,
I
would
like
to
discuss
something
on
a
past
couple
of
questions.
A
As
expected,
one
of
you
how
many
of
you
tens
of
clusters
like
ten
clusters,
twenty
plus
forty
clusters,
still
not
many,
how
many
open
hundreds
of
listeners
interesting?
Okay,
so
inspecting
off!
You
see
a
one
person
doing
everything
which
is
probably
not
a
best
way
of
doing
the
kubernetes.
If
you
have
many
clusters,
this
talk
is
about
you,
so
as
famously
this
is
from
the
movie
called
inception
and
inception
was
one
of
the
extremely
mind-boggling
movie.
But
don't
worry
this
talk
is
not
going
to
be
that
way
involved
this
inception.
A
This
is
about
the
inception
but
inception
for
the
cubicles,
and
this
is
very
famously
known
as
key
perception
in
the
industry
right
now.
So
let's
get
started.
I'll
just
refer
to
start
from
the
scratch,
so
a
very
simple
Cuban.
It
is
cluster.
What
does
it
get
brings
an
API
server,
scheduler
manager
at
sitting,
and
then
you
get
to
work
on
the
workers.
We've
got
queue,
proxy,
accumulates
and
so
on.
This
is
a
very
simple
structure:
I
like
it.
A
If
I
have
to
do
with
more
stuff,
if
I
want
highly
available
stuff,
I
would
say:
ok,
we'll
bring
more
mono
machines,
couple
of
more
master
machines
and
then
I
would
say:
ok,
I
need
more
worker
machines
as
well,
and
this
brings
us
a
good
happy
human
ideas,
question
or
Ben
happiest
person
in
the
world.
If
I
have
to
deal
with
only
these
things,
one
cluster
and
just
do
everything
with,
but
is
that
all
the
we
need
actually
what's
probably
not?
A
This
is
just
the
starting
I
think
this
when
you,
when
you
play
with
Cuban,
it
is
your
day.
One
is
this:
what
really
happens
is
that
on
day
two
day,
three,
you
create
more
clusters
for
different
landscapes
or
tasting
you're.
Getting
tens
of
clusters
in
you
usually
think
you're
getting
hundreds
of
clusters
and
eventually,
probably
if
you
have
a
much
larger
scale,
you
end
up
getting
thousands
of
communities
clusters
and
that's
where
the
whole
ball
game
starts.
A
So
now
you
want
to
have
this
clusters,
but
you
also
want
to
manage
a
certain
production
quality,
sound
comfort
them
right,
I
mean,
for
example,
if
you
have
so
many
clusters
running
and
replying
Dave,
you're,
sleeping
or
you're
on
vacation
and
one
clusters,
disk
is
corrupted
or
docker
is
hungry
in
one
of
the
cluster
or
cubelet
is
not
responding
in
one
of
the
machine
of
one
of
the
clusters.
So
you
know
anybody
sure
you
want
to
manually
go
there
in
this
stuff,
so
what
you
would
expect?
It's
some
kind
of
automation.
A
You
would
expect
that
everything
should
be
taken.
Care
of
Cuba
notice
is
bringing
a
new
era
of
managing
applications,
but
there
is
something
which
is
missing
yet
it's
the
management
of
the
infrastructure
itself,
the
management
of
how
long
exactly
many
securities
clusters
itself.
We
would
also
probably
want
to
have
a
nice
order
scanning
the
disaster
recovery,
which
is
begins
to
checkmark
in
the
industry,
said
if
your
entire
region
goes
down
and
our
data
center
goes
down.
What
will
you
do
you
want
to?
A
You
want
a
system
which
should
be
reliable
enough,
which
would
be
great,
which
it
should
be.
It
should
be
good
enough
to
deal
with
all
the
problems,
and
then
you
would
want
to
have
monitoring
logging
and
whatnot.
So,
of
course,
that
brings
a
little
bit
of
worry
to
us
that,
maybe
not
now,
maybe
tomorrow-
or
maybe
many
of
you
already
start,
seeing
it-
that
it's
hard
to
manage
many
many
clusters,
and
there
are
so
many
different
ways
to
manage
the
clusters
that
the
ecosystem
is
already
pretty
fragmented.
A
So
this
most
of
all
try
to
understand
what
do
we
exactly
need
when
we
have
many
clusters?
So
we
understand
the
problem
first
and
then
we
will
see,
for
the
solution
should
go
so
I
would
say
from
the
experiences
I
would
want.
We
have
a
universal
Kuban.
It
is
clusters
in
scale
where
my
clusters
should
be
completely
homogeneous.
So
if
I
have
in
the
cluster
across
different
clouds,
I
might
have
ten
clusters
running
on.
Aws
tentacles
are
spending
on
GCP
few
clusters
on
on-prem
and
the
private
clouds.
A
If
you
want
this
sphere
on
bare
metals,
what
not
I
would
like
to
homogeneous
clusters
across
different
clouds.
I
would
like
to
make
sure
I'm
using
a
very
fundamental
resources
which
are
standard
communities,
resources
like
vn0,
cleansers
and
so
on.
I
would
like
to
also
have
a
way
to
configure
the
operating
system.
I
would
not
want
to
have
a
system.
I
would
not
make
sure
that
I
always
know
non-core
ways.
You
know
what's
happening
the
core
OS,
if
you're
having
the
trend
is
gonna,
be
it's
gonna
be
close
so
now.
A
The
question
is
that
why
I
should
have
a
system
which
supports
multiple
operating
system?
I
should
have
a
system
which
can
manage
oculis
cluster,
irrespective
of
what
I'm
doing
today
only
holiday.
It
should
have
a
fleet
management
system.
Total
cost
of
ownership
should
be
as
nice
as
possible.
Can
a
Cypriot
management
is
basically
if
I
have
an
existing
equivalents.
Cluster
I
would
want
that
cluster
to
be
managed,
as
well
by
by
the
orchestrator
or
by
the
system,
which
I
am
bringing
Vinnie
so
now.
A
The
question
is
that
how
we
can
exactly
do
that
and
to
this
please
we
go
now
we're
going
to
talk
about
that
home,
and
you
know
these
are
the
modern
problems
and
modern
problems
is
very
well
say
it
requires
modern
solutions,
so
we
know
something
interesting.
We
have
a
very
important
information
that
we
sometimes
ignore
is
that
humanities
is
an
amazing
software
for
managing
the
application
workload.
A
If
you
have
internalized
workload,
it
just
amazing,
it
does
the
job
so
well,
if
we
know
this
information,
like
you
know,
if
we
not
learn
to
burn
IDs
by
our
communities,
yeah,
that's
where
the
inception
part
of
it
comes
that.
Why
cannot
just
not
create
the
kubernetes
as
a
yet
another
application
on
top
of
the
cuban?
It
is
it's
a
slightly
mind-boggling
wake.
You
will
see
how
it
can
actually
be
achieved
so,
rather
than
talking
the
theoretical
details,
what
I
will
do
is
that
I'll
I'd
have.
A
We
can
walk
through
from
the
very
high
level
of
an
open
sourced
project
which
exactly
implement
the
cubes
option
model
and
which
does
exactly
what
we
stated
in
the
previous
slides
with
a
universal
cubed
as
clusters
SK,
so
it's
getting
to
it
and
we're
not
talking
about
the
project.
Cardinal
was
a
gardener,
is
basically
an
open
source
project
from
SP.
It
has
been
couple
of
years
already
and
it's
basically
from
the
high
level
it.
A
Okay,
so
when
I
say
the
cubes
option,
what
do
you
exactly
mean
meant?
So,
let's
get
started
I
would
like
to
have
some
gives
us
as
well
from
you,
because
it
would
be
interesting
to
know
for
you
to
that.
As
always,
we
have
let's,
let's
assume
we
have
one
cluster
of
this
in
management
cluster,
which
is
basically
we
call
it
a
seed
cluster
which
has
few
monster
machines
running
and
which
has
three
worker
machines
running,
which
is
perfectly
fine.
Now
I'm
going
to
use
this
cluster
to
create
more
cluster
using
an
inception
water
can
any.
A
Okay,
I
think,
if
some,
let
me
give
you
some
hints
so
when
the
working
clusters,
what
we
basically
need
for
the
worker
clusters,
one,
what
is
your
cuban?
It
is
clustered.
So
basically,
your
Cuban
dispersed
areas,
mainly
your
worker
machines,
where
you
run
your
worker.
Would
you
don't
necessarily
need
to
know
what
exactly
your
control
plane
is
doing
at
any
point
in
time?
A
A
Okay,
dead
working
machines-
you
generally
for
is
a
worker
cluster.
Now
this
trend
is
not
just
trained
from
couple
of
guys.
This
is
the
industry
trained,
so
the
Cuban
leaders
community
are
closely
following.
This
is
actually
happening
as
part
of
the
community
where
we
are
having
a
management
cluster
and
we
are
trying
to
identify
different
potential
approaches
using
which
we
can
manage
multiple
other
clusters.
So
the
worker
clusters,
here,
as
you,
can
see
how
worker
machines,
but
where
are
them
controlled
planes,
and
it
gives
us
now.
A
So
you
can
see
I
have
one
clusters
already
running
well
and
I
have
already
a
worker
machines
for
the
worker
clusters.
So,
as
expected,
is
you
think
it
could
be?
Is
that
the
worker
clusters
control
plane,
basically
drones
inside
the
management
cluster,
probably
just
as
in
your
namespace.
So
what
is
the
control
plane?
It's
good,
deeper,
Co
control
plane
actually
is
nothing
but
a
set
of
ports
right
there,
just
set
of
controllers
running
somewhere.
A
So
now
you
are
saying
that
this
one
worker
machine,
which
would
have
previously
take
only
one
control
plane
on
it,
can
now
manage
multiple
deployments
of
multiple
worker
clusters,
and
that
model
is
what
we
refer
is
a
cube,
subshell
model.
But
we
run
one
cuban
in
this
cluster,
this
cuban.
It
is
clustered
on
the
management
cluster,
which
is
again
the
cuban.
It
is
again
so
this
keyboard
is
cluster
is
basically
using
the
standard
cuban.
It
is
primitives
like
deployments,
Nappanee
car
sensors
and
everything
which
is
better
tested.
A
You
know
it's
gonna
work,
it's
not
gonna
fail.
Now
we
discussed
one
one
part
of
the
solution.
Only
yet
we
talked
about
the
universal
statement
that
we
talked
about
is.
Is
it's
too
broad?
So
in
this
case
also,
if
you
see
there
is
still
a
possibility
that
one
of
the
merkel
machine
can
still
get
unhealthy
like
to
have
what
revealing
godself
inning
on
its
own,
it's
still
not
covered
here,
right
and
I
might
want
to
have
the
auto-scaling
of
each
and
every
different
worker
pools
running
in
different
different
clusters
and
so
on.
A
A
How
many
of
you
have
heard
the
term
plus
really
clear
before
okay,
interesting,
so
the
cluster
API
in
the
machine
API
are
a
very
interesting
way
of
managing
the
infrastructure
resources.
So
we
just
discuss
something
back
that
hope:
Ward
works
right,
you
create
a
pod,
and
then
there
is
a
queue
bleep
running
somewhere
which
place
is
the
port.
It
was
all
amazing.
We
know
that
it
works
for
the
communities
and
why
can
we
not
do
the
same
thing
for
the
clusters
as
well?
A
So
if
you
can
create
a
pod
30ml
and
expect
a
controller
to
create
a
container
for
you,
you
can
also
create
a
cluster
dirty
animal.
For
me,
internally,
color
shoe
brought
you
basically
a
cluster
or
tml
you
all
the
necessary
information
inside
you.
What
domain
it's
it's
for!
What
infrastructure
needs
to
run
on
what
community's
version
you
want
Tony
and
stuff
like
that,
and
if
you
can
implement
a
controller
which
can
understand
this
information
and
do
the
necessary
job
for
you.
A
Basically,
you
are
bringing
the
entire
imperative
way
of
managing
clusters
from
one
area
to
the
other
area,
which
is
a
declarative
way
of
managing
clusters
over
tomorrow.
If
you
want
to
create
ten
clusters,
you
would
only
be
creating
ten
different
shoot,
dirty
animals
or
cluster
or
animals,
and
the
controllers
will
do
the
job
in
a
similar
way.
A
So
the
machine
is,
we
say
for
the
cluster,
you
can
have
a
machine
object
where
you
mentioned
the
Machine
image,
the
type
of
the
machine
and
so
on,
and
you
basically
give
this
information
to
some
cluster
Orchestrator,
which
could
be
Gardner
or
which
could
be
what
is
being
developed
in
the
community
or
someone
else.
You
provide
this
information
to
a
cluster
Orchestrator
and
that
Orchestrator
basically
interfaces
the
information
to
the
real
clone
or
agra,
and
you
at
the
end
of
the
day
you
get
your
cluster
running.
That's
the
high
level
idea.
A
That's
the
second
part
of
this
solution
for
the
big
problem
that
we
talked
about,
but
that's
not
actually
all
we
we
have.
You
covered
the
technical
aspects
very
well
now
we
know
that,
with
the
cube
substring
model,
you
can
manage
many
clusters
control
plane
all
together.
We
make
sure
that
our
infrastructure
is
going
to
be
very,
very
taken
care
of
so
far,
but
the
third
thing
is
more
important
in
the
in
the
morning
against.
There
was
a
very
good
discussions
about
the
extensions
mechanism
in
Cuban.
It
is
right.
So
what
is
the
extension
mechanisms
physically?
A
So
you
can
think
of
it
in
a
way
that
you
have
a
certain
things
running
very
well
on
a
certain
cloud
with
certain
environment.
But
that's
not
all
you
want
if
you
want
a
larger
or
community
adoption.
If
you
want
people
to
customize
your
software,
you
need
to
be
able
to
swept
each
and
every
component
that
we
have
download.
So
if
I,
if
I
write
a
software
which
runs
well
on
AWS,
they
should
be
writing
in
a
way
that
it
should
work
exactly
in
a
similar
view
on
TCP
as
well.
A
So
I
should
be
able
to
write
a
good
extensions
for
that
right
in
a
similar
way.
The
cuban
environment
works
well
for
calico
or
core
OS.
There
should
be
a
good
extension
model
which
can
simply
swept.
Can
you
commit
panel,
maybe
over
s
with
winter
or
whatnot,
just
examples,
and
that's
where
the
power
of
the
extension
is
comes
so
the
bottom
line
is
that
three
key
things
conception,
model
plug
and
play
mechanisms
and
cluster
apne
machine
api.
A
If
you
can
combine
these
three
amazing
part
of
the
solutions,
then
you
get
the
next
generation
features
for
tier
clusters,
so
you
would
not
have
a
time
when
your
your
hundreds
of
clusters
running
somewhere,
but
something
is
going
down
some
something
unexpected
is
happening
when
you
we
go
there
and
fix
it
manually
that
those
days
are
not
gone.
You
know
I
would
expect
that
we
understand
it,
because
everything
can
feel
that
was
also
also
told
in
the
previous
talks.
A
You
have
to
come
with
the
assumption
that
everything
we
build
is
always
going
to
fail
and
our
system
should
be
reliable
enough
to
fix
them
on
their
own.
It
should
be
a
complete
automation.
So
what
are
these
next-generation
features?
We
are
then
talking
about.
So,
let's
take
a
look.
Oh
one
cluster,
let's
say
created
with
three
worker
machines
and
it's
being
managed
by
the
cluster
or
that's
my
point
now.
This
three
are
from
this
store
worker
machines.
A
As
I
say,
if
any
of
the
worker
machines
Dockery's
gets
a
honey
or
any
of
them
work
robust
uses
gone
out
of
the
blue,
this
happens
sometimes
in
the
on-ramps
right,
then
it
should
just
be
replaced
on
its
own
if
there
is
a
very
high
load
on
my
cluster.
So
if
there
are
three
worker
machines
and
if
I
see
that
there
are
many
ports
which
are
not
getting
scheduled
because
three
machines
are
not
you
know,
then
I
should
have
a
bit
auto
scaling
which
automatically
scales
it
up
in
a
similar
way.
A
I
should
have
a
put-down
scaling
as
well,
where
it
should
scale
back
to
save
the
resources
and
eventually
with
gardiner.
We
also
implemented
an
interesting
feature
of
the
ruling
updates.
Now
what
can
be
achieved?
What
new
things
can
be
achieved
comes
here
so
so
far
we
we
know
that
we
have
always
been
seeing
an
infrastructure
in
the
application
layer
of
the
Kuban.
It
is
as
two
different
layers,
but
if
you
follow
the
best
practices,
then
what
you
can
do
is
just
just
to
good.
Why?
A
Because,
let's
say
you
have
an
application
running
with
certain
version
on
given
cuboids
cluster,
and
now
you
have
set
up
some
PD
B's
on
it.
Some
SL
is
set
up
on
there
and
you're
saying
I
have
finest.
This
is
running
at
any
point
in
time,
not
more
than
two
instances
should
go
down.
That's
my
idea
to
be
maintained,
and
then
you
hit
the
security
patch.
Some
some
bug
is
found
and
you
get
the
information
from
that.
All
the
machines
has
to
be
rolled
out.
How
do
you?
How
do
you?
How
do
you
reliably
do
that?
A
This
is
actually
a
very,
very
complicated
problem,
so
we're
gonna
just
go
ahead
and
turn
on
the
machine
and
turn
on
all
the
machines.
Again,
that's
I,
don't
I,
don't
think
that's
a
great
way
of
doing
neither
you
can
just
be
so
many
new
machines
on
one
hand
and
then
try
to
migrate
all
the
applications
from
one
place
to
another
place
that
in
that
case
also,
you
will
end
up
messing
up
with
your
SNS.
So
here
when
we
talked
about
machine
API,
we
have
implemented
something
called
machine
deployment.
A
So
if
you
know
the
port,
you
must
know
port
replica
set
and
deployment
model
right.
How
deployment
does
this
things
so
nicely
with
the
ports?
You
must
have
tried
to
upgrade
your
application.
Why
are
the
deployment
controller
of
the
chip
on?
It
is
right
from
version
one
to
version
two.
Just
imagine
you
can
do
the
same
thing
with
your
machine,
so
when
machines
nets
a
while
running
some
version
of
operating
system
or
cuban
activities,
let's
say
one
point,
a
one
on
one
point
well
and
now:
I
figure
and
upgrade
so
with
the
upgrade.
A
All
these
machines
are
gonna
be
swept
one
by
one,
making
sure
that
if
there
are
fine
machines,
if
I
will
bring
down
the
first
machine,
there's
gonna
break
the
SLA,
so
I
will
not
bring
down
the
function.
First
notion
you
will
wind
on
the
second
machine.
First
I
migrated
the
workload
of
the
second
machine,
then
ever
bring
down
the
third
ocean
and
so
on.
A
So
those
kind
of
very
intelligent
logic
thing
can
be
built
upon
if
we
have
a
good
machine,
EPN
cluster
apiary
aspects
implemented
and
then,
of
course,
the
cost
any
we
we
see
this
very
often
that
when
we
run
many
many
clusters,
we
know
that
we
don't
want
to
keep
them
running
all
the
time.
So
many
clusters
are
created
by
developers
and
you're
just
for
playing
around
not
always
use.
So
many
clusters
are
just
not
required
to
be
done
all
the
time.
A
Then
we
implement
the
stuff
like
hibernation,
where
all
the
worker
machines
are
simply
brought
down.
You
can
just
bring
down
your
deployment
of
your
control
plane
and
you
see
a
lot
of
cost
here.
I
will
also
see
that
this
cluster
orchestrators
does
not
really
necessarily
be
connected
to
your
worker
machines.
All
the
time,
if
there
is
a
connectivity
issue
in
between
then-governor,
simply
assumes
or
deployments,
are
done
in
a
way
that
one.
A
If
there
is
a
connection
issue,
this
guy
will
work
really
functional
on
its
own,
so
availability
should
not
have
unnecessary
dependencies
on
each
other
and
last
but
not
least,
is
very
important
part
where
your
clusters
should
be
properly
backed
up
because,
as
I
said,
everything
in
fields,
so
if
you're
actually
is
running
on
certain
set
of
volumes,
you
may
say:
I
haven't
reacted
instances
running,
but
then
it's
very
likely
to
be
possible
that
your
volumes
of
all
of
the
city
volumes
are
simultaneously
corrupted
or
whatnot.
So
it
should
be.
A
Have
a
good
backup
and
restore
mechanisms
done
well
in
case
everything
is
gone.
Your
equity
backups
are
continuously
stole
into
the
clouds.
Any
of
the
cloud
store
could
be
s3
or
any
other
object
store,
and
it
can
be,
it
can
be
restored
immediately
and
then
you
can
take
it
in
this
cluster
back
soon
with
a
very
minimal
loss
operator.
So
that's
what
we
overall
do.
We
basically
implement
such
aspects
very
properly
in
the
project.
It's
called
project
coordinates
is
a
link.
A
A
Okay,
this
is
a
dashboard
for
project
Gardner
and,
as
you
see,
there
are
many
many
clusters
already
created
if
you
are
already
running
and
we're
concealing
and
if
you
are
already
in
the
hibernate
state.
So
today's
demo
I
want
to
show
something
more
than
what
you
have
seen.
So
you
can
simply
click
on
the
create
button
and
you
see,
as
I
said,
there
are
many
extensions
available,
so
we
have
created
the
extension
of
oil
different
clouds.
These
are
a
big
four
clouds,
open
stoically
clouds
in
the
sphere
and
so
on.
A
But
you
also
added
a
support
for
a
metal,
so
metal
is
basically
a
bare
metal
provider
in
Europe.
So
you
can
think
of
any
way
that
there
is
and
as
this
project
is
open
source
in
a
European
company
called
FIDs,
they
took
up
the
project
and
they
do
everything
on
the
bare
metal
machines.
So
they
created
a
plugin
or
they
created
the
extension
which,
which
was
headed
into
the
gardener,
and
now
gardener
can
create
the
clusters
on
the
bare
metal
mushrooms,
which
means
when
you
need
to
create
a
machine.
A
A
This
is
similar.
We,
you
can
add
the
add-ons
in
it.
You
can
have
maintenance
time.
We
know
as
we
discuss
about
the
security
patches
right,
safe
security
patches,
you
roll.
There
should
be
an
option
to
the
user.
The
user
can
say
that
only
during
this
time
period
the
operating
system
patches
should
be
Doyle
to
my
cluster,
because
other
point
in
time
I
would
not
like
to
be
likely
to
be
rolled
or
the
Kuban
dispatch.
It
should
be
automatically
or
what
time
it
should
be
rolled.
A
A
Maybe
the
metal
cluster,
as
you
can
see,
all
the
information
that
we
provided.
There
is
more
or
less
populated
here
on
top
I
mentioned
about
the
monitoring,
the
only
clogging
and
so
on,
so
very
likely.
When
you
manage
many
clusters,
you
don't
want
to
go.
You
want
to
know
how
to
manage
all
these
things
manually
with
different
clusters
right
so
with
the
automation
like
gardiner
all
these
things.
All
these
things
are
installed
on
each
and
every
cluster.
If
you
want,
you
can
opt
out
or
you
or
you
or
you
or
you
can
opt
in
separately.
A
A
Not
just
that
UI
is
good
to
have,
but
at
the
end
of
the
day
we
you
we
deal
with
Yemenis.
So
at
the
end
of
the
day,
you
can
also
just
move
the
shoe
tml.
All
the
clustering
general
terms.
You
can
modify
any
of
the
fields
of
the
shoe
PML
and
then
the
backend
reconsideration
looks
at
it
for
if
I
have
a
modify,
modify
any
of
the
networking
information
or
any
of
the
machine
type
information,
workflow
informations
and
the
backend
reconsideration
can
look
at
it
quickly.