►
Description
Don’t miss out! Join us at our upcoming events: EnvoyCon Virtual on October 15 and KubeCon + CloudNativeCon North America 2020 Virtual from November 17-20. Learn more at https://kubecon.io. The conferences feature presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Controllers at Chaos - Kesavan Subramanian & Gaurav Gupta, SAP
Gardener implements the automated management and operation of Kubernetes clusters as a service (shoot) by hosting its the control plane components (Etcd, API server, controller manager, scheduler) in a different K8s cluster called seed cluster which delivers a robust Control-Plane-as-a-Service leveraging existing Kubernetes capabilities. In this session, the speakers will cover how kubernetes in kubernetes works in production and typical challenges faced by this approach.
https://sched.co/ZejB
A
Hello,
welcome
to
kubecon.
Today,
myself,
k7
with
me,
gavru
from
sap
will
be
taking
you
to
some
of
the
interesting
things
about
our
controllers
and
garner
and
how
we
leverage
the
existing
kubernetes
capabilities
to
operate.
Kubernetes
cluster
search
scale,
let's
start
from
basics,
so
how
you
run
a
container.
A
A
Now,
let's
go
with
a
big
bang.
Now,
when
you
have
to
operate
thousands
of
such
containers,
what
will
be
the
good
solution?
The
quicken
will
be
kubernetes.
So
kubernetes
helps
you
to
architect
these
thousands
of
containers
across
across
the
kubernetes
workloads
and
then
helps
you
to
manage
those
containers.
A
Kubernetes
are
nothing
but
bunch
of
controllers,
like
controller
manager,
cloud
controller,
qpa,
server,
hct.
Now,
let's
take
some
big
organizations
where
they
want
to
operate
thousands
of
clusters.
What
is
the
solution?
We
have
in
place
to
operate
these
many
kubernetes
clusters
rather
than
just
operating
clusters.
We
need
across
cloud
providers
like
aws,
gcp,
azure
and
even
on
private
clouds.
A
A
The
other
thing
to
be
noticed
like
there
are
lot
many
cloud
providers
available
in
the
market
like
aws
gcp,
a
vsphere
azure
ali
cloud.
Each
of
them
have
their
own
design
structures
in
providing
them
provisioning,
the
networks,
volumes,
virtual
machines
etc,
but
as
a
kubernetes
developer,
we
need
a
homogeneous
clusters
so
that
this
helps
the
developers
to
be
more
cloud.
Agnostic
next
comes
for
the
operators.
A
There
should
be
some
backup
and
restore
in
place
for
the
kubernetes
clusters
itself,
so
that
this
will
help
them
in
hand
during
disaster
recovery
times,
and
people
working
on
kubernetes
are
more
familiar
with
going
off
declarative
way
of
creating
a
kubernetes
or
clothes
like
just
a
spec
which
shows
your
containers
and
how
many
replicas
you
want
to
run
and
just
deploy
it
on
kubernetes,
similar
approach.
Why
don't
we
can't
have
for
our
kubernetes
say
a
kubernetes
spec
would
be
like
giving
a.
What
is
the
kubernetes
version,
which
you
want
to
run?
A
What
is
the
worker
node
type
on
what
cloud
provider
you
want
to
run?
What
is
the
os
image
you
want
to
run
and
those
steps?
This
will
be
a
good
one.
Right
next
thing
comes
more
important,
is
about
extensibility,
a
solution
which
needs
to
be
more
extensible
to
adopt
new
features
when
needed,
and
it
should
be
extensible
to
adopt
new
crowd
providers
whenever
in
place
in
the
future.
A
Here
you
can
see
we
have
for
leader
nodes
to
make
our
for
our
control,
plane
components.
Let's
take
some
example
of
different
kubernetes
clusters
of
varying
sizes.
Here
you
could
see
there
are
few
worker
nodes
in
one
kubernetes
cluster
and
there
are
lot
of
kubernetes
worker
nodes
in
some
other
kubernetes
cluster.
A
A
A
Here
you
can
see
the
normal
kubernetes
workloads
and
our
leader
node,
which
has
the
control,
pin
components
we'll
have
one
more
cd
cluster,
that
this
is
nothing
but
another
kubernetes
cluster.
We
just
take
the
control,
pane
components
of
the
shoot
cluster
inducer
cluster
and
place
it
as
work
in
this
seed
cluster.
A
A
Here
you
could
see
garner
comprises
of
some
more
components
like
similar
to
kubernetes,
like
you,
garner,
ap,
server,
ghana,
controller
manager,
garden
scheduler
and
gardner
dashboard.
This
looks
will
take
into
a
simple
approach.
What
we
want
to
deliver
we
want
here,
you
can
see
everywhere.
We
speak
about
kubernetes,
not
nothing.
Apart
from
that
here,
we
delegate
we
delegate
only
the
worker
notes
to
the
end
users
and
the
garner
will
take
care
of
managing
the
worker
nodes
and,
as
well
as
the
control
plane
components.
A
Let's
speak
about.
Is
this
more
simple
architecture
light?
Let's
add
some
kayas
in
it
here
you
could
see.
This
is
a
complete
garner
architecture
which
we
comprise
of
all
control,
plane,
components
of
garner
and
the
controlling
components
of
particular
suit
clusters
and
how
they
communicate
with
each
other.
A
B
Yeah,
so
let's
have
a
recap
of
how
kubernetes
works
so
that
we
can
draw
parallel
from
there
for
the
design
of
a
gardener
solution.
So
we
know
that
we
have
a
cube
api
server
running
in
the
kubernetes
cluster
and
we
also
have
cubelets
running
in
each
of
the
nodes.
Now,
when
a
user
wants
to
create
a
port
it,
the
user
can
provide
a
pod,
yaml
or
resource
yaml.
Cubescheduler
can
schedule
this
part
in
in
a
node
and
then
cubelet
can
take
care
of
running
the
part
in
the
in
the
node
sim.
B
On
the
similar
lines,
we
have
gardner
api
server,
which
which
manages
the
custom
resources,
and
we
have
garden
lids
which
are
running
in
each
of
the
seed
cluster
seed
clusters.
Now,
when
a
user
provides
a
shoot
resource,
yaml
shoot
custom
resource
yaml,
which
is
nothing
but
a
kubernetes,
create
a
cluster
request
from
the
user.
The
gardner
scheduler
can
pick
up
the
pick
up
this
request
and
can
schedule
the
control
plane
of
the
shoot
cluster
in
a
seat.
B
So
this
garner
controller
manager
itself
is
enough
to
manage
lots
and
lots
of
kubernetes
cluster,
but
for
a
true
scalability,
we
would
want
to
distribute
the
logic
of
deploying
the
control
plane
components
in
the
seat.
Cluster
we
take
the
learnings
from
cubelet.
Cubelet
is
a
primary
node
agent,
which
runs
on
every
node
and
is
responsible
for
keeping
a
pod
running
and
healthy.
B
Similarly,
a
garden
lit
is
also
a
garden
agent
which
runs
on
each
of
the
seat
cluster
and
can
manage
the
shoot
clusters
in
which
are
running
or
which
are
responsible
which
are
running
in
the
seat
cluster.
It
takes
over
the
job
from
garner
controller
manager
for
reconciling
the
shoot,
controller,
shoot
cluster
resources.
B
B
It
is
not
necessary
to
run
this
garden,
let
in
inside
a
seed
cluster
as
long
as
it
can
talk
to
the
seeds
api
server,
which
is
which
is
running
in
the
gardner
cluster.
It
also
opens
up
the
doors
for
running
a
shoot
clusters
under
firewall,
since
the
control
plane
is
managed
outside
separately
by
gardner.
B
Now,
since
we
also
want
to
provide
extensibility,
which
means
that
we,
it
should
be
very
easy
for
a
gardener
provider
to
add
support
for
multiple
cloud
providers,
we
want,
they
are
something
called
extension
controllers.
These
are
cloud
specific
controllers
which
know
how
to
talk
to
the
cloud
providers
and
manage
the
dns,
vpcs
or
other
infrastructure
needs.
B
Now
the
extension
controllers
template
can
be
provided
as
a
custom
resource
to
the
gardener,
and
then
gardner
can
take
care
of
for
deploying
these
extension
controllers
in
the
seat
cluster,
so
that,
when
a
request
comes
for
a
shoot
cluster
for
a
specific
cloud
provider.
In
that
case,
these
controllers
can
deploy
the
required
infrastructure
in
the
in
in
the
cloud
providers.
B
Now
kubernetes
does
not
have
a
way
or
it
does
not
support
a
way
to
create
a
provision
or
deprovision
machines
in
the
infrastructure.
So
we
rely
on
a
component
called
machine
controller
manager
which
works
on
the
same
principles
as
of
for
a
cube
controller
manager.
So
it
provides
a
declarative
way
of
managing
the
vms
of
creating
the
vms.
B
So
in
the
same
way
that
we
have
deployment
controller,
we
have
machine
deployment
controller
as
part
of
machine
controller
manager,
machine
deployment,
custom
resource
takes
the
number
of
replicas,
that
of
the
machines
that
needs
to
be
provisioned,
and
it
also
takes
a
machine
template
which,
which
includes
details
about
the
machine,
the
size
or
the
os
image
that
that
the
machine
should
have
this
machine
deployment
controller
then
creates
a
machine
set
control.
A
machine
set
object,
this
machine,
such
object,
is
picked
up
by
the
machine
set
controller
and
it
creates
the
machines.
B
B
B
We
also
rely
on
cluster
auto
scaler,
which
we
have
for
the
cluster
autoscaler
and
adapted
it
to
work
with
machine
deployment
objects.
With
this
we
can
auto
scale
our
shoot
clusters,
basically
the
user
clusters
and
the
seat
cluster,
which
is
managing
the
shoot
clusters
themselves,
but
a
potential
problem.
There
could
be
that
if
a
hcd
pod
is
scheduled
on
a
node
that
is
getting
scaled
down,
it
can
lead
to
long
downtimes.
B
B
Some
of
the
components
like
ubpi
server
can
scale
both
horizontally
and
vertically,
but
since
vp
and
hpa
don't
work
well
together.
When
it
comes
down
to
the
cpu
and
memory
matrix
alone,
we
we
rely
on
a
controller
that
is
hvpa
controller.
It
provides
a
way
to
it,
provides
a
user
with
a
way
to
provide
the
ways
how
hp
and
vpa
recommendations
can
be
used
together
using
a
weight-based
scaling,
so
user
can
provide
weights
that
needs
to
be
given
to
hp
and
vp
recommendations
it.
It
also
does
some
extra
things.
B
It
also
provides
some
extra
flexibility
and
functionality.
So,
for
example,
a
user
can
provide
the
thresholds
which
can
be
used
to
so
which
provides
a
way
for
hvpa
controller
to
apply
the
recommendations
only
when
certain
delta
or
the
changes
is
reached
in
the
recommendations
by
hp
or
vpa
it
also.
We
can
also
provide
scaling
policies
which
can
with
which
user
can,
for
example,
mention
a
time
slot
or
during
which
only
a
certain
kind
of
scaling
should
take
place.
B
So,
let's
see
how
all
of
this
comes
together
to
provide
resilience
and
a
disaster
recovery
scenario,
so
in
case
a
shoe
cluster
has
issues.
So
if
the
user's
end
users
cluster
has
issues
in
those
kind
of
cases,
the
kubernetes
itself,
since
all
of
the
control
pane
components
are
running
in
pods
and
is
managed
by
kubernetes,
the
control
plane
parts
can
be
brought
back
up
by
the
kubernetes
itself.
B
So
in
case,
the
seed
cluster,
which
hosts
the
control
planes
of
many
many
shoot
clusters
itself
is
down.
So,
in
those
kind
of
cases
we
we
rely
on
the
series,
backup
and
restore
functionality.
So,
for
example,
let's
say
we
have
two
seed
clusters
and
one
of
the
seed
cluster
has
the
hosts
the
control
plane
of
the
shoot
cluster.
Now
this
control
plane,
the
state
of
the
cluster
itself,
is
backed
up
in
a
block
store
and
in
case
a
seed
goes
down.
A
Yeah,
so
here
you
can
see
how,
as
simple
as
that
to
create
a
kubernetes
cluster
with
our
garner
beautiful
dashboard,
just
to
mention
what
is
the
cluster
name,
and
this
is
a
given
the
worker
nodes.
What
is
the
machine
type
and
where,
on
the
cloud
provider
you
just
select
and
gana
dashboard,
gives
you
the
cluster
in
few
minutes?
A
Yeah
here
you
can
see
a
short
spec
of
shoot
custom
resource
here
you
can
see
the
add-on,
kubernetes,
dashboard,
ingress
controller
and
the
cloud
profiler
is
mentioned
as
aws
gonna
provide
a
feature
of
adding
hibernations
as
well,
so
that
you
can,
for
the
end,
users
can
save
the
cost
for
creating
clusters.
Next
comes
for
the
networks.
Here
you
can
mention
the
network
types
and
the
controller
worker
groups.
What
is
the
machine
type?
You
need.
What
is
the
machine
class?
You
want
to
create
a
number
of
ochre
nodes.
A
That's
it.
We
just
apply
the
shoot
yaml
in
the
above,
and
you
can
see
the
garnet
cluster
where
we
create
a
shoot
custom
resource
and
below
pen.
You
could
will
have
a
watch
on
the
control
pin
components
which
will
be
provisioned
in
one
of
the
cd
cluster.
You
just
need
to
have
a
watch
on
the
namespace
in
the
upper
pan.
You
could
see
a
progressing
bar
for
having
the
how
much
the
cluster
has
been
created.
Here.
A
A
Here
you
could
see
cloud
controller
manager
and
csi
driver
in
cash
loop,
backup
because
the
qb
server
is
still
not
a
trending.
So
how
how
like
governor
mentioned,
the
dependency
watchdog
will
keep
on
watching
this
crash
loop
back
of
parts
and
once
the
qbp
server
is
up
and
running,
it
just
kills
the
cloud
controller
manager
and
those
things.
So
now
we
could
see
the
replica
set
replaced
with
the
new
controller
manager
without
waiting
for
us
to
go
for
an
exponential
crash
loop
backup.
A
Now
we'll
wait
for
a
few
more
minutes.
We
just
made
a
speed
up
so
that
we
can
have
a
quick
view
of
where
the
cluster
is
created.
You
can
see
a
progress
is
at
87
percentage
and
almost
all
the
control
plane
components
like
ap
server,
cube
controller
manager.
Our
mission
controller
managers
are
ready,
and
now
the
cluster
is
ready
to
use.
A
So
with
garner,
we
are
able
to
deliver
fully
managed
homogeneous
clusters
everywhere
at
sap
we
operate
thousands
of
kubernetes
clusters
across
multiple
cloud
providers
like
alibaba,
aws,
microsoft,
azure
and
even
on
specific
government
cloud
as
well.
We
run
workloads
like
hana,
which
is
an
in-memory
database.
A
A
A
A
A
Next,
how
do
you
support
on-prem
clusters,
so
gardner
provides
an
whatever
ready,
extensibility
to
profile,
to
offer
across
different
cloud
providers,
and
we
can,
even
if
you
have
your
own
cloud
providers
or
anything,
it's
just
an
extensibility
to
create
your
own
cloud
provider.
Extensions
and
we
can.
You
can
be
even
gonna
canon
and
non-prominent
clusters.
A
Okay
does
garner
work
with
other
os
like
flat
car
tallows,
so
there
is
an
option,
so
it
garner
is
completely
extensible
as
a
gun,
as
gauro
told
you
can
bring
your
own
os
inside
this
right.
Now
we
have
our
own
specific
os
like
garner
linux,
and
it
can
be
on
any
other
os
systems
for
the
occur
notes
can
be
added
to
garner.
B
Hi
yeah,
I
I
had
a
network
issue,
so
I
am
back
so
I
I'll
also
try
to
go
through
some
of
the
questions.
So
one
is
how
does
gardner
compare
to
cluster
api,
so
we
have
concrete
plans
to
be
compliant
with
last
cluster.
Api
cluster
api
today
supports
extensible
control,
plane
providers,
so
the
default
is
right
now
cube
adm
and
the
plan
is
to
extend
the
control
plane
provider
crd
with
gardner
provider.
A
Manage
kubernetes
cluster
version
upgrades
yeah.
How
does
garner
manage
a
kubernetes
cluster
version
upgrade,
so
you
can
see
from
the
thing
whatever,
whichever
the
upstream
things
coming
from
kubernetes
releases
garner
make
sure
it's
also
as
part
of
garner.
We
take
the
kubernetes
release,
oceans
and
keep
upstream
to
the
governor,
so
you
can
get
the
upstream
kubernetes
resources
on
garner
as
well.
B
Okay,
another
question
is:
do
the
shoot
cluster
need
to
be
homogeneous
or
they
can
be
from
multiple
managed
providers?
So,
as
of
now,
the
shoot
clusters
can
only
be
configured
a
bunch
single
shoot.
Cluster
can
only
be
configured
in
one
cloud
provider,
but
one
gardner
cluster
can
manage
to
shoot
clusters
across
cloud
providers,
so
you
can
at
the
same
time
create
multiple
kubernetes
clusters,
which
can
be
managed
by
a
single
gardner
cluster
which
can
span
across
cloud
providers
or
even
bare
metal.
So
the
support
is
there
for
bare
metal.
A
Yeah
one
more
question:
is
digital
ocean
supported
yeah?
There
are
some
pull
requests,
I
believe
which
are
on
the
part.
We
can
it's
just
like
how
you
onboard
a
new
cloud
provider.
We
just
have
to
create
an
extension
for
the
particular
pro
providers
and
it's
definitely
possible
to
onboard
work
with
a
digital
ocean
as
well.
B
So
once
the
recommendations
from
hp
and
vpa
are
in
hvpa,
controllers
takes
those
recommendations
and
the
idea
is
to
scale
according
to
a
a
spec
that
is
provided
by
the
user
to
the
hvpa
controller,
and
so
this
is
how
we
we
have
been
trying
to
run
both
hp
and
vpa
together
and
until
now
it
we.
We
are
very
happy
with
the
way
we
were
able
to
make
it
work
and
it's
and
the
results
have
been
promising.
A
B
A
B
B
A
One
more
question:
if
I
want
to
want
a
short
cluster
to
be
upgraded
to
your
new
cube
version
on
the
run
without
downtime
does
garner
automate,
the
upgrade
process
can
garner,
do
it
in
a
canary
fashion
so
that
the
version
upgrade
is
rolled
back
in
case
of
trouble
yeah.
What
happened
means
they
might
have
seen
in
gaurav.
A
Gauros
speech
like
that
is
a
component
machine
controller
manager
which
takes
care
of
draining
the
nodes
and
setting
the
workloads
on
the
another,
bringing
up
the
new
worker
nodes
with
the
particular
kubernetes
version,
so
it
works
seamlessly
as
a
canary
version
in
garner
it's
just
in
the
dashboard.
You
just
have
to
just
press
on
a
button
press
to
upgrade
your
kubernetes
cluster,
and
you
can
even
mention
the
maintenance
window
time
where
the
cluster
upgrade
has
to
be
happen.
B
B
B
B
Yeah,
I
I
don't
see
if,
in
case
like
I,
have
miss,
I'm
missing
some
of
the
questions.
If
we
are,
if
we
have
missed
some
of
the
questions,
please
feel
free
to
reach
out
on
the
slack
channel,
or
we
can
also
reach
out
to
the
gardener,
our
team,
from
the
slack
channel
link
that
is
provided
on
the
slide.