►
From YouTube: How We Run Kubernetes in Kubernetes, aka Kubeception [I] - Timo Derstappen, Giant Swarm
Description
How We Run Kubernetes in Kubernetes, aka Kubeception [I] - Timo Derstappen, Giant Swarm
At Giant Swarm our users want fully-managed Kubernetes clusters without any limitations (incl. privileged access to the nodes). We deploy and manage these clusters either in our data center, in the preferred cloud of the customer, or even on-premise. Both for ourselves as well as for enterprise customers we need full isolation between clusters and a easy way to manage and update clusters without downtime.
In this talk we explain how we use a “mother” Kubernetes to deploy and manage fully-isolated and encrypted Kubernetes clusters for different customers or teams - aka Kubeception. Our model treats (inner) Kubernetes clusters as a third party resource and manages them with a custom controller. This way we have an automated way of provisioning and managing clusters without additional tooling or complex monitoring setups. Further, through our API, we are to be able to spin clusters up and down on demand, scale them, update them, keep track of which clusters are available, and be able to assign them to organizations and teams flexibly.
About Timo Derstappen
Timo Derstappen is CTO and co-founder of Giant Swarm. He has many years of experience in building scalable and automated cloud architectures.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
Thank
you
so
yeah
we're
running
communities
in
kubernetes
and
it's
a
little
bit
different
from.
There
have
been
talks
around
this,
where
it's
actually
about
having
a
self-hosted
communities.
This
is
a
little
bit
different
because
we
spawn
new
communities.
Instances
with
communities
and
I
want
to
tell
you
a
little
bit
about
this
and
some
backgrounds
and
motivation.
Why
we're
doing
this
so
we're
managing
communities,
clusters,
24/7
and
both
on
board
bare
metal
and
not
on
bark
and
in
the
cloud?
And
we
have
our
own
data
center.
A
So
we
have
a
data
center
in
Frankfort.
We
have
some
wrecks
there
and
we
spawn
bonitas
clusters
there
for
customers
as
a
service,
so
they
don't
have
to
worry
about
the
infrastructure
and
but
we
also
deploy
communities
instances
in
data
centers
of
our
customers,
and
that
means
yeah
either.
So
a
non-prime
deployment
for
us
could
also
be
on
AWS,
because
it's
the
account
of
a
company
and
it's
inside
their
VPC
infrastructure
and
a
little
bit
different
from.
A
How
would
we
how
we
would
run
it
on
AWS,
and
so,
but
it
happened
to
be
that
in
those
deployments
we
are
also
running
more
than
one
communities
cluster,
and
there
are
a
couple
of
reasons
for
that,
because
you
know
soft
multi-tenancy
sometimes
is
not
enough.
There
are
companies
that
are
big
enterprises
that
have
to
gain
some
trust
into
the
new
technology
to
actually
understand
that
they
can
run
workloads
on
the
same
cluster
that
they
have
been
separating
very
carefully
in
the
past.
A
So
there
are
different
reasons,
so
sometimes
it's
just
that
praat
workloads
shouldn't
be
interfered
by
testing
workloads,
so
you
want
to
have
separate
clusters
for
that.
Sometimes
there
are
teams
that
are
working
on
different
services
that
sometimes
are
like
working
with
privacy
data,
sometimes
with
financial
data,
and
those
seems
teams
should
be
separated
and
that's
why
we
end
up
having
multiple
clusters
in
those
employ
on
prime
deployments
as
well
and
yeah.
A
We
believe
that
people
must
come
to
the
to
things
in
their
own
time
in
their
own
way
for
their
own
reason,
or
they
never
truly
come
at
all.
So,
in
the
end,
we
encourage
everyone
to
use
our
back
network
policies
because
it
doesn't
make
sense
to
have
small
clusters,
because
if
you
have
like
death
protests
and
and
for
each
and
every
team,
then
the
end
of
having
a
lot
of
machines,
but
very
small
clusters
on
it
and
in
the
end
that
doesn't
make
sense.
A
I
mean
if
you
tear
down
one
node
of
a
three
node
cluster
30
percent
of
your
cluster
is
down
and
that's
no
fun
and
yeah
and
obviously
those
those
technologies.
We
have
one
one
at
six
now
we
already
have
our
back
enabled
in
one
at
five
and
it's
now
you
know
it's
maturing.
It's
coming.
People
can
adapt
their
processes,
they
will
learn
how
it
works,
and
so
we
hopefully
don't
run
too
many
communities
instance.
A
But
on
the
other
hand,
you
can
also
easily
spin
up
a
new
cluster
and
cluster
with
a
new
version
to
test
your
build
chain,
because
you're
Jiang
Jenkins
is
may
be
integrated
with
Corinne
with
the
communities
API
and
you
need
to
this
on
some
environment.
So
it's
just
one
click
to
install
a
new
version
and
test
your
bill
chain.
A
So
introducing
giant
Nietzsche's
and
that's
our
like
main
communities
we
use
and
up
the
motivations
rather
obvious.
We,
you
know
providing
communities
for
people
and
we
thought
using
ansible,
for
it
doesn't
make
any
sense
for
for
us.
We
want
to
use
the
same
thing,
we're
providing
to
others
and
yeah
so
definition,
because
it's
on
a
couple
of
slides
and
a
couple
of
diagrams
so
g8s
is
the
host
cluster.
It's
the
giant
eaters
and
k-8
is
the
kubernetes,
the
guest
cluster.
We
provide
to
the
end
users
and
don't.
A
Don't
go
to
Kuban
at
ease.
If
you
misunderstand
American
pronunciation
of
the
Greek
people
told
me
it's
proven:
nate
kubernetes.
Actually,
anyway,
so
architecture
we
have
a
control
plane
and
the
control
plane
is
actually
the
master.
So
let
us
the
giants.
Johnny
D's,
admins,
we're
running
the
Chinese
masters,
we're
running
a
couple
of
infrastructure
services.
We,
like
networking
storage,
we
run
involved
for
certificates,
we
run
EEPROM
ethos
and
you
know
elk
stack
and
such
things
in
within
the
control
plane.
A
And
then
we
have
a
couple
of
micro
services
that
form
our
API
and
then
the
kubernetes
masters
are
started
in
within
the
like
infrastructure
management
and
on
the
other
hand,
we
have
an
application
zone
where
the
the
actual
nodes
of
the
clusters
are
spawned
and
we
have
the
edge
agents
running
and
where
the
application
workloads
can
be
scheduled
to.
And
if
you
zoom
out
a
little
bit
and
I'm
not
going
into
details,
you
have
a
bunch
of
services
there
that
talk
to
each
other
and
in
the
end,
you
have
the
clusters
created
on
this
side.
A
So
what
do
we
do?
We
sew
on
bare
metal?
We?
We
don't
have
an
API
to
spawn
instances
for
the
workers,
so
we
decided
to
use
to
use
a
deployment
with
the
pot
and
the
pot
comes
within
the
docker
image
which
has
chemo
tooling
in
it
and
spawns
KVM
and
within
the
KVM.
We
again
have
a
core
s
and
we
configure
the
KBM
via
cloud
config
and
we're
giving
it
some.
A
A
The
English
controller
in
the
guest
clusters
is
either
integrated
into
the
locals
or
directly
or
we
do
some
collation
of
ingresses.
So
we
have
ingress,
is
read
from
the
get
guest
clusters
and
then
add
it
to
the
host
cluster,
so
that
also
works
and
yeah.
We
run
in
DNS,
cube
DNS
and
we're
running
the
calico
Coleco
policy
controller
to
enable
the
network
policies
and
yeah
to
go
a
little
bit
into
networking,
because
it's
complex
because
of
this
like
inception
thing
in
the
host
cluster
for
ourselves.
A
We
were
just
using
plain
calico
and
within
the
guest
clusters,
we're
using
flannel
and
calico,
but
we're
using
it
a
different
way.
Then
you
usually
use
it
in
any
community
to
set
up.
So
we
have
on
one
flannel
server
per
host.
So
in
the
recent
versions,
flannel
editors
serve
a
client
model,
so
we
have
one
flannel
server
running
per
host
and
then
we
have
clients
per
tenant
on
the
machine
and
we're
using
the
X
line
as
a
back-end.
A
A
And
if
we
add
another
so
yeah
inside
we
have
the
Calico,
then
so
the
users
of
the
clusters
can
use
network
policies
and
such
inside
of
the
cluster.
And
then,
if
we
add
another
tenant,
you
can
see.
We
have
another
flan
of
client
running.
So
the
flan
our
client
runs
outside
of
the
VM
and
usually,
if
you
use
flannel
within
your
communities,
installation
you
would
have
flannel
inside
of
your
community's
insulation.
So
it's
just
a
way
how
we,
you
know,
use
VX
line
in
between
the
VMS
and
isolate
the
clusters
from
each
other
and
yeah.
A
We,
we
also
have
a
certificate
infrastructure
and
we
use
involved
for
that.
So
we're
creating
a
root
CA
per
cluster,
and
then
you
have
so
you
have
a
PKI
back-end,
so
you
can
actually
issue
like
create
a
root
CA,
an
issue
certificates
and
attach
roles
that
allow
you
to
issue
certain
certificates,
certain
C
names
and
in
the
upcoming
or
in
the
latest
version,
even
organizations
which,
which
is
important
for
the
role
based
access
of
communities.
A
A
First,
because
we
we
have
been
using
fleet
a
lot
in
the
past.
We
we
just
decided
to
launch
different
communities
clusters
on
top
of
fleet,
but
since
nobody
else
is
using
fleet
anymore,
we
stopped
doing
that
and
in
the
beginning
we
started
with
just
having
like
Yama
fights
and
like
do
some
templating
and
then
start
communities
clusters.
And
then
finally,
we
thought
like
yeah.
We
want
to
use.
A
We
don't
want
to
do
the
templating
anymore,
but
we
want
to
go
for
writing
our
own
controllers
and
we
like
the
term
operators
because,
as
actually
tasked,
we
have
been
doing
as
operators
manually
before
and
we
want
to
just
codify
everything
inside
the
cluster.
So
I
want
to
go
into
a
little
bit
of
detail
what
we're
doing
there.
So,
yes,
it's
codifying
your
operational
tasks
and
who
have
you
had
has
heard
about
party
resources
cool,
so
I
don't
have
to
explain
it
that
much
who
is
using
third-party
resources.
A
Cool
so
yeah,
it's
a
way
to
easily
extend
the
API
and
we're
doing
this.
So
we
have
a
cluster
GPR,
so
you
can
actually
create
a
cluster
object
in
communities
and
then
we
have
a
KVM
operator.
That
watch
is
the
cluster
TPR
and
actually
then
goes
and
creates
the
deployment
for
the
cluster.
So
the
point
plots
are
spawned
and
we
actually
have
a
guest
cluster
on
and
we
recently
also
started
working
on
an
AWS
operator.
So
we
can
do
the
same
thing.
A
But
instead
of
talking
to
talking
back
to
the
giant
need
this
API
to
create
the
cluster
in
parts.
We
actually
talk
into
the
ec2
API,
the
AWS
FBI,
to
spin
a
cluster
in
in
AWS
and
yeah
again
a
little
bit
of
visualization
visualization.
What
what
happens
here
so
yeah?
We
have
the
cluster
admin.
He
wants
to
create
a
new
cluster,
so
he
talks
to
the
giant
swarm
API.
A
A
So
we
have
a
cluster
TPO
and
CA
and
multiple
certificates
and
then-
and
we
already
create
a
namespace
for
the
new
cluster
and
then
in
the
next
step.
The
operators
start
seeing
that
there's
like
a
new
TPO
and
so,
for
instance,
the
CA
operator
goes
and
to
vault
mounts
a
PKI
back
and
creates
a
root
certificate,
creates
the
policies
and,
and
then,
as
the
certificate
TP
o--'s
are
created,
a
certificate
operator
can
actually
issue
those
certificates
and
the
KVM
operator
will
create
the
deployment
for
the
cluster
and
yeah
same
for
AWS.
A
So
what
does
a
cluster
TPR
define?
It's
like
a
lot
of
a
lot
of
configuration
options,
everything
that
we
can
actually
configured
our
communities
with
or
all
the
components
are
involved.
For
instance,
like
configuration
options
for
Calico,
we
want
to
have
the
ingress
deployed
already
inside
of
the
cluster
yeah.
We
have
some
node
configurations.
So
how
much?
A
How
much
course
should
the
VM
have?
How
much
memory
should
the
VM
have
and
stuff
like
that
and
as
same
for
the
certificate?
So
we
have
like
a
TTL
F,
a
common
name.
We
have
maybe
the
organization
we
have
with
to
which
cluster
and
component
this
the
certificate
belongs:
yep
and
a
short
demo.
How
how
this
looks
so
I
need
to
switch
the
screen.
I,
don't
see
my
mouse.
A
No
that's
wrong,
but
there
it
is
so
yeah
I
need
to
look
there.
So
if
you
go
to
our
web
interface,
you
have
this
shiny
green
button
to
create
a
new
cluster.
So
I
can
do
Q,
Khan
cluster
and
then
I
have
the
configuration.
I
can
add.
Some
workers
or
I
can
actually
remove
some
workers
or
I.
Can
you
know,
change
some
corporation
and
then
I
can
go
into
my
cluster.
A
And
now
I
have
a
cluster
and
I
can
actually
go
and
create
a
key
pair
so
because
I
want
to
have
access
as
a
cluster
user,
so
I
create
my
own
key
pair
and
I
get
some
configuration.
I
can
put
into
my
cube
config
while
the
cluster
is
booting
up.
So,
let's
see
so
so.
First
of
all,
we
can
just
use
our
command
line
to
see.
A
A
A
B
A
Yeah
I
also
I
also
have
a
local
cluster
where
I
happen
to
have
the
AWS
operator
running
and
I
can
do
the
same
thing
here,
just
without
our
API
just
plainly
on
the
Annika
vanitas.
So
here
we
have
a
cluster
definition.
It's
very
small,
make
it
bigger.
So
now
we
have
a
cluster
definition
for
an
AWS
cluster
and
we
have
actually
one
master
one
one
worker
here
and
if
I
do
cube.
Ctl
create
example,
cluster
cool.
B
A
So
yeah
what's
coming
next,
so
we
figured
out
that
you
know
having
one
big
giant
configuration
in
in
the
closet.
Epa
at
EPR
doesn't
make
sense
for
us,
so
you
want
to
have
like
that.
If
there
are
things
that
have
like
different
life
cycles
different
lifetimes
and
why
not
manage
them
with
like
different
operators,
why
not
go
for
like
micro
services
in
terms
of
operators
so
having
one
responsibility
per
operator?
That
makes
a
lot
of
sense
so
we're
breaking
that
down.
A
We
have
already
separated
out
the
certificate
management
into
its
own
operator
before
we
have
been
using
that
in
init
containers
within
the
pots.
So
now
we're
we're
doing
that
and
then
the
next
thing
is
like:
how
do
we
manage
the
clusters
and
that
may
be
independent
from?
Is
it
on
AWS
on
or
is
it
on
on
bare
metal?
So
how
do
you
roll
you
upgrade?
A
Why
not
change
just
a
resource
in
the
cluster
and
say
like
I,
want
to
have
a
new
calico
version
rolled
out
in
this
cluster
and
then
our
operator
is
there
and
just
takes
care
of
rolling
that
out
and
for
that
we're
actually
planning
to
to
put
all
the
boilerplate
plate.
We
have
in
every
single
operator
into
a
library,
so
you
can
make
it
simpler
to
have
to
create
those
libraries.
We
did
the
same
thing
for
our
micro
services
and
they
were
based
on
gold
kit.
A
A
A
So
at
so
yeah
we
have
pixie
boot,
and
so
mostly
it's
done
by
having
a
pixie
boot
that
configures
the
cloud
configs
to
spin
up
the
machines,
but
that's
also
the
reason
why
we
go
more
into
self-hosted.
So
we
want
to
make
this
main
cluster
self-hosted.
So
it's
easier
to
upgrade
and
easier
to
install
for
us.