►
Description
Managing Thousands of Clusters and Their Workloads with Flux - Max Jonas Werner, D2iQ
With Kubernetes becoming more and more popular, so is managing clusters at scale. Applying GitOps principles with Flux simplifies provisioning clusters and managing workloads deployed onto them, including tenant and RBAC management. In this session I will demonstre a best practice approach towards GitOps with a management cluster handling the provisioning and further maintenance of clusters, tenants and workloads, employing the CNCF projects Flux, OPA Gatekeeper and the Kubernetes Cluster API sub-project. The benefits of such an approach are: * It creates a simplified way to declaratively define thousands of clusters and perform operations on those clusters * Makes it easy to have a multi-tenancy approach where each team or group of applications gets their own cluster or individual namespace on a certain cluster * Operations against clusters are fully audited and attributable, as reverting changes is hard.
A
Yeah
thanks
for
joining
my
talk-
and
I
must
admit-
I'm
really
grateful
to
god
that
I'm
standing
here
right
now
because,
like
on
sunday,
I
woke
up
to
a
very,
very
bad
food
intolerance
like
two
bad
symptoms
of
it,
from
something
I
ate
on
saturday
and
was
laying
in
bed
and
was
like.
I
can't
make
it
to
gitobscon
and
then
on
monday.
I
woke
up.
Nothing
happened
like
as,
if
nothing
happened,
I
completely
recovered,
and
now
I'm
here
so
yeah.
I'm
really
grateful
to
be
to
be
able
to
do
that
here.
A
For
the
first
time
in
years
I
am
max.
I
work
at
data
iq,
I'm
a
kubernetes
engineer
there,
I'm
also
a
flux.
Maintainer
have
been
for
like
half
a
year,
nine
months,
maybe
and
at
daddy
iq.
We
basically
have
two
products,
one
is
or
they
they
are
both
sold
under
the
umbrella
term
dkp,
which
is
the
data
iq
kubernetes
platform.
One
is
enabling
single
cluster
experience
and
the
other
product
building
on
top
of
that
is
for
facilitating
multi-cluster
experience
right.
So
we
basically
have
multi-cluster
running
through
our
veins.
A
A
The
the
main
customers
of
data
iq
are
big
corporation
enterprises,
government
agencies,
and
they
are
slowly
ramping
up
the
kubernetes
deployments
in
in
a
single
cluster
way
right
and
those
that
do
they
already
have
kind
of
mature
day
two
operations
with
the
help
of
of
vendors,
like
data
iq
or
with
the
help
of
an
awesome
platform
team
that
they
want
them
themselves.
Internally,
that's
pretty
much
where
single
cluster,
let's
say,
like
that's
problem
solved.
B
A
Enterprises
are
now
entering
like
this,
this
phase
of
experimenting
with
multi-cluster
environments
right,
so
they
want
to
use
multiple
clusters,
kubernetes
clusters,
in
particular,
for
for
heart
multi-tenancy-
something
michael
nicely
laid
out
in
in
the
talk
today
here
as
well,
but
it's
only
like
dipping
their
toes
into
the
waters
of
multi-cluster
right.
It's
it's
a
very
early
stage
and
very
few
have
mature
day
two
operations
of
these
in
place
with,
like
day
zero
being
an
experimentation
phase
day,
one
being
a
setup
phase.
You
install
your
first
clusters.
A
From
the
experiences
that
we
made
with
our
customers
and
what
I've
been
talking
to
other
people
with
there
are
mainly
four
use
cases
that
I
was
able
to
derive
that
you
want
to
facilitate
and
multi-cluster
environments.
You
want
to
treat
your
clusters
as
cattle
right.
I
just
had
this
had
this
quick
conversation
with
chris
before
the
talk
about
customers,
loving
their
clusters
right
and
he
said,
customers
shouldn't
love
their
clusters.
No
one
should
love
that
cluster,
treat
it
as
cattle.
A
If
it
breaks
shut
it
down,
create
a
new
one.
If
the
version
is
too
old,
it
gets
too
far
behind,
don't
try
to
upgrade
it
just
kill
it
and
create
a
new
one
right,
treat
it
as
cattle.
But
that
means
you
need
to
make
creation
and
deletion,
and
modification
of
a
cluster
are
very,
very
simple
and
replacement
of
an
existing
one
as
simple
as
creating
a
deployment
and
deleting
it
again.
A
You
also
want
to
deploy
a
fixed
set
of
components
onto
each
managed
clusters.
So
the
scenario
I'm
laying
out
here
is
you
have
a
management
cluster,
and
then
you
have
several
attached
clusters
or
adopted
clusters.
However,
you
want
to
call
it
and
in
order
to
make
them
manage
to,
for
example,
roll
up
metrics
to
have
insights
into
the
cost
of
each
cluster.
You
need
to
deploy
a
certain
fixed
set
of
management
components
onto
each
cluster.
A
That
is
something
that
is
mandated
by
the
management
cluster,
but
you
also
want
to
provide
something,
as
I
call
it
here
like
a
centrally
managed
application
catalog.
So
you
have
like
an
ops
team
or
a
vendor.
Maybe
that
maintains
an
application
catalog
for
you
that
you
can
provide
to
your
users
to
deploy
to
every
cluster.
Something
like
I
don't
know.
A
But
I
mean
central
to
a
multi-cluster
environment.
Is
that
you
want
to
make
it
easy
for
users
for
the
developers
to
deploy
their
workloads
onto
the
clusters
right?
You
have
a
tenant,
maybe
multiple
tenants
per
cluster,
and
they
just
want
to
run
their
applications
and
obviously
we're
at
get
ups.
Con
get
ups
comes
to
the
rescue
here
and,
and
I
liked
how
how
michael
said
it
and
his
talk
before
they
they
kind
of
converged
to
get,
because
it's
just
such
or
to
get
ops.
A
It
doesn't
have
to
be
good
in
in
a
natural
fashion.
Right.
It
just
comes
natural
that
you
that
you
converge
to
the
get-ups
model,
but
there
are
some
things
that
would
have
to
change.
With.
Regards
to
the
mindset
that
you
like
apply
to
to
the
model
you
can't
exclusively
rely
on,
kubernetes
are
back
anymore.
A
A
And
you
may
want
to
provide
a
way
to
configure
certain
applications
in
a
different
way.
Maybe,
like
you,
have
the
central
application
catalog.
But
you
want
users
to
be
able
to
say
like
oh
yeah,
I
want
to
deploy
metal
lb,
but
I
need
to
be
able
to
configure
the
address
range
of
metal
lb
yeah.
Something
like
that.
So
that's
pretty
important
users
need
to
be
able
to
configure
the
applications
that
they
deploy
in
a
very
simple
manner
and
of
course
you
want
to
isolate
tenants
from
each
other
on
each
cluster.
A
And
that's
like
one
principle
that
stuck
to
me
from
the
first
time.
I
heard
it,
you
don't
need-
and
maybe
you
don't
want
to
give
users
access
to
the
kubernetes
api
anymore,
maybe
only
in
a
read-only
fashion,
so
they
can
see
the
status
of
their
resources
right,
but
other
than
that
they
shouldn't
be
able,
or
at
least
shouldn't
shouldn't-
want
to
create
edit
or
delete
any
resources,
because
everything
is
going
through
git
anyway,
right
with
flux.
A
If
you
added
a
resource,
the
customization
controller
will
kick
in
right
after
that
and
just
delete
your
changes
again
or
roll
back
changes
again
anyway,
and
what
I
wanted
to
create
first,
like
as
a
prototype
for
myself
and
something
that
we
wanted
to
experiment
with
internally
was
something
like
a
multi-cluster
template,
something
like
we
define
a
certain
set
of
works,
workflows
and
and
personas,
and
then
we
implement
that
model.
However,
we
wanna
you,
we
can
use
flux
for
that.
A
A
A
Then
you
want
to
be
able
to
adopt
the
cluster,
so
you
have
like
several
clusters
and
you
you
want
to
make
them
managed,
and
you
want
to
have
this
like
single
pane
of
glass.
That
shows
you
all
your
clusters
that
shows
you
how
much
each
cluster
costs,
how
much
each
workload
costs
you
want
to
see.
Metrics
you
want
to
see
the
logs.
Maybe
you
want
to
roll
up
the
logs
to
the
central
management
cluster.
A
As
I
said
before,
you
maybe
want
to
be
able
to
group
your
clusters
and
deploy
the
same
set
of
workloads
onto
this
group
right.
So
you
have
several
groups,
you
have
several
clusters
in
each
group
and
then
define
the
workloads
that
each
cluster
in
each
group
would
would
get
right.
So
you
would
have
like
I
don't
know.
Maybe
a
dev
group
and
all
the
dev
clusters
would
get
this
set
of
applications
deployed
and
then
a
staging
group
that
would
get
a
different
set
of
applications
deployed.
A
And
also
you
want
to
be
able
to
configure
the
workloads
per
cluster
and
which
workloads
are
deployed
onto
each
cluster,
so
you
still
need
to
be
able
need
to
be
able,
even
though
you
have
groups
to
to
target
a
single
cluster
yeah.
Oh
yeah,
that's
about
configuration
of
the
workloads,
and
then
you
also
want
to
want
to
be
able
to
like
just
deploy
like
your
development
team,
and
you
want
to
be
able
to
like
just
deploy
your
application
onto
a
single
cluster
and
then
there's
different
personas
acting
on
each
cluster.
A
So
the
management
cluster
administrator.
That's
like
the
person
that
has
like
them
the
the
highest
privileges
in
in
this
hierarchy
of
multi,
multiple
clusters:
that's
the
person
that
deploys
the
management
cluster,
that's
the
person
that
makes
sure
the
set
of
the
fixed
set
of
components,
of
common
components
that
each
cluster
receives
is
well
maintained
and
everything
like
that.
And
then
you
have
the
workspace
administrator.
That's
the
person
that
administrates
a
certain
group
of
clusters.
A
A
And,
of
course,
I
didn't
want
to
leave
it
at
that
and
just
say:
yeah,
that's
a
nice
concept,
one
with
it.
I
also
wanted
to
dip
my
toes
into
it
and
and
actually
implement
it
and
see
how
hard
it
actually
is.
Given
the
tools
we
have
in
the
community
right
now
and
as
a
flux
maintainer,
it
came
quite
natural
to
me
that
I
used
flux
for
that.
A
If
you
don't
know
what
flux
is
yet
it's
basically
a
set
of
components
that
one
on
a
cluster
that
let
you
reconcile
get
repositories
or
other
artifact
storages
like
a
helm,
repository
or
an
s3
bucket.
Then
you
have
a
home
controller
over
there
that
creates
helm
releases
from
manifests
in
that
get
repository
or
in
that
source,
and
you
have
a
customized
controller
running
that
basically
uses
customized
to
create
resources
on
the
cluster.
From
like
bare
bones,
kubernetes
manifests
in
the
repository
right,
so
that's
flux
and
that's
why?
What
I?
A
A
Each
component,
like
the
source
controller,
comes
with
their
own
crds
and
it's
easy
to
understand
right.
You
don't
need
days
or
weeks
to
understand
flux,
you
just
run
flux,
install
it
has
great
cli
and
you
can
go
go
right
away
if
you
want
to
dig
deeper
into
flux,
there's
a
great
documentation,
page
from
created
from
by
kingdom
barrett
from
the
flux
project.
A
It's
a
huge
page,
it's
a
lot
of
text,
but
it
explains
flux
from
a
to
z,
and
it's
a
really
great
page.
I
really
love
it.
I
really
love
that
we
have
that
up
now
for
like
since
two
months
or
so
so
the
demo,
I'm
I'm
showing
right
now,
it's
pretty
quick
because
you
don't
have
too
much
time
is
hosted
in
this
repository.
A
A
A
On
the
right
hand,
side
you
can
see
it's
a
bare
bones.
I'm
gonna
make
this
bigger.
It's
a
bare
bones
kind.
Cluster!
Nothing
exceptional
is
running
in
it.
On
the
white
side,
we
have
some
customizations
already
running
in
it.
There
is
a
script
in
this
repository.
That's
called
create
management
cluster,
and
what
I
wanted
to
do
with
this
demo
is
see.
How
far
can
I
go
without
or
with
programming
as
little
as
possible
right
with
just
using?
A
What's
out
there
as
much
as
possible
and
again,
as
michael
said
before,
they
didn't
have
many
programmers
in
their
department,
and
I
guess
that's
true
for
many
enterprises.
So
what
I
try
to
do
is
use
what's
out
there
and
program
as
little
as
possible,
so
create
management.
Cluster
is
basically
these
four
lines
of
shell
script.
That
creates
a
kind
cluster,
it
installs
flux,
and
then
it
applies
two
resources
to
it
right,
that's
it.
A
So,
let's
see
how
I
attach
a
cluster,
that's
a
little
more
shell
scripting,
but
that's
the
the
closest
I
could
get
to
making
it
simple
right.
Maybe
we
can
find
ways
to
make
it
simpler,
so
it
also
installs
flux
using
the
cli.
That's
something
that's
actually
not
extremely
necessary
because
you
can
keep
all
the
flux
manifest
and
the
get
repository
and
I
have
attached
clusters
just
reconcile
that
and
install
flux.
That
way,
but
bootstrapping
is
always
kind
of
difficult.
A
A
This
customization
is
special
in
that
it
uses
a
very,
very
awesome
flux
feature
where
you
can
say
this
is
a
customization
that
targets
a
different
cluster
and
not
the
one
that
that
flux
itself
is
running
in
right.
So
you
can
pass
a
secret
to
it,
a
kubernetes
secret
to
it
that
has
a
cube
config
file,
so
flux
will
take
that
and
apply
all
the
resources
from
that
customization
onto
the
other
cluster
right.
So
and
and
that's
like
the
starting
point
from
where
everything
gets
installed
from
the
management
clusters,
point
of
view.
A
A
So
that's
a
git
repository,
and
this
is
the
one
that
is
cluster
specific
and,
for
simplicity's
sake,
I'm
using
the
same
repository
in
just
a
different
branch
right
and
and
an
actual
production
environment.
You
would
use
a
different
repository,
obviously
and
provide
the
team
that
is
to
be
the
tenant
on
that
cluster.
Access
to
that
get
repository,
but
yeah
I
wanna,
I
I
didn't
wanna
create
like
10
giga
repositories,
just
for
a
demo,
so
yeah.
I
hope
it
gets
my
point
across
and
then
it
creates
a
customization.
A
This
is
one
of
the
intricacies
with
flux.
You
have
flux
customizations
and
you
have
kubernetes
customizations
and
when
we
talked
about
customizations
at
data
iq
in
the
beginning,
everyone
was
so
confused
because
nobody
knew
what
the
other
person
was
talking
about.
Like
is
this
a
flux
customization,
or
is
this
a
customized
customization
yeah?
I?
I
guess
if
you
start,
when
you
start
using
it,
get
that
right
first
and
then
it
pushes
all
this
stuff
and
for
speed's
sake,
it
kicks
off
a
reconciliation
right
away.
A
A
A
Configs
one
is
the
internal
one
to
be
used
by
flux
and
the
other
one
is
the
one
to
be
used
by
the
shell
script
directly,
for
example,
to
run
flux,
install
that's
not
necessary
if
you're,
using
like
actual
clusters
and
the
w
flag
lets,
you
add
the
cluster
to
a
certain
workspace.
As
I
said,
a
workspace
is
a
group
of
clusters
that
has
a
different
repository
and
receives
all
like
all
the
clusters
and
network
space
receive
the
contents
of
that
gateway
story.
So
it's
just
no.
A
Let's
just
do
it
and
watch
on
the
white
side.
What
is
happening
so
flux
is
installed.
So
it's
not
saying
anymore.
It
doesn't
find
the
resource
the
like
the
definitions,
the
crds
on
the
right
side.
On
the
left
side,
it's
still
verifying
flux,
installation
waiting
for
it
to
get
ready.
Now
it's
cloning,
the
the
management
cluster
repository.
A
A
A
Okay
and
there's
a
single
hem
release
running
right
now,
which
is
gatekeeper,
which
is
like
a
prerequisite
to
using
workspaces
and
isolating
tenants.
So
now
the
cluster
is
attached.
Now
it's
managed.
We
can
see
it
in
the
repository
like
in
the
management
repository
there's
a
directory
here
that
has
the
secret
right
here,
which
holds
the
cube
config
of
the
cluster.
A
Right
so
it
points
to
the
remote
directory
of
clusters
test,
so
that's
specific
to
each
cluster
and
it
uses
the
secret
ref
right
and
then
remote
has
in
turn
the
sync
manifest
of
the
cluster
specificate
repository
all
right,
everybody
following
it's
when
I
did
this:
let's,
let's
see
how
the
attachment
works,
it's
pretty
simple!
It
basically
just
removes
the
remote
customization
as
soon
as
it's
done
with
that.
A
You
will
see
that
all
the
resources
on
the
attach
cluster
will
vanish.
Everything
is
gone,
the
cluster
is
detached
and
it's
in
the
same
state
as
it
was
before
you
attached
it,
which
is
pretty
nice
and
it's
basically
just
removing
stuff
from
the
repository
waiting
for
flux,
to
reconcile
and
be
done
with
it.
A
How
does
this
look
like
from
like
a
maybe
a
10
000
feet
view
you
have
the
management
cluster?
You
have
the
management
cluster
git
repository
and
the
persona,
which
is
the
management
cluster
administrator,
having
access
to
the
management
repository
and
to
the
common
repository
which
is
holding
the
stuff
that
every
cluster
receives.
Then
we
create
a
workspace
which
basically
means
I
create
a
workspace
repository
and
some
resources
in
the
management
repository.
A
A
A
Another
cluster
in
the
same
workspace
would
look
like
this.
It
would
also
be
connected
to
the
common
repository
and
the
workspace
repository,
but
have
its
own
repository
for
cluster-specific
stuff
right,
so
these
may
be
two
different
teams
working
on
different
applications,
but
still
in
the
same
workspace
for
various
reasons,
like
there's
a
million
reasons
to
be
to
to
wanting
to
group
clusters
and
there's
a
third
cluster
there,
that's
outside
of
the
workspace
so
only
reconcile
the
common
repository
to
make
it
manageable
and
the
cluster-specific
repository.
A
If
that
sounds
complicated,
I
tried
to
come
up
with
a
visualization
of
what
my
demo
repository
actually
does
and-
and
this
is
like
all
the
resources
that
are
being
created
and
the
different
colors
my
which
cluster
receives
them,
so
the
blue
ones
are
for
the
remote
cluster.
The
red
ones
are
on
the
management
cluster,
so
it
is
really
confusing
like
I,
I
got
so
confused
when
I
initially
bootstrapped
that
that
model
like
okay,
which
customization
does
what
and
at
what
point.
A
And
what
do
they
depend
upon
and
it
turns
out
that
making
things
simple
like
for
each
persona
right.
They
just
have
access
to
that
repository
and
just
put
their
stuff
in
there.
But
making
this
possible
is
really
hard
right
using
the
tooling
that's
out
there
in
the
world
today
and
that's
that's
like
one
of
the
key
takeaways
and
and
why
I
wanted
to
come
up
with
like
a
multi-cluster
template,
because
everyone's
doing
it
differently
and
because
it's
getting
complicated
really
quickly.
A
A
Oh
that's-
that's,
maybe
an
interesting,
an
interesting
project
to
to
look
at
and
we
actually
incorporated
it
into
our
product
at
data
iq
right.
I
love
making
it
simple
for
users,
so
I
love
creating
very
fine-grained
and
very
focused
clies
and
using
the
tooling
that's
out
there
out
there.
It's
so
simple
right.
We
at
data
iq
we
build
clies
with
go
and
there's
like
all
the
tooling,
like
viper
and
and
cobra
to
to
build
clies
in
a
very
standardized
way,
make
it
simple
for
your
users
by
creating
these
small
little
clis.
A
Play
around
with
stuff,
and
maybe
eventually
it'll
be
incorporated
into
your
platform
one
day.
Maybe
you
just
throw
it
away,
but
you
always
gain
knowledge
and
that's
it.
I.
It
was
really
hard
for
me
to
pack
everything
I
wanted
to
bring
across
in
this
half
hour.
Let's
talk
you
can
reach
out
to
me
via
email,
old
school
way.
I'm
on
twitter
and
marcus
is
basically
my
nickname
everywhere
else
as
well.
I'm
I'm
active
in
the
cncf
slack
and
there
you
can
post
your
question
in
the
flux
channel.
A
A
A
D
Thank
you.
So
I
was
noticing
that,
when
we're
watching
multiple
repositories,
like
the
common
repository
in
the
workspace,
you
gave
the
example
of
using
cube
cost
as
something
that
you
might
want
to
apply
in
all
your
cases.
What
if
you
had
a
coup,
cost
version
1.0
and
then
in
another
repository,
coupe
cost
version
2.0?
A
You
have
this
responsibility
for
that
repository
and
make
sure
okay
cubecast,
and
that
repository
works
across
my
clusters,
but
sure
you,
you
have
to
find
a
certain
way
how
to
how
to
polish
that
right-
and
that
is
also
something
I've
been
thinking
about
and
not
really
like
found
a
solution
for
what
happens.
If
something
is
deployed
in
a
workspace
and
also
in
a
cluster-specific
repository,
they
would
obviously
be
overriding
each
other
right.
So
that's
something
like
that's
a
problem.