►
Description
Don’t miss out! Join us at our upcoming events: EnvoyCon Virtual on October 15 and KubeCon + CloudNativeCon North America 2020 Virtual from November 17-20. Learn more at https://kubecon.io. The conferences feature presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Managing Multi-Cluster/Multi-Tenant Kubernetes with GitOps - Chris Carty, Independent
Trying to go fully cloud native all at once can be intimidating. Learning GitOps best practices is a relatively small investment in time that has big operational payoffs. Good news! If you’re using Git, you’ve already started! This talk with be a guide to effectively implementing GitOps through a real world example of modernizing deployments using FluxCD in a multi-cluster/multi-tenant environment. It will also introduce CI tools such as conftest/Kubeval, Open Policy Agent and Kind to make pipelines more consistent and secure.
https://sched.co/ZerJ
A
A
When
I
came
up
with
this
talk,
I
was
kubernetes
administrator
and
operator.
So
this
is
based
off
of
my
experiences
as
from
that
position.
So
today
we'll
be
talking
about
managing
multi-cluster,
multi-tenant
kubernetes
with
git
ops.
I
know
it's
a
bit
of
a
mouthful,
but
it's
kind
of
more
or
less
what
we're
doing.
I
can
think
of
a
shorter
name,
so
by
multi-cluster,
probably
already
talking
about
there
but
multi-tenant.
A
It's
the
problem
that
I
was
looking
at
solving
when
at
the
time
was
we
had
multiple
clusters
and,
of
course
we
had
multiple
teams
that
would
need
to
be
deploying
to
those
clusters.
So
this
is
how
we
kind
of
came
up
with
a
pattern
to
address
that
issue
and
some
of
the
fun
things
that
we
came
up
came
up
with
along
the
way.
A
So
this
is
the
first
time
I've
done
a
live
or
not
a
live
presentation
or
recorded
presentation,
which
is
quite
fun,
so
unfortunately
I
don't
have
all
you
find
people
to
present
to
so.
Instead,
we
have
our
friends
optimus,
prime
and
a
couple
other
autobots
and
a
few
decepticons
mixed
in.
There
will
be
my
my
audience
today,
not
quite
the
same,
but
I
expect
some
encouraging
words
and
claps
from
optimus
prime
over
there
on
the
left
so
now
with
we'll
get
into
it.
So
I
said
so.
A
This
is
based
off
of
my
experience,
managing
multiple
clusters
and
a
large,
a
large
number
of
teams.
So
when
we
first
started
out
the
problem
that
we
needed-
or
I
was
looking
to
solve-
was
how
do
I
manage
let
the
teams
maintain
their
autonomy
without
causing
this
kind
of
situation,
every
team
had,
theoretically
their
own
ci
version
or
ci
cd
pattern.
They
had
their
source
control
in
its
own
specific
location.
A
So,
while
we
were
all
on
git,
things
could
be,
you
know,
get
lab
github
bitbucket
a
few
different
places
we
couldn't
as
a
commercial
administrator.
I
couldn't
you
know,
get
them
to
centralize
on
one.
That
would
be
a
bit
too
big
of
an
ask
for
for
me
and
we
wanted
to
be
able
to
let
them
do
what
they
want,
while
getting
things
into
the
cluster,
with
as
little
friction
as
possible
and,
of
course,
without
causing
more
fires,
more
pain
for
the
developer.
A
We
wanted
to
do
less
automate
all
the
things
and
have
things
move
slowly
as
moved
smoothly.
So
one
of
the
awesome
parts
about
kubernetes
is
that
everything
is
declarative,
so
we
were
able
to
look
at
leveraging
tools
that
could
use
that
to
our
advantage.
So
we
had
to
initially
start
looking
at
this
more
traditional
cd
aspect,
so
pushing
the
changes
to
the
cluster.
So
with
gitlab
and
jenkins,
it
worked
really
nice,
but
at
some
point
it
didn't
quite
didn't
quite
scale.
A
You
know
we
had
to
load
up
the
those
cds
with
our
cd
tools
with
the
configuration
files
for
specific
clusters-
and
you
know
if
any
of
those
clusters
went
down,
came
up
or
were
replaced.
We'd
have
to
update
all
the
intuos
cis
cd
tools,
which
is
a
bit
of
a
bit
of
a
headache,
especially
when
we
knew
that
you
know
we'd
be
we'd.
Have
multiple
zones,
you'd
have
different.
A
You
know
dev
qa,
prod
clusters.
It
could
be
multiple
of
them.
There
could
be
on-prem
could
be
in
the
cloud
it
could
be
anywhere.
We
wanted
to
not
have
that
burden
of
managing
and
maintaining
where
the
cluster
was
on
the
development
team.
We
just
wanted
them
to
focus
on
this.
Is
my
config
deploy
it
right?
A
So
as
kelsey
here
says,
gitops
is
the
best
thing
since
configuration
as
code
and
probably
key
point
here
for
me,
is
it
our
same
similar
to
kelsey?
Is
this
is
key
to
dealing
with
infrastructure
at
scale
and
we
found
this
was
super
beneficial
for
that,
because
the
development
team
for
us
didn't
need
to
know
where
those
clusters
were.
A
That
was
that
burden
was
put
on
the
the
ops
stops
team
or
if
there
are
kubernetes
and
min
team,
and
so
we
were
able
to
work
with
them
with
this,
so
get
ops
is
a
workflow
which
emphasizes
git
as
your
single
source
of
truth.
I'm
assuming
most
people
at
this
point
are
storing
things
in
some
kind
of
source
control
primarily
get,
and
so
this
isn't
a
big
leap
to
do,
which
is
really
nice.
A
So
there
the
workflow
is,
you
know,
there's
an
operator
of
some
kind
that
sits
in
your
cluster
and
its
job
is
to
monitor
your
git
repo
and
just
apply
the
changes
that
it
sees
there.
So
you
have
a
single
source
of
truth
and
then
an
agent
which
tries
to
make
that
so
similar
now
kubernetes
works
when
you
give
it
the
manifest
right.
Just
like
this
is
the
state
of
the
world,
I'm
going
to
make
sure
that
this
happens.
A
So
now
we
have
kubernetes
trying
to
do
that
with
the
manifest
that
the
agent
deploys
and
then
the
agent
is
telling
kubernetes
constantly.
This
is
the
state
of
the
world
that
I
see
and
get
so
you
have
this
another
for
loop,
essentially
and
one
of
the
other
nice
of
the
other
benefits
this
process
is.
The
developer
doesn't
necessarily
need
to
have
direct
access
to
a
cluster.
Now,
there's
going
to
be
instances
where
you're
going
to
need
to
do
this.
A
If
something
goes
down
or
you
have
to
do
some
debugging
on
the
fly,
but
predominantly
the
changes
will
come
from
a
merge
request
or
pr
and
the
changes
will
get
merged
in
and
then
deployed
via
git.
Once
those
have
been
approved
and
passed,
your
passes
your
ci,
so
you
reduce
the
reliance
on
having
people
imperatively,
deploying
things
to
a
cluster.
A
Instead,
everything
goes
through
git
and
then
you
have
the
kind
of
the
baked
in
benefits
of
git.
Where
you
know
it's
trace,
the
changes
are
traceable.
Auditable
you'll,
hopefully
have
good
practice
around
that.
So
you
have
your
pr
and
any
discussion
that
comes
over
changes,
so
you
can
understand
what
what's
happening
within
your
cluster
system
now
the
cncf
has
also
started
adopting
github,
so
we
have
two
primary
projects
within
the
get
ops
field.
A
So
flux,
cd,
as
well
as
rgo
cd
flux,
is
the
tool
that
I
settled
on
when
was
building
out
our
internal
platform.
Argo
is
a
very
close.
Second,
it
was
really
cool
tool.
Just
it
did
a
few
extra
things
that
we
didn't
need
it
to
do,
but
it
was
still
still
great.
Flux
was
a
little
bit
more
lightweight
and
flexible,
and
it
was
just
focused
on
doing
the
deployment
aspect
which,
which
is
what
we
certainly
needed,
as
we
had
other
things
that
filled
in
the
argo
part.
A
I
do
check
out
both
both
of
the
tools
as
they're,
both
really
great.
They
sell
different
use
cases,
but
both
fantastic.
So
this
is
what
the
the
flux
system
looks
like.
So
you
have
a
single.
You
have
your
flux
agent,
which
does
the
syncing,
and
then
it
also
has
the
added
benefit
of
being
able
to
monitor
a
container
registry
and
pull
in
any
new
tags
that
have
been
pushed
that
image.
So
you
can
have
this
nice
little
automated
loop,
so
your
cd
for
your
application
creates
a
new
container
push
it
up
to
the
registry.
A
A
So
it's
pretty
tight,
which
is
nice,
and
that
was
one
of
the
features
that
we
wanted
to
to
leverage,
especially
within
dev
qa.
Just
so
we
have
that
quick
release
cycle,
which
is
nice,
so
the
other
part.
So
we
kind
of
so
we
found
the
tool
we
did.
Some
dry
runs
to
figure
out
which
how
it
worked
and
different
processes
but
we'd
started.
A
You
know
what
was
the
appropriate
way
to
structure
project.
What
was
the
logical
way
to
put
things
buckets
to
put
the
code
in?
How
do
we
divvy
that
up,
so
digging
around
looking
at
the
best
practices
and
like
it
came,
became
quite
clear
that
the
best
way
to
start
structuring
the
application
code
was
to
keep
the
application
code
separate
from
the
the
manifest.
A
A
So
the
the
intent
of
both
the
ci's
on
those
repos
are
different.
So
it's
just
it's
nicer.
If
they're,
if
they're
kept
separate
sure
for
smaller
projects
and
teams,
it's
not
a
big
deal
having
them
together,
but
once
you
start
getting
bigger
it,
it's
nicer
to
keep
them
in
their
own
buckets,
and
this
is
what
the
that
app
operator.
Repo
look
like
in
very
simple
scenario:
just
have
a
folder
with
your
ammo
in
it
and
all
the
agent
is
going
to
do
is
or
yeah.
So
there's
nothing
fancy
happening
here.
A
Loop,
it'll
just
start
deploying
things,
and
then
there
you
there,
you
have
it
so
as
a
developer,
wouldn't
need
to
touch
the
cluster
at
all,
which
is
really
nice
and
we're
able
to
leverage
this
for
onboarding.
So
cluster
access
was
heightened,
heightened
status,
only
certain
people
to
have
access
to
those
resources
so
for
the
developer
they
wouldn't
need
to
get
access.
A
A
It
allows
you
to
run
those
policies
within
your
either
your
local
machine
or
through
a
ci
process.
So
you
can
start
shifting,
left
and
capturing
those
those
violations
sooner
in
the
process.
Instead
of
having
to
wait
for
the
force
to
hit
the
cluster
to
get
get
your
pass
fails,
cube
val
is
a
thing,
I'm
from
the
same
team
that
from
the
confidence
originator,
gareth
thrushgrove,
so
it
allows
you
to
validate
whether
or
not
your
yaml
is
compliant
or
api
compliant
and
then
kind.
A
So
that's
for
those
who
don't
know
that
stands
for
kubernetes
and
docker.
So
it's
a
tool
that
lets
you
spin
up,
kubernetes
in
docker
containers.
So
this
is
a
really
nice
way
of
getting
a
disposable
kubernetes
cluster
in
your
ci.
You
deploy
against
that
and
test
things.
So
you
get
a
better
understanding
of
how
your
application
is
going
to
behave
before
it
even
hits
your
cluster
and
so
a
example.
This
is
very
bare
bones.
A
pipeline
would
look
something
like
this.
You
know
you
do.
A
A
validation
run,
cube
valve
comp
test
to
make
sure
things
are
all
good
and
dandy,
and
then
you
test
things
in
a
kubernetes
cluster,
so
demand
your
budget
either.
Do
a
provision
something
on
the
fly
in
a
cloud
provider
or
you
could
just
spin
up
kind,
really
quick,
just
lower
cost
and
test
against
that,
and
then
finally,
would
be
your
release
pipeline
sorry
release
stage,
and
that
would
just
be
consist
of
a
manual
approval
on
a
pr
you
know.
Xyz
person
has
signed
off
on
it
so
say
its
production.
A
You
know
security,
networking
and
whoever
is
in
charge
of
the
change
approval
process.
If
your
company
has
one
so
the
next
hurdle
that
we
started
looking
at
or
started
hitting
was
once
he
had
that
or
sorry.
Once
we
had
the
kind
of
folder
structure
in
the
ci
and
decided
and
how
that
was
going
to
work
was
well.
How
do
we
kind
of
manage
all
the
different
teams?
So
this
is
where
we
kind
of
bumped
into
our
multi-tenant
issue.
A
Or
do
we
continue
letting
the
teams
have
more
autonomy
right,
so
each
team
would
have
their
own
specific
git
repo,
that
they'd
be
storing
those
against
and
they'll
find
some
way
to
reconcile
against
those,
so
those
are
kind
of
the
general
patterns
I've
seen.
I
think
I
think
argo
does
something
different
and
something
around
those
lines
too.
A
You
can
organize
that.
However,
you
want,
depending
on
your
team
structure
or,
however,
your
projects
are
organized.
In
this
case
I
have
a
folder
for
cluster
admins
and
then
we
have
some
monitoring
networking
security
folders
under
there.
So
presumably
we'd
have
you
know
something
like
elasticstack
network
policies,
maybe
something
like
falco
oppa
with
gatekeeper
all
organized
up
there,
and
then
each
team
would
have.
A
However,
they
have
want
to
set
things
up,
so
it
would
be
in
this
case.
I
have
a
folder
for
app
under
each
each
team
and
then
all
the
flux
agent
would
do
is
sync
against
that
and
just
recursively
deploy
all
the
ammo
right.
So,
depending
on
how
large
your
team
is,
this
could
get
quite
big
at
some
point.
Given
that
flux
is
just
doing
a
get
clone,
so
you
know
all
it
takes
would
be
one
for
one
team
to
misconfigure
something,
and
then
your
deployment
might
fail,
or
something
like
that.
So
you
know
smaller
teams.
A
This
is
what
it
would
look
like
from
a
multi-cluster
point
of
view,
so
we
have
all
of
our
teams
they
deploy
against
the
single
git
repo,
and
then
we
have
an
agent
in
each
cluster
or
clusters
with
an
agent,
sorry
yeah,
with
an
agent
pointing
to
this
to
the
right
branch.
So
in
this
case
we
have
a
just
a
three
cluster
set
up,
so
each
branch
is
each
cluster
is
thinking
with,
say:
dev
qa,
prod,
pretty
straight
forward.
A
There
are
different
ways:
you
can
do
this
if
you
have
different
folders
use.
Customize
to
you,
know,
pull
off
of
base
and
you
can
break
things
up
like
that
which
is
nice,
and
then
you
can
start
doing
kind
of
organizing
clusters
by
kind.
So,
for
example,
you
want
to
have
everything.
Safe
security
has
a
group
of
policies.
You
want
to
have
everything
going
to
a
heightened
security
area
and
you
can
start
doing
neat
things
like
that.
A
A
Or
they
would
open
could
open
pr.
You
could
create
it,
so
they'd
just
be
doing
those
things.
Then
there'd
be
an
agent
which
gets
provisioned
in
that
name,
space
which
would
sync
with
their
own
repo,
and
that
would
be
namespace
and
scope.
So
you
give
them
a
lot
more
freedom
to
do
what
they
want,
but
you
have
from
that
one
cluster
repo,
a
lot
of
control
over
what
they
can
do
and
then
you
also
open
up
a
lot
more
visibility.
A
So,
depending
on
how
you
want
to
work,
they
could
open
up
a
pr
if
they
want
to
change
anything
against
their
main
repo
or
you
can
open
up
tickets
and
you
can
change
it,
but
they
would
still
be
able
to
see
what's
happening
within
their
namespace,
even
though
they
couldn't
change
it
directly,
and
this
is
so.
I
mentioned
this
briefly
within
the
a
couple
slides
ago,
but
it's
nice,
where
you
can.
A
A
A
The
fun
part,
is
the
clusters
become
a
little
bit
more
ephemeral
to
the
teams
are
abstracted
away
so
they're.
They
know
they're
there,
but
they
don't
need
to
know
where
they
are
exactly
or
what
they
are.
It's
just
that
the
policies
are
getting
targeted
and
deployed
effectively,
which
is
really
cool
and
quite
powerful,
especially
if
you're
trying
to
affect
change
on.
You
know
tens
or
hundreds
of
clusters,
depending
on
what
the
kind
of
organization
size
you're
looking
at
and
just
be
able
to
do,
that
with
a
single
pr
or
commit
is,
is
quite
powerful.
A
A
A
A
And
now
we'll
get
into
a
quick
demo
just
to
show
off
what
this
could
look
like
in
practice
and
just
to
show
you
it's
not,
I
mean
it's
pretty
cool,
just
to
be
able
to
see
see
it
works
so
I'll
have
three
clusters
running,
two
of
which
will
already
be
synced,
with
with
our
configuration
and
I'll
just
bootstrap
more
last,
that
individual
cluster
or
that
single
cluster.
That's
not
synced,
just
to
show
you
that
how
we
can
inherit
the
process
or
inherit
all
the
features
of
those.
A
A
So
right
now
we
have
three
clusters:
dev,
qa,
prod
and
they're.
All
saying
came
to
the
main
repo
on
this
on
the
same
branch
probably
want
to
have
it
separated
out
in
our
actual
scenario,
but
this
is
just
a
demonstration,
so
cheating
a
little
so
I'll.
Take
a
look
just
to
show
you
what's
in
this
cluster,
so
I'm
in
the
dev
or
sorry,
the
prod
cluster.
A
We
have
a
few
different
name
spaces
right
now,
I'm
in
the
demo
app
name
space
which
simply
has
a
little
web
app
that
I
wrote
the
falco
namespace
our
falco
system
has
falco
running
and
then
gatekeeper
has
gatekeeper
running.
You
can
see
it's
a
there's,
several
nodes
in
the
cluster
they're,
all
all
doing
things.
So
the
same
thing
goes
for
the
qa
cluster
or
sorry,
the
dev
cluster,
the
qa
cluster
is
the
one
we're
going
to
bring
up
online.
A
So
if
we
do
that,
it's
going
to
be
in
the
there
we
go
same
scenario
as
the
prod
cluster,
so
we're
in
the
demo
app
name
space
yeah.
We
got
same
app.
So
if
I
switch
my
name
space
over
to
falco,
just
to
show
you
what's
going
on
there
same
setup,
different
contacts
different
cluster,
so
I
go
over
to
my
qa
cluster.
A
A
So
it's
pulling
all
the
resources
in
from
this
folder
here
and
then
each
one
of
these
application
folders
is
going
to
generate
a
flux
agent,
which
will
sync
up
to
a
specific
repository.
So
the
demo
app
here
we'll
point
to
the
this
repo
here,
which
has
the
configurations
stored
in
the
deployment
repo
you
can
see
here.
If
I
go
into
flux
patch
everything
is
set
there.
A
So
it's
pointing
everything
at
the
deploy
folder
on
the
main
branch
at
that
that
target
the
falco
repo
is
going
to
look
a
little
different
because
it's
running
using
the
it's
being
deployed
by
helm.
So
let's
just
take
a
look
here.
It's
only
read
only
and
then
it's
being
pulled
from
this
helm,
branch
or
repo
that
I
have
set
up
so
I'll
just
go
there
just
to
show
you.
A
A
Go
back
to
the
helm
release
here,
so
it's
kind
home
release
and
then
those
values
right
there.
So
you
can
see
I
have
down
here.
Our
back
is
enabled
psp
should
be
enabled,
and
then
I
have
I
think,
evpf
turned
on
so
it
works
with
gcp
all
right.
So
now,
if
I
go
back
to
the
cluster,
hopefully
everything
has
booted
up
as
expected,
and
there
you
go
so
I've
deployed
all
the
all
this
infrastructure
or
most
of
the
infrastructure
without
having
to
touch
much.
A
So
if
I
look
at
the
falco
system,
so
I'll
just
switch
over
to
falco,
I'm
not
picking
on
falco
for
any
other
reason
other
than
I
like
saying
it
right
now,
so
you
can
see
that
the
agent
is
there
and
it's
not
doing
or
nothing
else
has
been
deployed.
That's
because
the
agents
that
have
that
were
deployed
by
the
master
don't
have
their
respective
ssh
keys
to
access
the
clusters
or
sorry
the
git
repos.
A
So,
in
order
to
kind
of
bypass
that
I
just
have
a
little
a
script
that
just
deletes
the
secret:
that's
in
there
and
recreates
it
with
the
right
right
key,
which
is
stored
locally,
so
nothing
too
magical
happening
there.
Now,
if
I
go,
do
this
it'll
take
just
a
couple
minutes
just
for
that
agent
or
the
five
minute
cooldown
time
for
everything
to
take
effect.
A
A
So
let's
go
check
and
take
a
look
on
gatekeeper
there.
We
go
so
it's
already
starting
to
roll
out.
So
now
we
have
gatekeeper
up
and
running
in
the
the
cluster
everything's
going
to
be
thinking
from
its
repository.
So
if
I
or
say
the
security
team
set
up
a
central
control
for
all
the
gatekeeper
policies,
I
would
automatically
be
getting
those
and
the
configuration
which
they've
set
up
in
that
specific
branch
or
policy
same
goes
for
falco
agents.
A
A
Logs
all
right,
so
there
you
can
see
everything,
that's
putting
out
there,
it's
getting
set
up,
so
we're
we're
good
to
go
and
that's
kind
of
how
you'd
bootstrap
everything
and
get
it
going,
and
that's
bringing
things
in
sync
from
different
clusters
into
a
single
single
cluster
or
sorry
across
multiple
clusters
from
multiple,
multiple
repos.
Okay.
A
I
hope
it
gave
you
an
idea
of
how
you
would
approach
building
your
multi-tenant
solution
going
forward.
We
have
our
multi-multi-git
repo
pattern
or
even
a
single
single
git
repo
and
some
of
the
pros
and
cons
of
approaching
approaching
that,
and
how
would
you
apply
those
to
managing
multi-multi-tenant
multi-cluster
kubernetes,
and
so
I
will
be
in
the
chat
for
a
little
while.
A
A
Also
a
few
resources
here
for
you
to
take
a
look
at
and
that
I
found
found
quite
helpful
in
my
journey
and
the
the
one
via
steve
wade
from
metal
is
quite
new,
but
it's
a
really
good
example
of
how
you
can
do
this
in
a
practical
way,
and
thank
you
for
for
having
me
today
and
tuning
in.
I
hope
you
enjoy
the
rest
of
kubecon
and
I
hope
that
we'll
be
able
to
meet
up
in
or
see
each
other
all
in
the
reel
sometime
in
the
near
future.