►
From YouTube: Creating A Landlord for Multi-tenant K8s Using Flux, Gatekeeper, Helm, and Friends - Michael Irwin
Description
Creating A Landlord for Multi-tenant K8s Using Flux, Gatekeeper, Helm, and Friends - Michael Irwin, Docker
Supporting multi-tenant environments in Kubernetes is easy, right? (insert laugh here) Well, it can be. But, it takes organization, structure, and proper policy enforcement.
At Virginia Tech, I helped build a "Common Application Platform" that gives each tenant its own manifest repo and deploys those manifests into isolated namespaces using Flux. By leveraging Gatekeeper and Karpenter, we can properly isolate workloads into node pools and ensure tenants don't step on each other's toes. And best of all, our tenant config is in a simple Helm chart that we call "the landlord."
In this talk, we'll dive into how we've built the landlord, the various policies and mutations we're using, and how it works... all with the intent that you can build your own platform too! We'll have live demos and even try to break a thing or two!
A
Hello,
everybody
so
just
a
heads
up.
If
I
sound
a
little
crafty
it's
because
last
week
I
was
battling
a
cold
and
just
sinus
mess
and
don't
worry.
I
was
covered
testing
the
whole
way
through
and
I
finally
got
through
it
all
just
to
get
on
an
airplane
and
sleep
with
masks
on
and
everything
for.
You
know
15
hours
and
all
the
craft
came
right
back,
so
I'm
still
working
through
that
a
little
bit
and
so
you'll
just
have
to
bear
with
me
a
little
bit
on
this.
A
So,
as
was
just
mentioned,
I
I
work
at
docker
now
and
I
just
started
there
about
two
months
ago,
so
still
getting
my
feet,
wet
and
everything.
But
prior
to
that
I
was
a
docker
captain
for
about
five
years,
and
so
I've
been
involved
with
the
docker
community
space
for
a
long
time
now
and
done
a
lot
of
conference
talks
and
and
one
and
in
fact,
actually
the
dockercon
two
weeks
ago.
A
You
can
find
me
pretty
much
anywhere
online
at
mikesar87
on
twitter
and
github
and
all
that
kind
of
stuff
too.
So
so
what
are
we
going
to
talk
about
today?
I'm
going
to
talk
just
a
little
bit
about
this
common
application
platform,
because
it
sets
the
stage
a
little
bit
of
what
we're
going
to
be
working
on
today
and
then
we're
going
to
talk
a
multi-tenancy.
A
Because
it's
easier
said
than
done
for
sure,
then
we're
going
to
talk
about
creating
our
actual
landlord
and
then
we're
going
to
wrap
up
in
q
a
and
the
the
source
code
for
this
will
be
found
here
and
it's
it's
already
there
and
we're
going
to
be
adding
to
it
as
we
go
throughout
this
session
to
you.
So
I'm
very
much
a
big
believer
in
live
demos,
we're
just
going
to
build
stuff
and
try
stuff
live,
and
so
hopefully
the
conference
wi-fi
supports
me.
If
not,
we
could
fall
back
and
that'll
work
too.
A
So
so,
first
off
the
goals
of
the
effort,
this
common
application
platform
was
really
designed
to
be
a
hey.
You
know
we
recognize
that
there
are
many
teams
that
are
deploying
applications
in
many
different
ways
at
the
university.
Can
we
start
to
pseudo
standardize
this
now
that
containers
have
have
really
given
us
a
higher
level
abstraction
point:
can
we
have
a
platform
that
builds
on
top
of
that
idea?
A
And
so
we
we
wanted
to
build
this
platform
and
and
make
it
available
to
all
of
our
application
development
teams
around
the
university
and,
at
the
end
of
the
day,
what
we
wanted
was
really
to
satisfy
this.
You
you
build
a
container
image,
should
be
a
build
container
image,
but
bring
a
container
we'll
run
it
for
you,
and
that
was
really
what
we
wanted
to
do.
One
of
the
first
things
we
had
to
do
was
identify
kind
of
the
separation
of
concerns.
A
What
does
the
platform
team
own
and
what
does
the
application
development
teams
own
and
we
pretty
much
decided
that
the
the
platform
team
owns
everything
below
the
applications
and
what
I
mean
by
that
is
what
we've
got
here
on
the
slide
here.
Let
me
grab
the
laser
pointer,
okay,
so
the
platform
team
won't
own
any
of
the
cloud
infrastructure,
networking
et
cetera
the
cluster
infrastructure
itself.
A
We
built
on
top
of
kubernetes
any
core
cluster
services,
so
obviously
our
like
ingress
controller
and
prometheus
operators
and
and
all
the
kind
of
cluster-wide
services
that
are
available.
We
would
also
manage
all
the
node
and
compute
resources,
so
our
our
development
teams
shouldn't
have
to
think
about
machine
patching
or
anything
that
comes
along
with
that
and
then
from
there.
A
We
want
to
have
an
abstraction
point
where
then,
the
application
teams
can
just
say
here's
my
app
but
here's
my
deployment,
my
service,
my
ingress
or
whatever,
and
they
don't
have
to
worry
about
all
the
other
glue
that
holds
it
all
together,
and
so,
with
this
kind
of
separation
concerns,
it
basically
kind
of
said
hey.
If
there's
you
know
new
cds,
new
security
updates,
if
it
was,
you
know
with
cubelet
or
kubernetes
itself,
or
you
know
a
machine
node.
Okay,
that's
on
us.
A
If
there's
a
problem
with
the
application
which
a
rest
endpoint
isn't
validating
users
correctly.
Well
that
that's
an
application
concern,
and
so
that's
in
this
blue
blue
box
and
that's
up
to
the
application
teams.
So
we
we
try
to
make
it
really
clear
on
on
how
this
is
is
going
to
work
out,
then
the
kind
of
the
question
is:
is
how
do
we?
Actually?
A
How
do
we
build
this
abstraction
point?
What
what
does
that?
Actually,
look
like
and
and
we'll
get
into
that
in
a
second
so
actually,
maybe
before
I
do
this
I'd
like
to
have
a
little
bit
of
audience
participation,
anyways
who's
ever
tried
to
do
multi-tenancy
in
kubernetes
before
okay,
so
maybe
not
quite
a
third
who
found
it
easy
to
do.
A
That's
that's
the
reaction.
I
expected
there
okay,
so
the
first
thing
I
want
to
answer
is:
what
do
I
mean
by
tenant,
because
everybody's
got
different
answers
there
and
we
kept
the
definition
really
really
loose
and
we
basically
just
said:
hey
application
teams.
You
tell
us
how
you
want
us
to
carve
things
up
and
as
we
go
through,
this
you'll
see
how
we
made
various
design
decisions
around
around
the
kind
of
sandboxing
around
each
tenant,
but
really
it's
up
to
the
individual
teams
to
figure
out.
We
had
some
teams.
A
That
said
all
right,
I
wanted
just
a
dev
staging
prod
and
we'll
throw
all
of
our
development
one
all
of
our
staging
and
another
one
and
all
of
our
prod
another
one.
Okay,
whatever
we
had
other
teams.
That
said,
I
want
a
different
tenant
per
application
and
again
we
could
support
really
any
of
them
to
us.
Tenants,
don't
cost
anything,
it's
just
extra
metadata
extra
policy,
extra
procedure
and
that
kind
of
stuff.
A
A
A
We
don't
want
one
team
to
make
changes
that
affect
another
team,
whether
that's
to
you
know,
change
deployments
or
change
ingress
or
whatever
you
don't
want
one
team
accessing
the
secrets
and
config
of
another
team.
So
there's
just
a
lot
of
things
that
you've
got
to
start
thinking
about.
How
do
we
properly
isolate
things
and
there's
there
are
various
aspects
built
into
kubernetes.
You
know
the
rbac
and
role
bindings,
etc.
That
can
help
with
that.
But
there
are
some
things
that
it
doesn't
cover.
A
And
so
how
do
you
just
make
sure
I
mean,
according
to
the
rbac
sure
everybody
can
create
ingress
objects,
but
how
do
you
get
a
little
bit
more
granular
that
and
we'll
talk
about
some
of
that,
as
well
as
a
platform
team.
We
also
didn't
want
to
allow
teams
to
create
their
own
node
port
and
load
balancer
services,
but
they
should
be
able
to
create
services.
A
So
how
do
we
get
more
granular
than
that
and
we'll
talk
about
that-
and
this
is
just
a
quick
list-
there's
so
many
more
things
that
you've
got
to
start
thinking
about
once
you
start
getting
into
multi-tenancy
now
this
isn't
a
fully
comprehensive
spectrum
here.
This
is
something
I
kind
of
just
threw
together
and
and
and
you'll
see
various
opinions
on
this.
A
A
So,
as
you
go
further
on
to
the
the
soft
side,
and
it's
the
more
trust
and
the
you
know,
we
we
trust
everybody
to
behave
well
and,
for
the
most
part,
that's
probably
true,
but
again
we're
in
a
university
setting
where
we
may
be
start
working
with
students-
and
I
don't
know
if
I
trust
all
those
students
out
there
and
or
those
students
develop
something
they've
got
it
deployed
and
now
they've
gradually
they
moved
on
and
now
some
other
professors
having
to
maintain
that,
and
they
don't
have
any
idea.
What's
going
on.
A
We
had
to
think
about.
You
know
where
in
the
spectrum
do
we
want
to
go
and,
as
you
move
further
over
to
the
hard
side
of
things,
you
start
sandboxing
and
basically
removing
a
lot
of
the
trust
and
you're,
putting
more
validation
and
you're,
putting
just
more
constraints
into
place.
It
certainly
makes
things
more
secure,
but
also
it
does
make
things
much
more
costly.
So,
for
example,
if
I'm
doing
a
micro
vm
per
pod
well,
now
I've
got
I'm
running
a
lot
more
micro
vms.
A
It's
a
lot
more
resources
than
I'm
having
to
maintain,
but
man.
If
anybody
bus
out
of
a
pod
well,
they
can't
jump
into
somebody
else's
pod
because
well
they're
in
their
own
little
micro
vm.
At
that
point,
and
then
there's
some
pretty
cool
stuff
going
on
virtual
clusters.
I
won't
get
into
that,
but
again,
there's
there's
a
spectrum
here
and
for
you
and
your
organization,
you
have
to
figure
out.
Where
is
it
that
you
want
to
land?
A
What's
an
appropriate
level
of
constraints
and
controls
and
also
permissioning
that
we
want
to
give
back
so
for
us
we
landed
at
about
this
point
where
we
didn't
go
to
the
full
micro
me
and
per
pod,
but
we
also
wanted
to
support
the
ability
to
say
hey.
We
want
to
group
various
applications,
various
tenants
together,
maybe
in
their
own
node
pool-
and
you
know
I
have
one
node
pool
for
this
team
and
another
node
pool
for
another
team
and
again
we'll
dive
into
that
here
in
just
a
few
minutes.
A
Okay,
so,
as
we
start
thinking
about
all
these
different
things
and
all
the
different
configuration
points,
we
decided
we're
going
to
build
something
called
a
landlord,
and
why
do
we
call
it
a
landlord?
Well
because
we're
in
a
college
town
and
that's
what
most
you
know,
it's
all
apartments
everywhere
and
it
just
fit
the
the
model
pretty
well.
A
But
when
you
look
at
an
apartment
and
you
think
about
a
landlord,
that's
having
to
manage
it,
there's
there's
a
lot
of
things
that
they're
having
to
keep
in
mind.
First
off
how
many
buildings
do
they
have?
How
many
floors
are
in
each
building?
How
big
are
the
units
you've
got
see
some,
maybe
studios
some,
you
know
three
or
four
bedroom
units.
You've
got
lots
of
different
sizes.
You've
got
shared
infrastructure
in
each
of
these
buildings.
A
So
do
you
you
know:
do
you
trust
everybody
in
the
same
building
to
utilize
that
shared
infrastructure
well-
or
you
say,
nope,
and
I'm
going
to
create
another
building
and
put
these
tenants
over
in
that
other
book?
So
it
just
matched
the
analogy
really
really
well
for
us
and
just
fyi
for
those
that
are
still
staying.
There's
plenty
of
seats
over
here
too
so
feel
free
to
cut
over
any
point
as
well,
so
for
us
when,
as
we
were
thinking
about
the
goals
of
the
landlord,
we
wanted
to
have
quite
a
few
different
things
here.
A
The
landlord
should
allow
us
to
define
all
of
our
tenants,
but
also
bm
be
able
to
do
so
in
a
way.
That's
configurable.
Okay,
this
this
tenant
needs
slightly
different
rules
than
another
tenant.
How
do
we
set
that
up?
We
also
wanted
to
be
able
to
do
everything
declaratively
and
item
potentially.
So
if
I
redefine
all
my
tenant
configuration
and
if
I
add
new
tenant-
and
I
read-
you
know
reapply
everything
it
shouldn't
should
mess
things
up.
A
But
it's
it's
neat
to
see
how
these
line
up
the
the
last
point
here
was
something
that
was
pretty
big
on
us,
because
for
the
most
part,
our
team
didn't
have
a
lot
of
developers
on
our
core
platform
team.
So
the
last
thing
we
wanted
to
do
was
say:
hey
we're
going
to
create
some
crds
and
we're
going
to
create
a
controller
we're
going
to
have
to
like
manage
all
this
stuff
ourselves.
A
So
we
wanted
to
be
able
to
define
a
landlord
in
a
way
that
could
utilize
tools
and
things
that
we're
already
familiar
with
with
again
not
having
to
do
custom,
programming
and
maintenance
here.
So
as
we
did
some
research
and
after
actually
deploying
tenants
a
couple
times,
we're
like
helm,
just
solves
all
this.
A
For
us
and
we'll
see
this
here
in
just
a
minute,
but
it
checks
all
the
boxes
for
us
that
we're
able
to
you
know
theoretically
create
a
chart
that
basically,
is
our
landlord
and
help
chart
and
then
giving
it
a
custom
values
that
defines
the
tenants
that
we
need.
Then
we
can
customize
each
tenant
based
on
on
their
specific
needs
and
with
that
again
I'm
a
big
believer
of
a
programming
on
the
spot.
A
Here
too,
so
we're
gonna,
we're
gonna,
build
a
landlord
helm
chart
together
here
and
we're
gonna
walk
through
the
process
a
little
bit
here.
Okay,
so
first
things
first
well:
every
tenant
needs
a
namespace
that,
hopefully
everybody
at
least
knows
that
at
this
point,
okay,
you
should
give
every
tenant
their
name
space.
A
I'm
I
have
a
top
level
key
of
tenants
and
then
I've
got
team,
awesome,
team
cats
and
team
dogs
here,
and
so
just
starting
from
here,
I
can
have
a
helm
template
that
what
it's
going
to
do
is
it's
going
to
do
a
range.
That's
basically
going
to
iterate.
All
through
all
the
tenants
pull
up
the
key
value
pairs
and
I'm
just
going
to
basically
create
this
namespace
object
once
for
every
tenant.
A
A
If
I
template
this
out,
we'll
see
that,
as
expected,
I've
got
name
space
for
team
dogs,
one
for
team
cats,
one
for
team
awesome,
okay.
So
at
this
point
I've
got
a
really
basic
helm
chart
I'm
super
on
the
soft
side,
and
I
would
just
assume
I'm
going
to
give
everybody
credentials
to
this
and
hooray.
I've
got
a
you
know
landlord
or
multi-tenant
hooray.
Obviously,
it's
not
a
good
setup,
but
let's
go
ahead
and
install
this.
A
All
right
so
that's
been
installed,
and
if
I
look
at
my
namespaces,
I
see
my
my
three
team
namespaces
there
cool.
So
let's,
let's
keep
moving
through
our
spectrum
here.
So
the
the
the
next
thing
we
actually
have
to
ask
is:
how
are
our
tenants
going
to
actually
deploy
things
into
our
cluster
and
it'd?
Be
pretty
bad?
If
I,
I
might
get
ops
con
and,
I
didn't
say,
get
ops
right,
otherwise,
it'd
be
like
wait.
Why
is
he
even
speaking
here?
So?
A
A
We
would
hand
that
to
them
and
basically
say:
hey
you've,
got
maintainer
rights
to
this,
and
then
whatever
you
put
in
there
is
what
we're
deploying.
So
we
we
own
those
manifest
repos
and
then
we
kind
of
give
it
back
to
the
application
teams
to
do
whatever
they
want
with.
And
so
you
know,
some
app
teams,
you
know,
went
through
the
full
ci
pipelines.
A
We
were
a
git
lab
shop,
and
so
they
would
take
the
get
lab
ci
and
would
clone
that
manifest
repository,
push
new
manifest
into
it
and
commit
push
it
all
that
kind
of
stuff
and
and
have
it
as
part
of
a
fully
automated.
And
then
we
had
other
teams
that
were
like
we
don't
trust
any
ci,
cd
and
and
so
we're
just
going
to
make
all
of
our
changes
manually,
which
okay
cool.
So
for
us,
one
of
the
the
big
advantages
is
by
using
get
ops
again.
A
It
allowed
us
to
support
all
the
different
ways
that
teams
wanted
to
to
deploy
things.
We
had
other
teams
that
wanted
a
little
bit
more
change
management
process
and
so
okay
cool,
here's,
your
manifest
repository
and
they
went
through.
Like
a
whole,
pull
request
review
process
internal
to
their
team.
Again,
we
as
a
platform
team.
We
don't
care
how
your
manifest
get
updated
as
long
as
whatever's
in
the
main
branch
is
what
you
want,
the
one,
that's,
what
we're
we're,
treating
as
a
source
of
truth.
A
However,
it
gets
there
that's
up
to
you,
and
so
that
was
one
of
the
kind
of
really
gratifying
things
for
us
as
we
started.
Seeing
more
of
our
our
customers.
Using
this
that
get
ops
was
the
right
choice
here.
But
in
order
to
do
this
as
a
as
a
platform
team,
there's
there's
quite
a
few
different
things
that
we
need
to
do
so
for
each
tenant.
We
need
to
create
a
service
account
within
that
namespace
that
has
access
to
be
able
to
create
resources
within
that
namespace.
A
A
By
creating
a
service
account
within
that
tenant's
namespace,
it
sandboxes
them.
It
keeps
them
within
their
their
namespace.
So
even
flux
can't
apply
things
outside
of
their
namespace.
Then
we
create
a
git
repository
that
fetches
the
source
material
and
create
a
customization.
That
then
applies
the
manifest.
A
So
here's
some
quick
little
examples
here
so
for
team
awesome
we're
going
to
and
again
this
is
some
of
the
things
that
we
can
adjust
over
time.
We
have
some
teams
in
their
manifest
repos.
They
just
dropped
on
the
manifest
at
the
root,
while
others,
you
know,
had
a
whole
directory
structure
and
and
and
so
we
wanted
to
be
able
to
to
customize
the
paths
for
each
of
the
different
groups.
A
Our
back
plugged
in
here,
and
so
what
this
is
going
to
do,
is
it's
again
going
to
loop
through
all
the
tenants
and
I'm
going
to
create
a
service
account
within
that
tenant's
namespace,
and
I'm
going
to
create
a
role
binding.
That
then
says
for
that
service
account.
We
just
created,
give
it
the
admin
cluster
role
access,
but
since
this
is
a
role
binding,
it's
it's
giving
them
admin
access
just
inside
that
namespace,
okay,
and
one
of
the
reasons
we
did.
A
Our
actual
config
objects
here
and-
and
so
in
this
case,
we're
going
to
create
a
git
repository
and
a
customization,
and
in
this
case
again
for
a
platform
team,
we
created
a
separate
repository
for
each
tenant
for
demo
purposes.
I
don't
want
to
have
to
create
tons
of
different
repositories,
and
I
also
want
each
of
you
to
be
able
to
go,
take
the
source
code
and
actually
run
with
it
without
having
to
create
tons
of
different
repositories
so
that
there
is
an
adjustment
here
in
which
I'm
actually
saying.
A
Awesome
cats
and
dogs,
okay,
and
so
what
this
is
going
to
do
is
once
we
apply
this
in
this
code
repository
here,
I've
got
a
directory
called
t
manifest,
which
basically
is
a
simulation
of
those
individual
team
repositories
and
so
in
in
team
cats,
for
example.
If
I
open
this
up
I'll
see
a
certificate
I'll
see
a
deployment,
that's
going
to
deploy
just
a
silly
little
cats
app
that
I've
got
defined,
ingress,
etc.
A
A
A
A
So,
let's
talk
about
locking
things
down
just
a
little
bit
here,
quick
show
of
hands
how
many
people
have
used
gatekeeper
before
okay,
so
there's
there's
quite
a
few
hands.
I'd
say
about
half,
okay,
so
gatekeeper.
First
off
the
first
thing
I'd
say
is:
don't
try
to
write
your
own
policy
engine?
Don't
just
don't
gatekeeper
for
those
that
aren't
familiar
is
basically
a
kubernetes
wrapper
around
the
open
policy
agent
engine
so
use
an
opa
or
open
policy
agent.
You
can
write
policies
using
a
language
called
rego
and
then
what
gatekeeper
does?
A
Is
it
wraps
it
around?
So
then
it's
basically
an
emission
controller
says
all
requests
are
coming
through
the
kubernetes
api
to
make
changes
to
state
whether
to
create
a
new
object
or
delete
an
object.
Whatever
gatekeeper
can
be
notified
and
you
can
basically
add
your
additional
policies
so
remember
earlier,
I
was
saying
I
want
to
allow
teams
to
create
services,
but
I
don't
want
them
to
be
able
to
create
node
port
services.
A
Okay,
so
one
of
the
things
that
I
want
to
do
is
as
a
platform
team,
we
created
gatekeeper
policies
that
basically
satisfy
all
the
pod
security
standards,
at
least
the
baseline
level.
So
we
didn't
allow
any
of
our
tenants
to
run
privileged
pods
or
to
mount
the
host
volume
host
file
system
into
the
container
use
host
namespaces.
Basically,
all
those
kind
of
standard,
pod
security
policies,
and
so
what
we
want
to
do
is
say
all
right,
landlord,
you're
in
charge
of
making
sure
that
all
these
policies
are
actually
being
defined.
A
In
this
case,
a
kate's,
pss
baseline
privilege
container
object
and,
and
then
I'm
going
to
apply
it
to
all
the
name,
spaces
that
are
being
used
by
tenants
and
so
within
these
namespaces,
it's
not
going
to
allow
any
privileged
container
to
run
okay
and
then,
after
that,
then
what
we
can
do
is
we
can
have
some
parameterized
ones
so
again,
going
back
to
the
idea
earlier
that
we
don't
want
to
allow
teams
to
to
take
ingress
names
for
which
they're
not
authorized
to
use.
Well,
that's
just
another
gatekeeper
policy
at
this
point.
A
Gatekeeper
policies
allow
us
to
provide
additional
parameters
to
them,
to
say
for
this
tenant
you're
allowed
to
use
domains
a
b
and
c
while
another
tenant
may
be
allowed
to
use
x,
y
and
z,
and
and
so
when
the
policy
runs.
It'll
use
those
those
parameters
as
part
of
the
policy
enforcement
and
so,
for
example,
I've
got
team
cats
over
here.
That's
able
to
use
cats.local.mike
here
now.
A
If
I
a
in
the
the
dogs,
try
to
use
this
name,
assuming,
obviously
it's
being
cut
off
a
little
bit
here,
but
that
they
they
wouldn't
be
allowed
to
use
that,
and
so
when
the
policy
runs,
it
would
prevent
anybody
else
from
using
that
domain
name
and
so
gatekeeper
again
being
a
mission
controller
would
totally
block
that
from
happening.
So,
let's.
A
A
So
if
it
has
a
field
host
path,
then
hey
that
that's
going
to
be
night
here
and
we're
going
to
prevent
that
from
happening.
So
this
chart
just
defines
all
those
policies
and
all
the
rego
one
of
the
things.
So
most
of
these
I
inlined
directly
into
the
object,
but
one
of
the
things
that
we
learned
as
a
platform
team
was.
It
actually
made
a
lot
more
sense
to
extract
the
rego
out
into
separate
files,
separate
directories
here,
so
we,
for
example,
the
authorized
domain
policy
that
I
was
just
talking
about.
A
Our
ci
pipelines
could
actually
run
all
of
our
tests
for
it
and
then,
as
part
of
the
helm,
build
it
would
just
extract
the
file
and
line
it
into
our
home.
Chart
too.
So
there's
an
example
of
that
happening
here
as
well
too.
If
anybody
wants
to
dive
into
that
anyways,
okay,
so
let's
go
back
to
our
landlord
chart.
Let's
get
all
of
our.
A
A
A
Tenets
so
one
of
the
things
that
we
we
did
from
the
start
is
we
would
authorize
domains
for
their
tenant,
namespace
dot
sum
identifier,
and
so
in
this
case,
what
I'm
going
to
do
is
basically
say:
hey
team
awesome
is
automatically
authorized
to
use
team
awesome.tenants.local.microsoft87.training.
So
basically
here's
a
default
domain
that
you
you
can
use,
and
you
don't
have
to
worry
about
anything
else,
and
so
we
would
put
this
within
our
own
cluster
dns
and
all
that
kind
of
stuff.
So
if
teams
just
wanted
to
prototype
things
quickly,
they
could
do
that.
A
Back
over
here
and
let's,
let's
go
to
team
cats
and
I'm
going
to
try
to
break
this
a
little
bit
and
I'm
going
to
try
to
just
run
a
privileged
pod
here
and
normally
I
would.
I
would
commit
this
I'd
push
it
up.
You
know
let
it
go
through
the
get
ops
thing,
but
just
due
to
time
and
whatnot
I'll.
Just
do
it
manually
here
so
teen
cats.
A
A
A
C
A
A
But
we've
we
quickly
adopted
carpenter
and
we're
a
big
fan
of
carpenter,
because
carpenter
allows
us
to
define
all
these
node
pools
and
the
provisioners
and
everything
using
config
and-
and
so
as
an
example.
Here,
here's
a
provisioner
that
I
can
create
the
taints
and
and
put
the
the
various
taints
and
the
labels
that
need
to
be
on
those
specific
pods
and
then
to
force
the
pods
into
the
node
pools.
Gatekeeper
also
can
be
a
mutating
controller,
and
so
what
we
would
do
is
we'd
say
hey
for
this
particular
tenant.
A
So
again,
the
teams
didn't
have
to
add
these
taints
and
tolerations
and
node
selectors
and
all
that
kind
of
stuff
we
did
automatically
for
them
just
by
running
on
the
platform
and
again
this
was
something
we
could
script
out
and
just
put
into
our
landlord
just
wrapping
up
here
too,
a
couple
other
things
that
we
did
in
our
landlord.
We
we
ran
file,
beat
in
our
cluster
and
we
would
gather
up
all
the
logs
and
send
them
off
to
splunk.
Every
team
had
different
splunk
indexes
that
they
wanted
to
use
so
cool.
A
We
would
just
say
tell
us
the
annotations
that
you
want
and
we'd
mutate,
those
on
as
well
and
and
then
so
all
pods
that
spun
up
in
their
their
particular
name
spaces
would
would
ford
off.
We
also
integrated
our
back
with
our
our
idp
on
campus
as
well,
and
this
is
a
pretty
common
pattern.
We
use
a
keyboard
dc
proxy
and
kind
of
go
from
there
too.
So,
and
so
one
last
thing.
Obviously,
in
this
demo
I
was
doing
everything
with
just
helm,
install
helm
upgrade.
A
B
D
So
about
like
the
poll
like
you
say,
you
use
gatekeepers,
especially
like
admission
web
hooks
so
like
I
always
like.
Maybe
it's
like
just
like
in
the
github's
pattern.
Thinking
right
like
I
ultimately
would
block,
then
the
release
of
my
my
tenants
that
would
use
a
invalid
configuration
right.
So
what's
your
preferation
like
actually
blocking
the
full
sink
or
allowing
it
into
and
kind
of
like
and
then
afterwards
having
a
control
understand,
I
cannot
just
take
it
and
do
something
with
it
or
I
need
to
ignore
it.
A
A
web-based
ui
word
that
the
all
the
tenants
can
log
in
it
was
basically
kind
of
set
up
for
a
more
get
ups
flow,
where
the
dashboard
is
fully
read
only
and
here's
your
deployments
and
all
that
kind
of
stuff.
But
one
of
the
things
that
we
did
is
up
in
the
top
left
corner.
It
would
basically
was
the
status
of
the
customization,
so
it
would
tell
you
what
commit
had
synced.
Was
it
successful
or
was
it
not?
A
And
so,
if
you
committed
it
like
an
ingress
object
or
that
user,
a
wrong
domain
or
a
node
port
service,
you
know
anything
that
invalidates
yeah.
The
customization
sync
would
fail
because
the
policy
prevented
it
from
being
applied,
but
then
we
would
have
that
right
there
on
the
screen
and
and
here's
the
the
error
message
you
got
from
gatekeeper
and
and
why
not
too
so
so?
Yes,
we
did
block
that
sync
from
because
the
emission
controller
blocked
it,
but
then
we
also
gave
them
here's
the
visibility
on
why
it
didn't
happen.
A
A
Yeah,
so
that
that's
a
great
question
the
we
had
a
couple
different
reasons
to
do
it.
Some
teams
didn't
trust,
other
teams
to
be
running
on
the
same
nodes
with
with
them
and
so
that
that
was
part
of
it.
So
how
much
cross-team
trust
was
there,
but
also
we
use
separate
node
pools
also
as
a
cost
accounting
measure.
So
we
would
spin
up
each
of
the
different
node
pools
and
put
cost
allocation
tags
since
we're
running
aws,
and
then
we
can
know
hey.