►
From YouTube: Multi-Tenancy: Tips, Tricks, Tools And Tests - Adrian Ludwin, Tasha Drew, Ryan Bezdicek, Fei Guo
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe in Amsterdam, The Netherlands from April 17-21, 2023. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Multi-Tenancy: Tips, Tricks, Tools And Tests - Adrian Ludwin, Google; Tasha Drew, VMware; Ryan Bezdicek, Twilio; Fei Guo, Alibaba
Speakers: Adrian Ludwin, Tasha Drew, Ryan Bezdicek, Fei Guo
Join the maintainers and leaders of the upstream Kubernetes working group for Multi-Tenancy for an overview of the tools, documentation, tests, and capabilities you can achieve to share Kubernetes clusters between teams and users. We'll also save time for audience questions, so bring your multi-tenancy hopes, dreams and woes!
A
Hi
everybody
and
welcome
to
the
multi-tenancy
working
group
update
for
kubecon
North
America
in
Detroit.
We're
super
excited
to
be
here
today
and
we're
going
to
be
doing
a
brief
review
of
all
of
the
tools
and
capabilities
that
the
multi-tenancy
working
group
has
been
releasing
and
give
you,
and
so
like
kind
of
glibly,
we're
going
to
be
doing
tips,
tricks
tools
and
tests,
and
also
talking
about
both
where
we've
gotten
in
multi-tenancy
and
kubernetes
with
the
working
group
and
where
we
think
we're
going
to
go
next
so
quickly.
A
B
C
A
So
the
goal
of
multi-tenancy
working
group
in
in
the
kubernetes
community
working
groups
are
set
up
as
sort
of
time-bounded
groups
that
come
from
a
bunch
of
different
special
interest
groups
and
are
looking
to
solve
a
specific
problem
or
challenge
for
the
kubernetes
community.
So
in
this
case,
when
the
working
group
began,
we
were
really
looking
at
how
are
people
going
to
achieve
multi-tenancy
in
kubernetes,
and
so
the
goals
of
the
working
group
here
were
to
define
the
models
of
multi-tenancy.
That
kubernetes
will
just
support,
discuss
and
execute
upon
any
remaining
work.
A
That
needs
to
be
done
to
support
these
models
and
create
conformance
tests.
That
will
prove
that
these
models
can
be
built
and
used
in
production
environments
so
towards
that
what
we
have
worked
on
is
a
project
that
was
originally
called
virtual
cluster
and
is
now
a
cluster
API
provider
called
nested.
We've
also
had
the
hierarchical
namespace
controller
hnc.
It
sounds
like
agency,
but
it's
the
letters
agent
and
see
and
then
the
multi-tenancy
Benchmark
project.
A
If
you
would
like
to
check
out
any
of
these
code
we're
on
GitHub,
you
can
see
the
link
on
this
page,
just
kubernetes.
Sigs
multi-tenancy
we
have
a
Google
Group,
which
you,
where
you
can
see
all
of
the
community
kind
of
regular
Communications
on
our
mailing
list.
That's
also
how
you
get
an
invitation
to
our
regular
meetings
and
all
of
our
documentation,
and
we
have
a
slack
channel
in
the
kubernetes
slack
that
folks
are
welcome
to
join.
We
do
have
regularly
scheduled
meetings
every
Tuesday
at
11,
A.M,
every
other
Tuesday.
E
E
We
found,
of
course,
that
people
could
mix
and
match
them
in
various
different
ways,
but
what
we
have
done
now
is
we've
codified
these
ways
of
talking
about
tenancy
into
documentation
that
went
into
the
official
kubernetes
docs
earlier
this
summer,
and
so
that
was
one
of,
as
we
said,
the
the
original
goals
of
this
group
was
to
to
categorize
all
the
different
types
of
tenancy.
We
feel
that
this
captures
the
vast
majority
of
the
tenancy
models
we
see.
We
also
when
we
started.
E
We
found
people
talking
about
things
like
hard
versus
soft
multi-tenancy
and
again
this
was
very
clear
to
any
one
user.
What
that
meant.
But
what
we
found
again
was
that
everybody
had
a
slightly
different
idea
of
what
that
meant,
and
so
what
we've
done
is
we've
broken
this
down
along
three:
let's
call
them
axes,
control,
plane,
isolation,
data,
plane,
isolation
and
then
other
considerations,
and
each
one
of
these
is
not
binary.
E
And
so
again
there
are
isolation
in
the
control.
Plane
is
on
a
spectrum.
You
can
either
use
exactly
what
kubernetes
gives
you,
which
is
okay,
but
not
a
ton.
If
you
want
to
completely
isolate
control
planes,
you
can
use
projects
such
as
virtual
cluster
or
Cappy
nested,
which
we'll
talk
about
later
and
then
in
between.
Then
you've
got
things
like
hnc,
which
allows
you
to
group
namespaces
into
hierarchies.
E
Hence
they
name
hierarchical
namespace
controller
and
share
things
like
access
controls
and
shortlist
quotas
across
different
tenants,
and
so
so
many
people.
This
is
what
multi-tenancy
means,
but
what
we
discovered
the
more
we
looked
into
it
is
that
this
is
only
part
of
the
story,
and
so
Jim
now
will
be
talking
about
the
other
parts
of
the
story.
C
Yes,
in
kubernetes,
the
data
plane
or
what's
referred
to
as
the
data
plane
is
really
the
worker
nodes
and
the
pods
and
the
applications
that
run
on
it.
So,
just
like
Adrian
covered
the
isolation
techniques,
kubernetes
provides
on
the
control
plane.
There's
several
different
resources,
different
configuration
options.
C
You
can
leverage
to
start
isolating
the
data
plane,
so
the
first
one
which
is
notable
here,
is
Network
policies,
of
course,
so
having
some
default
set
of
network
policies,
where
tenants,
don't
you
know,
can't
see
or
can't
get
to
traffic
from
other
tenants
and
there's
strong
isolation
across
those
that's
important.
Similarly,
on
the
storage
side,
following
some
best
practices
for
configurations
which
the
docs
go
into,
details
on
making
sure
PVCs
are
configured
PBS
and
PVCs,
so
that
one
tenant
can't
either
inadvertently
or
through
other
mechanisms,
claim
you
know
volumes
of
other
tenants
beyond
that.
C
There's
also,
of
course,
things
like
container
sandboxing,
which
are
which
is
a
way
of
creating
stronger
isolation
for
data
and
for
container
runtimes.
So
through
tools
like
gviser
and
other,
you
know
Solutions
like
data
containers
for
sandboxing
and
then
finally,
you
know
what
we've
seen
is
patterns
in
the
community,
where
some
users,
some
large
deployments,
have
used
node
isolations
for
specific
tenants
through
labeling
through
placement
through
other
mechanisms
that
kubernetes
provides
itself.
C
So
all
of
these
combined
and
again
like
Adrian,
you
know
emphasizes
it's
a
spectrum
you
can
choose,
which
ones
make
sense
for
your
deployment
and
come
up
with
the
right
data
isolation
techniques,
but
beyond
that
there's
a
few
other
things
kubernetes
provides
which
are
important
to
think,
consider
and
leverage
as
well.
One
of
it
is,
of
course,
we've
all
heard
of
noisy
neighbors
and
tenancy.
C
Now,
of
course,
if
this
this
is
teams
sharing
the
same
cluster,
maybe
you
have
a
little
little
bit
more
control
of
this
within
an
Enterprise,
but
if
you
have,
if
you're,
using
multiple
customers
as
your
use
case,
you
want
to
make
sure
that
there's
priority
fairness
across
those
customers
across
the
different.
You
know
types
of
tenants
that
you
have.
Similarly,
you
can
configure
quality
of
service
for
your
workloads
and
kubernetes
gives
you.
You
know
an
extensible
set
of
classes
for
this,
which
you
can
leverage
across
different
tenant
types
and
within
your
tenancy
models
itself.
C
And
finally,
one
interesting
thing
to
call
out
is
kubernetes,
of
course,
in
a
cluster
you
have
your
cluster
constructs.
You
have
shared
services
like
DNS
is
a
good
example
of
a
shared
service
which
is
necessary.
You
know
to
complete
the
runtime
operations
and
though
you
have
to
think
about
whether
you
would
you
know
you
can,
you
know,
manage
a
shared
DNS
service
or
you
would
want
a
DNS
per
tenant
so
that
you
know
you
can't
do
a
lookup
or
other
things
across
tenants
itself,
but
all
of
these
constructs
are
available.
B
This
is
generally
the
easiest
lift
for
Dev
teams
to
start
consuming,
since
they
don't
need
to
learn
kubernetes
as
much
as
they
may
and
others,
but
it
does
have
some
trade-offs
of
more
operational,
toil
for
teams
and
versioning
of
operators
Etc
the
next
kind
of
level
down
from
that
would
be
a
cluster
as
a
service,
which
is
what
many
teams
I
think
broadly
start
with.
Each
teams
get
their
own
cluster.
B
They
have
a
lot
more
broad
control,
but
as
a
Dev
team,
you
either
are
responsible
for
a
lot
more
of
that
operational
task,
or
you
have
a
centralized
SRE
team,
which
gives
you
a
lot
of
clusters
for
all
and
a
kind
of
a
happy
medium
between
that
it
would
be
the
control
plan
as
a
service.
An
example
here
would
be
the
virtual
cluster
service,
where
each
team
has
their
own
control
plane,
but
the
workloads
are
shared
across
many
worker
nodes
with
within
isolation,
meaning
from
an
operational
perspective.
B
And
so
those
are
our
three
tenancy
models
that
we've
we've
kind
of
outlined
in
our
blog
post
here
and
within
our
docs,
and
with
that
we
are
going
to
move
on
to
a
panel
discussion
asking
the
other
team
members
about
these
different
models
and
and
what
projects
we've
created
that
help
you
implement.
These.
B
And
our
first
question
will
go
to
Faye
Faye,
being
a
cork
contributor
to
virtual
clusters
and
cluster
API
provider
nested,
and
now
that
you
know
virtual
cluster
did
merge
with
cluster
API
provider
nested.
How
do
they
the
projects
cooperate?
You
know
what
new
features
can
we
leverage
now
that
is
in
cluster
API,
you
know:
can
you
kind
of
briefly
discuss
the
differences
and,
what's
going
on
there,
no
you're
unmuted,
okay,.
D
Sex
right,
so
I
can
show
us
a
little
bit
about
the
background
so
to
watch
across
the
project
was
originally
have
a
crd
called
cluster
version,
which
is
used
to
do
the
tenant
control
plan
management.
Well,
that's
theory
is
pretty
simple,
which
which
is
kind
of
with
a
building
in
memory
evcd,
and
there
is
no
probability
of
doing
upgrades
so
because
of
that
we
are
trying
to
see
if
we
can
leverage
the
existing,
the
big
Cappy.
D
You
know
Community
for
doing
the
computer
management,
so
that
is
a
purely
conductive,
deploying
the
project
and
without
any
workloading
building.
So
we
should,
we
can
see
places
like
water
cluster,
where
you
don't
need
a
dedicated,
node
resources,
but
that's
the
reason
we
think
of
these.
Two
projects
are
good
fit
together
and
we
put
this
new
project
in
my
repo
for
easy
developers
and
management
from
the
future
perspective,
I
think
from
the
virtual
for
the
virtual
class
itself,
it
kind
of
mergeful.
D
In
the
past
years,
our
major
Focus
was
on
the
bug,
fixes
and
test
coverage,
and
we
do
have
several
enhancements
in
a
sense,
especially
for
the
extensibility,
for
other
other
features
that
people
count.
One
example
is
The,
Talented
scope,
the
quartile
management,
so
we
are.
We
are
supporting
this
in
a
slightly
different
way
that
we
don't
Define.
The
API
for
this
capability
is
that
we
use.
D
The
way
like,
like
you
know,
kubernetes
schedulers
is
that
we
have
some
essential
point
in
our
virtual
cluster
synchronization
controller,
so
people
can
extend
some
other
Dean
checks
when
we
do
part
provisioning.
So
by
that,
by
having
this,
people
can
develop
their
own
quota
Checkmate
mechanism
using
their
own
crds,
but
that
solution
can
be
integrated
with
virtual
classes
so
so
that
we
can
come
with
work
together
with
customized
quota
capability
checks,
the
virtual
cluster.
D
From
the
calcium
perspective,
we
are
working
towards
a
speech
to
use
a
class
API
style
control
plan
provisioning
model,
but
the
challenge
is
that
we
trying
to
leverage
the
most
of
the
code
of
the
kubi
admin
control
plan,
deployment
implementation,
but
we
are
in
well
in
kpn.
There
is
one
channel
that
we
have
to
mark
the
machine
providing
infrastructure,
because
the
virtual
control
plan
doesn't
need
the
dedicated
note.
So
this
requires
some.
You
know
certain
kind
of
implementation
efforts
on
that
we
are
still
working
it
here.
D
B
E
Yeah,
so
we
hit
one
point
out
earlier
this
year
and
so
we've
been
working
on
1.1,
it's
a
little
slow
going
because,
as
as
is
always
the
case
with
open
source,
a
lot
of
us
are
volunteers,
and
so
the
the
big
feature
that
we've
got
planned
for
1.1
is
hierarchical.
Resource
code
is,
and
so
these
are
drop-in
replacement
for
regular
resource
quotas,
but
they
do
not
apply
to
only
one
namespace.
E
E
Is
it
copies
policy
objects
from
parents
to
children,
so
you
put
in
our
back
permission
in
the
parent
it'll
get
copied
to
the
child,
and
what
we
have
allowed
in
the
past
is
this
concept
of
exceptions,
which
is
that
well
it'll
get
copied
to
all
of
the
children,
except
that
one
people
wanted
basically
to
invert
that
they
say
well
by
default.
I,
actually
don't
want
to
copy
any
of
these
things.
Unless
I
specifically
ask
for
it
to
be
propagated,
so
that
is
a
new
feature
that
will
be
going
in.
E
That's
probably
all
that's
going
to
make
it
into
1.1.
Beyond
that
we,
the
the
list
of
features,
is
actually
getting
fairly
short
in
terms
of
future
features
that
we
want
to
add
and
we'll
be
looking
at
things
like
startup
latency
improvements
and
stability
improvements,
not
that
there
are
that
many
issues,
but
you
can
get
some
Corner
cases,
especially
on
Startup.
The
only
other
thing
that
we've
been
asked
for
from
time
to
time
is
the
ability
to
propagate
objects
that
don't
actually
exist.
E
E
Which
is
good,
hnc
was
not
designed
to
be
a
project
that
would
constantly
grow
and
flower.
It
was
supposed
to
to
plug
a
fairly
narrow
problem
set,
which
is
that
it's
hard
to
manage
groups
of
namespaces.
What
is
the
minimal
construct
that
we
can
add
to
that
in
a
pretty
solid
and
well-contained
way?
That
would
allow
us
to
do
that,
so
we're
getting
pretty
close
to
that
with
1.1
and
I.
Don't
anticipate
there
being
a
ton
of
changes
after
that,
of
course,
the
more
people
who
use
it.
E
B
And
and
to
to
face
Point
as
well
I'd,
say
any
anybody
willing
to
volunteer
or
contribute
reach
out
on
Slack
great
moving
on
to
Jim
Jim.
You
are
also
a
big
player
in
the
policy
working
group,
and
you
know
one
of
the
the
founders
of
keeper.
No,
can
you
kind
of
describe
how
kuberno
can
maybe
help
people
implement
multi-tenancy.
C
Absolutely
yeah,
so
we
talked
about
you
know,
maybe
about
eight
to
twelve
different
types
of
resource
configurations
that
are
available
in
kubernetes
for
both
control
plane,
as
well
as
data
plane,
isolation.
Some
of
the
other
things
like
API
fairness.
So
the
simple
way
to
think
about
it
is
policy.
Engines
can
be
used
to
enforce
or
even
automate
like
kuberno
has
features
from
mutation
as
well
as
generation
of
resources,
and
you
can
validate
for
proper
configurations
right.
C
So
using
policy
engines
in
conjunction
with
hnc
in
conjunction
with
Cappy,
allows
a
lot
of
flexibility
and
power.
Because
now
you
can,
you
know,
enable
self-service
to
your
tenants
if
they're
internal
tenants
and
dealing
directly
with
kubernetes
and
Kube
cuddle,
and
but
you
still
have
the
right
guard
rails
in
place
and
you're
automating.
All
the
required
configurations.
B
Awesome
yeah
I'm,
not
I'm,
a
big
fan
of
giver,
no
and
I
use
it
heavily.
So
I
appreciate
it
great
thanks,
Jim
and
and
Tasha
over
to
you.
You
know,
as
you
mentioned,
we've
been
working
on
these
projects.
For
you
know
almost
four
years,
we've
done
a
lot
what's
next
for
the
multiple
times,
you're
working
through
yeah.
A
Great
question
so
when
we
all
got
together
and
I
think
most
of
the
people
on
the
zoom
were
there
in
Seattle,
when
we
all
met
in
a
tiny
office
just
to
kind
of
talk
about
multi-tenancy
and
kubernetes
and
the
working
group,
and
it
was
really
interesting
because
at
that
time
we
had
a
lot
of
questions
coming
to
us
from
various
parts
of
the
kubernetes
community.
As
far
as
are
we
going
to
need
to
change
the
API
mechanics
of
kubernetes?
A
Are
we
going
to
do
need
to
do
like
a
major,
refactor
or
really
re-architect
the
way,
kubernetes
itself
Works?
In
order
for
us
to
meet
what
users
of
kubernetes
are
telling
us?
They
need
from
a
multi-tenancy
perspective,
and
so
that
was
really
what
the
working
group
was
created
for
was
to
kind
of
dig
into
that
problem
and
figure
out.
A
What
did
we
need
to
do
and
what
potential
re-architecture
might
need
to
happen,
and
then
how
critical
would
that
be
and
over
time,
as
we
put
a
project
plan
together
and
kind
of
started
to
think
about
these
elements
of
namespace
as
a
service
control
plane
as
a
service
clusters
as
a
service
and
started
to
build
out
these
projects
with
the
virtual
cluster
project?
That's
now
become
a
cluster
API
provider
and
the
hierarchical
namespace
controller,
and
then
we
now
have
have
an
entire
Sig
devoted
to
just
automating
the
delivery
of
multiple
clusters.
A
We
now
also
have
multi-tenancy
benchmarks
for
folks
to
basically
be
able
to
run
against
their
their
clusters
to
see
have
I
correctly,
if
I
correctly
set
up
this
cluster
for
multi-tenancy
have
I
left
any
sort
of
gaping
holes
anywhere
and
then,
in
addition
to
that,
the
the
working
group,
also
just
in
the
past,
quarter
to
came
together
to
finally
add
official
documentation
to
the
kubernetes
documentation.
Saying
hey.
This
is
what
multi-tenancy
is,
and
these
are
the
tools
you
have
to
address
it.
A
A
We're
the
multi-working
groups
in
general
in
kubernetes.
Upstream
are
not
intended
to
stick
around
forever,
they're
really
intended
to
answer
a
question,
and
so
we're
kind
of
starting
to
feel.
Like
hey,
we've
answered
this
question
fairly
conclusively,
and
so,
unless
people
really
kind
of
jump
up
and
say
no,
no
there's
more
like
here's
something
that
we
need
to
come
together
and
work
on.
It
may
be
time
for
us
to
kind
of,
say,
yay
we
won
and
go
tackle
something
new.