►
From YouTube: Multi-Tenancy in Kubernetes: How We Avoided Clusters Sprawl W... Dario Tranchitella & Maksim Fedotov
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon North America 2021 in Los Angeles, CA from October 12-15. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Multi-Tenancy in Kubernetes: How We Avoided Clusters Sprawl With Capsule - Dario Tranchitella, CLASTIX & Maksim Fedotov, Wargaming.net
Kubernetes is great when you have to deal with it as is and plays well with isolating workloads and limiting resources due to its primitives as ResourceQuota, LimitRange, NetworkPolicy, Namespace, and so on. But is it enough? Can you create a real isolated multi-tenant environment in it? With our experience with many production environments, we discovered it's not entirely. That is the reason why Capsule, an open-source Kubernetes Operator for multi-tenancy was born. Expanding the authentication Kubernetes capabilities, Capsule provides a viable and robust solution to avoid the hyped cluster sprawl while maintaining a native Kubernetes UX. And, last but not least, allowing a BYOD to push compute, storage, and network isolation and avoiding the noisy neighbors' effect. After a brief overview of the project by Dario Tranchitella (maintainer), Max Fedotov will explain how Capsule enhanced the operations and supercharged the provisioning mechanism for their k8s cluster at Wargaming.net
A
A
Well,
well,
a
little
bit
excited
because
it's
my
first
time
at
cubecom
as
speaker
and
also
as
attendee,
and
I'm
really
happy
to
be
here
with
you,
because
we
are
going
to
we
did
with
capsule.
And
it's
really
as
astonishing.
I'm
really
free
to
be
here
and
I'd
like
to
thank
all
the
people
that
is
joining
this
session,
because
we
did
a
great
job
together
as
developers
and
also
as
architect
of
the
software
that
we
built
so
yeah.
I'd
like
to
show
you
how
this
presentation
is
made
of.
A
So
this
is
obviously
the
introduction
and
we
are
going
to
have
two
session,
because
this
is
a
presentation
between
me
and
max
and
I'm
going
to
introduce
you
what
is
capsule,
because
capsule
is
an
open
source
project
and
it's
able
to
provide
a
multi-tenancy.
Also,
it's
the
topic
of
this
session
and
after
that
max
is
going
to
share
what
they
were
able
to
achieve
at
wargaming.net
with
capsule
and
is
going
to
share
some
really
nice
insights
regarding
the
abilities
of
capsule
and
the
platform
that
they
built.
A
A
Hi
people,
my
name,
is
zero
tranquitella
and
I'm
an
open
source
software
engineer
at
classics.
An
italian
startup
based
in
italy,
with
the
mission
of
bringing
kubernetes
everywhere
in
this
section,
I'd
like
to
introduce
you
to
capsule
the
multi-tenancy
operator,
but
first
I
think
I
should
give
you
a
brief
introduction
about
the
multi-tenancy
dilemma
and
the
current
state
of
multi-tenancy
in
kubernetes
itself
and
the
ecosystem.
A
It's
pretty
easy
to
understand.
This
is
common
and
best
practice,
dividing
components
and
applications
in
several
name
spaces,
but
doing
so
doesn't
mean
you're
adopting
multi-tenancy.
There
are
some
aspects
you
have
to
consider
since
it's
like
the
sure
dosing
in
the
mid
to
thousands
and
I'm
referring
to
the
noisy,
neighbor
effects.
A
We
all
know
kubernetes
is
great
at
crafting
distributed
systems
and
provides
out
of
the
box
some
resources
that
help
you
to
tackle
the
downside
of
dealing
with
a
multi-talent
environment,
the
resources
I'm
referring
to
are
limit
range
resource
quota
and
the
well-known
and
famous
network
policy.
I'm
not
going
to
do
a
one-on-one
introduction
about
this,
because
documentation
is
there
for
this
kind
of
stuff,
but
I'm
going
to
provide
a
brief
overview
since,
with
the
combination
of
these
resources,
you
can
achieve
complex
multi-tenant
strategies
said
so.
A
Limit
range
is
the
resource
you
have
to
use
to
set
a
minimum
and
or
a
maximum
amount
of
resource
requests,
also
known
as
cpu
memory
and
storage.
These
policies
are
useful
to
set
the
minimum
amount.
A
part
has
to
use
really
useful
when
dealing
with
unconstrained
containers
that
can
lead
to
cpu
throttling
or
out
of
memory
of
node
as
well
for
the
upper
limits.
Since
few
things
are
infinite
and
resources
pretty
sure
are
not
except
you
are
using
a
cloud
provider
and
it's
not
your
business
anyway.
A
Following
we
got
the
resource
quota,
this
api
allows
you
to
specify
some
constrained
policies
for
a
specific
set
of
resources.
Let's
imagine
that
you
have
to
limit
the
amount
of
pvc
requested
by
a
namespace
persistent
volume
claims.
This
is
the
perfect
use
case,
as
well
as
limiting
the
amount
of
pods
to
avoid
the
container
sprawl
and,
finally,
the
well-known
network
policy.
A
I
guess
we
all
know
them,
because
it's
used
as
a
sort
of
firewall
for
the
applications
that
we
deploy
in
our
clusters,
so
sometimes
we
are
biased
to
consider
them
as
a
sort
of
security
thing,
and
this
is
absolutely
true,
though,
however,
network
policy
must
be
used
to
deny
communication
between
services
that
don't
belong
to
the
same
tenant.
For
obvious
reason,
I'd
say:
security
in
first
place.
A
Okay,
great
now,
we
all
know
all
the
tools
to
build
up
your
tenant
platform.
Besides
the
downside
that
all
these
resources
are
namespaced
scope
and
it
means
that
the
enforcement
of
these
policies
is
applied
just
to
the
namespace,
where
the
resource
is
living
and
this
doesn't
apply
well
in
a
multi-tenant
environment,
but
exactly
how
a
tenant
can
be
defined.
A
Let's
try
to
summarize
how
it
can
be
interpreted
properly
in
kubernetes
ecosystem,
and
I
think
the
most
basic
definition
could
be
a
collection
of
namespaces
where
each
namespace
is
collecting
the
resources
requested
for
a
specific
component
or
application
owned
by
a
team
or
a
group
of
developers
and
in
the
end,
if
you
start
thinking
about,
if
you
are
giving
developers
access
to
your
kubernetes
clusters,
you
are
just
granting
administrator
capabilities
to
the
namespace.
They
have
to
manage,
and
sometimes
just
in
read-only
mode
and
working
with
multi-tenancy
with
just
a
team
is
pretty
easy
and
straightforward.
A
You
prepare
the
environments,
creating
the
namespaces
set,
the
kubernetes
resources
we
presented
early
and
I'm
referring
to
the
limit
range
and
through
policy
resource
quota
and
giving
access
to
them.
But
dealing
with
this
at
scale
could
be
a
nightmare,
and
the
automation
must
kick
in
for
several
reasons.
A
Capsule
is
our
non-opinionated
solution
to
solve
this
suitation.
You
got
a
custom
resource
definition
named
tenant
that
assigns
a
namespace
creation
right
to
their
owners
and
of
the
story.
What
you
just
need
to
do
is
specify
all
the
business
logic
in
the
tenant
manifest
and
the
tenant
owners
can
create
on
their
own,
the
namespaces
that
will
be
treated
as
a
sort
of
single
name
space,
the
tenant
abstraction,
but
before
proceeding
with
capsule,
is
capable
of,
I
think
it's
better
to
explain
how
capsule
was
born
and
yeah.
A
I
still
recall
the
first
call
I
had
with
adriano
the
classic
the
classics
founder
and
he
asked
me
directly
hey
do
you
want
to
deploy
and
develop
a
kubernetes
operator
for
the
multi-tenancy,
and
my
reply
was
even
more
direct.
Why
do
we
need
yet
another
project
to
deal
with
this
if
there
is
red,
dot,
openshift
and
ranger
doing
this
stuff,
because
multi-tenancy
in
kubernetes
is
broken?
A
All
these
projects
are
living
in
the
multi-tenancy
special
interest
group,
and
I
said
earlier
that
kubernetes
is
great
at
designing
distributed
systems,
since
it
allows
to
just
use
the
requested
resources
and
components
to
build
your
infrastructure.
It
doesn't
mean
it's
out
of
the
box.
You
have
to
put
in
place
on
your
own
and
the
same
applies
also
for
the
multi-tenancy
and
that's
the
reason
why
we
started
this
lovely
adventure
together.
A
At
that
time,
classics
was
doing
scouting
for
one
of
its
customers
to
provide
a
public
container
platform,
a
sort
of
heroku
but
built
on
top
of
kubernetes.
If
you
prefer
fancy
names
and
acronyms,
I
think
that
naas
is
the
right
one
namespace
as
a
service
and
juxtaport.
The
prerequisite
of
the
project
was
delivering
a
true
and
native
kubernetes
experience
to
developers
no
need
to
add
further
apis
binaries
to
install
yaml
definitions
and
anything
else
that
called
puzzle
turbines.
A
The
goal
was
issue,
keep
ctrl
from
your
terminal
and
there
you
go
and
we
started
the
project
with
this
in
mind
and
after
some
time
we
realized
that
we
were
building
something
really
disrupting
disrupting,
because
I
was
working
on
it.
Absolutely
not
but
being
non-opinionated
on
the
multi-tenancy
strategy
provided
showed
us
that
we
were
crafting
a
framework
allowing
any
or
almost
level
of
multi-tenancy.
A
But
let's
try
to
step
down
from
the
emotional
talk
and
let's
try
to
drill
down
the
the
capsule
capabilities.
I
don't
want
to
let
this
talk.
Look
like
a
safe
speech.
So,
okay,
what
are
the
capsule
super
capabilities?
I'm
trying
to
summarize
in
small
and
short
sentences
to
leave
enough
room
to
max
presentation.
That
is
absolutely
stunning.
Okay,
you
can
limit
the
name
spaces
that
can
be
created
by
a
tenant
owner
really
useful
for
the
pay-per-use
scenario.
A
You
can
specify
which
ingress
or
search
classes
a
tenant
can
use
as
providing
a
tire
ring
for
the
banking
source
used
by
applications
or
forcing
to
announce
applications
using
a
specific
load
balancer,
and
since
we
are,
we
are
talking
about
networking
at
ingress.
You
can
put
in
place
a
policy
for
the
allowing
resource
names
as
using
a
specific
domain
for
the
exposed
applications
or
you
could
mitigate
the
cve
2020
8554
enforcing
a
specific
set
of
ip
addresses
or
ranges
that
can
be
used
by
the
service
in
the
tenant
and
also
from
the
security
perspective.
A
Since
it's
using
the
pod,
node
selector
admission
controller
available
in
kubernetes
since
the
1.5
release,
if
I
recall
correctly,
however,
when
all
these
features
kick
in
together,
you
can
achieve
for
real
bring
your
own
device
scenario,
because
what
you
just
do
need
to
do
is
to
spin
up
some
worker
notes
provision
with
a
specific
label.
Let
them
join
the
cluster
and
you
can
manage
your
reserved
partitions
of
the
cluster
and
it's
absolutely
amazing,
since
you
are
totally
decreasing
any
side
effect
of
a
multi-tenant
cluster.
A
No
noise
enables
your
own
resources,
it's
like
being
on
a
managed,
kubernetes
cluster,
but
owned
by
your
organization
anyway,
I
know
what
some
of
you
start
mumbling
about.
I
can
already
do
that
on
my
own.
I
just
need
to
write
down
the
dynamic
admission,
controls
and
yeah.
That's
true,
but
there
is
something
you
have
to
consider
the
declarative
way,
also
known
as
the
ammo
determinant.
Yaml
definition,
as
you
can
see,
is
easy
to
understand
and
easy
to
grow
as
any
any
other
resource
in
kubernetes,
but
more.
It
allows
to
combine
all
these
policies
together.
A
That
will
require
otherwise
a
big
effort
on
wiring
all
the
web
books
together
and
building
a
way
to
group
together
the
namespaces
in
abstraction
lieutenant
one
and
keep
a
look
to
the
node
selector
key.
That's
the
keystone,
enabling
the
bring
your
own
device
scenario
beside
the
provisioning
process,
though
this
is
still
required
to
be
configured
at
no
startup
trying.
A
To
recap
all
the
features,
I'm
pretty
confident
to
say
that
capsule
can
be
considered
as
a
sort
of
framework
to
build
your
own
multi-tenancy
solution
without
disrupting
the
developer
experience
that
will
enjoy
a
fully
compatible
api
server,
managing
their
resources
using
cubectl
and
without
dealing
with
any
complex
crds.
Since
there
is
just
detailed
one
and
it's
up
to
the
cluster
administrator
to
to
deal
with
it
and
said
so
yeah.
A
B
Hello,
everyone,
as
you
may
be
know,
my
name
is
max
vidotov
and
I'm
working
as
an
infrastructure
engineer
in
a
maintenance
department
of
war
gaming.
In
my
part
of
talk,
I
would
like
to
share
with
you
how
we
were
able
to
redesign
our
internal
kubernetes
clusters
with
the
help
of
the
capsule,
why
we
decided
to
adopt
multi-tenancy
and
which
benefits
we
got
from
this
decision.
B
But
before
we
start,
I
would
like
to
tell
a
few
words
about
wargaming
for
those
who
are
maybe
not
a
big
fan
of
video
games
so
who
we
are.
We
are
a
global
online
game,
dev
and
publisher,
with
over
220
millions
of
players
who
are
enjoying
our
games
on
all
major
gaming
platforms,
and
I
think
everyone
had
heard
or
maybe
even
played,
our
flagship
products
like
world
of
tanks
or
world
of
warships
and
in
my
department
we
are
in
charge
of
supporting
all
the
infrastructure
which
is
used
in
order
to
run
these
games.
B
B
As
we
are
operating
in
multiple
regions,
we
provisioned
a
single
kubernetes
cluster
per
region
because
it
is
much
easier
to
manage
and
we
were
able
to
provide
additional
in-cluster
services
and
integrations
with
our
internal
system
like
monitoring,
logging,
authentication
and
so
on.
We
used
shared
nodes,
a
mix
of
hardware
and
vms,
and
we
created
a
separate
namespace
per
customer
which
our
internal
development
teams
in
order
to
provide
some
type
of
isolation.
B
We
also
used
kalika
global
network
policies
for
network
installation
and
at
the
beginning,
when
we
hadn't
got
a
lot
of
customers.
This
setup
was
quite
good
for
us,
but
things
changed
when
clusters
start
to
grow
and
we
got
more
customers
on
board.
So
what
was
wrong?
The
first
problem
was
that
namespaces
are
too
granular.
Most
of
our
customers
needed
more
than
a
single
namespace.
They
needed
a
namespace
per
project
or
namespace
per
application,
so
we
had
to
manage
namespace
provisioning
for
them
and
we
also
managed
access
and
permissions.
B
Node
provisioning
was
also
on
us,
so
we
so
we
had
to
do
some
type
of
capacity
planning
and
management,
and
another
problem
was
billing.
So
when
you
had
shared
nodes,
you
need
to
implement
complex
building
rules
and
that's
not
easy
at
all,
really
and
also
due
to
shared
resources.
It
was
very
easy
to
get
a
noisy,
neighbor
effect
that
darya
already
mentioned
in
his
talk
when
customer
workloads
began
to
interfere
with
each
other.
B
So
you
will
ask
me
an
obvious
questions.
If
the
shared
module
was
not
so
good
for
us,
why
not
to
use
a
cluster
per
customer,
and
that
was
one
of
our
initial
thoughts
too.
But
in
this
scenario
there
were
also
some
disadvantages
for
us.
The
first
one
is
that
we
have
six
regions
of
present,
so
in
reality
for
each
customer
we
need
to
provision
not
a
single
cluster
but
six
clusters.
B
So
we
multiply
the
number
of
customers
by
number
of
regions,
and
we
understood
that
it
would
be
too
much
really
for
us,
so
the
maintenance
cost
for
this
solution
will
be
quite
high,
for
another
problem
was
that
we
had
to
support
different
version
of
kubernetes.
Why?
Because
you
know
like
it's
always
in
a
big
companies,
there
is
a
customer
a
who
wants
to
have
the
newest
version,
and
there
is
a
customer
b
who
wants
to
have
the
more
stable
version
and
that's
a
problem
for
us,
because
we
had
different
tools.
B
B
Also,
we
were
a
bit
afraid
that
when
everyone
will
be
sitting
in
their
separate
clusters,
it
will
be
hard
to
share
some
knowledge
of
services
or
ideas.
So
what
we
wanted
to
is
that
if
one
of
our
customers
had
adopted
some
new
technology
or
had
brought
or
implemented
some
operator,
he
should
be
able
to
provide
this
as
a
service
to
all
other
customers.
B
B
So
we
understood
that
having
a
single
cluster
per
region
is
the
best
model
for
us
and
what
we
needed
was
to
broad
some
advanced
multi-tenancy
capabilities
to
kubernetes.
So
we
wrote
down
some
critical
points
that
we
wanted
to
have
in
our
new
cluster
design.
So
the
first
one
was
that
we
wanted
to
have
a
bring
your
own
device
approach,
so
customer
can
bring
his
own
hardware
nodes
or
he
can
provision
a
vm
in
some
of
the
cloud
provider
and
his
workloads
will
run
only
on
these
dedicated
nodes.
B
B
But
then
I
found
capsule
and
it
was
really
a
perfect
fit
for
our
demands.
So
we
became
one
of
the
early
capsule
adopters
and
soon
it
became
a
potential
component
of
our
internal
kubernetes
clusters
and
what
I
really
love
about
the
capsule
not
saying
about
some
technical
details,
a
technical
part.
It
is
its
community
and
maintainers.
B
So
when
you
are
adopting
some
new
open
source
technology,
you
always
found
out
some
use
cases
in
your
environment
that
are
not
already
covered
by
it.
So
you
need
to
add
some
functions,
or
maybe
you
even
don't
understand
how
better
to
use
this
new
product
and
dara
and
other
maintenance
were
always
so
helpful
and
they
helped
us
not
only
with
the
adopting
git,
but
they
also
helped
us
to
contribute
different
features,
and
I'm
really
very
proud
that
now,
I'm
also
one
of
the
capsule
maintainers
too.
B
So
with
the
help
of
capsule,
we
were
able
to
formalize
two
core
concepts
that
we
are
now
using
in
our
internal
kubernetes
clusters.
First,
one
is
tenant,
it's
our
customer,
so
it
has
some
name,
a
group
of
owners
who
had
an
admin
privileges
in
the
tenant
and
the
cost
center,
which
is
a
label
that
we
applied
to
tenants,
node
as
attack,
and
then
later
we
use
it
for
billing.
So
we
can
directly
send
bills
for
a
for
our
customers.
B
Another
one
is
a
nod
group.
It
is
a
set
of
nodes
with
the
common
flavor
that
are
created
in
one
of
the
cloud
providers
and
they're
always
bound
to
for
some
tenant.
So
you
can
think
about
it
like
a
cloud
agnostic,
obstruction
of
aws
after
scaling
groups
or
a
google
cloud.
After
scaling
non-groups,
a
node
group
also
had
a
name
which
we
apply
as
a
label
to
all
nodes,
so
our
customers
can
use
it
later
in
label
selectors.
B
It
is
created
in
some
of
the
cloud
providers
and
all
nodes
in
the
node
group
had
the
same
common
flavor
for
node
group.
We
also
introduced
a
concept
called
roll,
it's
some
type
of
a
predefined
template
so,
for
example,
for
ingress
role.
We
install
and
configure
an
ingress
controller
on
each
of
the
nodes
of
the
node
group,
and
the
last
one
is
mean
not
in
max
nodes.
B
First,
one
is
called
provisioner
it's
our
internal
component,
which
is
in
charge
of
provisioning
everything,
so
it
subscribes
using
web
sockets
to
our
internal
configuration
management
databases
database,
where
information
about
tenants
and
non-groups
is
stored,
and
then
it's
managed
10
on
singles
and
storage
classes
provisioning,
as
well
as
knowledge
group
provisioning
in
different
cloud
providers.
So
this
is
the
core
company
that
communicates
with
all
cloud
providers.
B
B
B
It
has
also.
We
also
had
an
implementation,
a
custom
implementation
of
cloud
provider
interface
for
it.
So,
instead
of
directly
interacting
with
specific
cloud
providers,
it
talks
to
our
provisioner,
and
that
is
the
way
that
we
are
able
to
support
nodes
from
different
cloud
providers
in
a
single
tenant.
B
A
Okay,
welcome
back
people
and
yeah.
It
has
been
really
enjoyable.
I
really
enjoyed
the
presentation
by
max
because
what
they
achieved
that
wargaming
is
absolutely
stunning.
I'm
really
happy
that
max
helped
us
also
shaping
the
capsule
capabilities
and
features,
and
what
do
you
think
max?
Did
we
do
a
great
job.
A
A
A
You
can
do
whatever
you
want
and
obviously
we
are
planning
further
improvements
regarding
also
you
can
see
some
facts
max
because
you
shared
also
the
capsule
proxy
and
the
capsule
proxy
is
playing
a
big
role
in
2d
multi
tenancy,
because
in
the
end
we
are
using
the
capsule
proxy
just
to
provide
the
least
of
the
namespaces
owned
by
the
tenant.
But
we
got
also
another
project
and
that
we
at
the
time
to
share
and
it's
the
lens
extension
so
lens.
A
It's
an
ide
developed
by
mirantis,
and
it's
really
useful,
because
with
this
ide
you
can,
you
can
see
all
the
stuff
that
is
running
in
your
kubernetes
cluster
and
there
is
also
the
ability
to
add
some
extensions
and
we
at
classics.
We
developed
that
with
our
partner,
nikolai
and
adriano,
and
it's
quite
interesting
and
I'd
say
that
it's
really
useful.
A
Well,
it's
not
quite,
but
it's
absolutely
useful
and
take
a
look
to
that,
and
I
would
like
also
to
take
this
chance
to
do
to
ask
for
people
if
you
like
capsule,
if
you
are
also
considering
the
multi-tenancy
topic
really
interesting.
Please
drop
us
a
call
on
twitter
on
linkedin
or
github,
because
we
are
looking
for
contributors
and
also,
if
you
have
feature
requests
or
you
are
facing
some
bugs-
don't
hesitate
to
open
the
issues
because,
as
I
said,
we
are
really
happy
to
get
feedback
from
people.