►
From YouTube: Kubernetes Working Group Multi-Tenancy 20190604
Description
Self-nominations for co-chair:
- Would you like to be the next co-chair of the working group? Attend the meeting, introduce yourself, and say a few words about why you'd be interested! I'm also happy to talk to anyone offline who would like more info about the role
-- Self-nominate slides (Harry): https://speakerdeck.com/resouer/self-nominate-wg-multitenancy-lei-harry-zhang
Further discussion of Propose Virtual Cluster Based Multi-tenancy Solution (By Fei Guo, Alibaba)
-- https://docs.google.com/document/d/1EELeVaduYZ65j4AXg9bp3Kyn38GKDU5fAJ5LFcxt2ZU/edit#heading=h.7tna1yo4dzv (This has been presented to David/Dawn/Daniel)
Tom Runyon has built a multitenant manager for a client of his that is similar to the proposed Tenant CRD, except it:
- Runs on Openshift so he could leverage ClusterResourceQuotas
- Manages tenants across multiple clusters
- He will go over his requirements and walk through a couple pieces of his implementation
-- https://docs.google.com/presentation/d/1x9Db5NGkvj1hzWsUKP4bTlKWWp-830A9VPOR4BDHyic/edit?usp=sharing
B
Yes,
my
name
is
Harry
and
I,
actually
am
staff
engineer
from
Ali
Baba
and
you
can
check
my
github
account
and
restore
I'm.
One
of
the
main
contributor
Kublai's
memory
post
since
2013
with
173
convinced
merge
in
could
be
scheduler
and
correct.
I'm,
also
one
of
the
reviewer
of
sick
note
and
sick
scheduling
and
workers
on
contain
random
interface,
resource
management
and
scheduling
performance,
tearing
stuff
I'm.
Also
one
of
the
maintainer
of
CR
spec,
especially
for
cut
containers.
Since
country
2017
I
began
and
began
working
with
working
group
multi-tenancy.
B
Since
2070
you
can
shake
one
in
my
speak
of
the
working
to
determine
died
during
cubicle,
North
America.
So
so,
and
I
speak
about
the
Alibaba
teaming
related
to
kubernetes.
We
are
actually
highly
invested
in
the
working
group.
What
it
can
see
which
really
figure
out
the
answer,
long
term
strategy.
That's
why
we
actually
proposed
a
virtual
cluster
based
mechanism.
Although
they
are
working
group
and
in
Alibaba.
Actually
they
have
two
models
to
fix
the
harbor
contingency
problems
and
the
basics
have
different
multi-tenancy
views,
but
we
all
have
stem
resource
acceleration
models.
B
So,
first
of
all,
in
the
initial
cluster
in
truck
has
their
wishes
hot,
similar
to
what
we
are
proposing.
Welcome
to
the
community
housing
during
the
VD
working
group,
Sigma,
D,
tenancy
working
group
on
the
Tennessee,
and
also
we
are
proposing
other
model
which
named
a
virtual
cluster,
which
will
give
the
tenants
pertinent
per
watch
cost
model,
but
underlying
in
structure
is
actually
same
because
me
to
fix
the
same
networking
service
model
and,
of
course,
with
the
runtime
model.
B
So
you
can
see
they
are
labelled
back,
should
have
very
solid
you,
the
cases
the
product
for
both
of
the
multitronic
multi-tenancy
structure,
the
first
of
always
alibi
person,
kubernetes
featured
one
of
the
over
public
contact
and
also
we
also
have
another
product
which
name
is
Ali
Baba
soliciting
for
truck
sure,
which
is
a
virtual
cluster,
moti
Tennessee
models
and
stir
over
with
web-scale
clustered
internal.
So
for
the
community
part.
We
think
that
colleague
on
the
team
and
meek
also
still
continue
to
contribute
and
engage
with
the
working
remote
in
Tennessee.
B
We
hope
to
move
over
into
no
experience
to
it
to
the
community
and
to
cooperate
and
accrue
shows
in
Pakistan,
which
will
cost
model
to
real
practice
and
error
issues.
Think
you
multi
planning
something
as
the
multi-tenancy
seeing
is
the
excitation
and,
of
course,
standard
box,
the
runtime
and
how
anything
future.
Eventually
we
will
make
crew
blades
and
interview
so
just
the
best
introduction
and
myself,
and
if
you
have
any
questions,
please
print.
We
feel
free
to
ask.
A
C
Yeah
should
I
go
again,
yeah
hi,
so
I
actually
was
thinking
of
producing
a
couple
of
slides
but
I
in
the
past,
we've
been
less
formal,
so
I
apologize,
I
haven't
produced
slides,
but
I
will
actually
go
ahead
and
share
the
deck
that
we
did.
Just
so
I
have
something
to
show
and
if
that's
okay,
just
to
reflect
on
some
of
the
work
30
terms.
So
if
I
can
take
the
ball.
C
C
C
C
C
So
my
name
is
Sanjeev.
Rampal
have
been
involved
in
this
group,
particularly
over
the
last
several
months,
although
being
on
and
off
involved
with
this
working
group
for
longer
than
that
and
I've
been
at
the
core
of
driving
the
activity
for
the
last
six
months.
Essentially,
this
group
was
flailing
to
be
honest
without
without
meaning
to
be
critical.
This
group
was
actually
flailing
for
a
while
and
I
feel
like
helped
in
a
significant
way,
bring
some
structure.
C
Since
the
meeting
we
had
in
Seattle
putting
together
the
working
group
direction,
as
well
as
working
with
folks
like
history,
on
prototypes
of
cannon
controllers
and
tenant
models,
and
then
we've
also
created
this
framework
for
sort
of
having
a
set
of
architecture.
Models
which
I'm
showing
here
in
terms
of
running
multi-tenant
cubanelle
is
on
is
leveraging
the
as
Tennessee
versus
leveraging
Sierra
DS
versus
the
virtual
cluster
model
that
had
he
just
mentioned,
and
and
also
working
towards
introducing
Tennessee
as
a
first-class
object.
C
C
Yeah
so
yeah
I
and
actually
produced
a
formal
slide
deck
to
nominate
myself,
and
if
you
guys
need
that
I
I
mean
I.
Think
you'll
be
good
to
do
that.
So
I
could
send
out
a
couple
of
slides
after
the
call,
as
well
so
I'm.
Just
sharing
the
work
that
we
presented
on
behalf
of
the
group
to
illustrate
the
fact
that
I've
been
strongly
involved
in
this
working
group.
For
for
a
while
and.
C
You
know
to
just
to
jump
to
this
other
slide
here.
We
are
so
so.
As
I
said
earlier,
this
working
group
has
been
flailing
for
a
while
and
I
I
do
believe
that
I
gave
a
strong
push
to
create
some
structure
and
that
has
led
to
our
getting
better
organized
and
having
a
better
framework
and
having
some
concrete
development
over
the
last
six
months.
C
You
can
refer
to
my
talks,
both
at
the
aqua
conference,
as
well
as
cube
con
prior
to
that
I've,
been
involved
in
in
terms
of
the
in
terms
of
the
open
source
community
have
been
involved
in
products
like
the
context.
You
know,
network
plug-in
I
have
contributed
to
the
installers
for
projects
like
open
shift
and
I
have
done
a
lot
of
product
development
in
the
close
source
area
for
Cisco.
Developing
products
that
are
deployed
all
over
the
world.
C
I
have
over
20
years
of
experience
and
strongly
motivated
in
helping
this
team
move
forward
in
this,
and
then
also
worked
with
closely
with
folks,
like
you
and
others,
on
on
some
of
the
initial
controller
prototypes
for
tenant
templates
and
tenancy
our
days
and
so
on,
some
of
which
we
documented
and
demoed
at
the
conference
as
well.
So
that's
that's
my
background.
C
I'm
sorry,
I
don't
have
a
formal
deck
to
introduce
myself,
but
hopefully
that
give
you
all
a
reasonable
picture
and
I'm
happy
to
send
out
one
or
two
slides
after
the
call
giving
you
more
summary
of
my
background,
I'm,
actually
taking
pointers
from
handy
here,
because
I
haven't
actually
produced
and
I'm
producing
a
formal
slide
for
myself.
So
thank
you.
Any
questions.
A
A
D
Okay,
so
yeah
last
meeting
I
have
a
brief
introduction
to
the
would
show
cluster
architecture,
because
I
only
have
10
minutes
so
last
time,
I
think
last
meeting
I
cover
mainly
this
figure
about
the
high
level
Hecht.
So
today,
I
made
take
another
10
to
15
minutes.
To
first
recap:
this
figure
a
little
bit
and
then
go
through
more
about
the
Margot
Hennessey
service
model
is
well
the
solution
that
we
have
think
about
how
to
you
private,
providing
or
narrow
isolation
in
the
area.
D
D
It
manager
or
the
physical
knows
resource
and
the
tenant
master
is,
and
it
is,
is
deployed
using
a
standard
Canaries
on
kubernetes
Kok
model
and
each
so
so
so
then,
the
tenant
who
owns
the
tenant
master,
which
has
a
four
you
know
API
control
as
IPS
server,
and
the
key
here
is
each
node.
We
have
a
agent,
so
the
the
via
a
generator,
obviously
is.
It
is
just
there's
a
mapping
from
the
VI
computer
actually
knows
in
a
tenant
master
and
the
basically
in
a
tenant
must
ready
to
be
used
via
agent
as
a
node.
D
So
what
he
does
is
every
time
so
the
tenant.
We
are
welcome
the
Terran
omastar
to
do
the
part
part
resource
provisioning
and
the
there
are
10
an
editor
or
the
tenant.
You
know
the
posi
formation.
We
are
going
to
be
sync
to
receive
a
master
using
the
sinker
you're,
using
a
sync
manager
in
the
10-man
controller.
So
then
the
third
coup
bleeds
the
cupola.
D
Is
that
going
to
talk
to
you
to
actually
talk
to
the
super
monster
either
watch
for
the
changes
in
a
super
master,
four-part
creation,
division
addition
is
centra
and
that
you
actually
do
actual
pot
provisioning
wrong,
actual
stars,
all
the
contrary
in
the
past.
In
the
end,
the
results
will
be
sinking
back
using
using
sinking
back
to
ten
in
master,
so
so
that
a
tenant
must
actually
knows
the
parties
actually
is
up
and
the
world
is
the
status
of
the
pot.
So
all
information
are
synced
by
the
tenant
controller.
D
Sync
back
form
a
super
master
community,
video
tenant
master.
So
this
is
the
this
is
the
highlight
of
architecture.
So
as
a
recap,
so,
if
you,
if
you
see
with
this,
we
you
the
creation
workflow-
is
accurate
on
coming
this
way,
the
talent
that
creates
pausing
at
end
of
master,
the
sync
manager
copied
up
pass
back
from
ten
remastered
super
master,
but
it
also
has
a
tenant
prefix,
your
con
name,
all
the
label
and
key
very
eccentric
award,
any
conflict
in
a
single
master
and
the
actual
post.
D
Scheduler
scheduling
is
done
by
the
unified
scheduler
rounds
in
a
super
master,
the
suit
the
universe,
Scalia
choose
a
node,
but
a
party
in
a
single
master.
The
computer
creates
pause
in
sticky
notes
and
the
sync
manager
creates
a
pod
another,
particularly
in
a
tenant
master
using
as
text
nodes
of
an
agent
rank
I
mean
for
instead
of
the
cookie
scene
for
then
updates
across
days
back
to
ten
master.
So,
overall
from
the
tenant
perspective,
the
pod
is
actually
runs
in
a
node
whose
cool
it
is
view
as
a
reagent.
D
So
this
is
the
the
the
general
idea
of
how
the
you
know:
their
tenant
master
in
degrees,
the
super
master
for
the
product,
lifecycle
management,
III,
didn't
a
fully
attached.
I
didn't
fully
describe
all
the
other
workflows
for
the
pod
lifecycle
management.
Unlike
addition
like
how
the
how
the
how
we
you
know
a
forward
or
the
API
calls
to
public
through
the
V
agent,
so
this
part
of
work,
I
I,
have
a
plan
to
you
know
feel
the
rest
of
in
this
dark.
Muddy
people
actually
indeed
interested
I,
have
slides
for
this.
D
The
problem
of
the
current
and
the
service
model
is
a
it
is
designed
based
on
the
you
know,
the
network
model
and
that
the
kubernetes
was
originally
designed,
basically
is
a
flat
network
model
means
every
pausing
that,
on
the
cross,
is
supposed
to
talk
to
any
other
parts
in
extant
cluster
and
all
the
agents
on
the
node
can
communicate
with
all
the
past
agents,
including
including
other,
not
in
the
couplet
or
the
order,
CICS
plugins,
but
this
model
is
largely
simplified
order.
You
know,
design
after
networking
infrastructure
for
the
in
the
coolant
is
budding.
D
It
actually
doesn't
work
very
well
for
the
multi-tenancy,
especially
in
harmony,
sentencing
cases.
The
problem.
The
problem
is,
you
know,
either
in
the
multi
currency
cases.
The
past
should
supposed
to
owning
talk
to
the
parties
in
the
same
Bulow's
to
the
same
tenant.
They
shouldn't
have
any
other
ways:
I
mean
indirect
ways
or
implicit
ways
to
talk
for
other
parts
to
that
below
the
other
tenant.
So
and
in
addition,
we
have
especially
in
some
of
the
use
cases
in
Alibaba.
D
So
so
so
so
this
comically
in
higher
you
know
design
for
the
the
node
in
an
old
site
for
the
EPC
configuration
in
the
muhammadi
tennessee
scenario.
So
the
biggest
form
of
the
the
web
sees
things.
Is
you
know
if,
once
people
create
a
vbc
for
the
past,
you
know
the
coverlet
doesn't
have
a
you
know,
direct
connection
to
the
to
the
past
narrower
anymore.
These
are
any
any
additional
configuration
and
the
the
current
a
service
model
in
helsing
based
on
the
so
the
service.
D
So
once,
if
you
you,
you
you
you,
if
you
use
vp
c
and
in
some
cases
in
some
cases,
seeing
I
if
seeing
a
directive
hogging
the
network
interface
to
the
partner
network
namespace,
the
host
network
IP
table
may
not
even
go
through
for
the
network
traffic,
so
the
taking
hire
you
know,
service
model
may
not
work.
So
this
is
a
kind
of
a
harder
problem,
because
because
network
configuration
is
so
flexible
and
people
can
create,
you
know
networking
their
own
way,
you
to
predict
producing
on
isolation
connectivity
central.
So
there
is
no.
D
I
don't
think
there
is
a
one
very
standard
way
to
solve
this
problem.
So
our
way
of
thinking
is,
we
are
trying
to.
You
know,
propose
another
layer
of
the
obstruction
we
Co
eight
to
ten
in
the
network
name
space.
The
idea
is
we
try
to
group
all
the
passed
below
student.
You
know
same
same
same
tenant
and
for
the
natural
connection,
we'd
connect
them
all
of
them
not
directly
to
the
host
breach.
D
Instead,
we
cracked
everything
in
our
new
neighbor
namespace
and
the
this
now
that
when
the
namespaces
is
credit
for
each
individual
tenant,
as
you
see
this
figure
and
this
figure
Chu,
so
the
idea
is
the
other
parts
they
have
a
weenie
spell
connect
to
the
narrow,
namespace
and
then
never
enemy
space
have
another
mini
spear.
To
connect
you,
the
house,
breach
on
the
the
good
thing
of
this
are
the
hierarchical
you
know.
D
Network
namespaces
is
if
we
can
have
another
solution
to
work
on
the
the
service
problem,
because
we
in
each
in
animal
namespace
order
traffic
traffic
needs
to
go
through
the
IP
tables.
So
instead,
so
we
need
to
slightly
we
as
long
as
we
slightly
change
the
could
be
proxy
to
allow
them
to
update
a
particular
IP
table
close
to
or
narrow
a
namespace,
so
this
problem
can
work
wrong.
D
Basically,
for
so
the
Kuby
proxy
needs
to
have
a
way
to
identify,
okay,
that
the
tenant
of
this,
the
certain
honor
of
the
service,
and
if
they
know
the
tenant
owner,
they
can
find
their
narrow
namespace
that
it
belong
to
the
tenant.
Then
they,
instead
of
directly
manipulated
their
IP
tables
in
the
house,
narrow
namespace.
They
came
anybody
that
the
net
one
a
dive,
cable
in
the
tenant,
a
network
namespace.
So
in
this
way
the
IP
table
the
their
IV
table
based
service
from
your
still
work.
So
that's
it.
That
is
the
wrong
way,
I'm.
D
Basically,
in
a
nutshell,
with
what
we
want
to
do
is
we
want
to
eat.
We
proposed
you
introduced
another
layer
or
another,
he
to
group
or
the
tenant
attitude
to
group,
all
the
parts
below
to
one
tenant
and
they
are
named
in
space
into
one
ten
and
then
when
them
space
and
we
proposed
you
have
to
East
tend
to
the
Kobe
proxy.
You
have
a
way
to
specify
the
net
or
namespace
whose
IP
level
is
going
to
be
updated.
So
this
is
the
idea.
Oh,
this
is
just
one
very
high
level
concept.
D
E
I
just
joined
some
I
might
have
missed,
but
today
there
is
only
one
network,
namespace
or
so
says
one
network
name,
space
per
pod
right
and
for
each
part
whether
they
belong
to
different
tenants
or
not
on
a
single
host.
They
all
they
still
have
different
network
namespaces.
Is
that
correct
or
not
each.
D
Pod
has
his
own
net
1mm
space,
but
once
he
is
mean
to
you,
but
the
thing
I
need
to
set
up
the
network
in
order
to
allow
the
inbound
outbound
traffic.
The
way
they
set
is
they
connect
each
phone,
net1
namespace
cue,
the
host
reach,
which
is
a
typical
hierarchy.
So
right
there
is
only
one
or
bond
you
know
hub,
which
is
a
house
breeze
which
is
host
breeze,
then
the
what
so.
This
is
a
default
configuration,
so
that
could
be
prongs
actually
leverages.
D
E
Sorry,
I'm
not
following
so
I
mean,
what's
the
difference
between
what
you're
suggesting
and
what
is
currently
there.
If
we
already
have
a
separate,
Network
namespace
per
pod,
then
how
are
we
or
are
you
suggesting
that
for
all
parts
that
belong
to
one
tenant,
they
should
all
belong
to
the
same
network
name
space
rather
than
each
one
having
its
own
network
namespace?
So
each
path.
E
D
Can
you
see
my
arrow
got
it
yeah
yeah?
If
you
look
at
this,
why
so?
Yes,
this
is
a
narrow,
namespace,
the
meanest
one
billows
to
the
partner,
never
namespace!
That
is
okay,
but
if
the
party
needs
to
send
traffic
out
outside
the
note
they
have
to
go
through
to
the
note:
isn't
it.
This
is
the
ether
zero
right
wheeze
on
my
proposal.
What
what
he
does
is
this
mini
spell
will
be
directly
connect
to
the
host
breach.
D
So
this
is
the
default
default
hierarchy
right,
so
the
partner
one
is
placed
directly
connect
to
the
host
bridge
using
the
link
in
his
pal.
So
every
path
goes
through.
The
single
host
agreed
to
talk
with
the
ethernet
in
the
host.
This
is
a
different
configuration
so
why
we
have
this
default
conversation
because
now
because
every
traffic
you
need
to
go
to
go
through
this
mini
spear
goes
to
breach
and
every
traffic
need
to
go
through
the
IV
table
for
the
traffic
routing,
then
the
KU
epoxy
they
are
owning
change.
D
D
F
D
F
It
means
that
you
have
to
modify
the
sandbox
code
of
the
cube
later
right.
The
the
couplet
is
basically
currently
it's
the
Linux
namespace,
and
then
it
will
bind
that
sandbox.
So
when
you
run
your
doctor
or
something
if
you
create
a
sandbox
container
and
that
container
holds
the
sandbox
in
this
case,
what
you're
doing
is
like
you
punching
all
the
tenant
port
in
your
separate
namespace.
So
it
means
that
you
will
be
modifying
some
of
the
kubernetes
core
components
right.
F
F
Okay,
so
in
the
CNI
you
want
to
change
so,
okay,
let
me
get
it
like
this,
so
in
this
case
the
sandbox
creation
right
that
is
done
by
the
tool
it's
in
with
docker
shim
right.
That
is
the
one
creating
the
sandbox
right.
The
couplet
is
basically
the
scene
is
basically
just
doing
the
network
coming
with
that
sandbox,
which
is
provided
by
the
docker
shim,
so
I'm
just
trying
to
understand
how
you
can
do
it
only
with
CNI.
F
F
D
We
do
have
some,
you
know
talk
if
you
look
at
a
dog.
We
have
some
discussion
about
what
need
to
be
change.
Yeah.
Okay,
let's
see
if
anybody
else
has
other
questions
on
there.
I
can
go
to
that
particular
issue.
You
see
if
we
need
a
coordination
or
cooperation
for
community.
What
needs
to
be
changed
but
at
least
is
a
cookie
proxy
in
our
proposal
needs
to
be
tenant.
Oh
well,
so
quite
so,.
C
C
Cube
DNS
or
whether
it's
logging
or
so
on
that
that
need
to
be
then
made
a
tenant
of
air
I.
Think
the
basic
isolation
can
be
done
purely
in
the
CNI
plugin,
but
probably
needs
a
more
detailed
study,
which
you
may
have
already
done,
perhaps
of
other
sort
of
components
that
need
to
to
be
made
tenant
of
it
and
I
think
you
were
just
saying
that
maybe
queue
proxy
at
a
minimum
would
need
to
be
tenant.
Aware.
Yes,.
D
D
Indeed,
actually,
in
order
to
make
service
work,
so
Kuby
pox
is
just
one
way,
but
but
there
were
some
other
functionality,
I.
Think
if
you
look
at
these
comments,
you
know
Sony
lie
actually
bring
out
a
very
good
question,
because
the
is
he's
talking
about
the
Reno
health
check.
So
the
health
check
is
a
function
at
KU,
but
it
actually
provide
it
needs
to
have
a
direct
connection
to
the
to
tutor
to
depart
for
for
this
purpose.
But
there
is
a
few
problem
in
this
approach
because
tenant
you
know
we
tenant,
we
see
network.
D
Actually,
the
IP
can
overlap
is
actually
possible
now
making
the
competitive,
injecting
the
individual
pass
through
IP
difficult.
Even
you
provide
the
network
connection.
So
one
will
what
we
think
about
is
we
have
to
change
slightly
change
the
cognate
to
once
they
do
network
like
leaven
is
flowing,
they
have
to
end
her,
the
hot
namespace
and
and
do
the
Curie.
So
this
is
the
one
with
that
cool.
It
has
to
be
tenant
Noel.
So
this
is.
This
is
just
one
example.
We
probably
have
some
other
examples
about
how
Cabrini's
be
change.
D
I
didn't
list
all
of
them,
but
at
least
we
already
thought
about
at
least
a
few
places.
Companies
needs
to
be
change
in
order
to
make
sure
all
the
service
account
service
and
the
lives
or
sub
functionality.
That
acrylic
provides
that
really
Klaus
event
connection
to
the
past
work,
but
the
bottom
line
is
and
Rose
changer
is
I.
Think
once
people
buying
the
idea
we
can
make
those
changes.
G
H
I
had
a
question
along
those
lines
you
know
with
this:
do
it
actually
alters
IP
tables?
So
in
this
situation,
if
one
of
the
tenants
deploy
something
like
sto,
does
your
modify
DQ
proxy
pick
up
changes
to
IP
tables,
or
does
it
only
push
them
out
or
I?
Guess
how
do
you
handle
that
and
how
do
you
prevent
conflicts?
Well,.
H
Yeah
so
IP
tables
from
the
per
node
to
the
actual,
underlying
OS
or
for
pertinent
to
the
inner
I
know
s.
Unless
I'm
are
you
duplicating
I,
guess
I
I'm
keeping
them
this
diagram
doesn't
really
I?
Guess
they
like
you,
might
have
how
these
IP
tables
are
working
with
in
in
terms
of
the
actual
underlying
OS.
So.
D
H
D
Right
the
idea
is
with
this,
which
is
my
proposal.
The
KU
approximate
change,
the
IP
table
data
pillows
to
the
tenant
and
narrow
namespace,
so
different
Tanana
have
even
2010
annette,
narrow
namespace,
which
means
they
are
cubed,
is
actually
manipulating
different
IP
tables
for
different
tenant.
Okay,.
D
Every
nerrvana
namespace,
so
the
idea
is,
you
has
to
know
where
to
go.
That's
a
key,
because
currently
you
look
at
the
API.
The
cuvee
proxy
doesn't
have
a
way
to
provide
the
tenant
information.
So
that's
the
problem.
So
that's
that's
actually,
motivation
that
we
are
in
the
modification
that
we
actually
suggest.
I
G
Well,
the
idea
would
be
to
add
your
own
cool
proxy
right,
so
you
don't
have
to
change
the
standard
cool
proxy
like,
for
example,
opening
control
deployed
your
song
cool
proxy,
so
cool
parks
is
just
a
plugin.
So
in
a
way
you
don't
have
to
modify
the
standard
cut
proxy
with
whatever
skin
are
you're
doing
you,
you
develop
the
coop
proxy
as
part
of
that
CNR
I.
C
C
So
really
the
zyre
original
desire
seems
to
be
and
correct
me
if
I'm
wrong
PHA
is
to
try
to
keep
as
much
changes
limited
to
just
the
CNI
plug-in
as
possible,
but
it
definitely
does
appear
like
we'll
have
to
evaluate
every
other
component
as
well.
But
it's
more
in
limited
to
the
scene,
I
plug
in
the
the
less
disruptive
it
would
be
yeah.
D
Yeah
plug-in
selves
cannot
solve
a
problem
for
sure,
especially
in
a
service
side,
so
I
mean
I
mean
you
have
some
other
ways
to
work
on
this
problem
because
you
can
choose
some
other
types
of
service
like
don't
use
IP
table
at
all,
so
the
entire
solution.
What
I'm
talking
about
is
try
to
solve
the
problem
which
you
go
using
IP
table
and.
C
C
B
This
is
down
from
so
I
think
we
can
start
with
Tonya
and
we
can
try
our
ideas
with
the
CN
phin,
which
I
think.
The
second
time
that
we
can
do
is
to
clearly
define
the
model
for
attendance
so
that
we
can
try
to
push
the
afternoon
to
make
more
component
to
be
an
awareness
aware,
so
actually
a
frame
engine,
then
okay
cool.
B
If
you
see,
but
if
you
I
don't
know,
did
you
know
it's
fallen,
oh,
like
karate,
so
there
many
other
areas
that
we
actually
want
improve
for
the
that
working
out
for
kubernetes.
So,
although
this
problem
is
actually
dicovered
from
the
virtual
hauser
approach
that
we
mentioned,
this
is
a
separate
problem
of
the
networking
isolation.
So
I
think
the
way
to
treated
for
is.
We
can
work
together
on
defining
a
networking.
B
And
we
try
to
see
what
other
columns
that
we
need
to
improve
more
than
Delaware.
We
already
talked
with
the
collect,
Korean
team
and
also
the
networking
team,
so
I
think
fair.
Fidesz
is
also
like
that
asked
to
define
the
earth
in
the
model,
and
then
we
go
back
to
them
and
try
to
push
some
10
years
to
livestream.
G
I
think
I
might
I
didn't
make
my
point
clear
before
you
know
so
there
are
CNI
plugins
which
for
replace
cool
proxy.
They
have
their
own
code
proxy.
So
there's
definitely
cases
where
cool
proxies
being
replaced
with
specialized
cool
proxy,
so
I
think
it's
it's
sort
of.
It
doesn't
make
sense
to
talk
about
calico
and
changing
the
standard
cool
proxy
to
handled
multi-tenancy
I.
Think,
it's
perfectly
fine
to
say
we
have
a
multi-tenant
version
of
calico,
for
example,
that
comes
with
you,
some
cool
proxy.
G
G
C
I
believe
that
is,
that
is
what
is
I
think
we
want
to
say
here
that
you
can't
take
any
any
old
CNI
and
expect
it
to
work.
It
would
work
better
with
CNI,
that
is
more
tech
a
little
bit,
but
in
as
much
change
is
limited
to
the
CNI
would
be
better
rather
than
spreading.
The
changes
through
other
cuban
areas,
components
sure.
B
I
think
for
the
proxy.
We
have
our
old
one,
because
it's
on
top
of
the
clog,
buoy
and
different
sea
ice
and
have
their
own
like
a
kamikaze,
but
for
other
problems
always
you
need
to
include
net
I
think
if
not
that
hard
to
put
them
to
change
stuff.
We
have
a
clear
item
of
tendency
or
networking
story
for
them.
A
K
K
One
of
my
clients
is
the
administrative
offices
of
the
US
courts
and
they
landed
on
open
shift
for
their
kubernetes
environment.
The
administrative
offices
are
setting
up
several
clusters
to
be
used
by
courts
for
deploying
their
court
specific
applications,
as
well
as
some
centralized
national
apps,
as
they
call
them.
They
want
hard
multi-tenancy
for
isolation
and
security,
but
also
from
a
finance
perspective.
K
The
courts
have
their
own
budgets
and
they
need
to
therefore
pay
for
the
the
resources
that
they
want
to
consume
inside
of
the
centrally
managed
environment,
but
they
don't
want
to
have
to
stand
up
there,
their
environment
themselves,
so
that's
the
kind
of
overview
of
the
environment
so
for
our
particular
multi
tenant
solution.
We
were
looking
at
a
explicit
approval
process.
Courts
can't
run
on
openshift
without
getting
approval
from
higher-ups,
as
well
as
finances
and
getting
those
those
pipes
together,
getting
approval
to
run
in
some
clusters
and
does
not
gave
you
permission
to
run
everywhere.
K
So
we
need
kind
of
differences
between
times
since
admins
don't
want
to
be
doing
things,
so
we
want
tenants
to
be
able
to
provision
their
own
projects.
We
found
that
project
namespace
provides
a
nice
isolation
for
your
application
or
for
your
cords,
its
application
for
your
one
court.
So
we
wanted
to
keep
that
model
rather
than
having
single
projects
and
namespaces
for
for
each
tenant.
We
want
to
integrate
nicely
with
LDAP
or
SSO
or
sam'l
or,
however,
you
guys
want
to
do
often
and
Ozzie.
K
This
is
like
tenant,
specific
project
templates.
It's
not
explicitly
built
that
way.
It's
kind
of
ad
hoc,
as
we've
added
things,
but
essentially
each
tenant,
is
allowed
to
create
some
custom
objects
that
get
inserted
into
each
project
as
its
as
its
created
and
the
ones
that
we've
done
for
now.
Our
network
permissions
and
default
role
bindings,
so
that
when
projects
are
provisions,
the
developer
teams
inside
of
each
of
those
courts,
would
have
access
by
default.
K
So
the
the
high-level
overview
is
that
we
have
a
controller
that
does
all
the
work
and
actually
manages
these
tenants
across
multiple
clusters.
We
have
an
API
that
essentially
is
allowing
us
to
control
our
back
on
properties
inside
of
our
tenants.
So
we
want
tenants
to
be
able
to
modify
the
project
list
of
what
projects
should
be
created,
but
we
don't
want
them
to
be
able
to
modify
the
cluster
resource
quotas
that
are
defined
for
those
those
tenants.
K
So
we
have
a
specialized
our
back
model
that
gets
implemented
in
an
API
call,
so
that
people
can
still
you
see,
allies
and
the
UI
that
tenant
admins
love
will
be
there.
There's
a
swagger
psyche
to
allow
the
to
the
hook
together
and
I'll,
show
the
UI
and
zoom
some
objects,
the
the
tenant
controller
we
built.
Actually
is
could
be
builder,
but
it's
very
open
shift
specific
and
what
I
mean
by
that
is
that
we
we
specifically
create
projects,
not
namespaces.
K
We've
leveraged
cluster
resource
quotas
for
managing
resources
on
individual
clusters,
so
it
does
not
directly
translate
to
something
that
would
work
on
Cooper
Nettie's,
vanilla
by
default,
but
I
would
like
that
flexibility
going
forward
if
this
is
a
value
to
other
other
groups,
as
I
mentioned,
the
API
allows
for
these
specialized
are
back
rules.
So
when
a
change
is
implemented
are
requested
through
the
API,
it
looks
at
who's
making
that
request
and
then
determines
based
on
some
business
logic,
whether
that
person
is
allowed
to
modify
that
property
of
that
tenant.
K
It
also
works
with
the
get
so
that
you
can
only
see
you
know
your
tenant
or
tenants
that
you
are
an
admin
of
and
you're
allowed
to,
delegate
permissions
down
to
the
cluster
level
so
that
some
people
can
provision
projects
on
the
development
clusters,
but
not
on
the
production
cluster.
So
you
can
have
that
isolation,
their
UI,
everyone
loves
the
UI.
K
So
the
high-level
overview
of
the
tenant
object
is
that
there's
an
overarching
admin
group
that
has
ability
to
modify
the
the
tenant
specific
capabilities.
Here
we
manage
multiple
clusters,
so
we
have
a
list
of
clusters
that
that
tenant
is
approved
for
it.
This
cluster
object
is
referenced
as
a
separate
CRD
that
manages
the
connections
there.
K
So
the
base
age
admin
portal
is
the
what
a
user
would
log
into,
and
this
is
the
self
provisioning
I
see
special
tabs
because
I
miss
admin
here,
but
a
tenant
would
be
able
to
come
on
here
and
see
what
projects
they
have
and
in
what
clusters
see
their
cluster
resource
usages
across
those
clusters
to
see
whether
they're
getting
close
as
well
as
provide
links
to
the
open
shift
console
for
where
that
project
is
on
that
cluster,
so
provides
a
centralized
location
where
a
tenant
can
view
what
what
they
have
available.
They
can
also.
K
In
terms
of
monitoring
we're
using
Prometheus
and
Griffin,
and
we
hooked
that
up
so
that
we
can
see
which
tenants
are
using
a
high
percentage
of
their
resources,
so
we
can
see
that
there
are
some
tenants.
This
J
application
tenant
is
using
all
of
their
cluster
storage
and
they've
already
reached
out
about
asking
for
more
and
we're
working
through
the
finances
and
the
approval
process
there.
K
But
this
gives
us
cluster
admins
the
ability
to
get
a
quick
overview
of
who's
close
to
asking
for
more
resources
and
what
resources
those
would
be
for
a
particular
tenant
they
might
and
I
haven't,
figured
out
exactly
how
to
do
this
thing
or
fun.
I
would
like
this
page
to
only
be
limited
to
the
tenant,
so
that
only
someone
in
SDS,
OAP
B
would
be
able
to
view
this
particular
dashboard,
but
that
particular
parameter,
but
that's
not
exactly
available.
So
there's
a
whole
separate
conversation
and
project
problem.
K
They
can
get
an
overview
of
what
they're
using
in
different
clusters
and
then,
if
they
maybe
have
a
question
on
well.
How
much
is
this
Jenkins
project
using
they
can
see
that's
using
four
of
their
15
CPUs
and
get
a
little
more
single
pane
view
on
where
those
resources
are
being
consumed,
and
then,
from
our
perspective,
we
have
an
overview
on.
You
know
this
cluster
has
this
many
CPUs
and
how
many
we
allocated
and
how
many
being
utilized
so
that
we
can
get
a
better
idea
about
what
our
utilization
should
be.
K
K
C
Tom,
this
is
great.
This
actually
is
very,
very
much
aligned
with
the
thoughts
that
we've
had
for
the
model.
B
of
those
four
options
that
we
are
tracking
two
quick
questions
is:
is
desert
tenant
map
to
an
open
chef
project
in
each
cluster,
one
one-to-one
mapping
between
tenant
and
openshift
project,
no.
K
C
J
A
related
question:
there
I
am
Yoshi
from
Google
on
we're
debating
about
the
sort
of
hierarchical
the
need
of
the
hierarchical
namespace
is.
Did
you
actually
choose
this
model
of
tenant
at
the
above?
The
namespace,
because
there
is
a
demand
of
people
wanted
to
create
namespace
as
they
won
once
they
get
the
tenant.
Is
that
the
right
interception
that.
K
Was
the
original
mission
we
had
several
people
who
like
to
have
lots
of
projects
because
they
feel
like
it's
their
own
little
sandbox
and
especially
an
open
ship
where
you
know
you
do
have
that
network
isolation
for
project
kind
of
give
your
developers
their
own
little
sandbox
in
the
project.
Without
you
know
having
this
big
project
that
people
then
worry
about
naming
conflicts
as
they're
testing
the
same
map
and
different
grade
stages.
J
F
Hi,
this
is
growth
from
Intel.
Actually,
we
are
working
on
the
sangeun
you
shin
from
google's
tenant
controller,
so
we
are
looking
to
have
a
tenant
manager
on
one
app
and
to
have
some
kind
of
a
tenant
resource
Kota.
So
we
took
the
proposal
and
we
were
working
on
that.
It's
I
just
put
the
link
here.
Actually,
so
it's
whenever
saying
you
were
demo
I.
It
was
same.
What
we
are
proposing
actually
in
the
own
app
actually
is
basically
to
have
a
tenant
quota,
which
is
one
layer
above
the
Cabana.
I
K
Now
have
cluster
quotas
that
sysadmin
men
needs
to
manage
individually.
It'd
be
great
if
we
had
a
tenant
quota
safely,
all
development
environments,
both
east
and
west,
and
then
there's
that
single
place.
That
kind
of
lets
you
move
your
resources
between
the
environments
without
having
to
update
the
CR
cues
of
each
individual
cluster
and
I.
Think
that
would
be
a
really
neat
feature
value
for
that.
Okay,.
A
K
C
K
B
I
see
the
tendency
but
I
think
actually
a
mole
of
B
and
we'll
see
under
the
hood.
They
are
actually
the
themes
in
the
mostly
actually
just
add
a
real
layer
on
top
of
malobi.
So
actually,
if
you
can
share
me
with
your
slice,
I
can
make
that
clear,
because
I
think
will
be
a
mostly
in
centrally
bouncing
thing
will
say:
just
add
another
layer
on
top
of
model
P,
because
under
the
hood,
they're
thinking
so
I
just
want,
and
my
own
opinion
that
yeah.