►
Description
Want to view more sessions and keep the conversations going? Join us for KubeCon + CloudNativeCon North America in Seattle, December 11 - 13, 2018 (http://bit.ly/KCCNCNA18) or in Shanghai, November 14-15 (http://bit.ly/kccncchina18).
Multi-Tenancy WG Deep Dive – David Oppenheimer, Google (Intermediate Skill Level)
"About David
David Oppenheimer is a software engineer working on Kubernetes and GKE at Google. He is co-lead of the newly-formed Kubernetes multi-tenancy working group, and was previously co-lead of the Kubernetes scheduling SIG. He has been working on Kubernetes since 2014, and prior to that worked on Google’s Borg and Omega cluster management systems."
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
Thanks
everyone
for
coming
this
is
the
kubernetes
multi-tenancy
working
group,
deep
dive
and
I'm
David
Oppenheimer
I'm,
a
software
engineer
at
Google
and
unfortunately,
my
co-lead
of
the
working
group,
jester's
l
was
not
able
to
come
to
Cuba
con,
otherwise
she'd
be
here.
Coley
ting
this
session
with
me,
so
we
kind
of
have
two
parts
of
the
session
planned.
One
is
a
few
folks
had
presentations
that
they
wanted
to
do
a
few
slides
about
ideas.
A
They've
been
thinking
around
about
around
multi-tenancy
and
policies
and
things
in
that
area,
and
then
also
we
had
some
collected
some
some
some
brainstorming
ideas
and
a
doc
that
people
were
interested
in
just
having
freeform
discussion
about,
but
we
don't
have
to
stick
to
what's
there.
We
can
also
talk
about
whatever
folks
in
the
audience
are
interested.
In
talking
about
one
of
the
things
that
went
pretty
well
last
cube
con.
We
did
a
multi-tenancy
working
group,
deep
dive.
A
That
was
actually
the
first
deep
dive
session
we
had
for
the
working
group
was
people
who
talked
about
how
they're,
using
multi-tenancy
and
kubernetes
today
and
like
what
kinds
of
features
are
missing
from
the
system
or
are
difficult
to
use
that
they'd
like
to
see,
and
so
that's
something
that
we
could
do
if
people
are
interested
in
talking
about
their
use
cases,
because
we
obviously
one
to
the
the
feature
set
in
kubernetes
to
be
driven
by
the
things.
People
actually
need,
not
just
something.
That's
you
know
created
in
some
kind
of
ivory
tower.
A
So
maybe
we
can
do
that
when
we,
when
we
get
to
the
discussion
section.
So
the
first
of
the
four
I
think
it's
four
presentations
is
something
that
I'll
be
presenting.
So
it's
an
idea.
We've
been
developing
at
Google
and
proposed
in
the
Pali
Tennessee
working
group
and
sig
off,
and
the
policy
working
group
called
kubernetes
security
profile,
and
actually
the
work
was
done
by
my
colleague
esuite,
who
who's
another
engineer
at
Google
in
the
kubernetes
and
container
engine
team.
A
A
And
so
the
idea
of
a
security
profile
is
to
create
a
small
menu
of
versioned
community
curated
policy
profiles
to
enable
turnkey
cluster
creation
with
the
desired
security
and
tenant
isolation.
So
that's
kind
of
a
lot
of
words
but
kind
of
conveys
what
the
idea
is
that
were
that
we're
getting
at,
and
by
everywhere
we
mean
like
these
should
run
the
same
way,
whether
you're
running
on
any
of
the
cloud
providers
or
on
Prem,
using
a
tool
like
cube,
atom
and
so
on.
A
A
You
know
strong
isolation
between
namespaces,
because
you're
running,
say
instances
of
a
single
tenant
application,
multiple
replicas
of
that
one
for
each
of
the
users
of
the
system
and
so
on,
and
and
then
also
this
would
work
across
upgrades
both
of
the
cluster
and
of
the
security
profiles
themselves.
And
so
you
can
see
there's
kind
of
two
pieces
to
the
names
of
these
profiles.
There's
the
the
name
that
describes
the
use
case
like
saz,
multi-tenancy
or
default,
and
then
also
a
version
associated
with
it.
A
A
And
so
the
idea
behind
the
security
profile
what's
going
on
under
the
covers,
is
that
it
kind
of
wraps
three
places
where
policies
get
specified
today.
So
one
is
what
we
call
bootstrapping
rules,
so
these
are,
for
example,
command
line
flags.
So
if
you
want
a
secure
cluster,
you
have
to
have
the
right
Flags
pass
to
the
API
server,
the
cubelet
and
so
on,
and
so
today
you
read
these
in
some
blog
posts
on
the
internet
or
occasionally
they'll
be
documentation,
saying
what
the
right
choices
are.
A
The
idea
here
is,
to
you
know,
kind
of
encode
these
in
in
kind
of
curated
and
and
well-accepted
profiles.
So
bootstrapping
rules
is
the
first
one.
Cluster
scope
policy
objects
like
pod
security
policy,
and
then
the
third
is
namespace.
Co
policy
objects
like
network
policy,
and
then
we
don't
dictate
how
these,
how
the
security
profile
gets
unwrapped
and
how
these
policies
that
it
embeds
get
applied
to
the
system.
The
idea
would
be
that
there
would
be
a
variety
of
what
we
were
calling
enforcers.
A
That
could
apply
them
to
the
system
and
it
depends
on
what
kind
of
policy
you're
talking
about
so,
for
example,
I
showed
that
you
know
the
command
line.
Flags
would
be
applied
by
say,
cube
Adam.
When
you
bring
up
the
cluster,
the
cluster
scope
policy
objects
could
be
injected
into
the
system
by
the
add-on
manager
or
just
through
invoking
acute
control
apply,
and
then
the
namespace
Co
policy
objects,
the
idea
being
that
you
could
automatically
populate
new
namespaces
with
particular
policy
objects
when
they
get
created.
A
That
could
be
done,
the
dirts
through
something
like
admission
control
and
then
the
little
cube
earn
a
DS
conformance
test
box.
That's
off
to
the
side
is
just
to
mention
that
we
want
these
profiles
to
be
integrated
into
kubernetes
conformance
so
that
when
you
bring
up
a
cluster
using
one
of
these
named
profiles,
it
behaves
the
same
everywhere,
no
matter
where
you're
running
the
security
profile,
and
so
we
have
a
demo
that
may
or
may
not
work.
A
Let's
see
yes,
so
I
showed
this
the
other
day
and
found
that
going
to
1.25
speed
is
slightly
better,
and
so
anyway,
the
demo
is
going
to
show.
Unfortunately,
the
text
of
the
top
left,
which
is
kind
of
the
most
important,
was
covered
up,
but
it
was
doing
that
queue
bottom
command
that
that
I,
that
I
did
before,
where
you're
passing
in
the
name
is
security
profile,
and
here
it's
bringing
up
the
cluster
and
we're
gonna,
add
a
node
to
the
cluster
and
install
Calico.
A
A
A
And
right,
and
so
the
security
profiles
are
represented
as
CRD,
so
that
that
kind
of
thing
that
box,
that
was
at
the
top
of
the
chart,
that
was
the
security
profile
itself,
that
wraps
the
other
policies,
is
represented
as
a
custom
resource.
You
can
see
here
that
we
installed
three
of
these
custom
resources.
A
Three,
you
know
objects
of
that
custom
resource
type
and
then
separately
we
can
select,
which
one
should
be
enabled
in
the
cluster
I.
Think
that's
the
command,
that's
going
to
be
next,
or
maybe
that
was
already
run,
but
anyway,
the
idea
is
that
there
is
a
separate
object,
called
a
selector
that
lets
you
choose
which
of
the
installed
policies,
security
profiles
rather
to
enable,
so
that
allows
you
to
sort
of
install
the
policies
without
running
them
and
then
activate
them
separately.
A
And
then
here
we
create
a
namespace,
and
you
know
normally
a
namespace
would
be
empty,
but
with
this
security
profile,
it
automatically
creates
a
network
policy
in
the
namespace,
and
so
when
we
do
get
name
space
over
here,
you
can
see
that
in
in
this
demo,
one
namespace
it
has
a
security
profile
and
actually
also
role
bindings
were
automatically
added
to
the
namespace
when
it
got
created
so
and
then
I
think.
The
last
piece
of
the
demo
is
just
showing
that
there's
also
a
pod
secure,
pod
security
policy,
which
is
a
cluster
level.
A
A
You
know
everybody,
everybody's
organization
is
gonna,
have
different
quotas,
and
so,
if
we
want
to
avoid
having
some
way
of
doing
like
templates
and
and
parameter
substitution,
we
kind
of
have
to
only
stick
to
policies
that
could
be.
You
know
very
generic
and
applicable
to
all
say
all
use
cases
of
a
particular
type.
So
it's
possible
that
we
could
extend
this
to
handle
something
like
resource
quota.
A
A
No,
so
there
is
I
didn't
show
that
enough
of
the
I
didn't
show
an
example
of
how
you
define
the
pod
security
policy,
because
I
was
supposed
to
just
be
five
minute
talk,
but
there
is
a
way
to
specify
exceptions,
and
we
haven't
really
decided
like
right
now.
You
can
list
which
namespaces
by
name
and
then
also
exceptions
by
name.
We've
also
talked
about
just
selecting
the
namespaces
by
label
selector,
but
yeah.
Definitely,
for
example,
the
cube
system
namespace.
You
probably
may
want
different
policies
on
than
other
namespaces,
for
example,
and
so
yeah.
A
B
The
repo
is
going
to
represent
the
configuration
the
cluster
state
and
you're
gonna
want
to
go
ahead
and
push
out
policies
via
this,
this
this
config,
so
how
we
do
API
design
and
what
we
kind
of
do
with,
especially
with
the
admission
control
really
matters.
So
I'll
just
give
a
quick
overview
of
kind
of
the
configures
code.
B
Workflow
right,
so
API
objects
are
represented
as
file
the
workflow
generally.
Is
you
edit
a
file
you
get?
It
reviewed
you
push
it
to
your
CI
DC
staging
repo
tests
run
on
it.
You
wait
for
them
to
pass
that
gets,
pushed
to
your
source
of
truth
repo
and
then
the
CI
CD
system
invokes
a
config
management
tool
and
then
the
config
management
tool
calls
an
API
to
reify
the
changes
to
the
platform
right
and
sometimes
CI
CD
systems
may
repeat
steps
to
allowing
slow
rollout
and
things
like
that.
B
So
that's
generally
the
workflow,
so
things
can
go
wrong
in
this
workflow
and
where
they
generally
go
wrong
and
then
to
the
worst
cases
at
the
last
stage
of
the
pipeline.
So
you
made
a
change,
you
got
it
reviewed,
you
pushed
it
up
there
and
it
went
all
the
way
through
and
in
that
last
step
which
I
bolded
in
the
previous
one.
It
fails,
and
so
you
end
up
with
an
error.
B
The
error
is
generally,
you
know
some
sort
of
kubernetes
error,
wrapped
in
let's
say
a
terraform
error
wrapped
in
you
know
jenkins
and
so
you're
kind
of
looking
at
that
and
wondering
what
to
do
and
then
you
find
out.
Oh,
you
know
what
some
admission
control
are
fired.
I
didn't
know
that
I
was
going
to
be
restricted
on
this,
and
so
therefore
I
have
to
go
back
and
change
my
thing
and
then
you
got
to
repeat
the
whole
process
again,
get
another
review
and
go
through
and
hope
right.
B
So
the
other
problem
that
happens
is
is
that
that
thing
has
changed,
that
the
change
that
you
made
is
in
the
actual
source
of
truth
repo
and
if
you're
not
diligent,
it
will
stay
there
forever,
and
so
now
you
have
differing
between
your
source
of
truth,
repo
and
what's
in
in
production.
Okay,
so
causes
of
this
is
that
your
config
management
tools,
validation,
do
not
match
the
API
validation,
and
so
this
is
where
the
you
know
and
then
pipelines
that
aim
to
test
and
validate
separate
separate.
B
These
are
from
the
actual
deployment
and
reduce
the
ability
to
react
quickly.
So
what
we
want
to
do
is
get
that
validation
to
happen
as
early
in
the
process
as
possible,
and
so,
if
we
can
do
that,
then
we
don't
have
to
go
through
these
steps
and
we
can
guarantee
that
we
can
have
a
better
guarantee
that
once
it
makes
it
through
the
CI
CD
pipeline,
it
will
apply.
B
So
what
is
our
general
approach
that
we
recommend
when
doing
and
thinking
about
ap
iodine's
and
especially
around
admission
control
and
how
we
want
to
control
and
constrain
our
our
api's?
So
so
we
want
to
make
the
API
validation
code
portable
right.
So
not
only
is
it
run
as
an
emission
controller
on
the
API
server,
but
it
also
can
run
as
a
pre-commit
hook
in
your
git
system
or
whatever
your
source.
Repo
system
is,
and
ideally
it's
the
same
code
so
when
they
rev
they
Rev
together.
B
Write
validation
code
needs
to
give
very
explicit
errors
about
what
went
wrong
and
where
to
go,
to
get
documentation
to
understand
what
the
constraints
are
so
you're
not
guessing.
We
want
to
reduce
the
guest,
because
the
the
make
a
change
and
apply
cycle
is
so
long
on
these
configures
code
pipelines.
You
want
to
reduce
that
as
as
much
as
possible,
and
then
you
know
again
where
we're
at
is
CIC
D
pipeline
is
there
to
ensure
the
commit.
So
what
you?
B
The
reason
why
you
have
a
CIC
D
pipeline
in
this
case
is
because
you
can
do
you
know,
get
pre,
submit
dash
n
and
ignore
pre-commit
hooks,
so
you
need
something
there
that
can
actually
run
those
and
guarantee
that
they
happen.
That's
what
your
CI
C
D
pipeline
is
doing
its
effectively
verifying
that
you
have
done
the
appropriate
setup
so
that
the
error
fails
in
the
in
the
in
the
pipeline
rather
than
at
commit
time.
So
that's,
basically
all
I
got
just
a
super
quick
overview
on
what
they
think
about.
Go
ahead.
B
This
is
like
apply
moving
to
server-side
yeah,
so
the
the
question
was
is
the
API
Machinery's
consider
is,
is
working
to
move
cube,
apply
into
the
server
side,
rather
than
it
currently
being
a
client
side?
And
how
does
this
play
with
that?
So,
unfortunately,
well
I
mean
we
should
we
should
find
out.
So,
at
the
end
of
the
day,
what
you
want
to
be
able
to
do
is
give
someone
the
permission
to
be
able
to
validate
something
without
the
ability
to
actually
run
the
command.
B
So
the
recommendation
I
would
make
is
to
is
to
have
a
lower,
privileged
command.
That
says,
hey
I
can
do
like
an
apply
dry
run
right,
it'll
run
that
dry,
run,
it'll,
run
it
against
all
the
the
validations
and
return
it,
and
then
I
can
put
that
in
like
in
my
pre-commit
hook
right.
So
that
would
be
the
recommendation
to
do
there
and
they
could
do
that
server-side,
but
it
would
introduce
a
new
set
of
roles
which
are
basically
these
lower
privilege,
role
that
let
you
do
dry
runs
so
good
question
all
right.
A
C
We
want
to
give
them
all
the
security
profiles,
which
basically
is
a
very
good
idea,
I
think
to
eat
up
our
lives.
We
want
to
provide
peace,
peace
for
them.
We
want
to
enable
certain
projects
to
have
access
to
certain
privileged
stuff
like
GPUs,
etc.
So
the
motivation
here
really
was
to
come
up
with
a
specialized
implementation
that
would
allow
us
to
do
so
in
a
very
common
way.
So
what
we
have
at
the
moment,
it's
a
bit
down
there
at
the
example
is
we
have
an
integration
with
git
lab
as
our
source
of
truth.
C
So
how
that
works?
Is
you
create
a
project
and
get
a
lab,
and
then
we
have
a
piece
of
code,
basically
just
a
service
that
then
creates
a
namespace
and
the
according
rules
in
the
cluster,
which
is
a
bit
of
a
fast-paced
solution
that
we
just
need
it,
because
we
needed
something
to
allow
for
all
those
students
to
have
a
secure
way
to
access
the
cluster
to
lock
into
the
cluster
and
then
have
all
the
rights
in
place.
C
So
just
to
give
you
some
numbers
here
that
the
currently
the
cluster
has
2000
namespaces
and
about
12,000
role
bindings.
So
that's
the
numbers
we
run
and
5,000
parts
or
6,000
parts
exactly
so.
What
we
faced
here
was
that
we
have
now
a
tight
integration
between
gitlab
as
our
source
of
truth
and
the
cluster,
so
get
a
lab
in
our
case
is
the
organization
specific
implementation
of
where
stuff
comes
from,
whereas
the
common
functionality
here
is
that
we
provide
access
to
namespaces
and
to
allow
people
to
do
things.
C
So
we
wanted
to
petition
these
concerns
and
that's
where
we
thought
a
data
Tanen
operator
would
actually
be
quite
nice
and
at
the
multi-tenancy
working
group
we
had
the
discussion
about
what
should
be
the
leave.
If
you
think
about
your
organization
structure
as
a
tree,
what
should
be
the
leaf
that
reaches
into
the
cluster,
and
how
far
should
we
take
the
tree
and
let
it
drip
into
the
cluster
and
our
approach,
or
our
suggestion
is
to
just
have
a
CR
D
that
represents
the
tenant
as
being
the
leaf
of
this
tree.
C
That
is
actually
in
the
cluster
and
the
rest
remains
into
your
organizational
structure.
So
the
idea
was
to
finally
be
able
to
make
an
opinionated
behavior
that
is
explicit
and
configurable
from
your
organizational
structure
into
the
cluster.
So
the
intended
mode
of
operation
is
quite
simple:
you
have
some
kind
of
external
system.
C
So
we
came
up
with
the
definition
of
a
tenant
in
our
case,
which
says
that
a
tenant
consists
of
a
set
of
namespaces
in
which
any
account
with
sufficient
permissions
may
create
Cuba
Nettie's
objects.
The
number
of
these
objects
and
Teresa
sumption
are
totaled
overall
namespaces
of
a
tenant,
because
it's
also
a
problem
we
have.
C
C
C
C
So
the
name
needs
to
be
only
one
time
there
and
a
list
of
namespaces
that
that
should
be
valid
for
and
then,
of
course,
at
least
all
the
things
that
cause
memory,
API
storage
volumes-
maybe
that
could
also
include
the
type
of
storage
that
the
tenant
may
use,
but
also,
of
course,
the
amount
of
storage.
It
may
use
reek,
respect
I'm
unit,
because
we
had
some
students
that
code
it
against
the
kubernetes
api
and
just
hammered
the
api
server
and
brought
it
down
or
I
essentially
at
CD.
C
So
we
also
might
want
to
restrict
here
on
appurtenant
base,
which
is
also
I
mean
you
can't
restrict
the
API
server,
but
not
on
a
pro
tenant
pace,
because
I
think
the
control
plan
is
not
really
a
multi-tenancy
aware
at
the
moment,
yeah
and
optionally.
It
may
contain
a
map
of
user
name
and
roles
to
be
able
to
map
back
and
forth
between
those
in
the
future.
C
So
we
have
a
little
proposal
in
here
that
is
also
editable,
I
think
or
at
least
comment
below,
and
we
would
really
like
to
have
your
thoughts
on
that
and
see
what
else
we
can
add
to
that
and
if
you
find
it
useful
or
not,
and
also
around
the
discussion
of,
is
that
tenancy?
Are
these
thing
really
what
we
need
or
do
we
need
further
levels
of
abstraction
in
the
cluster
itself
or
not?
We
had
some
discussions
with
Ray
on
that
on
on
the
zoom
talk
so
also
so,
yeah
I
think
that's
it
yeah.
C
C
D
C
Thinks
that
is
an
approach
that
we
think
the
idea
of
having
it's
actually
not
in
here.
We
see
this
as
a
problem
as
well,
because
we
of
course
have
different
sized
teams.
That's
basically
research
projects
that
have
their
grant
money
turned
into
into
computing
power,
and
they
have
different
amounts
of
computing
power
here,
of
course,
and
so
we
think
that
we
need
a
different
tool
for
that.
We
adapt
that
currently
Tamia's,
which
is
the
name
that
we
have
for
it.
We
have
some
outlines
on
that.
We
didn't
get
develop
it.
C
It
would
be
the
system
way
or
define
who
gets
how
much
resources
on
which
and
which
guarantees
to
those
resources
are
delivered,
and
you
can
also
share
resources
or
lease
resources
for
a
period
of
time.
So
we
see
these
two
systems
and
separate
it.
This
is
basically
to
enforce
the
stuff
and
to
connect
to
the
kubernetes
thing
here
and
then
the
other
thing
would
be
to
how
do
you
I
fill
these
resource
Kota
limits
to
to
adjust
that
for
different
team
sizes
requirements,
etc.
That's
the
idea.
A
E
We
recently
roll
out
our
clock
containing
instant
service
and
it's
you
under
preview,
but
we'd
like
to
share
ours
and
some
of
our
experience
and
some
issue.
We
are
facing
two.
So
the
first
is
how
we
use
that
current.
Currently,
the
clouding
container
instance,
a
little
bit
different
to
others,
is
that
we
also
expose
kubernetes
api.
So
basically,
here
we
just
roughly
block
all
this
route
scope
api
because
that's
kind
of
issue
under
multi-tenancy,
so
it's
too
limited
preview
and
only
available
in
china.
E
So,
sadly,
you
cannot
try
easily
so
first
off
voice
about
the
Teaneck
concept.
So
currently
we
don't
explicitly
define
a
tenant
with
just
maybe
two
namespace,
so
each
user,
our
tenant,
have
one
namespace
and
for
name
we
just
generate
by
the
platform.
Actually,
the
CCI
controller
and
actually
the
name
I.
Think
it's
not
very
big
issue.
Imagine
you
creating
a
character
on
multiplayer
online
game.
You
would
always
get
some
notification
that
now
this
name
is
taken
by
someone
else
right.
E
So
API
access
currently
all
like
cubelet
it's
it
will
directly
talk
to
the
true
banana
ceiba,
so
a
bar
for
and
the
user.
We
just
let
them
only
able
to
talk
to
the
a
packet
way.
So
we
can
view
the
pertinent
API
call
rate
limiting.
Also
our
Becca
are
enforced
that
their
level.
That
makes
easier
to
start
because
currently
we
don't
have
a
pertinent
rate
limiting
or
put
her
name
space
very
limiting
in
kubernetes
api,
maybe
a
silver
okay.
For
the
note,
that's
right:
okay,
okay,.
E
So
for
the
note
currently,
because
it's
a
shared
the
mounting
tenon
clusters,
though
we
also
are
manage
the
notes
and
user,
don't
have
any
access
to
the
machine
and
we
use
a
circular
container
as
a
runtime.
So
we
are
facing
some
detailed
issue
because,
for
example,
some
of
the
matrix
currently
not
they
cannot
be
easily
collected
for
the
network.
We
do
have
a
network
API
as
NC
Rd
and
we
currently
it's
a
one
network
for
namespace
and
the
underlying
container
network
work.
E
We
have
our
own
implementation,
it's
an
how
to
isolation,
which
is
very
similar
to
a
neutron
but
for
the
network
creation.
Currently,
we
don't
block
party
creation,
but
it
will
get
ready.
Finally,
for
storage,
actually,
here's
just
a
some
issue,
as
we
know
that
empty
dir
by
default.
We
can
indicate
the
limit,
but
but
the
behavior
is
that
if,
if
the
part
is
using
too
much
space
and
it
exceeds
the
limit,
kubernetes
will
evict
a
victor,
the
pod.
E
E
We
don't
want
to
evict
the
container
or
reboot
container
so,
but
there
are
do
some
issues
like
what
do
we
do
if,
if
the
couplet
needs
to
be
upgraded
or
sometimes
we
need
to
to
do
some
OS
upgrade,
the
node
may
have
to
be
shut
down,
but
so
we
we
have
not
yet
solved
that
yet.
But
it's
we
think,
that's
a
very
important
issue.
E
Another
one
is
the
root
scope,
API
access,
for
example,
CRD.
Currently
we
know
that,
like
on
the
cuban
s
engine
because
end-user
has
the
full
access
to
the
whole
cluster,
as
well
as
the
kubernetes
api,
so
they
can
easily
create
a
CR,
D
and
install
whatever
operator
they
want,
but
on
on
a
multi-tenant
to
analyse
cluster,
it's
because
CR
DS
root
scope.
So
currently,
our
solution
is
just
we
store
some
of
actually.
We
just
register
some
of
the
CR
d2
to
available
for
end
user,
but
for
the
operator
they
depends
on
and
the
user.
E
If
they
want
to
use,
they
can
run
operated
operator
on
in
their
namespace
and
use
the
CR
and
an
otherwise
the
PV
PV
API,
because
end
user.
Currently
we
for
computer
resources,
actually
we
just
charge
by
can
pie
container.
So
that's
fine.
It's
kind
of
resale
from
reselling
the
the
computing
resources,
but
the
storage
and
user
could
directly
could,
for
example,
create
volume
on
how
a
clown,
but
it's
it's
kind
of
weird
that
they
have.
They
cannot
manage
persistent
the
elements.
No,
they
can.
They
can
create
PVC.
But
that's
not
enough.
A
Thanks,
let's
give
Kevin
a
round
of
applause,
and
if
you
have
questions
for
him,
please
come
up
and
talk
afterwards
or
any
of
the
folks
who
talked
thanks
a
lot.
We
hope
you'll
join
us
at
the
working
group.
Bi-Weekly
meetings
they're
at
11
a.m.
on
Wednesdays
I
think
Pacific
time,
but
you
can
look
that
up
on
the
github
site.
All
right
thanks.
Everyone.