►
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Panel Discussion: Multi-Tenancy in Kubernetes: Current State and Future Roadmap - Sanjeev Rampal, Cisco; Ryan Bezdicek, Cray Inc.; Erica von Buelow, Red Hat; and Tasha Drew, VMware
Kubernetes doesn’t currently formally support the notion of multi-tenancy. However, many companies today are using Kubernetes to provide ad hoc multi-tenant solutions, to share kubernetes cluster resources between multiple independent teams. In this panel, we will have a discussion with a mix of experts on this topic including panelists from kubernetes users, open source community contributors, vendors and providers. The panel will discuss both soft and hard multi-tenancy models and will cover solutions that are available today (such as pod security policies and rbac) as well as new features being developed by the community Multi-tenancy working group and related work such as Kata containers. Importantly we will have an open discussion between panel members and audience on requirements and issues related to end to end aspects of multi-tenancy in real world deployments.
https://sched.co/MPcY
B
A
lot
of
the
panelists
are
also
working
in
the
upstream
Cuban
ideas,
multi-tenancy
working
group
and
a
lot
of
the
work
reflects
what's
happening
in
that
group.
But
really
this
me
if
we
would
really
welcome
a
lot
of
interaction.
Give
us
your
feedback,
your
thoughts,
your
requirements.
How
can
this
work
best?
Have
the
community
I'll
let
the
panel
introduce
themselves
go
ahead:
Russia
cool.
C
D
B
B
How
many
of
you
would
save?
You
would
say
that
you
have
a
good
feel
for
current
multi-tenancy
capabilities
in
cuban.
It
is,
would
you
say
you
you
understand
it
pretty?
Well,
okay,
so
maybe
about
half
how
many
of
you
feel
like
the
current
functionality
in
Cuba
notice
is
adequate
to
build
a
multi-tenant
solution
and
the
other
alternative
being
no.
We
need
to
have
more
work
in
Cuba
notice
to
additionally
support
multi-tenancy.
So
how
many
feel
that
current,
given
it
is
adequate?
B
Okay,
not
too
many
and
just
to
confirm,
do
the
rest
of
you
feel
like
more
work
is
needed
or
okay,
so
there's
a
strong
audience
preference
for
that,
that's
great!
So
what
we
felt
in
the
community
is
that
there's
a
lot
of
custom
solutions
that
combine
namespaces
are
back
Network
policy,
but
everyone's
solution
seems
to
be
different
and
officially
CNC.
If
doesn't
really
have
a
position
on
whether
kubernetes
does
actually
support,
multi-tenancy
or
not,
and
whether
I
can
take
my
application
from
one
kubernetes
cluster
from
vendor
X
and
run
it
on
vendor
wise
cuban.
B
It
is
cluster
and
I'll
still
get
the
same,
multi-tenant
behavior,
probably
not
so
maybe
our
panel.
We
can
start
with
what
you
feel
like
the
community
needs
to
do
through
the
working
group,
as
well
as
through
other
community
events,
to
further
the
state
of
multi-tenancy
so
Tasha.
Maybe
we
can
start
with
you
and
go
around
yeah.
C
Absolutely
so
I
think
the
best
thing
that
everyone
can
do
if
you
see
a
need
for
additional
multi-tenant
solutions
within
upstream
kubernetes,
is
participate
in
the
working
group.
We
have
a
bunch
of
documents,
all
of
which
you
can
check
out
on
our
repo,
which
is
in
the
kubernetes
cig,
slash
multi-tenancy
repo.
C
Basically,
if
you
go
to
docs,
we
just
have
a
links
page
that
has
all
of
our
Doc's
and
what
we've
been
doing
is
collecting
user
stories
and
use
cases
from
the
community
to
understand
how
people
want
to
use
multi-tenancy
and
what
their
use
cases
are,
and
so
that
is
super
open.
We
also
have
some
proof
of
concept
work
that
we're
working
on
that
we'll
probably
talk
about
more,
but
just
at
the
like.
D
But
yeah
I
mean
shelter,
meetings
right
and
we
need
help
and
the
other
thing
is
be
vocal.
So
clearly,
a
lot
of
people
want
more
with
multi-tenancy,
but
there's
a
lot
of
people
that
aren't
convinced
that
any
more
work
is
needed.
So
I
guess
vocalize
your
guys's
desires
for
multi-tenancy
and
kubernetes.
B
E
Reiterate,
definitely,
please
show
up
and
speak
up
there
within
the
working
group
and
across
kubernetes
I.
Think
one
thing:
what
really
we
are
all
very
hungry
for
is
real
use
cases
and
users
and
what
the
real
trade-offs
that
you
have
to
make.
Are
it's
hard
to
make
the
decisions
technically
of
what
is
worth
the
effort
not
without
getting
a
better
sense
of
what
that
will
actually
mean
to
real
users.
F
Yeah
so
I
think
coming
up
everyone
and
speak
about
it
will
be
able
for
to
make
a
decision
on.
What's
the
best
option,
there's
many
options
to
make
multi-tenancy
there's
also
still,
maybe
not.
Everyone
speaks
the
same
when
they
say
hurt,
multi-tenancy
and
so
multi-tenancy.
So
I
think
this
is
really
important
to
for
everyone
to
speak
the
same
language.
B
Okay,
thank
you
so
essentially,
the
takeaway
is
that
a
lot
of
people
have
semi
similar
ideas,
but
there
hasn't
been
a
standardization
effort.
Yet
maybe
the
working
group
can
be
a
common
forum
for
all
of
us
all
of
you
to
bring
your
thoughts
so
that
we
can
consolidate
that
into
a
commonly
accepted
multi-tenancy
architecture.
B
Well,
maybe
the
next
point
that
maybe
we
can
Gover
is
so
there's
this
perception
that
there's
soft
multi-tenancy
requirements
for
enterprises
and
hard
multi-tenancy
for
hosting
your
thoughts
on
whether
that's
a
that's
a
genuine
demarcation
and
and
do
we
want
to
proceed
on
both
directions
in
parallel
and
so
on.
Sure.
C
B
Thing
I'll
add,
is
that
if
you
go
to
the
working
group,
github
page,
there's
there's
a
bunch
of
documentation,
there's
also
some
prototypes
that
the
group
has
started
putting
together.
So
please
visit
that
and
see
whether
that
works
for
you,
please
feel
free
to
join
the
meetings
and
provide
your
requirements.
Input
comments,
criticism
so
that
we
can
collectively
work
on
moving
this
forward.
Yeah.
C
And
we
have
some
really
cool
potential
implementations
of
how
we
might
achieve
both
soft
and
hard
multi-tenancy.
All
of
our
videos
from
our
regular
meetings
are
online
and
a
YouTube
channel
which
you
can
check
out,
but
just
at
our
last
meeting
we
had
a
team
from
Alibaba
demonstrating
basically
a
nested
API
server
implementation
to
allow
them
to
have
a
greater
degree
of
control
to
their
users
and
there
looking
for
feedback
on
that
to
see
if
that
would
solve
a
lot
of
people's
multi-tenancy
use
cases.
C
B
F
So,
in
our
case,
we
decided
against
putting
like
har
multi-tenancy
or
having
a
cluster
for
each
one,
and
the
decision
is
made
in
order
to
be
able
to
have
a
single
control
plane
and
not
having
multiple
quartum
control.
Planes
allows
better
resource
usage,
so,
for
example,
we
can
serve
the
seeming
rest.
But
later
we
can
discuss
if
this
is
good
or
bad
okay.
B
And
you're
running
Kate's
PennDOT
cloud,
so
that's
an
interesting
initiative
for
anybody
that
wants
to
try
it
out
thanks
Erika,
so
you
work
with
Red
Hat.
They
you've
had
a
lot
of
deployments
already
with
some
level
of
open
chef
multi-tenancy.
Can
you
comment
what
you're
seeing
in
the
field
and
what
I'd
had
is
seen
sure.
E
Red
Hat,
with
Oakland
shift,
has
a
wide
range
of
support
for
multi
tenant
isolation.
Concepts
for
one
is
just
this
kind
of
security
and
controls
needed
for
the
enterprise
users
available
in
the
open
ship
container
platform
that
our
customers
run
on
various
clouds
in
bare
metal
situations,
then
we
ought.
They
open
shift.
Also
has
open
ship
dedicated
where
they
work.
We
manage
and
run
the
clusters
for
you.
So
that's
more
of
a
multi
cluster
solution.
E
On
top
of
that,
there's
OpenShift
online,
which
is
a
multi-tenant
salute
platform
that
we
offer
where
it's
kind
of
you
know
you
get
a
limited
number
of
namespaces
that
you
can
play
around
with.
So
in
that
sense,
OpenShift
online
is
really
our
place
where
we
explore
and
try
to
harden
as
much
as
possible
what
we
can
do
as
a
multi-tenant
platform
in
Cooper
net
with
kubernetes
any.
E
Talking
with
a
lot
of
some
of
the
customers
and
such
the
think,
when
they
get
there,
they'll
be
excited
to
have
those
problems,
but
so
many
other
technical
issues
are
the
biggest
concerns.
As
how
I'm
I
pv
6
is
the
first
one
that
most
people
bring
up.
I'll
give
them
a
little
bit
time
to
work
that
out
and
then
I'm
sure
they'll
have
a
lot
more
to
say
so.
B
The
is
it
fair
to
say
that
the
takeaway
is
that
what
you're
seeing
so
far
is
OpenShift
online.
Is
your
primary
multi
tenant
vehicle
at
the
point
at
the
moment,
although
you
support
it
in
multiple
offerings,
but
people
still
tend
to
use
a
virtualization
platform
under
open
shift
to
provide
the
tenancy
sort
of
multiple
clusters
built
on
OpenStack
tenants,
for
example,
is
that
fair
statement.
E
F
D
So
so
right,
so
we
have
supercomputers.
So
you
know
a
supercomputer
is
generally
like
an
entire
data
center,
that's
kind
of
seen
as
one
object,
and
so
in
order
to
kind
of
manage
the
entire
supercomputer.
We
have
a
kubernetes
cluster
running
in
front
of
it
on
you
know,
commodity
hardware,
that
is
handling
all
the
micro
services
to
you,
do
boot
up
in
the
OSCE's
and
firmware,
etc,
etc.
But
with
that
we
also
need
to
provide
access
to
the
supercomputer
with
it
and
part
and
partition
our
tenant,
the
the
supercomputer.
D
D
So
we
have
kind
of
a
concept
of
a
tenant
is,
is
both
allowing
the
the
users
the
end
users
to
deploy
their
own
workloads
and
services
within
our
kubernetes
cluster
to
to
interact
with
the
computer,
but
as
well
as
tenant
in
partitioning
the
actual
underlying
Hardware
to
to
make
that
happen
so
that
to
the
single
tenant
it
looks
like
they
have
their
own
supercomputer.
But
it's
really,
you
know
one
giant,
one.
That's
yard,
yeah.
B
I'm
sure
a
lot
of
folks
would
be
really
curious,
both
from
an
intellectual
perspective,
as
well
as
from
many
other
aspects
as
how
supercomputing,
cuban
artists
and
multi-tenancy
come
together
and
I'm
sure
we
would
love
to
hear
more
from
you
over
the
coming
months.
Tasha
from
a
woman's
perspective.
Do
you
have
any
any
color
you
can
provide
to
the
audience
on
what
you're,
seeing
with
VMware
customers
yeah.
C
Absolutely
VMware,
customers
are
obviously
tend
to
be
large
enterprises
and
the
way
that
they
interact
with
kubernetes
clusters
and
multi-tenancy
as
they
want
to
share,
and
this
is
a
really
common
use
case
across
many
companies.
They
want
to
share
kubernetes
clusters
between
departments
or
between
teams
and
so
they're,
just
trying
to
figure
out
like
how
can
I
restrict
access
to
this,
but
have
similar
tool
sets
and
not
have
to
have
hundreds
of
kubernetes
clusters
running
around
okay.
G
B
B
C
H
B
So
yeah,
so
the
project
is
still
early.
We
have
some
initial
information
on
profiles.
We
have
some
initial
POC
of
controller
that
enforces
tenancy.
So
let
me
kind
of
take
a
step
back
here,
and
one
of
the
things
we've
found
is
that
there
are
basically
four
architecture
approaches
to
doing
multi-tenancy
and
communities
approach.
Number
one
is
multiple
clusters
running
on
and
I
as
okay,
so
you
could
be
running
OpenStack
and
just
giving
multiple
clusters
and
folks,
that's
one
form
of
multi-tenancy
that
people
are
using.
Let's
call
that
solute
option
a
or
model
a
model.
B
B
is
a
single
kubernetes
cluster,
but
with
new
CR
DS
for
tenants,
tenants
templates,
namespace
templates,
and
that
is
what
we
are
prototyping,
so
that
what
you've
seen
in
the
repo
has
sort
of
the
early
prototype.
But
we
want
feedback
from
the
community
to
see
whether
we
should
push
forward
further
so
far,
the
feedback
we've
got
is
that,
yes,
we
would
like
to
see
that
model
develop
further.
A
third
model
that
we
have
received
from
the
community
is
communities
on
communities
with
multiple
clusters
running
on
top
of
a
single
master
communities,
cluster
and
each
tenant.
B
Getting
their
own
kubernetes
cluster,
but
not
running
on
top
of
ni,
as
rather
running
on
top
of
another
kubernetes
cluster,
so
more
cloud
native
solution
than
combining
with
ayahs.
That
is
a
proposal
from
Alibaba
that
we
only
have
an
initial
proposal
in
the
repo.
So
we
expect
to
see
more
over
the
coming
months
and
finally,
we
have
an
option
for
option
D.
We
call
it
where
we
would
require
a
change
to
core
kubernetes
to
add
the
concept
of
a
tenant
into
each
API
call
make
tenancy
a
first-class
citizen.
B
That
requires
a
lot
more
work
in
the
core
community
communities,
and
that
will
happen
only
if
the
community
expresses
strong
interest
that
that's
what
we
want.
So
we
are
bringing
this
early
so
that
we
want
you
all
to
work
together,
all
of
us
to
pick
one
or
more
of
these
options
to
develop
further.
So,
yes,
the
repo
still
has
very
early,
very
early
information.
Please
contact
the
team
on
slack.
We
can
work
with
you
further.
E
As
hard
as
we
can
make
it,
okay,
that
makes
sense
the
distinction
between
hard
and
soft.
The
online
is
our
best
attempt
to
do
a
hardened
freef.
Anyone
can
use
it
untrusted,
but
with
limits
since
its
new
technology,
yeah
I
think
the
line
between
hard
and
soft
is
very
fuzzy.
Very
fuzzy.
These
days,
where
even
VMs
have
some
level
of
leakage
that
we
worry
about,
but.
B
E
So
then,
there's
the
centralized
kind
of
versus
decentralized
or
delegated
API
models,
so
openshift
online
is
very
centralized
and
very
tightly
controlled
and
you
have
a
limited
number
of
things
and
you
know
unprivileged
pods
that
you're
allowed
to
run
versus
open,
shipped
dedicated
where
components
that
run
on
and
open
shift
cluster
yeah.
Thank
you
provide
provision,
a
full
cluster
for
each
tenant.
That
way,
you
could,
you
know,
configure
more
your
own
policies,
use
and
decide
what
you
want
to
run
it
and
have
your
own
control,
plane
and.
B
J
So
we
have
a
multi
master
multi
workers
deployment.
Basically,
we
have
a
different
business
units,
different
teams
right
now
they
are
to
publish
their
own
API
and
their
own
containers,
and
they
should
be
able
to
run
it
to
only
by
themselves
and
they
I
mean
others
should
not
see
others
and
should
not
delete
or
I
mean.
That
is
a
requirement.
What
we
have
so,
instead
of
running
multi
multiple
clusters
for
a
different
team,
we
want
to
run
one
cluster
for
everybody.
J
C
So
something
that
we're
working
on
right
now
is
defining
a
standard
multi-tenancy
profile
which
I
think
kind
of
gets
back
to
an
earlier
question.
So
if
you
look
at
sto
Docs,
for
example,
you
can
click
on
a
link
and
sto
Docs
that
will
tell
you
how
to
configure
is
do
for
a
multi
tenant,
so
multi
tenant,
configuration
and
kubernetes
doesn't
have
that
in
upstream.
So
everybody
is
configuring,
their
own
cluster
with
how
they
think
a
good
multi
tenant
solution
would
be.
C
So
what
we're
looking
to
do
is
take
all
of
those
examples
from
the
community
make
a
standard
profile
and
then
share
that
not
only
in
kubernetes
Doc's,
but
also
with
the
bug,
bounty,
Graham
and
so
then
get
feedback
from
the
bug
bounty
program
from
the
security
researchers
so
that
they
would
tell
us.
You
know
these
were
the
problems
we
found
with
the
soft
multi-tenancy
solution,
and
then
we
can
iterate
on
it
from
there.
C
B
G
C
K
Yes,
okay,
I
just
wanted
to
know
if
you
have,
if
you
think
about
any
limitation
in
terms
of
container
continent
sea,
actually,
whether
the
network,
how
can
you
limit
the
network
access
to
other
namespaces
or
other
resources?
How
can
you
Limit
leakage,
memory
leakage
or
whatever?
What
are
the
limitations
that
you
see
using
existing
communities,
resources
to
make
multi-tenancy
work
so.
D
The
biggest
one
I
mean
is
cluster
wide
resources.
You
know
there's
kind
of
hard
to
prevent
that
so,
especially
like
storage
and
seen
eyes
and
a
lot
of
the
the
cluster
wide
resources
that
are
deployed-
and
you
know
if
a
tenant
wants
to
deploy
an
operator
and
they
have
their
on
C
or
DS.
It
gets
difficult
when
you
have
multiple
names
faces
and
cluster
wide.
So
that's
really
the
big
one
I
that
I
can
think
of
I.
Don't
know.
E
Yeah
I
think
some
of
the
things
are
delegated
like
authentication
and
authorization
schemes
that
can
guarantee
that
it's
kind
of
scoping
down,
as
the
delegation
goes
resource,
controls
and
quotas
are
a
big
concern.
I
think
almost
everyone
struggles
with
that
right
now
of
how
to
properly
cap
and
keep
things
from
over
running
and
as
well
as
kind
of
securing
and
making
sure
that
you
keep
the
control
plane
needs
to
be
available.
E
B
F
So
one
of
the
first
things
we
found
is
that
the
API
is
was
not
designed
in
a
multi-tenant
in
mind,
so
we've
been
working
really
hard
in
order
to
extend
it
in
some
ways,
especially
using,
for
example,
open
policy
agent
where
we
are
using
validate
sirs
and
mutators
in
order
to
give
it
some
capabilities
that
are
needed
for
enclosing
some
of
the
api's.
And
then
you
are
also
losing
some
of
the
end
points.
For
example,
you
cannot
list
namespaces
either
you
see
them
all
or
you
cannot
see
anyone.
B
Ok,
let's
go
on
to
what
new
technology
are
you
all
looking
forward
to
in
terms
of
enhancing
current
Cuba
natives
multi-tenancy
capabilities?
It
could
be
something
like
open
policy
agent
or
it
could
be
some
of
the
newer
container
runtimes
like
cryo
and
Kara
containers
for
isolation,
which
do
you
feel,
is
the
most
important
new
technology
that
means
to
fill
the
gap
of
the
current
kubernetes
multi-tenancy
solution.
So
maybe
we
can
start
with
talk.
F
Yes,
sir,
so
when
we're
thinking
about
this
multi-donor
solution,
we
one
of
the
options
that
we
were
thinking
about
was
assigning
a
note
to
a
user
trying
to
make
notes
assign.
But
then
this
is
something
that
you
started
losing
with
some
resources
and
then
some
boxing
came
up
with
some
things
like
devisor,
where
you
are
spinning,
like
a
small
and
you
a
new
layer.
F
B
F
B
E
For
all
cryo
we're
never
sent
in
on
that
one
which
is
fun
I,
don't
have
to
worry
about
it
as
much
I.
Think
I
am,
but
the
thing
that
I'm
kind
of
excited
about
is
adding
some
formalization
and
to
the
types
that
we're
talking
about,
so
that
we
can
unify
and
sort
of
program.
On
top
of
all
of
these
solutions,.
D
B
B
So
BPF
berkeley
packet
filters,
Linux
kernel
capability,
allows
more
flexible
ability
to
set
up
filters
and
policies
without
actually
having
to
modify
the
kernel.
Eb
PF
actually
extended
me
PF
and
so
there's
new
network
plugins
that
are
leveraging.
That
plus
the
question
was:
is
that
where
you're
looking
forward
to
the
use
of
eb
PF
in
network
plugins
or
in
other
aspects
of
cuban
areas
as
well,
the.
E
E
What
I
say
vpf,
because
it's
a
bit
of
a
hack
but
really
what
it
is,
is
that
we
have
containers
as
this
isolation,
but
then
managing
a
lot
of
what
goes
on
in
the
life
cycle
of
those
containers
is
not
as
easy
to
manage
and
deal
with,
and
so
here's
kind
of
a
a
way
to
deal
with
that
input,
something
that
layer
between
the
outside
world
and
your
containers
and
that's
I,
think
why
it
soaks.
It
has
a
lot
of
potential
for
weathers
monitoring
or
dealing
with
the
network,
layer
and
dynamic
kind
of
routing.
B
Let's
talk
about
existing
communities
features
that
people
may
typically
want
to
use.
So
part
security
policies
is
one
it's
a
little
bit.
Different
people
have
different
opinions
about
particular
policies.
Let's
my
show
fans
how
many
people
are
actually
using
part
security
policies.
Today,
that's
a
that's
a
good
number,
so
I
mean
there
is
right
now
active
discussions
on
how
to
take
part
security
policies
forward
and
how
much
of
that
sort
of
may
overlap
with
more
dynamic
admission
controllers.
So
any
thoughts
from
anybody
on
the
on
the
panel
on
part
security
policies
and
how
they
I.
E
Have
a
lot
of
thoughts
on
this
because
OpenShift
cat
before
pod
security
policies
had
a
concept
called
security
context
constraints
which
are
lately
more
feature
than
pod
security
policies
that
makes
it
means
as
much
as
we
would
like
to
be
full
upstream.
We
would
need
to
work
with
the
community
to
move
pod
security
policies
along
just
like
a
small
example,
is
having
some
priorities
so
that
it's
not
such
a
random
ordering
of
what
gets
applied
to
how
the
policies
are
applied.
E
B
F
I
think
resource
quotas,
as
I
said
before,
is
something
really
important,
because
once
you
divide
the
cluster
you
to
have
resources
for
all
of
them,
but
it's
actually
something
very
limited.
We
know
because,
for
example,
you,
if
you
have
one
pot
in
a
namespace,
you
can
have
resource
allocation
and
then
resource
limits,
but
it's
actually
a
pot
level,
because
at
the
resource
quota
is
just
a
limit,
is
just
a
number
that
doesn't
make
anything
actually
what
it's
making
it
work
is
at
the
pot
level.
B
L
M
E
Pod
security
policies,
what
they're
really
doing
is
allow
you
to
kind
of
classify
pods
by
the
level
of
privileges
that
they
can
have.
You
would
like
to
on
certain
nodes
and
with
certain
users
and
certain
namespaces,
where
you
have
very
controlled
network
or
whatever
else
that
they
can
not
run
as
root,
for
instance,
that
they
have
utilized
all
of
the
best
of
the
security
isolation.
That
Linux
provides
whether
it's
a
farmer
or
SELinux,
as
opposed
to
when
you
do
want.
E
Now
that
we're
running
more
kubernetes
on
kubernetes
and
the
you
know,
including
even
like
OS
configuration,
you
need
more
privilege
and
then
so
you
need
to
separate
out.
So
if,
once
you
can
classify
your
pods
by
these
privilege
levels,
you
can
more
easily
and
dynamically
define
and
how
those
work
together,
whether
it's
you
know
privileged
no
unprivileged,
a
privileged
pod,
can
only
run
on
the
master,
and
only
in
these
namespaces
things
like
that.
Yeah.
F
B
One
other
point
is
that
there
is
a
certain
point
of
view
that
multi-tenant
kubernetes
clusters
should
only
support
non-privileged
containers,
non-root
containers,
but
there's
a
feeling
that
that
may
be
too
restrictive
to
application
teams.
The
my
show
fans
how
many
feel
it's
okay
to
have
multi
tenant
clusters
which
require
all
containers
to
run
as
non-root
non-privileged
containers.
B
B
A
B
E
F
B
So
please,
let's
make
this
collaborative
from
the
initial
show
of
hands.
Everybody
thinks
this
is
an
important
problem.
Everybody
thinks
that
this
is
not
yet
fully
defined.
So
please,
let's
work
on
it
together
and,
let's
hope
next
year,
we'll
be
able
to
say
that
we
have
made
a
lot
of
progress.
Also.