►
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Kubernetes at Cruise: Two Years of Multitenancy - Karl Isenberg, Cruise
Cruise has been working on self-driving cars for six years and growing exponentially for most of that time. Two years ago they started using Kubernetes, betting on namespace-level multitenancy to provide isolation between teams and projects. Today they have over 40 internal tenants, 100,000 pods, 4,000 nodes, and… an embarrassing number of KubeDNS replicas. This session will take you through the motivations, story, and results of migrating to multitenant Kubernetes, along with some hard-earned Pro Tips from the trenches. You’ll also learn about the open source tooling they built around Spinnaker, Vault, Google Cloud, and Istio in order to integrate with our multitenant Kubernetes. Come see how they went from barely isolated to very isolated and saved a few million dollars doing it!
https://sched.co/UaaO
A
You've
ever
been
to
a
talk
where
you
watched
it
later
at
2x
speed.
This
is
not
gonna,
be
that
one
because
I'm
gonna
be
speaking
to
XP
to
get
through
all
these
slides.
So
at
cruise
we
are
building
the
most
advanced
autonomous
vehicles
and
running
back
in
on
kubernetes.
We
have
self-driving
cars
in
San,
Francisco
right
now
and
there's
a
couple
hundred
of
them,
navigating
the
streets
autonomously
and
pretty
soon
we'll
have
some
of
those
available
for
you
in
a
ride
hailing
service.
A
But
here
I'm
not
going
to
talk
about
the
cars
too
much
mostly
talking
about
the
backend
which
runs
on
kubernetes.
So
this
is
just
an
example
of
how
our
cars
talk
to
kubernetes
in
the
backend.
There's
a
couple
different
ways:
either
over
the
internet
or
over
our
private
fiber
backbone
through
some
physical
routers
online,
in
our
data
centers
or
in
our
garages
and
then
over
interconnects
for
high-speed
transfer
data,
because
it
turns
out
you
can't
send
enough
data
over
LTE
or
even
5g,
when
you're
recording
with
so
many
different
devices
on
the
cars.
A
So,
a
little
bit
about
cruises
paths,
we
have
a
constellation
of
different
components
that
make
up
our
path,
so
that
includes
bits
about
ingress
and
egress
security,
observability
deployments
and
management
infrastructure.
A
lot
of
these
are
open
source
tools
or
SAS
products
that
we
integrate
with
and
a
bunch
of
them.
We
brought
ourselves.
A
This
is
sort
of
reaching
capacity
where
real
workloads
start
to
struggle
on
kubernetes,
and
there
are
larger
synthetic
tests
that
show
you
can
get
to
like
3,000
or
5,000
nodes,
but
generally,
what
we
find
is
that
when
you
get
above
a
thousand,
all
a
bunch
of
different
things
start
to
break
down.
So
this
is
where
we
are
and
most
of
our
load
input
and
runs
on
one
of
these
big
clusters.
A
This
blue
line-
these
are
pod,
count
graph
and
it's
showing
we
have
a
high
dynamicism
with
pods
that
get
scheduled
and
deleted
pretty
frequently
along
with
more
consistent
workloads.
We
have
a
very
wide
variety
of
workloads,
including
services
and
jobs
and
agents
that
process
jobs
and
databases
and
semi
stateful
storage,
a
whole
bunch
of
things.
So
we
like
to
use
multi-tenancy
because
it
gets
us
a
little
more
higher
utilization
on
our
nodes,
among
other
things,
so
a
little
bit
about
multi-tenancy,
first,
because
it's
poorly
understood
or
communicated
in
different
ways
by
different
people.
A
When
I
talk
about
multi-tenancy
I
talk
about
multiple
applications
operating
in
a
shared
environment
and
usually
when
you
have
multiple
applications,
you
also
have
multiple
operators,
so
there's
a
little
bit
of
contention
over
whether
the
applications
or
the
tenants
or
the
operators
or
the
tenants,
but
it
doesn't
really
matter
because
you're
gonna
have
more
of
these
anyway
and
chances
are
everybody.
Who's
running
kubernetes
is
running
a
lot
of
different
applications
on
them,
not
just
one
application,
so
you
can
call
the
application
the
tenant.
You
can
call
a
system
of
applications
attend.
A
It's
really
up
to
you.
How
you
define
this,
but
the
real
the
the
crux
of
multi-tenancy,
is
the
logical
isolation
in
order
to
get
from
logical
isolation,
or
rather
to
build
logical
isolation.
You
have
to
put
in
a
whole
bunch
of
components
on
top
of
kubernetes
kubernetes
has
some
of
this
isolation
built
in
and
some
of
it
you
bring
yourself
if
you
go
all
the
way
to
physical
isolation,
then
you're,
not
really
in
multi-tenancy
anymore.
If
you
have
full
physical
isolation,
that's
really
just
single
tenancy
or
multi
instance.
A
So,
the
more
physical
integration
you
have,
the
harder
it
is
to
preserve
logical
isolation
and
the
more
expensive
it
is
with
more
sorry,
the
less
expensive
it
is.
So
if
everything
is
running
on
the
same
hardware,
it's
cheaper
to
run
until
it
breaks
and
then,
if
everything
is
separate,
then
it's
more
expensive
to
run
and
you've
sort
of
reached
the
edges
of
multi-tenancy.
So
why
multi-tenancy,
obviously
cloud
cloud
cost
is
one
of
them.
I
won't
go
too
deep
into
that
one,
because
it's
a
little
obvious,
lower
operational
cost
is
another
one.
A
This
is
just
the
standard
idea
that
a
rising
tide
lifts
all
boats.
We
wanted
to
make
in
our
investments
in
a
focused
area
so
that
it
applied
to
as
many
of
our
internal
tenants
as
possible
to
get
them
to
production
readiness.
Specifically
so
I
talked
about
multi-tenancy
in
layers.
These
are
not
necessarily
checkboxes
where
you
like,
oh
I,
did
this
and
done
I
never
have
to
touch
this
again
more
like
areas
of
concern
in
isolation,
so
starting
at
the
bottom.
A
A
A
A
This
is
one
of
our
early
open-source
projects
and
it
allows
you
to
bind
your
secrets
to
your
application
containers
with
a
in-memory
volume,
so
it
never
gets
written
to
disk.
So
you're
not
Luke
leaking
your
secrets
and
it
gets
these
from
vault
using
logging
in
with
the
kubernetes
service
account,
and
then
you
can
do
this
thing.
A
Called
identity,
translation,
that's
my
word,
not
necessarily
an
industry
industry
term
yet,
but
the
idea
is
I
can
get
from
kubernetes
service
account
to
GCP
service
account
to
AWS
service
account
to
whatever
kind
of
service
accounts
you're
using
without
having
to
inject
them
all
at
configuration
time.
It
allows
for
a
little
bit
looser
configuration
and
it
pulls
us
in
at
a
10-8
time
or
deploy
time.
A
A
So,
with
the
existing
tools
in
the
marketplace,
we
found
it
difficult
to
validate
on
existing
clusters,
figure
out
who
was
violating
those
policies,
apply,
exemptions
and
then
rollout
to
an
enforcement
policy
and
then
backfill
the
people
who
were
in
exemptions
to
get
them
to
stop
doing
those
crazy
things
and
some
of
those
crazy
things
are
like
you
know,
mounting
the
dr.
socket,
which
is
completely
insecure
or
using
bind
mounts
to
places
that
are
on
the
operating
system
that
you
don't
want
them
to
have
access
to
or
acting
as
host.
A
Pin
I
won't
go
through
all
of
these,
but
one
of
the
things
we
noticed
when
we
did
this
was
that
the
default,
docker
or
second
profile
gets
disabled
by
kubernetes
because
of
some
reverse
compatibility
concerns,
so
we
go
and
enable
that,
with
with
the
default
mutating
web
hook,
so
the
web
hook
applies
these
things.
When
the
pods
are
created,
then
they
automatically
get
these
security
features
for
free,
that's
also
an
open-source
tool
and
it's
available
on
github.
So
a
next
layer
of
isolation
is
domain.
Isolation,
there's
a
lot
of
different
ways
to
slice
this.
A
So
when
I
talk
about
domain
isolation,
I'm
talking
about
environmental,
organizational
and
architectural
domains,
so
kubernetes,
you
basically
have
two
options
for
slicing
these
orthogonal.
You
have
clusters
and
you
have
namespaces
the-the-the
sig
for
multi-tenancy
is
working
on
hierarchical
namespaces,
but
they're
not
there
yet.
So
what
you
have
now
is
a
choice.
You
got
to
pick
two
of
these,
so
this
this
island
has
fit
beyond
one
and
logs
on
the
other.
Just
so
you're
aware
the
the
way
we've
sliced.
This
is
originally
environmental
versus
organization.
A
Also,
we
have
clusters
for
our
environments
in
different
BP
seas,
and
then
we
have
namespaces
for
the
teams.
So
we
thought
that's
worked
great
for
awhile
until
we
realized
that
in
hyper-growth
your
teams
change
really
frequently
and
then
you
have
to
refer
it
refactor,
all
your
namespaces
all
the
time.
So
what
we
switch
to
instead
was
a
project-based
namespace
which
allows
for
more
granularity
for
tighter
security
and
still
gets
you
the
benefit
of
an
orthogonal
domain.
A
We
started
with
one
cluster
in
each
environment
and
we
moved
to
multi
cluster
as
we've
as
we've
needed,
but
I'll
talk
about
that
a
little
later
so
permission,
isolation
is
the
next
layer
of
multi-tenancy,
and
this
really
comes
down
to
authorization
and
authorization
management.
Kubernetes
gives
you
the
tools
to
do
our
back
to
do
our
back,
but
it
doesn't
really
give
you
the
tools
to
manage
your
permissions
above
that
so
oftentimes.
You
need
some
other
layer.
A
On
top
of
that
to
make
that
work,
one
of
the
tools
that
we
wrote
to
help
this
out
was
our
back
sync
and
our
beck
sync
allows
us
to
bind
groups.
Google
groups
specifically
to
role
bindings,
sorry
use
row
bindings
on
groups
to
give
them
roles
to
give
them
permissions,
because
the
roles
are
a
set
of
permissions
and
then
this
one
other
feature
we
threw
into
this
was
the
ability
to
add
service
accounts
to
groups,
so
we
can
do
the
same
thing
for
service
accounts.
A
There
is
a
new
feature
you
can
sort
of
see
also
at
the
bottom,
that
gke
has
immigration
recently
with
Google
Groups,
but
they
still
don't
have
service
accounts
in
groups
yet.
So,
if
you
want
that,
we
still
provide
it
here,
and
this
is
also
an
open-source
project,
vault
workspaces.
So
in
terms
of
managing
your
secrets,
right
now,
with
kubernetes
you're,
effectively
managing
them
at
the
namespace
level,
which,
if
you're
having
a
team
level,
might
not
be
as
secure
as
you
want.
If
you
had
a
project
level,
it
might
be
fine,
it
kind
of
depends.
A
We
wanted
more
security
than
that
and
more
isolation,
so
we
isolate
our
secrets
at
the
application
level
or
at
the
workload
level.
So
one
of
the
ways
we
do,
that
is
with
a
standard,
hierarchical
path
in
volt,
and
then
that
allows
us
to
have
an
application
space
in
vault,
and
then
your
application
has
its
own
service
account
and
then
it
can
log
in
and
grab
those
secrets
for
your
application.
But
nobody
else
is
so
in
terms
of
managing
these.
A
Isopod
is
a
tool
that
we
wrote
specifically
for
add-on
management
and
one
of
those
add-ons
was
managing
permissions
for
awhile.
So
this
is
similar
to
helm
in
the
idea
that
it's
used
for
deploying
things
and
but
instead
of
templates,
we
write
code
in
skylark,
which
is
a
Python
subset.
So
skylark
allows
us
to
have
a
nearly
Turing
complete
language
in
order
to
programmatically
make
templates
for
our
add-ons,
because
we
didn't
really
want
to
go
through
the
hassle
of
releasing
every
single
atom
when
we're
running
20
per
cluster
and
after
deploy
them
took
20
clusters.
A
So
this
allows
us
just
to
push
out
master
to
the
top
of
our
set
and
deploy
it
everywhere,
all
at
once,
which
may
not
be
your
requirements,
because
you
might
have
more
diversity
than
that.
It
works
for
us.
Also,
Alessi
ammo
makes
people
happy
sometimes,
and
the
helmet
iller
alternative
was
insecure
at
the
time.
Thankfully,
home
3
is
out
now
and
the
terraform
kubernetes
plugin
at
the
time
we
started.
This
wasn't
really
usable.
I've
heard
they've
invested
in
a
little
more.
A
We
need
to
check
on
it
again,
but
this
is
a
fun
tool
and
it's
also
open
source.
Next
thing
we
wrote
was
Juno,
unfortunately,
for
you
guys
this
guy.
This
is
not
open
source,
but
it's
an
internal
project
for
now,
and
this
allows
us
to
configure
a
lot
of
different
things
for
our
tenants
to
increase
their
isolation
and
and
improve
our
operational
speed.
So,
for
example,
here
we
see
we
can
enable
GCP
projects.
I
can
add
them
to
a
share.
Bbc
I
can
create
a
vault
workspace
and
I
can
create
a
Koopa
Nettie's
namespace.
A
So
we
hope
to
allow
this
to
self-service,
to
create
namespaces
on
demand
for
our
users,
and
ideally
you
can
either
you
know,
authorize
people
to
do
that
by
using
groups,
or
you
can
do
a
sort
of
request.
Approval
process
where
the
platform
team
approves
these
resources
granted,
because
each
of
these
resources
comes
with
an
effective
quota
and
you
need
to
do
some
capacity
planning
around
it,
which
is
still
it
yet
to
be
isolated.
A
So
resource
isolation
is
sort
of
the
next
tier
here
and
there's
some
built-in
types
of
the
CPU
GPU
memory,
everybody
kind
of
knows
those
when
they
use
good
burn.
It
is
the
ones
that
are
sort
of
less
isolated.
Is
persistent
storage
and
ephemeral
storage.
Let
me
step
back
on
the
persistent
storage
for
a
second.
You
can
isolate
that
really.
Well,
because
it's
one
disk
to
one
application
unless
you're
using
local,
persistent
storage
and
then
it
gets
a
little
more
complicated
yet
to
mount.
A
You
have
to
pre
mount
your
applications
or
have
some
way
to
runtime
mountain
mount.
Sorry,
pre
mount
your
drives
or
have
some
some
runtime
drive
mounting
it's
pretty
complicated
to
manage
and
the
storage
classes
are
sorry.
The
the
quota
is
all
storage
class
specific.
So
if
you
have
like
two
storage
classes
because
they're
in
two
different
zones-
but
your
quota
is
regional
at
the
project
level,
then
you
have
to
like
add
those
together
to
get
your
quota
makes
capacity
planning
a
little
harder.
A
But
another
thing
you
have
to
concern
yourself
with
the
ephemeral
storage
here
is
that
all
these
storage
volumes
here
in
the
middle
all
share
your
root.
Drive,
unless
you've
subdivided
them
some
more,
so
you
might
want
to
consider
isolating
those
so
that
your
doctor
images
don't
take
a
ball
your
space
on
your
storage
volumes
or
your
storage
volumes.
Don't
you
know
block
you
from
be
able
to
launch
more
pods
or
your
OS
crashes,
because
one
of
your
pods
is
using
too
much
disk.
A
We
had
a
fun
bug
where
we
crashed
docker,
because
we
were
using
too
much
disk
and
then
doctor
slowed
down
and
the
health
check
for
docker
waited
30
seconds
and
then
killed
it
and
then
waited
for
it
to
come
back
up.
But
if
you're
using
too
much
disk,
it
would
never
come
back
up.
So
you
ended
up
with
a
cascade
failure
where
you
get
evicted
from
the
node
and
crash
the
entire
cluster.
So
don't
do
that.
A
A
So
we
do
this
with
Juno
or
isopod
and
allows
us
to
set
a
default
quota
for
each
namespace
when
we
create
the
namespace
and
then
we
set
a
really
low
limit
for
resource
limits
on
CPU
and
memory
to
force
people
to
specify
that
creates
a
feedback
mechanism
that
requires
them
to
know
about
resources
or
have
some
tool
that
does
that
for
them
next
level
is
network
isolation.
I
wish
our
data
centers.
Look
like
this
they're,
really
cool
got
a
nice
air
gap.
There
we're
gonna
have
that.
Instead,
we
have
shared
tunnels
so
share
tunnels.
A
Here,
one
two
call
out
is
nat
gateways.
Not
gateways
like
to
fall
over
especially
be
using
them
with
static,
IPS
static.
Ips
then
get
wait-listed
by
people
who
are
doing
things
that
maybe
they
shouldn't
do.
Then
you
have
wait,
lists
strewn
across
everybody
and
you
can't
auto
scale
your
or
now
that's
because
then
you'd
have
to
go
update
all
the
white
lists
or
things
break,
so
don't
do
that
either.
White
lists
are
bad,
but
the
we
switched
from
a
bill.
A
Sorry,
a
self
deployed
terraform
module
managed
in
that
gateway
that
didn't
scare
scale
very
well
to
cloud
nap
when
that
was
offered
by
Google,
and
so
that's
worked
a
little
bit
better
for
us,
but
it
still
has
the
same
whitelist
problem.
Ingress/Egress
QPS
is
completely
nice
elated.
So
unless
you
are
running
something
like
Sto,
you
basically
don't
get
QPS
or
request
isolation
between
containers
or
pods
on
the
same
node.
A
So
what
we
did
was
we
went
from
Broadwell
to
sky
lake
and
got
twice
as
much
network
speed
to
avoid
the
problem,
but
hopefully
we're
working
towards
increasing
QPS
isolation
or
just
ingress/egress
worth
noting
if
you're
using
Network
shared
network
storage
that
is
shared
on
the
network
like
it's
in
the
name.
So
your
network
and
your
storage
are
now
on
isolated
from
each
other,
much
less
the
pods
they're
on
so
there's
a
little
bit
of
complexity
there.
A
That
needs
to
be
separated
and
you
might
want
to
use
like
actual
physically
attached
disks
instead
of
network
attacks
disk.
If
you
need
more
isolation,
bandwidth
here,
there's
a
CNI,
bye
and
bandwidth
plugin,
but
it
doesn't
work
on
gke
because
the
version
of
calico
isn't
new
enough.
So
we've
asked
them
to
increase
that
and
we'll
try
that
out
pretty
soon.
In
the
meantime,
I
working
on
sto,
where
we
can
get
rate
limits
with
the
layer,
7
load
balancer
worth
calling
out
that
layer,
7
load
balancing,
is
slower
than
layer
layer,
4
almost
universally.
A
So
speaking
of
this
do
network
policies
built
into
kubernetes
are
great,
except
for
when
they're,
not,
which
is
when
you're
managing
them,
because
managing
IP
blocks
is
effectively
managing
whitelists.
It's
no
fun.
Namespace,
selecting
and
pod
selectors
tends
to
be
a
blunt
instrument,
and
it's
not
really
great.
Unless
you
have
very
fine-grain
namespaces,
where
your
namespaces
are
your
application
and
then
you
want
to
you
know,
control
them
between.
We,
like
the
sto
model,
a
little
bit
better,
although
it's
kind
of
complicated,
but
you
can
effectively
do
denier
list
checker.
A
That's
basically
black
lists
and
white
lists.
You
can
also
do
service
authentication
which
allows
you
to
say
you
know
this
service
account
has
access
to
access
this
service
run
by
there's
other
service
account
and
then
make
policies
that
enforce
what
can
talk
to
what
integration
isolation.
This
one
is
the
biggest
unsolved
problems.
I
think
when
you
have
kubernetes
kubernetes
is
not
enough
and
you
need
a
whole
bunch
of
other
things
like
I,
said
a
constellation
of
components
and
applications
and
operators
all
sorts
of
stuff.
A
A
We
found
that
we
didn't
like
it
running
on
the
same
nodes
as
the
application,
so
we
moved
into
its
own
node
pool,
and
then
we
found
that
we
didn't,
like
isolating
our
own,
sorry
make
managing
a
new
node
pool
and
having
it,
you
know,
have
different
rules
and
everybody
else
and
also
then
have
two
layers
of
auto
scaling.
So
we
moved
it
out
to
GCE.
Now
we
have
the
load
balancers
running
on
VMs,
and
then
we
have
one
layer
of
auto
scaling
and
it's
much
more
performant
and
just
as
easy
to
manage.
A
A
Unfortunately,
there's
no
current
mechanism
for
spinning
up
or
requesting
a
new,
actual
physical
load
balancer
for
every
namespace,
so
I
know,
there's
an
investment
in
the
API
machinery,
sig
to
work
on
gateways
and
gateway
classes,
so
I'm
hoping
that
pans
out
pretty
soon
so
that
we
can
say
I
need
to
new
one
of
these
for
this
namespace
or
this
tenant.
Another
problem
here
that
we
don't
have
very
well
isolated
is
DNS.
A
This
is
just
just
kind
of
a
lull
slide
in
how
DNS
is
is
set
up,
because
you
have
a
cache
in
front
of
a
cache
in
front
of
a
cache
in
front
of
a
cache
in
front
of
DNS,
and
so
this
is
just
private
zones.
But
basically
your
DNS
on
the
cluster
is
really
easy
to
overwhelm
with
any
of
your
tenants
and
that
will
affect
every
other
tenant
and
unfortunately,
that's
just
the
way
it
is.
A
All
the
logs,
all
the
metrics,
all
the
tracing
tends
to
go
to
a
SAS
provider
or,
if
you're
running
your
own.
It
all
goes
in
the
same
place,
so
there's
there's.
Usually
the
SAS
providers
will
have
multi-tenancy
because
you
are
a
tenant,
but
they
don't
have
two
layers
of
multi-tenancy,
so
you
can't
give
tenancy
to
your
users
underneath
you
or
that
you're
hosting,
for
so,
if
you're
running
your
own,
also
like
Prometheus
and
Ravana
have
the
same
problems:
they're
not
really
designed
for
it.
A
They
might
have
authentication
low-level
multi-tenancy,
but
they
don't
have
authorization
rules
for
most
of
their
stuff.
This
is
just
a
fun
hack.
We
made
to
get
logs
into
a
different
project
in
GCP,
so
we
we
actually
double
admit
container
logs
so
that
they
can
get
into
a
team
project,
and
that
team
has
access
to
do
log
based
metrics
that
way
they
can
self-service
their
metrics
and
those
metrics
then
go
to
data
dog,
and
then
they
can
do
the
alert
flow.
A
The
way
they
normally
would,
rather
than
having
to
come
through
us
and
do
some
like
get
ops
management
of
the
log
based
metrics
also
more
critically,
they
can
do
dashboards
in
their
project,
whereas
we
don't
want
to
give
them
edit
to
our
project,
because
then
they'd
stomp
on
each
other's
toes.
That's
like
the
opposite
of
isolation.
A
So,
as
was
mentioned,
you
can
go
from
no
isolation.
Sharing,
sharing
a
cluster
admin
account
which
I
don't
recommend,
do
logical.
Isolation,
which
has
been
called
sort
of
soft
multi-tenancy
before
hard
multi-tenancy
is,
is
kind
of
the
idea
that
you
don't
trust
the
people
that
are
underneath
you
I,
don't
recommend
that
for
a
kubernetes
install
because
of
all
these
sort
of
isolation,
constraints
or
problems
or
yet
to
be
implemented,
features
and
then
system.
Isolation
is
really
single
tenant
to
you
or
multi
instance
strategy.
A
So
you
choose
what
layer
you
want
to
do
and
what
works
for.
You
may
not
work
for
everybody.
So
there's
a
lot
of
complexity
in
that
and
it's
hard
to
provide
it
like
a
single
done
and
sold
solution.
You
can
isolate
at
the
machine
level.
You
can
isolate
it.
The
node
pool
level
you
can
isolate
at
the
class
you
can
isolate
at
the
network
level.
Any
one
of
these
would
be
real
system,
isolation,
the
components
it's
good
to
call
out
that
each
one
of
these
components,
I
listed
you
can
get
to
scale
issues
pretty
quickly.
A
The
the
goal
of
this
whole
strategy
was
to
use,
save
us
money,
lower
operational
cost,
get
scale,
validation,
and
you
know
Raisa,
tired
of
all
boats,
so
I
think
we
did
that
the
biggest
cost
was
shared
downtime,
so
anyone
users
bugs
or
anyone
users
system
like
running
into
somebody
else's
bugs
or
you
know,
running
into
an
isolation
boundary
that
doesn't
exist
and
they
can.
They
can
take
out
everybody
else.
So
that
becomes
a
security
issue
if
you
get
compromised,
but
it's
also
an
availability
issue
just
with
your
regular
tenants.
A
So
generally,
it's
good,
but
you
have
to
make
make
sure
that
your
use
cases
are
good
for
it.
So
what
we've
done
is
we
started
big
one
big
cluster
I
mean
we
have
Deb
stage
prod,
but
our
problem
is
the
biggest
and
we
put
everybody
on
that
until
it
hurts,
and
then
we
move
them
out.
So
we
have
a
couple
other
tenants
with
their
own
clusters,
but
I
wouldn't
call
them
single
tenant
clusters,
because
it's
really
like
a
team,
they
get
their
cluster
and
then
that
team
has
different
projects.
They
get
different
namespaces.
A
So
we
really
have
a
bunch
of
multi
dining
clusters.
They're
just
you
know
isolated
on
different
boundaries.
Another
thing
we
learned
here
is
that
we
got
a
bunch
of
security
improvements
prioritized
because
they
were
also
operational
and
isolation.
'el
concerns
and
I
think
that's
a
win
for
everybody
is
investing
in
these
security
tools
which
might
might
have
been
overlooked
otherwise
and
then
expertise
building
we
we
have
broken
kubernetes
many
times,
and
so
that
allows
us
to
know
where
the
edges
are
and
what
not
to
do
in
the
future
or
what
to
work
around.
A
What
to
prioritize.
In
terms
of
fixes
and
I
think
that
was
valuable,
especially
before
we
have
customers
that
we
would
embarrass
ourselves
in
front
of
there.
The
cost
here
is
incompatible
tooling
turns
out
that
not
just
integrations
but
tooling
tend
to
be
incompatible.
One
of
them
I
mentioned
earlier,
was
like
helm,
tiller.
It
didn't
have
any
officee
in
it,
so
it
was
really
like
you
install
a
helm
and
it
can
install
the
whole
cluster,
bypassing
your
tenancy
rules
or
you
install
it
in
the
namespace,
and
you
run
multiple
incidents.
It's
not
very
great
helm.
A
A
So
we
can't
really
install
it
with
everybody
else's
namespaces,
for
example,
because
it
creates
and
deletes
namespaces,
so
it
basically
has
to
have
an
equivalent
to
cluster
admin
or
more
permissions
than
a
tenant
would
have
so
you
can't
put
it
in
a
tenancy
box.
So
we
give
that,
as
its
own
clusters
to
see
are
the
installations
another
one.
The
platform
team
has
to
manage
the
IDs
because
they're
outside
of
the
namespace
and
they're
shared
by
the
whole
cluster.
A
A
That's
going
to
break
down
all
the
other
tenants,
and
so
one
of
the
things
that
this
also
gives
us
in
general
is
not
just
to
see
IDs
but
the
chance
to
review
tenants
that
we
pull
into
the
system
and
do
some
capacity
planning
with
them
and
make
sure
that
we
have
quota
for
them.
Make
sure
that
they
are
not
doing
anything
crazy,
like
a
lift
and
shift
without
understanding
what
cloud
native
means.
A
But
that's
that's
a
time
intensive
process,
and
so
it
also
slows
down
the
the
self-service
aspect
so
that
it's
kind
of
a
challenge
pro
and
con
and
kubernetes
itself.
It's
hard
not
just
for
the
platform
team,
but
for
everyone
using
it,
and
so
they
all
have
to
learn
it.
They
all
have
to
learn
the
pads
tools
that
we
integrate
with
and
we're
constantly
building
new
abstractions
to
that
they
either
have
to
learn
in
parallel,
or
that
you
know,
hide
some
complexity
from
them,
but
there's
no
one
system
that
the
overlays
that
makes
this
simple.
C
Hi
Carl's,
thanks
for
sharing
this
with
us
and
a
very
informative,
so
I'm
Yan
Cheng
from
JD
dot-com
I
have
two
questions.
The
first
one
is:
can
you
talk
a
little
bit
more
about
your
use
cases?
Workloads.
The
second
one
is
in
terms
of
resource
isolation,
and
do
you
see
the
the
disk
I/o
and
like
the
Panda
way,
offender
is
a
key
change
there
and
that's
a
problem.
You
know
a
multi-tenant
sender
that
is
platform
Thanks,
so.
A
I
guess
two
questions.
One
is
on
whether
the
bandwidth
and
QPS
isolation
is
required
or
a
problem,
and
the
other
one
was
based
on
our
workload,
styles
or
types.
So
we
have
a
lot
of
different
workloads.
We
have
standard
web
applications
micro
services.
We
also
have
jobs
that
we
get
kicked
off
by
an
external
scheduler.
We
have
cron
jobs
that
happen
nightly
for
batch
job
stuff.
We
have
sort
of
semi,
stateless
stuff
like
elastic,
search
or
a
cache,
a
layer.
We
also
read
its
databases,
for
example.
A
So
we
have
all
those
systems
together
and
a
couple
more
stragglers
that
I
didn't
mention,
but
it's
it's
enough
that
we
can
get
pretty
good
isolation,
but
we
also
have
some
jobs
that,
like
want
to
use
sixty
CPU,
we
had
to
give
them
their
own
namespace.
In
order
to
isolate
them
a
little
more
otherwise
they'll
block,
scheduling
or
won't
be
able
to
schedule
themselves,
and
then
we
have,
in
terms
of
the
bandwidth
and
QPS,
we
sort
of
solved
the
immediate
problem
that
we
had
by
giving
more
bandwidth.
But
it's
not
isolation.
A
So,
there's
a
little
bit
of
complexity
with
us
running
on
gke,
which
means
that
we're
not
running
an
upstream
kubernetes
version
and
gk
is
a
little
a
little
behind
on
versions
and
we're
behind
on
their
versions.
And
so,
if
there's
a
new
patch
into
kubernetes,
it'll
take
like
a
year
to
get
to
us.
So
that's
that's
something
that
needs
to
be
improved
both
by
upstream
and
Google.