►
From YouTube: Running Isolated VirtualClusters With Kata & Cluster API - Chris Hein & Eric Ernst, Apple, Inc
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2023 in Amsterdam, The Netherlands from April 17-21. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Running Isolated VirtualClusters With Kata & Cluster API - Chris Hein & Eric Ernst, Apple, Inc
Speakers: Chris Hein, Eric Ernst
Kubernetes is generally considered a single-tenant container orchestrator, but as companies have been running it and realizing the benefits of the Kubernetes architecture contrasted with the nontrivial level-of-effort of managing many single tenant clusters we’ve seen a spike in use cases & projects that support the need for multi-tenant & zero-trust deployments. You can see this in the growth of “Sandboxed Runtimes” like Kata, gVisor & Firecracker. As well as tools like vCluster, Kamaji & HNC. In this talk Chris Hein & Eric Ernst will demonstrate one way hard multi-tenancy can be achieved by leveraging Cluster API Nested with VirtualCluster running inside a Kubernetes cluster with workload isolation & virtual networking being provided by the Kata runtime. Users of this architecture get the benefits of per-tenant Kubernetes control planes to use CRDs, Admission Webhooks, Cluster level RBAC, Aggregate APIServers along with workload & network segregation while reducing the overall maintenance burden. Modeled after the ICDCS paper by folks from Alibaba - https://bit.ly/3tfnWnA If you are interested in sandboxed runtimes, hard multi-tenancy, scaling Kubernetes, Cluster API or multi-cluster Kubernetes this is the talk for you.
A
A
A
So
to
get
us
started,
we
are
in
the
multi-tenancy
track
right
now,
so
as
I'm
gonna
just
make
an
assumption
that
all
of
you
are
interested
in
running
multi-tenant
kubernetes.
If
not,
hopefully
it's
still
interesting
to
you,
but
as
everybody
here
knows,
multi-tenancy
and
kubernetes
is
it's
hard.
It's
not
easy!
A
This
talk
is
going
to
be
split
into
two
sections:
we're
gonna
first
start
talk
about
control,
plane,
multi-tenancy
and
then
we're
going
to
talk
about
data,
plane,
multi-tenancy
and
then
we're
going
to
talk
about
some
some
improvements
that
we've
done
or
features
that
we've
added
to
cada
recently
to
make
that
multi-tenant
story
even
more
advanced
in
kubernetes
itself.
So
to
get
us
started,
we're
going
to
first
talk
about
multi-tenant
control
planes
and
there's
a
bunch
of
tools
in
this
space
these
days.
A
To
do
this
and
we're
only
going
to
focus
on
one,
it's
one
that
I'm
a
maintainer
for
it's
called
virtual
cluster.
There's
another
project
called
the
cluster
that
you
could
use
some
of
these
same
exact,
Technologies
and
tools
to
to
implement
with
that
lots
of
other
tools
in
this
space
as
well.
So
before
we
get
started
on
that,
we're
going
to
first
talk
about
some
issues
that
you
have
running:
multi-tenant
control
planes
itself,
one
of
the
biggest
things
that
you
run
into
or
I
hope.
A
A
lot
of
you
have
run
into
is
Clumsy
and
thrashy
clients.
This
is
something
that
like,
if
you
have
a
kubernetes
API
server,
you
expose
it
to
a
bunch
of
different
tenants,
they're
going
to
do
things
that
you
don't
want
and
things
like
not
setting
resource
version,
zero
and
getting
basically
direct
NCD
hits
on
say,
for
instance,
a
pod
list,
maybe
in
a
2000
pod
namespace,
where
it's
just
having
to
do
the
Json
serialization
it's
hitting
at
CD.
A
It's
causing
a
lot
of
churn
and,
in
essence,
causing
problems
for
other
tenants
through
those
through
those
requests,
and
so
in
this
example.
We're
basically
just
saying
three
tenants
doing
this
pod
list
and
basically
hammering
SCD,
causing
problems
for
other
tenants
and,
in
essence,
taking
down
that
single
that
single
control
plane
from
that
standpoint,
on
top
of
that
users,
these
days
as
kubernetes
has
become
this
base
platform
that
we
can
build
platforms
on
top
of
they
want
access
to
all
of
it
and
out
of
the
box,
you
can't
do
this.
A
You
end
up,
then,
in
a
place
where
you
can't
do
Cloud
native
tools,
things
like
normal
CI,
CD
tools
that
want
access
to
the
entire
cluster
or
multiple
namespaces,
or
the
ability
to
create
a
whole
namespace
to
deploy
like
a
red,
green
deployment
or
a
blue
green
deployment.
And
so
you
end
up
in
this
place,
where
it's
just
hard
to
implement
these
strategies.
I
hope,
that's
things
that
you
all
have
actually
experienced
as
well.
A
Now
in
talking
about
isolating
these
again.
This
is
only
one
strategy
to
do
it.
There's
many
here
behind
this,
but
this
is
one
strategy
that
that
we
have
been
working
on
for
a
bit
which
is
running
virtual
cluster
and
I'm,
going
to
talk
high
level
I'm
not
going
to
go
into
the
in-depth
of
this,
because
this
is,
in
essence,
rehashing
a
talk
from
fanguo
from
Alibaba
or
now
at
Microsoft,
who
who
did
a
presentation
about
this
in
2020
in
kubecon?
A
But
then
you
deploy
two
more
main
components
that
VC
Sinker
up
at
the
top
and
VC
manager.
Vc
managers
behind
the
scenes
what's
actually
orchestrating
creating
those
control
planes,
along
with
cluster
API
and
the
VC
Sinker,
is
sitting
there
as
a
multi-tenant
scheduler
that
listens
to
all
tenant
control
planes
and
takes
pod
scheduleable
resources
and
syncs
them
down
to
the
super
cluster.
A
Technically,
all
the
green
boxes
are
virtual
cluster
and
there's
a
piece
in
there
called
VN
agent
or
virtual
node
agent,
and
this
is
an
on
on
node
agent
that
acts
as
a
proxy
that
takes
in
requests
from
each
one
of
those
tenant.
Api
servers
there's
a
lot
behind
this
I.
Don't
expect
you
to
understand
all
of
these
pieces,
but
if
you're
interested
phase
talk
goes
really
deep
into
it
and
it's
it's
a
fantastic
Deep
dive
behind
the
scenes,
the
rest
of
it.
A
The
blue
boxes,
are
all
kubernetes
events
that
we
all
know
and
use
Cris,
cubelets
so
on
and
so
forth,
and
we
got
workloads
that
come
from
those
tenant
control
planes.
At
the
end
of
the
day,
a
user
of
this
system
would
talk
to
one
of
those
dedicated
control,
planes,
tenant
a
tenant
b
or
tenant
C,
and
they
would
end
up
as
pod
scheduleable
resources
in
that
lower
level
cluster.
A
Now
what
what
this
actually
looks
like
behind
the
scenes
is
something
like
this.
So
imagine
you
have
a
massive
cloud
of
infrastructure
and
you
want
to
leverage
that
and
you
want
to
start
bin
packing
workloads
into
it,
and
you
now
have
something
like
tenant
a
b
and
c,
and
you
can
even
have
things
like
a
malicious
tenant
and
a
clumsy
tenant
all
talking
to
their
own
API
servers
and
those
workloads
are
then
getting
scheduled
down
in
the
super
cluster.
But
all
of
that's
isolated.
A
So
any
of
those
interactions
and
we're
only
talking
about
the
data,
the
control
plane
here,
all
of
those
interactions
are
isolated.
So,
if
somebody
goes
and
says
is
the
clumsy
client
and
does
massive
pod
list,
they
don't
affect
anybody
but
themselves,
because
it's
just
hitting
their
SED
instance
they're,
not
then
causing
problems
for
everybody
else
or
that
malicious
tenant.
That's
trying
to
do
something
bad.
A
We
actually
have
pieces
in
the
Sinker
that'll
allow
us
to
stop
pods
from
being
scheduled
if
we
don't
want
them
to,
for
whatever
reason,
pod
security
policies,
for
example,
that
they
can't
change
again
being
a
cluster
level
resource.
Yes,
pod
security
policies
are
going
away,
but
there's
other
resources
that
are
like
it
I
forget
what
it's
called
so
anyways.
This
then
takes
you
infrastructure
and
allows
you
to
start
bin
packing
it
in
a
much
denser
way.
A
So
you
can
say
if
you
look
at
this
and
say
that
that
first
node
or
that
first
rack
now
has
some
workloads
from
the
malicious
tenant,
the
the
clumsy
tenant
and
that
tenant
a-
and
this
can
be
done
with
anything
from
run,
C
to
Kata
to
G
visor.
Any
of
those
Stacks
under
the
hood,
but
we
can
get
some
benefits
out
of
actually
working
on
the
data
plane
and
making
it
more
isolated.
A
That's
really
what
we're
trying
to
protect
against
you
then
get
access
to
Cluster
level
resources,
because
you
have
cluster
admin
to
that
tenant
control,
plane
and
it
doesn't
all
those
resources
don't
end
up
in
the
supercluster
and
then
you
can
also
use
all
those
Cloud
native
tools
like
you
normally
would
any
of
the
operators
that
you
want
to
deploy
things
like
cross
plane.
You
could
deploy
into
this
because
you
have
cluster
admin
capabilities
and
you
can
deploy
cluster
scoped
resources,
police,
solves
half
the
problem.
B
If
we're
to
look
at
the
most
basic
level,
people
want
to
be
able
to
run
their
workload
on
a
machine
and
because
we
do
multi-tenancy,
you
can
both
run
on
the
same
exact
machine
and
they're.
Just
the
process
running
on
the
host
running
on
a
host
Linux
kernel
on
your
one
node
one
of
many
nodes
where
they're
all
running
we're
going
to
use
containers,
because
we
want
to
have
a
view
that
they're
the
only
thing
running
on
the
machine
and
that
it's
constrained
appropriately,
there's
no
denial
of
service
between
workloads,
everything
else.
B
So
these
are
great
features
of
the
Linux
kernel
using
namespaces
using
c
groups
judiciously
picking
what
capabilities
are
appropriate
for
the
workload
filtering
out.
Different
sys
calls
things
like
that
and
it
works
pretty
well.
The
concern
from
a
security
perspective
is
that
this
is
a
single
interface,
so
all
of
these
capabilities
are
based
out
of
the
host
kernel.
So
if
you
do
have
a
zero
day
in
any
kind
of
privilege
escalation,
you
are
now
rude
on
the
host
me
as
a
provider.
That's
concerning,
but
you
as
maybe
one
of
many
tenants
on
that.
B
Node
should
be
pretty
concerned
about
this
as
well,
so
neutral
and
trusting
tenants
yeah.
That's
that's
bad
for
them
in
that
case,
so
we
have
different
options
here.
The
first
option
is
essentially
YOLO,
like
the
security
profile
that
we
have
like
what
is
the
cost
of
an
escape?
It's
pretty
low
cost.
B
If
we
don't
have
very
sensitive
information,
if
the
tenants
kind
of
do
trust
each
other
a
little
bit,
it
doesn't
matter,
don't
pay
the
cost
for
extra
isolation,
we're
talking
about
hard
multi-tenancy,
so
that
that's
out
the
other
option
you
have
is
don't
run
multiple
workloads
from
different
tenants
on
the
same
node
so
give
everybody
a
different
node
pool.
This
has
a
challenge
one.
You
can
have
fragmentation
or
resources
where
two
tenants,
one
of
them
uses
100
of
the
capacity
another
needs
is
five
percent.
B
That's
that's
unfortunate
too.
If
you
do
have
an
escape
infrastructure
provider,
that's
concerning
I.
Don't
have
two
layers
of
isolation
anymore,
also,
if
you
have
an
escape
and
yeah
you're
you're
stuck
on
your
one
kubernetes
worker
node,
but
a
worker
node
has
a
lot
of
capabilities
as
far
as
authorization
to
kubernetes
apis
as
well
as
whatever
the
network
has
access
to.
So
for
this
we're
going
to
say,
that's,
that's
not
quite
enough
and
we're
going
to
look
at
providing
stronger
workload
isolation.
B
So
let's
have
two
layers
of
isolation
and
kind
of
in
kubernetes
and
container
area.
We
call
this
like
generally
sandboxed,
runtimes
being
that
we
have
two
layers
of
isolation.
A
good
example
is
G
visor
for
this
another
one.
Since
we're
talking
about
Kata,
you
can
guess
it
is
we're
going
to
be
talking
about
cotta
containers.
So
in
the
example
going
from
a
traditional
container
on
on
that
side,
I
guess
it's
your
left
side
versus
the
right
hand,
side
with
kind
of
containers.
B
We
launch
a
minimal
virtual
machine,
so
one
layer,
hardware,
virtualization
and
then
within
that
on
a
guest
Linux
kernel.
They
then
we're
creating
regular
containers
using
names
to
AC
groups,
all
the
things
second
layer
of
isolation,
the
guest
kernel,
a
touch
more
detail.
What
that
ends
up?
Looking
like
cubelet,
toxic
container,
D
or
cryo
talks
to
a
task
shim
below
that,
so
like
a
Kata
shim
or
it
could
be
traditionally
like
a
run
c
one
or
it
could
be
anything
really
in
the
case
of
Kata.
B
We'll
talk
about
these
little
components
a
little
bit
and
that's
why
I'm
kind
of
introducing
it
on
the
bottom
there's
way
too
much
information.
You
shouldn't
care
about
other
than
to
know
that
networking
just
works.
Usually
you
drop
a
v
eth
into
a
network
name
space.
We
then
all
traffic
is
piped
directly
into
the
gas
such
that
the
container
workload
has
access
to
it
without
any
configuration
necessary,
and
then
we
can
get
into
how
we
can
leverage
these
things
to
come
together.
Cool.
A
Thank
you
for
that.
So
virtual
cluster
introduces
one
thing
which
I
left
completely
out
until
we
got
to
this
out
of
the
box.
Kubernetes
does
one
thing
that
is
kind
of
odd
in
essence,
Services,
because
they're,
a
virtual
component
are
actually
allocated
the
IPS
from
them
are
allocated
from
a
cider
range,
that's
hard-coded
into
the
API
server
Flags,
so
you
set
that
and
it
basically
gets
allocated
now.
If
we
go
back,
if
you
think
back
to
that
that
diagram
before
you
in
essence,
have
three
tenants
talking
to
another
tenant.
A
What
tenants
are
you
supposed
to
basically
make
those
those
service
IPS
routable
from
that's
the
space
where
things
start
to
break
down,
and
so
we
got
together
basically
tried
to
figure
out
how
we
can
make
this
all
possible
using
Kata
under
the
hood
and
that's
what
we're
going
to
dive
into
so
again
this
that
architecture.
Imagine
those
three
tenants:
those
service,
those
service,
cluster
IP
ranges
that
are
in
those
API
server
Flags.
A
At
the
same
time,
when
that
pod
gets
created
in
the
API
server,
we
actually
have
an
instance
of
cube
proxy
running.
Alongside
that,
this
is
in
a
cotta
container.
That's
considered
a
privilege
caught
a
container
for
the
control
plane
and
it
has
a
one
single
side
car
along
with
it,
so
we're
not
making
any
changes
to
any
of
that
core
code
base
and
that's
called
the
exporter
and
the
exporter
sits
there
and
it
says:
hey
I
got
an
update
on
my
IP
tables.
I
need
to
do
something.
A
Now,
that's
jumping
ahead
a
little
bit
because
at
this
point,
virtual
cluster,
like
the
architecture
that
I
was
showing
before,
is
actually
syncing,
that
pod
object
down
into
the
API
server
and
it's
doing
its
normal
kubernetes
events
hitting
the
cubelet
or
the
cube.
That's
getting
a
notification
saying:
oh
I
got
to
schedule
this
workload.
It
hits
container
D
or
cryo.
It
schedules
the
Pod
and
it
creates
that
entire
space
in
this
world
now
we're
showing
akata
workload
running
there
rather
than
something
like
run
C.
A
So
going
back
to
cube
proxy
over
here,
we
I'm
going
to
use
the
fun
things
there
Cube
proxy
over
here,
we're
actually
going
to
get
an
update
on
that
and
we're
going
to
push
iptables
rules
back
into
the
API
server.
Now.
Some
of
this
is
a
little
bit
is
not
the
most
scalable
architecture
as
it
currently
stands,
and
we'll
talk
about
that
in
the
in
a
couple
slides
later,
but
right
now,
what
we're
doing
is
basically
taking
those
iptables
syncing
them
up
to
the
tenant
control.
Plane,
saying
hey
this!
A
This
tenant
has
this
set
of
Ip
tables
and
I
want
you
to
apply
that
to
every
single
workload
that
it
has
now.
I
talked
about
this
VN
agent
before
being
an
on
node
agent.
That
basically
takes
in
those
those
IP
tables,
and
it
says
Hey
katashem
set
these,
because
this
is
that
same
tenant.
So
now
we
have
per
tenant
iptable
rules
that
are
getting
pushed
to
every
single
workload
that
it
goes
alongside.
A
So,
if
you
repeat
this
entire
process
you're
pushing
a
new
set
of
rules
into
those
into
those
Kata
containers,
so
that
it
has
that
new
space-
and
this
creates
a
routable
service.
Cluster
IP
range
I
forgot
one
last
slide,
which
is
it
pushing
through
the
kadashim
to
the
Cod
agent
and
pushing
the
rules
in
yeah
and
at
that
point
yeah.
A
If
you're
in
one
of
these
tenants
we'll
show
you
a
little
bit
deeper
dive
of
that,
so
imagine
now
we
have
two
nodes
back
in
that
example,
and
we
have
two
different
Kata
workloads
running
one's
at
ten
zero,
zero
one
and
ten
one
zero
one
and
those
are
on
some
random
node
IPS
that
we
don't
really
care
about
in
this
one.
So
you're
coming
from
that
one,
where
we're
doing
that
curl
to
the
Food
Service.
This
is
one
of
those
things
that
gets
pushed
into
the
iPad
tables
rules.
A
So
quick
demo
of
how
this
all
functions
and
then
I'll
come
back
to
that
slide.
Just
to
give
you
a
little
bit
of
a
refresher-
and
this
is
all
recorded
because
I
am
not
going
to
do
this
live
so
this
is
going
to
go
through
and
do
exactly
what
we're
talking
about.
I'll
show
you
the
deployments
here
just
to
show
what
a
cluster
is
like
to
create.
So
if
we
actually
look
at
this,
this
is
what
a
virtual
cluster
spec
looks
like
am
I
doing,
time-wise.
Okay!
A
A
I
think
that
started
cool.
So
this
is
now
going
to
go
and
it's
going
to
go
VC.
It's
going
to
use
a
cube,
CTL
virtual
cluster
plug-in
that
allows
it
to
actually
write
out
a
cubeconfig
for
this
tenant
control
plane.
Specifically,
so
it's
going
to
bring
up
an
NCD
instance.
Remember,
I
was
telling
you
before.
It
only
brings
up
a
couple
components,
so
it
brings
up
the
the
NCD
instance,
the
API
server
and
controller
manager.
A
A
Yeah
I
guess
I
should
probably
speed
this
up,
but
anyways
we're
going
to
now
show
you
what
that
virtual
cluster
looks
like.
So,
if
you
Cube
cuddle,
get
all
Dash
a
so
all
namespaces.
This
is
all
you
see.
I'm
gonna
do
one
more
time
where
I'm
actually
hitting
the
super
cluster.
Just
to
give
you
the
difference.
So
we're
talking
to
a
tenant
control,
plane
running
in
the
cluster
that
I'm
now
hitting
which
has
all
of
these
resources
and
there's
a
bunch
of
pieces
here
that
are
that
are
important
that
I'll
show
later.
A
But
in
essence,
we
now
have
a
couple
new
name
spaces,
so
that
default
Dash
with
a
hash
VC
sample
one,
that's
the
cluster
name
and
the
prefixed
namespaces,
and
this
is
how
we
can
basically
create
namespaces
on
behalf
of
users
in
a
lower
level
super
cluster
and
isolate
them.
So
we
prefix
all
those
namespaces.
If
you
have
long
name
spaces.
This
is
always
a
question
that
gets
asked.
If
you
have
long
namespaces
it
truncates
it
for
you,
but
still
makes
it
unique.
So
we're
going
to
clear
that
up.
A
So
we
get
it
back
up
to
the
top
and
now
we're
going
to
go
and
we're
going
to
deploy
something
into
that
cluster
first
thing:
we're
going
to
do
is
deploy
core
DNS
because
who
needs
a
cluster
if
you
don't
have
DNS
in
there
and
what's
important
about
this?
Is
those
namespace
prefixes
that
I
just
showed
those
are
actual
parts
of
the
fqdn
when
you're
in
a
cluster?
And
so
we
need
this,
because
we
configure
this
coordiness
instance
as
the
name
server
on
every
single
pod
that
comes
from
your
tenant
control
plane.
A
So
everybody
has
their
own
isolated,
DNS
servers
you
can
set
that
to
resolve
to
whatever
behind
the
scenes.
You
want
again
I'm
going
to
show
you
the
difference
in
clusters.
So
what's
cool
here
is
I'm
going
to
highlight
that
there's
replica
sets
and
deployments,
but
nothing
from
that
tenant,
and
this
is
why
we
deploy
a
controller
manager,
because
we
only
care
about
pod,
scheduleable
resources.
We
can
take
that
and
say
once
the
controller
manager
and
the
Tenant
scheduled
it
or
created
a
pod
for
it
we'll
take
that
and
push
it
into
the
cluster.
A
So
next
thing
we're
going
to
do
is
we're
going
to
deploy
a
Kata
workload,
and
this
actually
isn't
calling
out
Kata
in
here
from
a
runtime
class
standpoint,
because
what
we've
done
is
to
make
it
extra
secure.
You
don't
even
have
to
set
a
runtime
class.
We
inject
that
automatically.
You
deploy
a
pod
like
a
normal
kubernetes
deployment
and
it
ends
up
in
Kata
behind
the
scenes
and
what
I'm
deploying
here
is
two
different
applications:
PHP
Apache,
because
we
all
love,
PHP,
right,
right,
cool
and
a
networking
pod.
A
C
A
Now
we're
going
to
Speed
it
I,
wanna
I,
think
I
could
probably
speed
this
up
a
little
bit
so
now
we're
gonna
go,
get
pods
and
services
in
that
tenant
control
plane
show
that
those
things
were
created
and
show
the
service
cluster
IPS.
So
you'll
notice
here
that
we
have
10
32.01
for
the
kubernetes
service,
10
32
0106,
that's
allocated
from
the
tenant
control
plane
on
the
services
and
then
those
IPS
for
the
pods
are
that
192.168
range
that's
exposed
by
the
super
cluster.
A
So
in
this
world
right
now
we
have
routable
pod
ranges
and
we
expect
things
like
cnis
to
do
zero
trust
out
of
the
box
at
that
same
point
now,
you'll
see
in
the
tenant
control
plane
same
eyepiece
for
the
pods
but
different
IPS
for
the
actual
super
cluster
services.
So
these
are
here
but
they're
non-routable
from
the
the
tenant
standpoint.
A
Now,
if
we
go
and
say
exec
into
this
pod
and
we
hit
iptable,
save
just
to
show
what's
happening
and
I'm
going
to
use
default,
kubernetes
just
to
show
what's
actually
happening.
So
this
wrote
a
rule
into
that
Kata
container.
That
is
that
103201
range,
not
the
Super
clusters
range
with
the
comments
about
that
Cube
proxy
normally
injects
and
I'll
do
a
couple
more
tests
in
here
just
to
show
show
how
things
function.
A
No,
it
doesn't
ever
hit
any
of
those
services
that
are
exposed
at
the
lower
level
and
then
we're
going
to
try
one
more
thing:
we're
going
to
grab
the
service
cluster
IP
of
that
kubernetes
service.
Do
the
exact
same
request,
slash
to
that
Health
endpoint
again,
this
is
not
routable
in
the
super
cluster.
If
I
went
to
one
of
those
nodes
and
tried
to
curl
this,
it
wouldn't
work
and
I'm
getting
that
Health.
Okay,
endpoint
I'm
gonna
do
a
couple
more
tests
in
here
because
it
gets
fun.
C
A
And
okay,
bang,
sweet
okay,
so
now
we're
basically
are
showing
that
this
is
now
a
new
routable
range
for
all
these
pods
we're
gonna
do
one
more
test,
which
is
again
that
IP
address
that
we
had
above
so
we
showed
this
IP
address
before,
but
this
is
that
test
networking
pod
one
more
time
getting
an
okay,
Bang,
Yeah
sweet.
So
now
we're
actually
able
to
hit
the
pods
directly
if
we
want
to
and
then
we're
going
to
do.
A
One
other
thing
which
is
I,
think
an
interesting
layering
of
the
architectures
here
so
now,
I
can
take
this
load
balancer
service.
So
if
you
noticed
before,
when
I
was
talking
about
that
virtual
cluster,
this
is
on
on
AWS.
We
have
the
cloud
controller
manager
deployed
for
AWS
configured,
but
because
we
do
dual
Services.
When
we
actually
deploy
this,
we
can
leverage
the
platform
to
actually
expose
things
at
a
platform
level.
A
So
if
you
wanted
to
do
a
platform
level
Ingress,
for
example,
you
can
do
that
kind
of
function
or
in
this
one
a
type
load.
Balancer
that's
going
to
go
and
actually
create
an
EOB
and
attach
it
to
the
same
search
to
the
same
pods
that
are
in
the
cluster
and
be
routable
as
a
a
platform
level
load
balancer
rather
than
having
each
tenant
need
to
own
those
type
of
tools.
So
you
can
layer
this
and
build
build
pieces
that
are,
in
essence
invisible
to
the
user,
but
act
kind
of
like
magic.
A
A
This
is
showing
that
entire
same
structure
so
behind
the
scenes
and
when
you
deploy
any
of
those
pods
or
any
of
those
services,
this
entire
flow
is,
is
happening
behind
the
scenes
for
you
pushing
those
iptables
rules
into
the
Kata
containers
and
making
them
all
routable
now
I'm
talking
about
the
feature-
and
this
is
where
I
was
saying-
that
this
isn't
the
most
scalable
architecture.
If
you
remember
116
days
with
things
like
iptables
the
amount
of
the
amount
of
network
traffic
that
goes
through
each
time.
One
of
these
things
is
updating.
A
A
That
is
still
a
cap,
but
it's
called
kaping
and
kaping
is
at
sigs.ks.io
kpng,
and
this
is
the
new
node
group
proxy
or
an
NG
proxy,
and
this
allows
you
to
actually
set
up
a
network
control
plane
which
every
single
workload
or
every
single
node
at
that
point
would
then
have
a
network
agent
rather
than
using
something
like
the
VN
agent
or
or
any
of
the
cnis.
It
then
gets
all
or
sorry
Cube
proxy.
What
it's
going
to
do
is
it's
going
to
actually
have
a
grpc
connection
back
to
that
Network
control
plane.
A
So
our
long-term
plans
are
to
integrate
with
this
instead,
because
we
can
actually
do
really
interesting
things
like
pushing
single
rules
through,
because
that's
a
piece
that
they
in
they've
started
to
implement
in
in
the
kaping
world,
rather
than
pushing
the
entire
IP
rules
set,
there's
also
a
bunch
of
syncs
that
they're
calling
it
behind
the
scenes
on
the
Node
on
the
Node
agent
side,
where
you
can
do
push
it
straight
into
ebpf
and
all
of
that
stuff.
It's
a
really
really
cool
project.
A
D
A
Yep,
so
they
could
actually
technically
run
as
long
as
they
have
the
Privileges
to
be
able
to
access
that
guest
OS.
They
could
technically
run
a
service
mesh
in
there
and
do
some
of
those
same
functions.
The
pieces
where
service
meshes
don't
allow
us
to
do
the
same
type
of
function
is
the
way
that
the
informers
work
for
those
types
of
interactions
and
are
set
up
to
talk
to
a
kubernetes
API
server
out
of
the
box,
they're
going
to
talk
to
the
super
cluster,
and
so
you
have
to
break
that
down.
A
E
So
the
second
part
of
the
talk
is
pretty
much
about
the
techniques
that
you
use
to
isolate
the
the
workloads
in
that
big
data
plane
right,
but
that's
only
a
problem
that
you
need
to
solve,
because
you
were
with
the
v-cluster
approach
that
has
the
supercluster
with
the
massive
data
playing.
So
you
could
have
done
something
different.
Like
you
mentioned
a
few
options
at
the
beginning,
maybe
have
some
hosted
control
planes,
manage
cluster
and
then
have
completely
separate
and
dedicated
data
planes
for
each
of
them,
like
whatever.
E
A
Absolutely
so
why
virtual
cluster
is
a
a
good
use
case
to
actually
leverage
control.
Planes
can
be
expensive,
especially
if
you
live
in
a
in
a
on-premise
world,
where
you're
deploying
into
bare
metal
host
yeah,
it's
a
very
expensive
thing
and
so
being
able
to
layer.
This
architecture
allows
you
to
build
massive
clusters
that
you
support
at
that
level
and
then
smaller,
more
single
purpose
control
planes
that
you
can
easily
isolate.
A
F
Cool
hey
two
questions,
starting
with
the
V
cluster,
so
once
you
set
up
V
cluster,
you
know
we
think
all
your
problems
are
solved.
You
can
give
it
to
your
customers
and,
like
hey,
you
have
basically
your
own
kubernetes
cluster.
Do
whatever
you
want
two
minutes
later:
they're
gonna
go
and
try
to
deploy
for
me
if
you
stack
or
the
Nvidia
dcgm
exporter,
or
something
like
that.
That
requires
a
Daemon
set
with
host
access
and
they're
going
to
be
really
annoyed
and
said:
hey.
F
You
didn't
give
me
a
true
true,
kubernetes
cluster
I,
wonder
if
both
with
and
without
cattle
containers,
you
have
been
doing
any
work
around
that
like.
If
you
say
that
you
run
with
cattle
containers,
then
you
really
would
need
to
schedule
that
Pub
inside
the
same
container
as
the
workload
is
running,
to
be
able
to
extract
metrics
for
that
workload.
The
same
goes
with
like
log
collection
fluent
bit
and
that
type
of
stuff.
B
A
A
C
A
Of
stuff
yeah,
so
out
of
the
box,
which
you
you
get
access
to
logs
through
normal,
like
through
that
VN
agent,
it's
basically
a
proxy
that
allows
you
to
get
to
the
to
the
actual
logs
on
that
on
an
individual
container.
It
injects
the
prefixes,
which
is
what
keeps
it
so
that
you
can't
talk
to
another
tenant's
workloads
for
running
things
like
a
Daemon
set
that
has
a
logging
agent
out
of
the
box
into
it.
That's
something
that
we
don't
have
yet
yeah,
okay,.
F
Yeah
I
mean
you
can
run
your
own
logging
agent.
Then
yeah
collect
that
on
their
behalf
and
do
some
multi-tenancy
there
I
guess,
which
is
the
same
thing
that
we
do.
The
next
question
is
around
performance
and
cattle
containers.
Yeah
no
I
haven't
looked
at
cattle
containers
in
a
while,
but
both
both
in
terms
of
kind
of
GPU,
pass-through
and
storage.
It
looked
like
it
would
have
a
lot
of
overhead,
like
the
cattle
container
way
of
doing
storage
password
with
vertile
FS
seemed
very
unoptimized,
at
least
a
while
back.
B
F
B
B
F
B
F
G
A
In
This
World
Behind,
the
Scenes.
These
are
actually
being
deployed
into
cotton
containers,
we're
just
not
talking
about
that.
Okay,
that
piece
under
the
hood,
because
those
are
being
deployed
into
one
or
many
of
those
super
clusters
that
are
configured
as
like
the
default
runtime
classes
being
set
to
microvm.
It's
just
automatically
deploying
we're
just
not
calling
it
out
in
this
slide.
So,
okay.