►
Description
As the rapid transition to remote work happened, VFX studios and designers used to beefy workstations, on-site storage clusters and high performance networking have had to scramble to make those resources available to people at home.
This presentation details how a VFX studio with 60 designers transitioned from a fully on-prem environment to complete cloud workflow. Combining KubeVirt powered Virtual Workstations with render nodes and storage running natively in Kubernetes provided a solution that beat expectations. Being able to manage all components via the same Kubernetes API allowed for a quick integration into existing systems.
We will be discussing our experience integrating KubeVirt under a strict deadline while leveraging bleeding edge features such as Virtio-FS.
Presenter: Peter Salanki, Director of Engineering, CoreWeve, Inc, @salanki (Twitter / Github)
A
We
we
focus
primarily
on
very
compute
and
tested
workloads
and
and
specifically
everything
that
leverages
gpus,
as
our
customers
are
people,
doing
rendering
molecular
dynamics,
neural
net
training
and
also
real-time
type
workloads
with
nor
net
inference.
Video
transcoding
game
streaming
and
virtual
desktops
and
virtual
desktops
and
also
game
streaming
is,
is
where
we
have
started
using
kuber
to
provide
you
know
very
integrated
platform
into
our
our
community
stack.
A
I'm
gonna
talk
a
lot
more
about
about
that.
In
a
little
bit.
Our
client
size
is
ranges
from
anywhere
to
50
to
4000
gpus
per
client
or
around
2000
cp
cores.
So
we
target
kind
of
mid
to
large
consumers
of
compute
as
our
clients,
our
primary
offering
and
the
only
kind
of
offering
we
had
up
until
we
deployed
kubwork
was
kubernetes
environmental.
So
we
run
large
multi-tenant
clusters
directly
on
bare
metal
on
linux,
and
this
allows
us
to
do
really
fast
spin
up
of
customer
workloads.
A
So
we
kind
of
you
know,
put
gpus
everywhere
and
we'll
talk
more
about
that
in
a
second
as
well.
So
for
this
presentation,
I'm
going
to
talk
to
what
one
of
our
large
customer
segments,
which
is
visual
effects
studios.
So
these
are
kind
of
3d
designers
and
artists
and
video
editing,
artists
that
you
know
make
all
the
cool
explosions
and
stuff
you
see
in
all
their
movies
and
tv
shows.
A
A
There
are
people
who
scale
up
to
aws,
but
that
becomes
really
expensive,
with
s3
costs
and
egress
fees
and
also
the
spin-up
times
I
mean
instantiating,
vms
and
so
on.
Aws
aren't
great,
there's,
also
shift
kind
of
in
the
rendering
industry
where
gpus
are
becoming
more
and
more
prevalent
than
than
just
cpus.
A
So
this
all
changed
about
a
year
ago,
a
little
bit
less
than
a
year
ago.
I
guess
when,
when
you
know
kobe
started
becoming
a
thing
all
of
a
sudden.
You
have
all
these
companies
that
have
so
much
compute
locally
needing
to
work.
Remote
and
working
remote
is
not
not
super
easy.
When
you're
talking
about
this,
this
much
data
and
this
much
compute.
So
what
a
lot
of
people
did
is
they?
You
know
they
did
remote,
so
software
to
their
on-premise
workstations.
You
know.
A
A
Other
people
moved
some
of
their
workloads
and
some
other
virtual
virtual
desktops
to
aws
or
gcp
to
be
able
to
scale
them
moving.
Your
entire
production
pipeline
to
to
aws
is
pretty
cost
prohibitive,
and
it
also
has
a
lot
of
performance
bottlenecks
with
around
how
s3
works
and
has
free
transfer
speeds
and
so
on,
so
that
they
kind
of
common
design.
Is
you
run
your
virtual
desktops
on
amazon
and
then
you
connect
over
nfs
to
your
on-premise
storage
and-
and
you
know
it's-
it's-
not
performance
optimal,
to
say
the
least.
A
So
we
had
some
of
our
clients
and
come
to
us
this
summer
and
say:
hey
can't.
We
just
move
everything
to
you
guys,
but
do
we
need
ever?
You
need
to
have
a
virtual
desktop
infrastructure
for
us,
where
we
can.
You
know,
access
remotely
systems
that
have
powerful
gpus
in
them
powerful,
cpus
and
are
directly
connected
to
our
storage,
and
that's
that's
when
we
we
kind
of
found
kubevirt
and
in
the
end,
designed
a
solution
that
looks
kind
of
like
this,
so
everything
runs
in
the
same
kubernetes
cluster.
A
A
This
was
a
bit
interesting
because
vertifs
was
very
new
when
we
started
doing
this,
but
once
we
got
everything
working,
that
performance
is
really
good.
We
run
tens
of
thousands
of
iops
over
virtio
fs,
and
this
allows
the
artists
to
work
directly
against
the
the
shared
file
system,
the
set
fs,
and
that
also
the
render
pods
connect
to
directly
to
actually
render
the
jobs.
A
A
So
why
did
we
choose
kubert
I've?
I've
worked
with
openstack
in
the
past
and
when
I
got
got
the
the
request
to
build
to
build
a
bm
infrastructure.
My
first
thought
was
that,
oh
god,
we
gotta
deploy
openstack
and
not
not
only
do
I
think
that
openstack
is
extremely
complex.
It's
also
would
be
another
kind
of
complex
stack
to
run
in
parallel
with
already
a
pretty
complex
community.
A
We
also
sat
down
and
looked
at
all.
The
integrations
we'd
have
to
do
and
basically
rebuild
a
lot
of
what
we
already
built
for
kubernetes
and
how
it
handles
storage
and
even
worse,
how
to
get
these
two
to
interact.
If
we
run,
you
know
the
bare
metal
render
jobs
in
kubernetes
but
ran
the
virtual
machines,
the
virtual
desktops
in
openstack.
A
How
would
they
communicate
with
each
other?
How
would
orchestration
work
and-
and
you
know,
would
look
to
be
a
lot
of
development
work,
and
so
then
we
started
researching
alternative
solutions
and
kind
of
came
across
kubert
I'd
seen
kubrick
earlier,
but
didn't
really
realize
how
mature
kubward
had
gotten.
A
So
we
started
development
on
this
this
summer
and
and
basically
had
a
proof
of
concept
product
done
in
two
months,
so
say
that
development
really
started
for
a
production
product
in
october,
and
we
have
migrated
the
customer
fully
from
their
on-premise
setup,
plus
aws
to
us
in
december.
So,
by
leveraging
kubert,
we
could
just
shave
off
so
much
development
time
and
and
also
provide
a
much
better
user
experience
for
the
user.
We
get
the
you
know
the
seamless
integration
between
very
bare
metal
and
virtualized
workloads.
A
We
talked
about
sharing
storage,
with
with
the
persistent
volume
claims
and
and
vert
iofs
that
just
kind
of
works
out
of
the
box,
and
you
know
users
just
need
to
manage
the
storage
in
one
place,
which
is
the
persistent
volume
claims
there
is
the
same.
We
can
use
the
same
tools,
cube
ctl
and
apis
to
manage
both
vms
and
manage
manageable
vms
and
manage
them.
A
The
pods-
and
you
know
we
can
manage
our
internal
cooperative
infrastructure
like
we
manage
anything
else,
we
do
through
githubs
using
argo
cd
network
policies
using
our
calico
cni.
Just
works
applies
to
the
vms.
Don't
have
to
do
anything
special
there.
We
do
billing
by
looking
at
resource
consumption
for
the
kubernetes
for
prometheus
and
communities,
metrics
that
also
just
worked
out
with
box.
A
The
customers
love
it.
We
have,
I
think,
70
people
as
we
speak,
running
cougar
vdis
with
anything
between
one
and
four
gpus
in
them,
which
is
pretty
neat.
So
we
also
had
to
build
some
some
stuff
around
cooper
to
make
it
a
bit
easier
digestible.
So
we
use
a
tool
from
bitnami
called
kubaps
to
kind
of
give
a
visual
way
to
deploy
home
charge.
A
So
we
package
up
all
of
our
different
applications
like
the
render
software
deadline,
in-home
charts
and
give
users
a
user
interface
for
goog
apps
to
deploy
these
themselves.
So
we
did
the
same
thing
for
our
our
coop
vdis.
We
built
a
hamster
for
it
and
built
a
custom
ui
component
to
deploy
this
this.
This
gave
the
users
a
graphical
interface
to
be
able
to
deploy
a
vm
select.
What
region
they
want
on
that?
What
os
image,
what
gpu,
how
much
cpu
and
so
on,
and
then
this
in
the
end
just
generates
there.
A
The
virtual
machine
manifests
it
generates.
The
service
manifests
and
applies
a
bunch
of
different
some
mapping
tables.
You
know
which,
which
os
image
and
version.
Should
you
run
what
node
type
should
this
run
on?
It
turned
into
kind
of
a
franken
chart.
It
is
way
too
too
complex
for
what
helm
was
really
intended
for
so
what
we
deliver.
A
What
we
developed
after
that
was
that
what
we
call
the
virtual
server
operator,
it
should
really
be
virtual
machine
operator,
but
you
know
the
conflicts
with
a
with
a
convert
virtual
machine
name.
So
we
have
to
pick
something
else,
even
though
I
think
virtual
server
sounds
pretty
ugly,
so
this
is
just
an
abstraction
over
both
the
virtual
machine
crd
and
the
server's
crd,
and
also
pu
data
volume,
crd
where
users
can,
with
once
with
one
single
yama,
manifest
create
them.
A
A
Machines,
we
also
had
to
do
a
bunch
of
modifications
to
kubrick
and
cdi.
We
were
very
early
using
virtio
fs.
I
think
the
vertifs
was
merged
by
vladik
about
two
weeks
before
we
started
using
it
and-
and
I
don't
believe,
there's
a
lot
of
other
people
in
us
who
have
actually
tried
to
use
it
at
any
scale.
We
found
a
bunch
of
interesting
bugs
that
we
tried
to
supply
bug,
fixes
for.
A
A
And
and
most
of
them
have
been
merged
once
we
got
vertifs
working
properly,
we
have
had
really
good
good
experience
with
it,
even
though
it
definitely
had
some
quirks
in
them
in
the
beginning.
We
also
found
the
pr
for
cdi
to
be
able
to
do
cross
namespace
volume,
cloning
using
the
the
csi
csi
clone
api,
and
we
want
to
do
this
obviously,
to
spin
up
vms
really
quickly.
When,
when
you
know
our
customers
go
and
spin
up
a
new
virtual
desktop,
they
wanted
to
launch
within
30
seconds
and
waiting
for
data
to
root.
A
This
a
copy
is
not
really
not
really
feasible.
Cdi
didn't
have
any
cross
namespace
functionality
built
in,
and
you
know
the
way
we
do
it
is
we
keep
one
namespace
with
our
kind
of
base
virtual
desktop
images
with
you
know
we
sent
os
and
ubuntu
and
whatever
we
want
people
to
be
able
to
clone
these
to
their
own
namespace.
A
So
we
filed
the
pr
for
that
that
we're
waiting
get
merged.
We
also
had
to
fork
the
nvidia
device
plugin,
because
the
default
nvidia
device
plugin
has
a
static
map
of
pci
ids,
which
is
a
pain
to
maintain,
and
it
also
uses
different
resource
labels
than
the
regular
nvidia
device
plug-in
for
kubernetes.
And
since
we
mix
kubernetes
native
or
non-virtualized
workloads
together
with
kubert,
we
want
the
resource
labels
for
gpus
to
be
the
same
across
both
for
kind
of
documentation,
ease
of
use
and
billing
purposes.
A
How
has
our
experience
been
contributing
to
kubert?
It's
been
pretty
easy
getting
into
the
code.
To
be
honest,
whenever
we
find
issues,
it's
very,
very
quick
for
us
to
begin
figure
out
where
we
need
to
change
stuff
and
and
make
the
changes
compiling
it.
You
know
that
the
build
scripts
are
very
easy,
very
easy
to
push
your
own
docker
images
and-
and
just
you
know,
update
your
coupe
convert
operator
manifests.
A
We
have
had
a
bit
more
challenges
with
getting
our
pull
requests
merged.
The
build
pipeline
often
fails
and-
and
you
know,
can
have
to
assume
that
it's
nothing.
We
did
it's
sometimes
hard
getting
feedback
on
the
pull
requests
from
from
the
maintainers
vladik
has
been
really
responsive,
but
we
have
had
a
bit
more
trouble
on
the
cdi
part
where
I
think
that
the
merging
our
cross
namespace
volume
cloning
feature
will
be
helpful
for
a
lot
of
people,
but
right
now
we're
we
can
of
course
run
our
own
forks
forks
of
everything.
A
B
C
A
Yeah,
I'm
not
sure
I
like
developing
in
cooper
or
outside
of
cooper,
so
let's
yeah
I
mean
in
general,
you
know
it.
It
was
good.
I
think
that
that
the
fact
that
it
was
a
as
an
nvidia
device
plug-in
is
kind
of
what
led
us
down
this
path.
Otherwise
we
probably
wouldn't
have
tried
to
do
it
ourselves,
but
you
know
getting
the
gpu
support.
Working
was
really
a
non-issue.
We
you
know
they
there
are.
There
are
some
quirks
depending
on
gpu
types
and
so
on.
A
A
A
And
yeah
bloody
guy,
I
know
about
the
generic
device
plugins
and
you
know
we
might,
we
might
slash-
will
migrate
to
that
in
the
future.
We
just
you
know
the
the
issue
around
pci
ids.
Is
there
as
well.
We
have
an
easy
way
to
manage
that.
We
have
a
lot
of
different
gpu
types
and
we
we
we
buy
all
the
new
gpus
that
nvidia
released
we're
navigating
a
cloud
service,
part
provider
partner.
A
A
Yeah
so
that
you
know
also
the
kubrick
clusters:
they're
they're,
not
specific
clusters,
so
we
run
in
the
beginning.
We
run
really
large
multi-tenant
clusters,
so
each
cluster
will
have
about
1000
nodes
and
then
we
we
change
these
nodes
from
being
kind
of
hypervisor
nodes,
which
really
means
that
we
reboot
them
and
pci
pass
through
all
the
gpus
or
their
regular
kubernetes
nodes.
And
then
we
split
that
back
and
forth
based
on
demand.
A
So
if
there's
a
lot
of
demand
for
virtual
machines,
we
we
have
a
system
based
on
a
cluster,
auto
scaler
that
basically
reboots
the
nodes
and
they
boot
up
with
a
new
flag
and
then
passes
through
all
the
gps.
Right
now
I
think
we
have
around
150
the
vms
total
running
with
at
least
one
gpu
each
and
that's
spread
out
over.
A
Yeah,
so
it's
gonna
be
either
or
since
you
know
we
do
huge
one
gig
huge
pages
for
performance,
so
it's
just
try.
I
mean
I
looked
into.
We
looked
into
doing
it
dynamically
and
kind
of
slice
and
dice
things
back
and
forth,
but
it's
not
really
worth
it.
We're
just
gonna.
You
know
it's
either
gonna
be
kubrick
mode
when
everything
is
in
huge
paces
who
all
the
memories
are
located.
C
A
Have
no
no,
not
not
in
terms
of
number
of
two
root
notes.
You
know.
That's
that's
also,
one
of
the
reasons
that
I
like
uber,
there's
a
you
know
piggybacks
as
much
as
you
guys
can
on
the
kubernetes
native
infrastructure.
So
I
kind
of
I
know
what
I
get
so
you
know
it
scales
as
well
as
any
other
pod.
You
know,
there's
obviously
some
traffic
to
your
api
servers
and
so
on,
but
it's
so
light
that
no
I
mean
no.
C
B
C
A
Yeah
significantly-
and
you
know
it
depends-
depends
a
bit
on
what
you're,
booting
and
os.
But
you
know
the
pod
and
the
from
kind
of
vmi
start
to
the
pod
is
running
it's
10,
15
20
seconds
and
then
another
30
seconds.
On
top
of
that,
until
the
the
os
is
booted,
so
you
know
va,
we
can
have
a
vdi
up
in
in
30
seconds.
Assuming
that
you
know
all
the
vert
launchers
are
cached
and
so
on.
That's.
C
B
Yeah
there
is
a
question
following
you
know.
Actually
there
is
a
comment
before
the
question
by
vladik
back
to
the
topic
of
of
device
plugins.
I
think
the
comment
was
you
know:
vladik
had
a
session
yesterday
in
the
tbr
summit
about
this
topic
about
generic
device
plugins,
so
that
might
be
relevant
to
the
previous
comments.
So
that's
yeah.
A
It
will-
and
I
I
I
I
I
commented
on
that
bit
earlier-
that
we
will
definitely
look
into
migrating.
To
that.
You
know.
What's
important
to
us
is
that
we
can
mask
pci,
ids
and
so
on
from
the
user
to
provide
a
very
simple
user
experience
and
also
for
that
to
comply
into
how
we
do
billing.
So
we
have
a
lot
of
different
nvidia
gpus,
so
we
don't
really
want
to
have
to
like.
Have
a
bunch
of
you
know
a
bunch
of
pci
ids
flying
around
in
a
bunch
of
different
configurations.
B
A
B
A
Having
set
this
kind
of
the
optimal
solution-
and
you
know
we
use
the
the
kubernetes
csi
there
to
orchestrate
both
read,
write
many
volumes
using
cfs
and
the
volume
mode
block
block
volumes
for
the
root
root
drives
and
yeah.
It
works
great
interacts
very
well
with
cooper
interacts
well
with
cai
cdi
and
volume
cloning,
so
we
rely
heavily
on
on
kind
of
volume,
cloning
and
cdi
to
to
clone
root
storage
from
uvms
and
doing
that
with
zero
coffee
and
really
quickly.
A
You
want
me
to
answer
that.
Then
there
was
this
question
about
bgpu,
which
I
think
was
pretty
elegant.
So
we
we
as
a
company
we
stay
away
from
the
gpus,
both
because
yeah
for
a
bunch
of
reasons.
We
want
to
give
the
users
full
access
to
gpu,
so
they
can
do
whatever
they
want
with
it.
Also
from
a
stability
point
of
view,
we
just
don't
have
a
need
for
for
pgp
functionality,
and
our
users
traditionally
won't
want
to
have
all
the
performance
of
the
gpus
they
get.
A
We
try
to
stock
a
lot
of
different
gpus,
so
we
have
everything
from
kind
of
rtx
4000
through
vu,
100s
to
a100s
and
a6000s,
and-
and
we
just
do
pci
passthrough,
which
is
an
easier
easier
design
and
our
users
like
it.
So
that's
why
we
kind
of
just
have
architecture
to
stay
away
from
bgpu,
and
even
if
we
did
that,
we
would
still
have
to
deal
with
a
huge
page
allocation
for
the
memory,
which
is
not
super
easy
to
allocate
and
de-allocate
one
gigahertz
pieces
at
runtime.
A
So
just
you
know
dedicating
nodes
to
be
either
covered
or
not.
It's
just
a
simpler
architecture.
For
us
and
then
the
final
question
is:
will
it
run
crisis?
Yes,
it
will.
I
don't
have
any
performance
specs
for
you,
but
we
also
have
a
couple
of
of
career
test
machines
that
we
use
in
office
to
to
test
performance
on
on
different
games.
It's
you
know
the
two
hours
of
qa.
We
do
every
day,
I'm
available
to
slack
to
answer
any
questions.
Thank
you.
Everyone
for
for
listening
to
me.