►
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Deep Dive: KubeVirt BoF - Scott Collier, Red Hat & Ben Warren, Cisco
KubeVirt extends Kubernetes with the ability to orchestrate and run virtual machine workloads, as Kubernetes itself is not able to launch those. Building a cloud-native virtualization API which supportsexisting and new virtualized workloads, but at the same time also fits seamlessly into Kubernetes, turns out to work out quite fine most of the time. However, there are also areas where virtualizationrequirements are crossing the boundaries of Kubernetes existing features or cloud-native concepts. On the other hand crossing the boundaries, is an opportunity to work with the Kubernetes community to enable additional use-cases for both containers and virtual machines.In this BoF we’ll be taking a look at existing features but also at features which are currently driven by the community, and also features which are currently blocked. To
Learn More: https://sched.co/Ixs8
A
Welcome
everyone
to
the
Qbert
deep-dive
bird
of
a
feather
session,
just
a
quick
check
who
is
here
for
the
intro
session
earlier
today.
Very
good.
Welcome
back
I
won't
ask
the
same
questions
that
I
did
then
about
who's
running
kubernetes
and
and
who
has
existing
virtual
virtualization
infrastructure?
Is
there.
A
My
name
is
Scott
Collier,
again,
I
work
for
Red,
Hat
and
I
focus
on
basically
the
cnv.
Well,
the
Qbert
upstream
community
side
of
this,
and
we
have
David
Basel
here
from
Red
Hat
as
well
focuses
on
virtualization
Ben
Warren
from
Cisco
and
Adam
from
Red
Hat
as
well.
So
we're
gonna
take
a
little
bit
of
a
different
approach
to
this
one
than
we
did
the
earlier
session.
A
We
had
some
bad
time
management
and
we
just
didn't
leave
enough
time
for
questions
at
the
end.
So
on
this
one
we
have
a
lot
fewer
slides.
We
want
to
have
more
questions.
We
have
more
t-shirts
for
folks
who
ask
questions.
The
first
eight
folks
get
a
t-shirt,
and
we
want
to
really
make
this
an
engaging
session
here.
So
have
a
little
bit
of
conversation
about
architecture,
just
the
core,
virtualization
architecture,
the
storage
architecture
and
then
Ben's
going
to
cover
some
of
the
use
cases
they're
looking
at
at
Cisco
and
then
QA
time.
A
So,
just
for
those
who
weren't
here
in
the
previous
session
I'll
just
cover
quickly
what
Qbert
is,
and
it
basically
just
enables
kubernetes
to
manage
virtual
machines
in
a
common
shared
environment
alongside
your
application
containers,
okay,
it's
a
drop-in
add-on
by
CR,
DS
custom
resource
definitions.
To
give
you
a
similar,
look
and
feel
for
managing
your
applications
and
your
virtual
machines.
A
It's
not
all
the
way
there,
yet
there's
still
a
lot
of
work.
That
needs
to
be
done,
but,
as
you
can
see,
the
the
current
feature
set
looks
really
robust
and
tonight
there's
going
to
be
a
Qbert
gathering
where
we
are
gonna
have
a
chance
to
talk
about
the
roadmap
and
some
of
the
features
that
you
don't
see
here
now
that
we
need
to
look
at.
You
know
developing
in
prioritizing
over
the
short
and
midterm.
B
C
Hear
me:
oh
all
right
so
I'm
David,
bustle,
I
work
at
Red,
Hat,
I
guess
I
was
already
introduced.
Kubernetes
has
done
a
really
great
job
at
giving
us
extension
points.
Essentially,
let
us
develop
Cubert
as
if
it
was
just
part
of
the
kubernetes
core
API.
So
just
like
kubernetes
has
things
like
deployments
and
pods.
C
We
can
register
our
own
third
party
API.
Is
you
some
custard
resource
definitions?
That's
the
extension
point.
The
communities
provides
us
in
addition
to
that,
we're
also
using
things
like
mutation
and
validation,
web
hooks
and
even
an
aggregate,
aggregated
API
server,
and
all
these
things
together.
Let
us
create
an
API
that
really
maintains
the
exact
same,
look
and
feel
as
a
core
kubernetes
api.
So,
from
a
user
perspective,
it's
gonna,
look
really
familiar
to
people
and
people
they're
developing
and
want
develop
against.
C
Cubert
are
gonna
see
what
we're
doing
at
submit
make
a
lot
of
sense.
Addition
to
extension
points
we're
using
just
patterns
that
we
see
kubernetes
do
so
we're
leveraging
these
patterns
that
we've
already
laid
out
for
us.
One
of
those
is
this
concept
of
stacking
api's.
So
in
kubernetes
we
have
this
stacking.
Where
you
know
applies
at
the
bottom
of
the
stack,
you
have
a
replica
set,
the
staff
and
type
of
pods,
and
you
have
deployments
they're
stacked
on
top
of
replicas,
so
there's
stacking
of
deployments
they
create
replica
sets
which
then
create
pods.
C
We're
doing
the
same
thing
in
our
API
is
so
we
have
a
virtual
machine
instance
which
is
kind
of
our
equivalent
of
a
pod.
It
represents
an
instance
of
a
virtual
machine
and
then
layered
on
top
of
that
we
have
a
higher-level
api's
that
manage
things
like
virtual
machine
replication
and
stateful
persistent
virtual
machines.
You
can
turn
off
and
on
and
they
maintain
their
state.
So
the
whole
point
about
this
is
users
can
interact
with
Qbert
using
the
standard
tooling
that
they're
used
to
with
kubernetes,
and
it's
going
to
be
familiar
to
everyone.
C
Looking
at
the
back
end,
what
we
have
is
we
have
a
set
of
controllers
that
we
have
implemented
that
implement
our
API
s
and
coordinate
the
virtual
machine
launch
flow.
So
when
you
post
a
virtual
machine
instance
to
the
cluster,
our
controller
is
going
to
observe
the
creation
of
that
object
within
the
cluster
and
in
response
it's
going
to
generate
a
pod
for
that
virtual
machine
to
live
in
that
pod.
C
Those
resources
is
that
we've
attached
to
the
pod
to
the
virtual
machine
process,
that's
being
launched
within
the
pod
an
end
result
here
is
we
have
a
virtual
machine
process,
that's
kind
of
assumed
the
role
of
the
pod
and
it
really
just
kind
of
looks
like
the
pod
at
that
point
and
to
kind
of
drive
that
home
our
default
networking,
if
you
ping
the
pod
IP
address
for
the
virtual
machine
instance
you're
getting
a
response
from
the
virtual
machine
guest
and
not
the
pod,
exactly
so
from
a
10,000
foot
view.
That's
that's
our
architecture.
C
D
Yep,
it's
still
on
so,
basically
when,
when
we
get
to
storage,
we
want
to
continue
that
theme
of
a
native
kubernetes
experience
for
virtualization.
So
a
couple
of
design
decisions
that
we
made
early
on
when
we
thought
about
how
to
give
persistent
storage
to
virtual
machines
was
the
first
one
being
that,
when
you
think
of
a
VM
disk,
you
should
think
of
a
PV
C.
It's
a
really
handy,
API
point
within
kubernetes
that
we
can
use.
D
So
basically,
what
we
chose
to
do
is
that
a
single
PV
C
maps
to
a
single
virtual
machine
machine
disk.
So
if
you
want
two
disks
with
your
VM,
you
could
have
two
PVCs.
This
allows
us
to
take
advantage
of
all
of
the
work
that's
going
on
in
the
kubernetes
SIG's
storage,
to
provide
things
like
snapshots
and
cloning,
and
all
these
other
things
that
we'll
just
be
able
to
get
for
free
for
our
VM
disks,
another
thing,
and
so
basically
how
this
works
is
on
a
PV
C.
D
It
started
off
that
we
just
had
file
file
mode
PVCs.
In
that
case,
we
just
create
a
file
on
the
root
of
that
volume
that
uses
all
the
space
of
the
PV
C
in
order
to
put
the
virtual
machine
disk,
and
then
Cubert
knows
how
to
how
to
map
that
into
the
configuration
to
launch
the
VM
when
it
comes
to
block
mode
Peavey's,
which
is
something
that's
coming
up,
we
can
do
the
same
thing,
except
in
that
case
we
just
exposed
that
block
device
directly
to
the
virtual
machine
when
it
launches.
D
So
this
really
gives
us
a
nice
integration
with
the
base,
kubernetes
storage.
Another
design
decision
that
we
made
was
that
we
would
use
raw
image
format
instead
of
something
like
qk
2.
For
those
of
you
familiar
with
that
detail,
and
the
reason
is,
is
we
really
wanted
to
push
all
of
the
brains
and
the
smarts
of
the
storage
down
to
the
storage
providers?
And
we
want
cube
ver
2
work
with
any
storage
that
you
bring
to
the
table
so
whether
it
be
Gluster
or
local
storage
or
cloud
provider
storage?
D
It
doesn't
matter
because
we
abstract
at
the
PVC
layer.
So
this
means
we
won't
be
doing
software-defined
storage
things
like
implementing
the
nightmare
that
haunts
me
from
the
past.
You
know:
live
deletion
of
snapshots
using
cue
cow
to
image
and,
like
the
cue
mu
and
libvirt
api's,
we
assume
that
the
storage
will
take
care
of
all
that
for
you.
So
that's
basically
I
guess
an
overview
of
of
that.
D
Then
there
was
a
unique
problem
when
it
comes
to
persistent
storage
for
virtual
machines,
so
it
normally,
when
you
think
of
something
like
a
database,
the
database
or
the
application.
That's
using
the
storage
is
kind
of
generating
the
data
and
you
want
to
save
it
over
time,
but
with
a
virtual
machine,
you
kind
of
have
to
start
with
something
you
need
that
disk
and
it
you
might
want
to
have
an
operating
system
pre-loaded
on
that,
for
example.
D
So
then
the
obvious
question
comes:
how
do
we
get
that
data
into
the
PVC
before
we
hand
it
off
to
the
virtual
machine?
So
in
order
to
solve
this
problem,
we
also
wanted
to
use
a
general-purpose
solution
and
we
kind
of
thought
of
three
different
ways
that
you
could
get
data
into
that
PVC.
In
a
general-purpose
way,
one
would
be
what
we're
calling
import,
which
is
that
the
cluster
pulls
that
image
down
from
a
hosted
location
such
as
a
web
server
like
a
pre-installed
disk
image.
D
Another
one
would
be
that
you
would
want
to
push
that
or
upload
it
from
your
local
machine.
Let's
say:
you've
prepared
a
disk
image
and
then
you
want
to
push
it
up
into
the
cluster
into
a
PVC.
So
we
call
that
our
upload
method
and
then
the
third
way
would
be
to
actually
clone
between
and
IG's
from
an
existing
volume.
So,
for
example,
I
have
a
virtual
machine
template
with
a
prepared
disk
image
ready
to
go
and
I
just
want
to
copy
that
and
on
multiple
places.
D
Do
cloning
we're
participating
with
the
group
developing
the
snapshot
capability,
kind
of
pushing
these
things
into
the
CSI
spec,
and
also
things
like
moving
storage
between
namespaces?
You
might
want
to
prepare
golden
images
for
your
virtual
machine
disks
in
one
namespace
and
then
transfer
it
to
your
users
namespace.
So
a
few
different
things
like
that
and
I
think
that's
enough
talking,
because
if
you
have
more
questions,
we
can
answer
the
specific
ones
that
you
have
inside.
A
A
B
A
B
A
D
Yes,
we've
actually
yeah,
so
the
question
the
question
is:
have
we
looked
into
using
9p
to
export,
basically
a
file
system
directly
into
the
virtual
machine?
It's
something
that
we've
talked
about,
not
something
that
we've
pursued
directly
I
think
it
would
definitely
be
pretty
doable
as
the
architecture
should
allow
for
that
to
happen.
If
we,
if
there
is
interest
in
it.
E
Sure,
okay,
so
my
name
is
Ben
I
work
in
the
compute
distributed
storage
group
at
Cisco,
I'm
gonna
talk
a
little
bit
about
an
engineering
project.
This
is
something
that
doesn't
have
a
name
or
a
release
date
or
anything
like
that.
So
it's
it's
really.
Just
things
were
playing
around
with,
but
Cisco's
a
pretty
major
vendor
of
HCI.
E
Here's
a
diagram,
so
we
have
a
cluster,
a
compute
cluster.
That's
a
compute
and
storage
cluster
of
three
UCS
servers,
sharing
the
distributed
storage,
hyperflex
distributed
storage,
and
we
have
multiple
appliance
VMs
running
on
these
servers
and
we're
using
Qbert
for
the
VM
or
or
whatever
orchestration
I.
Guess
you
could
say,
really
simple
diagram.
Hopefully
it's
clear
feel
free
to
ask
questions
if
you
have
them,
and
so
we
found
this
to
be
a
pretty
good
system.
E
So
the
things
that
we're
important
to
us
is
we
need
to
have
very
configurable
networking
storage,
so
we're
using
CNI.
These
different
appliances
have
very
different
networking
needs,
as
you
could
imagine.
Generally.
Multiple
interfaces
is
a
biggie,
so
we've
been
using
Maltese
and,
and
then
the
various
you
know
either
a
bridge
or
any
of
the
CNI
plugins
that
are
available
out
there
to
to
provide
multiple
interfaces
to
our
VMs.
We're
also
using
si
si
si
si
plugin
for
our
hyper
flex
to
do
this
storage
plug-in
stuff
we're
we
have
a
lot.
E
So
one
of
the
things
that
one
of
the
reasons
we
really
like,
kubernetes,
not
not
just
because
it's
trendy
but
I,
mean
it-
has
the
the
idea
of
8
of
H
a
so
we
like
the
idea
of
if
a
VM
goes
down,
it
will
quickly
spin
up
somewhere
else
or
even
better.
It
will
memory
coherently,
move
over
without
glitching
on
another
node,
so
we
like
to
see
like
fast
fail
failure,
detection
within
kubernetes
and
Qbert
and
the
live
migration.
E
These
are
things
that
we're
looking
forward
to
you
know
these
things
are
getting
better
every
day
and
the
last
thing
that's
important
to
us
is
VM
configurability.
There's
there's
always
trade-offs
in
the
project
like
this,
where
you're
trying
to
you
know,
stick
with
the
the
kubernetes
way
of
doing
things
or
you're
trying
to
bring
us
our
iove
whatever
or
some
type
of
hardware
pass
through
into
your
vm.
E
And
how
are
you
going
to
do
that
and
so
we're
looking
at
ways
it's
important
to
us
to
be
able
to
change
pretty
much
anything
even
down
to
ki-moon
line.if,
because
libvirt
doesn't
even
necessarily
support
any
of
the
new
things
as
they
Akemi
I've.
Certainly,
I've
worked
on
some
cameo
features
that
never
made
their
way
into
libvirt,
but
you
know
I
might
want
to
exercise
them
so
I'd
like
to
be
able
to
do
that
with
Qbert
and
that's
all
I
have.
A
C
So
we
do
have
a
higher
level
controller
called
virtual
machine.
It
was
a
virtual
machine
instance.
Replica
set
I
believe
is
what
we've
called
it.
It
works
similar
to
a
replica
set
and
in
Cooper
Nettie,
since
gonna
allow
us
to
scale
out
as
many
virtual
machines
as
we
want.
We're
also
investigating
ways
to
reuse
the
things
like
deployments
and
replica
sets
within
Cooper
neighs
to
launch
virtual
machines
and
have
that
seamlessly
work
in
an
Qbert.
C
So
you
would
define
your
virtual
machine
instance
and
actually
have
a
deployment
somehow
meant
replicating
that
so
that's
kind
of
future
things
that
we're
looking
at
right.
Now
we
have
a
virtual
machine
instance
replica
set,
though
that
lets
us
do
a
lot
of
the
things
that
replica
set
does
with
pots.
A
D
Right
so
basically,
we
support
in
CDI
being
able
to
import
the
cue
cow
to
or
raw
images
that
can
be
compressed
if
in
the
case
of
raw.
So
if
you
had
a,
for
example,
a
library
of
in-house
images
that
you
wanted
to
produce
virtual
machines
from
one
easy
way
to
do
would
be
to
just
host
those
up
on
an
on
an
internal
web
server.
That's
you
know
local
to
the
cluster
or
quickly
accessible
via
the
cluster.
D
F
C
Like
clout,
you
turn
my
cloud
Annette
yeah,
so
yeah
you
can
do
that
in
line
in
the
virtual
machine
instance
back.
You
actually
define
the
cloud
init
data.
You
can
also
do
it
through
like
a
secret
or
something
like
that.
You
can
define
the
cloud
Annette
data
in
a
secret
and
then
map
it
to
the
virtual
machine
instance.
D
B
D
It
will
spawn
a
pod
that
attaches
to
the
PVC,
performs
the
work
it'll
annotate,
the
PVC
afterwards
to
indicate
the
status,
and
this
is
a
good
time
to
kind
of,
as
some
of
you
might
be
thinking,
how
do
you
synchronize
the
process
of
I,
want
this
data
to
be
in
that
virtual
or
in
that
PVC
prior
to
the
VM
starting?
Because
if
you
just
declare
this
stuff
together,
the
ordering
is
not
guaranteed.
So
for
that
we
actually
have
something
called
data
volume
and
it's
an
abstraction
on
top
of
the
PVC.
D
But
everything
is
still
using
the
PVCs
and
what
happens
is
cube
fir.
It
integrates
with
a
data
volume
object,
you
create
this,
and
the
data
volume
controller
controller
will
actually
create
the
PVC
and
managed
the
preparation
of
it
and
Qbert
will
not
start
the
virtual
machine
until
the
process
is
completed.
So
it's
all
managed
in
a
declarative,
orchestrated
fashion.
C
That's
a
lot,
so
he
templates
we
would
have
to
come
up
with
some
I.
Don't
know
how
a
whole
lot
about
heat
templates,
but
I
know
that
there
had
to
be
some
sort
of
mapping
of
what
you
define
a
heat
template
which
could
be
maybe
an
auto
scale
group
or
something
like
that
to
a
COO
Bernays
deployment
or
a
replica
set
of
some
sort.
We
could
investigate
something
like
that.
That
might
be
interesting.
You
know
we're
accepting
patches
if
you're
interested
and
prototyping
something
like
that
that'd
be
interesting.
C
It's
similar
to
what
you
would
expect
in
kubernetes.
So
if
you
have
a
deployment-
and
you
say
you
want
three
replicas
and
if
one
of
those
goes
down,
then
you
know
it's
going
to
bring
one
backup.
So
when
we
have
a
virtual
machine
instance,
replicas
say
yeah,
it
brings
it
brings
it
back
up
we're
also
looking
at
things.
Well,
it's
a
little
bit
different
and
that
we're
talking
about
mortal
or
closed
versus
immortal
workloads.
Whether
a
pod
is
a
mortal
workload.
Once
it's
gone,
it's
gone,
we're
a
virtual
machine.
C
We
think
of
that
as
more
persistent
workload
and
a
lot
of
times.
So
there
are
some
h.a
considerations
where
you
things
like:
if
you're
evacuating,
a
node
you
might
want,
might
wanna
live
migrate,
those
resources
away
or
if
it
fails,
you
might
want
to
bring
it
back
up
with
the
exact
same
storage,
similar
to
like
a
stateful
set
yeah.
C
So
the
question
is,
if
you're,
evacuating
or
draining
a
node,
if
we
be
able
to
use
the
live
migration
as
a
way
to
evacuate
that
and
absolutely
we're.
Looking
at
that
now
that
that's
a
important
use
case,
because
it
lets
you,
you
know,
do
hardware
maintenance
without
having
to
actually
kill
stateful
processes.
Yeah.
Thank.
C
Right
the
question
is
with
live:
migration.
Cni
is
when
you
start
a
pod.
It
gets
a
unique
IP
address
and
if
you're
going
to
live
migrate
between
two
pods,
that
means
that
that
IP
address
isn't
following
it.
You
have
two
IP
addresses
at
that
point.
So
when
you
live
migrate,
your
migraine
to
an
environment
has
a
separate
interface.
We
haven't
solved
that
yet
we're
looking
at
other
options
for
networking.
E
A
These
are
simply,
these
are
just
getting
started
like
proof-of-concept
environments.
So,
if
you
go
to
the
Qbert
website,
we
basically
build
out
like
a.m.
eyes
and
images
for
GCP
that
just
lower
the
barrier
to
entry
for
you
to
kick
the
tires
on
this,
we
aren't
suggesting
that
you
actually
run
your
Qbert
infrastructure
on
top
of
this.
In
fact,
it
would
probably
be
leaning
more
towards
the
bare
metal
deployment
scenario.
C
The
question
is
running
virtual
machines
on
top
of
the
public
cloud.
Is
that
well,
the
problem
with
that
is
nested
virtualization,
so,
like
GCP
I,
believe
uses
KVM
and
it
might
let
us
do
nested
virtualization,
which
means
that
you'd
have
the
the
same
performance.
You
can
run
Cubert
emulation
mode
on
AWS
today,
it's
just
it's
just
not
gonna
be
performin.
So
that's
what
we
do.
C
A
C
A
A
G
G
C
Well,
if
you
already
have
this
the
question:
I,
don't
know
how
to
summarize
that
question.
At
this
point,
the
question
was
about
how
to
provision
virtual
machines
legacy
virtual
machines.
I
believe-
and
my
assumption
is
the.
If
somebody
has
a
legacy
virtual
machine,
they
all
right
have
a
system
in
place
for
provisioning
them.
I
I
think
you
would
use
the
existing
system
that
you
have,
or
else
you're
you're
inventing
something
new.
So
if
you
were
using
ansible,
wouldn't
we
just
kick
off
ansible
to
provision.
C
C
So
you
have
to
have
privileges
to
create.
The
question
was:
what
privileges
do
you
need
to
start
a
virtual
machine?
You
have
to
have
the
privileges
to
create
and
modify
virtual
machine
instance
objects
and
that's
it
and
the
controller
will
create
the
virtual
machine
pods
on
the
user's
behalf.
We
think
about
separation
between
users
and
cluster
admins,
though
so
something
like
a
migration
action
would
be
something
that
we're
thinking
might
be
reserved
for.
C
C
A
So
Steve
our
timekeeper
is
said,
cut
it
off.
Thank
you,
everyone
for
joining,
just
as
a
reminder.
There's
a
demo
at
the
booth
tomorrow,
the
Red
Hat
booth
at
8:30
a.m.
and
then
tonight
there's
going
to
be
a
Qbert
gathering
at
6:45
at
the
Sheraton
on
the
second
floor
in
the
Aspen
room.
So
thank
you.
Everybody
for
attending.